“Virtualized Computing Platform For Inferencing, Advanced Processing, And Machine Learning Applications” in Patent Application Approval Process (USPTO 20200027210)

2020 FEB 07 (NewsRx) -- By a News Reporter-Staff News Editor at Hospital & Nursing Home Daily -- A patent application by the inventors Haemel, Nicholas (San Francisco, CA); Vukojevic, Bojan (Pleasanton, CA); Haukioja, Risto (Palo Alto, CA); Feng, Andrew (Cupertino, CA); Cheng, Yan (Great Falls, VA); Alle, Sachidanand (Cambridge, GB); Xu, Daguang (Potomac, MD); Roth, Holger Reinhard (Rockville, MD); Israeli, Johnny (San Jose, CA), filed on July 18, 2019, was made available online on January 23, 2020, according to news reporting originating from Washington, D.C., by NewsRx correspondents.

This patent application is assigned to NVIDIA Corporation (San Jose, California, United States).

The following quote was obtained by the news editors from the background information supplied by the inventors: “Throughout recent history, imaging operations have been integral to research and diagnostics in a wide variety of industries, healthcare and medical research being among them. Medical imaging devices--such as computed tomography (CT) scan machines, positron emission technology (PET) scan machines, nuclear imaging machines, X-ray machines, ultrasound machines, and magnetic resonance imaging (MRI) machines--are widely used to aid medical professions and data scientists in visualizing a subject’s anatomy for identifying anomalies, determining diagnoses, and researching organ function and disease. Modern, well-equipped hospitals and labs may have any number of these medical imaging devices on-premises. As innovations in computing--and especially graphics processing--have advanced, graphics processing units (GPUs) have increasingly been used in medical imaging devices to improve their performance and functionality.

“Despite these advances in technology, a healthcare or medical practitioner is still required to perform accurate diagnoses--often relying on each practitioner’s perception, understanding, and specific experience. In some instances, machine learning--e.g., using deep neural networks (DNNs)--has been used to capture these perceptions of on-premises practitioners to perform classification, segmentation, and object detection tasks in the medical imaging field. However, building, training, deploying, and executing machine learning to perform these tasks is exceedingly complex and difficult, typically requiring extensive and costly upgrades to the computing infrastructure. As a result, it may be prohibitively expensive or time-consuming to deploy on hospital premises. In addition, because various machine learning tasks--e.g., classification, segmentation, reconstruction, etc.--may each be a responsibility of a different team at an individual hospital, collaboration and integration between the various tasks may increase the complexity of deployment on-premises.

“Moreover, computing capabilities for medical imaging devices are often capped years before the devices are available for shipment. For this reason, the technology being used at hospitals is often at least a few years outdated, creating a latency or gap between cutting edge medical imaging technology that is actually available and the current install base of medical imaging devices in clinics and hospitals. This latency or gap carries over to the machine learning capabilities of a hospital, as the programming and training of the machine learning models is created and deployed for use with already outdated medical imaging devices. As a result, as more accurate imaging techniques and devices are developed, in addition to more optimized and informative machine learning models, hospitals are constrained to the technology of their current medical imaging devices--thereby reducing the ability of the hospital to provide the most accurate and informative diagnoses and treatment of patients.”

In addition to the background information obtained for this patent application, NewsRx journalists also obtained the inventors’ summary information for this patent application: “Embodiments of the present disclosure relate to a virtualized computing platform for inferencing, advanced processing, and machine learning. Systems and methods are disclosed that allow for customized inference or processing pipelines by selecting, organizing, and/or adapting container hosted applications for local, on-premise implementations. In embodiments, machine learning models that may be trained, updated, and/or generated at a first facility may be leveraged and updated for location specific implementation to perform image processing and/or inferencing operations at a second, different facility.

“In contrast to conventional systems, such as those described above, the system of the present disclosure accelerates inferencing, imaging operations, and informatics while minimizing the complexity of on-premise compute infrastructure requirements. For example, a virtual computing platform that continually adapts to new advancements in technology may be used to improve patient care through advanced diagnostics and research. The system allows for selection, organization, and deployment of containers--hosting instantiations of applications--in inference and/or image deployment pipelines. The pipelines may be configured for receiving (for example, medical) imaging data, processing the data, and outputting meaningful and informative results to practitioners. As such, because the pipelines may be dynamically customizable, outputs of imaging devices, radiology devices, gene sequencing devices, genomics devices, and or processing devices may be used by updated, state-of-the-art technology within the virtualized platform to provide accurate and efficient results while reducing the burden of deployment on-premise as compared to conventional approaches.

“In addition, within various containers, machine learning models may be deployed for image inferencing and training. In contrast to conventional systems, the machine learning models that are deployed may be selected from a remote database of existing machine learning models. As these existing machine learning models are deployed and updated at various locations, a crowd sourcing approach may be used to generate more universal machine learning models for simplifying and expediting deployment across locations. By training and/or updating the machine learning models on-premise, at various locations, the confidentiality of patient records--in view of state and federal laws and regulations (e.g., the Health Insurance Portability and Accountability Act (HIPAA)) for the handling and use of patient information--may be maintained while more robust and accurate machine learning models may be generated. Further, by providing existing machine learning models, the compute resources, expense, and time required for local, on-premise creation, training, and deployment of machine learning models is drastically reduced, and the resulting models may be better optimized for their respective applications.”

The claims supplied by the inventors are:

“1. A method for processing data from a device in a distributed processing system, the method comprising: receiving a selection of one or more applications to perform processing requests; instantiating a deployment pipeline to perform the processing requests, the deployment pipeline including one or more containers comprising executable instantiations of the one or more applications from the selection; determining, for at least one container of the one or more containers, one or more services to perform one or more operations for an executable instantiation of an application comprised by the at least one container; receiving a processing request corresponding to data generated by the device; receiving the data; and processing the data according to the deployment pipeline and using the one or more services to generate processed data.

“2. The method of claim 1, wherein the deployment pipeline includes receiving the data from an on-premise computing system, preparing the data for each of the one or more containers, applying the data to each of the one or more containers, and preparing the processed data from the deployment pipeline for use at the on-premise computing system.

“3. The method of claim 1, wherein: the data is generated by the device at a first location; the selection is from a computing system at the first location; and the receiving the processing request, the receiving the data, and the processing the data are performed at one or more second locations remote from the first location.

“4. The method of claim 3, wherein at least one of: the one or more second locations hosts a cloud computing platform; or the one or more second locations includes a datacenter.

“5. The method of claim 3, wherein at least a portion of the data is initially processed at an edge device embedded with the device prior to the receiving the data at the one or more second locations.

“6. The method of claim 1, further comprising: implementing the deployment pipeline within a computing system on-premise with respect to the device, wherein at least one of the one or more services is hosted remotely from the computing system.

“7. The method of claim 1, wherein at least one of the one or more containers leverages an image processing algorithm or a machine learning model for performing inference on the data.

“8. The method of claim 1, wherein each of the one or more applications are stored as an image file, and the instantiation of the application executed by the container is generated from the image file.

“9. The method of claim 1, wherein the data comprise imaging data, the method further comprising: for each container, configuring the container for use with the device and a computing system co-located with the device, the configuring including: determining a data format for the data; and based at least in part on the data format, determining one or more pre-processing operations for the data to prepare the data for use by the container.

“10. The method of claim 9, wherein the data format is based on at least one of an imaging modality of the device or a data type generated by the computing system.

“11. The method of claim 1, wherein the data include at least one of digital imaging and communications in medicine (DICOM) data, remote procedure call (RPC) data, data substantially compliant with a REST interface, data substantially compliant with a file-based interface, raw data, or radiological information system (RIS) data.

“12. The method of claim 1, further comprising generating a visualization based at least in part on the processed results, and transmitting display data corresponding to the visualization to a computing system for display by the computing system.

“13. The method of claim 12, wherein the computing system comprises at least one of an augmented reality (AR) system or a virtual reality (VR) system.

“14. A system comprising: a virtual instrument including an advanced processing pipeline selectively customized for processing imaging data generated for a device, the advanced processing pipeline to execute a discrete instantiation of an application for performing one or more operations on the data; a service manager to route processing requests to services, the processing requests received from the virtual instrument to cause one or more of the services to perform one or more operations for the instantiation of the application; and an data communication interface to receive the data from a computing system communicatively coupled to the device, input the data to the data deployment pipeline, receive processed data as an output of the data deployment pipeline, and transmit the processed imaging data to the computing system.

“15. The system of claim 14, wherein the advanced processing pipeline is customized by: receiving a selection of at least the application from a pool of applications for processing the data; and configuring the advanced processing pipeline to include the application and at least one other processing operation separate from the application in a sequence of processing operations.

“16. The system of claim 15, wherein the at least one other processing operation includes a pre-processing operation for preparing the data for the application.

“17. The system of claim 14, wherein the services include an inference service, and the inference service performs deep learning inferencing for the application using artificial intelligence.

“18. The system of claim 17, wherein the artificial intelligence includes at least one of a computer vision algorithm, an object detection algorithm, an image reconstruction algorithm, a gene sequencing algorithm, or a neural network.

“19. The system of claim 17, wherein the artificial intelligence is used to perform one or more tasks including an object detection task, a feature detection task, a segmentation task, a reconstruction task, a calibration task, or an image enhancement task.

“20. The system of claim 14, wherein the services include a visualization service, and the visualization service generates a visualization for displaying the processed data.

“21. The system of claim 20, wherein the visualization is transmitted to a display device of the computing system or another display device of a remote computing system.

“22. The system of claim 21, wherein the computing system comprises at least one of: an augmented reality (AR) system; or a virtual reality (VR) system.

“23. A method comprising: receiving an input corresponding to a selection of a neural network from a model registry, the neural network trained using first data generated at least one first facility; based at least in part on the selection, receiving the neural network at a second facility remote from the at least one first facility; receiving second data generated by one or more imaging devices at the second facility; generating ground truth data corresponding to the second data; retraining the neural network using the second data and the ground truth data to generate an updated neural network; and deploying the updated neural network for use with a containerized instantiation of an application of an data deployment pipeline, the containerized instantiation of the application performing one or more processing tasks on third data generated by at least one of the one or more imaging devices or other imaging devices at the second facility.

“24. The method of claim 23, wherein the at least one first facility is a first medical facility and the second facility is a second medical facility, the first medical facility and the second medical facility each including one of a hospital, a clinic, a diagnostic lab, or a research lab.

“25. The method of claim 23, wherein the model registry includes machine learning models trained to perform at least one processing task with respect to digital imaging and communications in medicine (DICOM) data, radiology information system (RIS) data, clinical information system (CIS) data, remote procedure call (RPC) data, fourth data substantially compliant with a representation state transfer (REST) interface, fifth data substantially compliant with a file-based interface, or raw data.

“26. The method of claim 23, wherein the generating the ground truth data includes: applying the second data to a neural network trained to generate annotation data; and editing the annotation data to generate updated annotation data, wherein the ground truth data includes the updated annotation data.

“27. The method of claim 23, wherein the updated neural network is deployed as part of a service of a imaging platform, and the containerized instantiation of the application calls on the service during execution of the data deployment pipeline to perform inference using the updated neural network.

“28. The method of claim 23, wherein the retraining the neural network includes applying the second data to the neural network and using one or more loss functions to update parameters of the neural network based at least in part on a comparison of predictions of the neural network to the ground truth data.”

URL and more information on this patent application, see: Haemel, Nicholas; Vukojevic, Bojan; Haukioja, Risto; Feng, Andrew; Cheng, Yan; Alle, Sachidanand; Xu, Daguang; Roth, Holger Reinhard; Israeli, Johnny. Virtualized Computing Platform For Inferencing, Advanced Processing, And Machine Learning Applications. Filed July 18, 2019 and posted January 23, 2020. Patent URL: http://appft.uspto.gov/netacgi/nph-Parser?Sect1=PTO1&Sect2=HITOFF&d=PG01&p=1&u=%2Fnetahtml%2FPTO%2Fsrchnum.html&r=1&f=G&l=50&s1=%2220200027210%22.PGNR.&OS=DN/20200027210&RS=DN/20200027210

(Our reports deliver fact-based news of research and discoveries from around the world.)