Patent Issued for Systems and methods for labeling 3D models using virtual reality and augmented reality (USPTO 11983817): State Farm Mutual Automobile Insurance Company

2024 MAY 31 (NewsRx) -- By a News Reporter-Staff News Editor at Insurance Daily News -- From Alexandria, Virginia, NewsRx journalists report that a patent by the inventors Carnahan, Jeremy (Normal, IL, US), Knuffman, Ryan (Danvers, IL, US), Nussbaum, Bryan (Hershey, PA, US), filed on December 6, 2021, was published online on May 14, 2024.

The patent’s assignee for patent number 11983817 is State Farm Mutual Automobile Insurance Company (Bloomington, Illinois, United States).

News editors obtained the following quote from the background information supplied by the inventors: “Machine learning (ML) and artificial intelligence (AI) are techniques increasingly utilized by computer systems for processing data and carrying out tasks. ML and AI often involve training models which may then be used to process raw data and generate useful outputs. For example, in supervised machine learning, a model may be trained to identify pictures that include an image of a cat. The model may be trained using numerous labeled images, where the labels denote whether or not the image contains a cat. The trained model may then be used to identify if another image contains a cat.

“As ML and AI become more ubiquitous in technological applications, be it big data analysis, robotics, autonomous vehicles, digital personal assistants, or image recognition systems, the amount of data required for training ML and AI systems is increasing. Without a more efficient way to generate training data, the development of ML and AI systems may be stunted. Efficient methods for generating reliable training data, then, are of great importance for promoting the continued advancement of AI and ML technologies.

“Simultaneous to the evolution of ML and AI systems, virtual reality (VR) and augmented reality (AR) systems are quickly developing as new mediums for interacting with a digital environment. VR systems immerse users in a digital environment (e.g., through a VR headset), such that the user is effectively positioned within a “virtual” world. AR systems overlay real-world environments with digital content (e.g., aspects of a digital environment), such that the user experiences the real world “augmented” by digital data. In addition to applications for user enjoyment (e.g., VR video games), VR and AR allow a user to experience and interact with a digital environment in a new way.

“Currently, conventional systems for generating labeled training data for ML and AI face a number of challenges. First, labeling is time intensive for users manually labeling objects and features in 3D models, especially given difficulties in navigating 3D environments in conventional systems. Additionally, difficulties in labeling may lead to increased user error, such as not labeling unnoticed objects, mislabeling objects, or bounding objects in the wrong position or with the wrong size/orientation boundary. Identified objects and labels in 3D models are also difficult to verify against potential real-world counterparts. As such, a system and process for alleviating these difficulties are desired.”

As a supplement to the background information on this patent, NewsRx correspondents also obtained the inventors’ summary information for this patent: “The present embodiments may relate to systems and methods for labeling objects in three dimensional (3D) models using virtual reality (VR) and augmented reality (AR). The system may include a VR computing device, a model processing (MP) computing device, a third party computing device, and a database.

“In one aspect, a computer-implemented method for labeling a three-dimensional (3D) model using virtual reality (VR) techniques may be provided. The method may be implemented by a computer system including at least one processor. The method may include: (i) receiving, by the processor, a 3D model including at least one environmental feature that is unlabeled, (ii) displaying, through a VR device in communication with the processor, a VR environment to a user representing the 3D model, (iii) prompting a user to input labeling data for the at least one environmental feature displayed within the VR environment of the VR device by prompting the user to select the at least one environmental feature through user interaction with the VR device, and input labeling data for the at least one environmental feature, wherein the labeling data identifies the at least one environmental feature, and/or (iv) generating, by the processor, a labeled 3D model by embedding the labeling data associated with the selected at least one environmental feature into the 3D model. The method may include additional, less, or alternate actions, including those discussed elsewhere herein.

“In another aspect, a virtual reality (VR) labeling computer system for labeling a three dimensional (3D) model using VR techniques may be provided. The VR labeling computer system may include at least one processor in communication with at least one memory device, and the at least one processor may be configured to: (i) receive a 3D model including at least one environmental feature that is unlabeled, (ii) display, through a VR device in communication with the processor, a VR environment to a user representing the 3D model, (iii) prompt a user to input labeling data for the at least one environmental feature displayed within the VR environment of the VR device by prompting the user to select the at least one environmental feature through user interaction with the VR device, and input labeling data for the at least one environmental feature, wherein the labeling data identifies the at least one environmental feature, and/or (iv) generate a labeled 3D model by embedding the labeling data associated with the selected at least one environmental feature into the 3D model. The system may include additional, less, or alternate functionality, including that discussed elsewhere herein.

“In yet another aspect, at least one non-transitory computer-readable storage media having computer-executable instructions embodied thereon for labeling a three dimensional (3D) model using a virtual reality (VR) technique may be provided. When executed by at least one processor, the computer-executable instructions may cause the processor to: (i) receive a 3D model including at least one environmental feature that is unlabeled, (ii) display, through a VR device in communication with the processor, a VR environment to a user representing the 3D model, (iii) prompt a user to input labeling data for the at least one environmental feature displayed within the VR environment of the VR device by prompting the user to select the at least one environmental feature through user interaction with the VR device, and input labeling data for the at least one environmental feature, wherein the labeling data identifies the at least one environmental feature, and/or (iv) generate a labeled 3D model by embedding the labeling data associated with the selected at least one environmental feature into the 3D model. The instructions may direct additional, less, or alternate functionality, including that discussed elsewhere herein.

“Advantages will become more apparent to those skilled in the art from the following description of the preferred embodiments which have been shown and described by way of illustration. As will be realized, the present embodiments may be capable of other and different embodiments, and their details are capable of modification in various respects. Accordingly, the drawings and description are to be regarded as illustrative in nature and not as restrictive.”

The claims supplied by the inventors are:

“1. A computer-implemented method for labeling a three-dimensional (3D) model using virtual reality (VR) techniques, the method implemented by a computer system including at least one processor, the method comprising: receiving, by the processor, a 3D model including at least one environmental feature that is unlabeled; displaying, through a VR device in communication with the processor, a VR environment to a user representing the 3D model, the VR environment modifiable by the user; displaying, in the VR environment, a prompt requesting the user to input labeling data for the at least one environmental feature displayed within the VR environment and one or more rules associated with the at least one environmental feature, the labeling data inputted by selecting the at least one environmental feature through gesture icons interacting with the VR environment displayed on the VR device, the labeling data identifying the at least one environmental feature, the one or more rules indicating one or more predicted outcomes associated with the at least one environmental feature; and generating, by the processor, a labeled 3D model by embedding the labeling data associated with the selected at least one environmental feature into the 3D model along with the one or more rules inputted for the selected at least one environmental feature.

“2. The computer-implemented method of claim 1, the method further comprising: extracting the at least one environmental feature and associated labeling data from the labeled 3D model; and generating, by the processor, training data based upon the at least one extracted environmental feature of the labeled 3D model.

“3. The computer-implemented method of claim 2, the method further comprising: training a machine learning model using the training data; and training a machine learning model using the labeled 3D model.

“4. The computer-implemented method of claim 1, wherein the 3D model is a point cloud, and wherein the point cloud is generated based upon aerial photos of a real-world location.

“5. The computer-implemented method of claim 1, wherein the 3D model is processed using object recognition and includes processing the 3D model using a segmentation technique, and wherein the segmentation technique is a semantic segmentation technique.

“6. The computer-implemented method of claim 1, wherein generating the labeled 3D model further comprises updating meta-data of data points representing the at least one environmental feature.

“7. The computer-implemented method of claim 1, wherein generating the labeled 3D model further comprises: generating data points representing the at least one environmental feature; and embedding the data points representing the at least one environmental feature into the 3D model, wherein the data points representing the at least one environmental feature are a surface mesh.

“8. The computer-implemented method of claim 7, wherein displaying the VR environment further comprises altering an appearance of the surface mesh, and wherein generating the labeled 3D model by embedding the labeling data into the 3D model further comprises updating meta-data associated with the surface mesh.

“9. The computer-implemented method of claim 1, wherein displaying the VR environment further comprises altering an appearance of the at least one environmental feature.

“10. The computer-implemented method of claim 9, wherein altering the appearance of the at least one environmental feature includes shading the at least one environmental feature.

“11. The computer-implemented method of claim 9, wherein altering the appearance of the at least one environmental feature includes outlining the at least one environmental feature.

“12. The computer-implemented method of claim 1, wherein the user inputs data into the VR environment using hand gestures made within the VR environment.

“13. The computer-implemented method of claim 1, wherein the user inputs data into the VR environment using eye movement detected by the processor.

“14. The computer-implemented method of claim 1, wherein the user inputs data into the VR environment using spoken commands.

“15. The computer-implemented method of claim 1, wherein generating the labeled 3D model by embedding the labeling data into the 3D model further comprises updating meta-data associated with the 3D model.

“16. The computer-implemented method of claim 1, wherein generating the labeled 3D model by embedding the labeling data into the 3D model further comprises generating a surface mesh, embedding the surface mesh in the 3D model, and updating meta-data associated with the surface mesh.

“17. The computer-implemented method of claim 1, further comprising generating training data, wherein generating the training data further comprises: receiving a second labeled 3D model; aggregating the labeled 3D model and the second labeled 3D model as the training data; capturing a first and second element of the labeled 3D model; and aggregating the first and second elements of the labeled 3D model as the training data.

“18. The computer-implemented method of claim 17, wherein generating the training data includes translating the labeled 3D model from a first file format to a second file format.

“19. A virtual reality (VR) labeling computer system for labeling a three dimensional (3D) model using VR techniques, the VR labeling computer system including at least one processor in communication with at least one memory device, wherein the at least one processor is configured to: receive a 3D model including at least one environmental feature that is unlabeled; display, through a VR device in communication with the processor, a VR environment to a user representing the 3D model, the VR environment modifiable by the user; display, in the VR environment, a prompt requesting the user to input labeling data for the at least one environmental feature displayed within the VR environment and one or more rules associated with the at least one environmental feature, the labeling data inputted by selecting the at least one environmental feature through gesture icons interacting with the VR environment displayed on the VR device, the labeling data identifying the at least one environmental feature, the one or more rules indicating one or more predicted outcomes associated with the at least one environmental feature; and generate a labeled 3D model by embedding the labeling data associated with the selected at least one environmental feature into the 3D model along with the one or more rules inputted for the selected at least one environmental feature.

“20. At least one non-transitory computer-readable storage medium having computer-executable instructions embodied thereon for labeling a three dimensional (3D) model using a virtual reality (VR) technique, wherein when executed by at least one processor, the computer-executable instructions cause the at least one processor to: receive a 3D model including at least one environmental feature that is unlabeled; display, through a VR device in communication with the processor, a VR environment to a user representing the 3D model, the VR environment modifiable by the user; display, in the VR environment, a prompt requesting the user to input labeling data for the at least one environmental feature displayed within the VR environment and one or more rules associated with the at least one environmental feature, the labeling data inputted by selecting the at least one environmental feature through gesture icons interacting with the VR environment displayed on the VR device, the labeling data identifying the at least one environmental feature, the one or more rules indicating one or more predicted outcomes associated with the at least one environmental feature; and generate a labeled 3D model by embedding the labeling data associated with the selected at least one environmental feature into the 3D model along with the one or more rules inputted for the selected at least one environmental feature.”

For additional information on this patent, see: Carnahan, Jeremy. Systems and methods for labeling 3D models using virtual reality and augmented reality. U.S. Patent Number 11983817, filed December 6, 2021, and published online on May 14, 2024. Patent URL (for desktop use only): https://ppubs.uspto.gov/pubwebapp/external.html?q=(11983817)&db=USPAT&type=ids

(Our reports deliver fact-based news of research and discoveries from around the world.)