Patent Issued for Augmented reality system for remote product inspection (USPTO 11893836): Massachusetts Mutual Life Insurance Company

2024 FEB 27 (NewsRx) -- By a News Reporter-Staff News Editor at Insurance Daily News -- Massachusetts Mutual Life Insurance Company (Springfield, Massachusetts, United States) has been issued patent number 11893836, according to news reporting originating out of Alexandria, Virginia, by NewsRx editors.

The patent’s inventors are DePaolo, Damon (Barkhamsted, CT, US), John, Jiby (Suffield, CT, US), Knas, Michal (Monson, MA, US), Shubrick, Payton (Springfield, MA, US).

This patent was filed on January 13, 2023 and was published online on February 6, 2024.

From the background information supplied by the inventors, news correspondents obtained the following quote: “Various product regulations require that each product (for example, a vehicle) comply with certain record keeping procedures, which may include maintenance of vehicle inspection reports and vehicle service information. To comply with these regulations, a driver of a vehicle must frequently inspect their vehicle for periodic scheduled maintenance and/or keep a record of repair and service information of the vehicle based on the inspection. Current inspection methods generally involve the inspection of the vehicle at a vehicle service center, which requires the vehicle to be physically present at the vehicle service center. Such an in-center inspection process typically takes several hours to complete. Accordingly, the in-center inspection process of taking the vehicle to the vehicle service center for the inspection is quite burdensome and time consuming for the user.

“Transmitting images of the vehicle to be inspected to the vehicle service center has proven to be a less than desirable solution to the above-described problem. First, when receiving images of the vehicle from the user, a receiving server of the vehicle service center cannot identify whether the images are indeed captured from the vehicle, which has to be inspected. Therefore, authenticity of the images of the vehicle is not readily apparent, which is undesirable. Second, the vehicle service center uses a human-driven process where an administrator validates the images of the vehicle. Such validation process is heavily reliant upon the administrator’s subjective understanding and skills, which is also undesirable.”

Supplementing the background information on this patent, NewsRx reporters also obtained the inventors’ summary information for this patent: “For the aforementioned reasons, what is therefore needed is a product inspection apparatus that eliminates a need for an in-center inspection of a product, such as a vehicle. What is also needed is an efficient computer-implemented product inspection system, which is configured to validate an authenticity of images of the vehicle and identify whether the images visually pass an inspection in real time.

“The product inspection apparatus may be configured to operate a visual inspection software application, which may enable a remote inspection of every element of the vehicle using augmented reality for all periodic, scheduled inspections of the vehicle. The features of the product inspection apparatus and the visual inspection software application may be applicable for remote inspection of various products other than the vehicles, such as home appliances.

“The product inspection apparatus may include an electronic device, which may execute the visual inspection software application. The visual inspection software application may activate a camera of the electronic device to capture a live video or images of the vehicle. The visual inspection software application may activate the camera when the electronic device is within a predetermined distance from the vehicle and at a particular field of view with respect to the vehicle. A display screen of the electronic device may present the video or the images of the vehicle captured by the camera. The video or the images may be digitally processed, and augmented by the addition of computer-generated images associated with a status of inspection of each element (for example, a rear view and/or a side view mirror) of the vehicle. Each computer-generated image may include a graphical indicator associated with the status of inspection of a particular element. Each computer-generated image may be projected on top of (e.g., as an additional layer to) a real world image of the particular element presented on the display screen.

“The electronic device may select the graphical indicator based on a result of inspection of the particular element. For instance, the electronic device may select a check mark (I) as the graphical indicator when a captured image of a first element of the vehicle matches with a reference image of the first element. The computer-generated image of the check mark is super-imposed as an overlay image on the captured image of the first element presented on the display screen. The computer-generated image of the check mark can also be placed adjacent to the captured image of the first element presented on the display screen. In another instance, the electronic device may select a cross mark (X) as the graphical indicator when the captured image of the first element of the vehicle does not match with the reference image of the first element. The computer-generated image of the cross mark is super-imposed as the overlay image on the captured image of the first element presented on the display screen. The computer-generated image of the cross mark can also be placed adjacent to the captured image of the first element presented on the display screen. When captured images of all elements are checked against reference images of all elements, the electronic device may generate and present a graphical component on the display screen. The graphical component may correspond to a web link to download an inspection report of the vehicle.

“In one embodiment, a method comprises establishing, by a server associated with an electronic device, an electronic communication session with a processor of a vehicle, wherein the server receives a unique identifier of the vehicle during the electronic communication session, and wherein the server monitors a distance between the electronic device and the vehicle; retrieving, by the server from a database, a multi-dimensional wireframe of the vehicle based on the unique identifier; determining, by the server, a position and a direction of an electronic device relative to the vehicle using a direction sensor of the electronic device when a user operating the electronic device scans the vehicle using a camera; displaying, by the server on the electronic device, a layout of a first element of the vehicle, and wherein the server prompts the user to align the first element displayed on the electronic device with a corresponding first element of the vehicle from a particular position and direction relative to the vehicle; when the corresponding first element of the vehicle is aligned with the first element displayed on the electronic device, causing, by the server, the camera to capture a first image of the vehicle comprising the first element; retrieving, by the server, a set of reference images associated with the vehicle based on the unique identifier; and hen the first element in the first image matches a corresponding first element in the set of reference images, displaying, by the server on the electronic device, a composite image not captured by the camera, wherein the composite image is overlaid on the first image, the composite image comprising a notification corresponding to a status of an inspection of the first element.”

The claims supplied by the inventors are:

“1. A method comprising: determining, by a server, a position and a direction of an electronic device in relation to a vehicle using a direction sensor of the electronic device when a user operating the electronic device scans the vehicle using a camera of the electronic device; displaying, by the server on the electronic device, a layout of a first element of the vehicle, wherein the server prompts the user to align the first element displayed on the electronic device with a corresponding first element of the vehicle by adjusting the position and direction of the electronic device relative to the vehicle; when the corresponding first element of the vehicle is aligned with the first element displayed on the electronic device, causing, by the server, the camera to capture a video comprising a set of images of the vehicle; retrieving, by the server, a set of reference images associated with the vehicle; responsive to the first element in at least one image within the set of images matching the corresponding first element in the set of reference images, displaying, by the server on the electronic device, a composite image overlaid on the first at least one image comprising a first graphical component corresponding to a status of an inspection of the first element; and displaying, by the server, a second graphical component corresponding to a successful completion of the inspection when a predetermined number of elements within a set of captured images of the vehicle matches a corresponding element within the set of reference images.

“2. The method of claim 1, wherein the layout is displayed using an augmented reality protocol.

“3. The method of claim 1, further comprising: establishing, by the server, an electronic communication session with a processor of the vehicle, wherein the server receives a unique identifier of the vehicle during the electronic communication session used to identify the vehicle.

“4. The method of claim 1, further comprising: monitoring, by the server, a distance between the electronic device and the vehicle.

“5. The method of claim 1, wherein the first graphical component corresponds to a vehicle inspection report.

“6. The method of claim 1, further comprising: updating, by the server, a data record corresponding to the first element in the at least one image regarding the status of the inspection corresponding to the first element.

“7. The method of claim 1, further comprising: generating, by the server, an inspection report comprising a list of elements that passed the inspection.

“8. The method of claim 1, wherein the first graphical component is represented as an animated character.

“9. The method of claim 1, wherein the second graphical component includes a web link to download a vehicle inspection report.

“10. A system comprising: a server in communication with an electronic device, the server comprising a processor and a non-transitory computer-readable medium containing instructions that when executed by the processor causes the processor to perform operations comprising: determine a position and a direction of the electronic device in relation to a vehicle using a direction sensor of the electronic device when a user operating the electronic device scans the vehicle using a camera of the electronic device; display, on the electronic device, a layout of a first element of the vehicle, wherein the server prompts the user to align the first element displayed on the electronic device with a corresponding first element of the vehicle by adjusting the position and direction of the electronic device relative to the vehicle; when the corresponding first element of the vehicle is aligned with the first element displayed on the electronic device, cause the camera to capture a video comprising a set of images of the vehicle; retrieve a set of reference images associated with the vehicle; responsive to the first element in at least one image within the set of images matching the corresponding first element in the set of reference images, display, on the electronic device, a composite image overlaid on the first at least one image comprising a first graphical component corresponding to a status of an inspection of the first element; and displaying, by the server, a second graphical component corresponding to a successful completion of the inspection when a predetermined number of elements within a set of captured images of the vehicle matches a corresponding element within the set of reference images.

“11. The system of claim 10, wherein the layout is displayed using an augmented reality protocol.

“12. The system of claim 10, wherein the instructions cause the processor to: establish an electronic communication session with a processor of the vehicle, wherein the server receives a unique identifier of the vehicle during the electronic communication session used to identify the vehicle.

“13. The system of claim 10, wherein the instructions cause the processor to: monitor a distance between the electronic device and the vehicle.

“14. The system of claim 10, wherein the first graphical component corresponds to a vehicle inspection report.

“15. The system of claim 10, wherein the instructions cause the processor to: update a data record corresponding to the first element in the first at least one image regarding the status of the inspection corresponding to the first element.

“16. The system of claim 10, wherein the instructions cause the processor to: generate an inspection report comprising a list of elements that passed the inspection.

“17. The system of claim 10, wherein the first graphical component is represented as an animated character.

“18. The system of claim 10, wherein the second graphical component includes a web link to download a vehicle inspection report.

“19. A system comprising a server configured to: determine a position and a direction of an electronic device in relation to a vehicle using a direction sensor of the electronic device when a user operating the electronic device scans the vehicle using a camera of the electronic device; display, on the electronic device, a layout of a first element of the vehicle, wherein the server prompts the user to align the first element displayed on the electronic device with a corresponding first element of the vehicle by adjusting the position and direction of the electronic device relative to the vehicle; when the corresponding first element of the vehicle is aligned with the first element displayed on the electronic device, cause the camera to capture a video comprising a set of images of the vehicle; retrieve a set of reference images associated with the vehicle; responsive to the first element in at least one image within the set of images matching the corresponding first element in the set of reference images, display, on the electronic device, a composite image overlaid on the first at least one image comprising a first graphical component corresponding to a status of an inspection of the first element; and displaying, by the server, a second graphical component corresponding to a successful completion of the inspection when a predetermined number of elements within a set of captured images of the vehicle matches a corresponding element within the set of reference images.

“20. The system of claim 19, wherein the layout is displayed using an augmented reality protocol.”

For the URL and additional information on this patent, see: DePaolo, Damon. Augmented reality system for remote product inspection. U.S. Patent Number 11893836, filed January 13, 2023, and published online on February 6, 2024. Patent URL (for desktop use only): https://ppubs.uspto.gov/pubwebapp/external.html?q=(11893836)&db=USPAT&type=ids

(Our reports deliver fact-based news of research and discoveries from around the world.)