Patent Issued for System, method, apparatus, and computer program product for utilizing machine learning to process an image of a mobile device to determine a mobile device integrity status (USPTO 11704887): Assurant Inc.

2023 AUG 07 (NewsRx) -- By a News Reporter-Staff News Editor at Insurance Daily News -- Assurant Inc. (New York, New York, United States) has been issued patent number 11704887, according to news reporting originating out of Alexandria, Virginia, by NewsRx editors.

The patent’s inventors are Breitsch, Nathan (New York, NY, US), Coble, Anthony (New York, NY, US), Ionescu, Mircea (Wooster, OH, US), Saunders, Stuart (Lakewood, OH, US).

This patent was filed on September 16, 2020 and was published online on July 18, 2023.

From the background information supplied by the inventors, news correspondents obtained the following quote: “Computer vision enables computers to see and understand an image. In some instances, computer vision may be used to detect and analyze the content of an image, such as recognizing an object within an image. However, existing technology is inadequate to meet the speed and precision requirements of many industries, and there is a need for improvement in computer vision techniques and technology to enable sophisticated image processing. Moreover, human analysis is incapable of the speed and precision required for computer vision tasks. Through applied effort, ingenuity, and innovation, many of these identified problems have been solved by developing solutions that are included in embodiments of the present invention, many examples of which are described in detail herein.”

Supplementing the background information on this patent, NewsRx reporters also obtained the inventors’ summary information for this patent: “BRIEF SUMMARY OF EXAMPLE EMBODIMENTS

“Systems, methods, apparatuses and computer program products are therefore provided for utilizing machine learning to train and utilize a model to determine a mobile device integrity status based on electronic processing of images.

“In some use cases, a system must review images of an object to verify the integrity of the object (e.g., to determine information about the object, to verify the operability or functionality of the object, to verify the identity of the object, or the like). The computer vision and image processing must occur rapidly and with a high-degree of precision, which is lacking in many conventional image processing techniques. A further challenge may be when the system cannot select the imaging device that captures the image and cannot control the image capture process directly, and thus, the computer vision must be sufficiently robust to account for and/or detect issues with the capture process. In an example working environment, a system may seek to verify the identity and integrity of an object using only an image of the object (e.g., a mobile device) or using an image in combination with one or more data objects transmitted from the object or from another device. An example of such an environment may be when a user registers for a service, protection plan, or the like, which requires a remote system to verify the object (e.g., a mobile device) without having the device physically present. According to some processes for purchasing aftermarket coverage, a consumer must visit a retailer, insurance provider, or mobile device service provider to have the device inspected and to verify the integrity of the device before the insurer will issue the policy and enroll the device for coverage. Other processes for purchasing and/or selling coverage allow a consumer to utilize a self-service web application or mobile application to take photographs of their mobile device and submit the images for manual review prior to enrollment. However, such processes require review time and may delay the confirmation of coverage to the consumer. Such processes may further expose the provider to fraud, such as when the consumer submits a photo of a different, undamaged mobile device and tries to obtain coverage for a previously damaged device.

“An additional implementation provides a time-sensitive bar code, quick response (QR) code, or other computer-generated code to be displayed by the device, and captured in a photo using a mirror, thereby linking the photo submission to the device that displayed the code. However, such implementations may be susceptible to fraud, such as by enabling a user to recreate the code on another undamaged device and to capture a photo of the undamaged device. Still further, the code implementation may only provide for validation of the front (e.g., display side) of the device without reliably verifying the condition or status of the rear of the device and/or bezel of the device. Another drawback of such implementations is that when a code is displayed on a device display, it may obscure cracks or other damages present on the screen.

“Example embodiments of the present disclosure provide for improved determination of mobile device integrity status. Example embodiments may prompt a user to capture images of their device in a mirror or other reflective surface using a sensor or camera of the device itself. Identifying information of the mobile device may be processed along with the images to confirm the images were indeed taken of the subject device from which the images were captured, and to confirm the device has no pre-existing damage that disqualifies the device from coverage.”

The claims supplied by the inventors are:

“1. A method comprising: receiving a device integrity verification request associated with a mobile device; receiving mobile device identifying data objects comprising information describing the mobile device; causing display on the mobile device of a prompt to capture at least one image of the mobile device using one or more image sensors of the mobile device and a reflective surface; receiving the at least one image captured by the one or more image sensors mobile device; with at least one trained model, processing the at least one image to determine a mobile device integrity status by determining whether there is at least one of: one or more concave occlusions in the at least one image, or one or more corners blocked in the at least one image, by performing: generating a mobile device mask comprising a reduced number of colors relative to the at least one image; extracting a polygonal subregion P of the mobile device mask; determining a convex hull of P; and utilizing the convex hull to determine whether there is at least one of (a) one or more concave occlusions in the at least one image, or (b) one or more corners blocked in the at least one image; and in an instance it is determined there is at least one of one or more concave occlusions in the at least one image, or one or more corners blocked in the at least one image, causing display on the mobile device of a prompt to capture images without occlusions.

“2. The method of claim 1, wherein processing the at least one image to determine mobile device integrity status comprises: utilizing the at least one trained model to determine whether there is damage to the mobile device; and in response to determining there is damage to the mobile device, determining a mobile device integrity status as not verified.

“3. The method of claim 1, wherein processing the at least one image to determine mobile device integrity status comprises: determining an angle of the mobile device relative to the reflective surface when the at least one image was captured; and determining, based on the angle, that the at least one images includes a different mobile device than the mobile device associated with the mobile device identifying data object.

“4. The method of claim 1, wherein processing the at least one image to determine a mobile device integrity status comprises: determining whether the at least one image includes the mobile device associated with the mobile device identifying data object.

“5. The method of claim 4, wherein determining whether the at least one image includes the mobile device comprises: identifying a suspected mobile device in the at least one image; generating a prediction of an identity of the at least one suspected mobile device, and comparing the mobile device identifying data objects to the prediction of the identity of the at least one suspected mobile device to determine whether the suspected mobile device is the mobile device, and in an instance in which the suspected mobile device is determined to be the mobile device, determining a mobile device integrity status as verified.

“6. The method of claim 1, wherein the mobile device integrity status is determined as inconclusive, and the method further comprises: transmitting the device integrity verification request and the at least one image to an internal user apparatus for internal review.

“7. The method of claim 3, further comprising: in response to determining, based on the angle, that the at least one images captures a different mobile device, (a) causing display on the mobile device of a message instructing the user to recapture the mobile device; and (b) determining that the mobile device integrity status is not verified.

“8. The method of claim 1, wherein processing the at least one image to determine mobile device integrity status comprises: determining a location within the at least one image of the mobile device, wherein the location is defined as a bounding box; and in an instance the bounding box has a first predefined relationship with a threshold ratio of the at least one image, causing display on the mobile device of a message indicating to move the mobile device closer to the reflective surface.

“9. The method of claim 8, further comprising: in an instance the bounding box has a second predefined relationship with the threshold ratio of the at least one image, cropping the at least one image according to the bounding box.

“10. The method of claim 1, wherein processing the at least one image to determine a mobile device integrity status comprises: determining with the at least one trained model, whether the at least one image includes a front of the mobile device, a back of the mobile device, or a cover.

“11. The method of claim 1, further comprising: in response to receiving the at least one image, providing in real-time or near real-time, a response for display on the mobile device, wherein the response provided is dependent on the determined mobile device integrity status.

“12. The method of claim 1, further comprising: causing display on the mobile device of a test pattern configured to provide improved accuracy in predicting a characteristic of the at least one image captured when the mobile device displays the test pattern, relative to an accuracy in predicting the characteristic of the at least one image captured when the mobile device displays another pattern of display.

“13. The method of claim 1, further comprising: identifying a subset of conditions to be satisfied in order to determine a mobile device integrity status as verified; in an instance all the conditions in the subset of conditions are satisfied in a particular image, setting an image status of the particular image to verified; and in an instance respective image statuses for all required images are verified, determining the mobile device integrity status as verified.

“14. The method of claim 13, wherein at least one condition of the subset of conditions to be satisfied is performed on the mobile device.

“15. The method of claim 1, wherein receiving the at least one image comprises receiving at least two images captured by the mobile device, wherein a first image of the at least two images is of a front side of the device, and a second image of the at least two images is of the rear side of the device, and wherein processing the at least one image to determine a mobile device integrity status comprises; with the at least one trained model, processing both the first image and the second image; in an instance the processing of both images results in respective image statuses of verified, determining the determine mobile device integrity status as verified.

“16. The method of claim 1, further comprising: training the at least one trained model by inputting training images and respective labels describing a characteristic of the respective training image.

“17. An apparatus comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code configured to, with the processor, cause the apparatus to at least: receive a device integrity verification request associated with a mobile device; receive mobile device identifying data objects comprising information describing the mobile device; cause display on the mobile device of a prompt to capture at least one image of the mobile device using one or more image sensors of the mobile device and a reflective surface; receive the at least one image captured by the one or more image sensors mobile device; with at least one trained model, process the at least one image to determine a mobile device integrity status, by determining whether there is at least one of: one or more concave occlusions in the at least one image, or one or more corners blocked in the at least one image, by performing: generating a mobile device mask comprising a reduced number of colors relative to the at least one image; extracting a polygonal subregion P of the mobile device mask; determining a convex hull of P; and utilizing the convex hull to determine whether there is at least one of (a) one or more concave occlusions in the at least one image, or (b) one or more corners blocked in the at least one image; and in an instance it is determined there is at least one of one or more concave occlusions in the at least one image, or one or more corners blocked in the at least one image, cause display on the mobile device of a prompt to capture images without occlusions.

“18. The apparatus of claim 17, wherein processing the at least one image to determine mobile device integrity status comprises: utilizing the at least one trained model to determine whether there is damage to the mobile device; and in response to determining there is damage to the mobile device, determining a mobile device integrity status as not verified.

“19. The apparatus of claim 17, wherein processing the at least one image to determine mobile device integrity status comprises: determining an angle of the mobile device relative to the reflective surface when the at least one image was captured; and determining, based on the angle, that the at least one images includes a different mobile device than the mobile device associated with the mobile device identifying data object.

“20. The apparatus of claim 17, wherein processing the at least one image to determine a mobile device integrity status comprises: determining whether the at least one image includes the mobile device associated with the mobile device identifying data object.”

There are additional claims. Please visit full patent to read further.

For the URL and additional information on this patent, see: Breitsch, Nathan. System, method, apparatus, and computer program product for utilizing machine learning to process an image of a mobile device to determine a mobile device integrity status. U.S. Patent Number 11704887, filed September 16, 2020, and published online on July 18, 2023. Patent URL (for desktop use only): https://ppubs.uspto.gov/pubwebapp/external.html?q=(11704887)&db=USPAT&type=ids

(Our reports deliver fact-based news of research and discoveries from around the world.)