Patent Issued for Biometric Characteristic Application Using Audio/Video Analysis (USPTO 10,825,564)

2020 NOV 16 (NewsRx) -- By a News Reporter-Staff News Editor at Insurance Daily News -- A patent by the inventors Zhang, Dingchao (Normal, IL); Bernico, Michael (Bloomington, IL); Laube, Peter (Bloomington, IL); Pamuksuz, Utku (Champaign, IL); Myers, Jeffrey S. (Normal, IL); Bokshi-Drotar, Marigona (McKinney, TX); Breitweiser, Edward W. (Bloomington, IL), filed on December 11, 2017, was published online on November 16, 2020, according to news reporting originating from Alexandria, Virginia, by NewsRx correspondents.

Patent number 10,825,564 is assigned to State Farm Mutual Automobile Insurance Company (Bloomington, Illinois, United States).

The following quote was obtained by the news editors from the background information supplied by the inventors: “Today, a user’s health status may be determined based on several biometric characteristics, such as the user’s age, gender, blood pressure, heart rate, body mass index (BMI), body temperature, stress levels, smoking status, etc. These biometric characteristics are typically obtained through self-reporting from the user (e.g., by filling out a form indicating the user’s gender, birth date, etc.) and/or medical examinations that include taking measurements conducted by various instruments, such as a thermometer, scale, heart rate monitor, blood pressure cuff, etc.

“This process of filling out forms and taking measurements with several different instruments may be difficult and time consuming for the user. Users may also withhold information or report incorrect information which may lead to inaccuracies in the health status assessment (e.g., from errors in self-reporting or uncalibrated instruments).”

In addition to the background information obtained for this patent, NewsRx journalists also obtained the inventors’ summary information for this patent: “To efficiently and accurately predict a user’s health status and corresponding longevity metric, a biometric characteristic system may be trained using various machine learning techniques to create predictive models for determining biometric characteristics of the user based on video of the user. The determined or predicted biometric characteristics may be combined to generate an overall indication of the user’s health which may be used to generate a longevity metric for the user. The biometric characteristic system may be trained by obtaining audiovisual data (e.g., videos or images) of several people having known biometric characteristics at the time the audiovisual data is captured (e.g., age, gender, BMI, etc.). The people may be referred to herein as ‘training subjects.’ For example, the training data may include public audiovisual data such as movies, television, music videos, etc., featuring famous actors or actresses having biometric characteristics which are known or which are easily obtainable through public content (e.g., via Internet Movie Database (IMDb.RTM.), Wikipedia.TM., etc.).

“In some embodiments, the training data may include feature data extracted from the audiovisual data using computer vision techniques and the training data may include the known biometric characteristics that correspond to each set of feature data. In any event, the training data may be analyzed using various machine learning techniques to generate predictive models which may be used to determine biometric characteristics of a user, where the user’s biometric characteristics are unknown to the system.

“After the training period, a user may capture audiovisual data such as a video of herself via a client computing device and provide the video to the biometric characteristic system. The biometric characteristic system may analyze the video using computer vision techniques to identify a portion of each frame that corresponds to the user’s face and to extract feature data from the identified portions. The extracted feature data for the user may be compared to the predictive models to determine the user’s biometric characteristics. Additionally, the biometric characteristics may be used to determine an overall health indicator for the user and/or a longevity metric. Then the biometric characteristics, the overall health indicator, and/or the longevity metric may be provided for display on the user’s client computing device.

“In this manner, a user’s health status may be predicted efficiently (e.g., in real-time or at least near-real time from when the video is provided to the biometric characteristic system) and accurately without relying on self-reporting, medical examinations, or readings from various instruments. The present embodiments advantageously streamline the health status assessment process and increase ease of use for users who may simply submit a short video clip of themselves instead of engaging in a lengthy process of filling out forms and providing medical records. Moreover, by capturing video rather than still images, the present embodiments advantageously extract movement data which may be used to predict additional biometric characteristics such as heart rate, blood pressure, galvanic skin response (GSR), etc. Furthermore, video may be more difficult for users to modify in attempts to alter their physical appearances, and therefore using video may prevent fraud.

“In an embodiment, a client device for automatically determining biometric characteristics of a user is provided. The client device includes a user interface, one or more image sensors, one or more processors communicatively coupled to the user interface and the one or more image sensors, and a non-transitory computer-readable memory coupled to the one or more processors and storing instructions thereon. When executed by the one or more processors, the instructions cause the client device to capture, via the one or more image sensors, audiovisual data corresponding to a user, wherein the audiovisual data includes a plurality of frames captured over a threshold time period and provide the audiovisual data corresponding to the user to a server device that analyzes the audiovisual data to identify a plurality of features within the audiovisual data and apply the plurality of features to a model for determining one or more biometric characteristics of the user. The instructions further cause the client device to receive an indication of the determined one or more biometric characteristics from the server device without providing textual information and display, via the user interface, the indication of the determined one or more biometric characteristics.

“In another embodiment, a method for automatically determining biometric characteristics of a user is provided. The method includes capturing, via one or more image sensors in a client device, audiovisual data corresponding to a user, wherein the audiovisual data includes a plurality of frames captured over a threshold time period and providing, by the client device, the audiovisual data corresponding to the user to a server device that analyzes the audiovisual data to identify a plurality of features within the audiovisual data and apply the plurality of features to a model for determining one or more biometric characteristics of the user. The method further includes receiving, at the client device, an indication of the determined one or more biometric characteristics from the server device without providing textual information and displaying, by the client device, the indication of the determined one or more biometric characteristics.

“In yet another embodiment, a non-transitory computer-readable memory is provided. The computer-readable memory stores instructions thereon. When executed by one or more processors, the instructions cause the one or more processors to capture, via one or more image sensors communicatively coupled to the one or more processors, audiovisual data corresponding to a user, wherein the audiovisual data includes a plurality of frames captured over a threshold time period and provide the audiovisual data corresponding to the user to a server device that analyzes the audiovisual data to identify a plurality of features within the audiovisual data and apply the plurality of features to a model for determining one or more biometric characteristics of the user. The instructions further cause the one or more processors to receive an indication of the determined one or more biometric characteristics from the server device without providing textual information and display, via the user interface, the indication of the determined one or more biometric characteristics.”

The claims supplied by the inventors are:

“We claim:

“1. A client device for automatically determining biometric characteristics of a user, the client device comprising: a user interface; one or more image sensors; one or more processors communicatively coupled to the user interface and the one or more image sensors; a non-transitory computer-readable memory coupled to the one or more processors, and storing thereon instructions that, when executed by the one or more processors, cause the client device to: capture, via the one or more image sensors, audiovisual data corresponding to a user, wherein the audiovisual data includes a plurality of frames captured over a threshold time period; provide the audiovisual data corresponding to the user to a server device that identifies a plurality of features within the audiovisual data, and applies the plurality of features to a model, the model determining one or more biometric characteristics of the user based on the plurality of features; receive, from the server device and wihtout providing textual information, a longevity metric determined based on the one or more biometric characteristics wherein the longevity metric indicates: a life insurance quote corresponding to the user and associated with a policy type, and a coverage amount generated based on a life expectancy of the user; and display, via the user interface, the longevity metric.

“2. The client device of claim 1, wherein the one or more biometric characteristics includes an overall health risk score determined by combining individual health risk scores for each of the one or more biometric characteristics.

“3. The client device of claim 1, wherein the model for determining the one or more biometric characteristics of the user is generated using one or more machine learning techniques.

“4. The client device of claim 1, wherein the model is trained based on a plurality of sets of audiovisual data corresponding to a plurality of people and one or more biometric characteristics for each of the plurality of people.

“5. The client device of claim 1, wherein the one or more biometric characteristics include at least one of: age, gender, body mass index (BMI), heart rate, body temperature, galvanic skin response (GSR), smoking status, or emotional state.

“6. The client device of claim 1, further comprising an audio sensor wherein the audiovisual data includes a video of the user with an audio component.

“7. The client device of claim 1, wherein each frame in the audiovisual data includes a representation of the user’s face and the plurality of features includes feature vectors corresponding to the user’s face.

“8. A method for automatically determining biometric characteristics of a user, the method comprising: capturing, via one or more image sensors in a client device, audiovisual data corresponding to a user, wherein the audiovisual data includes a plurality of frames captured over a threshold time period; providing, by the client device, the audiovisual data corresponding to the user to a server device that identifies a plurality of features within the audiovisual data, and applies the plurality of features to a model, the model determining one or more biometric characteristics of the user based on the plurality of features; receiving, from the server device and without providing textual information, at the client device, a longevity metric determined based on the one or more biometric characteristics, wherein the longevity metric indicates: a life insurance quote corresponding to the user and associated with a policy type, and a coverage amount generated based on a life expectancy of the user; and displaying, by the client device, the longevity metric.

“9. The method of claim 8, wherein the one or more biometric characteristics includes an overall health risk score determined by combining individual health risk scores for each of the determined one or more biometric characteristics.

“10. The method of claim 8, wherein the model for determining the one or more biometric characteristics of the user is generated using one or more machine learning techniques.

“11. The method of claim 8, wherein the model is trained based on a plurality of sets of audiovisual data corresponding to a plurality of people and one or more biometric characteristics for each of the plurality of people.

“12. The method of claim 8, wherein the one or more biometric characteristics include at least one of: age, gender, body mass index (BMI), heart rate, body temperature, galvanic skin response (GSR), smoking status, or emotional state.

“13. The method of claim 8, wherein each frame the audiovisual data includes a representation of the user’s face and the plurality of features includes feature vectors corresponding to the user’s face.

“14. A non-transitory computer-readable memory storing thereon instructions that, when executed by one or more processors, cause the one or more processors to: capture, via one or more image sensors communicatively coupled to the one or more processors, audiovisual data corresponding to a user, wherein the audiovisual data includes a plurality of frames captured over a threshold time period; provide the audiovisual data corresponding to the user to a server device that identifies a plurality of features within the audiovisual data, and applies the plurality of features to a model, the model determining one or more biometric characteristics of the user based on the plurality of features; receive, from the server device without providing textual information, a longevity metric determined based on the one or more biometric characteristics, wherein the longevity metric indicates: a life insurance quote corresponding to the user and associated with a policy type, and a coverage amount generated based on a life expectancy of the user; and display, via the user interface, the indication of the longevity metric.

“15. The computer-readable memory of claim 14, wherein the one or more biometric characteristics includes an overall health risk score determined by combining individual health risk scores for each of the determined one or more biometric characteristics.

“16. The computer-readable memory of claim 14, wherein the model for determining the one or more biometric characteristics of the user is generated using one or more machine learning techniques.

“17. The computer-readable memory of claim 14, wherein the model is trained based on a plurality of sets of audiovisual data corresponding to a plurality of people and one or more biometric characteristics for each of the plurality of people.

“18. The computer-readable memory of claim 14, wherein the one or more biometric characteristics include at least one of: age, gender, body mass index (BMI), heart rate, body temperature, galvanic skin response (GSR), smoking status, or emotional state.

“19. The computer-readable memory of claim 14, wherein each frame of the audiovisual data includes a video of the user with an audio component, and the one or more processors further: identify a portion of the plurality of frames that correspond to the user’s face; extract the plurality of features from the portion of the plurality of frames; and determine the one or more biometric characteristics of the user by comparing the plurality of features with the model.

“20. The computer-readable memory of claim 14, wherein the audiovisual data includes a representation of the user’s face and the plurality of features includes feature vectors corresponding to the user’s face.”

URL and more information on this patent, see: Zhang, Dingchao; Bernico, Michael; Laube, Peter; Pamuksuz, Utku; Myers, Jeffrey S.; Bokshi-Drotar, Marigona; Breitweiser, Edward W. Biometric Characteristic Application Using Audio/Video Analysis. U.S. Patent Number 10,825,564, filed December 11, 2017, and published online on November 16, 2020. Patent URL: http://patft.uspto.gov/netacgi/nph-Parser?Sect1=PTO1&Sect2=HITOFF&d=PALL&p=1&u=%2Fnetahtml%2FPTO%2Fsrchnum.htm&r=1&f=G&l=50&s1=10,825,564.PN.&OS=PN/10,825,564RS=PN/10,825,564

(Our reports deliver fact-based news of research and discoveries from around the world.)