Patent Issued for Method And System For Identifying Biometric Characteristics Using Machine Learning Techniques (USPTO 10,503,970)

2019 DEC 25 (NewsRx) -- By a News Reporter-Staff News Editor at Insurance Daily News -- From Alexandria, Virginia, NewsRx journalists report that a patent by the inventors Zhang, Dingchao (Normal, IL); Bernico, Michael (Bloomington, IL); Laube, Peter (Bloomington, IL); Pamuksuz, Utku (Champaign, IL); Myers, Jeffrey S. (Normal, IL); Bokshi-Drotar, Marigona (McKinney, TX); Breitweiser, Edward W. (Bloomington, IL), filed on December 11, 2017, was published online on December 23, 2019.

The patent’s assignee for patent number 10,503,970 is State Farm Mutual Automobile Insurance Company (Bloomington, Illinois, United States).

News editors obtained the following quote from the background information supplied by the inventors: “Today, a user’s health status may be determined based on several biometric characteristics, such as the user’s age, gender, blood pressure, heart rate, body mass index (BMI), body temperature, stress levels, smoking status, etc. These biometric characteristics are typically obtained through self-reporting from the user (e.g., by filling out a form indicating the user’s gender, birth date, etc.) and/or medical examinations that include taking measurements conducted by various instruments, such as a thermometer, scale, heart rate monitor, blood pressure cuff, etc.

“This process of filling out forms and taking measurements with several different instruments may be difficult and time consuming for the user. Users may also withhold information or report incorrect information which may lead to inaccuracies in the health status assessment (e.g., from errors in self-reporting or uncalibrated instruments).”

As a supplement to the background information on this patent, NewsRx correspondents also obtained the inventors’ summary information for this patent: “To efficiently and accurately predict a user’s health status and corresponding longevity metric, a biometric characteristic system may be trained using various machine learning techniques to create predictive models for determining biometric characteristics of the user based on video of the user. The determined or predicted biometric characteristics may be combined to generate an overall indication of the user’s health which may be used to generate a longevity metric for the user. The biometric characteristic system may be trained by obtaining audiovisual data (e.g., videos or images) of several people having known biometric characteristics at the time the audiovisual data is captured (e.g., age, gender, BMI, etc.). The people may be referred to herein as ‘training subjects.’ For example, the training data may include public audiovisual data such as movies, television, music videos, etc., featuring famous actors or actresses having biometric characteristics which are known or which are easily obtainable through public content (e.g., via Internet Movie Database (IMDb.RTM.), Wikipedia.TM., etc.).

“In some embodiments, the training data may include feature data extracted from the audiovisual data using computer vision techniques and the training data may include the known biometric characteristics that correspond to each set of feature data. In any event, the training data may be analyzed using various machine learning techniques to generate predictive models which may be used to determine biometric characteristics of a user, where the user’s biometric characteristics are unknown to the system.

“After the training period, a user may capture audiovisual data such as a video of herself via a client computing device and provide the video to the biometric characteristic system. The biometric characteristic system may analyze the video using computer vision techniques to identify a portion of each frame that corresponds to the user’s face and to extract feature data from the identified portions. The extracted feature data for the user may be compared to the predictive models to determine the user’s biometric characteristics. Additionally, the biometric characteristics may be used to determine an overall health indicator for the user and/or a longevity metric. Then the biometric characteristics, the overall health indicator, and/or the longevity metric may be provided for display on the user’s client computing device.

“In this manner, a user’s health status may be predicted efficiently (e.g., in real-time or at least near-real time from when the video is provided to the biometric characteristic system) and accurately without relying on self-reporting, medical examinations, or readings from various instruments. The present embodiments advantageously streamline the health status assessment process and increase ease of use for users who may simply submit a short video clip of themselves instead of engaging in a lengthy process of filling out forms and providing medical records. Moreover, by capturing video rather than still images, the present embodiments advantageously extract movement data which may be used to predict additional biometric characteristics such as heart rate, blood pressure, galvanic skin response (GSR), etc. Furthermore, video may be more difficult for users to modify in attempts to alter their physical appearances, and therefore using video may prevent fraud.

“In an embodiment, a method for identifying biometric characteristics of a user based on audiovisual data is provided. The method includes obtaining a plurality of sets of audiovisual data corresponding to a plurality of people and one or more biometric characteristics for each of the plurality of people. For each of the one or more biometric characteristics, the method includes analyzing the plurality of sets of audiovisual data to identify a plurality of features and generate a model for determining an unknown biometric characteristic of a user based on the plurality of features and the obtained biometric characteristic for each of the plurality of people. The method also includes receiving a set of audiovisual data corresponding to a user, wherein the audiovisual data includes a plurality of frames captured over a threshold time period, applying features within the set of audiovisual data corresponding to the user to the one or more models to determine the one or more biometric characteristics of the user, and providing an indication of the determined one or more biometric characteristics of the user to a client computing device.

“In another embodiment, a server computing device for identifying biometric characteristics of a user based on audiovisual data is provided. The server computing device includes one or more processors and a non-transitory computer-readable memory coupled to the one or more processors and storing instructions thereon. When executed by the one or more processors, the instructions cause the server computing device to obtain a plurality of sets of audiovisual data corresponding to a plurality of people and one or more biometric characteristics for each of the plurality of people. For each of the one or more biometric characteristics, the instructions cause the server computing device to analyze the plurality of sets of audiovisual data to identify a plurality of features and generate a model for determining an unknown biometric characteristic of a user based on the plurality of features and the obtained biometric characteristic for each of the plurality of people. The instructions further cause the server computing device to receive a set of audiovisual data corresponding to a user, wherein the audiovisual data includes a plurality of frames captured over a threshold time period, apply features within the set of audiovisual data corresponding to the user to the one or more models to determine the one or more biometric characteristics of the user, and provide an indication of the determined one or more biometric characteristics of the user to a client computing device.

“In yet another embodiment, a non-transitory computer-readable memory is provided. The computer-readable memory stores instructions thereon. When executed by one or more processors, the instructions cause the one or more processors to obtain a plurality of sets of audiovisual data corresponding to a plurality of people and one or more biometric characteristics for each of the plurality of people. For each of the one or more biometric characteristics, the instructions cause the one or more processors to analyze the plurality of sets of audiovisual data to identify a plurality of features and generate a model for determining an unknown biometric characteristic of a user based on the plurality of features and the obtained biometric characteristic for each of the plurality of people. The instructions further cause the one or more processors to receive a set of audiovisual data corresponding to a user, wherein the audiovisual data includes a plurality of frames captured over a threshold time period, apply features within the set of audiovisual data corresponding to the user to the one or more models to determine the one or more biometric characteristics of the user, and provide an indication of the determined one or more biometric characteristics of the user to a client computing device.”

The claims supplied by the inventors are:

“We claim:

“1. A method for identifying biometric characteristics of a user based on audiovisual data, the method executed by one or more processors programmed to perform the method, the method comprising: obtaining, at one or more processors, a plurality of sets of audiovisual data corresponding to a plurality of people and one or more biometric characteristics for each of the plurality of people; for each of the one or more biometric characteristics: analyzing, by the one or more processors, each of the plurality of sets of audiovisual data to identify a plurality of features; analyzing, by the one or more processors, the plurality of features in each set of audiovisual data to identify movement data corresponding to changes in the plurality of features over time; and generating, by the one or more processors a model for determining an unknown biometric characteristic of a user based on the plurality of features, the movement data, and the obtained biometric characteristic for each of the plurality of people; receiving, at the one or more processors, a set of audiovisual data corresponding to a user, wherein the audiovisual data includes a plurality of frames captured over a threshold time period; applying, by the one or more processors, features within the set of audiovisual data corresponding to the user to the one or more models to determine the one or more biometric characteristics of the user; and providing, by the one or more processors, an indication of the determined one or more biometric characteristics of the user to a client computing device.

“2. The method of claim 1, further comprising: determining, by the one or more processors, an individual health indicator for each of the one or more determined biometric characteristics of the user; and combining, by the one or more processors, the individual health indicators for the user to generate an overall health indicator, wherein providing the indication of the determined one or more biometric characteristics of the user to the client computing device includes providing the overall health indicator to the client computing device.

“3. The method of claim 2, wherein the overall health indicator is used to predict remaining longevity for the user.

“4. The method of claim 1, wherein analyzing the plurality of sets of audiovisual data to identify a plurality of features includes: analyzing, by the one or more processors, each frame in the plurality of sets of audiovisual data to identify a portion of each frame depicting a face of the person using face detection techniques; and analyzing, by the one or more processors, the plurality of portions depicting faces to identify a plurality of feature vectors corresponding to the faces.

“5. The method of claim 1, wherein the model for determining an unknown biometric characteristic of a user is generated using one or more machine learning techniques.

“6. The method of claim 1, wherein the one or more biometric characteristics include at least one of: age, gender, body mass index (BMI), heart rate, body temperature, galvanic skin response (GSR), smoking status, or emotional state.

“7. A server computing device for identifying biometric characteristics of a user based on audiovisual data, the server computing device comprising: one or more processors; and a non-transitory computer-readable memory coupled to the one or more processors, and storing thereon instructions that, when executed by the one or more processors, cause the server computing device to: obtain a plurality of sets of audiovisual data corresponding to a plurality of people and one or more biometric characteristics for each of the plurality of people; for each of the one or more biometric characteristics: analyze each of the plurality of sets of audiovisual data to identify a plurality of features; analyze the plurality of features in each set of audiovisual data to identify movement data corresponding to changes in the plurality of features over time; and generate a model for determining an unknown biometric characteristic of a user based on the plurality of features, the movement data, and the obtained biometric characteristic for each of the plurality of people; receive a set of audiovisual data corresponding to a user, wherein the audiovisual data includes a plurality of frames captured over a threshold time period; apply features within the set of audiovisual data corresponding to the user to the one or more models to determine the one or more biometric characteristics of the user; and provide an indication of the determined one or more biometric characteristics of the user to a client computing device.

“8. The server computing device of claim 7, wherein the instructions further cause the server computing device to: determine an individual health indicator for each of the one or more determined biometric characteristics of the user; and combine the individual health indicators for the user to generate an overall health indicator, wherein providing the indication of the determined one or more biometric characteristics of the user to the client computing device includes providing the overall health indicator to the client computing device.

“9. The server computing device of claim 8, wherein the overall health indicator is used to predict remaining longevity for the user.

“10. The server computing device of claim 7, wherein to analyze the plurality of sets of audiovisual data to identify a plurality of features, the instructions cause the server computing device to: analyze each frame in the plurality of sets of audiovisual data to identify a portion of each frame depicting a face of the person using face detection techniques; and analyze the plurality of portions depicting faces to identify a plurality of feature vectors corresponding to the faces.

“11. The server computing device of claim 7, wherein the model for determining an unknown biometric characteristic of a user is generated using one or more machine learning techniques.

“12. The server computing device of claim 7, wherein the one or more biometric characteristics include at least one of: age, gender, body mass index (BMI), heart rate, body temperature, galvanic skin response (GSR), smoking status, or emotional state.

“13. A non-transitory computer-readable memory storing thereon instructions that, when executed by one or more processors, cause the one or more processors to: obtain a plurality of sets of audiovisual data corresponding to a plurality of people and one or more biometric characteristics for each of the plurality of people; for each of the one or more biometric characteristics: analyze the plurality of sets of audiovisual data to identify a plurality of features; analyze the plurality of features in each set of audiovisual data to identify movement data corresponding to changes in the plurality of features over time; and generate a model for determining an unknown biometric characteristic of a user based on the plurality of features, the movement data, and the obtained biometric characteristic for each of the plurality of people; receive a set of audiovisual data corresponding to a user, wherein the audiovisual data includes a plurality of frames captured over a threshold time period; apply features within the set of audiovisual data corresponding to the user to the one or more models to determine the one or more biometric characteristics of the user; and provide an indication of the determined one or more biometric characteristics of the user to a client computing device.

“14. The non-transitory computer-readable memory of claim 13, wherein the instructions further cause the one or more processors to: determine an individual health indicator for each of the one or more determined biometric characteristics of the user; and combine the individual health indicators for the user to generate an overall health indicator, wherein providing the indication of the determined one or more biometric characteristics of the user to the client computing device includes providing the overall health indicator to the client computing device.

“15. The non-transitory computer-readable memory of claim 13, wherein to analyze the plurality of sets of audiovisual data to identify a plurality of features, the instructions cause the one or more processors to: analyze each frame in the plurality of sets of audiovisual data to identify a portion of each frame depicting a face of the person using face detection techniques; and analyze the plurality of portions depicting faces to identify a plurality of feature vectors corresponding to the faces.

“16. The non-transitory computer-readable memory of claim 13, wherein the model for determining an unknown biometric characteristic of a user is generated using one or more machine learning techniques.

“17. The non-transitory computer-readable memory of claim 13, wherein the one or more biometric characteristics include at least one of: age, gender, body mass index (BMI), heart rate, body temperature, galvanic skin response (GSR), smoking status, or emotional state.”

For additional information on this patent, see: Zhang, Dingchao; Bernico, Michael; Laube, Peter; Pamuksuz, Utku; Myers, Jeffrey S.; Bokshi-Drotar, Marigona; Breitweiser, Edward W. Method And System For Identifying Biometric Characteristics Using Machine Learning Techniques. U.S. Patent Number 10,503,970, filed December 11, 2017, and published online on December 23, 2019. Patent URL: http://patft.uspto.gov/netacgi/nph-Parser?Sect1=PTO1&Sect2=HITOFF&d=PALL&p=1&u=%2Fnetahtml%2FPTO%2Fsrchnum.htm&r=1&f=G&l=50&s1=10,503,970.PN.&OS=PN/10,503,970RS=PN/10,503,970

(Our reports deliver fact-based news of research and discoveries from around the world.)