Patent Issued for Recipient customized generative voice modeling (USPTO 11967311): United Services Automobile Association

2024 MAY 15 (NewsRx) -- By a News Reporter-Staff News Editor at Insurance Daily News -- A patent by the inventors Jayapalan, Vijay (San Antonio, TX, US), Lyle, Ruthie (Durham, NC, US), Yarbrough, Gregory B. (San Antonio, TX, US), filed on June 30, 2021, was published online on April 23, 2024, according to news reporting originating from Alexandria, Virginia, by NewsRx correspondents.

Patent number 11967311 is assigned to United Services Automobile Association (San Antonio, Texas, United States).

The following quote was obtained by the news editors from the background information supplied by the inventors: “Individuals communicate electronically to operators associated with an entity for various reasons. For instance, a customer of an entity (e.g., a bank) can contact an operator of the entity via a phone call to enroll in one or more products offered by the entity. During the communication, the operator can provide information to the client verbally. Additionally, in this conversation, the audio of the operator can include various characteristics of the voice and environment around the operator that are unique to the operator. For example, the operator may speak at a specific audio level, have background noise, speak at a specific speed/cadence, etc.

“Further, in many cases, an operator during a conversation with an individual may provide mandated information to the individual. For example, the operator may read a series of disclosures or initiate a pre-recording of the disclosures to the individual responsive to determining that the disclosures are required (e.g., a specific product provided to the individual requires reading of the disclosures). In these cases, the reading of the mandated information may have different features/characteristics than that of the previous discussion with the operator. For example, a voice recording of the disclosures may include a different voice type, difference audio speed/cadence, volume level, etc., than that of the discussion with the operator. This may result in lower user experience.

“The techniques introduced here may be better understood by referring to the following Detailed Description in conjunction with the accompanying drawings, in which like reference numerals indicate identical or functionally similar elements. Moreover, while the technology is amenable to various modifications and alternative forms, specific embodiments have been shown by way of example in the drawings and are described in detail below. The intention, however, is not to limit the technology to the particular embodiments described. On the contrary, the technology is intended to cover all modifications, equivalents, and alternatives falling within the scope of the technology as defined by the appended claims.”

In addition to the background information obtained for this patent, NewsRx journalists also obtained the inventors’ summary information for this patent: “Customers call service providers for many reasons such as to obtain information, resolve bills, request new products. In some cases, the caller speaks with an interactive voice response (IVR) system or hears pre-recorded statements (e.g., specific disclosure statements). Listening to a monotone pre-recorded statement can be a bad experience for the user. To improve the user experience particularly with the pre-recorded statements, the present embodiments relate to generation of on demand curated audio content. As the caller speaks with the representative, the system can obtain insights about the caller (e.g., dialect, pace of speech, tone) and about the caller’s environment (e.g., noisy background, location). Using real-time data, the pre-recorded statements can be altered to best suit the caller. The pre-recorded statement can be played in the voice of the representative so that the pre-recorded statement feels seamless.

“More specifically, an insight module can analyze aspects of a communication between an operator of an entity (e.g., representative) and a client to identify various insights of the client (e.g., age of the client, emotional state, ambient noise level, cognitive capability, communication speed). A processing module can combine the insights identified on demand and audio content incorporating the voice of the operator to curate the audio content that corresponds to the insights relating to the client. Accordingly, the system can generate on-demand curated audio content that is customized using audio features (or insights) relating to the speech of the operator on demand (e.g., in real-time or near real-time).

“The curated audio content can be generated using a generative voice model to model the audio with the voice of the operator. Further, the generative voice model may process the insights to customize the audio. For example, if the client has a noisy background when speaking to the operator, the generative voice model can generate audio content to include background noise that is similar to the detected noisy background of the client. As another example, if the insights identify that the operator and/or the client is an older individual or is hard of hearing, the playback of the audio content may be slowed.

“The system as described herein can initiate insight generation and/or audio content customization responsive to detecting one or more triggers. For example, a trigger can include identifying that audio content is required for a conversation, identifying that the client has difficulty understanding the operator’s voice (e.g., detecting that the client has said “what did you say”), detecting an indication from the operator to initiate generation of the customized audio content, etc.

“Accordingly, the present embodiments can detect contextual features of the client and/or operator audio and generate curated audio content that is consistent with the detected contextual features. This can allow for playback of curated audio content that has features that match those of the operator’s voice but are suited for the client and the client’s environment. The playback of the curated audio content can increase user experience, as the user can better understand the disclaimer audio and comprehend the audio as the curated audio content matches the voice of the operator.”

The claims supplied by the inventors are:

“1. A method performed by a network-accessible server system to generate on demand curated audio content, the method comprising: detecting an initiation of an electronic communication between a client via a client device and an operator via an operator device; responsive to detecting the initiation of the electronic communication, processing, audio output from the client device, to derive a series of client audio insights defining client characteristics of the client and/or characteristics of a client environment in which the client device is operating; detecting a first triggering event to generate curated audio content; responsive to detecting the first triggering event, combining a portion of a registry of stored audio data with a first text-based script to generate output audio content; generating curated audio content by determining and applying, for each client audio insight defining client characteristics of the client and/or characteristics of the client environment, a curation rule defined to customize audio for curated reception by a client A) with the defined client characteristics and/or B) in the defined client environment; and initiating playback of the curated audio content from the client device via the electronic communication.

“2. The method of claim 1, further comprising: obtaining, for the client audio insights, the defined curation rules by: processing the audio outputting from the client device to identify a set of characteristics relating to the audio of the client, each of the set of characteristics relating to an insight type; for each characteristic of the set of characteristics, mapping each characteristic to a corresponding insight type; and selecting, for each insight type of the series of insight types, a rule associated with each corresponding insight type.

“3. The method of claim 1, wherein the series of client audio insights include any of: a volume level of the audio outputting from the client device, a background noise level of the audio outputting from the client device, a determined age of the client, a detected speech cadence of the client, and a detected emotional state of the client.

“4. The method of claim 1, further comprising: storing audio, outputting from the operator device, in the registry of stored audio data; and processing the audio, output from the operator device, to identify words included in the audio output from the operator device, wherein the audio output from the operator device comprises the portion of the registry of stored audio data that is combined with the text-based script to generate the set of audio content.

“5. The method of claim 1, wherein the first triggering event includes A) an indication provided by the operator device to initiate generation of the curated audio content or B) detecting a keyword included in audio output from the operator device or the audio output from the client device.

“6. The method of claim 1, further comprising: monitoring audio data transmitted in the electronic communication between the client device and the operator device to identify an audio content type indicator; and responsive to identifying the audio content type indicator, selecting the first text-based script included in a library of text-based scripts by mapping the audio content type indicator to the first text-based script.

“7. The method of claim 1, wherein the generating curated audio content comprises: processing one or more of the client audio insights using a generative voice model to modify at least one audio content feature according to the curation rule that corresponds to each client audio insight.

“8. The method of claim 1, further comprising: responsive to detecting a completion of playback of the curated audio content, transmitting, to the client device, a request for feedback; receiving a set of requested feedback, from the client device; and storing the set of requested feedback and the associated curated audio content in a curated audio content registry, wherein curated audio content with requested feedback exceeding a threshold rating level is configured to be utilized in generation of subsequent instances of curated audio content.

“9. A system to generate on demand curated audio content for an electronic communication between an operator device and a client device, the electronic communication facilitating voice communication between a client and an operator, the system comprising: a server configured to: process audio, output from a client device, to derive a series of client audio insights defining client characteristics of the client and/or characteristics of a client environment in which the client device is operating; detect a first triggering event to generate curated audio content; responsive to detecting the first triggering event, combine a portion of a registry of stored audio data with a first text-based script to generate output audio content; generating curated audio content by determining and applying, for each client audio insight defining client characteristics of the client and/or characteristics of the client environment, a curation rule defined to customize audio for curated reception by a client A) with the defined client characteristics and/or B) in the defined client environment; and causing playback of the curated audio content to the operator device via the electronic communication.

“10. The system of claim 9, wherein the server is further configured to: obtain, for the client audio insights, the defined curation rules by: processing the audio output from the client device to identify a set of characteristics relating to the audio of the client, each of the set of characteristics relating to an insight type; for each characteristic of the set of characteristics, mapping each characteristic to a corresponding insight type; and selecting, for each insight type of the series of insight types, a rule associated with each corresponding insight type.

“11. The system of claim 9, wherein the server is further configured to: store audio, output from the operator device, in the registry of stored audio data; and process the audio, output from the operator device, to identify words included in the audio output from the operator device, wherein the audio output from the operator device comprises the portion of the registry of stored audio data that is combined with the text-based script to generate the set of audio content.

“12. The system of claim 9, wherein the server is further configured to: monitor audio data transmitted in the electronic communication between the client device and the operator device to identify an audio content type indicator; and responsive to identifying the audio content type indicator, select the first text-based script included in a library of text-based scripts by mapping the audio content type indicator to the first text-based script.

“13. The system of claim 9, wherein the server is further configured to: responsive to detecting a completion of playback of the curated audio content, transmit, to the client device, a request for feedback; receive a set of requested feedback from the client device; and store the set of requested feedback and the associated curated audio content in a curated audio content registry, wherein curated audio content with requested feedback exceeding a threshold rating level is configured to be utilized in generation of subsequent instances of curated audio content.

“14. The system of claim 9, wherein the server is further configured to: detect an indication in audio, output from the client device, that the client has failed to comprehend audio that was included in the curated audio content; and responsive to detecting the indication in audio, output from the client device, that the client has failed to comprehend audio that was included in the curated audio content, modify a volume level and/or a speed of playback of the curated audio content.

“15. A non-transitory machine-readable medium having machine executable instructions stored thereon that, when executed by one or more processors, direct the one or more processors to perform a method comprising: detecting an initiation of an electronic communication between a client via a client device and an operator via an operator device; responsive to detecting the initiation of the electronic communication, processing, audio output from the client device, to derive a series of client audio insights defining client characteristics of the client and/or characteristics of a client environment in which the client device is operating; detecting a first triggering event to generate curated audio content; generating curated audio content by determining and applying, for each client audio insight defining client characteristics of the client and/or characteristics of the client environment, a curation rule defined to customize audio for curated reception by a client A) with the defined client characteristics and/or B) in the defined client environment; and initiating playback of the curated audio content from the client device via the electronic communication.

“16. The non-transitory machine-readable medium of claim 15, wherein the method further comprises: obtaining, for the client audio insights, the defined curation rules by: processing the audio output from the client device to identify a set of characteristics relating to the audio of the client, each of the set of characteristics relating to an insight type; for each characteristic of the set of characteristics, mapping each characteristic to a corresponding insight type; and selecting, for each insight type of the series of insight types, a rule associated with each corresponding insight type.”

There are additional claims. Please visit full patent to read further.

URL and more information on this patent, see: Jayapalan, Vijay. Recipient customized generative voice modeling. U.S. Patent Number 11967311, filed June 30, 2021, and published online on April 23, 2024. Patent URL (for desktop use only): https://ppubs.uspto.gov/pubwebapp/external.html?q=(11967311)&db=USPAT&type=ids

(Our reports deliver fact-based news of research and discoveries from around the world.)