Google has developed a method it calls audiovisual speech separation. It used machine learning to identify specific speech sounds from audio in video.

The technology does more than just isolate the verbalized words from the background and other sources. It can separate the speech patterns of two or more people talking.  

The details posted in a Google Developers’ blog describe the technology, which also includes several videos demonstrating how it works.

The technology can separate background and ambient noise on one audio track. Researchers believe this capability can have a range of applications, from speech enhancement and recognition in videos and voice search to video conferencing and the ability to improve hearing aids.

The technique combines auditory and visual signals of an input video to separate the speech. The visual signal aims to improve the speech separation quality in cases of mixed speech — but more importantly, it also associates the separated speech tracks with the visible speakers in the video.

Researchers explained that when training the model they used a large collection of 100,000 high-quality videos of lectures and talks from YouTube. From the videos they extracted segments with a clean speech, which means they removed any sounds of music and other speakers. It results in about 2,000 hours of video clips, each of one person visible to the camera and talking with no background interference.

Google provides several videos in which researchers were able to separate the speech of two people talking such as sports commentators and someone talking in a noisy cafeteria.

The technology also can be used for automatic video captioning.