Transcribing and Translating Audio | Generative AI Series

Introduction

Whisper is a foundation model from OpenAI. You can use this model to convert speech to text (transcription) or to automatically translate text from one language to another (translation).

Foundation models are trained on massive amounts of data and form the basis of more advanced or specialized models. For instance, OpenAI trains the Whisper model with an audio dataset containing more than 680,000 hours and 1.4 trillion words. The massive dataset allows Whisper to learn patterns and relationships when performing natural language processing (NLP) tasks.

In this article, you'll deploy a Vultr cloud GPU stack and install the required libraries to implement OpenAI's Whisper model with Python to transcribe audio and translate text.

Prerequisites

Before you begin:

• Deploy a new Ubuntu 22.04 A100 Vultr Cloud GPU Server with at least:

  • 80 GB GPU RAM
  • 12 vCPUs
  • 120 GB Memory

• Establish an SSH connection to the server.

• Create a non-root user with sudo rights and switch to the account.

• Create a HuggingFace account.

• Create a Hugging Face user access token.

Install the FFmpeg Package

The Whisper model requires the FFmpeg package. This package has many useful libraries for processing multimedia content, such as audio and video. Follow the steps below to install FFmpeg:

1. Choose the appropriate command to install FFmpeg. For Ubuntu and Arch, run the commands below:

   • Ubuntu or Debian:

     console Copy

     $ sudo apt update 
     $ sudo apt install ffmpeg
     

   • Arch Linux:

     console Copy

     $ sudo pacman -S ffmpeg
     

2. Use pip to install the openai-whisper model.

   console Copy

   $ pip install openai-whisper
   

3. Ensure you've installed the Whisper model by checking its version.

   console Copy

   $ pip show openai-whisper
   

   Output:

   Name: openai-whisper
   Version: 20231117
   Summary: Robust Speech Recognition via Large-Scale Weak Supervision
   Home-page: https://github.com/openai/whisper

Transcribe an Audio File With Python

In this section, you'll download a sample audio file from Steve Jobs, the visionary co-founder of Apple. Then, you'll use the Whisper model with Python to transcribe the audio file to text. Follow the steps below:

1. Download the sample steve-jobs.mp3 file using the Linux wget command.

   console Copy

   $ wget https://docs.vultr.com/public/doc-assets/new/implementing-audio-transcription-with-translation-genai-series/steve-jobs.mp3
   

2. Create a new transcribe.py file using a text editor like nano.

   console Copy

   $ nano transcribe.py
   

3. Enter the following information into the transcribe.py file. In the following file, you're loading the sample audio file from Steve Jobs and transcribing the audio into a text output.

   python Copy

   import whisper
   import IPython
   
   audio_file = "steve-jobs.mp3"
   
   IPython.display.Audio(audio_file)
   
   model = whisper.load_model("medium")
   
   result = model.transcribe(audio_file)
   
   print(result["text"].strip())
   

4. Save and close the file.

5. Run the transcribe.py file.

   console Copy

   $ python3 transcribe.py
   

6. Verify the following output.

   I'm honored to be with you today for your commencement from one of the finest universities in the world. Truth be told, I never graduated from college, and this is the closest I've ever gotten to a college graduation. Today, I want to tell you three stories from 
   ...
   down the road will give you the confidence to follow your heart even when it leads you off the well-worn path, and that will make all the difference.

Translate Audio File from Spanish to English

In addition to transcribing text, you can use the OpenAI's Whisper model to translate text into different languages. Follow the steps below:

1. Download a sample spanish.mp3 file.

   console Copy

   $ wget https://docs.vultr.com/public/doc-assets/new/implementing-audio-transcription-with-translation-genai-series/spanish.mp3
   

2. Create a new translate.py file.

   console Copy

   $ nano translate.py
   

3. Enter the following information into the translate.py file. You're using the Whisper model to load the Spanish audio sample file in the following file. Then, you're defining a task to convert the audio sample to English.

   python Copy

   import IPython
   import whisper
   
   audio_file = "spanish.mp3"
   
   IPython.display.Audio(audio_file)
   
   model  = whisper.load_model("medium")
   result = model.transcribe(audio_file, task = 'translate')
   
   print(result["text"].strip())
   

4. Run the translate.py file.

   console Copy

   $ python3 translate.py
   

5. Verify the following output.

   What do you think artificial intelligence is? What do I think it is? I don't know how to describe it. Something that is not natural, obviously. Artificial intelligence is, through data, introducing an algorithm

Conclusion

In this article, you explored how to use OpenAI's Whisper foundation model to transcribe and translate sample audio files. You started with transcribing an English audio file to English text. Then, you've also translated a Spanish audio sample file to English text.