Azure AI Speech Services

Every application that listens, speaks, or bridges language barriers needs a speech engine behind it. Azure AI Speech Services provides that engine: production-grade speech-to-text, natural-sounding text-to-speech with neural voices, real-time translation across dozens of languages, and the ability to create a custom voice that sounds uniquely like your brand. Whether you’re building a call center bot, a multilingual conferencing tool, or an accessibility layer for your product, these APIs handle the hard parts so you can focus on the experience.

This guide walks through the four pillars of Azure AI Speech, with working Python code for each one. You’ll learn how to transcribe audio in real time, generate speech with SSML control, translate conversations across languages on the fly, and understand when Custom Neural Voice makes sense for your project.

Architecture Overview

Azure AI Speech is not a single API but a family of services built on shared neural network infrastructure. Here is how the major components relate to each other and to the broader Azure AI ecosystem:

Key Metrics at a Glance

Speech-to-Text: Transcribing Audio

Speech-to-Text (STT) converts spoken audio into text. Azure supports two modes: real-time recognition for live microphone or stream input, and batch transcription for processing pre-recorded files at scale. Both use the same underlying neural models trained on thousands of hours of multilingual audio.

Setting Up the Azure Speech SDK

Install the SDK and create your Speech resource in the Azure portal before writing any code. You will need the subscription key and region.

pip install azure-cognitiveservices-speech

Real-Time Speech Recognition from Microphone

This example captures audio from the default microphone and prints recognized text as it arrives. The SDK handles silence detection, endpoint detection, and partial results automatically.

import azure.cognitiveservices.speech as speechsdk

# Configure the speech service
speech_config = speechsdk.SpeechConfig(
    subscription="YOUR_SPEECH_KEY",
    region="eastus"
)
speech_config.speech_recognition_language = "en-US"

# Use default microphone as audio input
audio_config = speechsdk.AudioConfig(
    use_default_microphone=True
)

# Create the recognizer
recognizer = speechsdk.SpeechRecognizer(
    speech_config=speech_config,
    audio_config=audio_config
)

print("Speak into your microphone...")
result = recognizer.recognize_once_async().get()

if result.reason == speechsdk.ResultReason.RecognizedSpeech:
    print(f"Recognized: {result.text}")
elif result.reason == speechsdk.ResultReason.NoMatch:
    print("No speech could be recognized.")
elif result.reason == speechsdk.ResultReason.Canceled:
    cancellation = result.cancellation_details
    print(f"Canceled: {cancellation.reason}")

Continuous Recognition for Long Audio

For conversations, meetings, or any audio longer than a few seconds, use continuous recognition. The SDK fires events as it processes the audio stream, giving you both partial (interim) and final results.

import azure.cognitiveservices.speech as speechsdk
import time

speech_config = speechsdk.SpeechConfig(
    subscription="YOUR_SPEECH_KEY",
    region="eastus"
)
speech_config.speech_recognition_language = "en-US"

# Enable detailed output with word-level timestamps
speech_config.request_word_level_timestamps()
speech_config.output_format = speechsdk.OutputFormat.Detailed

# Recognize from an audio file instead of microphone
audio_config = speechsdk.AudioConfig(
    filename="meeting-recording.wav"
)

recognizer = speechsdk.SpeechRecognizer(
    speech_config=speech_config,
    audio_config=audio_config
)

all_results = []

def on_recognized(evt):
    """Called when a final recognition result is received."""
    if evt.result.reason == speechsdk.ResultReason.RecognizedSpeech:
        all_results.append(evt.result.text)
        print(f"RECOGNIZED: {evt.result.text}")

def on_recognizing(evt):
    """Called for interim/partial results."""
    print(f"  [partial]: {evt.result.text}")

def on_canceled(evt):
    print(f"CANCELED: {evt.cancellation_details}")

# Connect event handlers
recognizer.recognized.connect(on_recognized)
recognizer.recognizing.connect(on_recognizing)
recognizer.canceled.connect(on_canceled)

# Start continuous recognition
recognizer.start_continuous_recognition()

# Wait for processing to finish (simple approach)
done = False

def stop_cb(evt):
    global done
    done = True

recognizer.session_stopped.connect(stop_cb)
recognizer.canceled.connect(stop_cb)

while not done:
    time.sleep(0.5)

recognizer.stop_continuous_recognition()

# Full transcript
transcript = " ".join(all_results)
print(f"\nFull transcript:\n{transcript}")

Transcribing from an Audio File (Batch API)

For offline processing of large audio archives, the batch transcription REST API lets you submit files stored in Azure Blob Storage and retrieve the results later. This is ideal for call center recordings, podcast transcripts, or legal depositions.

import requests
import json

SPEECH_KEY = "YOUR_SPEECH_KEY"
REGION = "eastus"
ENDPOINT = f"https://{REGION}.api.cognitive.microsoft.com"

# Submit a batch transcription job
headers = {
    "Ocp-Apim-Subscription-Key": SPEECH_KEY,
    "Content-Type": "application/json"
}

body = {
    "contentUrls": [
        "https://mystorage.blob.core.windows.net/audio/call01.wav",
        "https://mystorage.blob.core.windows.net/audio/call02.wav"
    ],
    "locale": "en-US",
    "displayName": "Call Center Batch Job",
    "properties": {
        "wordLevelTimestampsEnabled": True,
        "diarizationEnabled": True,
        "punctuationMode": "DictatedAndAutomatic"
    }
}

response = requests.post(
    f"{ENDPOINT}/speechtotext/v3.2/transcriptions",
    headers=headers,
    json=body
)

transcription = response.json()
print(f"Job created: {transcription['self']}")
print(f"Status: {transcription['status']}")

Text-to-Speech: Generating Natural Audio

Text-to-Speech (TTS) converts written text into lifelike spoken audio using neural networks. Azure offers over 500 neural voices across 100+ languages. The output quality is remarkably close to human speech, with natural prosody, intonation, and pacing.

Basic Speech Synthesis

import azure.cognitiveservices.speech as speechsdk

speech_config = speechsdk.SpeechConfig(
    subscription="YOUR_SPEECH_KEY",
    region="eastus"
)

# Select a neural voice
speech_config.speech_synthesis_voice_name = "en-US-JennyNeural"

# Output to default speaker
audio_config = speechsdk.audio.AudioOutputConfig(
    use_default_speaker=True
)

synthesizer = speechsdk.SpeechSynthesizer(
    speech_config=speech_config,
    audio_config=audio_config
)

result = synthesizer.speak_text_async(
    "Welcome to Azure AI Speech Services. "
    "This is a neural voice speaking naturally."
).get()

if result.reason == speechsdk.ResultReason.SynthesizingAudioCompleted:
    print("Speech synthesized successfully.")
    print(f"Audio duration: {result.audio_duration}")

Saving Synthesis Output to a File

# Save directly to a WAV file instead of playing on speakers
audio_config = speechsdk.audio.AudioOutputConfig(
    filename="output.wav"
)

synthesizer = speechsdk.SpeechSynthesizer(
    speech_config=speech_config,
    audio_config=audio_config
)

result = synthesizer.speak_text_async(
    "This audio will be saved to a WAV file."
).get()

print("Audio saved to output.wav")

Advanced Control with SSML

Speech Synthesis Markup Language (SSML) gives you fine-grained control over how the voice speaks: adjust rate, pitch, volume, add pauses, switch voices mid-sentence, or apply speaking styles like “cheerful” or “empathetic.”

<!-- SSML example with multiple voices and styles -->
<speak version="1.0"
       xmlns="http://www.w3.org/2001/10/synthesis"
       xmlns:mstts="http://www.w3.org/2001/mstts"
       xml:lang="en-US">

  <voice name="en-US-JennyNeural">
    <!-- Cheerful greeting -->
    <mstts:express-as style="cheerful">
      Welcome to our customer service line!
      We're happy to help you today.
    </mstts:express-as>

    <break time="500ms"/>

    <!-- Normal pace for instructions -->
    <prosody rate="-10%" pitch="+5%">
      For billing inquiries, press one.
      For technical support, press two.
    </prosody>

    <!-- Emphasize important info -->
    Your reference number is
    <say-as interpret-as="characters">ABC123</say-as>.
  </voice>

  <!-- Switch to a different voice -->
  <voice name="en-US-GuyNeural">
    <mstts:express-as style="empathetic">
      I understand your concern. Let me look
      into this for you right away.
    </mstts:express-as>
  </voice>

</speak>

Synthesizing SSML with Python

ssml_text = """
<speak version="1.0"
       xmlns="http://www.w3.org/2001/10/synthesis"
       xmlns:mstts="http://www.w3.org/2001/mstts"
       xml:lang="en-US">
  <voice name="en-US-AriaNeural">
    <mstts:express-as style="newscast-formal">
      Today in technology news: Azure AI Speech Services
      now supports over five hundred neural voices across
      more than one hundred languages and locales.
    </mstts:express-as>
    <break time="300ms"/>
    <prosody rate="medium" volume="loud">
      This marks a significant milestone in the
      democratization of speech AI technology.
    </prosody>
  </voice>
</speak>
"""

result = synthesizer.speak_ssml_async(ssml_text).get()

if result.reason == speechsdk.ResultReason.SynthesizingAudioCompleted:
    print("SSML synthesis completed.")
elif result.reason == speechsdk.ResultReason.Canceled:
    details = result.cancellation_details
    print(f"Synthesis canceled: {details.reason}")
    print(f"Error: {details.error_details}")

Available Speaking Styles

Not all neural voices support all styles. Here are the most commonly used styles with the voices that support them:

Real-Time Speech Translation

Speech Translation combines speech recognition and machine translation into a single streaming pipeline. Audio goes in one language, and you get text (or synthesized audio) out in another language, all in real time. This powers scenarios like live meeting interpreters, multilingual kiosks, and cross-language customer support.

Translating Speech Between Languages

import azure.cognitiveservices.speech as speechsdk

# Configure translation
translation_config = speechsdk.translation.SpeechTranslationConfig(
    subscription="YOUR_SPEECH_KEY",
    region="eastus"
)

# Source language: what the speaker is saying
translation_config.speech_recognition_language = "en-US"

# Target languages: translate to these simultaneously
translation_config.add_target_language("es")   # Spanish
translation_config.add_target_language("fr")   # French
translation_config.add_target_language("de")   # German
translation_config.add_target_language("ja")   # Japanese

# Optional: synthesize the translated output
translation_config.voice_name = "es-ES-ElviraNeural"

audio_config = speechsdk.AudioConfig(
    use_default_microphone=True
)

# Create the translation recognizer
recognizer = speechsdk.translation.TranslationRecognizer(
    translation_config=translation_config,
    audio_config=audio_config
)

print("Speak in English. Translations will appear in real time...")
print("=" * 60)

result = recognizer.recognize_once_async().get()

if result.reason == speechsdk.ResultReason.TranslatedSpeech:
    print(f"Recognized:  {result.text}")
    print(f"Spanish:     {result.translations['es']}")
    print(f"French:      {result.translations['fr']}")
    print(f"German:      {result.translations['de']}")
    print(f"Japanese:    {result.translations['ja']}")

Continuous Translation for Meetings

For longer sessions like meetings or conferences, use continuous translation with event handlers. This approach streams partial translations as the speaker talks, giving listeners a near-real-time experience.

import azure.cognitiveservices.speech as speechsdk
import time
import json

translation_config = speechsdk.translation.SpeechTranslationConfig(
    subscription="YOUR_SPEECH_KEY",
    region="eastus"
)
translation_config.speech_recognition_language = "en-US"
translation_config.add_target_language("es")
translation_config.add_target_language("pt")

audio_config = speechsdk.AudioConfig(
    use_default_microphone=True
)

recognizer = speechsdk.translation.TranslationRecognizer(
    translation_config=translation_config,
    audio_config=audio_config
)

# Store translations for export
translation_log = []

def on_translated(evt):
    if evt.result.reason == speechsdk.ResultReason.TranslatedSpeech:
        entry = {
            "original": evt.result.text,
            "translations": dict(evt.result.translations),
            "offset": evt.result.offset,
            "duration": evt.result.duration
        }
        translation_log.append(entry)
        print(f"[EN] {evt.result.text}")
        print(f"[ES] {evt.result.translations['es']}")
        print(f"[PT] {evt.result.translations['pt']}")
        print()

recognizer.recognized.connect(on_translated)

# Start continuous translation
recognizer.start_continuous_recognition()
print("Meeting translation active. Press Ctrl+C to stop.\n")

try:
    while True:
        time.sleep(0.5)
except KeyboardInterrupt:
    recognizer.stop_continuous_recognition()
    # Save the full translation log
    with open("translation_log.json", "w") as f:
        json.dump(translation_log, f, indent=2)
    print(f"\nSaved {len(translation_log)} entries to translation_log.json")

Custom Neural Voice

Custom Neural Voice (CNV) lets organizations create a unique, branded synthetic voice that sounds like a specific person. Instead of choosing from the catalog of prebuilt voices, you record a professional voice talent, upload the training data, and Azure trains a neural TTS model on that specific voice.

When to Use Custom Neural Voice

• Brand consistency — a banking app that always speaks in the same recognizable tone, no matter the platform.
• Accessibility — recreating a user’s personal voice for people at risk of losing their ability to speak (Personal Voice).
• Content creation — audiobook narration, e-learning, or video voiceovers at scale without booking studio time for every update.
• IVR systems — interactive voice response for call centers that matches your company’s personality rather than sounding generic.

The Training Pipeline

Building a Custom Neural Voice follows these steps:

1. Record training data — typically 300-2,000 utterances (about 30 minutes to 2 hours of clean studio audio) from your chosen voice talent.
2. Prepare transcripts — each audio file needs a matching text transcript with exact sentence-level alignment.
3. Upload and validate — the Azure Speech Studio inspects audio quality, signal-to-noise ratio, and transcript accuracy.
4. Train the model — Azure’s neural network trains on your data. Training typically completes in 2-4 hours.
5. Test and deploy — evaluate the voice in the Speech Studio playground, then deploy it as an endpoint you can call from the SDK.
6. Integrate via SSML — use your custom voice name in SSML or SDK calls exactly like any prebuilt neural voice.

# Using a Custom Neural Voice is identical to using a prebuilt voice
# Just reference your custom voice's deployment name

speech_config = speechsdk.SpeechConfig(
    subscription="YOUR_SPEECH_KEY",
    region="eastus"
)

# Set the custom voice endpoint
speech_config.endpoint_id = "YOUR_CUSTOM_VOICE_ENDPOINT_ID"
speech_config.speech_synthesis_voice_name = "MyBrandVoice"

synthesizer = speechsdk.SpeechSynthesizer(
    speech_config=speech_config
)

# SSML with your custom voice
ssml = """
<speak version="1.0"
       xmlns="http://www.w3.org/2001/10/synthesis"
       xmlns:mstts="http://www.w3.org/2001/mstts"
       xml:lang="en-US">
  <voice name="MyBrandVoice">
    Hello, thank you for calling Contoso. How can I help you today?
  </voice>
</speak>
"""

result = synthesizer.speak_ssml_async(ssml).get()
print(f"Custom voice synthesis: {result.reason}")

Capabilities at a Glance

Service Tiers and Pricing

Azure AI Speech uses a pay-as-you-go model based on the number of audio hours processed. Pricing varies by feature and whether you use standard or custom models.

Real-World Use Cases

Production Best Practices

1. Always handle cancellation errors — network interruptions, expired keys, and quota limits all surface as cancellation events. Log the cancellation_details reason and error code to diagnose issues quickly.
2. Use connection pooling — creating a new SpeechRecognizer or SpeechSynthesizer for every request wastes time on WebSocket handshakes. Reuse objects across requests within a session.
3. Choose the right region — deploy your Speech resource in the region closest to your users. Audio streaming is latency-sensitive, and every 50ms of extra round-trip adds noticeable delay.
4. Set audio format explicitly — for TTS, request compressed formats like Audio48Khz192KBitRateMonoMp3 to reduce bandwidth. WAV is uncompressed and fine for local playback, but wasteful over the network.
5. Enable profanity filtering when appropriate — the STT engine can mask, remove, or pass through profanity. Set this based on your content policy: speech_config.set_profanity(ProfanityOption.Masked).
6. Implement retry logic with exponential backoff — transient failures happen. Retry on HTTP 429 (rate limit) and 503 (service unavailable) with jittered backoff, but never retry on 401 (bad key) or 400 (bad request).
7. Monitor usage and set budget alerts — a bug in continuous recognition can leave a session running indefinitely, accumulating costs. Set up Azure Monitor alerts for unexpected usage spikes.
8. Test with diverse audio conditions — your lab microphone sounds perfect, but production audio includes background noise, echo, accents, and variable volume. Test with realistic samples before launch.

Integrating Speech with Azure OpenAI

One of the most powerful patterns combines Speech Services with Azure OpenAI to create voice-enabled AI assistants. The user speaks, STT transcribes the input, OpenAI generates a response, and TTS speaks the answer back naturally.

import azure.cognitiveservices.speech as speechsdk
from openai import AzureOpenAI

# --- Configuration ---
SPEECH_KEY = "YOUR_SPEECH_KEY"
SPEECH_REGION = "eastus"
OPENAI_ENDPOINT = "https://your-openai.openai.azure.com/"
OPENAI_KEY = "YOUR_OPENAI_KEY"

# Set up Speech
speech_config = speechsdk.SpeechConfig(
    subscription=SPEECH_KEY,
    region=SPEECH_REGION
)
speech_config.speech_recognition_language = "en-US"
speech_config.speech_synthesis_voice_name = "en-US-JennyNeural"

# Set up Azure OpenAI
client = AzureOpenAI(
    azure_endpoint=OPENAI_ENDPOINT,
    api_key=OPENAI_KEY,
    api_version="2024-06-01"
)

# Create recognizer and synthesizer
recognizer = speechsdk.SpeechRecognizer(speech_config=speech_config)
synthesizer = speechsdk.SpeechSynthesizer(speech_config=speech_config)

conversation_history = [
    {"role": "system", "content": "You are a helpful voice assistant. "
     "Keep responses concise (2-3 sentences) since they "
     "will be spoken aloud."}
]

print("Voice assistant ready. Speak to begin...")

while True:
    # Step 1: Listen
    result = recognizer.recognize_once_async().get()
    if result.reason != speechsdk.ResultReason.RecognizedSpeech:
        continue

    user_input = result.text
    print(f"\nYou: {user_input}")

    if "goodbye" in user_input.lower():
        synthesizer.speak_text_async("Goodbye!").get()
        break

    # Step 2: Think (Azure OpenAI)
    conversation_history.append(
        {"role": "user", "content": user_input}
    )

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=conversation_history,
        max_tokens=150
    )

    assistant_reply = response.choices[0].message.content
    conversation_history.append(
        {"role": "assistant", "content": assistant_reply}
    )

    print(f"AI: {assistant_reply}")

    # Step 3: Speak the response
    synthesizer.speak_text_async(assistant_reply).get()

Next Steps

1. Create a Speech resource in the Azure Portal — start with the free tier (F0) and install the Python SDK with pip install azure-cognitiveservices-speech.
2. Try the Speech Studio at speech.microsoft.com — test STT, TTS, and pronunciation assessment directly in the browser with no code required.
3. Build a transcription pipeline — connect real-time STT to your application for live captioning or voice commands.
4. Experiment with SSML — use the Speech Studio’s Audio Content Creation tool to visually design SSML documents before wiring them into code.
5. Explore the samples — the Azure Speech SDK samples repository on GitHub has working examples in Python, C#, Java, JavaScript, and C++.
6. Read the documentation: learn.microsoft.com/azure/ai-services/speech-service