Category: Azure AI

Training a model in a notebook is the easy part. Getting that model into production — with reproducible pipelines, automated retraining, version control, fairness checks, and zero-downtime deployments — is where most ML projects fail. Azure Machine Learning is Microsoft’s enterprise platform for the full machine learning lifecycle, and MLOps is the discipline that ties it all together. This guide covers the end-to-end journey from raw data to monitored production models.

We will walk through workspace provisioning, component-based training pipelines, AutoML, the model registry, managed endpoints, the Responsible AI dashboard, and CI/CD automation with GitHub Actions. Every section includes production-ready code using the Azure ML SDK v2 and CLI v2.

The MLOps lifecycle

MLOps applies DevOps principles to machine learning. The goal is to make model training, validation, deployment, and monitoring as repeatable and automated as building and shipping software. The diagram below shows how each stage feeds into the next in a continuous loop:

Each stage in this lifecycle maps to a specific Azure ML capability. The platform layer at the bottom provides the shared infrastructure that every stage relies on.

Provisioning an Azure ML workspace

The workspace is the top-level resource in Azure ML. It groups compute, data, models, endpoints, and experiments. Behind the scenes it creates a Storage Account, Key Vault, Application Insights, and an optional Container Registry.

CLI v2 setup

# Install the Azure ML CLI v2 extension
az extension add -n ml -y

# Create a resource group
az group create \
  --name rg-mlops-prod \
  --location eastus2

# Create the workspace
az ml workspace create \
  --name mlw-production \
  --resource-group rg-mlops-prod \
  --location eastus2 \
  --display-name "Production ML Workspace"

# Create a compute cluster for training
az ml compute create \
  --name gpu-cluster \
  --resource-group rg-mlops-prod \
  --workspace-name mlw-production \
  --type AmlCompute \
  --size Standard_NC6s_v3 \
  --min-instances 0 \
  --max-instances 4 \
  --idle-time-before-scale-down 120

SDK v2 setup

from azure.ai.ml import MLClient
from azure.ai.ml.entities import Workspace, AmlCompute
from azure.identity import DefaultAzureCredential

# Connect to the workspace
ml_client = MLClient(
    credential=DefaultAzureCredential(),
    subscription_id="your-subscription-id",
    resource_group_name="rg-mlops-prod",
    workspace_name="mlw-production",
)

# Create a compute cluster programmatically
gpu_cluster = AmlCompute(
    name="gpu-cluster",
    size="Standard_NC6s_v3",
    min_instances=0,
    max_instances=4,
    idle_time_before_scale_down=120,
)
ml_client.compute.begin_create_or_update(gpu_cluster)

Datasets and data assets

Azure ML treats data as a first-class, versioned resource. Every dataset registered in the workspace is immutable and traceable, so you can always reproduce which data a given model was trained on.

Registering data assets

from azure.ai.ml.entities import Data
from azure.ai.ml.constants import AssetTypes

# Register a file dataset (e.g., CSV in blob storage)
training_data = Data(
    name="customer-churn-train",
    description="Training data for churn prediction model",
    path="azureml://datastores/workspaceblobstore/paths/data/churn_train.csv",
    type=AssetTypes.URI_FILE,
    version="1",
    tags={"source": "crm-export", "date": "2025-07-01"},
)
ml_client.data.create_or_update(training_data)

# Register a folder dataset (e.g., images)
image_data = Data(
    name="product-images-v2",
    description="Product catalog images for classification",
    path="azureml://datastores/workspaceblobstore/paths/images/products/",
    type=AssetTypes.URI_FOLDER,
    version="2",
)
ml_client.data.create_or_update(image_data)

# Register an MLTable for structured tabular data
tabular_data = Data(
    name="sales-forecast-features",
    path="./data/sales_features/",  # folder with MLTable file
    type=AssetTypes.MLTABLE,
    version="3",
)
ml_client.data.create_or_update(tabular_data)

Training with component-based pipelines

Azure ML pipelines decompose training workflows into reusable components. Each component is a self-contained step (data prep, training, evaluation) with defined inputs, outputs, and a runtime environment. This makes pipelines modular, testable, and easy to maintain.

Defining a training component

from azure.ai.ml import command, Input, Output
from azure.ai.ml.entities import Environment

# Define a curated environment or custom one
env = Environment(
    name="sklearn-training-env",
    image="mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu20.04",
    conda_file="./envs/conda.yml",
)

# Training component
train_component = command(
    name="train_model",
    display_name="Train churn prediction model",
    inputs={
        "train_data": Input(type="uri_file"),
        "learning_rate": Input(type="number", default=0.01),
        "n_estimators": Input(type="integer", default=100),
    },
    outputs={
        "model_output": Output(type="mlflow_model"),
        "metrics_output": Output(type="uri_file"),
    },
    code="./src/train",
    command=("python train.py "
             "--train-data ${{inputs.train_data}} "
             "--learning-rate ${{inputs.learning_rate}} "
             "--n-estimators ${{inputs.n_estimators}} "
             "--model-output ${{outputs.model_output}} "
             "--metrics-output ${{outputs.metrics_output}}"),
    environment=env,
    compute="gpu-cluster",
)

Building the full pipeline

from azure.ai.ml.dsl import pipeline

@pipeline(
    description="End-to-end churn prediction pipeline",
    default_compute="gpu-cluster",
)
def churn_pipeline(raw_data, learning_rate, n_estimators):
    # Step 1: Prepare data
    prep_step = prep_component(
        raw_data=raw_data,
        test_split_ratio=0.2,
    )

    # Step 2: Train model
    train_step = train_component(
        train_data=prep_step.outputs.train_data,
        learning_rate=learning_rate,
        n_estimators=n_estimators,
    )

    # Step 3: Evaluate model
    eval_step = eval_component(
        model=train_step.outputs.model_output,
        test_data=prep_step.outputs.test_data,
    )

    return {
        "trained_model": train_step.outputs.model_output,
        "evaluation_report": eval_step.outputs.report,
    }

# Instantiate and submit
pipeline_job = churn_pipeline(
    raw_data=Input(path="azureml:customer-churn-train:1"),
    learning_rate=0.01,
    n_estimators=200,
)
submitted_job = ml_client.jobs.create_or_update(pipeline_job)
print(f"Pipeline submitted: {submitted_job.studio_url}")

YAML pipeline definition (alternative)

# pipeline.yml — declarative pipeline definition
$schema: https://azuremlschemas.azureedge.net/latest/pipelineJob.schema.json
type: pipeline
display_name: churn-prediction-pipeline
experiment_name: churn-experiment

settings:
  default_compute: azureml:gpu-cluster

inputs:
  raw_data:
    path: azureml:customer-churn-train:1
    type: uri_file

jobs:
  prep_data:
    type: command
    component: ./components/prep/component.yml
    inputs:
      raw_data: ${{parent.inputs.raw_data}}

  train_model:
    type: command
    component: ./components/train/component.yml
    inputs:
      train_data: ${{parent.jobs.prep_data.outputs.train_data}}
      learning_rate: 0.01

  evaluate:
    type: command
    component: ./components/eval/component.yml
    inputs:
      model: ${{parent.jobs.train_model.outputs.model_output}}
      test_data: ${{parent.jobs.prep_data.outputs.test_data}}

AutoML: automated model selection

When you do not have a strong prior on which algorithm to use, AutoML evaluates dozens of models and hyperparameter combinations for you. It handles featurization, model selection, and hyperparameter tuning — then returns the best model with full explainability.

from azure.ai.ml import automl, Input

# Classification task with AutoML
classification_job = automl.classification(
    compute="gpu-cluster",
    experiment_name="churn-automl",
    training_data=Input(
        path="azureml:customer-churn-train:1",
        type="mltable",
    ),
    target_column_name="churned",
    primary_metric="AUC_weighted",
    # Guardrails
    enable_model_explainability=True,
    enable_early_termination=True,
    # Time and iteration limits
    limits=automl.ClassificationLimits(
        timeout_minutes=120,
        max_trials=50,
        max_concurrent_trials=4,
        enable_early_termination=True,
    ),
    # Allowed algorithms (optional filter)
    allowed_training_algorithms=[
        "LightGBM", "XGBoost", "RandomForest",
        "LogisticRegression", "GradientBoosting",
    ],
    # Cross-validation
    n_cross_validations=5,
)

# Submit the AutoML job
returned_job = ml_client.jobs.create_or_update(classification_job)
print(f"AutoML job: {returned_job.studio_url}")

# After completion — retrieve the best model
best_run = ml_client.jobs.get(returned_job.name)
print(f"Best algorithm: {best_run.properties['best_model_algorithm']}")
print(f"Best AUC: {best_run.properties['best_primary_metric_score']}")

AutoML supports classification, regression, time-series forecasting, computer vision (image classification, object detection), and NLP (text classification, NER) tasks. For vision and NLP tasks, it fine-tunes pre-trained deep learning models automatically.

Model registry: versioning and promotion

The model registry is a central repository for all trained models. Every model gets a name, version, and metadata. You can tag models with stage labels to track their lifecycle from development through staging to production.

from azure.ai.ml.entities import Model
from azure.ai.ml.constants import AssetTypes

# Register a model from a training run
model = Model(
    name="churn-prediction",
    version="3",
    path=f"azureml://jobs/{training_job.name}/outputs/model_output",
    type=AssetTypes.MLFLOW_MODEL,
    description="LightGBM churn classifier — AUC 0.94",
    tags={
        "algorithm": "LightGBM",
        "auc": "0.94",
        "dataset_version": "1",
        "stage": "staging",
    },
    properties={
        "training_job": training_job.name,
        "approved_by": "",
    },
)
ml_client.models.create_or_update(model)

# List all versions of a model
versions = ml_client.models.list(name="churn-prediction")
for v in versions:
    print(f"v{v.version} — {v.tags.get('stage', 'dev')} — {v.description}")

# Promote to production (update tags)
prod_model = ml_client.models.get(name="churn-prediction", version="3")
prod_model.tags["stage"] = "production"
prod_model.properties["approved_by"] = "ml-lead@company.com"
ml_client.models.create_or_update(prod_model)

Managed endpoints: online and batch

Azure ML provides two types of managed endpoints that handle infrastructure, scaling, and networking for you.

Online endpoints (real-time inference)

from azure.ai.ml.entities import (
    ManagedOnlineEndpoint,
    ManagedOnlineDeployment,
    CodeConfiguration,
)

# Create the endpoint
endpoint = ManagedOnlineEndpoint(
    name="churn-prediction-endpoint",
    description="Real-time churn prediction API",
    auth_mode="key",
    tags={"environment": "production"},
)
ml_client.online_endpoints.begin_create_or_update(endpoint).result()

# Deploy the model (blue deployment)
blue_deployment = ManagedOnlineDeployment(
    name="blue",
    endpoint_name="churn-prediction-endpoint",
    model="azureml:churn-prediction:3",
    instance_type="Standard_DS3_v2",
    instance_count=2,
    request_settings={
        "request_timeout_ms": 3000,
        "max_concurrent_requests_per_instance": 10,
    },
)
ml_client.online_deployments.begin_create_or_update(blue_deployment).result()

# Route 100% traffic to the blue deployment
endpoint.traffic = {"blue": 100}
ml_client.online_endpoints.begin_create_or_update(endpoint).result()

# Test the endpoint
result = ml_client.online_endpoints.invoke(
    endpoint_name="churn-prediction-endpoint",
    request_file="./test-request.json",
)
print(result)

Blue-green deployment (zero-downtime updates)

# Deploy new model version as "green"
green_deployment = ManagedOnlineDeployment(
    name="green",
    endpoint_name="churn-prediction-endpoint",
    model="azureml:churn-prediction:4",  # newer version
    instance_type="Standard_DS3_v2",
    instance_count=2,
)
ml_client.online_deployments.begin_create_or_update(green_deployment).result()

# Canary: send 10% traffic to green
endpoint.traffic = {"blue": 90, "green": 10}
ml_client.online_endpoints.begin_create_or_update(endpoint).result()

# After validation — shift all traffic to green
endpoint.traffic = {"blue": 0, "green": 100}
ml_client.online_endpoints.begin_create_or_update(endpoint).result()

# Clean up old deployment
ml_client.online_deployments.begin_delete(
    name="blue",
    endpoint_name="churn-prediction-endpoint",
)

Batch endpoints (large-scale scoring)

from azure.ai.ml.entities import BatchEndpoint, BatchDeployment

# Create a batch endpoint for scoring millions of records
batch_endpoint = BatchEndpoint(
    name="churn-batch-scoring",
    description="Score customer churn risk in bulk",
)
ml_client.batch_endpoints.begin_create_or_update(batch_endpoint).result()

batch_deployment = BatchDeployment(
    name="lgbm-v3",
    endpoint_name="churn-batch-scoring",
    model="azureml:churn-prediction:3",
    compute="gpu-cluster",
    mini_batch_size=100,
    max_concurrency_per_instance=2,
    output_action="append_row",
    output_file_name="predictions.csv",
)
ml_client.batch_deployments.begin_create_or_update(batch_deployment).result()

# Invoke the batch endpoint
job = ml_client.batch_endpoints.invoke(
    endpoint_name="churn-batch-scoring",
    input=Input(
        path="azureml://datastores/workspaceblobstore/paths/scoring/july_customers.csv",
        type="uri_file",
    ),
)

Responsible AI dashboard

The Responsible AI (RAI) dashboard provides a unified view of model fairness, interpretability, error analysis, and causal reasoning. It is not optional — it is a requirement for any model that affects people’s outcomes (credit scoring, hiring, healthcare, etc.).

from azure.ai.ml import Input
from azure.ai.ml.entities import (
    ResponsibleAiInsights,
    ErrorAnalysisConfig,
    ExplanationConfig,
    FairnessConfig,
    CausalConfig,
)

# Configure the RAI dashboard components
rai_config = ResponsibleAiInsights(
    model="azureml:churn-prediction:3",
    train_data=Input(path="azureml:customer-churn-train:1", type="mltable"),
    test_data=Input(path="azureml:customer-churn-test:1", type="mltable"),
    target_column="churned",
    components=[
        # Error analysis: find where the model fails
        ErrorAnalysisConfig(
            max_depth=4,
            num_leaves=31,
        ),
        # Explanations: feature importance per prediction
        ExplanationConfig(),
        # Fairness: check for bias across groups
        FairnessConfig(
            sensitive_features=["age_group", "gender", "region"],
            fairness_metric="demographic_parity",
        ),
        # Causal: what-if analysis
        CausalConfig(
            treatment_features=["discount_offered", "support_calls"],
        ),
    ],
)

# Submit the RAI dashboard job
rai_job = ml_client.jobs.create_or_update(rai_config)
print(f"RAI dashboard: {rai_job.studio_url}")

The dashboard answers four critical questions:

• Error analysis — Where does the model make the most mistakes? Which subgroups have the highest error rates?
• Explanations — Which features drive each individual prediction? Is the model relying on the right signals?
• Fairness — Does the model perform equally across demographic groups? Are there disparate impact patterns?
• Causal inference — What would happen if we changed a feature? Does offering a discount actually reduce churn?

CI/CD for ML with GitHub Actions

MLOps automation connects your Git repository to the Azure ML platform. When code is merged to main, the pipeline should automatically retrain, evaluate, and — if metrics pass — deploy the new model.

GitHub Actions workflow

# .github/workflows/mlops-pipeline.yml
name: MLOps Training Pipeline
on:
  push:
    branches: [main]
    paths:
      - "src/train/**"
      - "src/evaluate/**"
      - "data/**"
      - "pipeline.yml"

jobs:
  train-and-evaluate:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read
    steps:
      - uses: actions/checkout@v4

      - name: Azure Login (OIDC)
        uses: azure/login@v2
        with:
          client-id: ${{ secrets.AZURE_CLIENT_ID }}
          tenant-id: ${{ secrets.AZURE_TENANT_ID }}
          subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}

      - name: Install Azure ML CLI
        run: az extension add -n ml -y

      - name: Submit training pipeline
        run: |
          az ml job create \
            --file pipeline.yml \
            --resource-group rg-mlops-prod \
            --workspace-name mlw-production \
            --stream

      - name: Validate metrics
        run: |
          python scripts/check_metrics.py \
            --min-auc 0.90 \
            --min-f1 0.85

  deploy:
    needs: train-and-evaluate
    runs-on: ubuntu-latest
    if: success()
    steps:
      - uses: actions/checkout@v4

      - name: Azure Login (OIDC)
        uses: azure/login@v2
        with:
          client-id: ${{ secrets.AZURE_CLIENT_ID }}
          tenant-id: ${{ secrets.AZURE_TENANT_ID }}
          subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}

      - name: Install Azure ML CLI
        run: az extension add -n ml -y

      - name: Deploy to staging (green)
        run: |
          az ml online-deployment create \
            --file deploy/green-deployment.yml \
            --resource-group rg-mlops-prod \
            --workspace-name mlw-production

      - name: Smoke test
        run: |
          python scripts/smoke_test.py \
            --endpoint churn-prediction-endpoint \
            --deployment green

      - name: Shift traffic
        run: |
          az ml online-endpoint update \
            --name churn-prediction-endpoint \
            --traffic "blue=0 green=100" \
            --resource-group rg-mlops-prod \
            --workspace-name mlw-production

Platform capabilities at a glance

Azure ML vs. the competition

Choose Azure ML when your organization is invested in the Microsoft ecosystem, needs built-in Responsible AI tooling, or requires tight integration with Azure DevOps and GitHub. Choose Databricks when your workflows are Spark-heavy and you want a unified data + ML platform. Each platform has matured significantly — the deciding factor is usually which ecosystem your team already lives in.

Production MLOps checklist

1. Version everything — data assets, environments, pipeline definitions, and models should be in source control with immutable versions.
2. Automate the pipeline — manual training runs do not scale. Use CI/CD triggers so that code changes automatically kick off training and evaluation.
3. Gate deployments on metrics — set minimum thresholds for AUC, F1, precision, or recall. A model that fails the gate never reaches production.
4. Run the Responsible AI dashboard — before any model that affects real people goes live, verify fairness across sensitive groups and review error patterns.
5. Monitor for data drift — production data evolves. Configure drift monitors that alert you when input distributions shift beyond a threshold, triggering retraining.
6. Use blue-green deployments — never replace a production model in-place. Deploy the new version alongside the old, validate, then shift traffic.
7. Scale compute to zero — training clusters and batch compute should auto-scale to zero when idle. GPU hours left running can consume budget rapidly.
8. Secure the workspace — use private endpoints, managed identity (not keys), and RBAC. Store secrets in Key Vault, never in code or environment variables.

Next steps

1. Create a workspace using the CLI v2 and register your first data asset.
2. Build a two-step pipeline — data preparation followed by training — and submit it to a compute cluster.
3. Run AutoML on a tabular dataset to establish a baseline, then compare against your custom model.
4. Deploy to a managed endpoint and test the REST API with sample data.
5. Generate a Responsible AI dashboard and review the fairness analysis before promoting the model.
6. Read the docs: learn.microsoft.com/azure/machine-learning

Every application that works with images, documents, or video eventually needs a way to understand visual content programmatically. Azure AI Vision provides that layer: from generating captions and detecting objects to extracting text from scanned receipts and searching videos by natural language. At its core is Florence, Microsoft’s foundation model for vision that powers multimodal embeddings, visual search, and the latest generation of image analysis capabilities.

This guide walks through the full Azure AI Vision service family, shows you how to integrate each capability with Python, and explains how Florence changes what’s possible with visual understanding at scale.

The Azure AI Vision Ecosystem

Azure AI Vision is not a single API but a collection of services built on shared infrastructure. Image Analysis 4.0 handles general-purpose understanding, the Read API specializes in OCR, and Custom Vision lets you train domain-specific classifiers. Florence ties them together as the foundational model that generates the embeddings and representations used across these services.

Image Analysis 4.0

Image Analysis 4.0 is the latest generation of Azure’s general-purpose image understanding API, powered by Florence. Unlike the older v3.2 API that relied on fixed model architectures, 4.0 uses the Florence foundation model to deliver significantly better accuracy across captioning, tagging, object detection, and smart cropping.

Key capabilities

• Captions — generate a natural-language sentence describing the image content.
• Dense captions — produce captions for every detected region within the image, not just the whole frame.
• Tags — return a list of content tags with confidence scores.
• Object detection — identify and locate objects with bounding boxes.
• Smart cropping — find the most interesting region for automatic thumbnail generation.
• People detection — locate individuals in an image with bounding boxes (no identification).

Caption and tag generation

from azure.ai.vision.imageanalysis import ImageAnalysisClient
from azure.ai.vision.imageanalysis.models import VisualFeatures
from azure.core.credentials import AzureKeyCredential

# Initialize the client
client = ImageAnalysisClient(
    endpoint="https://your-vision.cognitiveservices.azure.com",
    credential=AzureKeyCredential("YOUR_API_KEY"),
)

# Analyze an image for captions, tags, and objects
result = client.analyze(
    image_url="https://example.com/warehouse-photo.jpg",
    visual_features=[
        VisualFeatures.CAPTION,
        VisualFeatures.DENSE_CAPTIONS,
        VisualFeatures.TAGS,
        VisualFeatures.OBJECTS,
    ],
    language="en",
    gender_neutral_caption=True,
)

# Display the main caption
print(f"Caption: {result.caption.text}")
print(f"Confidence: {result.caption.confidence:.2%}")

# Display dense captions for each region
for dc in result.dense_captions.list:
    print(f"  Region: {dc.text} ({dc.confidence:.2%})")
    print(f"  Bounding box: {dc.bounding_box}")

# Display tags sorted by confidence
for tag in sorted(result.tags.list, key=lambda t: t.confidence, reverse=True):
    print(f"  {tag.name}: {tag.confidence:.2%}")

Object detection with bounding boxes

# Detect objects and draw bounding boxes
result = client.analyze(
    image_url="https://example.com/street-scene.jpg",
    visual_features=[VisualFeatures.OBJECTS],
)

for obj in result.objects.list:
    box = obj.bounding_box
    print(
        f"Object: {obj.tags[0].name} "
        f"({obj.tags[0].confidence:.0%}) "
        f"at [{box.x}, {box.y}, {box.width}, {box.height}]"
    )

# Example output:
# Object: car (97%) at [120, 280, 340, 180]
# Object: person (95%) at [450, 150, 80, 220]
# Object: traffic light (89%) at [590, 40, 35, 90]

Florence: The Foundation Model Behind It All

Florence is Microsoft’s large-scale vision foundation model that underpins Image Analysis 4.0. Instead of training separate models for captioning, tagging, and detection, Florence learns a shared visual-language representation from billions of image-text pairs. This approach provides several advantages over previous-generation models.

How Florence works

• Unified architecture — a single model handles captioning, tagging, detection, and embedding generation through different task heads.
• Contrastive learning — trained on massive image-text datasets to align visual and language representations in a shared embedding space.
• Zero-shot transfer — the model can recognize concepts it was never explicitly trained to label, because it understands the semantic relationship between images and text.
• Multimodal embeddings — generate vector representations of images and text that can be compared directly for similarity search.

Multimodal embeddings for visual search

from azure.ai.vision.imageanalysis import ImageAnalysisClient
from azure.core.credentials import AzureKeyCredential
import numpy as np

client = ImageAnalysisClient(
    endpoint="https://your-vision.cognitiveservices.azure.com",
    credential=AzureKeyCredential("YOUR_API_KEY"),
)

# Generate an embedding vector for an image
image_embedding = client.vectorize_image(
    image_url="https://example.com/red-dress.jpg",
    model_version="2024-02-01",
)

# Generate an embedding vector for a text query
text_embedding = client.vectorize_text(
    text="a person wearing a red dress in a garden",
    model_version="2024-02-01",
)

# Compare image and text embeddings via cosine similarity
def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

similarity = cosine_similarity(
    image_embedding.vector, text_embedding.vector
)
print(f"Similarity score: {similarity:.4f}")

# Use case: search a catalog of product images by text query
# Store image embeddings in Azure AI Search or Cosmos DB
# At query time, vectorize the search text and find nearest neighbors

OCR: The Read API

The Read API extracts printed and handwritten text from images, PDFs, and TIFF files. It supports 164 languages, handles rotated text, mixed-language documents, and complex layouts with tables and columns. Unlike simple OCR that returns raw character output, the Read API preserves document structure with line groupings, word boundaries, and confidence scores per word.

Extracting text from an image

from azure.ai.vision.imageanalysis import ImageAnalysisClient
from azure.ai.vision.imageanalysis.models import VisualFeatures
from azure.core.credentials import AzureKeyCredential

client = ImageAnalysisClient(
    endpoint="https://your-vision.cognitiveservices.azure.com",
    credential=AzureKeyCredential("YOUR_API_KEY"),
)

# Extract text using the Read feature
result = client.analyze(
    image_url="https://example.com/receipt.jpg",
    visual_features=[VisualFeatures.READ],
)

# Process extracted text blocks
for block in result.read.blocks:
    for line in block.lines:
        print(f"Line: '{line.text}'")
        print(f"  Bounding polygon: {line.bounding_polygon}")

        # Access individual words with confidence
        for word in line.words:
            print(
                f"  Word: '{word.text}' "
                f"(confidence: {word.confidence:.2%})"
            )

# Output example:
# Line: 'ACME COFFEE SHOP'
#   Word: 'ACME' (confidence: 99.80%)
#   Word: 'COFFEE' (confidence: 99.50%)
#   Word: 'SHOP' (confidence: 99.70%)

Processing local files

# Analyze a local image file instead of a URL
with open("invoice_scan.pdf", "rb") as f:
    image_data = f.read()

result = client.analyze(
    image_data=image_data,
    visual_features=[VisualFeatures.READ],
)

# Combine all extracted text into a single string
full_text = "\n".join(
    line.text
    for block in result.read.blocks
    for line in block.lines
)
print(full_text)

Custom Vision: Training Domain-Specific Models

When the pre-built Image Analysis API does not recognize the specific objects or categories your application needs, Custom Vision lets you train your own classifiers and object detectors with minimal labeled data. You can build production-quality models with as few as 15 images per class.

Two project types

• Image classification — “Is this a defective part or a good part?” Assigns one or more labels to the entire image.
• Object detection — “Where are the cracks in this concrete surface?” Identifies objects with bounding boxes and labels.

Training workflow

from azure.cognitiveservices.vision.customvision.training import (
    CustomVisionTrainingClient,
)
from azure.cognitiveservices.vision.customvision.training.models import (
    ImageFileCreateBatch, ImageFileCreateEntry,
)
from msrest.authentication import ApiKeyCredentials

# Connect to the training endpoint
credentials = ApiKeyCredentials(
    in_headers={"Training-key": "YOUR_TRAINING_KEY"}
)
trainer = CustomVisionTrainingClient(
    endpoint="https://your-cv.cognitiveservices.azure.com",
    credentials=credentials,
)

# Create a classification project
project = trainer.create_project(
    name="quality-inspection",
    classification_type="Multiclass",
)

# Create tags for your categories
tag_good = trainer.create_tag(project.id, "good-part")
tag_defective = trainer.create_tag(project.id, "defective-part")

# Upload training images
image_entries = []
for img_path in good_part_images:
    with open(img_path, "rb") as f:
        image_entries.append(
            ImageFileCreateEntry(
                name=img_path.name,
                contents=f.read(),
                tag_ids=[tag_good.id],
            )
        )

trainer.create_images_from_files(
    project.id,
    ImageFileCreateBatch(images=image_entries),
)

# Train the model
iteration = trainer.train_project(project.id)
print(f"Training status: {iteration.status}")

Once training completes, you can publish the model to a prediction endpoint and call it from your application. Custom Vision also supports exporting models to ONNX, TensorFlow, and CoreML formats for edge deployment on IoT devices, mobile apps, or offline scenarios.

Video Analysis

Azure AI Vision extends beyond still images into video understanding. Two capabilities stand out for production use cases.

Spatial analysis

Spatial analysis processes live video feeds to detect people and track their movement through defined zones. Typical deployments include retail foot traffic counting, occupancy monitoring for safety compliance, and queue length detection. The analysis runs on-premise using a Docker container connected to your cameras, sending only aggregated event data to Azure.

Video retrieval with natural language

Video retrieval uses Florence embeddings to index video content and make it searchable by natural language queries. Instead of manually tagging hours of footage, the service automatically generates embeddings for video frames and lets you search with queries like “person carrying a red box near the loading dock.”

import requests

# Create a video retrieval index
endpoint = "https://your-vision.cognitiveservices.azure.com"
headers = {
    "Ocp-Apim-Subscription-Key": "YOUR_API_KEY",
    "Content-Type": "application/json",
}

# Ingest a video into the index
ingest_payload = {
    "videos": [
        {
            "mode": "add",
            "documentId": "warehouse-cam-01",
            "documentUrl": "https://storage.blob.core.windows.net/videos/cam01.mp4",
        }
    ]
}

response = requests.put(
    f"{endpoint}/computervision/retrieval/indexes/my-index/ingestions/run1",
    headers=headers,
    json=ingest_payload,
    params={"api-version": "2024-02-01"},
)

# Search the indexed video with a natural language query
search_payload = {
    "queryText": "forklift near shelf area",
    "top": 5,
}

results = requests.post(
    f"{endpoint}/computervision/retrieval/indexes/my-index:queryByText",
    headers=headers,
    json=search_payload,
    params={"api-version": "2024-02-01"},
)

for match in results.json()["value"]:
    print(f"Video: {match['documentId']}, "
          f"Timestamp: {match['start']}s - {match['end']}s, "
          f"Relevance: {match['relevance']:.4f}")

Full Capabilities at a Glance

Vision Tiers Comparison

Use Cases by Industry

Production Best Practices

1. Use managed identity instead of API keys — configure DefaultAzureCredential for your deployed services. API keys are convenient during development, but managed identity eliminates the risk of key leakage in production.
2. Batch your requests wisely — the Image Analysis API processes one image per call. For bulk workloads, use Azure Functions or Batch to parallelize calls while respecting rate limits.
3. Cache embedding vectors — Florence embedding calls cost money and add latency. Store vectors in Azure AI Search or Cosmos DB rather than regenerating them on every query.
4. Set confidence thresholds — never trust raw model output blindly. Define minimum confidence scores for tags and detections; surface low-confidence results for human review.
5. Handle image preprocessing — compress oversized images before sending them to the API. The maximum file size is 20 MB, but smaller images reduce latency and cost without sacrificing accuracy for most tasks.
6. Deploy containers for sensitive data — if your images cannot leave your network (medical, defense, classified), run the Vision containers on-premise and keep all processing local.
7. Monitor and set alerts — track API error rates, latency percentiles, and monthly spend in Azure Monitor. Set budget alerts before a runaway pipeline drains your credits.
8. Version your Custom Vision models — always publish new iterations alongside the previous version. Test with held-out data before routing production traffic to the new model.

Next Steps

1. Provision the service — create a Computer Vision resource in the Azure Portal and grab your endpoint and key.
2. Install the SDK — run pip install azure-ai-vision-imageanalysis and try the caption generation sample above.
3. Build a visual search prototype — generate Florence embeddings for 50-100 images, store them in Azure AI Search, and query with natural language.
4. Train a Custom Vision model — pick a domain-specific classification task, upload 15+ images per class, and test the trained model.
5. Explore the documentation — the full API reference is at learn.microsoft.com/azure/ai-services/computer-vision.

Every application that listens, speaks, or bridges language barriers needs a speech engine behind it. Azure AI Speech Services provides that engine: production-grade speech-to-text, natural-sounding text-to-speech with neural voices, real-time translation across dozens of languages, and the ability to create a custom voice that sounds uniquely like your brand. Whether you’re building a call center bot, a multilingual conferencing tool, or an accessibility layer for your product, these APIs handle the hard parts so you can focus on the experience.

This guide walks through the four pillars of Azure AI Speech, with working Python code for each one. You’ll learn how to transcribe audio in real time, generate speech with SSML control, translate conversations across languages on the fly, and understand when Custom Neural Voice makes sense for your project.

Architecture Overview

Azure AI Speech is not a single API but a family of services built on shared neural network infrastructure. Here is how the major components relate to each other and to the broader Azure AI ecosystem:

Key Metrics at a Glance

Speech-to-Text: Transcribing Audio

Speech-to-Text (STT) converts spoken audio into text. Azure supports two modes: real-time recognition for live microphone or stream input, and batch transcription for processing pre-recorded files at scale. Both use the same underlying neural models trained on thousands of hours of multilingual audio.

Setting Up the Azure Speech SDK

Install the SDK and create your Speech resource in the Azure portal before writing any code. You will need the subscription key and region.

pip install azure-cognitiveservices-speech

Real-Time Speech Recognition from Microphone

This example captures audio from the default microphone and prints recognized text as it arrives. The SDK handles silence detection, endpoint detection, and partial results automatically.

import azure.cognitiveservices.speech as speechsdk

# Configure the speech service
speech_config = speechsdk.SpeechConfig(
    subscription="YOUR_SPEECH_KEY",
    region="eastus"
)
speech_config.speech_recognition_language = "en-US"

# Use default microphone as audio input
audio_config = speechsdk.AudioConfig(
    use_default_microphone=True
)

# Create the recognizer
recognizer = speechsdk.SpeechRecognizer(
    speech_config=speech_config,
    audio_config=audio_config
)

print("Speak into your microphone...")
result = recognizer.recognize_once_async().get()

if result.reason == speechsdk.ResultReason.RecognizedSpeech:
    print(f"Recognized: {result.text}")
elif result.reason == speechsdk.ResultReason.NoMatch:
    print("No speech could be recognized.")
elif result.reason == speechsdk.ResultReason.Canceled:
    cancellation = result.cancellation_details
    print(f"Canceled: {cancellation.reason}")

Continuous Recognition for Long Audio

For conversations, meetings, or any audio longer than a few seconds, use continuous recognition. The SDK fires events as it processes the audio stream, giving you both partial (interim) and final results.

import azure.cognitiveservices.speech as speechsdk
import time

speech_config = speechsdk.SpeechConfig(
    subscription="YOUR_SPEECH_KEY",
    region="eastus"
)
speech_config.speech_recognition_language = "en-US"

# Enable detailed output with word-level timestamps
speech_config.request_word_level_timestamps()
speech_config.output_format = speechsdk.OutputFormat.Detailed

# Recognize from an audio file instead of microphone
audio_config = speechsdk.AudioConfig(
    filename="meeting-recording.wav"
)

recognizer = speechsdk.SpeechRecognizer(
    speech_config=speech_config,
    audio_config=audio_config
)

all_results = []

def on_recognized(evt):
    """Called when a final recognition result is received."""
    if evt.result.reason == speechsdk.ResultReason.RecognizedSpeech:
        all_results.append(evt.result.text)
        print(f"RECOGNIZED: {evt.result.text}")

def on_recognizing(evt):
    """Called for interim/partial results."""
    print(f"  [partial]: {evt.result.text}")

def on_canceled(evt):
    print(f"CANCELED: {evt.cancellation_details}")

# Connect event handlers
recognizer.recognized.connect(on_recognized)
recognizer.recognizing.connect(on_recognizing)
recognizer.canceled.connect(on_canceled)

# Start continuous recognition
recognizer.start_continuous_recognition()

# Wait for processing to finish (simple approach)
done = False

def stop_cb(evt):
    global done
    done = True

recognizer.session_stopped.connect(stop_cb)
recognizer.canceled.connect(stop_cb)

while not done:
    time.sleep(0.5)

recognizer.stop_continuous_recognition()

# Full transcript
transcript = " ".join(all_results)
print(f"\nFull transcript:\n{transcript}")

Transcribing from an Audio File (Batch API)

For offline processing of large audio archives, the batch transcription REST API lets you submit files stored in Azure Blob Storage and retrieve the results later. This is ideal for call center recordings, podcast transcripts, or legal depositions.

import requests
import json

SPEECH_KEY = "YOUR_SPEECH_KEY"
REGION = "eastus"
ENDPOINT = f"https://{REGION}.api.cognitive.microsoft.com"

# Submit a batch transcription job
headers = {
    "Ocp-Apim-Subscription-Key": SPEECH_KEY,
    "Content-Type": "application/json"
}

body = {
    "contentUrls": [
        "https://mystorage.blob.core.windows.net/audio/call01.wav",
        "https://mystorage.blob.core.windows.net/audio/call02.wav"
    ],
    "locale": "en-US",
    "displayName": "Call Center Batch Job",
    "properties": {
        "wordLevelTimestampsEnabled": True,
        "diarizationEnabled": True,
        "punctuationMode": "DictatedAndAutomatic"
    }
}

response = requests.post(
    f"{ENDPOINT}/speechtotext/v3.2/transcriptions",
    headers=headers,
    json=body
)

transcription = response.json()
print(f"Job created: {transcription['self']}")
print(f"Status: {transcription['status']}")

Text-to-Speech: Generating Natural Audio

Text-to-Speech (TTS) converts written text into lifelike spoken audio using neural networks. Azure offers over 500 neural voices across 100+ languages. The output quality is remarkably close to human speech, with natural prosody, intonation, and pacing.

Basic Speech Synthesis

import azure.cognitiveservices.speech as speechsdk

speech_config = speechsdk.SpeechConfig(
    subscription="YOUR_SPEECH_KEY",
    region="eastus"
)

# Select a neural voice
speech_config.speech_synthesis_voice_name = "en-US-JennyNeural"

# Output to default speaker
audio_config = speechsdk.audio.AudioOutputConfig(
    use_default_speaker=True
)

synthesizer = speechsdk.SpeechSynthesizer(
    speech_config=speech_config,
    audio_config=audio_config
)

result = synthesizer.speak_text_async(
    "Welcome to Azure AI Speech Services. "
    "This is a neural voice speaking naturally."
).get()

if result.reason == speechsdk.ResultReason.SynthesizingAudioCompleted:
    print("Speech synthesized successfully.")
    print(f"Audio duration: {result.audio_duration}")

Saving Synthesis Output to a File

# Save directly to a WAV file instead of playing on speakers
audio_config = speechsdk.audio.AudioOutputConfig(
    filename="output.wav"
)

synthesizer = speechsdk.SpeechSynthesizer(
    speech_config=speech_config,
    audio_config=audio_config
)

result = synthesizer.speak_text_async(
    "This audio will be saved to a WAV file."
).get()

print("Audio saved to output.wav")

Advanced Control with SSML

Speech Synthesis Markup Language (SSML) gives you fine-grained control over how the voice speaks: adjust rate, pitch, volume, add pauses, switch voices mid-sentence, or apply speaking styles like “cheerful” or “empathetic.”

<!-- SSML example with multiple voices and styles -->
<speak version="1.0"
       xmlns="http://www.w3.org/2001/10/synthesis"
       xmlns:mstts="http://www.w3.org/2001/mstts"
       xml:lang="en-US">

  <voice name="en-US-JennyNeural">
    <!-- Cheerful greeting -->
    <mstts:express-as style="cheerful">
      Welcome to our customer service line!
      We're happy to help you today.
    </mstts:express-as>

    <break time="500ms"/>

    <!-- Normal pace for instructions -->
    <prosody rate="-10%" pitch="+5%">
      For billing inquiries, press one.
      For technical support, press two.
    </prosody>

    <!-- Emphasize important info -->
    Your reference number is
    <say-as interpret-as="characters">ABC123</say-as>.
  </voice>

  <!-- Switch to a different voice -->
  <voice name="en-US-GuyNeural">
    <mstts:express-as style="empathetic">
      I understand your concern. Let me look
      into this for you right away.
    </mstts:express-as>
  </voice>

</speak>

Synthesizing SSML with Python

ssml_text = """
<speak version="1.0"
       xmlns="http://www.w3.org/2001/10/synthesis"
       xmlns:mstts="http://www.w3.org/2001/mstts"
       xml:lang="en-US">
  <voice name="en-US-AriaNeural">
    <mstts:express-as style="newscast-formal">
      Today in technology news: Azure AI Speech Services
      now supports over five hundred neural voices across
      more than one hundred languages and locales.
    </mstts:express-as>
    <break time="300ms"/>
    <prosody rate="medium" volume="loud">
      This marks a significant milestone in the
      democratization of speech AI technology.
    </prosody>
  </voice>
</speak>
"""

result = synthesizer.speak_ssml_async(ssml_text).get()

if result.reason == speechsdk.ResultReason.SynthesizingAudioCompleted:
    print("SSML synthesis completed.")
elif result.reason == speechsdk.ResultReason.Canceled:
    details = result.cancellation_details
    print(f"Synthesis canceled: {details.reason}")
    print(f"Error: {details.error_details}")

Available Speaking Styles

Not all neural voices support all styles. Here are the most commonly used styles with the voices that support them:

Real-Time Speech Translation

Speech Translation combines speech recognition and machine translation into a single streaming pipeline. Audio goes in one language, and you get text (or synthesized audio) out in another language, all in real time. This powers scenarios like live meeting interpreters, multilingual kiosks, and cross-language customer support.

Translating Speech Between Languages

import azure.cognitiveservices.speech as speechsdk

# Configure translation
translation_config = speechsdk.translation.SpeechTranslationConfig(
    subscription="YOUR_SPEECH_KEY",
    region="eastus"
)

# Source language: what the speaker is saying
translation_config.speech_recognition_language = "en-US"

# Target languages: translate to these simultaneously
translation_config.add_target_language("es")   # Spanish
translation_config.add_target_language("fr")   # French
translation_config.add_target_language("de")   # German
translation_config.add_target_language("ja")   # Japanese

# Optional: synthesize the translated output
translation_config.voice_name = "es-ES-ElviraNeural"

audio_config = speechsdk.AudioConfig(
    use_default_microphone=True
)

# Create the translation recognizer
recognizer = speechsdk.translation.TranslationRecognizer(
    translation_config=translation_config,
    audio_config=audio_config
)

print("Speak in English. Translations will appear in real time...")
print("=" * 60)

result = recognizer.recognize_once_async().get()

if result.reason == speechsdk.ResultReason.TranslatedSpeech:
    print(f"Recognized:  {result.text}")
    print(f"Spanish:     {result.translations['es']}")
    print(f"French:      {result.translations['fr']}")
    print(f"German:      {result.translations['de']}")
    print(f"Japanese:    {result.translations['ja']}")

Continuous Translation for Meetings

For longer sessions like meetings or conferences, use continuous translation with event handlers. This approach streams partial translations as the speaker talks, giving listeners a near-real-time experience.

import azure.cognitiveservices.speech as speechsdk
import time
import json

translation_config = speechsdk.translation.SpeechTranslationConfig(
    subscription="YOUR_SPEECH_KEY",
    region="eastus"
)
translation_config.speech_recognition_language = "en-US"
translation_config.add_target_language("es")
translation_config.add_target_language("pt")

audio_config = speechsdk.AudioConfig(
    use_default_microphone=True
)

recognizer = speechsdk.translation.TranslationRecognizer(
    translation_config=translation_config,
    audio_config=audio_config
)

# Store translations for export
translation_log = []

def on_translated(evt):
    if evt.result.reason == speechsdk.ResultReason.TranslatedSpeech:
        entry = {
            "original": evt.result.text,
            "translations": dict(evt.result.translations),
            "offset": evt.result.offset,
            "duration": evt.result.duration
        }
        translation_log.append(entry)
        print(f"[EN] {evt.result.text}")
        print(f"[ES] {evt.result.translations['es']}")
        print(f"[PT] {evt.result.translations['pt']}")
        print()

recognizer.recognized.connect(on_translated)

# Start continuous translation
recognizer.start_continuous_recognition()
print("Meeting translation active. Press Ctrl+C to stop.\n")

try:
    while True:
        time.sleep(0.5)
except KeyboardInterrupt:
    recognizer.stop_continuous_recognition()
    # Save the full translation log
    with open("translation_log.json", "w") as f:
        json.dump(translation_log, f, indent=2)
    print(f"\nSaved {len(translation_log)} entries to translation_log.json")

Custom Neural Voice

Custom Neural Voice (CNV) lets organizations create a unique, branded synthetic voice that sounds like a specific person. Instead of choosing from the catalog of prebuilt voices, you record a professional voice talent, upload the training data, and Azure trains a neural TTS model on that specific voice.

When to Use Custom Neural Voice

• Brand consistency — a banking app that always speaks in the same recognizable tone, no matter the platform.
• Accessibility — recreating a user’s personal voice for people at risk of losing their ability to speak (Personal Voice).
• Content creation — audiobook narration, e-learning, or video voiceovers at scale without booking studio time for every update.
• IVR systems — interactive voice response for call centers that matches your company’s personality rather than sounding generic.

The Training Pipeline

Building a Custom Neural Voice follows these steps:

1. Record training data — typically 300-2,000 utterances (about 30 minutes to 2 hours of clean studio audio) from your chosen voice talent.
2. Prepare transcripts — each audio file needs a matching text transcript with exact sentence-level alignment.
3. Upload and validate — the Azure Speech Studio inspects audio quality, signal-to-noise ratio, and transcript accuracy.
4. Train the model — Azure’s neural network trains on your data. Training typically completes in 2-4 hours.
5. Test and deploy — evaluate the voice in the Speech Studio playground, then deploy it as an endpoint you can call from the SDK.
6. Integrate via SSML — use your custom voice name in SSML or SDK calls exactly like any prebuilt neural voice.

# Using a Custom Neural Voice is identical to using a prebuilt voice
# Just reference your custom voice's deployment name

speech_config = speechsdk.SpeechConfig(
    subscription="YOUR_SPEECH_KEY",
    region="eastus"
)

# Set the custom voice endpoint
speech_config.endpoint_id = "YOUR_CUSTOM_VOICE_ENDPOINT_ID"
speech_config.speech_synthesis_voice_name = "MyBrandVoice"

synthesizer = speechsdk.SpeechSynthesizer(
    speech_config=speech_config
)

# SSML with your custom voice
ssml = """
<speak version="1.0"
       xmlns="http://www.w3.org/2001/10/synthesis"
       xmlns:mstts="http://www.w3.org/2001/mstts"
       xml:lang="en-US">
  <voice name="MyBrandVoice">
    Hello, thank you for calling Contoso. How can I help you today?
  </voice>
</speak>
"""

result = synthesizer.speak_ssml_async(ssml).get()
print(f"Custom voice synthesis: {result.reason}")

Capabilities at a Glance

Service Tiers and Pricing

Azure AI Speech uses a pay-as-you-go model based on the number of audio hours processed. Pricing varies by feature and whether you use standard or custom models.

Real-World Use Cases

Production Best Practices

1. Always handle cancellation errors — network interruptions, expired keys, and quota limits all surface as cancellation events. Log the cancellation_details reason and error code to diagnose issues quickly.
2. Use connection pooling — creating a new SpeechRecognizer or SpeechSynthesizer for every request wastes time on WebSocket handshakes. Reuse objects across requests within a session.
3. Choose the right region — deploy your Speech resource in the region closest to your users. Audio streaming is latency-sensitive, and every 50ms of extra round-trip adds noticeable delay.
4. Set audio format explicitly — for TTS, request compressed formats like Audio48Khz192KBitRateMonoMp3 to reduce bandwidth. WAV is uncompressed and fine for local playback, but wasteful over the network.
5. Enable profanity filtering when appropriate — the STT engine can mask, remove, or pass through profanity. Set this based on your content policy: speech_config.set_profanity(ProfanityOption.Masked).
6. Implement retry logic with exponential backoff — transient failures happen. Retry on HTTP 429 (rate limit) and 503 (service unavailable) with jittered backoff, but never retry on 401 (bad key) or 400 (bad request).
7. Monitor usage and set budget alerts — a bug in continuous recognition can leave a session running indefinitely, accumulating costs. Set up Azure Monitor alerts for unexpected usage spikes.
8. Test with diverse audio conditions — your lab microphone sounds perfect, but production audio includes background noise, echo, accents, and variable volume. Test with realistic samples before launch.

Integrating Speech with Azure OpenAI

One of the most powerful patterns combines Speech Services with Azure OpenAI to create voice-enabled AI assistants. The user speaks, STT transcribes the input, OpenAI generates a response, and TTS speaks the answer back naturally.

import azure.cognitiveservices.speech as speechsdk
from openai import AzureOpenAI

# --- Configuration ---
SPEECH_KEY = "YOUR_SPEECH_KEY"
SPEECH_REGION = "eastus"
OPENAI_ENDPOINT = "https://your-openai.openai.azure.com/"
OPENAI_KEY = "YOUR_OPENAI_KEY"

# Set up Speech
speech_config = speechsdk.SpeechConfig(
    subscription=SPEECH_KEY,
    region=SPEECH_REGION
)
speech_config.speech_recognition_language = "en-US"
speech_config.speech_synthesis_voice_name = "en-US-JennyNeural"

# Set up Azure OpenAI
client = AzureOpenAI(
    azure_endpoint=OPENAI_ENDPOINT,
    api_key=OPENAI_KEY,
    api_version="2024-06-01"
)

# Create recognizer and synthesizer
recognizer = speechsdk.SpeechRecognizer(speech_config=speech_config)
synthesizer = speechsdk.SpeechSynthesizer(speech_config=speech_config)

conversation_history = [
    {"role": "system", "content": "You are a helpful voice assistant. "
     "Keep responses concise (2-3 sentences) since they "
     "will be spoken aloud."}
]

print("Voice assistant ready. Speak to begin...")

while True:
    # Step 1: Listen
    result = recognizer.recognize_once_async().get()
    if result.reason != speechsdk.ResultReason.RecognizedSpeech:
        continue

    user_input = result.text
    print(f"\nYou: {user_input}")

    if "goodbye" in user_input.lower():
        synthesizer.speak_text_async("Goodbye!").get()
        break

    # Step 2: Think (Azure OpenAI)
    conversation_history.append(
        {"role": "user", "content": user_input}
    )

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=conversation_history,
        max_tokens=150
    )

    assistant_reply = response.choices[0].message.content
    conversation_history.append(
        {"role": "assistant", "content": assistant_reply}
    )

    print(f"AI: {assistant_reply}")

    # Step 3: Speak the response
    synthesizer.speak_text_async(assistant_reply).get()

Next Steps

1. Create a Speech resource in the Azure Portal — start with the free tier (F0) and install the Python SDK with pip install azure-cognitiveservices-speech.
2. Try the Speech Studio at speech.microsoft.com — test STT, TTS, and pronunciation assessment directly in the browser with no code required.
3. Build a transcription pipeline — connect real-time STT to your application for live captioning or voice commands.
4. Experiment with SSML — use the Speech Studio’s Audio Content Creation tool to visually design SSML documents before wiring them into code.
5. Explore the samples — the Azure Speech SDK samples repository on GitHub has working examples in Python, C#, Java, JavaScript, and C++.
6. Read the documentation: learn.microsoft.com/azure/ai-services/speech-service

Every organization deals with paperwork — invoices, receipts, contracts, ID cards, tax forms. Extracting data from these documents manually is slow and error-prone. Azure AI Document Intelligence uses machine learning to read, understand, and extract structured data from documents automatically.

This article covers what the service offers, which models to use for different scenarios, and how to process your first document with just a few lines of code.

What is Azure AI Document Intelligence?

Azure AI Document Intelligence (formerly Form Recognizer) is an AI service that extracts text, key-value pairs, tables, and structures from documents. It handles:

• Scanned PDFs and images (via built-in OCR).
• Digital PDFs with complex layouts.
• Photos of receipts, business cards, and forms.
• Handwritten text in multiple languages.

Unlike basic OCR that just reads text, Document Intelligence understands the structure of your documents — it knows which text is a header, which values belong to which fields, and how tables are organized.

Prebuilt models vs. custom models

The service comes with two categories of models:

Prebuilt models

Ready to use with zero training. Microsoft has already trained these on millions of documents:

Custom models

When your documents don’t match any prebuilt model — like internal forms, proprietary reports, or industry-specific paperwork — you can train a custom model using as few as 5 labeled samples.

Real-world use cases

Accounts payable automation

Process incoming invoices automatically: extract vendor, amount, line items, and PO numbers, then push the data into your ERP system. Teams that used to spend hours on manual data entry can process hundreds of invoices in minutes.

Expense report processing

Employees snap photos of receipts. Document Intelligence reads the merchant, date, total, and category, then populates the expense report automatically.

Contract analysis

Extract key clauses, dates, and parties from contracts. Combine with Azure OpenAI to summarize terms or flag unusual conditions.

Healthcare intake

Read insurance cards and patient forms at check-in. Extract member IDs, group numbers, and patient details to reduce front-desk workload and data entry errors.

Getting started

Step 1: Create the resource

1. Go to the Azure Portal and search for “Document Intelligence”.
2. Click Create.
3. Choose your subscription, resource group, region, and pricing tier.
4. Click Review + Create.

Step 2: Try it in Document Intelligence Studio

Before writing any code, explore Document Intelligence Studio at documentintelligence.ai.azure.com. Upload a sample document, pick a prebuilt model, and see the extracted data instantly. It’s the fastest way to evaluate whether a prebuilt model fits your documents.

Step 3: Analyze a document with Python

Here’s how to extract data from an invoice using Python:

from azure.ai.documentintelligence import DocumentIntelligenceClient
from azure.core.credentials import AzureKeyCredential

client = DocumentIntelligenceClient(
    endpoint="https://YOUR-RESOURCE.cognitiveservices.azure.com/",
    credential=AzureKeyCredential("YOUR-API-KEY")
)

# Analyze an invoice from a URL
poller = client.begin_analyze_document(
    "prebuilt-invoice",
    analyze_request={"url_source": "https://example.com/invoice.pdf"}
)
result = poller.result()

for doc in result.documents:
    print(f"Vendor: {doc.fields['VendorName'].content}")
    print(f"Total: {doc.fields['InvoiceTotal'].content}")
    print(f"Date:  {doc.fields['InvoiceDate'].content}")

    if "Items" in doc.fields:
        for item in doc.fields["Items"].value:
            desc = item.value["Description"].content
            amount = item.value["Amount"].content
            print(f"  - {desc}: {amount}")

Install the SDK:

pip install azure-ai-documentintelligence

Step 4: Extract tables and layout

For documents that don’t match a prebuilt model, the Layout model extracts all text, tables, and structure:

poller = client.begin_analyze_document(
    "prebuilt-layout",
    analyze_request={"url_source": "https://example.com/report.pdf"}
)
result = poller.result()

# Extract tables
for table in result.tables:
    print(f"Table: {table.row_count} rows x {table.column_count} columns")
    for cell in table.cells:
        print(f"  [{cell.row_index},{cell.column_index}] {cell.content}")

Combining with Azure OpenAI

Document Intelligence and Azure OpenAI are a powerful combination. A common pattern:

1. Extract text and tables from a PDF using Document Intelligence.
2. Send the extracted content to GPT-4o with a prompt like “Summarize this contract” or “Find all penalties and deadlines.”
3. Get structured, actionable output that would have taken hours to compile manually.

This is especially effective for contracts, financial reports, and regulatory filings where you need both extraction accuracy and natural language understanding.

Pricing overview

At these prices, processing 1,000 invoices costs about $10. Compare that to the cost of manual data entry and the ROI becomes obvious.

Next steps

1. Open Document Intelligence Studio and upload a real document to see extraction results instantly.
2. Start with a prebuilt model — invoices and receipts cover the most common automation scenarios.
3. Train a custom model if your document type isn’t covered — 5 samples is all you need to start.
4. Read the official docs: learn.microsoft.com/azure/ai-services/document-intelligence

Conclusion

Azure AI Document Intelligence eliminates the tedious work of reading and typing data from documents. Whether you’re processing 10 invoices a week or 10,000, the service scales to match your workload. Combined with Azure OpenAI, it turns raw documents into structured, actionable data — the kind of automation that delivers measurable ROI from day one.

Next up, we’ll explore Copilot Studio, Microsoft’s platform for building custom AI assistants without writing code.

Imagine asking a question in plain English and getting an accurate answer pulled directly from your company’s own documents. That’s the power behind Retrieval-Augmented Generation (RAG), and Azure AI Search is the service that makes it possible at scale on Microsoft Azure.

In this guide, you’ll learn what Azure AI Search is, how RAG works, and how to build your first search-powered AI solution.

What is Azure AI Search?

Azure AI Search (formerly Azure Cognitive Search) is a fully managed search service on Azure. It goes far beyond traditional keyword search by offering:

• Full-text search: classic keyword matching with filters, facets, and scoring profiles.
• Vector search: find results based on meaning, not just exact words.
• Hybrid search: combine keyword and vector search for the best of both worlds.
• Semantic ranking: a built-in AI layer that re-ranks results by relevance.
• Integrated vectorization: automatically generate embeddings from your content using Azure OpenAI.

Think of it as the intelligent retrieval engine sitting between your data and your AI models.

What is RAG and why does it matter?

RAG stands for Retrieval-Augmented Generation. It’s a pattern that solves one of the biggest challenges with large language models: they don’t know your private data.

Here’s how it works:

1. The user asks a question (e.g., “What’s our refund policy for enterprise clients?”).
2. The system searches your documents using Azure AI Search to find the most relevant content.
3. The retrieved content is sent to a language model (like GPT-4o) along with the question.
4. The model generates a grounded answer based on your actual documents, not its general training data.

The result? Accurate, up-to-date answers that cite your own sources, with far fewer hallucinations.

RAG vs. fine-tuning: when to use each

For most enterprise use cases, RAG is the recommended starting point. Fine-tuning is complementary, not a replacement.

Key components of a RAG solution on Azure

1. Data sources

Azure AI Search can pull data from Azure Blob Storage, Azure SQL Database, Cosmos DB, SharePoint, and many other sources using built-in indexers.

2. Search index

Your data is processed and stored in a search index. During indexing, you can apply skillsets that enrich the data — extract text from PDFs, detect languages, split documents into chunks, and generate vector embeddings.

3. Query pipeline

When a user asks a question, the query is converted into a vector (using the same embedding model), and Azure AI Search retrieves the most relevant chunks using hybrid search.

4. Language model

The retrieved chunks are passed to Azure OpenAI (GPT-4o or similar) as context, and the model generates a natural language answer.

Setting up your first RAG pipeline

Step 1: Create an Azure AI Search resource

1. Go to the Azure Portal (portal.azure.com).
2. Search for “AI Search” and click Create.
3. Select your subscription, resource group, and region.
4. Choose a pricing tier (Free works for testing, Basic for small production workloads).
5. Click Review + Create.

Step 2: Upload your documents

Upload your files (PDFs, Word docs, text files) to an Azure Blob Storage container. This will be your data source.

Step 3: Create an index with integrated vectorization

In the Azure portal, use the “Import and vectorize data” wizard on your AI Search resource. It will:

• Connect to your Blob Storage.
• Chunk your documents automatically.
• Generate embeddings using an Azure OpenAI embedding model.
• Create the search index with both text and vector fields.

Step 4: Query with Python

from azure.search.documents import SearchClient
from azure.core.credentials import AzureKeyCredential

client = SearchClient(
    endpoint="https://YOUR-SEARCH-SERVICE.search.windows.net",
    index_name="your-index",
    credential=AzureKeyCredential("YOUR-API-KEY")
)

results = client.search(
    search_text="refund policy for enterprise",
    top=3,
    query_type="semantic",
    semantic_configuration_name="my-semantic-config"
)

for result in results:
    print(result["chunk"], result["@search.score"])

Step 5: Connect to Azure OpenAI for the full RAG flow

import openai

# 1. Retrieve relevant chunks (from the search above)
context = "\n\n".join([r["chunk"] for r in results])

# 2. Send to Azure OpenAI with the retrieved context
ai_client = openai.AzureOpenAI(
    api_key="YOUR-OPENAI-KEY",
    api_version="2024-10-21",
    azure_endpoint="https://YOUR-RESOURCE.openai.azure.com/"
)

response = ai_client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": f"Answer using only this context:\n\n{context}"},
        {"role": "user", "content": "What's the refund policy for enterprise clients?"}
    ]
)

print(response.choices[0].message.content)

To install the required libraries:

pip install azure-search-documents openai

Pricing overview

Vector search and semantic ranking are included at no extra charge on Basic and above. Embedding generation costs depend on your Azure OpenAI pricing.

Next steps

1. Try the “Import and vectorize data” wizard in the Azure portal — it’s the fastest way to see RAG in action.
2. Explore Azure AI Studio: it provides a visual RAG pipeline builder with built-in chat evaluation.
3. Experiment with chunking strategies: document splitting has a big impact on answer quality.
4. Check the official docs: learn.microsoft.com/azure/search

Conclusion

Azure AI Search combined with RAG lets you build AI solutions that actually know your data. Instead of hoping a language model has the right answer, you give it the right context. It’s the most practical way to bring generative AI into your organization without exposing sensitive data or dealing with hallucinations.

In the next article, we’ll look at Azure AI Document Intelligence, the service that extracts structured data from forms, invoices, and documents automatically.