Azure AI Vision & Florence

Every application that works with images, documents, or video eventually needs a way to understand visual content programmatically. Azure AI Vision provides that layer: from generating captions and detecting objects to extracting text from scanned receipts and searching videos by natural language. At its core is Florence, Microsoft’s foundation model for vision that powers multimodal embeddings, visual search, and the latest generation of image analysis capabilities.

This guide walks through the full Azure AI Vision service family, shows you how to integrate each capability with Python, and explains how Florence changes what’s possible with visual understanding at scale.

The Azure AI Vision Ecosystem

Azure AI Vision is not a single API but a collection of services built on shared infrastructure. Image Analysis 4.0 handles general-purpose understanding, the Read API specializes in OCR, and Custom Vision lets you train domain-specific classifiers. Florence ties them together as the foundational model that generates the embeddings and representations used across these services.

Image Analysis 4.0

Image Analysis 4.0 is the latest generation of Azure’s general-purpose image understanding API, powered by Florence. Unlike the older v3.2 API that relied on fixed model architectures, 4.0 uses the Florence foundation model to deliver significantly better accuracy across captioning, tagging, object detection, and smart cropping.

Key capabilities

• Captions — generate a natural-language sentence describing the image content.
• Dense captions — produce captions for every detected region within the image, not just the whole frame.
• Tags — return a list of content tags with confidence scores.
• Object detection — identify and locate objects with bounding boxes.
• Smart cropping — find the most interesting region for automatic thumbnail generation.
• People detection — locate individuals in an image with bounding boxes (no identification).

Caption and tag generation

from azure.ai.vision.imageanalysis import ImageAnalysisClient
from azure.ai.vision.imageanalysis.models import VisualFeatures
from azure.core.credentials import AzureKeyCredential

# Initialize the client
client = ImageAnalysisClient(
    endpoint="https://your-vision.cognitiveservices.azure.com",
    credential=AzureKeyCredential("YOUR_API_KEY"),
)

# Analyze an image for captions, tags, and objects
result = client.analyze(
    image_url="https://example.com/warehouse-photo.jpg",
    visual_features=[
        VisualFeatures.CAPTION,
        VisualFeatures.DENSE_CAPTIONS,
        VisualFeatures.TAGS,
        VisualFeatures.OBJECTS,
    ],
    language="en",
    gender_neutral_caption=True,
)

# Display the main caption
print(f"Caption: {result.caption.text}")
print(f"Confidence: {result.caption.confidence:.2%}")

# Display dense captions for each region
for dc in result.dense_captions.list:
    print(f"  Region: {dc.text} ({dc.confidence:.2%})")
    print(f"  Bounding box: {dc.bounding_box}")

# Display tags sorted by confidence
for tag in sorted(result.tags.list, key=lambda t: t.confidence, reverse=True):
    print(f"  {tag.name}: {tag.confidence:.2%}")

Object detection with bounding boxes

# Detect objects and draw bounding boxes
result = client.analyze(
    image_url="https://example.com/street-scene.jpg",
    visual_features=[VisualFeatures.OBJECTS],
)

for obj in result.objects.list:
    box = obj.bounding_box
    print(
        f"Object: {obj.tags[0].name} "
        f"({obj.tags[0].confidence:.0%}) "
        f"at [{box.x}, {box.y}, {box.width}, {box.height}]"
    )

# Example output:
# Object: car (97%) at [120, 280, 340, 180]
# Object: person (95%) at [450, 150, 80, 220]
# Object: traffic light (89%) at [590, 40, 35, 90]

Florence: The Foundation Model Behind It All

Florence is Microsoft’s large-scale vision foundation model that underpins Image Analysis 4.0. Instead of training separate models for captioning, tagging, and detection, Florence learns a shared visual-language representation from billions of image-text pairs. This approach provides several advantages over previous-generation models.

How Florence works

• Unified architecture — a single model handles captioning, tagging, detection, and embedding generation through different task heads.
• Contrastive learning — trained on massive image-text datasets to align visual and language representations in a shared embedding space.
• Zero-shot transfer — the model can recognize concepts it was never explicitly trained to label, because it understands the semantic relationship between images and text.
• Multimodal embeddings — generate vector representations of images and text that can be compared directly for similarity search.

Multimodal embeddings for visual search

from azure.ai.vision.imageanalysis import ImageAnalysisClient
from azure.core.credentials import AzureKeyCredential
import numpy as np

client = ImageAnalysisClient(
    endpoint="https://your-vision.cognitiveservices.azure.com",
    credential=AzureKeyCredential("YOUR_API_KEY"),
)

# Generate an embedding vector for an image
image_embedding = client.vectorize_image(
    image_url="https://example.com/red-dress.jpg",
    model_version="2024-02-01",
)

# Generate an embedding vector for a text query
text_embedding = client.vectorize_text(
    text="a person wearing a red dress in a garden",
    model_version="2024-02-01",
)

# Compare image and text embeddings via cosine similarity
def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

similarity = cosine_similarity(
    image_embedding.vector, text_embedding.vector
)
print(f"Similarity score: {similarity:.4f}")

# Use case: search a catalog of product images by text query
# Store image embeddings in Azure AI Search or Cosmos DB
# At query time, vectorize the search text and find nearest neighbors

OCR: The Read API

The Read API extracts printed and handwritten text from images, PDFs, and TIFF files. It supports 164 languages, handles rotated text, mixed-language documents, and complex layouts with tables and columns. Unlike simple OCR that returns raw character output, the Read API preserves document structure with line groupings, word boundaries, and confidence scores per word.

Extracting text from an image

from azure.ai.vision.imageanalysis import ImageAnalysisClient
from azure.ai.vision.imageanalysis.models import VisualFeatures
from azure.core.credentials import AzureKeyCredential

client = ImageAnalysisClient(
    endpoint="https://your-vision.cognitiveservices.azure.com",
    credential=AzureKeyCredential("YOUR_API_KEY"),
)

# Extract text using the Read feature
result = client.analyze(
    image_url="https://example.com/receipt.jpg",
    visual_features=[VisualFeatures.READ],
)

# Process extracted text blocks
for block in result.read.blocks:
    for line in block.lines:
        print(f"Line: '{line.text}'")
        print(f"  Bounding polygon: {line.bounding_polygon}")

        # Access individual words with confidence
        for word in line.words:
            print(
                f"  Word: '{word.text}' "
                f"(confidence: {word.confidence:.2%})"
            )

# Output example:
# Line: 'ACME COFFEE SHOP'
#   Word: 'ACME' (confidence: 99.80%)
#   Word: 'COFFEE' (confidence: 99.50%)
#   Word: 'SHOP' (confidence: 99.70%)

Processing local files

# Analyze a local image file instead of a URL
with open("invoice_scan.pdf", "rb") as f:
    image_data = f.read()

result = client.analyze(
    image_data=image_data,
    visual_features=[VisualFeatures.READ],
)

# Combine all extracted text into a single string
full_text = "\n".join(
    line.text
    for block in result.read.blocks
    for line in block.lines
)
print(full_text)

Custom Vision: Training Domain-Specific Models

When the pre-built Image Analysis API does not recognize the specific objects or categories your application needs, Custom Vision lets you train your own classifiers and object detectors with minimal labeled data. You can build production-quality models with as few as 15 images per class.

Two project types

• Image classification — “Is this a defective part or a good part?” Assigns one or more labels to the entire image.
• Object detection — “Where are the cracks in this concrete surface?” Identifies objects with bounding boxes and labels.

Training workflow

from azure.cognitiveservices.vision.customvision.training import (
    CustomVisionTrainingClient,
)
from azure.cognitiveservices.vision.customvision.training.models import (
    ImageFileCreateBatch, ImageFileCreateEntry,
)
from msrest.authentication import ApiKeyCredentials

# Connect to the training endpoint
credentials = ApiKeyCredentials(
    in_headers={"Training-key": "YOUR_TRAINING_KEY"}
)
trainer = CustomVisionTrainingClient(
    endpoint="https://your-cv.cognitiveservices.azure.com",
    credentials=credentials,
)

# Create a classification project
project = trainer.create_project(
    name="quality-inspection",
    classification_type="Multiclass",
)

# Create tags for your categories
tag_good = trainer.create_tag(project.id, "good-part")
tag_defective = trainer.create_tag(project.id, "defective-part")

# Upload training images
image_entries = []
for img_path in good_part_images:
    with open(img_path, "rb") as f:
        image_entries.append(
            ImageFileCreateEntry(
                name=img_path.name,
                contents=f.read(),
                tag_ids=[tag_good.id],
            )
        )

trainer.create_images_from_files(
    project.id,
    ImageFileCreateBatch(images=image_entries),
)

# Train the model
iteration = trainer.train_project(project.id)
print(f"Training status: {iteration.status}")

Once training completes, you can publish the model to a prediction endpoint and call it from your application. Custom Vision also supports exporting models to ONNX, TensorFlow, and CoreML formats for edge deployment on IoT devices, mobile apps, or offline scenarios.

Video Analysis

Azure AI Vision extends beyond still images into video understanding. Two capabilities stand out for production use cases.

Spatial analysis

Spatial analysis processes live video feeds to detect people and track their movement through defined zones. Typical deployments include retail foot traffic counting, occupancy monitoring for safety compliance, and queue length detection. The analysis runs on-premise using a Docker container connected to your cameras, sending only aggregated event data to Azure.

Video retrieval with natural language

Video retrieval uses Florence embeddings to index video content and make it searchable by natural language queries. Instead of manually tagging hours of footage, the service automatically generates embeddings for video frames and lets you search with queries like “person carrying a red box near the loading dock.”

import requests

# Create a video retrieval index
endpoint = "https://your-vision.cognitiveservices.azure.com"
headers = {
    "Ocp-Apim-Subscription-Key": "YOUR_API_KEY",
    "Content-Type": "application/json",
}

# Ingest a video into the index
ingest_payload = {
    "videos": [
        {
            "mode": "add",
            "documentId": "warehouse-cam-01",
            "documentUrl": "https://storage.blob.core.windows.net/videos/cam01.mp4",
        }
    ]
}

response = requests.put(
    f"{endpoint}/computervision/retrieval/indexes/my-index/ingestions/run1",
    headers=headers,
    json=ingest_payload,
    params={"api-version": "2024-02-01"},
)

# Search the indexed video with a natural language query
search_payload = {
    "queryText": "forklift near shelf area",
    "top": 5,
}

results = requests.post(
    f"{endpoint}/computervision/retrieval/indexes/my-index:queryByText",
    headers=headers,
    json=search_payload,
    params={"api-version": "2024-02-01"},
)

for match in results.json()["value"]:
    print(f"Video: {match['documentId']}, "
          f"Timestamp: {match['start']}s - {match['end']}s, "
          f"Relevance: {match['relevance']:.4f}")

Full Capabilities at a Glance

Vision Tiers Comparison

Use Cases by Industry

Production Best Practices

1. Use managed identity instead of API keys — configure DefaultAzureCredential for your deployed services. API keys are convenient during development, but managed identity eliminates the risk of key leakage in production.
2. Batch your requests wisely — the Image Analysis API processes one image per call. For bulk workloads, use Azure Functions or Batch to parallelize calls while respecting rate limits.
3. Cache embedding vectors — Florence embedding calls cost money and add latency. Store vectors in Azure AI Search or Cosmos DB rather than regenerating them on every query.
4. Set confidence thresholds — never trust raw model output blindly. Define minimum confidence scores for tags and detections; surface low-confidence results for human review.
5. Handle image preprocessing — compress oversized images before sending them to the API. The maximum file size is 20 MB, but smaller images reduce latency and cost without sacrificing accuracy for most tasks.
6. Deploy containers for sensitive data — if your images cannot leave your network (medical, defense, classified), run the Vision containers on-premise and keep all processing local.
7. Monitor and set alerts — track API error rates, latency percentiles, and monthly spend in Azure Monitor. Set budget alerts before a runaway pipeline drains your credits.
8. Version your Custom Vision models — always publish new iterations alongside the previous version. Test with held-out data before routing production traffic to the new model.

Next Steps

1. Provision the service — create a Computer Vision resource in the Azure Portal and grab your endpoint and key.
2. Install the SDK — run pip install azure-ai-vision-imageanalysis and try the caption generation sample above.
3. Build a visual search prototype — generate Florence embeddings for 50-100 images, store them in Azure AI Search, and query with natural language.
4. Train a Custom Vision model — pick a domain-specific classification task, upload 15+ images per class, and test the trained model.
5. Explore the documentation — the full API reference is at learn.microsoft.com/azure/ai-services/computer-vision.