Azure AI Foundry

You’ve built a prototype that calls Azure OpenAI and gets decent results. Now you need to ship it: ground the responses in your data, evaluate quality systematically, deploy behind an API, monitor for drift and abuse, and do all of this under enterprise governance. Azure AI Foundry is Microsoft’s unified platform for this entire lifecycle — from prototype to production AI applications.

This guide covers the platform architecture, walks through building a production RAG application with Prompt Flow, setting up systematic evaluation, and deploying with monitoring and content safety built in.

What is Azure AI Foundry?

Azure AI Foundry (formerly Azure AI Studio) is the central hub for building, evaluating, and deploying AI applications on Azure. It unifies services that were previously scattered across the portal:

• Model catalog — deploy OpenAI, Meta, Mistral, Cohere, and open-source models from a single interface.
• Prompt Flow — visual and code-based tool for building LLM pipelines (RAG, agents, chains).
• Evaluation — systematic quality assessment with built-in and custom metrics.
• Content safety — detect and filter harmful content, PII, and jailbreak attempts.
• Tracing & monitoring — end-to-end observability for production AI applications.
• Fine-tuning — customize models with your data without managing infrastructure.

Setting up a project

Everything in AI Foundry is organized by projects. A project is a workspace that groups your models, data, evaluations, and deployments:

1. Go to ai.azure.com and sign in.
2. Click Create project.
3. Select or create an AI Hub — the parent resource that provides shared compute, storage, and networking.
4. Choose your Azure subscription, region, and associated resources (Azure OpenAI, AI Search, Storage).

The AI Hub handles infrastructure; the project is where your team builds and iterates.

The model catalog

AI Foundry provides access to models from multiple providers:

Each model can be deployed in minutes. Serverless API deployments are the fastest — pay per token with no infrastructure to manage. Managed compute deployments give you dedicated GPUs for fine-tuned or open-source models.

Building a RAG pipeline with Prompt Flow

Prompt Flow is the core tool for building production LLM applications. It’s a visual + code pipeline builder that connects data retrieval, prompt engineering, and model inference:

The pipeline

# Prompt Flow Python node: retrieve relevant documents
from azure.search.documents import SearchClient
from azure.identity import DefaultAzureCredential

def retrieve_documents(question: str, index_name: str, top_k: int = 5) -> list:
    client = SearchClient(
        endpoint="https://your-search.search.windows.net",
        index_name=index_name,
        credential=DefaultAzureCredential(),
    )

    results = client.search(
        search_text=question,
        query_type="semantic",
        semantic_configuration_name="default",
        top=top_k,
        select=["title", "content", "url"],
    )

    return [
        {"title": r["title"], "content": r["content"], "url": r["url"]}
        for r in results
    ]

The prompt template

system:
You are a helpful assistant for {{company_name}}.
Answer the user's question based ONLY on the provided context.
If the context doesn't contain the answer, say "I don't have
information about that in our documentation."
Always cite your sources with [Title](URL) format.

context:
{% for doc in documents %}
### {{doc.title}}
{{doc.content}}
Source: {{doc.url}}
{% endfor %}

user:
{{question}}

Systematic evaluation

Before deploying, you need to measure quality. AI Foundry provides built-in evaluators:

Running an evaluation

from azure.ai.evaluation import evaluate
from azure.ai.evaluation import GroundednessEvaluator, RelevanceEvaluator

# Test dataset: questions + expected answers + context
test_data = "test_dataset.jsonl"

result = evaluate(
    data=test_data,
    target=my_rag_pipeline,  # your Prompt Flow or function
    evaluators={
        "groundedness": GroundednessEvaluator(model_config),
        "relevance": RelevanceEvaluator(model_config),
    },
    evaluator_config={
        "default": {
            "question": "${data.question}",
            "answer": "${target.answer}",
            "context": "${target.context}",
        }
    },
)

print(f"Groundedness: {result.metrics['groundedness.score']:.2f}")
print(f"Relevance: {result.metrics['relevance.score']:.2f}")

Run evaluations on every change to your prompt, retrieval logic, or model. Track scores over time to catch regressions before they reach production.

Content safety

Azure AI Content Safety filters sit between your application and the model:

• Hate, violence, sexual, self-harm — categorized at low/medium/high severity levels.
• Jailbreak detection — identifies attempts to bypass the model’s instructions.
• Protected material detection — flags output that matches copyrighted text.
• PII detection — identifies personal information in inputs and outputs.
• Groundedness detection — flags responses that aren’t supported by the provided context.

Configure severity thresholds per deployment. For a customer-facing application, you might block medium+ severity across all categories. For an internal research tool, you might only block high severity.

Deployment and monitoring

Deploying your application

from azure.ai.ml import MLClient
from azure.ai.ml.entities import ManagedOnlineEndpoint, ManagedOnlineDeployment

# Create an endpoint
endpoint = ManagedOnlineEndpoint(
    name="support-rag-endpoint",
    auth_mode="key",
)
ml_client.online_endpoints.begin_create_or_update(endpoint)

# Deploy your Prompt Flow
deployment = ManagedOnlineDeployment(
    name="v1",
    endpoint_name="support-rag-endpoint",
    model="azureml://flows/support-rag/versions/1",
    instance_type="Standard_DS3_v2",
    instance_count=2,
)
ml_client.online_deployments.begin_create_or_update(deployment)

Production monitoring

Once deployed, AI Foundry provides:

• Tracing — full request traces showing each step in your pipeline (retrieval time, model latency, token count).
• Token dashboards — track consumption by endpoint, deployment, and user.
• Quality monitoring — run periodic evaluations on sampled production traffic to detect drift.
• Alerts — configure alerts on latency spikes, error rates, or content safety triggers.

AI Foundry vs. using services directly

Use AI Foundry when you’re past the prototype stage and need systematic quality assurance, content safety, and production monitoring. For quick experiments or simple integrations, direct API calls are faster to get started.

Production architecture checklist

1. Use AI Hub for shared resources — deploy Azure OpenAI, AI Search, and storage once in the Hub; share across projects.
2. Version everything — prompt templates, retrieval configs, evaluation datasets, and flow definitions should be in source control.
3. Evaluate before every deployment — set minimum quality scores as deployment gates in your CI/CD pipeline.
4. Configure content safety early — don’t ship without content filters. The cost of a harmful output far exceeds the setup time.
5. Monitor token costs — set budget alerts. A misconfigured pipeline can burn through tokens quickly.
6. Plan for scale — use provisioned throughput for predictable workloads; pay-as-you-go for variable traffic. Consider a fallback deployment on a different model (e.g., GPT-4o-mini) for graceful degradation.

Next steps

1. Create a project at ai.azure.com and deploy a model from the catalog.
2. Build a RAG flow — connect Azure AI Search to your documents and create a Prompt Flow pipeline.
3. Run an evaluation — create 20-30 test questions with expected answers and measure groundedness.
4. Read the docs: learn.microsoft.com/azure/ai-studio