AI Toolkit en VS Code

You don’t need a cloud subscription to experiment with AI models. The AI Toolkit for Visual Studio Code brings model discovery, local inference, fine-tuning, and evaluation directly into your editor — the same place you already write code. It’s a complete AI development workbench without leaving VS Code.

This guide covers the full workflow: browsing the model catalog, running models locally, fine-tuning with your own data, and deploying to Azure when you’re ready for production.

What is the AI Toolkit?

The AI Toolkit is a VS Code extension developed by Microsoft that provides:

• Model catalog — browse and download models from Hugging Face, Azure AI, and ONNX collections directly in VS Code.
• Local playground — run models on your machine for testing, prototyping, and evaluation without any API calls.
• Fine-tuning — customize models with your own datasets using QLoRA or LoRA, running locally or on remote compute.
• Evaluation — benchmark model quality with built-in metrics before deployment.
• Deployment — push fine-tuned models to Azure AI endpoints with a few clicks.

It integrates with your existing VS Code workflow — terminal, source control, debugging — so AI development feels like any other coding task.

Development workflow

Installation and setup

1. Open VS Code and go to the Extensions panel (Ctrl+Shift+X).
2. Search for “AI Toolkit” and install the extension by Microsoft.
3. After installation, a new AI Toolkit icon appears in the activity bar (left sidebar).
4. Click it to open the toolkit panel with the model catalog, playground, and fine-tuning options.

Hardware requirements

Browsing the model catalog

The model catalog is the starting point. It aggregates models from multiple sources:

• Hugging Face — thousands of open models (Phi, Llama, Mistral, Gemma).
• Azure AI model catalog — Microsoft’s curated collection optimized for enterprise.
• ONNX Runtime models — optimized for fast local inference on CPU and GPU.

You can filter by task (text generation, classification, embeddings), size, license, and hardware requirements. Each model card shows parameter count, quantization options, and benchmark scores.

Downloading a model

Click any model in the catalog and select Download. The toolkit handles everything — downloading weights, setting up the runtime, and configuring the inference engine. Models are stored locally in your workspace.

For a quick start, try Phi-3-mini-4k-instruct-onnx — a capable, small model that runs well on most hardware.

The playground

Once you download a model, the playground lets you interact with it immediately:

• Chat interface — test conversational models with a familiar chat UI.
• System prompt editor — configure the model’s persona and behavior.
• Parameter controls — adjust temperature, top-p, max tokens, and other generation settings in real time.
• Batch testing — run multiple prompts and compare outputs side by side.

The playground runs entirely locally. No data leaves your machine, which makes it ideal for testing with sensitive prompts or proprietary data.

Example: testing a classification prompt

System prompt:
You are a support ticket classifier. Classify each ticket into exactly
one category: billing, technical, account, or general.
Respond with only the category name.

User:
I can't access my dashboard after changing my password yesterday.

Model output:
account

Test multiple inputs, tweak the system prompt, and adjust temperature until you’re confident in the model’s behavior — all without API costs.

Fine-tuning with QLoRA

When a pre-trained model almost works but needs domain-specific knowledge, fine-tuning closes the gap. The AI Toolkit supports QLoRA (Quantized Low-Rank Adaptation), which lets you fine-tune large models on consumer GPUs by training only a small set of adapter weights.

Preparing your dataset

Create a JSONL file with your training examples:

{"messages": [{"role": "system", "content": "You are a medical coding assistant."}, {"role": "user", "content": "Patient presents with acute bronchitis"}, {"role": "assistant", "content": "ICD-10: J20.9 - Acute bronchitis, unspecified"}]}
{"messages": [{"role": "system", "content": "You are a medical coding assistant."}, {"role": "user", "content": "Follow-up for type 2 diabetes with neuropathy"}, {"role": "assistant", "content": "ICD-10: E11.40 - Type 2 diabetes with diabetic neuropathy, unspecified"}]}

Configuring fine-tuning

1. In the AI Toolkit panel, click Fine-tune.
2. Select your base model (e.g., Phi-3-mini).
3. Upload your JSONL dataset.
4. Configure training parameters:
   • Epochs: 3-5 for most tasks.
   • Learning rate: 2e-4 is a good starting point for QLoRA.
   • LoRA rank: 16-64 (higher = more capacity, more VRAM).
   • Batch size: depends on your GPU memory.
5. Click Start training. The toolkit shows real-time loss curves and progress.

Remote fine-tuning with Azure ML

If your local hardware isn’t sufficient, the toolkit can offload fine-tuning to Azure ML compute:

# The toolkit generates this configuration automatically
# You just need to connect your Azure subscription
Compute: Standard_NC24ads_A100_v4
GPU: NVIDIA A100 (80 GB)
Estimated time: ~45 minutes for 1000 examples

The workflow is the same — select the model, upload data, configure parameters — but training runs on cloud GPUs. Results are downloaded back to your workspace when complete.

Evaluation

After fine-tuning, you need to measure quality before deployment. The AI Toolkit provides built-in evaluation metrics:

Create an evaluation dataset with expected outputs and run it against both the base model and your fine-tuned version. The toolkit shows a side-by-side comparison so you can confirm the fine-tuning improved quality without introducing regressions.

Deploying to Azure

When your model is ready for production:

1. Right-click your fine-tuned model in the toolkit and select Deploy to Azure.
2. Choose your Azure subscription and resource group.
3. Select the compute tier (GPU instances for inference).
4. The toolkit packages your model, creates an Azure AI endpoint, and deploys it.
5. You get a REST API endpoint that you can call from your application.

import requests

endpoint = "https://your-model.eastus.inference.ml.azure.com/score"
headers = {
    "Authorization": f"Bearer {api_key}",
    "Content-Type": "application/json",
}

response = requests.post(endpoint, headers=headers, json={
    "messages": [
        {"role": "user", "content": "Classify this ticket: I need a refund"}
    ]
})

print(response.json())

AI Toolkit vs. Azure AI Foundry

Use the AI Toolkit for fast experimentation on your machine, then move to Azure AI Foundry when you need team collaboration, managed infrastructure, and production-grade monitoring.

Next steps

1. Install the extension — search “AI Toolkit” in the VS Code marketplace.
2. Download Phi-3-mini — run it locally in the playground with zero setup.
3. Fine-tune a small model — prepare 50-100 examples in JSONL format and run a QLoRA training session.
4. Read the docs: learn.microsoft.com/windows/ai/toolkit