Microsoft Fabric + AI

Most organizations run their data stack across disconnected services — a data lake here, a warehouse there, separate tools for ETL, BI, data science, and real-time analytics. Microsoft Fabric unifies all of these into a single SaaS platform built on a shared OneLake foundation. One copy of data, one security model, one governance layer — from ingestion to machine learning to dashboards.

This guide covers Fabric’s architecture in depth, walks through the major workloads with real code, and shows how AI capabilities are woven into every layer of the platform.

Architecture: OneLake and the Lakehouse

Fabric’s central innovation is OneLake — a single, multi-cloud data lake for your entire organization. Think of it as “OneDrive for data.” Every Fabric workload reads and writes to OneLake, which means:

• No data duplication — the data warehouse, data science notebooks, and Power BI reports all reference the same underlying data.
• One security model — define access once in OneLake; it applies everywhere.
• Open format — data is stored in Delta/Parquet format, accessible via standard tools and APIs.

The Lakehouse: where SQL meets Spark

A Lakehouse in Fabric combines the scalability of a data lake with the query performance of a data warehouse. It’s the primary analytical data structure, and it gives you two interfaces:

• Spark notebooks — transform and process data with PySpark, Scala, or SparkSQL.
• SQL analytics endpoint — query the same data with standard T-SQL, no data movement needed.

Creating a Lakehouse and loading data

# PySpark notebook in Fabric

# Read CSV from external source into the Lakehouse
df = spark.read \
    .option("header", "true") \
    .option("inferSchema", "true") \
    .csv("abfss://raw@onelake.dfs.fabric.microsoft.com/sales_2024.csv")

# Clean and transform
from pyspark.sql import functions as F

df_clean = df \
    .withColumn("order_date", F.to_date("order_date", "yyyy-MM-dd")) \
    .withColumn("revenue", F.col("quantity") * F.col("unit_price")) \
    .filter(F.col("status") != "cancelled")

# Write as Delta table to the Lakehouse
df_clean.write \
    .mode("overwrite") \
    .format("delta") \
    .saveAsTable("sales_clean")

Once the Delta table exists, it’s immediately queryable through the SQL endpoint — no import step required:

-- SQL analytics endpoint (T-SQL)
SELECT
    YEAR(order_date) AS year,
    MONTH(order_date) AS month,
    SUM(revenue) AS total_revenue,
    COUNT(*) AS order_count
FROM sales_clean
GROUP BY YEAR(order_date), MONTH(order_date)
ORDER BY year, month;

Data pipelines with Data Factory

Fabric’s Data Factory handles ETL/ELT at scale. It supports 130+ connectors and provides a visual pipeline designer:

• Copy activity — move data from any source to OneLake (databases, APIs, SaaS apps, files).
• Dataflows Gen2 — low-code transformations with Power Query for business users.
• Notebook activity — run Spark notebooks as pipeline steps for complex transformations.
• Orchestration — schedule pipelines, set dependencies, handle retries and error paths.

Data Science: ML models in Fabric

Fabric’s Data Science workload provides a full ML lifecycle:

import mlflow
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.model_selection import train_test_split

# Load data from the Lakehouse
df = spark.sql("SELECT * FROM sales_clean").toPandas()

X = df[["quantity", "unit_price", "discount", "region_code"]]
y = df["revenue"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

# Track with MLflow (built into Fabric)
mlflow.set_experiment("revenue-prediction")

with mlflow.start_run():
    model = GradientBoostingRegressor(n_estimators=200, max_depth=5)
    model.fit(X_train, y_train)

    score = model.score(X_test, y_test)
    mlflow.log_metric("r2_score", score)
    mlflow.sklearn.log_model(model, "revenue_model")

    print(f"R² Score: {score:.4f}")

Models tracked in MLflow can be registered, versioned, and deployed directly as Fabric ML endpoints for batch or online predictions — without leaving the platform.

PREDICT function: ML in T-SQL

Once a model is registered, you can call it directly from SQL queries:

SELECT *,
    PREDICT(revenue_model, *) AS predicted_revenue
FROM sales_forecast_input;

This bridges the gap between data scientists who build models and analysts who consume them — no API calls, no separate infrastructure.

Real-Time Intelligence

For streaming data, Fabric provides Eventhouse (KQL-based) and Eventstream:

• Eventstream — ingest from Azure Event Hubs, IoT Hub, Kafka, or custom sources.
• Eventhouse — store and query streaming data with KQL (Kusto Query Language).
• Real-time dashboards — Power BI visuals that refresh every few seconds.
• Data Activator — trigger alerts and actions when data patterns match your conditions (e.g., “notify me if temperature exceeds 80°C for 5 minutes”).

// KQL query in Eventhouse
SensorReadings
| where Timestamp > ago(1h)
| summarize AvgTemp = avg(Temperature), MaxTemp = max(Temperature)
    by bin(Timestamp, 5m), DeviceId
| where AvgTemp > 75
| order by Timestamp desc

Copilot in Fabric

Copilot is integrated across Fabric workloads:

Capacity and pricing

Fabric uses a Capacity Unit (CU) model. All workloads share the same capacity pool:

Fabric vs. the alternatives

Next steps

1. Start a free trial at app.fabric.microsoft.com — 60 days with full capabilities.
2. Create a Lakehouse — load a CSV, transform it with Spark, and query it with SQL.
3. Build a pipeline — connect an external data source and schedule automatic ingestion.
4. Read the docs: learn.microsoft.com/fabric