Create Azure Data Factory

Go to Azure Portal

Create a Data Factory resource

• Region: Close to your data storage
• Enable GIT integration if you want versioning

Create Linked Services

Think of Linked Services as "connectors" to your data sources and sinks.

• Go to Manage > Linked Services
• Add a new Linked Service:
• For input: REST, HTTP, or Azure Blob (if using CSV)
• For output: Azure SQL Database, Data Lake, etc.

Create Datasets

Datasets define what data you're reading or writing.

Input Dataset (e.g., HTTP)

• Format: JSON
• Structure: Define schema or infer at runtime

Output Dataset (e.g., Azure SQL Table)

• Table name: covid_stats
• Schema: Define columns (country, date, confirmed, deaths, etc.)

Build a Pipeline

Go to Author > Pipelines, and create a new one.

Add Activities:

• Copy Data Activity:
• Source: REST API or CSV file
• Sink: Azure SQL
• Use Mapping to map fields between input and output
• Data Flow Activity:
• If you need transformations (pivot, joins, filters)
• Stored Procedure Activity:
• Use this to clean/aggregate your SQL table after insert

Trigger and Monitor

Create a Schedule Trigger (e.g., run daily at 2am)

Go to Monitor tab to view runs, logs, and errors

Monitoring and Logging

Use Azure Monitor or built-in Data Factory monitor

Configure alert rules for failed pipeline runs

Tips

Parameterize datasets and pipelines to make them reusable

Use Self-hosted Integration Runtime if your source is on-premises

Keep logs in Log Analytics for audit and performance tracking

Add Retry policies for network resilience