Azure Data Factory: Cloud ETL and Data Integration for Architects
Azure Data Factory (ADF) is Microsoft's cloud-based data integration and ETL service. It enables you to ingest data from over 90 built-in connectors, transform it at scale, and load it into Azure analytical stores like Synapse Analytics, Data Lake Storage, or Azure SQL. For architects designing modern data platforms on Azure, ADF is typically the data movement and orchestration layer that ties the rest of the stack together.
Core Concepts
ADF is built around four primary objects:
- Pipelines: Logical groupings of activities that together perform a data integration task. A pipeline defines the workflow — what to do and in what order
- Activities: The steps within a pipeline — data movement activities such as Copy Data, data transformation activities such as Data Flow and Databricks Notebook, and control flow activities such as If Condition and ForEach
- Datasets: Named views of the data structures that activities read from or write to. A dataset points to a table in SQL, a file path in ADLS, or a Blob Storage container
- Linked Services: Connection strings and credentials for external systems. Think of a Linked Service as the JDBC connection equivalent in ADF — it defines how to connect, and datasets define what to access
Integration Runtimes
The Integration Runtime (IR) is the compute infrastructure ADF uses to execute activities. There are three types: Azure IR — managed and fully serverless, used for cloud-to-cloud movements; Self-hosted IR — installed on an on-premises machine or private network VM to access non-public data sources; and Azure-SSIS IR — a managed cluster for running SQL Server Integration Services packages in Azure.
The choice of IR is one of the most important architectural decisions in an ADF design. Self-hosted IR is required when the source or destination is behind a firewall or in a private network — a very common requirement for enterprise hybrid architectures.
Triggers and Orchestration Patterns
ADF pipelines can be triggered on a schedule using cron syntax, on a tumbling window for processing time-partitioned data with dependency awareness, or via event-based triggers when a file lands in Azure Blob Storage or ADLS. For complex orchestration across multiple pipelines, ADF can call child pipelines and handle dependencies using If, Until, and ForEach activities.
Key Architectural Considerations
- Mapping Data Flows are the code-free transformation layer in ADF, suitable for column transformations, joins, aggregations, and schema changes without Spark expertise
- Use parameterisation to build generic pipelines that can handle multiple sources or destinations without code duplication
- Monitor with Azure Monitor — ADF emits pipeline run metrics and logs for alerting and diagnostics
- For heavy transformations, delegate to Azure Databricks or Synapse Spark via ADF activities rather than running complex logic in Data Flows
Key Takeaways
- ADF is the orchestration and data movement layer of the Azure data platform, not the compute layer
- Pipelines, activities, datasets, and linked services are the four core objects every ADF architect must understand
- Integration Runtime choice determines whether you can access on-premises and private network data sources
- Parameterise pipelines early to avoid duplication as your data estate grows
