Azure Data Factory — ETL, Orchestration, and the Modern Data Stack
Azure Data Factory is Microsoft's cloud-native data integration and orchestration service. It moves and transforms data across sources — on-premises databases, cloud storage, SaaS applications, and streaming systems — using a graphical pipeline authoring experience backed by a scalable execution runtime. In the modern Azure data stack, ADF occupies the ingestion and orchestration layer, sitting between your source systems and your analytical storage and compute layers.
Core Concepts
ADF's architecture has four main building blocks:
- Pipelines: The orchestration unit — a logical grouping of activities that execute together. Pipelines can be triggered on a schedule, by an event (file arrival in Azure Blob Storage, an HTTP call), or called from another pipeline.
- Activities: The individual steps within a pipeline. Copy Activity moves data between supported sources and sinks. Data Flow executes transformation logic. Control activities handle branching, looping, and error handling. Execute Pipeline activity enables modular, reusable pipeline composition.
- Datasets: Named references to data structures at your sources and sinks — a table in Azure SQL, a folder in ADLS Gen2, a Salesforce object.
- Linked Services: Connection definitions — credentials and endpoint configuration for each data source or destination. Linked services reference Azure Key Vault secrets rather than storing credentials directly.
Integration Runtime
The Integration Runtime (IR) is the compute infrastructure that executes ADF activities. Three types serve different scenarios:
- Azure IR: Fully managed, serverless runtime for cloud-to-cloud data movement and transformation
- Self-Hosted IR: An agent installed on on-premises or private network infrastructure — required for accessing data sources that are not internet-accessible
- Azure-SSIS IR: Lifts existing SQL Server Integration Services packages to run in Azure — the migration path for legacy SSIS workloads
Mapping Data Flows
Mapping Data Flows provide a no-code transformation authoring experience that compiles to Apache Spark and runs on a managed Spark cluster. They handle complex transformations — joins, aggregations, pivots, derived columns — without requiring Spark expertise. The trade-off is cold-start latency: Spark cluster provisioning takes 2 to 4 minutes. For latency-sensitive pipelines this matters; for batch workloads it rarely does.
ADF in the Modern Azure Data Stack
With the emergence of Microsoft Fabric, the positioning of ADF is evolving. Fabric's Data Factory experience (Fabric Pipelines) shares ADF's pipeline model but integrates natively with OneLake and Fabric's compute services. For new Fabric-native architectures, Fabric Pipelines are the forward direction. ADF remains the correct choice for complex hybrid ingestion scenarios, legacy SSIS migration, and environments not yet on Fabric.
Key Takeaways
- ADF handles ingestion and orchestration — it is not a transformation engine itself but executes transformations via Data Flows or external compute
- Self-Hosted IR is required for on-premises source connectivity — plan its deployment and network access carefully
- Store all credentials in Azure Key Vault and reference them from ADF Linked Services — never hardcode credentials in pipeline configuration
- Mapping Data Flows are powerful but carry Spark cold-start latency — not suitable for sub-minute SLA pipelines
- For new Fabric-native projects, evaluate Fabric Pipelines before defaulting to ADF
