Designing a Secure and Scalable Azure AI Landing Zone
An Azure AI Landing Zone is the specialised infrastructure foundation for enterprise AI workloads — built on top of the Enterprise-Scale Landing Zone baseline but with AI-specific controls for network isolation, identity, governance, responsible AI, and cost management. Getting this right before deploying AI services at scale prevents the security and governance debt that accumulates when AI infrastructure is provisioned ad hoc. Here is the real-world design guide.
Network Isolation for AI Services
The default configuration for Azure AI services exposes public endpoints. For enterprise deployments, this is not acceptable. Every AI service in your landing zone — Azure OpenAI, Azure AI Search, Azure AI Foundry, Azure Machine Learning — should be deployed with public network access disabled and access provided exclusively through private endpoints.
Private endpoints assign a private IP address from your VNet to the AI service, routing all traffic through your network rather than the public internet. Combined with private DNS zones (centralised in your hub subscription, linked to all spoke VNets), this creates a network isolation model where AI services are unreachable from the internet and accessible only from within your Azure network or from on-premises via ExpressRoute.
Private Endpoint Architecture
Deploy private endpoints for each AI service in the spoke VNet of the AI Landing Zone subscription. Do not deploy them in the hub. The hub handles shared networking services (firewall, DNS, gateways); AI service private endpoints belong in the workload subscription. Configure private DNS zone entries in the centralised hub DNS zones — this ensures all VNets linked to the hub can resolve AI service hostnames to private IP addresses automatically.
Azure AI Foundry requires multiple private endpoints: one for the hub workspace, one for storage, one for Key Vault, and one for Azure Container Registry if used. Plan the private endpoint count before designing VNet address space — each private endpoint consumes a private IP address from the subnet.
Managed Identity for AI Workload Authentication
Never use API keys for application-to-AI-service authentication in production. Use managed identities — either system-assigned for single-service deployments or user-assigned for applications that authenticate to multiple AI services with the same identity. Managed identities eliminate API key rotation risk, provide Entra ID-backed audit trails, and integrate with Azure RBAC for fine-grained access control.
Assign managed identities the minimum required Azure OpenAI role: Cognitive Services OpenAI User for inference-only access, Cognitive Services OpenAI Contributor only when model deployment management is required. Do not assign Cognitive Services Contributor at the account level — this grants excessive permissions.
RBAC for AI Workloads
Define RBAC assignments for four personas: AI service operators (who deploy and configure AI resources), AI developers (who consume AI services for application development), AI security reviewers (who audit AI configuration and usage), and AI cost managers (who review consumption and quotas). Map each persona to built-in Azure roles at the appropriate scope — resource group for workload-specific roles, subscription for platform roles.
Azure AI Foundry Governance
Foundry Hub and Project governance requires explicit design. Use Foundry Hubs as the governance boundary — one hub per business unit or compliance domain. Projects within a hub inherit the hub's network, identity, and policy configuration. Restrict project creation to approved identities; ad hoc Foundry project creation by developers bypasses the governance model.
Responsible AI Controls
Azure OpenAI content filters are enabled by default but require explicit configuration for enterprise requirements. Define content filter policies per use case — a customer-facing chatbot needs different filter sensitivity than an internal developer productivity tool. Document the content filter configuration decisions and the rationale for any modifications from defaults — this is essential for responsible AI audit trails.
Cost Management for AI at Scale
- Set per-deployment token quotas in Azure OpenAI — unconstrained quotas result in cost surprises in high-volume scenarios
- Use provisioned throughput for predictable, high-volume workloads — pay-as-you-go costs more per token at scale
- Tag all AI resources with workload, team, and cost-centre tags — AI spend accumulates quickly and is hard to attribute retroactively
- Monitor token consumption with Azure Monitor — set budget alerts at 70% and 90% of monthly token quota thresholds
