1 / 9
←
→

OCBC Airflow
Managed Service

SLA Documentation

NOTICE

Apache Airflow is community open-source software. There is no principal or vendor SLA backing this service. Response times represent best-effort acknowledgment targets, not guaranteed resolution times.

View SLA Details
Service Level Agreement

Severity Classification & Response Time

Service Coverage

Included

Existing on-premises Apache Airflow platform within contracted topology

Excluded

Infrastructure (Kubernetes, VM, network), third-party systems (database, LDAP), and new development

Key Points

  • Response Time refers to acknowledgment and beginning of investigation, NOT resolution time
  • Resolution time depends on root cause complexity, upstream fix availability from Apache community, and OCBC change approval process
  • Complex issues may require upstream patch or workaround from Apache community
Severity Impact Response Time
Severity 1 Production completely down or severely degraded 1 hour
Severity 2 Significant degradation, many users or critical DAGs impacted 4 hours
Severity 3 Limited impact, workaround available 8 hours
Severity Definitions

Severity Examples

Severity 1 (Critical)

Airflow production is completely down or severely degraded and cannot be used at all

  • Airflow Webserver down: Users cannot access UI, cannot monitor DAGs, cannot trigger manual runs
  • Airflow Scheduler down: All DAGs stop being scheduled, no tasks are executed
  • All DAGs failing: Corrupted Airflow metadata database causing all DAGs to fail parsing
  • Database connection pool exhausted: Airflow cannot query metadata, all operations hang
  • Celery workers unresponsive: Task queue is full, no tasks are being processed

Severity 2 (Major)

Airflow is running but has significant degradation impacting many users or critical DAGs

  • Multiple critical DAGs failing: 5 out of 10 daily reporting DAGs failing while others run normally
  • Task queue backlog growing: Tasks accumulating in queue, execution delay exceeding 30 minutes
  • SSO or LDAP authentication errors: Some users cannot login, but admin access remains functional
  • Scheduler lag exceeding 15 minutes: DAG scheduled at 08:00 only starts at 08:15 or later, impacting downstream SLA

Severity 3 (Moderate)

Issue exists but workaround is available, impact is limited to few users or non-critical processes

  • Single non-critical DAG failing: Weekly report DAG failing but can be triggered manually
  • UI slowness: Loading DAG list takes more than 10 seconds but still functional
  • Log parsing warnings: Warnings appearing in logs but not impacting execution
  • Configuration drift detected: Timezone config different from standard but DAGs still running correctly
Resolution

Resolution Factors (Outside SLA Scope)

The following factors affect resolution time and are outside the managed service SLA commitment

Factor Impact on Resolution Time
Root Cause Complexity Bug in Apache Airflow core, third-party dependencies, or infrastructure layer
Apache Community Response Time for upstream patch release, validation, testing
OCBC Change Approval Change control process, downtime approval, deployment window scheduling
Infrastructure Dependencies Kubernetes, VM, network, database fixes from other teams

Resolution Time Commitment

Critical Incident

  • Response time: Maximum 1 hour, including initial investigation process
  • Resolution: Best effort resolution with target workaround or mitigation within 24 to 72 hours, depending on complexity and root cause

Critical Vulnerability

  • ICS-controlled configuration/DAG: Assessment and workaround or mitigation as needed
  • Apache Airflow core: Target 72 hours for assessment and workaround if possible
  • ICS commitment: Provide temporary workaround → escalate to Apache community → monitor → apply official fix once available
Workflow

Incident Response Workflow

Workflow Participants

Customer (OCBC)

Submits requests and provides approval for changes requiring downtime

Technical Support Team

Handles investigation, troubleshooting, and resolution of incidents

Commercial Team

Handles requests that fall outside the contracted scope of work

Process Steps with Examples

Step Action Example
1 Submit Request Customer creates ticket via ServiceNow or email: "DAG sales_report failing since 08:00"
2 Classify Severity Tech Support assigns Sev 1/2/3. Critical DAG → Sev 2
3 Acknowledge Send initial response within SLA time: "Investigating. Will update within 2 hours"
4 Investigate Root cause analysis, log review. Found: DB connection timeout during peak load
5 Resolve/Workaround Apply fix or provide mitigation. Increased connection pool, DAG now running
6 Verify Confirm fix with customer: "Please confirm sales_report completed for today"
7 Close Update ticket with resolution details. Closed. RCA: pool exhaustion. Fix: pool_size 10→20
Scope

Service Scope Decision Tree

Decision Points

Incident

Something is broken or has an error that needs immediate attention. Routes directly to severity assessment.

Example: "Scheduler down", "DAG failing"

Service Request

A change request, enhancement, or assistance (not an error). Requires scope check first.

Example: "Add new DAG", "Upgrade Airflow version"

In Scope Check

Result Action SLA
Yes - In Scope Process within service request SLA, proceed to execution 72 hours
No - Out of Scope Identify work type, route to commercial process N/A (commercial proposal required)

DAG Addition and Replacement Policy

  • New DAG Addition: Any request to add new DAG is classified as new development and requires commercial process (new SOW or amendment)
  • Delete and Create New: If an existing DAG is deleted and a new DAG is created to replace it, this is NOT considered a replacement. It is treated as new DAG development and requires commercial process
  • Modification of Existing DAG: Troubleshooting, bug fixes, and minor configuration changes to existing DAGs are within scope
Exclusions

Out of Scope Items

The following items are explicitly excluded from the managed service and require a separate commercial process

Category Examples
New Development Create new DAG for Oracle→PostgreSQL ingestion; build custom operator for SAP integration
Infrastructure Changes Add new Kubernetes node; upgrade cluster to v1.28; expand disk from 500GB→1TB
Topology Expansion Add new Airflow environment (UAT); deploy to new region; add DR site
Integration with New Systems Connect to new Snowflake account; integrate with Salesforce API; add Slack webhook

In Scope Examples (for comparison)

  • DAG daily_sales_report has error, please investigate (Troubleshooting existing DAG)
  • Scheduler is slow, task backlog is accumulating (Performance troubleshooting)
  • New user cannot login via SSO (Authentication troubleshooting for existing configuration)
  • Request for quarterly health check (Included in service)
  • New CVE announced for Airflow, need patch (Security patch support)
Out of Scope Handling

Commercial Process

For out of scope requests, the following commercial process applies

6
Steps in Commercial Process
SOW
New SOW or Amendment Required
Step Action
1 Technical Support Team identifies the request as out of scope
2 Request is escalated to Commercial Team
3 Commercial Team discusses requirements with OCBC
4 Proposal is prepared with scope, timeline, and pricing
5 If approved, new SOW or Amendment to existing contract is created
6 After commercial agreement is finalized, work can begin

Example

OCBC requests to create new DAG for ingesting data from Oracle to PostgreSQL. This is new development, not troubleshooting existing DAG, therefore it goes through a commercial process.

Business Continuity

Force Majeure & Business Continuity

Force Majeure at OCBC

When force majeure affects OCBC operations (data center, network, infrastructure)

  • OCBC notifies ICS of the situation and expected impact on Airflow operations
  • ICS suspends SLA clock for the duration of the force majeure event
  • ICS remains on standby to provide remote support once OCBC systems are accessible
  • After restoration, ICS assists with Airflow health check and recovery validation

Force Majeure at ICS (CMBC)

Continuity of Managed Service during disruption at ICS side

Ticketing System Outage

  • Fallback to direct WhatsApp communication with spreadsheet tracking
  • Tickets migrated back after system restoration

Critical Cyber Event at ICS

  • Isolate affected systems, notify OCBC immediately
  • Switch to clean backup environment
  • Service to OCBC continues from isolated environment

Communication During Force Majeure

Trigger Timing
Immediate notification to the affected party Within 1 hour of force majeure event
Regular status updates during active incidents Every 4 hours
Post-incident report Within 5 business days after resolution
Lessons learned review and process improvement If applicable