Implement emergency levers

Emergency levers are rapid-response mechanisms that can be activated during critical incidents to mitigate the availability impact on your workload. These levers are designed to help manage system disruptions, reduce load, or redirect traffic during emergencies, thereby ensuring that core functionality remains available and that the overall impact is minimized. Implementing well-defined emergency levers can significantly improve the resilience and recovery capabilities of your workload.

Establish emergency lever champions in each team: Assign emergency lever champions within each workload team to oversee the development and testing of emergency levers. These champions ensure that well-defined emergency levers are in place and can be activated quickly to mitigate availability impacts during incidents.

Provide training on emergency lever implementation: Train builder teams on best practices for implementing and using emergency levers, including load reduction, traffic redirection, and scaling down non-essential services. Training should cover the scenarios in which emergency levers should be activated and how to safely revert changes once the emergency is resolved. Proper training helps teams understand the importance of having emergency response mechanisms to maintain system availability.

Develop emergency lever guidelines and standards: Create clear guidelines for designing and implementing emergency levers across services. These guidelines should include best practices for load shedding, service prioritization, and traffic management. Documented standards help ensure consistent emergency response practices across workloads, improving the resilience and recovery capabilities of systems.

Integrate emergency lever validation into CI/CD pipelines: Integrate validation checks into CI/CD pipelines to ensure that emergency levers are functional and can be activated when needed. Automated tests can verify the activation and deactivation of emergency levers under different scenarios, reducing the risk of failure during an actual incident.

Define automated guardrails for emergency lever activation: Use automated tools to enforce the implementation of emergency levers across services. Tools like AWS Lambda, AWS Step Functions, and Amazon Route 53 can help implement emergency measures such as load shedding, scaling, and traffic rerouting. Automated guardrails help ensure that emergency levers are available and can be activated in a timely manner during critical incidents.

Foster a culture of proactive incident management: Encourage builder teams to prioritize the design and implementation of emergency levers as part of their incident management strategy. Recognize and reward teams that effectively develop and test emergency levers to ensure system resilience. Open discussions about lessons learned from past incidents can help create a culture that values proactive planning and rapid response.

Conduct regular emergency lever reviews and drills: Schedule regular reviews to evaluate the effectiveness of emergency levers and conduct drills to test their activation. These reviews and drills help ensure that emergency levers are functional, that teams are familiar with their activation procedures, and that any potential issues are addressed before an actual incident occurs.

Leverage automation for consistent emergency lever implementation: Use Infrastructure as Code (IaC) tools like AWS CloudFormation or AWS CDK to automate the deployment of emergency levers. Automating these processes helps ensure consistency across environments and allows for rapid activation of emergency response mechanisms when needed.

Provide dashboards for visibility into emergency lever status: Use dashboards to provide visibility into the status of emergency levers, including their readiness and recent activations. Tools like Amazon CloudWatch and AWS X-Ray can help monitor the activation of emergency levers and assess their impact on system performance. Dashboards help builder teams proactively manage emergency lever readiness and ensure system resilience.

Supporting Questions

• How do you ensure that builder teams implement effective emergency levers for managing availability impacts during critical incidents?
• What mechanisms are in place to validate that emergency levers can be activated quickly and effectively when needed?
• How do you align emergency lever implementation with organizational standards for resilience and rapid response?

Roles and Responsibilities

Emergency Lever Champion (within Builder Team)

Responsibilities:

• Oversee the design, implementation, and testing of emergency levers to mitigate availability impacts.
• Ensure that emergency levers can be activated quickly and are well-documented for use during incidents.

Application Developer

Responsibilities:

• Implement emergency lever features in services to provide rapid-response mechanisms during incidents.
• Use automated tools to validate that emergency levers can be activated effectively during the development and testing phases.

Operations Team Member

Responsibilities:

• Assist builder teams with configuring emergency levers and managing their activation during critical incidents.
• Provide guidance and training to ensure alignment with best practices for emergency response and availability management.

Artifacts

Emergency Lever Guidelines and Standards: A document outlining best practices for designing and implementing emergency levers, including load shedding, service prioritization, and traffic management.

Training Resources for Emergency Lever Implementation: Hands-on labs, workshops, and documentation to help teams understand how to develop and use emergency levers effectively.

Automated Emergency Lever Validation Configurations: Scripts and configurations that help automate the validation of emergency levers across services and environments.

Relevant AWS Services

Training and Awareness Tools:

• AWS Skill Builder and AWS Well-Architected Labs: Resources for learning about implementing emergency levers, load shedding, and rapid-response mechanisms.
• AWS Trusted Advisor: Provides insights into workload configurations and recommendations for improving emergency response capabilities.

Emergency Lever Implementation and Guardrails:

• AWS Lambda: Implements automated responses for reducing load or scaling down non-essential services during critical incidents.
• AWS Step Functions: Orchestrates emergency workflows to manage system behavior during incidents.
• Amazon Route 53: Provides traffic management and failover capabilities to redirect requests during availability issues.

Monitoring and Visibility Tools:

• Amazon CloudWatch: Tracks metrics related to the activation of emergency levers, providing alerts for incidents that require emergency response.
• AWS X-Ray: Traces requests across services to verify that emergency levers are functioning as expected and that responses are appropriately managed.
• AWS CloudFormation: Codifies emergency lever configurations to automate and standardize their implementation across environments.