-
Operational Excellence
-
- Resources have identified owners
- Processes and procedures have identified owners
- Operations activities have identified owners responsible for their performance
- Team members know what they are responsible for
- Mechanisms exist to identify responsibility and ownership
- Mechanisms exist to request additions, changes, and exceptions
- Responsibilities between teams are predefined or negotiated
-
- Executive Sponsorship
- Team members are empowered to take action when outcomes are at risk
- Escalation is encouraged
- Communications are timely, clear, and actionable
- Experimentation is encouraged
- Team members are encouraged to maintain and grow their skill sets
- Resource teams appropriately
- Diverse opinions are encouraged and sought within and across teams
-
- Use version control
- Test and validate changes
- Use configuration management systems
- Use build and deployment management systems
- Perform patch management
- Implement practices to improve code quality
- Share design standards
- Use multiple environments
- Make frequent, small, reversible changes
- Fully automate integration and deployment
-
Security
-
- Evaluate and implement new security services and features regularly
- Automate testing and validation of security controls in pipelines
- Identify and prioritize risks using a threat model
- Keep up-to-date with security recommendations
- Keep up-to-date with security threats
- Identify and validate control objectives
- Secure account root user and properties
- Separate workloads using accounts
-
- Analyze public and cross-account access
- Manage access based on life cycle
- Share resources securely with a third party
- Reduce permissions continuously
- Share resources securely within your organization
- Establish emergency access process
- Define permission guardrails for your organization
- Grant least privilege access
- Define access requirements
-
- Build a program that embeds security ownership in workload teams
- Centralize services for packages and dependencies
- Manual code reviews
- Automate testing throughout the development and release lifecycle
- Train for application security
- Regularly assess security properties of the pipelines
- Deploy software programmatically
- Perform regular penetration testing
-
-
Reliability
-
- How do you ensure sufficient gap between quotas and maximum usage to accommodate failover?
- How do you automate quota management?
- How do you monitor and manage service quotas?
- How do you accommodate fixed service quotas and constraints through architecture?
- How do you manage service quotas and constraints across accounts and Regions?
- How do you manage service quotas and constraints?
- How do you build a program that embeds reliability into workload teams?
-
- How do you enforce non-overlapping private IP address ranges in all private address spaces?
- How do you prefer hub-and-spoke topologies over many-to-many mesh?
- How do you ensure IP subnet allocation accounts for expansion and availability?
- How do you provision redundant connectivity between private networks in the cloud and on-premises environments?
- How do you use highly available network connectivity for workload public endpoints?
-
- Monitor end-to-end tracing of requests through your system
- Conduct reviews regularly
- Analytics
- Automate responses (Real-time processing and alarming)
- Send notifications (Real-time processing and alarming)
- Define and calculate metrics (Aggregation)
- Monitor End-to-End Tracing of Requests Through Your System
-
- Monitor all components of the workload to detect failures
- Fail over to healthy resources
- Automate healing on all layers
- Rely on the data plane and not the control plane during recovery
- Use static stability to prevent bimodal behavior
- Send notifications when events impact availability
- Architect your product to meet availability targets and uptime service level agreements (SLAs)
-
-
Cost Optimization
-
- Establish ownership of cost optimization
- Establish a partnership between finance and technology
- Establish cloud budgets and forecasts
- Implement cost awareness in your organizational processes
- Monitor cost proactively
- Keep up-to-date with new service releases
- Quantify business value from cost optimization
- Report and notify on cost optimization
- Create a cost-aware culture
-
- Perform cost analysis for different usage over time
- Analyze all components of this workload
- Perform a thorough analysis of each component
- Select components of this workload to optimize cost in line with organization priorities
- Perform cost analysis for different usage over time
- Select software with cost effective licensing
-
-
Performance
-
- Learn about and understand available cloud services and features
- Evaluate how trade-offs impact customers and architecture efficiency
- Use guidance from your cloud provider or an appropriate partner to learn about architecture patterns and best practices
- Factor cost into architectural decisions
- Use policies and reference architectures
- Use benchmarking to drive architectural decisions
- Use a data-driven approach for architectural choices
-
- Use purpose-built data store that best support your data access and storage requirements
- Collect and record data store performance metrics
- Evaluate available configuration options for data store
- Implement Strategies to Improve Query Performance in Data Store
- Implement data access patterns that utilize caching
-
- Understand how networking impacts performance
- Evaluate available networking features
- Choose appropriate dedicated connectivity or VPN for your workload
- Use load balancing to distribute traffic across multiple resources
- Choose network protocols to improve performance
- Choose your workload's location based on network requirements
- Optimize network configuration based on metrics
-
- Establish key performance indicators (KPIs) to measure workload health and performance
- Use monitoring solutions to understand the areas where performance is most critical
- Define a process to improve workload performance
- Review metrics at regular intervals
- Load test your workload
- Use automation to proactively remediate performance-related issues
- Keep your workload and services up-to-date
-
-
Sustainability
-
- Optimize geographic placement of workloads based on their networking requirements
- Align SLAs with sustainability goals
- Optimize geographic placement of workloads based on their networking requirements
- Stop the creation and maintenance of unused assets
- Optimize team member resources for activities performed
- Implement buffering or throttling to flatten the demand curve
-
- Optimize software and architecture for asynchronous and scheduled jobs
- Remove or refactor workload components with low or no use
- Optimize areas of code that consume the most time or resources
- Optimize impact on devices and equipment
- Use software patterns and architectures that best support data access and storage patterns
- Remove unneeded or redundant data
- Use technologies that support data access and storage patterns
- Use policies to manage the lifecycle of your datasets
- Use shared file systems or storage to access common data
- Back up data only when difficult to recreate
- Use elasticity and automation to expand block storage or file system
- Minimize data movement across networks
-
- Articles coming soon
Use runbooks to perform procedures
Using Runbooks to Perform Procedures
Runbooks are documented processes that guide teams through performing specific tasks or achieving desired outcomes. By providing a series of clearly defined steps, runbooks help reduce operational risk, increase consistency, and ensure that tasks are completed correctly. Runbooks are particularly valuable in cloud operations, where repeatable processes are key to maintaining stability and reliability.
Document Procedures in Runbooks
Use runbooks to document procedures for common operational tasks, such as deploying updates, troubleshooting incidents, or configuring infrastructure components. Runbooks should be simple, clear, and actionable, providing all the steps necessary to complete a task without ambiguity. This documentation ensures that team members can perform tasks accurately, regardless of their experience level.
Reduce Operational Risk
Runbooks help reduce operational risk by ensuring that processes are standardized and repeatable. When team members follow the same documented steps for a given procedure, the risk of errors is minimized. This consistency is crucial during critical tasks, such as incident response or complex system changes, where mistakes could lead to significant downtime or performance issues.
Create Checklists for Consistency
At its simplest, a runbook can be a checklist used to complete a task. Checklists provide a clear, step-by-step guide that ensures all important actions are taken in the correct order. Runbook checklists are particularly useful for ensuring that nothing is missed during routine procedures, such as system maintenance or deployment tasks.
Achieve Desired Outcomes
Runbooks are used to ensure that specific outcomes are consistently achieved, regardless of who is performing the task. For example, a runbook for scaling an application ensures that all necessary configurations are applied correctly, while a runbook for incident response ensures that critical steps are taken to restore services promptly. By having runbooks for all critical processes, teams can achieve desired outcomes efficiently and reliably.
Use Runbooks for Cloud Operations
In cloud operations, runbooks are essential for handling both planned and unplanned activities. Runbooks can cover areas such as:
- Infrastructure Deployment: Step-by-step instructions for deploying cloud resources.
- Incident Response: Actions to take when responding to an outage or system alert.
- Maintenance Tasks: Procedures for tasks like patch management, scaling, or backups. By following documented steps, teams can ensure that operations are performed in a controlled manner, reducing the likelihood of errors.
Supporting Questions
- How are runbooks used to document procedures for operational tasks?
- What steps are taken to ensure that runbooks reduce operational risks?
- How do runbooks help achieve consistency in cloud operations?
Roles and Responsibilities
Operations Engineer
Responsibilities:
- Use runbooks to carry out operational tasks such as incident response, system maintenance, and deployments.
- Contribute to the creation and improvement of runbooks based on experience and feedback from operational activities.
DevOps Engineer
Responsibilities:
- Develop and maintain runbooks for critical infrastructure operations, ensuring that processes are clearly documented and up-to-date.
- Automate runbook procedures where possible to reduce manual intervention and improve efficiency.
Incident Responder
Responsibilities:
- Use runbooks during incident response to ensure that the necessary steps are taken to mitigate and resolve issues.
- Update runbooks based on lessons learned from incident response activities to improve future readiness.
Artifacts
- Runbook Document: A step-by-step document detailing procedures for specific tasks, such as incident response, deployment, or maintenance.
- Runbook Checklist: A checklist version of a runbook that provides a simplified view of the steps to be completed during a task.
- Runbook Update Log: A log documenting updates made to runbooks, including changes based on feedback or new best practices.
Relevant AWS Tools
Runbook Automation and Management Tools
- AWS Systems Manager Automation: Automates runbook procedures, allowing teams to execute operational tasks with minimal manual effort.
- AWS Systems Manager Runbook: Provides a central location for storing and managing runbooks, making them easily accessible to team members.
Incident Management and Monitoring Tools
- AWS Systems Manager Incident Manager: Uses runbooks to guide incident response efforts, ensuring that critical steps are followed during emergencies.
- Amazon CloudWatch: Triggers runbooks in response to specific events, such as alarms, enabling automated or semi-automated responses to operational issues.
Collaboration Tools
- Amazon Chime: Facilitates real-time communication and collaboration among team members working through a runbook, especially during incident response.
- AWS OpsCenter: Integrates with runbooks to provide contextual guidance for resolving issues identified during operational monitoring.