Integrate functional testing as part of your deployment

Controlled changes are essential for effective deployment of new functionalities. Integrating functional testing within your deployment pipelines ensures that only verified changes are applied, thereby maintaining the reliability of workloads and reducing the risk of introducing issues into the production environment.

Best Practices

Implement Comprehensive Functional Testing

• Define clear functional testing requirements upfront to align with project objectives and user needs.
• Automate the functional tests to ensure they are executed consistently and efficiently during each deployment cycle.
• Integrate these tests into your Continuous Integration/Continuous Deployment (CI/CD) pipeline to catch issues early in the development process.
• Establish success criteria for your functional tests to effectively halt or roll back deployments if thresholds are not met, thus maintaining system reliability.
• Regularly review and update tests to reflect changes in functionality and user requirements, ensuring continued relevance and effectiveness.

Questions to ask your team

• How often do you run functional tests in your deployment pipeline?
• What criteria do you use to define success for your functional tests?
• How do you ensure that functional tests cover all critical features of your application?
• What processes are in place to review and update functional tests as your application evolves?
• How do you handle failures during the functional testing phase in your deployment pipeline?
• Is there a way to quickly roll back to a previous version if functional tests fail?
• How do you monitor the effectiveness of your functional tests?
• Do you involve your development and operations teams in defining functional test scenarios?

Who should be doing this?

DevOps Engineer

• Design and implement the automated deployment pipeline including functional tests.
• Ensure that functional tests are comprehensive and cover all critical functionalities.
• Monitor and analyze test results and performance metrics to identify issues.
• Collaborate with development teams to continuously improve testing strategies.
• Manage rollback procedures for deployments that do not meet success criteria.

Quality Assurance Engineer

• Develop and maintain functional test cases based on user requirements.
• Run functional tests as part of the deployment process.
• Report and track defects found during testing, ensuring visibility to the development and DevOps teams.
• Provide feedback on test coverage and quality improvements.

Product Owner

• Define acceptance criteria and success metrics for functional testing.
• Collaborate with the DevOps and QA teams to prioritize features and changes that are to be deployed.
• Ensure that changes align with business goals and user needs.

System Administrator

• Monitor deployed environments to ensure they are running stable and known software versions.
• Implement patching and maintenance procedures to keep the system secure and reliable.
• Assist in managing deployment configurations and rollback capabilities.

What evidence shows this is happening in your organization?

• Automated Deployment Pipeline Template: A template outlining the steps and stages of an automated deployment pipeline that includes functional testing as a critical component. This ensures that any deployment executes a suite of tests to validate functionality before proceeding.
• Functional Testing Strategy Guide: A comprehensive document detailing the approach to integrating functional tests into the deployment process, including best practices, tool recommendations, and examples of success criteria.
• Change Management Policy: A formal policy that mandates the integration of functional testing in deployment workflows, outlining the procedures for controlled changes and rollback mechanisms.
• Deployment Readiness Checklist: A checklist to assess the readiness of deployments, including items such as ‘Functional tests developed and ready to execute’ and ‘Success criteria defined for each test’.
• Incident Response Runbook: A runbook that provides guidelines on how to handle issues that arise from deployments, including steps for halting or rolling back changes if functional tests fail.
• Dashboard for Deployment Metrics: A visual dashboard that tracks deployment success rates, test pass/fail rates, and rollback incidents to monitor the effectiveness of the functional testing integration in deployments.

Cloud Services

AWS

• AWS CodePipeline: AWS CodePipeline automates the build, test, and release process for applications, enabling integration of functional tests in the deployment pipeline, which can halt or roll back deployments based on test results.
• AWS CodeBuild: AWS CodeBuild allows you to compile code, run tests, and produce software packages that are ready for deployment, enabling automated functional testing as part of your deployment process.
• AWS CloudFormation: AWS CloudFormation enables you to use templates to define and provision AWS infrastructure. Coupled with automated tests, it ensures reliable and repeatable deployments.

Azure

• Azure DevOps: Azure DevOps provides a set of development tools for CI/CD including pipelines that can integrate functional tests, ensuring controlled and reliable deployments.
• Azure Test Plans: Azure Test Plans is a service within Azure DevOps that provides a browser-based test management solution to create tests and run them as part of the CI/CD process.

Google Cloud Platform

• Google Cloud Build: Google Cloud Build is a service that executes builds on Google Cloud, allowing you to integrate testing into your CI/CD workflows, thereby ensuring code quality and reliability.
• Google Kubernetes Engine (GKE): GKE allows you to deploy, manage, and scale containerized applications, and you can implement CI/CD pipelines with integrated testing for reliable application deployments.

Question: How do you implement change?
Pillar: Reliability (Code: REL)