How to Safely Roll Back AWS SAM Deployments
Master the art of rolling back AWS SAM deployments with our step-by-step guide and best practices for serverless applications.
Why Rollbacks Are Essential for Serverless Success
In the world of serverless development with AWS SAM, the ability to safely roll back deployments is crucial for maintaining system stability and minimizing downtime. Unlike traditional deployments, AWS SAM leverages CloudFormation stacks, making rollbacks more complex but also more powerful when done correctly.
Common scenarios requiring rollbacks include:
- New features causing unexpected errors in production
- Performance degradation after deployment
- Configuration errors in AWS SAM templates
- Third-party dependency failures
- Security vulnerabilities introduced in new code
AWS SAM Rollback Methods: Choosing the Right Approach
| Method | Best For | Complexity | Downtime Risk |
|---|---|---|---|
| CloudFormation Stack Rollback | Immediate recovery from failed deployments | Low | Medium |
| Blue/Green Deployment | Zero-downtime rollbacks | High | Low |
| Traffic Shifting with Lambda Aliases | Serverless-first applications | Medium | Low |
| Versioned S3 Bucket Rollback | Recovering from corrupted artifacts | Low | Medium |
Real-World Rollback Scenario
A fintech company deployed a new payment processing feature using AWS SAM. After deployment, they noticed a 30% increase in transaction failures. Using CloudFormation stack rollback combined with Lambda aliases, they:
- Reverted to the previous stable version in under 3 minutes
- Maintained all in-progress transactions
- Preserved debugging information for the failed deployment
Step-by-Step Rollback Guide
Identify the Failed Deployment
Use CloudFormation to list recent stack events and identify the failed deployment:
aws cloudformation describe-stack-events --stack-name my-sam-stack --query 'StackEvents[?ResourceStatus==`UPDATE_FAILED`]' --output table
Trigger Immediate Rollback
For deployments that failed during update, CloudFormation automatically rolls back. To manually trigger:
aws cloudformation cancel-update-stack --stack-name my-sam-stack
Monitor the rollback progress in the AWS Console or CLI.
Rollback Using SAM CLI
If you need to redeploy a previous version, use the SAM CLI with a previous template:
# List previous deployments aws s3 ls s3://my-deployment-bucket/ # Redeploy previous version sam deploy --template-file previous-template.yaml --stack-name my-sam-stack --s3-bucket my-deployment-bucket --capabilities CAPABILITY_IAM
Blue/Green Deployment Rollback
For zero-downtime rollbacks using blue/green deployment:
Resources:
MyLambda:
Type: AWS::Serverless::Function
Properties:
FunctionName: my-function
AutoPublishAlias: live
DeploymentPreference:
Type: Linear10PercentEvery1Minute
Enabled: true
Alarms:
- !Ref MyErrorAlarm
# Rollback on alarm trigger
EnableRollback: trueRollback Best Practices
Prevention Strategies
- Implement canary deployments to detect issues before full rollout
- Use AWS CodeDeploy for advanced deployment strategies
- Enable CloudFormation termination protection for critical stacks
- Version all artifacts in S3 for easy rollback
Automated Rollback Triggers
Configure CloudWatch alarms to trigger automatic rollbacks:
Resources:
RollbackAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmName: "High-Error-Rate"
ComparisonOperator: GreaterThanThreshold
EvaluationPeriods: 1
MetricName: Errors
Namespace: AWS/Lambda
Period: 60
Statistic: Sum
Threshold: 10
TreatMissingData: notBreaching
AlarmActions:
- !Sub "arn:aws:automate:${AWS::Region}:cloudformation:rollback-stack"
Dimensions:
- Name: FunctionName
Value: !Ref MyLambdaFunctionPost-Rollback Actions
- Preserve logs from failed deployment for debugging
- Notify stakeholders of rollback completion
- Update incident documentation with root cause analysis
- Schedule post-mortem meeting to prevent recurrence
Troubleshooting Common Rollback Issues
Stuck in ROLLBACK_FAILED State
Symptom: Stack stuck in ROLLBACK_FAILED state, preventing updates.
Solution:
# Continue rolling back aws cloudformation continue-update-rollback --stack-name my-stack # Skip problematic resources aws cloudformation continue-update-rollback --stack-name my-stack --resources-to-skip ResourceId1 ResourceId2
Resource Deletion Conflicts
Symptom: Rollback fails due to resources that can’t be deleted.
Solution:
- Manually remove dependencies blocking deletion
- Use CloudFormation resource import after rollback
- Implement resource retention policies
Advanced Rollback Techniques
Cross-Account Rollbacks
When deploying across multiple accounts, implement a coordinated rollback strategy:
# Assume role in target account aws sts assume-role --role-arn arn:aws:iam::123456789012:role/DeployRole # Set credentials export AWS_ACCESS_KEY_ID=ASIA... export AWS_SECRET_ACCESS_KEY=... export AWS_SESSION_TOKEN=... # Rollback stack in target account sam deploy --template-file previous-template.yaml --stack-name cross-account-stack --s3-bucket shared-deployment-bucket
Database Migration Rollbacks
For deployments with database changes:
- Use database migration tools with versioning
- Wrap schema changes in transactions
- Maintain backward compatibility during transitions
- Implement feature toggles instead of direct schema changes
Mastering the Rollback Process
Effective rollback strategies are essential for maintaining service reliability in AWS SAM environments. Key takeaways:
- Always have a rollback plan before deployment
- Automate rollback triggers using CloudWatch alarms
- Maintain previous deployment artifacts in versioned S3 buckets
- Implement blue/green deployments for critical services
- Regularly test your rollback procedures
By mastering these techniques, you’ll reduce downtime and maintain customer trust even when deployments don’t go as planned.
Additional Resources
`;
// Create a Blob and download
const blob = new Blob([htmlContent], { type: 'text/html' });
const url = URL.createObjectURL(blob);
const a = document.createElement('a');
a.href = url;
a.download = this.download;
document.body.appendChild(a);
a.click();
// Clean up
setTimeout(() => {
document.body.removeChild(a);
URL.revokeObjectURL(url);
}, 100);
});