Article series: DevOps part 3 of 3
DevOps - Deploy to Monitor
In this article, we describe the second half of a Software Development Lifecycle (SDLC) inspired by the eight parts of the DevOps loop, mapping out development and security activities to each part.
DevOps
The "correct" DevOps loop.
Deploy
Continuous Delivery - the idea and attitude that every build could be a potential release. Depending on how mature you are as a team, you might deploy to the production environment immediately (after it has passed your various test environments).
Make sure all environments are production-like, which is often easier if you work in a cloud environment. Also make sure they are built from code, so called "infrastructure as code".
Consider using feature toggles to enable/disable new functionality or features that are not yet ready to reach the end user.
Use A/B testing to experiment on what delivers the best value to your customers.
Release
"Release on demand" - making the value of your release available to customers at a time when it provides the most value. Depending on how mature you are as a team or what your business needs are, releases to customers can be either manual or automatic.
Frequent releasing leads to reduced risk. By releasing frequently, you practice the release process and troubleshooting any problems is much easier as there are fewer changes between the previous version and the new version.
Ensure that all releases are automated and do not require manual intervention. Even if the decision to make a release is manual, the procedure itself should be automated.
Make sure you can do releases to customers without downtime and during daytime when everyone involved is on site. Use Blue/Green or Canary releases and/or by using feature toggles.
Operate
Consider using "immutable infrastructure" where manual changes to an environment are not allowed (but must be done by updating code/configuration and making a new release). Perhaps it should even be a goal to not allow humans to log into production systems.
If you are building your application in one of the major cloud environments, they have security dashboards that you should check periodically (Azure Security Center, AWS Security Hub, etc.).
Dare to scan your application for vulnerabilities in the production environment, otherwise someone else will do it out of your control!
Conduct penetration tests at regular intervals, e.g. every 6-12 months. As you release more frequently, you need to decouple your penetration testing from your releases. It is simply not feasible if you release every week and a penetration test takes two weeks. After a penetration test, try to create automated test cases based on any vulnerabilities found in the test. This way you can ensure that these cannot happen again.
If you have a modern penetration testing provider, they may even be able to provide tickets to your backlog and automated test cases (instead of thick reports).
One goal could be to eventually open up your application to a bug bounty program to complement your penetration testing. This requires having a mature team and a mature application, otherwise bug bounty activities will probably consume much of your available time.
Monitor
Make sure you get safety-related metrics from your application or system environment, for example:
the ratio of successful to unsuccessful logins (large changes could indicate that your application is under attack)
an unusually high number of 4xx or 5xx errors may be an indication that you have a serious problem or are under attack
Make sure you create relevant alerts with well-described actions and try to find a good balance between too many and too few.
The goal of your metrics should be to proactively detect and address both operational and security issues before they occur and affect your customers.
Final words and more information
Now we've gone through a DevOps-inspired development lifecycle with lots of activities. The recommendation is not that you start with all of them at once. Choose the activities that you think fit you best or add the most value.
Remember that it's not just about tools, but first and foremost "people, process" and then "product". Changing behaviours may be what you should pay most attention to.
For further inspiration we would highly recommend:
The book "The DevOps Handbook"
The book "Accelerate: The Science of Lean Software and DevOps" which summarizes years of studies on how high-performance DevOps teams work
The book "The Phoenix Project" which is a short story about DevOps from an operations perspective
The book "The Unicorn Project" which is a short story about DevOps from a developer perspective
Omegapoint's series of articles "Application security - defence in depth"
The research behind years of State of DevOps reports is summarized here: