SynCore is a SaaS company that owns a pricing-based software platform for managing RFx processes, which include requests for information, proposals, and quotes. Their software helps businesses efficiently handle these tasks by automating the process of collecting and evaluating bids.
<1% Downtime and 95% Reduced Bugs: A DevSecOps Success Story
Client Location
USA
Industry
SaaS
Project Duration
3+ Months
Project Framework
DevOps
Engagement Model
Time and Material
About the Client
The Problem
The client team’s existing setup was mostly manual, with only an automated proof of concept (PoC) pipeline in place. They used Bitbucket for DevOps, but their deployment process was very inconvenient to manage. It depended on version control systems and updates the local branch on a virtual machine without proper packaging or version control.
Essentially, the entire process was like working on a local computer - a branch is pulled to a VM, updated, and then refreshed using PM2 and Nginx to apply changes without downtime. On top of this, database changes had to be shared through Slack, which is not very organized or efficient.
Initial Needs and Challenges
SynCore’s DevOps team knew that their deployment process was not well-organized. Managing and deploying the software was complex and they had security issues too, with access keys being visible and vulnerable to potential breaches. Their current API setup also limited their ability to switch to a better, more efficient deployment process.
Automating unit tests was another challenge. Their current method using Bitbucket pipelines wasn’t very handy for tracking changes. This made it difficult to make sure the code was tested properly. They definitely needed a better way to manage these tests.
Lastly, their entire workflow had to move to an effective process that included better packaging of their software. Ideally, they wanted to serve their UI files from a Content Delivery Network (CDN) to improve performance and manageability, rather than packaging everything together in one place.
They were already working on a DevOps transformation and wanted to integrate security earlier in the process. Our role was to help them build a team with the processes and technologies needed to achieve that goal.
Dhaval Dave
VP – Operations and Delivery
What We Proposed
- We suggested using DevSecOps, a methodology that combines development, security, and operations to make the whole process streamlined and more secure.
- While DevOps focuses on better teamwork between developers and system admins, DevSecOps goes further by adding a strong focus on security from the beginning.
- DevSecOps helps by automating security gates, so they don’t slow down the development process. This ensures security without extra hassle.
- By integrating security early, DevSecOps helps in maintaining high code quality and reliability throughout the development cycle.
- Overall, DevSecOps aims to make both the development and deployment process more efficient and secure for better and faster results.
Project Objectives
The DevSecOps solution we proposed consists of the following steps – Requirement Gathering, Desing, Development, Build, Test, Release, Deployment, Operate, and Maintain.
The above steps should be executed in a sequence. In case of failure in the DevSecOps process, the build will not move to the next stage and the development team will receive notifications of failure.
Higher-Quality Product
Our top-most goal was to improve the overall quality and security of the software. For that, the software must have fewer bugs and offer better performance so that the product works as intended and meets users’ needs.
Faster Delivery
We wanted to speed up the delivery process so that new features and updates reach users more quickly. The key strategy to achieve this is automating tasks and streamlining the workflow to cut down on delays.
Improved Deployment Frequency
Another task was to increase how often new updates and features are deployed. By making the deployment pipeline more efficient, we could release updates more regularly and the software up-to-date and relevant.
Collaborative Team Culture
Our engineers focused on creating a more holistic team environment. It was imperative to improve communication and cooperation between developers, operations staff, and other team members for seamless cross-departmental workflows.
Fewer Security Incidents
We aimed to reduce the number of problems or incidents that arise in the software. By catching issues earlier and automating processes, we can prevent issues from escalating and ensure smoother operations.
Lower Failure Rate of New Releases
We wanted to minimize the chances of new updates or releases causing problems. By enhancing testing and quality checks, we can reduce the likelihood of errors and ensure new features work correctly.
Faster Recovery Times
If something goes wrong, we have to make the recovery process quicker. For that to happen, the team should have a system in place to quickly fix issues and get the software back up and running without prolonged downtime.
Build Innovative Software Products with Security as a Top Priority.
Solutions to be Implemented
We recommended using 80 hours a month for March 2022 to set up the required processes and run some pilot tests. From April 2022, we managed and monitored the process effectively. Here are the strategies we implemented:
Accessibility Features
To achieve the desired process, we used Azure DevOps services and installed an agent on the server for builds and deployments. We set up CI/CD and test pipelines and used Firebase Hosting for the front-end Angular part. Azure DevOps is secure, ideal for enterprise apps, and has a marketplace for a variety of add-ons. It offers free usage of 1800 minutes, while Firebase handles CDN automatically and cost-effectively.
Static Code Analysis
We’ll integrate static code analyzers to help the client team write secure code that forms the architectural backbone of the system. Static Application Security Testing (SAST) tools like SonarQube and TSLint or any tool of their choice would work as an automated part of their development process and help detect and fix potential vulnerabilities early. The setup will be a one-time activity for our team.
Vulnerability Scan
Security testing solutions will be implemented into the development process to check for security issues throughout the CI/CD pipeline. The team can check the code for bugs and vulnerabilities before it’s released. We’ll set up an OWASP recommended tool for this, and it will be a one-time setup as well.
Unit Tests/Penetration Tests
The client team can write tests to check individual parts of the code. We have set up a test pipeline to automate running these tests with each build. Hence, they can check everything before releasing new versions. Our team will handle the setup for this, which will be done just once.
Compliance Testing
In DevSecOps, compliance is about continuously managing and fixing security settings in real time. Instead of just ticking boxes, we’ve made sure they get alerts when any security settings change. This task is optional, and our team can handle it if needed.
Deployment Pipeline
Our team will manage the process of deploying the final build to the production environment. This involves handling various types of build outputs, such as libraries and bundles. Our team will set up the deployment process once, and their development team will trigger the deployments using this setup.
<7 Hours
of Downtime in 720 Hours in a Month (1%)
2-3
Security Incidents Cut from 10 per Quarter (-82%)
10+ times
New Build a month Instead of 4 (+57%)
Final Achievements
The client accepted our recommendations and provided our DevOps team with the necessary server details, Azure DevOps account access, and Firebase account access. After the initial setup in March, we determined the ongoing hours needed for the DevOps engineer. The same engineer also managed the server infrastructure with periodic monitoring as needed.
Faster Deployments
The deployment time for new features and updates was cut by 57%. Previously it took 5 hours, but now it only takes 2.15 hours. As a result, SynCore has doubled their release frequency. They can now deploy new builds almost 10+ times a month instead of 4.
Improved Code Quality
Automated testing caught >95% of bugs before they reached production. The software now reportedly receives a lot less user-reported issues, from 20 per month to just 1 or 2. This is a significant improvement in code quality.
Better Security
Implementing DevSecOps reduced security vulnerabilities by 82%. The number of security incidents dropped from 10 to just 2-3 per quarter – a direct result of running over 450 vulnerability scans and security checks per month.
Increased Productivity
Automatic builds, testing, and deployment saved the team around 40+ hours per month. This efficiency gain equates to one full workweek saved every month.
Higher Uptime
System reliability improved dramatically. The current downtime is less than 1% per month, which means out of 720 hours in a month, the system was unavailable for less than 7 hours.
The transformation from DevOps to DevSecOps was absolutely critical for our product and team. All we were looking for was an experienced team to help us set the path and get our hands on the process. Radixweb did it so. Everyone is happy with the workflow and our clients hardly come up with major issues.
David Barnett
COO
Work with a Team That Puts Security Front-and-Centre in Software Development
