The most effective way of maintaining low technical debt is to write good quality code while developing any software. Rather than considering it as a time-consuming task, one should uphold it as an essential investment for the project and the company. The ROI of this investment keeps growing exponentially.
Over the last decade, Paytm launched many services under its umbrella. It took a huge amount of development efforts to serve the millions of customers using these services. Developers spend countless numbers of hours debugging the code along with the QA team. Paytm has 4700+ repositories, 162 projects, and 268 million lines of code at this moment. Imagine ensuring the code quality of this system to avoid downtimes without a central inspection tool. Usually, project owners at Paytm monitor the code quality at the project/repository level with tools such as SonarLint, SonarQube, StyleCop, ReSharper, etc. It lacked the Paytm wide standardized quality benchmarks. The code quality was often compromised to gain short term benefits.
This led to creating a central code quality inspection tool that would bring the same and highest level of quality for all the projects/repositories. In order to reconfirm the necessity of this tool, we monitored the Pull Requests (a merge request of new code) for 15 days. Out of 7613 PRs, 2895 were marked as ‘Passed’ by existing project-level code quality analyzers. But in reality, they failed at one of these three stages:
Introducing Central Pipeline
The Central Pipeline/Interface is nothing but a door through which any code being developed in Paytm should pass. To pass through this door, the code must clear the inspection checklist. This checklist consists of areas such as static code quality, potential bugs, reliability issues, security bugs, etc. If the code fails to complete this checklist, it is not allowed to be merged in the main codebase.
Quality/Monitoring Metrics
After multiple brainstorming sessions and discussions with technical leaders, 11 code monitoring metrics were finalized for implementation. These metrics are :
1. Bugs: Number of bug issues
2. Code Smells: A code smell is any character in the source code of a program that possibly indicates a deeper problem
3. Vulnerabilities: Number of vulnerability issues
4. Reliability Severeness (Lower the better): 1 = 0 Bugs, 2 = at least 1 Minor Bug, 3 = at least 1 Major Bug, 4 = at least 1 Critical Bug, 5 = at least 1 Blocker Bug
5. Security Severeness (Lower the better): 1 = 0 Vulnerabilities, 2 = at least 1 Minor Vulnerability, 3 = at least 1 Major Vulnerability, 4 = at least 1 Critical Vulnerability, 5 = at least 1 Blocker Vulnerability
6. Maintainability Severeness: The rating given to the project is related to the value of the Technical Debt Ratio. The default Maintainability Rating grid is: A=0-0.05, B=0.06-0.1, C=0.11- 0.20, D=0.21-0.5, E=0.51-1
7. Code Coverage: It is a mix of Line coverage and Condition coverage. Its goal is to provide an even more accurate answer to the following question: How much of the source code has been covered by the unit tests: Coverage = (CT + CF + LC)/(2*B + EL), where CT = conditions that have been evaluated to ‘true’ at least once, CF = conditions that have been evaluated to ‘false’ at least once, LC = covered lines = lines_to_cover – uncovered_lines, B = total number of conditions, EL = total number of executable lines (lines_to_cover)
8. Lines to cover: Number of lines of code that could be covered by unit tests (blank lines or full comments lines are not considered as lines to cover)
9. Uncovered lines: Number of lines of code that are not covered by unit tests.
10. Lines of code: Number of physical lines that contain at least one character which is neither whitespace nor a tabulation nor part of a comment
11. Duplicated lines (%): (Number of Duplicated lines) / (Total Number of Line) * 100
In addition to these metrics, softwares like SonarQube and Checkmarx are integrated to improve the overall inspection. SonarQube is an open-source platform developed by SonarSource for continuous inspection of code quality to perform automatic reviews with static analysis of code to detect bugs, code smells, and security vulnerabilities on 20+ programming languages.
Checkmarx is an enterprise-grade flexible and accurate static analysis solution used to identify hundreds of security vulnerabilities in custom code.
Prevention Is Better Than Cure
The idea behind this tool was neither to restrict any developer nor to intervene unnecessarily in the development process. Developers used to spend approximately 110 hours per month, per person just to resolve the production issues. The same production issues could have been prevented by the early detection of bugs. Many production outages have caused Paytm crores of rupees in terms of revenue in the last 5 years. The vision for Central Pipeline was to bring a healthy work environment which eventually leads to a better product.
Taiichi Ohno, founder of the Toyota Production System once said, “Standards (Quality) should not be forced down from above but rather set by the production employees themselves”. We followed the same and launched the CI Dashboard where project leaders can adopt custom quality inspection levels as per their needs and project. This dashboard not only provided the custom configuration options but also gave trend analysis of the above mentioned 11 monitoring metrics.
Success Stories
Based on the data collected over the period of two months, one of the top 10 highly active repositories was given a score. This score was based on the performance of all the PRs raised in a particular week. The scoring logic was as follows:
The score was given for 10 consecutive weeks. Throughout this period, on each PR raised, the developer was informed about the code quality issues as a comment on Bitbucket, as an alert on slack channels, and on email. Sample Email, Slack alert, and Bitbucket comment:
Developers and project owners worked on these alerts. Alerts helped them debug the exact issues much faster than usual. Here is the successful trend of Repo Score for this repository:
Note: The success story mentioned here is just an illustration. The actual information about projects is confidential and cannot be shared publically.
A Long Way To Go
The CI Pipeline will add great aid in early feedback and a fail-fast approach. An important goal of the fail-fast philosophy is to avoid the sunk cost effect, which is the tendency for humans to continue investing in something that clearly isn’t working. In the software world, poor code quality like lower unit test coverage, unintentional coding bugs, non-adherence to code readability standards are some of the areas which, if left untracked will have a significant impact. Hence, such practices are highly recommended in achieving faster Dev to Release cycle.
This tool will be Paytm’s Continuous Integration and Continuous Deployment platform in near future.
