The 10 building blocks of Pipeline Design

Every organisation, from the new-found startup to the established enterprise, started their development journey from a small piece of code. Since then, they developed into a thousand different directions. What all companies have in common though, whether they develop web applications or data models, big platforms or mobile apps, is that software goes through the same process: you build it, you ship it.
‍

Continuous Integration and Delivery/Deployment (CI/CD) is not something new; it is exactly because of the immense quantity of material and knowledge out there that grasping what is important and assessing what steps to take can become daunting.

This article provides a quick overview of 10 building blocks that you should consider to implement in your pipeline design. How many have you got yet?

‍

The 10 building blocks of CI/CD

1. Code management 👑

How you manage your code is probably the first step in the journey to get it shipped to your customers. Experience shows how getting some of these processes wrong can have a long-lasting impact on your organisation.

‍

core elementS:

• Structuring repositories and how you use your version control system (git) is of sheer importance. For example, when you have a microservice architecture, knowing whether to go for a monorepo structure, with a single repository to contain multiple projects, or to separate these projects into multiple repositories owned by different teams, can later on decrease communication issues and attrition in your release process.
• Git branching strategies are also one of those things that can cost you a lot if done wrong. It is very hard to change after you choose a strategy, and will oftentimes determine how fast you can ship your code. Gitflow vs trunk-based development are just two options here: while trunk-based development allows you to have shorter-lived branches and to iterate quickly, Gitflow might still be a choice for established projects with a fixed and stable release cycle.
• Choosing the right Release Strategy will determine the speed at which you can deliver. For example, to choose a good strategy for your product, you will need to take into account internal processes such as compliance and user acceptance testing, how fast your company can market and promote the new release, how costly it is for your users to migrate from one version to another, and whether they can integrate your changes fast enough. To know that, the objective is to evaluate bottlenecks in your delivery pipeline and find the fastest speed at which you can deliver updates that still has a return for your users, and that does not hinder your company's ability to operate effectively.
• Versioning is important at so many levels: by understanding API compatibility and choosing the right API and software versioning strategies, you can guarantee high availability when deploying, and you will be able to implement specific deployment strategies. This will also allow you to split your code in multiple components (microservices) that can be deployed independently. It is paramount to understand that your API is the contract between you and your users. A correct versioning strategy and an implementation that matches the compatibility requirements between versions are required to have the ability of running multiple versions in parallel (e.g., for A/B testing), or to roll back to a previous version in case of a deployment failure.
  
  ‍

2. Pipeline design 🎨

It seems obvious, but to design effective pipelines we need to first understand the principles. In this category we collect some patterns that you might refer to throughout your journey.

core elementS:

• Master Design principles such as separation of duties, how you test and build maintainable pipelines, how to foster modularity and reusability, and how to establish standards across your organisation. Understand the different stages of a pipeline (build, test, scan…), how to bundle steps into these different stages (e.g. through reusable components), and define the interactions to registries and 3rd-party tools. Learn how to scale your pipelines in your organisation and build a golden path of reusable components that your teams can debug and extend on their own, through templating, better release processes, writing effective documentation and making it accessible at the right location, branch protection rules…
• Build with Developer Experience in mind: learn how to optimize long running pipelines, e.g. by running steps in parallel, avoiding rebuilds, and through the introduction of caching.
• Make a conscious choice between self-hosted vs managed solutions: with managed solutions you don’t have to worry about the provisioning of the infrastructure, but you depend on a third party for a critical component and its uptime. With self-hosting, you are in control of everything, but at the cost of the extra overhead implied (maintenance and implementation time, runners image creation, labeling, upgrades, etc.). Remember that the choice is based on your regulatory and security requirements, and oftentimes defined by things such as what software you are allowed to install or network layer restrictions (e.g., do I need access to components in my company private network, such as a private container registry, or can I run the build on a managed runner without the need to access any private resources?).
  
  ‍

3. Code quality ✅

One of the best ways to decrease incidents in production is to ship better code. While this is simpler said than done, plenty are the ways to help improve code quality in our pipelines. Besides improving your software reliability, automated checks that enforce better code quality can also improve your development speed, by reducing the number of code reviews and iterations required to ship your code; in the long run, it will also decrease the amount of technical debt that your projects accumulate.
‍

Core elements:

• Testing: understand the difference between different types of tests and how to efficiently embed them in your pipelines; learn how to develop an effective testing strategy, how to improve results visibility, and understand the importance of logs/artifacts retention.
• Enforcing a unified code style reducing the cognitive load in switching between different parts of your code base allows developers to navigate between projects, and collaborate on the same components effectively. Things such as committing editor/formatter configurations to your git repository, automatically checking your code formatting rules in your pipelines, and choosing the right tools for each of your programming languages, can improve your teams' collaboration.
  ‍

What else:

• Ensuring source code quality can be done through static and dynamic code analysis, or by introducing coding practices such as mandatory pull request reviews, pair/mob programming.
• Code coverage is sometimes overrated, but once you understand the different types of coverage, what matters and what does not, you can monitor the coverage that matters, e.g. that which makes your tests fail when you break something and lets you refactor with confidence.
  
  ‍

4. Deployment strategies 🚀

After you build your code, you have to ship it. Many teams fear deployments, as that is the time when things break unexpectedly. It is paramount to understand what elements you need to deploy with confidence.
‍

Core elements:

• Understanding deployment strategies: continuous deployment might be a great strategy for a software company that releases new app versions daily, but it might not apply to highly-regulated environments, where compliance or documented approvals are more important. You will need to make the choice between going for a pull-based release approach, with for instance tools that monitor your container registries for changes and pull new versions into your deployment environment as soon as they are released, or to go for a push-based approach, where you trigger a pipeline to deliver your newly-packaged code to your target environment. The choice might depend on very many factors that are unique to your teams, experience, and ultimately your product domain and industry. To meet your customers' availability and quality of service expectations, you will also need to learn and master more advanced deployment strategies such as canary releases, blue/green deployment, A/B testing, the use of feature toggles, and understand when to choose which strategy.
• Disaster recovery planning is also very important. While this topic can fall into the “governance” category, your deployment strategies highly influence how fast you are able to recover, and what steps are in place to reduce the possibility of issues reaching production. This topic is all-encompassing, but the key is to have documented procedures to recover from failed deployments, to have a team that has the correct access to perform the operations that are required to recover the system, and to continuously learn through the implementation of practices such as incident postmortems.
• Testing your deployments (shakedown testing, load testing, ephemeral environments, ...) and having good metrics to monitor rollouts will help you locate issues faster, understand your ability to recover, and ultimately find the bottlenecks in your application and infrastructure.
  ‍

5. Managing stages ✅

Having different environments that are as similar to production as possible will allow you to test your new software version before it reaches your customers, let your partners test and integrate your software before it gets released, and ultimately reduce the number of issues that arise when releasing.

As an example, you could have three environments: development, user acceptance testing (UAT) and production. Development will have fewer restrictions and will let your devs experiment with software changes and continuously integrate and test them; UAT will be the place where you validate your software release, and where partners can test their integrations; releasing to production will be the same as releasing to UAT, as the two environments were designed to be the same.

Moreover, keeping strong isolation and enforcing the right access control decreases the attack surface of your applications, while limiting the blast radius if something goes wrong.

‍

core elementS:

• Acquire a strong understanding of Design Principles to know how to separate stages effectively, grasping their value and the trade-offs between cost and availability; also, understand who is responsible and who has access to different components and environments, to follow the least-privilege access rule.
  ‍

What else?

• Test data management: how to effectively manage test data and make sure that you stay compliant with regulation is important, as you cannot simply copy production data to your testing environment. Is anonymizing data really a good idea? Should we generate or copy data between stages? How do we keep our testing as “real” as possible?

‍

6. Infrastructure as Code 🏗️

Infrastructure as Code (IaC) is tangential to code pipelines. We place it here as oftentimes you will design pipelines to apply your IaC, and these pipelines have different requirements.

‍

core elementS:

• Pipelines for infrastructure: are different from normal pipelines, and learning how to design them to prevent deploying or destroying resources by mistake is essential; for example, introducing mandatory approvals, learning how to secure your state and credentials, and keeping your infrastructure code in a separate repository with stricter access rules, can lead you to design a better and secure solution.
• Transitioning to Pipeline as Code is also key, as many people are still relying on UI tools to define their pipelines. Migrating your pipeline to code can improve reusability, maintainability, modularity, can enable templating and sharing of reusable components across your organisation, and ultimately build a more scalable solution. YAML, Groovy, or similar languages offered by your provider of choice are the way to go.
  ‍

What else?

• Database management might require a different section, as learning how to perform schema migrations, manage data backups, and perform disaster recovery, can be implemented as part of your pipelines as well.
  ‍

7. Developer experience 👩‍💻

Developer experience is what makes your internal developer platform successful, your developers happy by letting them focus on what matters, and ultimately ship your code faster.
‍

core elementS:

• Learn what really matters to developers: time wasted waiting on pipelines, flaky tests, logs that are hard to read/access, misleading error messages, missing artifacts... What are the components of a good developer experience? Things such as ease of use, shallow learning curve, great support and documentation, easy debugging…
  ‍

What else?

• Every company seems to now need developer portals, and platforms where reusable components are offered to your developers. If you are asking yourself why and how to build them, and most importantly when you would need one, you can check out our lastest video on the topic here!
  
  ‍

8. Local environments 🌴

Local development environments ensure things break before they even reach your pipeline, enabling faster development and decreasing pipeline cost.
‍

core elementS:

• Learn what makes a local environment great: reproducibility, unified and automated setup, and the same set of tools for everyone (here I am not talking about IDE, but about a single path to build, run, and test your code).
  ‍

What else?

• Learn the tools: Docker Compose, local Kubernetes, Localstack, Testcontainers and Devcontainers… (bonus points 💯 if you tell us which tools you use so that we can add them to the list)
• On-demand environments: understand why your developers might need production-like on-demand environments to integrate their code and ensure that it works (and why most of the time they probably don’t).
  ‍

9. Security 🔒

Security is the category that encompasses everything. Implementing security around your processes, and at all stages of your software development lifecycle, is key to your success.
‍

core elementS:

• Secrets management encompasses the decision of where to store, rotate, version, and backup your secrets. You might also need to introduce governance and auditing procedures, to ensure your company’s compliance with regulation.
• Application security: the focus here is on introducing best practices such as scanning your software dependencies and container images, implementing static (SAST) and dynamic (DAST) application security testing (code analysis) in your pipelines, and shifting security left, as close to your developers as possible, to where the code is written, thereby reducing the risk of incidents and reputation loss for your company.
• Learn how to secure your pipelines: know what can hijack your pipelines, understand what different actors can do (e.g., did you know that people who have write access to a GitHub repository can get access and read your GitHub Actions secrets? See here), and correctly evaluate the risk of 3rd-party dependencies.
  ‍

What else?

• Signing artifacts and continuous scanning of deployed software: how to make sure that things are what they say they are, and how to monitor live deployments for new vulnerabilities.
• Using SBOMs and IBOMs for compliance and continuous scanning. A Software Bill of Material (SBOM) is a (usually) machine-readable list of all the software components that are used in your application, including dependencies, while an Infrastructure Bill of Material (IBOM) is a list of all your infrastructure components.
  ‍

10. Governance 🛂

Governance is the final step to ensure that everything works as planned, and that your company is compliant with the relevant regulation in your industry.
‍

core elementS:

• Monitoring and observability: you might want to integrate your pipelines into a holistic monitoring solution that is accessed by your security team. You will also need to provide documentation and access, and train your developers so that they can effectively debug issues and iterate on their code changes rapidly.
• Cost management: pipelines tend to become expensive over time. Understanding where cost comes from, such as the choice between expensive licenses and adding one more person to your team who can maintain a cheaper tool, can determine a huge portion of your bills in the mid-term. Pipelines are generally expensive to run, and optimizing runners management and licensing can reduce cost significantly.
• Ensuring ownership: if no one owns it, no one fixes it. Components with no owner get outdated. Weak ownership results in people fighting over problems and generally bad communication. Make sure that your company culture fosters collaboration in a way that will result in having people that care about their code, or at least take care of it when it breaks.
  ‍

What else?

• Auditing requires you to understand what you need to have at the ready when an audit comes in, and why you should not prepare everything in advance.
• Push for continuous improvement and make sure to monitor your progress, for example through the introduction of DORA Metrics, metrics that help you track the frequency of deployment, how fast your software can be shipped, how often deployments fail, and how fast it is to recover from a failed deployment...
• Find out for each of your services the Point in Time Objective (PTO), how much data you can lose in an incident, and Recovery Time Objective (RTO), how fast you can recover from it, and how to calculate your Service Level Objectives (SLOs), the concrete metrics and values that you guarantee your customers to receive in your contract, that is, your Service Level Agreement (SLA).
  ‍

Now what?

Wow, that was a sprint 🏃 A lot of ideas one after the other, and not much time to clarify 🥵
‍

This abstraction gives you the gist of what we look at when we help companies design and implement their delivery pipelines. Here at SQUER we try to focus on what matters first, to then iterate over the 10 building blocks, based on the evolution and maturity of the projects we work on.

‍
Bonus

Everything we said but in a couple slides: