Simplify CI/CD with a Basic-Goal Software program Catalog

To automate deployment processes, CI/CD wants context: deployment configurations, construct configurations, artifacts, model numbers, dependencies, setting variables, check outcomes and extra. This information doesn’t exist in a single place — it’s normally scattered throughout a number of methods and instruments.

For instance, deployment configurations could be saved in a separate YAML file, setting variables could be outlined in a script or within the deployment manifest and model numbers could be tracked manually in a spreadsheet.

Too many sources of reality can result in a number of issues, together with elevated complexity, metadata inconsistency, difficulties updating information and, most of all, an lack of ability to use automation. Software program catalogs, the core of inside developer portals, can provide a solution.

First Step: A Software program Catalog That Can Retailer CI/CD Information

Step one is making a software program catalog with the best information inside. It must be a general-purpose software program catalog that permits including information varieties with totally different properties and relationships, offering flexibility to allow everybody to bring their own data model to the catalog.

An internal developer portal is on the core of platform engineering. It presents builders with the self-service actions constructed as a part of the platform and likewise with a software program catalog.

That is the place it will get attention-grabbing. From the developer expertise viewpoint, the software program catalog might be defined as a redacted, whitelisted information retailer that’s curated to assist builders overcome cognitive load (for example, see how K8s data can be presented to developers).

However that isn’t the complete story. Truly, utilizing the software program catalog for CI/CD may be very highly effective. A software program catalog can retailer information about builds, environments, cloud and much more. Such a software program catalog might be extremely helpful to create a single supply of reality for CI/CD context.

Platform engineering groups we’re speaking to are actively realizing these advantages, particularly on the subject of CI/CD metadata. They use the software program catalog as a single supply of reality for CI/CD, and are additionally utilizing the CI/CD information within the software program catalog as a part of their automated workflows.

By together with related information concerning the clusters, environments, cloud areas and distributors within the software program catalog, the CI/CD course of might be extra clever and automatic, main to raised engineering. It decouples CI/CD from the contextual information it wants, separates controls and makes it simpler to troubleshoot failures and damaged pipelines.

Via the developer portal, these capabilities additionally assist platform engineering groups present builders with higher visibility into the deployment course of, as they will see the deployment standing and any errors that happen in actual time.

Subsequent Step: Model Management and Safety

As soon as the software program catalog is ready up, the advantages of 1 supply of reality for CI/CD information might be taken even additional when it’s additionally used for model management and safety,

Monitoring all of the modifications made to the metadata and configuration information improves the traceability of metadata modifications over time. This may be helpful for auditing functions and for understanding the evolution of the deployment course of.

Moreover, it drives higher collaboration (with model and alter monitoring), quicker situation decision and the flexibility to shortly revert to a earlier model and improved compliance. When CI/CD information is fragmented — consider scattered model historical past in git — it’s troublesome to do that, nevertheless it’s a lot simpler with the software program catalog.
A software program catalog normally ensures that solely licensed customers can entry and modify the metadata, decreasing the chance of unauthorized entry, information breaches and different safety incidents. Examples are a misconfiguration that ends in making an S3 bucket public or exposing a service with personally identifiable info to the web.

How It Works

The software program catalog is actually a centralized database that shops all of the metadata associated to the CI/CD course of. It may be accessed and modified by means of a REST API, which allows CI/CD pipelines to work together with the metadata retailer programmatically. Information varieties, properties and relationships might be simply added when wanted, since totally different organizations do DevOps otherwise.

What information must be accessed and saved? This is determined by what we name your information mannequin, that means the properties and classes which might be vital inside your pipelines. As an example:

• You possibly can manage the catalog by totally different classes, every containing metadata associated to a particular side of the CI/CD course of. For instance, there could be a class for deployment configurations, a class for setting variables and a class for model management.
• Inside every class, there can be totally different metadata objects or keys. For instance, inside the deployment configurations class, there could be metadata objects for the deployment goal, the deployment technique and the deployment model.

The CI/CD pipelines can work together with the metadata retailer by utilizing a REST API, specifying the class and metadata merchandise they need to entry. For instance, to retrieve the deployment goal for a particular utility, the CI/CD pipelines may ship a GET request to the deployment configurations class, specifying the metadata merchandise for the deployment goal.

The Significance of Graph Databases for Software program Catalogs

Graph databases turn out to be useful for software program catalogs. Because the totally different entities within the software program catalog have complicated relationships (for example, a service is deployed on a namespace in a K8s cluster in a cloud account) and people relationships are vital, you want the flexibility to natively question them. A graph database allows you to just do that. That is notably helpful within the context of a CI/CD pipeline, the place builders, DevOps and machines want to have the ability to shortly entry details about how totally different elements of the system are associated.

• Or let’s say we need to establish all of the providers that use a specific picture model. With out a metadata retailer, you will have to manually search by means of varied providers’ configurations and documentation to seek out those that match. However with a graph database, we will create nodes for every service and hyperlink them to the picture model they use. This enables us to shortly question the graph to seek out all of the providers that use the specified picture model. We will begin by querying the picture model node after which traversing its relationships to the service nodes. We will even add extra info to the nodes, such because the setting the service is operating in, the date it was final up to date and any related alerts or points. This offers a complete view of the complete system and permits us to simply observe and handle our providers.

• For instance, let’s say we need to establish all providers operating in a specific area (for example, in the event you’re working a large-scale cloud platform, serving prospects throughout totally different areas). With out a graph database, we would want to carry out a number of queries throughout totally different information sources and attempt to piece collectively the knowledge. Nonetheless, with a graph database, we will do it in a single question.

This capacity to natively question complicated relationships is important in enabling builders and machines to carry out impression evaluation, handle configurations, run steady checks and handle releases extra successfully. This not solely simplifies the CI/CD course of, it additionally helps to make sure the general stability and reliability of the system.

Software program Catalogs Have to Be API-First

Now we’d like to consider how one can simply get information into the software program catalog. Simply ingesting information into the software program catalog requires an API-first strategy. This consists of information from cloud providers,Kubernetes (for cluster information), git suppliers, Infrastructure- as-Code (IaC) instruments akin to Terraform or Crossplane and extra.

An API-first strategy additionally makes it straightforward to construct integrations with different instruments and methods, akin to making a dashboard with details about your infrastructure and purposes. This can assist you construct a extra complete and helpful metadata retailer that gives a holistic view of your infrastructure and purposes.

Conclusion

The rise of platform engineering and the interior developer portals which might be used as a core interface for builders additionally presents a possibility to create a software program catalog that may be helpful not only for builders. A software program catalog with CI/CD metadata can create a single supply of reality, remedy model and safety points, and permit automation of deployment processes and extra. To see what a general-purpose software program catalog can comprise, go to Port’s live demo here.

Zohar Einy is the CEO of Port , a non-code platform for internal developer portals, which he co-founded with Yonatan Boguslavski. Zohar began his career in the Israel Defense Forces’ 8200 unit as a platform engineer, where he and Yonatan…

Read more from Zohar Einy