Dimensional Data

In part one of our series, I showed you how to migrate from Atlassian’s Bamboo Server to GitLab CI/CD. In this blog post we’re going to take a deep dive into how it works from a user’s perspective. Get started You’ve deployed the demo so it’s time to play with it to understand how it works. Let’s imagine that one of the members of our project is John Doe. He is a software engineer responsible for developing some components (app1, app2, and app3) of the entire product, and he and his team would like to test those components in several combinations in myriad preview environments. So, what does that look like? First of all, let’s make some commits to the app1, app2, and app3 source code and get successful builds upon those commits. After that, we should create releases for those apps to be able to deploy them (as the deployment part of the apps CI config only shows when being triggered by a Git tag, i.e., a GitLab release). A release can be created by launching the last step (manual-create-release) in a commit pipeline. That would give us a new release with the ugly name containing the date and commit SHA in the patch part (in accord to semver scheme): On the Tags tab for the same app you now can see a deployment part of the pipeline has been triggered by the just created GitLab release but no actual environments to deploy are displayed (the _ item in the Deploy-nonprod stage is not an env): Create an environment But before that we have to briefly switch to another team who is responsible for preparing infrastructure IaC templates. Navigate to the infra/environment-blueprints project and pretend you are a member of that team doing their job. Namely, imagine you have just created some initial set of IaC files (they are already kindly prepared by me and present in the repository). You’ve tested them and now you feel that they are ready to be used by the other members of the project. You indicate such a readiness of a particular version of the IaC files by giving it a GitTag. Let’s put a tag like v1.0.0 onto the HEAD version. You will see how the tags are going to be used immediately. But first let’s make some changes to the IaC files (e.g., add a new resource for some of the apps) and create a second Git tag, let’s say v1.1.0. So, at this moment we have two versions of IaC templates (or blueprints) for our infrastructure – v1.0.0 and v1.1.0. Deploy an app into the environment Now we can return back to John and his team. We assume John is somehow informed that the version of the IaC templates he should use is v1.0.0. He wants to create a new preview environment out of the IaC templates of that version and put app1 and app2 into that env. (Here starts a description of how a user interoperates with the infrastructure-set Git repo. Notice that though the eventual idea is that it should be a Merge Request workflow – where you first get a Terraform plan within a Merge Request and can apply such a plan by merging the MR – which is widely advocated by GitLab but for the sake […]

When I faced a task of migrating from Atlassian Bamboo Server to GitLab CI/CD, I was not able to find any comprehensive information regarding something similar. So I designed a process on my own. This demo shows how to migrate a CI/CD structure for an existing multi-component application from a discontinued Atlassian Bamboo Server to GitLab CI/CD (Community Edition). The accompanying repository is https://gitlab.com/iLychevAD/ci-cd-for-a-multi-component-app. In this first part of a two-part series, you will find a description of the current state of affairs – i.e., how the CI/CD has been organized within Bamboo Server, how the Bamboo Build and Deploy plans are designed for bootstrapping infrastructure and deploying the components of the application, and the architecture of the application itself. And in part two, we’ll take a deeper look at the virtues of GitLab CI/CD. Initial state (Note: This is not a description of some particular project but more a kind of compilation of several projects I worked on.) The application solution allows the client to fulfill a particular business purpose (the nature of which is not relevant here and thus not specified) and consists of more than 50 discrete components (further referred to as applications or just apps or components). I refrain from calling them microservices as each of them looks more like a full-fledged application communicating with other siblings using REST API and messages in Kafka topics. Some of them expose a web UI to external or internal users and some are just utility parts serving the needs of other components or performing internal operations, etc. Code for each app is stored in its own Git repository (further just repo). So, a multi-repo approach is used for them. Each app may be written in different languages and packaged as one or several OCI-images for deployment. Each app repo looks like: 📦 ┣ 📂src

Continuous integration and continuous delivery (CI/CD) have become the keystone technical architecture of successful DevOps implementations. CI/CD has a reputation for being complex and hard to achieve, but that doesn’t have to be the case. Modern tools enable teams to get started with minimal configuration and infrastructure management. Here’s how you can “start fast” with CI/CD and get some quick, demonstrable performance wins for your DevOps team. What Does CI/CD Mean? CI/CD refers to a system or systems that enable software development to have continuous integration and continuous delivery capabilities. The architecture underpinning CI/CD is typically referred to as a pipeline, as software progresses through various stages akin to flowing through a pipe. What does continuous integration and continuous delivery actually mean? Taking some time to explore the more granular details will help us set some goals for getting a fast start with CI/CD. Starting on the left side of the pipeline, continuous integration encompasses a variety of automation that occurs over the course of multiple stages, designed to test and provide quick feedback on different aspects of code quality, functionality, and security. CI testing can run the gamut from unit tests and linting run locally on a developer workstation, to full integration testing suites and static analysis. Anyone that’s ever seen a small code change cause a significant outage or breakage upon reaching production knows the value of automated, repeatable testing, and the downsides of depending on manual testing. Once a code change has passed testing, it’s time to deploy. In legacy environments, system administrators and operations staff often had to manually transfer and install updates, and reboot servers to deploy new features. This type of manual work simply does not scale to the demands of the modern application ecosystem, and is error prone to boot. With continuous delivery, that code is automatically deployed to servers in a testable and deterministic way. Code can be staged in environments with less strict SLAs, such as development, staging, and QA. Once it has been verified, the new features can be launched as production workloads. In some environments, “continuous delivery” becomes “continuous deployment”, in which comprehensive testing automatically deploys new code through to production without human intervention. What’s the ultimate goal of all this automation? It’s what makes a successful software organization: faster deployment cadence. Getting Started with CI/CD With a little background established, now it’s time to focus on the key objective: getting up and running quickly. The primary goal here is to get a quick win with a CI/CD implementation to improve deployment velocity, and hopefully drive a larger effort towards standardizing on widespread and effective CI/CD usage. Getting started with CI/CD can appear daunting. There is a wealth of tools, services, and platforms available to provide specific functionality and end-to-end solutions for CI/CD. Some options like Jenkins are self-managed; others, including GitLab, have a holistic CI/CD pipeline with integrated version control. Build Your Pipeline Realistically, there is no magic bullet configuration for CI/CD. Each implementation will be highly dependent on a number of factors: the type of application being deployed, the size and skillset of the engineering team/s, the business requirements, and the scale of the application itself. The design and implementation considerations for an application that might see 100 users per day is vastly different from one that […]