I don’t know about you, but I personally have around 30+ projects I’m pulling and building off various version control platforms, and it is a very tiresome process.
Boot build VM -> git pull -> whatever sorcery is involved with getting it built -> deploy built artifacts and archive them -> rinse and repeat.
The Solution & The Slight Hiccup
The solution was actually pretty obvious, setup a CI/CD pipeline to automate the entire process. The only complication? Such pipelines are typically integrated into the project repository in question, and I’m not sure the project managers would be too appreciative if I open a pull request of something along the lines of “hey please accept this PR to add a CI config file so I can automate builds for your project for myself”.
That said, its actually a pretty easy problem to solve to a certain extent, but I’ll get to that.
Gitlab
CI/CD solutions are a dime a dozen these days, with the major git hosts providing their own spins as well. I did a bit of research and experimentation and I came up with a pretty simple answer – GitLab, specifically GitLab self-hosted. It is very well priced, and exceptionally powerful while being stupidly easy to deploy, maintain and use. It also doesn’t use any weird ports that can be problematic if you want to use Cloudflare or similar.
Why self-hosted? I’m a big fan of being in control, and I plan to use the platform for my own personal projects for research and development of things I would not in any way want to be in a public space.
Deployment
I’m running Gitlab on a pretty small Debian 10 virtual machine, hosted on a machine at home. 4GB RAM, 2 vCPUs and 200GB HDD. The deployment process is really straightforward.
Replace EXTERNAL_URL with the URL you plan to run Gitlab on, don’t forget to configure an A record for your DNS and point it to the IP you are hosting on. I’m configuring it this way so I can access my home lab from anywhere on a laptop or phone, I believe you can just do an intranet setup if that’s what you desire.
sudo apt-get update sudo apt-get install -y curl openssh-server ca-certificates curl https://packages.gitlab.com/install/repositories/gitlab/gitlab-ee/script.deb.sh | sudo bash sudo EXTERNAL_URL="https://gitlab.example.com" apt-get install gitlab-ee
After its installed (yes that’s literally it), head over to your EXTERNAL_URL, set your root password and login.
Usage
Using GitLab is really self explanatory. It is also extensively documented
Setup runners
GitLab runners are also really, really simple to deploy. I’m using a docker executor on both a Debian and Windows VM. Both of which have 8GB RAM, 4vCPUs and ~120GB HDD
Debian Docker Runner
GitLab provides an apt repo which is quick to setup, as does docker. Given that the VM will never have incoming WAN traffic, only LAN, I’m adding the user to the docker group to just make things simple. This is a security risk and you should never do this without understanding the risks involved.
curl -fsSL https://get.docker.com | sh get-docker.sh sudo usermod -aG docker $your_user curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh | sudo bash sudo apt-get install -y gitlab-runner
Once the runner is installed, you need to register it with your GitLab server.
sudo gitlab-runner register
Enter your GitLab instance URL when prompted (your EXTERNAL_URL), grab your CI token from the ‘Runners’ page in the GitLab admin section EXTERNAL_URL/admin/runners and paste it when prompted. Name the runner, tag it if needed, and when asked for a ‘runner executor’, enter docker and select a base docker image to use. I use alpine:latest as a base.
Caches
The docker executor will leave a lot of trash behind, mainly unused images, and they can grow very costly very quickly. The quick way to fix this is to enable the disable_cache option under the [runners.docker] section. read: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runnersdocker-section
Builds
At its very core, the whole concept of continuous integration is purely automation, and that automation is defined in the ci config file, which is in the case of Gitlab, the .gitlab-ci.yml file.
The YML structure is pretty intuitive and well documented, so I’m not going to go over the basics of it: https://docs.gitlab.com/ee/ci/yaml/
Now that the infrastructure is setup, what’s next? Well first things first a build target is needed. I’m going to work with the TreadTear project on GitHub. It has CI integration setup for Travis, but it doesn’t catch those artifacts, so the latest release will always be behind master.
NB: You might notice in my screenshots GItLab doesn’t look like it usually does. That’s because its currently night and I really don’t feel like burning my eyes out of their sockets using the standard white-mode theme, so I’m using https://github.com/darkreader/darkreader.
First things first, a new project needs to be created
After that is done, there is a convenient “Set up CI/CD” button available on the project details page that will throw us right into a web editor and a new .gitlab-ci.yml file.
Looking at the .travis.yml file, its quick to work out how exactly our ci file must be setup. Threadtear is a Java project that makes use of the gradle build system. Perfect and simple. There is even a gradle template that can be accessed from the “Apply a template” combobox.
# This file is a template, and might need editing before it works on your project.
# This is the Gradle build system for JVM applications
# https://gradle.org/
# https://github.com/gradle/gradle
image: gradle:alpine
# Disable the Gradle daemon for Continuous Integration servers as correctness
# is usually a priority over speed in CI environments. Using a fresh
# runtime for each build is more reliable since the runtime is completely
# isolated from any previous builds.
variables:
GRADLE_OPTS: "-Dorg.gradle.daemon=false"
before_script:
- export GRADLE_USER_HOME=pwd/.gradle
build:
stage: build
script: gradle --build-cache assemble
cache:
key: "$CI_COMMIT_REF_NAME"
policy: push
paths:
- build
- .gradle
test:
stage: test
script: gradle check
cache:
key: "$CI_COMMIT_REF_NAME"
policy: pull
paths:
- build
- .gradle
The source is obviously not located in the repository, so the quick and dirty way to get it there is simply to git clone threadtear and cd threadtear to move into that directory. The template is also horribly bloated and half of the stuff is not needed so a quick clean up is a good idea. Going through the build instructions provided by GraxCode there are two commands that need to be run to build threadtear: gradle build followed by gradle fatjar to build a standalone .jar. With these changes in mind, the gitlab-ci file now looks something like this. also the chances that the alpine gradle image includes git are next to none, so installing that is also a good idea
image: gradle:alpine
variables:
GRADLE_OPTS: "-Dorg.gradle.daemon=false"
before_script:
- export GRADLE_USER_HOME=pwd/.gradle
- apk add git
- git clone https://github.com/GraxCode/threadtear.git
- cd threadtear
build:
stage: build
script:
- gradle build
- gradle fatjar
Except wait… Try that and you will see that the gradle:alpine image doesn’t actually run as root, so there is no way to run apk add git because you can’t get root and there is no way to elevate without installing additional packages, nor is there a way to specify what user to run an image as… So the workaround is to simply swap from MARKDOWN_HASH7c9fb847d117531433435b68b61f91f6MARKDOWNjdk branch is based on adoptopenjdk:8-jdk-hotspot which is ubuntu based, meaning we have sudo to install packages with and git is installed by default.
image: gradle:jdk8
variables:
GRADLE_OPTS: "-Dorg.gradle.daemon=false"
before_script:
- export GRADLE_USER_HOME=pwd/.gradle
- git clone https://github.com/GraxCode/threadtear.git
- cd threadtear
build:
stage: build
script:
- gradle build
- gradle fatjar
The final part is to upload the build artifacts. Keep in mind that we are working in a subdirectory of the build-root
image: gradle:jdk8
variables:
GRADLE_OPTS: "-Dorg.gradle.daemon=false"
before_script:
- export GRADLE_USER_HOME=pwd/.gradle
- git clone https://github.com/GraxCode/threadtear.git
- cd threadtear
build:
stage: build
script:
- gradle build
- gradle fatjar
artifacts:
paths:
- ./threadtear/build/libs
The build pipeline runs automatically after every commit, and a few seconds later everything is done and dusted
Docker Container Registry
Setting up the container registry is really simple, as SSL is handled by GitLab the only thing you have to do is define the domain/port you want the registry to run on, and make sure to create a DNS record for it. Wait for it to propagate then reconfigure gitlab to pull SSL certificates and all done.