Our current goal is to create a new Jenkins pipeline that clones our github repo to jenkins workspace and build docker images that are defined in docker-compose.yml file by using docker compose commands in the jenkinsfile. After these images are built, they are going to be pushed to their corresponding DockerHub repositories. For now we will trigger jenkins pipeline manually to see if everything works properly. At the end of this article, we will create a github webhook and jenkins pipeline then will be triggered automatically by the webhook.
In one of my previous article, I prepared a django web application and used docker compose to build and run the containers. We will be working on that specific application in this article. You can see the previous article from here. The file & folder structure is as follows. Content of those files have already shared in the previous article.
I created 3 new repositories in DockerHub. Each image will be kept in different repos. I named the new repos like below.
The current docker-compose.yaml file has the following sections for the images. As you see there is no tag for the image.
image: selimica/website-app
image: selimica/website-postgres
image: selimica/website-nginx
It is better to tag the images while building them. I want to tag them with the Jenkins build number. Therefore I added IMAGE_TAG. In jenkins file you will see that IMAGE_TAG = "${BUILD_NUMBER}". On the other hand, when I am building the image locally on my laptop, because there is no ${BUILD_NUMBER} variable ,tag "dev" will be used.
Go ahead and change them like this
image: selimica/website-app:${IMAGE_TAG:-dev}
image: selimica/website-postgres:${IMAGE_TAG:-dev}
image: selimica/website-nginx:${IMAGE_TAG:-dev}
The complete new docker-compose.yaml file should look like this.
services:
app:
build: .
image: selimica/website-app:${IMAGE_TAG:-dev} #we tag the images with this tag while building
volumes:
- .:/web
- static:/static
ports:
- 8000:8000
env_file:
- .env
depends_on:
db:
condition: service_healthy
networks:
- mynet
db:
build: ./postgres
image: selimica/website-postgres:${IMAGE_TAG:-dev} #we tag the images with this tag while building
volumes:
- ./postgres_data:/var/lib/postgresql/data
ports:
- 5432:5432
environment:
- POSTGRES_DB=webdb
- POSTGRES_USER=dbuser
- POSTGRES_PASSWORD=dbpass
healthcheck:
test: ["CMD-SHELL", "pg_isready -U dbuser"]
interval: 5s
timeout: 5s
retries: 5
networks:
- mynet
nginx:
build: ./nginx
image: selimica/website-nginx:${IMAGE_TAG:-dev} #we tag the images with this tag while building
volumes:
- static:/static
ports:
- 80:80
depends_on:
- app
networks:
- mynet
networks:
mynet:
driver: bridge
volumes:
static:
postgres_data:
The following is the jenkinsfile that we use for building and pushing images. This pipeline assumes Docker Compose V2 is installed.
pipeline {
agent any
options {
timestamps()
disableConcurrentBuilds()
buildDiscarder(logRotator(numToKeepStr: '30'))
}
environment {
IMAGE_TAG = "${BUILD_NUMBER}"
DOCKER_BUILDKIT = '1'
COMPOSE_DOCKER_CLI_BUILD = '1'
}
stages {
stage('Cleanup Workspace') {
steps { cleanWs() }
}
stage('Checkout SCM') {
steps {
git url: 'https://github.com/selimatmaca/website.git',
branch: 'main',
credentialsId: 'GitHub' // Your Jenkins GitHub Credential ID
}
}
stage('Build images (Compose)') {
steps {
sh 'docker compose build --pull'
}
}
stage('Login & Push') {
steps {
withCredentials([usernamePassword(
credentialsId: 'DockerHub', // Your Jenkins DockerHub Credential ID
usernameVariable: 'DOCKERHUB_USR',
passwordVariable: 'DOCKERHUB_PSW'
)]) {
sh '''
echo "$DOCKERHUB_PSW" | docker login -u "$DOCKERHUB_USR" --password-stdin
for IMG in website-app website-nginx website-postgres; do
docker push selimica/$IMG:${IMAGE_TAG}
docker tag selimica/$IMG:${IMAGE_TAG} selimica/$IMG:latest
docker push selimica/$IMG:latest
done
'''
}
}
}
}
post {
always {
sh 'docker image prune -af || true'
sh 'docker builder prune -af || true'
sh 'docker logout || true'
}
}
}
Explaining the pipeline above:
agent any: Run on any available Jenkins node/agent.
timestamps(): Prefixes each console line with a time, useful for debugging
disableConcurrentBuilds(): Prevents two builds of this job from running at the same time (new ones will queue).
buildDiscarder(logRotator(numToKeepStr: '30')): Keeps only the last 30 builds’ metadata/logs/artifacts; older builds are discarded (saves disk).
IMAGE_TAG: The image tag you’ll use is the Jenkins build number (e.g., 42). This lines up with your docker-compose.yml using ${IMAGE_TAG:-dev}.
DOCKER_BUILDKIT=1: Enables Docker’s BuildKit for faster, more efficient builds.
COMPOSE_DOCKER_CLI_BUILD=1: Tells Docker Compose v2 to use the Docker CLI build (BuildKit/buildx under the hood) instead of legacy build.
Cleanup Workspace: Deletes all files from the workspace before the run.
Checkout SCM: Clones your repo into the workspace.
Build images (Compose): Runs Docker Compose v2 to build all images defined in your docker-compose.yml. "--pull" ensures Compose pulls newer base images first. If you do not use --pull, it could build images from the local cache.
Login & Push: withCredentials(usernamePassword): Securely exposes your Docker Hub username/password to the shell as env vars. docker login using --password-stdin avoids putting secrets in the process list. Push the versioned tag (:${IMAGE_TAG} = build number) for each image. Create/overwrite the latest tag pointing at the same image.
For Loop: IMG variable becomes website-app in the first loop, then it becomes website-nginx in the second loop and it becomes website-postgres in the third loop. Then it pushes the images to the registry as for example
selimica/website-app:42,
selimica/website-nginx:42,
selimica/website-postgres:42
Then it adds another tag (latest tag) to the existing image
Then it pushes the image that image to the registry
Post actions:
- always: Run these steps no matter if the build succeeded or failed.
- docker image prune -af Removes all unused images to free disk space without asking approval(f for force).
- docker builder prune -af: Cleans up the build cache
- || true is used, to ignore any errors such as no image found, so pipeline will not fail
Time to commit and push to Git. So new jenkinsfile will be uploaded to our git repo. We need this because when Jenkins job runs, it should copy the new jenkinsfile to its workspace.
git add .
git commit -m "jenkinsfile changed"
git push
On VSCode, Ctrl + Shift + P and run Jenkins Pipeline. Note that we are running pipeline manually at this point. We will automate all these later.
By using settings.json, VSCode triggers the pipeline named "website_CI".
Jenkins first cleans up the workspace. Then it fetches the files and folders from GitHub. Docker compose builds the images according to the docker-compose.yml which resides in the workspace. Then local images in workspace are tagged according to DockerHub repos. Finally we push the images to the correct DockerHub repos.
After Jenkins displayed success message, I checked DockerHub repos and saw each repo has its images. Everytime I run this Jenkins pipeline, it will build and push a new image to each repo with the new build number and it overwrites the latest tagged image.
Git Webhook:
We can use GitHub webhook to run jenkins jobs automatically. Let's say a developer changes some codes on his local machine and commits and pushes to github. By using webhook we can tell Jenkins server that there is a code change and jenkins job should be triggered automatically. We need Git plugin on our Jenkins server, wehave already installed that. Log on to your jenkins server > Manage Jenkins > System > Scroll down to GitHub > Click Advanced > click the checkbox "Specify another hook URL for GitHub configuration" and copy the url. We just need to copy that url. Don't save anything on this page.
Payload URL is the url we just copied. Change content type to application/json
I did not configure https communication for jenkins, if you run jenkins over https, you can select SSL verification. For production setups, always use HTTPS to protect credentials and webhook payloads.
When the GIT repo receives an event (for example push), it sends a notification to the Payload URL in JSON format. When Jenkins is notified with this event, it triggers the relevant pipeline thanks to GitHub plugin
Now go back to your Jenkins server and choose the jenkins job > Configure > Scroll down to "Build Triggers" > Check "GitHub hook trigger for GITScm polling" and Save.
Now I am going to test if webhook works properly. On my local machine, I just created a test filenamed "webhooktest.txt" and then used git commands to commit and push.
git add .
git commit -m "webhooktest"
git pushOn jenkins server, my jenkins job started automatically. Git repo cloned, images are created and pushed to DockerHub successfully. We have completed Jenkins section for our project (at least for now). next article we will discuss Argo CD.