Create a Jenkins CICD Pipeline to build a Docker Image with Splunk Integration

Hello all, in this blog of mine I would be talking about how to use Jenkins tool to create a CICD pipeline to build docker image along with the Splunk integration. Before jumping directly onto creating Jenkins pipeline, let us know what is CICD pipeline, some basic knowledge about the Docker Images, Docker Container and Splunk tool.

Continuous Integration and Continuous Deployment/Continuous Delivery

Continuous Integration is a process in which the developers push the code to a code repository (which can be any code repository like GitHub, CodeCommit). As soon as the code is pushed to the repository the testing or build server checks the code and gives the response to the developer about the tests and checks that have passed or failed. This helps the developers to find and fix the bugs at an early stage of the project development. Here the developers do not have to perform the build operation, they only have to push the code to build repository and build servers will perform the testing and building of the code.

Continuous Delivery ensures that the software is released reliably whenever needed. It ensures that the deployments of software happen often and are quick. It can be helpful in shifting away from like “one release in every 2 months” to something like “five releases in a single day”. This is known as automated deployment. A small difference between Continuous Deliver and Continuous Deployment is that, Continuous Delivery may involve manual step to “approve” deployment whereas Continuous Deployment has no manual interventions of approvals and every code changed is deployed all way to the production environment.

Process of Continuous Integration-Continuous Delivery

Jenkins

Jenkins is a popular tool for performing continuous integration of software projects. It allows continuous integration and continuous delivery of projects, regardless of the platform you are working on. It is a free source that can handle any kind of build or continuous integration. You can integrate Jenkins with a number of testing and deployment technologies. Currently Jenkins is the leading open-source automation server with some 1,600 plug-ins to support the automation of all kinds of development tasks.

Docker

Docker is a set of platform as a service (PaaS) products that use OS-level virtualization to deliver software. Docker is a software platform that allows you to build, test, and deploy applications quickly. Docker packages software into standardized units called containers that have everything the software needs to run including libraries, system tools, code, and runtime. Using Docker, you can quickly deploy and scale applications into any environment and know your code will run.

There are certain terms related to Docker which are discussed below:

Dockerfile

It all starts with a dockerfile. A Dockerfile is a file that you create which in turn produces a Docker image when you build it. A Dockerfile is a text file that Docker reads in from top to bottom. It contains a bunch of instructions which informs Docker HOW the Docker image should get built. A Dockerfile is a blueprint for building Docker images, and the act of running a separate build command produces the Docker image.

Docker Image

A Docker image is made up of a collection of files that bundle together all the essentials, such as installations, application code and dependencies, required to configure a fully operational container environment. A docker image is blueprint of the container. The docker images can be shared on Docker Hub.

Docker Container

A container is a standard unit of software that packages up code and all its dependencies so application runs quickly and reliably from one computing environment to another. The docker images become containers at runtime when they run on Docker Engine.

Splunk

Splunk is a software platform widely used for monitoring, searching, analyzing and visualizing the machine-generated data in real time. It performs capturing, indexing, and correlating the real time data in a searchable container and produces graphs, alerts, dashboards and visualizations. The machine data is generated by CPU running a webserver, IOT devices, logs from mobile apps, etc. It is not necessary to provide this data to the end users but they are extremely important to understand, monitor and optimize the performance of the machines. Splunk can read this unstructured, semi-structured or rarely structured data. After reading the data, it allows to search, tag, create reports and dashboards on these data. With the advent of big data, Splunk is now able to ingest big data from various sources, which may or may not be machine data and run analytics on big data.

Working of Splunk

Benefits of Splunk

• Offers enhanced GUI and real-time visibility in a dashboard
• It is a best-suited tool for root cause analysis.
• Splunk allows you to generate graphs, alerts, and dashboards.
• Splunk allows you to accept any data type like .csv, json, log formats, etc.
• The ingested data is indexed by Splunk for faster searching and querying on different conditions.

After getting knowledge about Docker, Jenkins and Splunk, the next section is about the complete implementation of building the pipeline and all the steps involved in doing so. After completing all the steps and building complete pipeline the flow of the project will be as shown in the below image.

Complete flow of the project

The steps in building the pipeline are as follows:

Launch EC2 Instance

i. Launch an Amazon Linux 2 AMI and choose an instance type of your choice, I have selected the default t2.micro instance type.

ii. In the Configure Instance Details step make sure the instance is launched in the public subnet with Public IP enabled. In the Advanced Details section on the same page below, add the following user data script:

#! /bin/sh
yum update -y
amazon-linux-extras install docker
service docker start
usermod -a -G docker ec2-user
chkconfig docker on

The above script installs docker, creates a docker group, adds the current ec2-user in it and starts the docker service on the instance.

iii. In the Add Storage choose the default value only, in Add Tags add a name tag if you want or skip the section.

iv. In Configure Security Group choose Create a new security group and add the rules as shown in the below image:

Add Custom TCP rule to allow connect on port 8080

v. Now Review and Launch the instance with existing key pair if you have or launch the instance with a new key pair.

vi. After launching the instance SSH into the instance.

Install Jenkins on EC2 Instance

i. Run the command to update all the packages.

sudo yum update 

ii. Check if java is installed or not using the command

java -version

If the version is not shown install java using the below command

sudo yum install java-1.8.0-openjdk

iii. Once java is installed the next step is to download latest Jenkins package. The command for the same is:

sudo wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat/jenkins.repo

iv. To enable the installation of the package, import the key file from Jenkins-CI:

sudo rpm --import https://pkg.jenkins.io/redhat/jenkins.io.key

v. Now, install Jenkins on the EC2 instance

sudo yum install jenkins

vi. Start the Jenkins service using command

sudo service jenkins start

vii. Access the Jenkins server using the public dns of EC2 instance or public ip of the instance on port 8080.

http://{ec2-public-dns}:8080​ or http://3.89.79.74:8080/

viii. Login using the username admin and to get the initial admin password execute the following command:

sudo su –
cd /var/lib/jenkins/secrets/
cat initialAdminPassword

ix. To stop jenkins the command is:

sudo service jenkins stop

Setting up CICD pipeline in Jenkins to Create Docker Image and push to Docker Hub

For setting up the pipeline firstly I have used a small Maven Project which is uploaded in my git hub repository. The link for the code is provided at the end. Now, the pipeline works in a manner that when a code is pushed from the local repository to the GitHub repository Jenkins will detect the change and start building the project and create a Docker image of the project and push it to the Docker Hub repository. Below are the steps to for setting up the mentioned pipeline:

i. First step which is required is to setup git in our local machine and push all the code to the remote Git repository.

ii. In the EC2 instance run the following commands to install git and maven required for our project.

To install git:

sudo yum install git -y
git version

To install Maven with yum:

sudo wget http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo -O /etc/yum.repos.d/epel-apache-maven.repo
sudo sed -i s/\$releasever/6/g /etc/yum.repos.d/epel-apache-maven.repo
sudo yum install -y apache-maven
mvn --version

iii. After that login to the Jenkins server and go to the Manage Jenkins section and click on the Manage Plugins.

iv. · In the Manage Plugins click on the Available tab and select the below listed plugins in the Available section and click on Install without Restart. These are the necessary plugins required for the getting the code from the git and creating and pushing the docker image to Docker Hub.

Plugins for git:

Plugins for docker:

• CloudBees Docker Build and Publish Plugin
• Docker Plugin
• docker-build-step

iv. After the plugins are installed, return to the Dashboard and click on New Item. Give the name Docker-Jenkins-Integration and choose Freestyle Project and click on OK.

Now to configure the pipeline :

• Select GitHub Project option and add your GitHub repository url.
• In Source Code Management select git and add your git repository url as shown in image below and provide credentials for git if the repository is private. Jenkins will now get the source code from the git repository which also has Dockerfile which is used for creating docker image.

• In the Build Triggers select Poll SCM and in Schedule type * * * * * this will trigger the pipeline as soon as the code is pushed in the git repository.
• Go to the Build section and from dropdown select the Invoke top-level Maven targets and in Goals mention install to perform the installs for maven.
• To create and push Docker image to Docker Hub click on Add build step and choose Docker Build and Publish. Enter the repository name of Docker Hub and add Docker Hub credentials and click on Apply and then Save to complete the pipeline.

Click on Build Now and see the output in Console and image pushed in Docker Hub registry.

Docker Image pushed to the the Docker Hub Registry

Now, the next step in the project is to integrate Splunk and Jenkins. For this first launch an AWS EC2 using Splunk Enterprise AMI. To do so the steps are as follows:

i. Go to EC2 -> Launch Instance and select AWS Market Place and search for Splunk and select Splunk Enterprise AMI as shown below. Choose an instance type of your choice, I have selected the default t2.micro instance type.

ii. In the Configure Instance Details step make sure the instance is launched in the public subnet with Public IP enabled.

iii. In the Add Storage choose the default value only, in Add Tags add a name tag if you want or skip the section.

iv. 1. In Configure Security Group choose Create a new security group you will the following rules enabled by Splunk AMI.

Security Group created by Splunk AMI

Add one more port i.e. 8088 to this list in to configure Splunk HTTP Event Collector (HEC) which will be used in the later part of project. The final security group will look like:

v. Now Review and Launch the instance with existing key pair if you have or launch the instance with a new key pair.

Login to Splunk Web UI

• Go to EC2 dashboard and copy the public DNS of Splunk Instance, paste in browser by accessing port 8000 of the server. The URL will be like, http://<PublicDNSOfSplunkInstance>:8000
• To login:

username: admin
password: SPLUNK-<InstanceID> e.g SPLUNK-i-011e2f22423cb2342

Installation of Splunk App for Jenkins

i. Login to Splunk and on left sidebar click on Manage Apps.

ii. Next Click on Browse More Apps on top right corner and search for Jenkins. You will get Splunk app for Jenkins click on Install.

iii. On clicking Install you will have to login with Splunk Account username and password. I you don’t have an Splunk Account go the link https://www.splunk.com/ and SignUp and enter the newly created account’s username and password.

iv. Once the installation is complete click on Go to Home and you can see Splunk App for Jenkins on the left sidebar as shown below.

Install Splunk Plugins and setup Splunk Configuration in Jenkins

i. Now to install Splunk Plugins in Jenkins go to Jenkins Master and click on Manage Jenkins.

ii. Click on Manage Plugins, go to Available section and search for Splunk, you will see Splunk Plugin select it and click on Install without restart.

iii. Next to configure Splunk in Jenkins go to back to Manage Plugins section and click on Configure System.

iv. In Configure System go to Splunk for Jenkins Configuration section and click on Enable.

v. In the HTTP Input Host enter the public ip of Splunk instance.

vi. To get HTTP Input Token go the Splunk Web UI and click on Settings -> Data Input->HTTP Event Collector.

vii. In the HTTP Event Collector page click on New Token on the top right corner. Next give the name for token and click on Next->Review->Submit.

viii. Go to the HTTP Event Collector page and copy the token number.

ix. Enter the copied token number in HTTP Input Token field in Jenkins.

The entire Configuration will look like:

x. Make sure that if you have SSL disabled in the Jenkins Configuration, then SSL is disabled in Global Settings of Splunk App also.

xi. Click on Test Connection in Jenkins Configuration and if the configuration it will show Splunk Connection Verified. Click on Save and Apply to apply the changes.

xii. Now restart both Jenkins and Splunk server to apply all the new changes.

Analyze the Output in Splunk

i. After restarting both the servers again build the Jenkins Pipeline and then go the Splunk Web UI and in the left sidebar click on Splunk App for Jenkins.

ii. Here it will show all the reports, graphical representation and many other details about the Jenkins build pipeline.

Graphical Representation of Build History of pipeline

Now explore the Splunk tool as much as you want and learn more!!!

So that’s how we create a Jenkins Pipeline to build Docker image with Splunk Integration.

GitHub Link: https://github.com/nidhi2802/docker-jenkins