Airflow, Kubernetes and Docker Documentation

How to create a Dockerfile to automate the build image process

Creating a Dockerfile

Dockerfile is a text file which contains set of commands which are used by user to build an image via command line. Dockerfile is an automated way to build an image with set of commands.

Docker build is a command which builds an image from Dockerfile. Let's create a docker file and put some commands inside it.

touch Dockerfile

Above touch command is used to create a new file on mac/linux. Make sure you create the docker file in current directory. Although you can create the docker file anywhere but for convenience I am creating it on current directory. So that I don't have to type the absolute path for docker file.

Always remember these three commands while creating a new docker file -

FROM : This creates a layer from docker image

RUN : builds your application

MAINTAINER : Best practice to put the name of maintainer of the image

CMD : Run the commands inside container

I have created the dockerfile and put the below text inside dockerfile.

FROM ubuntu

MAINTAINER DATATIPSS

RUN apt-get update

CMD ["echo","This is my first image using dockerfile"]

I am using ubuntu as base image and giving maintainer as datatipss. You can give your name.

Now run the docker file using docker build -t myapp . command

Here -t : tag your image or giving a name to your image and .(dot) is running dockerfile in current directory.

docker build -t myapp .

[+] Building 79.3s (6/6) FINISHED

=> [internal] load build definition from Dockerfile 0.0s

=> => transferring dockerfile: 152B 0.0s

=> [internal] load .dockerignore 0.0s

=> => transferring context: 2B 0.0s

=> [internal] load metadata for docker.io/library/ubuntu:latest 7.9s

=> [1/2] FROM docker.io/library/ubuntu@sha256:9d6a8699fb5c9c39cf71bd6219f0400981c570894cd8cbea30d3424a31f 7.3s

=> => resolve docker.io/library/ubuntu@sha256: 9d6a8699fb5c9c39cf71bd6219f0400981c570894cd8cbea30d3424a31f 0.0s

=> => sha256: 9d6a8699fb5c9c39cf71bd6219f0400981c570894cd8cbea30d3424a31f 1.42kB / 1.42kB 0.0s

=> => sha256: 9d6a8699fb5c9c39cf71bd6219f0400981c570894cd8cbea30d3424a31f 529B / 529B 0.0s

=> => sha256:54ab604fab8d1b3d1c8e02509cc7031f8541428051401f4122619e5968e16 1.48kB / 1.48kB 0.0s

=> => sha256:ab2d02b1ec420fdb84c9f52abda403b6a0f5de904a2ecda5ae4f5cd6e4d46 27.17MB / 27.17MB 6.6s

=> => extracting sha256:ab2d02b1ec420fdb84c9f52abda403b6ae0f5de904a2ecda5ae4f5cd6e4d46 0.7s

=> [2/2] RUN apt-get update 63.9s

=> exporting to image 0.0s

=> => exporting layers 0.0s

=> => writing image sha256:76ff7254c9ca346bfe34c42c3f7c17ad0958e70210f01f9e200cbdadb2e4f

Ok now you have successfully created your image. Let's check and run the image using docker run imageid_or_image_name command.

Check images first:

docker images

REPOSITORY TAG IMAGE ID CREATED SIZE

myapp latest 76ff7254c9ca 3 minutes ago 92.2MB

Now run the image:

docker run myapp

This is my first image using dockerfile

So we have successfully created our first image and ran it.

Also you can inspect the image and get insights of it. If you are using image build by someone else so before running it you can always inspect it to check if it is legitimate image. You can inspect any image and check the author and what commands are inside the image.

In this case you can check the author is datatipss and under cmd tag it's echo command only.

docker image inspect myapp

[

{

"Id": "sha256:76ff7254c9ca346bfe34c42c3f7c1958e70210f01f9e200cbdadb2e4f",

"RepoTags": [

"myapp:latest"

"RepoDigests": [],

"Parent": "",

"Comment": "buildkit.dockerfile.v0",

"Created": "2021-09-28T17:40:34.86456821Z",

"Container": "",

"ContainerConfig": {

"Hostname": "",

"Domainname": "",

"User": "",

"AttachStdin": false,

"AttachStdout": false,

"AttachStderr": false,

"Tty": false,

"OpenStdin": false,

"StdinOnce": false,

"Env": null,

"Cmd": null,

"Image": "",

"Volumes": null,

"WorkingDir": "",

"Entrypoint": null,

"OnBuild": null,

"Labels": null

"DockerVersion": "",

"Author": "DATATIPSS",

"Config": {

"Hostname": "",

"Domainname": "",

"User": "",

"AttachStdin": false,

"AttachStdout": false,

"AttachStderr": false,

"Tty": false,

"OpenStdin": false,

"StdinOnce": false,

"Env": [

"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

"Cmd": [

"echo",

"This is my first image using dockerfile"

"ArgsEscaped": true,

"Image": "",

"Volumes": null,

"WorkingDir": "",

"Entrypoint": null,

"OnBuild": null,

"Labels": null

"Architecture": "arm64",

"Variant": "v8",

"Os": "linux",

"Size": 92173773,

"VirtualSize": 92173773,

"GraphDriver": {

"Data": {

"LowerDir": "/var/lib/docker/overlay2/76ff7254c9ca346bfe34c42c3f7c1958e70210f01f9e200cbdadb2e4f/diff",

"MergedDir": "/var/lib/docker/overlay2/c7dp26x65obhv96p76ff9lbo1/merged",

"UpperDir": "/var/lib/docker/overlay2/c7dp26x65obhv96p76ff9lbo1/diff",

"WorkDir": "/var/lib/docker/overlay2/c7dp26x65obhv96p76ff9lbo1/work"

"Name": "overlay2"

"RootFS": {

"Type": "layers",

"Layers": [

"sha256: 76ff7254c9ca346bfe34c42c3f7c1958e70210f01f9e200cbdadb2e4f",

"sha256:ecd02c9154cf7dc6fa2257f9378687064ebbb87524aa7191a683650b0c"

]

"Metadata": {

"LastTagTime": "2021-09-28T17:43:44.89023209Z"

}

]

That way you can check what's going to run from the image. Once you build your image you can make it publicly available for everyone. You can upload it on DockerHub and anyone can use it.

You can build multiple images of multiple applications and distribute it over the internet so that anyone can use your docker image.

Running Jenkins in Docker Container

How to run Jenkins in Docker Container

Are you looking to start with Jenkins but you don't want to setup on your host machine. Instead you want to try it on Docker? Then you are on right place. We will run the Jenkins in Docker Container.

Let's understand the Jenkins first

Jenkins is an open source automation server which can be used to automate all sorts of tasks including building, testing and deploying the software.

Jenkins can be installed through Docker or can be run standalone by any machine with Java Runtime Environment(JRE).

Pull the Jenkins Image

Below docker command will be used to download/pull the image from DockerHub.

docker pull jenkins/jenkins

docker pull jenkins/jenkins

Using default tag: latest

latest: Pulling from jenkins/jenkins

ffacddc51d82: Pull complete

781188a9e523: Pull complete

69a15122fa9e: Pull complete

1ab7a0b3a992: Pull complete

990ebae0b67d: Pull complete

dabca586d152: Pull complete

f2d0d9a7d002: Pull complete

4ac45f07946b: Pull complete

afa4a0ea0cf8: Pull complete

a20c227120b1: Pull complete

5ed87337d302: Pull complete

a64886bb3f9f: Pull complete

52920e8e3e8d: Pull complete

2639292b661e: Pull complete

c001f3ea6952: Pull complete

bb7385fc1236: Pull complete

1b9f514ccf6a: Pull complete

Digest: sha256:76d67d16e06cf63c670405b0f08dcbbbc434ca49b4a5bc9a043b51d6c2b4d285

Status: Downloaded newer image for jenkins/jenkins:latest

docker.io/jenkins/jenkins:latest

Run the image using below command

docker run -p 8080:8080 -p 50000:50000 jenkins

This command will store the workspace under /var/jenkins_home. All the plugins and configuration stays under the same path. This can't be persistent so it's always recommended to use docker volume to create persistent data.

Instead you can use below command with bind mount for data persistency.

docker run --name homejenkins -p 8080:8080 -p 50000:50000 -v /var/jenkins_home jenkins/jenkins

Output will be like below. It has the password to start your jenkins.

*************************************************************

Jenkins initial setup is required. An admin user has been created and a password generated.

Please use the following password to proceed to installation:

4772035c16234e769eaa7bcf4278

This may also be found at: /var/jenkins_home/secrets/initialAdminPassword

*************************************************************

2021-09-27 18:51:46.657+0000 [id=29] INFO jenkins.InitReactorRunner$1#onAttained: Completed initialization

2021-09-27 18:51:46.666+0000 [id=23] INFO hudson.WebAppMain$3#run: Jenkins is fully up and running

2021-09-27 18:51:59.156+0000 [id=49] INFO h.m.DownloadService$Downloadable#load: Obtained the updated data file for hudson.tasks.Maven.MavenInstaller

2021-09-27 18:51:59.157+0000 [id=49] INFO hudson.util.Retrier#start: Performed the action check updates server successfully at the attempt #1

2021-09-27 18:51:59.161+0000 [id=49] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$0: Finished Download metadata. 58,870 ms

Now go to localhost:8080 on your browser and It will popup a screen asking for this password. Once you'll enter the password for Jenkins it will give you below screen.

Click on Install plugins and wait for couple of minutes as it takes some time to install recommended plugins. Once installation is done ,you'll get username and password screen. Just put your username and password and save it. You are all set to use Jenkins.

Tip: Make sure you take the backup of your directory volume of Jenkins so that you don't loose any data.

By default you get 2 executers but you can change it and modify your image to set more number of executers.

Once everything is done you'll se below home screen for Jenkins

Click on New Item and select freestyle project and give it name. now go to build environment and select Execute shell script.

Now paste below command inside build environment. We'll print hello Jenkins then print working directory and then the user, using shell script.

echo 'Hello Jenkins'

pwd

whoami

Click on Apply and save. Now navigate to project and click Build now and your build will be running.

Once your build is completed ,click on green check sign on your build number. It will open the output console and you will see below results on console.

Started by user Pankaj D
Running as SYSTEM
Building in workspace /var/jenkins_home/workspace/simple project
[simple project] $ /bin/sh -xe /tmp/jenkins86668692502344527.sh
+ echo Hello Jenkins
Hello Jenkins
+ pwd
/var/jenkins_home/workspace/simple project
+ whoami
jenkins
Finished: SUCCESS

That was it for running Jenkins on Docker. We can use Jenkins for other options as well like creating a pipeline and integrating with github.

Note: If you think this helped you and you want to learn more stuff on devops, then I would recommend joining the Kodecloud devops course and go for the complete certification path by clicking this link

Getting started with Docker swarm, the Docker Swarm way

Getting started with docker swarm

Docker Swarm is a container orchestration tool. Docker Swarm allows user to manage multiple containers deployed across multiple hosts/machines.

In Docker Swarm you create at least one manager and one/multiple workers. Manager manages all the workers.

One of the key benefit of Docker Swarm is the high availability. You deploy your application on multiple nodes and there could be one/more than one managers managing all worker nodes and ensuring the high availability of your application.

Second important benefit is the Load Balancing. Load balancing is very crucial and important part of any application deployed in production. Docker swarm makes sure that all containers have sufficient resources and load is balanced properly across all nodes to serve optimal efficiency.

Scalability also can serve very important role to keep application scale up when it's required.

Demo

Let's start with demo part on docker swarm. To perform demo you need some need to install docker first. If you are using older docker version then docker-machine gets installed along with the docker. But in recent versions docker machine does not get install along with docker. So you need to separately install the docker machine.

You also need to install virtualbox to run docker swarm. So make sure virtualbox is also installed on your machine.

Now just the below command to check if you have docker machine on your system. Make sure your docker is running on your system. This will give you docker version.

$ docker-machine version

docker-machine.exe version 0.16.2, build bd45ab13

If you get docker-machine command not found on windows then you need to install docker-machine separately. You also need to install GitBash first. Once you have installed gitbash you can run below commands.

On windows using GitBash:

$ if [[ ! -d "$HOME/bin" ]]; then mkdir -p "$HOME/bin"; fi && \

curl -L https://github.com/docker/machine/releases/download/v0.16.2/docker-machine-Windows-x86_64.exe > "$HOME/bin/docker-machine.exe" && \

chmod +x "$HOME/bin/docker-machine.exe"

Run above command on GitBash. For Linux and Mac users follow

https://github.com/docker/machine/releases

Now we have the setup, lets create the docker manager first.

docker-machine create --driver virtualbox manager1

If this command runs successfully then we are good but if this produces the below error,

Running pre-create checks...

Error with pre-create check: "This computer doesn't have VT-X/AMD-v enabled. Enabling it in the BIOS is mandatory"

Then you probably need to bypass the virtualbox check in your command. Run the below command.

docker-machine create -d virtualbox --virtualbox-memory=4096 --virtualbox-cpu-count=4 --virtualbox-disk-size=40960 --virtualbox-no-vtx-check manager

~ manager is name of node here. You can give any name.

Meanwhile it's running the checks, You can open virtualbox and see that it is creating a VM for you to run.

I kept the name default so it was created using default name. Follow the same command with different names like below.

docker-machine create -d virtualbox --virtualbox-memory=4096 --virtualbox-cpu-count=4 --virtualbox-disk-size=40960 --virtualbox-no-vtx-check worker1

docker-machine create -d virtualbox --virtualbox-memory=4096 --virtualbox-cpu-count=4 --virtualbox-disk-size=40960 --virtualbox-no-vtx-check worker2

Since we have created all the nodes lets check what are the nodes running on system using below command.

$ docker-machine ls

NAME ACTIVE DRIVER STATE URL SWARM DOCKER

ERRORS

default - virtualbox Running tcp://192.168.99.100 v19.03.12

worker1 - virtualbox Running tcp://192.168.99.101 v19.03.12

worker2 - virtualbox Running tcp://192.168.99.102 v19.03.12

That IP address will be same for everyone. Docker generates this as default. See the last octate which is only different ie. 100,101,102 .

Lets make the default node as manager node using below command.

$ docker-machine ssh default

( '>')

/) TC (\ Core is distributed with ABSOLUTELY NO WARRANTY.

(/-_--_-\) www.tinycorelinux.net

Above command will login to default node. Once you are in the node then you can run the below command to make it manager.

docker swarm init --advertise-addr 192.168.99.100:2376

Output below:

Swarm initialized: current node (c7if56abzumka79k8mg0xbnu0) is now a manager.

To add a worker to this swarm, run the following command:

docker swarm join --token xyz

You can run the command which was just generated by manager node. You need to run this command on worked node. To login to worker node run the below command.

$ docker-machine ssh worker1

Now run the docker swarm join --token command to join this worker node as worker to manager.

Check all the nodes using docker node ls command.

So far we have created one total three nodes where one will be manager and other two will be worker nodes. That was it for preparing a setup and running your docker swarm. We have just created multiple nodes here. Then defined one of the node as manager and other two as worker node.