How and why container monitoring is so important

What is container monitoring?

Containers are ephemeral in nature, they are difficult to monitor compared to bare metal server based applications or even those running on virtualized server. Monitoring is critical to ensure avalability, performance and security of containers. So containers infrastructure requires new monitoring tools and strategies.

Container observability

Visibility and monitoring are essential a running environment and to optimize resource usage and costs.

Because each container image can have a large number of running instances and due to high pace at which new images and versions are introduced, problems can be easily spread across containers and applications and can interrupt the entire architecture. So this makes it very critical to identify the root cause of a problem as soon as it occurs.

In large scale containerized environments, this is only possible through dedicated cloud native monitoring tools.

But if you are unable to achieve observability so this can result in below:

• It is very difficult for developers and operations task to understand what is running and how it is performing. So without observability it is very difficult to troubleshoot the problem and meeting the SLA for a production system.
• Scalability is also the major challenge to achieve without observability. Scaling your application on demand can enhance your user's experience. But if scalability is too slow it can make it poor.

Challenges with container monitoring 

There are few challenges in container monitoring:

• Containers are ephemeral so provisioning and destroying a container very quick process. This is one of the biggest advantage but for complex and big production system it makes very difficult to identify the issue.
• Containers share resources. These consume resources from host machine. If there is no monitoring of resources on host machine then any point of time high CPU or memory spike can scare you and can lead your production running application to stop.

Then how can we monitor containers

You can always use alerting system to monitor your containers. Setting up alert across the delivery pipeline can prevent the risk of system failure at early stage.

What are the common features in monitoring tools 

• Real time monitoring 
• Performance baseline
• Anomaly detection
• Network Performance monitoring 
• Config monitoring 
• Dashboards
• API monitoring
• Alerting
• Automation

Here are famous container monitoring tools used by modern industries

Prometheus

Prometheus is open-source systems monitoring and alerting toolkit and it was originally built at SoundCloud. Prometheus collects and stores it's metrics s time series data ie. metrics information was stored with the timestamp at which it was recorded alongside optional key value pairs called labels.

features:

• A multi-dimensional data model with time series data identified by metric name and key/value pairs
• PromQL is a flexible query language to query the dimensionality 
• Multiple modes of  graphing and dashboard support

Grafana

With Grafana you can visualise, analyse and alert on your system. No matter where your data is stored you can create dashboards and monitor. your data source can be anything like postgres, mysql, redis etc. 

Apart from above two there are few more popular tools like ElasticsSearch and Kibana, Zabbix, datadog etc.

Note: If you think this helped you and you want to learn more stuff on devops, then I would recommend joining the Kodecloud devops course and go for the complete certification path by clicking this link

How to run PostgreSQL on Docker

Postgres on Docker 

Postgres is most advanced object relational database management system(ORDBMS). Postgres implements majority of SQL:2011 standard. It's ACID compliant and It avoids locking issues using multiversion concurrency control. So today we are going to run Postgres on Docker.

To start with Postgres we first need to pull the image from DockerHub. DockerHub is image repository for all images. Let's run the below command and pull the image:

docker pull postgres

Using default tag: latest

latest: Pulling from library/postgres

a9eb63951c1c: Pull complete 

b192c7f382df: Pull complete 

e7ce3f587986: Pull complete 

4098744a1414: Pull complete 

4c98d6f3399d: Pull complete 

65e57fefc38a: Pull complete 

d61d9528cfd5: Pull complete 

de6b20f44659: Pull complete 

25db13ff0bef: Pull complete 

7f74f4b0e936: Pull complete 

144c847b11fb: Pull complete 

cf0afd1be009: Pull complete 

fe0c14991327: Pull complete 

Now let's check that we have downloaded the image.

docker images

REPOSITORY        TAG       IMAGE ID       CREATED       SIZE

postgres          latest    83ce63c594ee   5 days ago    355MB

Let's run the image and start a container.

docker run --name test -e POSTGRES_PASSWORD=Test@123 -d postgres

Just run the docker ps command to check if container is running

docker ps

CONTAINER ID   IMAGE      COMMAND                  CREATED         STATUS   

83ec4a222   postgres   "docker-entrypoint.s…"   2 minutes ago   Up 

Let's enter in bash shell of container by running below command

docker exec -it 83ec4a222 bash

root@83ec4a222:/# 

Connect to Postgres now:

psql -h localhost -p 5432 -U postgres -w

psql (14.0 (Debian 14.0-1.pgdg110+1))

Type "help" for help.

You are connected to Postgres now. Lets' create some tables and execute some queries.

postgres=# \l

                                 List of databases

   Name    |  Owner   | Encoding |  Collate   |   Ctype    |   Access privileges   

-----------+----------+----------+------------+------------+-----------------------

 postgres  | postgres | UTF8     | en_US.utf8 | en_US.utf8 | 

 template0 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +

           |          |          |            |            | postgres=CTc/postgres

 template1 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +

           |          |          |            |            | postgres=CTc/postgres

(3 rows)

postgres=# 

Let's check the current database name by running below command.

postgres=# select current_database();

 current_database 

------------------

 postgres

(1 row)

So current database is Postgres. We'll check now how many databases are there on the system.

postgres=# select datname from pg_catalog.pg_database;

  datname  

-----------

 postgres

 template1

 template0

(3 rows)

There are total 3 databases on system.

You can check all tables on a database by querying information schema.

postgres=# select table_name from information_schema.tables limit 10;

      table_name       

-----------------------

 pg_statistic

 pg_type

 pg_foreign_table

 pg_authid

 pg_shadow

 pg_statistic_ext_data

 pg_roles

 pg_settings

 pg_file_settings

 pg_hba_file_rules

(10 rows)

We can do a lot more than this on Postgres this was just a small part about Postgres. We can get all information about all tables and databases just by using information schema. Docker can be very useful in this case when we don't want to install it on system and want to run Postgres inside container and can leverage the power of Docker.

Note: If you think this helped you and you want to learn more stuff on devops, then I would recommend joining the Kodecloud devops course and go for the complete certification path by clicking this link

How to create a Dockerfile to automate the build image process

 Creating a Dockerfile

Dockerfile is a text file which contains set of commands which are used by user to build an image via command line. Dockerfile is an automated way to build an image with set of commands.

Docker build is a command which builds an image from Dockerfile. Let's create a docker file and put some commands inside it. 

touch Dockerfile

Above touch command is used to create a new file on mac/linux. Make sure you create the docker file in current directory. Although you can create the docker file anywhere but for convenience I am creating it on current directory. So that I don't have to type the absolute path for docker file.

Always remember these three commands while creating a new docker file - 

FROM : This creates a layer from docker image

RUN : builds your application

MAINTAINER : Best practice to put the name of maintainer of the image

CMD : Run the commands inside container 

I have created the dockerfile and put the below text inside dockerfile.

FROM ubuntu

MAINTAINER DATATIPSS

RUN apt-get update

CMD ["echo","This is my first image using dockerfile"]

I am using ubuntu as base image and giving maintainer as datatipss. You can give your name. 

Now run the docker file using docker build -t myapp .  command

Here -t : tag your image or giving a name to your image and .(dot) is running dockerfile in current directory.

docker build -t myapp .

[+] Building 79.3s (6/6) FINISHED                                                                                                                             

 => [internal] load build definition from Dockerfile                                                                                                     0.0s

 => => transferring dockerfile: 152B                                                                                                                     0.0s

 => [internal] load .dockerignore                                                                                                                        0.0s

 => => transferring context: 2B                                                                                                                          0.0s

 => [internal] load metadata for docker.io/library/ubuntu:latest                                                                                         7.9s

 => [1/2] FROM docker.io/library/ubuntu@sha256:9d6a8699fb5c9c39cf71bd6219f0400981c570894cd8cbea30d3424a31f                                          7.3s

 => => resolve docker.io/library/ubuntu@sha256: 9d6a8699fb5c9c39cf71bd6219f0400981c570894cd8cbea30d3424a31f                                          0.0s

 => => sha256: 9d6a8699fb5c9c39cf71bd6219f0400981c570894cd8cbea30d3424a31f 1.42kB / 1.42kB                                                           0.0s

 => => sha256: 9d6a8699fb5c9c39cf71bd6219f0400981c570894cd8cbea30d3424a31f 529B / 529B                                                               0.0s

 => => sha256:54ab604fab8d1b3d1c8e02509cc7031f8541428051401f4122619e5968e16 1.48kB / 1.48kB                                                           0.0s

 => => sha256:ab2d02b1ec420fdb84c9f52abda403b6a0f5de904a2ecda5ae4f5cd6e4d46 27.17MB / 27.17MB                                                         6.6s

 => => extracting sha256:ab2d02b1ec420fdb84c9f52abda403b6ae0f5de904a2ecda5ae4f5cd6e4d46                                                                0.7s

 => [2/2] RUN apt-get update                                                                                                                            63.9s

 => exporting to image                                                                                                                                   0.0s

 => => exporting layers                                                                                                                                  0.0s

 => => writing image sha256:76ff7254c9ca346bfe34c42c3f7c17ad0958e70210f01f9e200cbdadb2e4f   

Ok now you have successfully created your image. Let's check and run the image using docker run imageid_or_image_name command.

Check images first:

docker images

REPOSITORY        TAG       IMAGE ID       CREATED         SIZE

myapp             latest    76ff7254c9ca   3 minutes ago   92.2MB

Now run the image:

docker run myapp

This is my first image using dockerfile

So we have successfully created our first image and ran it. 

Also you can inspect the image and get insights of it. If you are using image build by someone else so before running it you can always inspect it to check if it is legitimate image. You can inspect any image and check the author and what commands are inside the image.

In this case you can check the author is datatipss and under cmd tag it's echo command only.

docker image inspect myapp

[

    {

        "Id": "sha256:76ff7254c9ca346bfe34c42c3f7c1958e70210f01f9e200cbdadb2e4f",

        "RepoTags": [

            "myapp:latest"

        ],

        "RepoDigests": [],

        "Parent": "",

        "Comment": "buildkit.dockerfile.v0",

        "Created": "2021-09-28T17:40:34.86456821Z",

        "Container": "",

        "ContainerConfig": {

            "Hostname": "",

            "Domainname": "",

            "User": "",

            "AttachStdin": false,

            "AttachStdout": false,

            "AttachStderr": false,

            "Tty": false,

            "OpenStdin": false,

            "StdinOnce": false,

            "Env": null,

            "Cmd": null,

            "Image": "",

            "Volumes": null,

            "WorkingDir": "",

            "Entrypoint": null,

            "OnBuild": null,

            "Labels": null

        },

        "DockerVersion": "",

        "Author": "DATATIPSS",

        "Config": {

            "Hostname": "",

            "Domainname": "",

            "User": "",

            "AttachStdin": false,

            "AttachStdout": false,

            "AttachStderr": false,

            "Tty": false,

            "OpenStdin": false,

            "StdinOnce": false,

            "Env": [

                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

            ],

            "Cmd": [

                "echo",

                "This is my first image using dockerfile"

            ],

            "ArgsEscaped": true,

            "Image": "",

            "Volumes": null,

            "WorkingDir": "",

            "Entrypoint": null,

            "OnBuild": null,

            "Labels": null

        },

        "Architecture": "arm64",

        "Variant": "v8",

        "Os": "linux",

        "Size": 92173773,

        "VirtualSize": 92173773,

        "GraphDriver": {

            "Data": {

                "LowerDir": "/var/lib/docker/overlay2/76ff7254c9ca346bfe34c42c3f7c1958e70210f01f9e200cbdadb2e4f/diff",

                "MergedDir": "/var/lib/docker/overlay2/c7dp26x65obhv96p76ff9lbo1/merged",

                "UpperDir": "/var/lib/docker/overlay2/c7dp26x65obhv96p76ff9lbo1/diff",

                "WorkDir": "/var/lib/docker/overlay2/c7dp26x65obhv96p76ff9lbo1/work"

            },

            "Name": "overlay2"

        },

        "RootFS": {

            "Type": "layers",

            "Layers": [

                "sha256: 76ff7254c9ca346bfe34c42c3f7c1958e70210f01f9e200cbdadb2e4f",

                "sha256:ecd02c9154cf7dc6fa2257f9378687064ebbb87524aa7191a683650b0c"

            ]

        },

        "Metadata": {

            "LastTagTime": "2021-09-28T17:43:44.89023209Z"

        }

    }

]

That way you can check what's going to run from the image. Once you build your image you can make it publicly available for everyone. You can upload it on DockerHub and anyone can use it. 

You can build multiple images of multiple applications and distribute it over the internet so that anyone can use your docker image. 

Running Jenkins in Docker Container

How to run Jenkins in Docker Container

Are you looking to start with Jenkins but you don't want to setup on your host machine. Instead you want to try it on Docker? Then you are on right place. We will run the Jenkins in Docker Container.

Let's understand the Jenkins first

Jenkins is an open source automation server which can be used to automate all sorts of tasks including building, testing and deploying the software.

Jenkins can be installed through Docker or can be run standalone by any machine with Java Runtime Environment(JRE).

Pull the Jenkins Image

Below docker command will be used to download/pull the image from DockerHub.

docker pull jenkins/jenkins

docker pull jenkins/jenkins

Using default tag: latest

latest: Pulling from jenkins/jenkins

ffacddc51d82: Pull complete 

781188a9e523: Pull complete 

69a15122fa9e: Pull complete 

1ab7a0b3a992: Pull complete 

990ebae0b67d: Pull complete 

dabca586d152: Pull complete 

f2d0d9a7d002: Pull complete 

4ac45f07946b: Pull complete 

afa4a0ea0cf8: Pull complete 

a20c227120b1: Pull complete 

5ed87337d302: Pull complete 

a64886bb3f9f: Pull complete 

52920e8e3e8d: Pull complete 

2639292b661e: Pull complete 

c001f3ea6952: Pull complete 

bb7385fc1236: Pull complete 

1b9f514ccf6a: Pull complete 

Digest: sha256:76d67d16e06cf63c670405b0f08dcbbbc434ca49b4a5bc9a043b51d6c2b4d285

Status: Downloaded newer image for jenkins/jenkins:latest

docker.io/jenkins/jenkins:latest                                           

Run the image using below command

docker run -p 8080:8080 -p 50000:50000 jenkins

This command will store the workspace under /var/jenkins_home. All the plugins and configuration stays under the same path. This can't be persistent so it's always recommended to use docker volume to create persistent data.

Instead you can use below command with bind mount for data persistency.

docker run --name homejenkins -p 8080:8080 -p 50000:50000 -v /var/jenkins_home jenkins/jenkins 

Output will be like below. It has the password to start your jenkins.

*************************************************************

*************************************************************

*************************************************************

Jenkins initial setup is required. An admin user has been created and a password generated.

Please use the following password to proceed to installation:

4772035c16234e769eaa7bcf4278

This may also be found at: /var/jenkins_home/secrets/initialAdminPassword

*************************************************************

*************************************************************

*************************************************************

2021-09-27 18:51:46.657+0000 [id=29] INFO jenkins.InitReactorRunner$1#onAttained: Completed initialization

2021-09-27 18:51:46.666+0000 [id=23] INFO hudson.WebAppMain$3#run: Jenkins is fully up and running

2021-09-27 18:51:59.156+0000 [id=49] INFO h.m.DownloadService$Downloadable#load: Obtained the updated data file for hudson.tasks.Maven.MavenInstaller

2021-09-27 18:51:59.157+0000 [id=49] INFO hudson.util.Retrier#start: Performed the action check updates server successfully at the attempt #1

2021-09-27 18:51:59.161+0000 [id=49] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$0: Finished Download metadata. 58,870 ms

Now go to localhost:8080 on your browser and It will popup a screen asking for this password. Once you'll enter the password for Jenkins it will give you below screen.

Click on Install plugins and wait for couple of minutes as it takes some time to install recommended  plugins. Once installation is done ,you'll get username and password screen. Just put your username and password and save it. You are all set to use Jenkins.

Tip: Make sure you take the backup of your directory volume of Jenkins so that you don't loose any data.

By default you get 2 executers but you can change it and modify your image to set more number of executers.

Once everything is done you'll se below home screen for Jenkins

Click on New Item and select freestyle project and give it name. now go to build environment and select Execute shell script. 

Now paste below command inside build environment. We'll print hello Jenkins then print working directory and then the user, using shell script.

echo 'Hello Jenkins'

pwd

whoami

Click on Apply and save. Now navigate to project and click Build now and your build will be running.

Once your build is completed ,click on green check sign on your build number. It will open the output console and you will see below results on console.

Started by user Pankaj D
Running as SYSTEM
Building in workspace /var/jenkins_home/workspace/simple project
[simple project] $ /bin/sh -xe /tmp/jenkins86668692502344527.sh
+ echo Hello Jenkins
Hello Jenkins
+ pwd
/var/jenkins_home/workspace/simple project
+ whoami
jenkins
Finished: SUCCESS

That was it for running Jenkins on Docker. We can use Jenkins for other options as well like creating a pipeline and integrating with github.

Note: If you think this helped you and you want to learn more stuff on devops, then I would recommend joining the Kodecloud devops course and go for the complete certification path by clicking this link