Docker Containers

Docker

Container allows developers to package up an application with all needed parts, such as libraries and other dependencies, and ship it all out as one package. Docker is the pioneer and the most popular solution for creating, sharing and using containers.
OddEye works with native Dockers HTTP API to get metrics from running containers.

Install

cd ${OE_AGENT_HOME}/checks_enabled
ln -s ../checks_available/check_docker_stats.py ./

Configure

Server

To collect Docker metrics about all your containers, you will run one OddEye Agent on every host.

In order to start collecting metrics from Docker you should first configure Docker server to expose metrics via HTTP socket. In general its just adding -H tcp://{SERVER_IP}:{PORT} parameter to dockerd at startup.

For SystemV and Upstart servers you should add following to /etc/default/docker file

DOCKER_OPTS="-H tcp://127.0.0.1:3141"

If you aur using SystemD servers like Debian 8 or newer, you should edit /lib/systemd/system/docker.service with following parameters :

ExecStart=/usr/bin/dockerd -H tcp://127.0.0.1:3141 -H fd://

After restarting docker daemon you can configure OE-Agent to get stats about containers from http://127.0.0.1:3141

Agent

Now when HOST server exposes its containers metrics via HTTP, you can enable check_docker_stats and restart OE-Agent. Config file of check_docker_stats module is conf/docker.ini. If you have followed docker configuration examples above, check_docker_stats will works with already set up default settings.

[Docker]
stats: http://127.0.0.1:3141/containers
detailed: True
memstats: True
prettynames: True

If you have configure another HOST:PORT for dockerd, please make sure to edit conf/docker.ini to match you configuration parameters.

Provides

Based on configuration of System and Agent, following metrics are collected from containers if
details flag is set to True and Memory metrics are enabled:

Name Description Type Unit
docker_bytes_rx_eth0 Amounts of received bytes rate Bytes
docker_bytes_tx_eth0 Amounts of sent bytes rate Bytes
docker_container_cpu_usage Containers CPU usage gauge Percent
docker_cpu_throttled_periods Throttled Periods of containers CPU counter Milliseconds
docker_cpu_throttled_time Throttled time in milliseconds of containers CPU counter Milliseconds
docker_master_cpu_usage Master CPU usage in percent gauge Percent
docker_mem_cur_usage Containers current memory usage gauge Bytes
docker_mem_max_usage Containers max memory usage gauge Bytes
docker_mem_total_active_anon The amount of anonymous active memory gauge Bytes
docker_mem_total_active_file The amount of anonymous file cache memory gauge Bytes
docker_mem_total_cache Block device/cache used memory of control group gauge Bytes
docker_mem_total_inactive_anon The amount of anonymous inactive memory gauge Bytes
docker_mem_total_inactive_file The amount of anonymous inactive file cache gauge Bytes
docker_mem_total_mapped_file Memory mapped by the processes in the control group gauge Bytes
docker_mem_total_rss Memory that doesn’t correspond to disk, heaps, and anon gauge Bytes
docker_mem_total_swap Total amount of containers swapped memory gauge Bytes
docker_mem_total_unevictable The amount of memory that cannot be reclaimed gauge Bytes
docker_rx_dropped_eth0 RX dropped traffic rate Integer
docker_rx_errors_eth0 RX network errors rate Integer
docker_rx_packets_eth0 RX packets rate Integer
docker_tx_dropped_eth0 TX dropped traffic rate Integer
docker_tx_errors_eth0 TX network errors rate Integer
docker_tx_packets_eth0 TX packets rate Integer

If details is set to False, these metrics will be collected from containers.

Name Description Type Unit
docker_bytes_rx_eth0 Amounts of received bytes rate Bytes
docker_bytes_tx_eth0 Amounts of sent bytes rate Bytes
docker_container_cpu_usage Containers CPU usage gauge Percent
docker_cpu_throttled_periods Throttled Periods of containers CPU counter Milliseconds
docker_cpu_throttled_time Throttled time in milliseconds of containers CPU counter Milliseconds
docker_master_cpu_usage Master CPU usage in percent gauge Percent
docker_mem_cur_usage Containers current memory usage gauge Bytes
docker_mem_max_usage Containers max memory usage gauge Bytes

Each of metrics contains container TAG which is ether container ID or Name.

If prettynames parameter in config file is set to False, you will see long container ID's at container TAG of dashboard

Containers Memory utilization

In order to be able to track containers memory utilization cgroup_enable=memory kernel parameter should be set at Servers boot. Because memory control via cgroups is resource intensive operation, most of Linux distributions, disables it by default. So make sure that server is booted with cgroup_enable=memory parameter, otherwise container's memory utilization metrics will not available.

To enable Kernel control via cgroups add following parameters to /etc/default/grub.

GRUB_CMDLINE_LINUX="cgroup_enable=memory swapaccount=1"

Run update-grub and reboot server.

Otherwise just set memstats: False in conf/docker.ini .