Kafka

Apache Kafka is distributed stream processing engine, written in Java/Scala, initially developed by Linked in. Indeed it's a top level Apache foundation product.

Enablling Kafka checks has two parts:

  • Configure Kafka to expose metrics via Jolokia
  • Kafka checks and configure Agent with Jolokia access parameters.
Install

At first download Jolokia jolokia-jvm-VERSION-agent.jar from https://jolokia.org/download.html and copy it to All nodes of Kafka cluster.

cd /usr/share/java/
wget -O jolokia-agent.jar http://search.maven.org/remotecontent?filepath=org/jolokia/jolokia-jvm/1.3.6/jolokia-jvm-1.3.6-agent.jar

Now we need to edit kafka-server-start.sh and add javaagent JVM option: lets assume that you have installed Kafka at /opt/kafka directory.

export  KAFKA_OPTS="$KAFKA_OPTS -javaagent:/usr/share/java/jolokia-agent.jar=config=/opt/kafka/config/jolokia/jolokia.properties"

Jolokia

Now we need to configure Jolokia: For packaged Kafka installations copy-paste of code below will do the trick. If you have tarball Kafka installation, just replace paths with actual location of your config files .

mkdir /opt/kafka/config/jolokia
cat > /opt/kafka/config/jolokia/jolokia.policy <<-EOF
<?xml version="1.0" encoding="utf-8"?>
<restrict>
  <http>
    <method>get</method>
    <method>post</method>
  </http>
  <commands>
    <command>read</command>
    <command>list</command>
    <command>search</command>
  </commands>
</restrict>
EOF
cat > /opt/kafka/config/jolokia/jolokia.properties <<-EOF
host=0.0.0.0
port=7777
agentContext=/jolokia
backlog=100
policyLocation=file:///opt/kafka/config/jolokia/jolokia.policy
historyMaxEntries=10
debug=false
debugMaxEntries=100
maxDepth=15
maxCollectionSize=1000
maxObjects=0
EOF

Restart Kafka node.

Agent

cd ${OE_AGENT_HOME}/checks_enabled
ln -s ../checks_available/check_kafka.py ./

Kafka check is enabled as any other check: so copy or symlink check_kafka.py check from checks_available to checks_enabled.

Configure

Edit conf/mq.ini, it already contains reasonable defaults to work out of the box, but if your Kafka and Jolokia are configured in other than described below way, just replace default parameters with desired ones.

[Kafka]
jolokia: http://127.0.0.1:7777/jolokia/read
Restart
{AGENT_HOME}/oddeye.sh restart
Provides

After several seconds you can start building your Kafka dashboards. Metrics will be prefixed with kafka_ .

Name Description Type Unit
kafka_bytesinpersec Amount of incoming bytes to Kafka node rate Bytes
kafka_bytesoutpersec Amount of outgoing bytes to Kafka node rate Bytes
kafka_bytesrejectedpersec Amount of rejected bytes to Kafka node rate Bytes
kafka_failedfetchrequestspersec Amount of failed fetch requests to node rate OPS
kafka_failedproducerequestspersec Amount of failed produce requests to node rate OPS
kafka_fetch Fetches on current node rate OPS
kafka_fetchconsumer Fetch consumers on current node rate OPS
kafka_fetchfollower Fetch followers on current node rate OPS
kafka_gc_old_collectioncount CMS or G1 Old gen garbage collections count counter None
kafka_gc_old_collectiontime CMS or G1 Old gen garbage collections time rate Milliseconds
kafka_gc_old_lastgcinfo CMS or G1 Old gen last garbage collections duration gauge Milliseconds
kafka_gc_young_collectioncount ParNew or G1 Young gen garbage collections count rate Milliseconds
kafka_gc_young_collectiontime ParNew or G1 Young gen garbage collections time rate Milliseconds
kafka_gc_young_lastgcinfo ParNew or G1 Young gen last garbage collections duration gauge Milliseconds
kafka_groupcoordinator Kafka group coordinators gauge None
kafka_heap_committed Java Heap memory committed gauge Bytes
kafka_heap_used Java Heap memory used gauge Bytes
kafka_joingroup Kafka join groups rate OPS
kafka_leaderandisr LeaderAndIsr requests performed ont his node rate OPS
kafka_messagesinpersec Incomming messages rae to current node rate OPS
kafka_nonheap_committed Java Non Heap memory committed gauge Bytes
kafka_nonheap_used Java Non Heap memory used gauge Bytes
kafka_offsetcommit Offset commits on current node rate OPS
kafka_offsetfetch Offset fetched on current node rate OPS
kafka_offsets Kafka Offsets gauge None
kafka_produce Prodeucers per second on current node rate OPS
kafka_totalproducerequestspersec Kafka total produced requess per second rate OPS