Apache Storm

Apache Storm is an Open Source distributed real time processing system.

OddEye Have two modules for monitoring Apache Storm

  1. check_storm (Monitors Storm worker processes and exposes Java Lang metrics)
  2. check_storm_api (Collect's Topology specific metrics from Storm UI-)

Storm API

Storm Workers

Install

cd ${OE_AGENT_HOME}/checks_enabled
ln -s ../checks_available/check_ceph.py ./

Configure

Storm API configuration parameters are in bigdata.ini file. Check supports Topology general aswell as per Spout/Bolt monitoring. By default per spout/bolt monitoring is turned on, if you do not need thesedetail, please set perspout or perbolt to False.

[Storm-API]
host: 127.0.0.1
port: 8080
perspout: True
perbolt: True

Restart

${OE_AGENT_HOME}/oddeye.sh restart

Provides

Name Description Type Unit
storm_acked Storm acked requests per second counter integer
storm_capacity "Shown runtime capacity should be around 1.0 " gauge
storm_complete_latency Latency for completing tasks gauge Milliseconds
storm_emitted Emitted messages per second counter integer
storm_execute_latency Message execution latency gauge Milliseconds
storm_executed Storm executions counter integer
storm_failed Failed requests counter integer
storm_process_latency Storm process latency gauge Milliseconds
storm_requested_cpu Requested CPU resources per topology member gauge integer
storm_requested_mem_off_heap Requested heap memory per topology member gauge Bytes
storm_requested_mem_on_heap Requested non heap memory per topology member gauge Bytes
storm_transferred Storm transfers counter integer

Storm Workers

Storm Workers

In order to gather Storm workers statistics, we need to download and enable Jolokia JVM Agent

Jolokia

cd /usr/share/java/
wget -O jolokia-agent.jar http://search.maven.org/remotecontent?filepath=org/jolokia/jolokia-jvm/1.3.6/jolokia-jvm-1.3.6-agent.jar

Configure Storm

As far, as Jolokia JVM Agent is downloaded, we should configure Apache Storm, to use it as JavaAgent for workers and expose metrics via HTTP/Json. To do this, edit storm.yaml. It should be in STORM_HOME/conf folder and add or append worker.childopts parameters with following:

-javaagent:/usr/share/java/jolokia-jvm-agent.jar=port=4%WORKER-PORT%,host=0.0.0.0

Based on configured supervisor.slots.ports Java agent will bind 4%WORKER-PORT% if you don't like 4, please feel free to change it to any suitable number.

By default, storm children binds on 670* ports, so after making changes in storm.yaml and restarting Storm Topology, you should see something like this, when you run netstat -nlptu | grep java

tcp     0   0 0.0.0.0:46700     0.0.0.0:*   LISTEN  6357/java       
tcp     0   0 0.0.0.0:46701     0.0.0.0:*   LISTEN  11281/java

tcp     0   0 0.0.0.0:4670x     0.0.0.0:*   LISTEN  11281/java

If you see these ports open, you can configure Agent, otherwise, something is wrong with configuration.

Configure Agent

To configure Agent, change working directory to AGENT_HOME and :

cd checks_enabled
ln -s ../checks_available/check_storm_workers.py ./

Edit conf/bigdata.init and set correct parameters for Storm :

[Storm]
host: node1
port: 46700,46701,4670x
path: /jolokia/read

You can set all ports, used by Jolokia or only those, that are interesting for you, separate them by comma and restart Agent .

cd ../ && ./oddeye.sh restart

Provides

Collected metrics will contain extra tag workerport, which will match ports configured in config file

Name Description Type Unit
storm_{g1_old/cms}_collectioncount G1 Old gen or CMS garbage collections count counter Integer
storm_{g1_old/cms}_collectiontime G1 Old gen or CMS garbage collection time rate Milliseconds
storm_{g1_young/parnew}_collectioncount G1 Young gen or ParNew garbage collections count counter Integer
storm_{g1_young/parnew}_collectiontime G1 Young gen or ParNew garbage collection time rate Milliseconds
storm_{g1_young/parnew}_lastgcinfo G1 Young gen or ParNew garbage collections last GC duration gauge MIlliseconds
storm_heap_committed Java Heap memory committed gauge Bytes
storm_heap_max Java Heap memory max gauge Bytes
storm_heap_used Java Heap memory used gauge Bytes
storm_nonheap_committed Java Non Heap memory committed gauge Bytes
storm_nonheap_max Java Non Heap memory max gauge Bytes
storm_nonheap_used Java Non Heap memory used gauge Bytes