Connecting to Petals JMX

Petals exposes its metrics on JMX, but Prometheus itself cannot natively gather metrics through JMX. So we need to expose those metrics on HTTP, which Prometheus can access.   
Luckily Prometheus maintains jmx_exporter which exposes metrics on HTTP. It can either act as a java agent injected into the JVM during Petals ESB startup or an independent server connecting to Petals ESB by RMI.

Installing jmx_exporter as java agent :

JMX to Prometheus exporter: a collector that can configurably scrape and expose MBeans of a JMX target.
This exporter is intended to be run as a Java Agent, exposing a HTTP server and serving metrics of the local JVM. It can be also run as an independent HTTP server and scrape remote JMX targets, but this has various disadvantages, such as being harder to configure and being unable to expose process metrics (e.g., memory and CPU usage). Running the exporter as a Java Agent is thus strongly encouraged.

• Copy jmx_prometheus_javaagent-XXX.jar in petals-esb-directory/lib folder
• Create a yaml config file in petals-esb-directory/conf folder, here it is named prometheus-jmx.yaml. The file can be empty for now, but this default config will display everything available :

  startDelaySeconds: 0
  rules:
    -pattern: ".*"
  

• Add the following line to petals-esb.sh, just before the “exec” command at the very end of the script. If necessary, change the version number to match the jar file you downloaded. 8585 is the port number on which HTTP metrics will be exposed (once gathered by the jmx_exporter), set is as you see fit.

  JAVA_OPTS="$JAVA_OPTS -javaagent:${PETALS_LIB_DIR}/jmx_prometheus_javaagent-0.3.1.jar=8585:${PETALS_CONF_DIR}/prometheus-jmx.yaml"

• Run petals-esb.sh
• Metrics are available at http://localhost:8585/metrics

Alternate jmx_exporter install: as HTTP server

• Download jmx_prometheus_httpserver. Be careful about the confusing version number, check the date to have the last version.
• Adapt the prometheus-jmx.yaml config file to connect by RMI. You can use either jmxUrl or hostPort, username and password are mandatory.

  startDelaySeconds: 0
  
  # jmxUrl: service:jmx:rmi:///jndi/rmi://localhost:7700/PetalsJMX
  hostPort: localhost:7700
  username: petals
  password: petals
  
  rules:
   - pattern: ".*"
  

• Start the server, with the exposition HTTP ip:port and config file as argument :

  java \-jar jmx_prometheus_httpserver-0.3.1-jar-with-dependencies.jar localhost:8585 prometheus-jmx.yaml

Install Prometheus

• Install : https://prometheus.io/docs/prometheus/latest/getting_started/
• Configure Prometheus, here is a sample prometheus.yml config:

  global:
    scrape_interval: 5s
    evaluation_interval: 5s
  
  scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
    - job_name: 'petals monitoring'
  
  # metrics_path defaults to '/metrics'
  # scheme defaults to 'http'.
    static_configs:
      - targets: ['localhost:8585']
      labels:
        groups: 'petals'
  

• Start Prometheus :

  ./prometheus --config.file=prometheus.yml

Configuring jmx agent

Jmx agent can be configured in its yaml config file.

Note that :

• Only numeric values are supported by Prometheus (though string can me interpreted as regexp to extract numeric values)
• Custom and complex objects may not be exported by the exporter, having ‘- pattern “.”’ as only rule will return every metric available* (useful for testing). 
• Petals ESB container MBeans metrics are all typed as Map, so are ignored by the jmx agent. As is, you can monitor some components metrics but cannot monitor container metrics with Prometheus.
• Rules order is important: Eventually, a single MBean is processed by a single rule! To decide which rule is applied: MBeans will be parsed by each rule (in order) until a pattern matches, then this rule is applied to the MBean. In other words, all rules are tested against each MBean the first one to match is kept for the MBean. So very specific rules should be put first, and generic/default rules last.
• Prometheus can make extensive use of labels through queries to determine metrics sources. Think about your needs when designing your labels, more explanations on the official documentation or this blog post.
• Metrics can be typed (conceptually, as gauge, counter or histogram) for Prometheus to know how to handle them. More details on the official documentation.

Rule samples :

rules:
- pattern: 'java.lang<type=Runtime, key=java.runtime.name><>SystemProperties: (.*)'
  name: aaaa_test
  value: 1
  labels:
    runtime: "$1"

 - pattern: '(.*)<(.*)><(.*)>(.*)'
   name: aaaa_petals_test
   value: 1
   labels:
     mouais: "$1 $2 $3 $3"

 - pattern: '(\w+)<type=(\w+), name=(\w+)><>Value: (\w+)'
   name: $1_$2_$3
   value: 1
   help: "$1 metric $2 $3"
   labels:
     value: "$3: $4"