Metric evaluation monitoring - the Splunk way


#1

One way to monitor metrics evaluation is through Splunk and its powerful query language.

For instance to look for how many metric evaluation your system perform every hour you can use something like

host=hostname-prefix* t_tenant=sometenant t_tag=sometag t_type=Facility t_action=evalMetrics | bucket _time span=1h | stats count by _time

The resulting graph will look like this:

Another question you may have is, how long it takes the system to evaluate the metrics and what the distribution looks like. For instance using this query:

host=hostname-prefix* t_tenant=sometenant t_tag=sometag t_type=Facility t_action=evalMetrics a_t < 1| bucket a_t span=0.05| stats count by a_t

The resulting graph will look like this:

Using the pie chart, you can see what percentage of evals took under certain time

Splunk is very powerful for investigating problems, use it as much as you can.