Problems with cached metrics


#1

On 7.8.4, I’ve a cached set of metrics at DAY and MONTH intervals for the last 24 months. For some reason, the cached version of the normalized data is stuck at Dec 6th, i.e. the metric result is in missing after this date.

I tried to use cache field in the EvalMetricsSpec but using true or false return same results:

1- When setting "cache": false:

c3Viz(Facility.evalMetrics({
  "expressions":["ElectricityConsumed","ChilledWaterConsumed","HotWaterConsumed"],
  "interval":"DAY","ids":["0088"],
  "start":"2018-11-17T00:00:00.000",
  "end":"2018-12-17T00:00:00.000",
  "cache": false
}))


2- When setting "cache": true:

c3Viz(Facility.evalMetrics({
  "expressions":["ElectricityConsumed","ChilledWaterConsumed","HotWaterConsumed"],
  "interval":"DAY","ids":["0088"],
  "start":"2018-11-17T00:00:00.000",
  "end":"2018-12-17T00:00:00.000",
  "cache": true
}))

Then if I evaluate the same expressions against a same version that’s and defined as follows:

var sm = SimpleMetric.make({
        "id" : "ElectricityConsumedNotCached",
        "name": "ElectricityConsumedNotCached",
        "srcType": "Facility",
        "path": "servicePoints.(resource.id == 'electricity').meters.measurements.(measurementType == 'consumed_totalizer')",
        "expression": "registerReadInterpolator(avg(avg(normalized.data.quantity)), 'GT', 0, null)",
        "unit": {
            "id": "kilowatt_hour"
        }
    })
res = Facility.evalMetricsWithMetadata({
    ids: ['0088'],
    expressions: ['ElectricityConsumed', 'ElectricityConsumedNotCached'],
    start: '2018-11-17T00:00:00.000',
    end: '2018-12-17T00:00:00.000',
    interval: 'DAY'
  }, [sm]);
c3Viz(res)

I get the following result:

I tried to clear caches on this cluster, refresh normalization for the PointPhysicalMeasurementSeries, but it didn’t help, the cached version of my metric is stuck at Dec 6th and all data after this data is missing. Once I remove the cache config from the metric I get good results, putting the config back and the problem reappear. Nothing seems to help!

The questions are:

  1. What’s wrong with the cache field of EvalMetricsSpec, What’s the purpose of this field?
  2. How I could refresh my cached version of these metrics?

#2

@bachr thanks for pointing this out, the cache field on the EvalMetricsSpec was meant for caching compound metrics (still an experimental feature) and we should either make it private or remove it. That field does not have anything to do with the cache field on the SimpleMetric. For now, I recommend NOT to use the cache field on the EvalMetricsSpec

When you do cache the values on the SimpleMetric, it is expected to get invalidated (if it follows the general rule of metrics - same as analytic invalidation). If it is not getting invalidated on new data arrival and it follows the rules for analytics getting triggered, then I would categorize that as a bug.

The way to invalidate the cache the SimpleMetric cache manually is using: SimpleMetric.invalidateCache


#3

thanks @rohit.sureka for the hint. I was able to invalidate this metric and get good results with help from @asaf

To check the metric is properly cached for a time range, used this

var interval = "MONTH";
var objId = "0088";
var metric = SimpleMetric.get("ElectricityConsumed_Facility");
var cacheRange = metric.cacheRange(objId, interval);
if (cacheRange == null) {
  console.log("cached");
} else {
  console.log("not cached");
}

Then to invalidate the cached data

var invalidateCacheSpecs = [{
  metricName: "ElectricityConsumed",
  objId: "0088",
  timeRanges: [{
    start: "2015-08-01T00:00:00",
    end: "2018-12-01T00:00:00"
  }],
  typeId: Tag.getTypeId(null, "Facility")
}];
SimpleMetric.invalidateCache(invalidateCacheSpecs);

#5

hmm, that is strange. How about Persistable.refreshMetrics ?