TimedValue, Treatment, duplicates, overlapping, time travel


#1

There is a tsDecl metric:

{
  "id": "KPCS_Facility",
  "name": "KPCS",
  "description": "KPCS evolution",
  "tsDecl": {
    "data": "characteristics",
    "filter": "name==ServicePointCharacteristicKey.KPCS &&
               exists(evalAt) ? (compareTo(meta.created, evalAt) <= 0) : true",
    "value": "number(value)",
    "treatment": "PREVIOUS",
    "start": "timestamp"
  },
  "srcType": "Facility",
  "path":"generalGasServicePoint",
  "unit": {
    "id": "kwh-per-m3"
  },
  "variables": [{ "name": "evalAt" }]
}

Suppose there are three values of a ServicePointCharacteristicSet:

            ServicePointCharacteristicSet.make({
                timestamp: '2018-03-01T00:00:00.000Z', // same as meta.created
                name: 'kPCS',
                value: '10',
                parent: { id: this.servicePoint.id }
            }),
            ServicePointCharacteristicSet.make({
                timestamp: '2018-04-03T00:00:00.000Z', // same as meta.created
                name: 'kPCS',
                value: '20',
                parent: { id: this.servicePoint.id }
            }),
            ServicePointCharacteristicSet.make({ // meta.created == '2018-04-14'
                timestamp: '2018-04-03T00:00:00.000Z', // same as previous timestamp
                name: 'kPCS',
                value: '30',
                parent: { id: this.servicePoint.id }
            })

First of all, when all the values are present, we want to drop the second value because the third is most recent (its meta.created is bigger). Treatment PREVIOUS does not do that, but outputs 25 as the AVG of the last two values, 20 and 30.

Taking only the value with the latest meta.created among those with the same timestamp means that that value is a correction of all previous values of kPCS, which should be ignored from that meta.created on. We cannot delete those old values because there is a requirement to evaluate the metric “in the past” when requested[1], at a date passed via evalAt: in such a case, values more recent than evalAt are filtered out regardless of their timestamp.

So, in the above example, when evalAt is 2018-05-01 we get the average of the last two values, which is wrong. When evalAt is 2018-04-10, the third value is filtered out and that is what’s expected.

Is there a corresponding Treatment or such, or we have to do it in a metric, removing “duplicates” if they overlap?[2] Can this logic be extracted into a MetricFunctionLibrary efficiently? Something else?

Thanks

[1] TimeMachine: Evaluating metrics in the past
[2] Let’s simplify and assume that only values with the same timestamp are involved…


#2

Try to add:

“dataVersion” : “meta.created * 1.0”

in your tsDecl


#3

Great, thanks! I wanted to do without dataVersion and filter using meta.created only, but it is still needed.


#4

Would be nice to add overlapHandling options EARLIEST and LATEST @rohit.sureka