Quantity in RegisterMeasurement as rate acts differently from other defined timeseries (same treatment)


#1

Hi,
I was trying to get the optimal RegisterMeasurement variant acting on treatment annotation.
I have a quantity which have to be missing when it’s the case.
other fields to represent coefficients must be present along the timeserie, even when not present, so I thought @ts(treatment='rate') should be the appropriate as for my understanding, rate or avg weights next and previous value when the current is absent.

The problem is that I never get it working. I turned to basic experimental tries. Here I have a reproducible case:

remix type RegisterMeasurement {

  @ts(treatment='rate')
  quantity : ExactDimension

  flagToZero: double

  @ts(treatment='rate')
  energyValue           :ExactDimension
  @ts(treatment='rate')
  ptaCoef               :ExactDimension
  @ts(treatment='rate')
  conversionCoef        :ExactDimension
  @ts(treatment='rate')
  volumeValue           :ExactDimension
}

Strangely, quantity is weighted when not present, in the third value:

* 0: 20
* 1: 35
* 2: 35
* 3: 50
* 4: 60

(20 + 50) / 2

Strangely, with the same treatment, ptaCoef and conversionCoef got results like:

  • 0: 9.05
  • 1: 9.05
  • 2: 0
  • 3: 9.05
  • 4: 9.05

So the reverse aimed results. I would like this behaviour for quantity, with missing at index 2. and average goes for ptaCoef and conversionCoef

Please bear with me, my data is like :

servicePointId ptaCoef conversionCoef startIndex start endIndex end flagToZero energyValue volumeValue
17755137471680 0.969 9.05 200 2018-01-01T00:00:00.000+01:00 210 2018-01-02T00:00:00.000+01:00 0 90.5 10
17755137471680 0.969 9.05 210 2018-01-02T00:00:00.000+01:00 230 2018-01-03T00:00:00.000+01:00 0 181 20
17755137471680 0.969 9.05 260 2018-01-04T00:00:00.000+01:00 300 2018-01-05T00:00:00.000+01:00 0 362 40
17755137471680 0.969 9.05 300 2018-01-05T00:00:00.000+01:00 350 2018-01-06T00:00:00.000+01:00 0 452.5 50
17755137471680 0.969 9.05 350 2018-01-06T00:00:00.000+01:00 410 2018-01-07T00:00:00.000+01:00 0 543 60

RegisterMeasurement object are mapped as:

id v# quantity.value quantity.unit.id flagToZero energyValue.value energyValue.unit.id ptaCoef.value ptaCoef.unit.id conversionCoef.value conversionCoef.unit.id volumeValue.value volumeValue.unit.id parent.id start isEstimated
RMSD_17755137471680_5G#BD6QNB 1 210 cubic_meter 0 90.5 kilowatt_hour 0.969 kwh-per-m3 9.05 kwh-per-m3 10 cubic_meter RMSD_17755137471680_5G 2018-01-03T00:00:00.000+01:00 false
RMSD_17755137471680_5G#BD6QNC 1 230 cubic_meter 0 181 kilowatt_hour 0.969 kwh-per-m3 9.05 kwh-per-m3 20 cubic_meter RMSD_17755137471680_5G 2018-01-04T00:00:00.000+01:00 false
RMSD_17755137471680_5G#BD6QND 1 300 cubic_meter 0 362 kilowatt_hour 0.969 kwh-per-m3 9.05 kwh-per-m3 40 cubic_meter RMSD_17755137471680_5G 2018-01-06T00:00:00.000+01:00 false
RMSD_17755137471680_5G#BD6QNE 1 350 cubic_meter 0 452.50000000000006 kilowatt_hour 0.969 kwh-per-m3 9.05 kwh-per-m3 50 cubic_meter RMSD_17755137471680_5G 2018-01-07T00:00:00.000+01:00 false
RMSD_17755137471680_5G#BD6QNF 1 410 cubic_meter 0 543 kilowatt_hour 0.969 kwh-per-m3 9.05 kwh-per-m3 60 cubic_meter RMSD_17755137471680_5G 2018-01-08T00:00:00.000+01:00 false

Any help would be much appreciated.


#2

The behavior you’re seeing with quantity is the normal behavior of the RegisterMeasurmentSeries, when it finds missing it will simply interpolate, i.e. draw a straight line between the points with values.

In your case a point is missing for 2018-01-05 but you have data for 2018-01-04 and 2018-01-06, so what happens is the value for interval 2018-01-04 to 2018-01-06 (which is 300-230=70) will be divided by the number of intervals (which is 2 DAYs) and thus:

  • 2018-01-04 to 2018-01-05 will have 70/2=35
  • 2018-01-05 to 2018-01-06 will have 70/2=35

This is weird, because those fields should be normalized similarly to quantity. Can you share your series header type definition and how you’re evaluating the metrics.


#3

Knowing that they have the same treatment, what differs quantity from other timeseries?
Let’s accept this behavior for quantity (with any treatment), I iterate over other possible treatments for other timseries, still getting 0 and not an interpolated value. In my case, coefficients should keep the same previous known value.
I tried

  @ts(treatment='previous')
  ptaCoef               :ExactDimension
  @ts(treatment='previous')
  conversionCoef        :ExactDimension

It gave the same exact behavior.
I reprocessed the file, and run normalization on the series.


#4
@canonicalTransform(condition="calendarId != '' && calendarId != 'INCONNU' && registerId != 'INCONNU' && registerId != ''")
type TransformCanonicalMeasurementGazDailyToMeasurementSeries mixes RegisterMeasurementSeries transforms CanonicalMeasurementGazDaily {
    id:               ~ expression "concat('RMSD_', servicePointId,'_5G')"
    servicePoint:   ~ expression {id: "concat(servicePointId,'_5G')"}
    resource :        ~ expression {id: "'naturalgas'"}
    measurementType:  ~ expression "'GasConsumption'"
    treatment :       ~ expression "'integral'"
    interval:         ~ expression "DAY"
    unitConstraint:   ~ expression {id: "'cubic_meter'"}
    measurementCategory: ~ expression "'DailyConsumption'"
}

RegisterMeasurementSeries.tsEval({
        filter: 'id=="RMSD_17755137471680_5G"',
        projection: 'avg(avg(normalized.data.quantity))',  //same goes for other ts...
        start: "2018-01-03T00:00:00.000",
        end: "2018-01-08T00:00:00.000",
        grain: 'DAY',
}))

#5

Well !! all goes good.
I attained rates behaviour for ptaCoef and conversionCoef with previous annotation and quantity with integral annotation. But, changed treatment of TransformCanonicalMeasurementGazDailyToMeasurementSeries
from integral to rate

Result
quantity:
* 0: 20
* 1: 35
* 2: 35
* 3: 50
* 4: 60

ConversionCoef:
* 0: 9.05
* 1: 9.05
* 2: 9.05
* 3: 9.05
* 4: 9.05

#6