Missing data in evalMetrics - doubts


#1

Hi, I have some doubts about the values returned as “missing array” in metric result.
I would understand what are the elements that are valuated in this percentage: if I set in RegisterMeasurementSeries / PointPhysicalMeasurementSeries / PhysicalMeasurementSeries the field “interval”, the evaluation of the missing in evalMetrics take care of this value? It seems no.

Case A
I have a RegisterMeasurementSeries with fields --> interval = HOUR ; treatment = INTEGRAL

If I have for a specific measurementSeries (Register) one single raw data per day and I launch evalMetrics with interval = HOUR , why the evalMetric returns missing array equals to “0”? And not always, because I have different percentages (as 2 or 97 or others) for different hours, even if I have always one raw data per day.

Case B
I have a PointPhysicalMeasurementSeries with fields --> interval = QUARTER_HOUR ; treatment = RATE

The logic is to mantain the same value arrived in the system until a new data arrives. Ok, but if I launch evalMetrics with interval = QUARTER_HOUR I see missing array always 0, even from one raw data to another (I mean, even for hour populated with previous value).

Case C
I have a PhysicalMeasurementSeries with fields --> interval = QUARTER_HOUR ; treatment = integral

In this case if I have a measurementSeries that have raw data every hour and I have some data missing in specific hour, I see the correct percentage of missing.


#2

Hi, any suggestions ?


#3

Please find my answers inline:

Case A:
The missing percentage is calculated as a percentage of the data available within the giving interval. Depending on the start date of the data point the percentages may be different per interval during the hours of that day. If you have a specific example it may be easier to reason through it.

Case B:
This is by definition, it means the previous point holds true until the next point is seen. If you do want the average but do not want the point to repeat, then use AVG as the treatment and you will see the right reporting for missing.

Case C:
Yes that is expected.

p.s. Small caveat, for high frequency data i.e. interval finer or equal to MINUTE we don’t record all the missing values in interest of storage