Missing in eval metric result

#1

about the “treatment” parameter of a Point Physical Measurement Series.
If it is “rate”, a lack of raw data in the normalization grain will lead to 100 % missing in this time slot.
If it is “previous”, what will happen with a lack of raw data in the normalization grain, when the metrics is evaluate. Will the value of the previous time slot be taken as the value of current time slot (missing value), leading to à zero % missing for it ?

0 Likes

#2

I have a similar question about Measurements (not PointPhysicalMeasurements) in a PhysicalMeasurementSeries with “previous” treatment.

If I run an evalMetrics on a SimpleMetric defined on this series (just returning normalized.data), all intervals before the last Measurement in DB have missing values properly populated.
However, each interval after the last Measurement is marked as 100% available (0% missing), even though no data is present.

As an example, if I have a single Measurement for 2017-12-29 and if I run an evalMetric from 2017-01-01 to 2019-01-01 with interval DAY, all days before 2017-12-29 will have 100% “missing”, and all days after 2017-12-29 will have 0% “missing”.
I am looking to change this behavior so that all days after 2017-12-29 have 100% “missing”. What do you suggest?

Thanks

0 Likes

#3

The treatment PREVIOUS means that after the first available data point, all data points have the value of the previous point, where value also includes “missing”.

Do you want to project the data value forward in time but not the missing value? so for example measurements like:

x, x, x, 10, x, 2, 4, x, x

would result in data like

x, x, x, 10, 10, 2, 4, 4, 4

but missing values like:

100, 100, 100, 0, 100, 0, 0, 100, 100

?

0 Likes

#4

Yes Riley, that’s exactly what I am looking for.
(I could live without the data projection forward but the 3 available treatments are “previous”, “rate” & “integral” (not “latest”))

0 Likes

#5

I’m still looking for a solution here. My PhysicalMeasurementSeries are defined with treatment “previous” but I could live without the forward projection, as long as missing behaves as you described it @rileysiebel. Thanks for any hint!

0 Likes

#6

@Ierela,

You are right you should expect a 100% missing values after the last data point as well in case of missing. The value field is populated with the “previous” value but the missing field should still say 100% missing. This seems like a bug to me.

There is a mode in C3 where the data after the last data point is marked with 0% missing when you set the “extrapolate” flag on the series header (e.g. PhysicalMeasurementSeries) which indicates extrapolate the data that you have seen last after the last data point in which case we don’t consider the values as missing. However, if that flag is not set, then we should be setting the missing value correctly after the last data point. It seems like when the treatment is previous for “interval” series (which has both start & end - a.k.a Measurement) - specifying a treatment previous automatically makes it extrapolatable based on the current system, but it seems unnatural and should be fixed.

Could add a ticket for the other behavior?

0 Likes

#7

PhysicalMeasurementSeries extends TimeseriesHeader which does not have an extrapolate flag (however PointPhysicalMeasurementSeries does, since it extends TimedDataHeader).
Therefore, do you have a fix in mind that wouldn’t require a platform patch?

I have opened a ticket. Thanks!

0 Likes

#8

@Ierela

We will prioritize and fix this in the platform. In the meanwhile, you can create a Metric function library function to manipulate the time series according to your desired result, using the example here: Metric referencing actionDecl-based metric returns empty timeseries

0 Likes