How to ignore some TimedDataPoint in normalization?

#1

Some data we received from the customer are bad and even though we want to keep those data, we want to ignore those data in the normalized time series. I can identify those bad data, how do I ignore those data in normalization?

0 Likes

#2

I am not sure in general, but I know that function normalize takes a stream of TSDataPoints (and the latter shares TimedDataPointFields with TimedDataPoint), with fields isEstimated and dataVersion. You might try and see if some values of these fields can make normalizer ignore corresponding data points.
Alternatively, see recent post on how to build a custom normalizer if you think it is worth it (not too difficult, but I am yet to get an idea of performance).

0 Likes

#3

Your options are

  1. If the corrected points have the same start/end as the bad datapoint, then make sure to have the bad data point to have a lower dataVersion than the right one.
  2. Handle that in the TSDecl metric (on raw points)
  3. Move the bad data to a different type all together.
2 Likes

How to filter data in a metric declaration?