Inputs for integrating measurements data

#1

We have measurements that comes at a day frequency, they are register kind of measurements. We are using PointPhysicalMeasurementSeries.
When integrating a new measurement we should respect the following requirements:

  1. Measurements should be integrated only if Value (day) < Value (day - 1) or Value (day - 1) hit the register max value and
    -idea: I guess I have to implement the register normalizer on top of the default normalizer.

  2. Measurement should not be integrated if value is very high compared to the previous one (daily max growth = 1/10 of the index max value)
    -no idea: how can I guarantee this?

  3. Measurement should not be integrated if duplicate, i.e same timestamp, same value, same meter.
    -idea: this is handled in normalization, right?

  4. Measurement should not be integrated if inconsistent, i.e. same timestamp, same meter but different values
    -no idea: note that the next measurement can came later and should made the previous one invalid!

  5. Measurement should not be integrated if timestamp is in the future, i.e. measurement timestamp > now()
    -idea: This can be handled in the transformation

  6. Measurement should not be integrated if the value is greater than the max value set in the meter.
    -idea: This is handled by RegisterMeasurementSeries, but we cannot use this as dataVersion and we need this field.

Any additional inputs on how I can address these requirements?

0 Likes

Build custom normalizer
#2

If by “integration” you mean “Data Integration” i.e. Sources, “Canonicals” and Transforms then you should “Integrate Data” as closely to source as possible.

What you are describing here is best if implemented as metrics.

@rohit.sureka thoughts?

1 Like

#3

@bachr

To echo @DavidT’s comment, yes it should be done via “Sources”, “Transforms”, “Canonicals” and handling of data should be a mix of normalization layer (for duplicates, values < previous value, etc) + metric combination.

If you are alluding to NOT storing the data itself, I’m not sure if that is a great idea, as we should try to map the source data as close to what the customer gives us and then any transforms, manipulation could/should be done in the platform.

1 Like

#4

If he is alluding to not storing the data itself, i am sure it is not a good idea

0 Likes

#5

thanks All I understand that the integration should be via canonicals/transforms. Sorry, if my initial question was not clear, I was more focused on describing the requirements, we have to store the measurements.

0 Likes

#6

you have to store all the measurements even the ones that don’t meet the requirements above

The requirements should all be implemented post data load e.g. normalization is a good strategy, so are metrics

1 Like

#7

This is the second iteration around normalization in a current project. We already have a normalization based on PointMeasurement and registerReadInterpolator that seems to work (we need to make sure that tests run well isolated). Now the customer is adding requirements based on a legacy code and we shall try to use after-normalization approaches to address them if possible. That is why I asked for the list of functions used in the standard normalization, we need to provide a solution next week.

0 Likes

#8

@AlexBakic I am not clear when you say list of functions used in normalization. May be you can elaborate? If you look at the In Depth documentation, it should describe in detail all the steps that take place during normalization. That is exactly how the data points get treated.

0 Likes

#9

I thought there would be, let’s say, seven “intermediate types” of measurements during a seven-phase normalization, to which there could correspond seven functions producing series of intermediate type Tn (say, without duplicates) from series of intermediate type Tn-1 (say, with units converted). Of course, for performance reasons an implementation may not look like that, but just at a high level.

With one function per phase, one could make a driver (chain) that does the same as the standard customization, and then just replace some functions with custom ones.

0 Likes

#10

@AlexBakic There aren’t type system functions available for all those today except for:

GrainDetector.detectGrain
GrainSplitter.align
Unit.getConversionFactor

0 Likes