Disable normalization on some PhysicalMeasurementSeries

#1

Hello,

I wish to disable the normalization for some specific PhysicalMeasurementSeries (I do not need to access normalized data, just the raw measurements). The only way I’ve found to do this so far is to use the Ext.Ts annotation but this requires to extend the measurement type and to change the PhysicalMeasurementSeries to use it, which is way too verbose.

I was hoping to just set a parameter of the PhysicalMeasurementSeries (for instance I see treatment or overlapHandling that configure the behavior of the series). Is there any way to do this?

Thanks

0 Likes

#2

We ran into this a few times, and the quick solution was to remove the series object itself (or re-upsert it with different ID). Measurements can still be fetched using {filter:"parent.id=='foo'"} even if foo does not exist.

@rohit.sureka we can consider adding a flag on TimeseriesHeader, something like “disableNormalization”?

0 Likes

#3

@yaroslav you are right, currently in order to do that you can remove the PhysicalMeasurementSeries object and no normalization will occur.

We could potentially add the flag to disable normalization but what is the motivation to keep the data around as time series but just use it just for fetch, why not categorize and model this data into a completely different type?

0 Likes

#4

I’ve considered that as well, of course, but what makes this workaround difficult is that we can’t use a MapReduce to clear all or some measurements if we don’t have a parent series to refer to.

We could use a different type as @rohit.sureka suggests it but then it requires a separate MapReduce job to clean the non-normalized series, not very DRY…

0 Likes

#5

@lerela yes a separate map reduce job to clean that kind of data. But as you mentioned it is truly data that is requiring special handling as you mentioned that this kind of data does not need to be normalized, right?

Also separate map reduce job is literally 3-4 lines of JsBatchJob code we are talking about right? I can send you a sample of that if you’d like.

One more thing that you can do is, since by default the normalization is on demand, the system will never Normalize the series unless accessed via evalMetrics. If this is truly data that is just for exploration and never be accessed it will never be normalized.

Also when running refresh normalization, you can add an extra flag on these series and make sure you exclude that from the filter during refreshNormalization.

Let me know if this makes sense / have any questions

0 Likes

#6

Thanks for those ideas Rohit. Never fetching the series is an interesting option.
I’m not familiar with JsBatchJob so a sample would be great, but indeed it’s not too complicated I was just looking for a clean way to do this.

0 Likes

#7

Here’s an example of writing a map reduce job without creating any types (a.k.a do it on the fly):

spec = JSMapReduceSpec.make({
 targetType : {typeName:"PhysicalMeasurementSeries"},
 include: "id",
 filter:"exists(id)",
 limit: -1, batchSize: 100,
 map : "function map(batch, objs, job) {if(objs && objs.size()>0) {var ids = objs.pluck('id'); for (var i = 0; i < ids.length; i++) {Measurement.fetch({filter:\"parent.id == '\" + ids[i] + \"'\", limit:-1});}}}"
});
JS.mapReduce(spec);

You can also write your function independently and just call toString for the map field

e.g.

var myMapFunc = function map(batch,objs,job) {
  your code
}

and then:

map: myMapFunc.toString()
2 Likes