Build custom normalizer

#1

It’s fairly straightforward to create Normalizers in case the default one does not fit your use cases, for instance when you have some complicated requirements to meet. In the following example creates a new normalizer for PointPhysicalMeasurementSeries.

First, create a type that mixes the Normalizer type:

type PointMeasurementNormalizer  mixes Normalizer {

    normalize: function(objs: stream<TSDataPoint>, spec: TSNormalizationSpec): [Timeseries] js server
}

Then, implement the logic for normalization in PointMeasurementNormalizer.js. The idea is to implement the Normalization all or few of the steps in javascript (check the In Depth documentation on Normalization) while using as much information available through input TSNormalizationSpec. For instance,

function normalize(objs, spec) {
  // Check if you have all needed information in spec
  if(someValidationFunction(spec)==false) throw new Error("Bla Bla");
  // Clean the objects
  var prev = null;
  var cleanedObjs = objs.map(function(obj){
    // 1. Initial Cleaning
    if(isBadData(obj)) return null;                            // ignore bad data
    obj = obj.putField("start", obj.start.withoutZone()); // remove TimeZone
    // 2. Unit Conversion: 
    obj = obj.putField("value", obj.value * conversionFactor);
    obj = obj.putField("unit", normUnit);
    // 3. De Duping
    if(isEqual(obj, prev)) return null;                       // compare if previous object is similar to current one
    // 4. De Overlapping
    if(isEqual(obj.start, prev.start)) { /* e.g. keep the point with higher dataVersion and reject the other one */ }
    return obj;
  }).filter(function(obj) {
    return obj != null;
  });
  // 5. Interval Detection:
  var normInterval = header != null ? header.interval || header.grain || spec.normInterval : GrainDetector.detectGrain(cleanedObjs);
  // 6. Interpolation:
  ///  - loop over data points, look for gaps i.e. 0s between valid points
  ///  - use the MeasurementSeries's resetOper (i.e. spec.header.resetOper) to decide which point to use as base value for filling the gaps.

  // Return the resulting timeseries.
  var missing = [];
  var ts = createTs(..., missing);
  return [ts];
}
function createTs(. . .) {
  return Timeseries.makeNorm(NormTimeseriesDoubleSpec.make({
    start : tsStart,
    end : tsEnd,
    interval : normInterval,
    data : newData,
    unit : unit,
    missing : NormTimeseriesDouble.range2norm(tsStart, tsEnd, normInterval, missing)
  }));
}

Lastly, attach the newly created normalizer to the

remix type PointMeasurement {

    @ts(normalizer='PointMeasurementNormalizer')
    quantity: !~
}

To test the implementation, create some data points, attach them to a series then call normalize on it.

var s = PointPhysicalMeasurementSeries.upsert({
  id: "S0001",
  unitConstraint: {
    id: "degrees_celsius"
  },
  indexMax: 50,
  grain: "DAY"
})
var p1 = PointMeasurement.upsert({
  parent: s,
  id: "S0001#0001",
  start: "2018-05-01",
  quantity: {
    value: 1.2,
  }
})
. . .
var p10 = PointMeasurement.upsert({
  parent: s,
  id: "S0001#0010",
  start: "2018-05-31",
  quantity: {
    value: 35.3,
  }
})
var ts = PointPhysicalMeasurementSeries.normalize(pms,
  DateTime("2018-05-01"),
  DateTime("2018-05-31"),
  "DAY",
  false
)
c3Viz(ts)
4 Likes