Move Measurements from one MS to another

#1

Hi, I have to move some measurements from some MeasurementSeries to others, the amount of data is:

  • About 250 MS to update
  • Each MS have about 250000 measurements associated

I tryied different ways to do this action, but all take a lot of time and I’m not able to close this activity in reasonable time. I tried to use (in a javascript script launched as snipped):

  • “createBatch” after doing “make” of all the measurements
  • “mergeBatch” after doing “make” of all the measurements
  • “create” called inside a loop over the measurements

How can I do this action in a reasonable time of execution?

Thanks,
Laura

0 Likes

#2

I think you can achieve this with a map reduce like the one below:
Before to run this you have to update in the MeasurementSeries the attribute “name” with the new ID of the MeasurementSeries.
So you have to update for all the 250 MS the attribute name with the new id and then run the below command.
you can test with one ID as in the example below if you want

function mapper(batch, objs, job) {

objs.each(function (obj) {

 var a = []; 

  var pmStm = PointMeasurement.fetchObjStream({filter: Filter.eq('parent.id', obj.id), limit:-1}) 

                          .map(function(measurement) { 

  return measurement.removeField("id").putField("parent", PointPhysicalMeasurementSeries.make({id: obj.name})); 

  }); 

  var pmBatch = []; 

  while (pmStm.hasNext()) { 

    pmBatch.push(pmStm.next()); 

    if (pmBatch.length == 5000) { // for my testing I used batch size = 10 

      PointMeasurement.createBatch(pmBatch); 

      pmBatch = []; 

    } 

  } 

  if (pmBatch.length > 0) { 

    PointMeasurement.createBatch(pmBatch); 

  } 

 }); 

}

mr = JS.mapReduce({

targetType: PointPhysicalMeasurementSeries,

filter: Filter.intersects(‘id’, [‘AS_TELENET-POWER_38790D_AP_P’]), //this is the old id, remember to change this

include: ‘id,name’,

batchSize: 1,

map: mapper

});

2 Likes

#3

It’s actually a bad idea to do fetch inside the map, it would be better to directly have the MapReduce run on the Measurement.

0 Likes

#4

in order to run on the measurement in this case you need somehow to pass the parent and the new parent id

0 Likes

#5

Here’s a solution with a map job on PointMeasuremnt (based on @marcosordi original solution)

Get the current series you want to move their data

var ppms = PointPhysicalMeasurementSeries.fetch({include: 'name', filter: 'your fitler to select the series to update', limit: -1}).at('objs');

Define the logic that moves PointMeasurement

// map function
function mapper(batch, objs, job) {
  var ctx = job.context.value;
  var newParent = ctx.get('parentId').value;
  var newObjs = objs.map(function(obj) {
    // it's better to create new objects then update old one
    return PointMeasurement.make({
      parent: {id: newParent},
      quantity: {unit: obj.quantity.unit, value: obj.quantity.value}
      // add other fields you care about
    });
  });
  PointMeasurement.createBatch(newObjs, {createDirect: true});
}
var mapperStr = mapper.toString();

Now you can trigger the mapreduce jobs with:

// for each series launch a map-reduce job
var jobs = ppms.map(function(series) {
  var oldParentId = series.id;
  var newParentId = series.name; // or any holder of the new parent id
  return JS.mapReduce({
    targetType: PointMeasurement,
    filter: Filter.eq('parent.id', oldParentId),
    include: 'quantity,start', // include things you need
    batchSize: 1000,
    context: { // use the context object to pass anything to your job
      parentId: newParentId
    }
    map: mapperStr
  });
})

Finally to look to the status of your jobs you can do:

var ids = _.pluck(jobs, 'id');
c3Grid(JSMapReduceJob.fetch({filter: Filter.intersects('id', ids)}))
2 Likes

How to pass parameters to a map() function of a JSMapReduce job
#6

I am not sure that there is any benefit in creating a separate MR job per series. We usually follow the pattern @marcosordi described, where a header type is used to seed a single MR job and the data points are only accessed within map().

0 Likes