PointMeasurement start is modified by system if not aligned to start of interval


#1

On 7.6, I’ve a strange behavior on the start of PointMeasurement. Basically when create a batch of points with a start that’s not aligned with start of the interval then the system change the inserted value.

First create a series:

PointPhysicalMeasurementSeries.create({id: 'test', unitConstraint: {id: 'degrees_celsius'}, treatment: 'RATE', interval: 'QUARTER_HOUR'})

Example 1: start of measurement not aligned to start of the interval

PointMeasurement.createBatch([
 {parent: {id: 'test'}, quantity: {value: 26}, start: '2018-09-01T05:00:00.042+02:00'},
 {parent: {id: 'test'}, quantity: {value: 26}, start: '2018-09-01T05:15:00.052+02:00'},
 {parent: {id: 'test'}, quantity: {value: 26}, start: '2018-09-01T05:30:00.047+02:00'}
])

Then looking at the inserted data I see (check the start of the third point how it’s altered):

"id","lastModification","lastEditor","parent.id","parent.bucketInterval","start","quantity.value"
"test#GX","Manual Input","BA","test","MONTH","2018-09-01T05:00:00.042+02:00","26"
"test#GY","Manual Input","BA","test","MONTH","2018-09-01T05:15:00.042+02:00","26"
"test#GZ","Manual Input","BA","test","MONTH","2018-09-01T05:29:59.042+02:00","26"

Example 2: now if I create measurements with timestamp aligned to start of the 15mn interval (i.e. 0 for seconds)

PointMeasurement.createBatch([
 {parent: {id: 'test'}, quantity: {value: 26}, start: '2018-09-01T05:00:00.000+02:00'},
 {parent: {id: 'test'}, quantity: {value: 26}, start: '2018-09-01T05:15:00.000+02:00'},
 {parent: {id: 'test'}, quantity: {value: 26}, start: '2018-09-01T05:30:00.000+02:00'}
])

Then looking at the inserted data I see (I see that all points have same timestamp as the one I provided):

"id","lastModification","lastEditor","parent.id","parent.bucketInterval","start","quantity.value"
"test#G2","Manual Input","BA","test","MONTH","2018-09-01T05:00:00.000+02:00","26"
"test#G3","Manual Input","BA","test","MONTH","2018-09-01T05:15:00.000+02:00","26"
"test#G4","Manual Input","BA","test","MONTH","2018-09-01T05:30:00.000+02:00","26"

Is this a normal behavior? does it mean that in my transform I have to make sure that my points have a start that is aligned to the interval of the series (here 15mn)?


#2

@bachr this is definitely not expected behavior. We should be preserving raw data as is. Can you try the same experiment in v7.8 server? There were a few changes in how we store millisecond values for types that go in cassandra.

If it continues to happen in the latest version of the code then it’s a definite bug and I suggest you file a ticket


#3

@rohit.sureka when testing on 7.8.1.4767-1,
The result for Example 1 the millis are reset to 0 (KO)

"id","parent.id","start","quantity.value"
"test#B","test","2018-09-01T05:00:00.000+02:00","26"
"test#C","test","2018-09-01T05:15:00.000+02:00","26"
"test#D","test","2018-09-01T05:30:00.000+02:00","26"

The result for Example 2 is the same as what was inserted (OK)

"id","parent.id","start","quantity.value"
"test#E","test","2018-09-01T05:00:00.000+02:00","26"
"test#F","test","2018-09-01T05:15:00.000+02:00","26"
"test#G","test","2018-09-01T05:30:00.000+02:00","26"

#4

To support millisecond resolution you will need to modify the data type declaration on the start field and add the with millis keywords. Here is an example that is supported in Platform 7.8.

/**
 * Enhance PointMeasurement to support millisecond resolution
 **/
remix type PointMeasurement {  
  start: datetime with millis  
}

#5

Awesome thanks a lot @scottk just made a test and it’s working :slight_smile:


#6

Is there a rationale for lossy storing of datetime?
Are there other types stored by dropping some info?


#7

datetime by default is not expected to store millis in C3 which is why there is a special datetime with millis value type


#8

I see, but the default seems wrong if someone wants to persist data, because they start with millis but lose them. Also, I noticed timestamps of type datetime that are stored yet without losing millis (either something is different in the server or I am mistaken).


#9

If there is a bug or inconsistent behavior in v7.8* please raise a ticket with clear steps to reproduce