Persistence of duplicate value objects in TimedValueHistory type


#1

I got this from @pavan.nandikonda’s reply to an earlier community post

The TimedValueHistory type is designed to maintain the latest value of the status on the account while also maintaining the history of changes to the status over time. However, depending on how you set up your canonical transform, a type with a [TimedValueHistory] field will behave differently.

There are 2 different ways to set up the transform - for the sake of explanation let’s call these

  1. Inline transform
  2. Dedicated transform type

If you set things up using an inline transform (1) on data load the platform will automatically check the value of the new TimedValue with the most recent and only create a new entry if those values are different. If you set it up using a dedicated transform type (2), there is no value check between the new TimedValue and most recent value, so the TimedValueHistory field could end up having duplicate values between entries

As an example, let’s say you have the following types

type MyTimedValue mixes TimedValue, ExactDimension


entity type MyTimedValueHistory mixes TimedValueHistory<ParentType>, ExactDimension


entity type ParentType {
   @db(timedValueHistoryField='myTimedValueHistory')
   myTimedValue: MyTimedValue
   
   @db(order='descending(toUTC(timestamp))')
   myTimedValueHistory: [MyTimedValueHistory] (parent)
}

and the following CanonicalMyParentType file:

id,timedValue,timestamp,unit
"ABC",5,"2018-09-01","skittles"
"ABC",5,"2018-09-02","skittles"

Using Method 1 (inline transform):

TransformCanonicalMyParentTypeToParentType {
  id: ~ expression 'id'
  myTimedValue: ~ expression '{value: timedValue, timestamp: timestamp, unit: { id: unit }}'
}

You would end up with a single MyTimedValueHistory entry after data load, with a value of:

{
  value: 5,
  timestamp: "2018-09-01",
  unit: {id: "skittles"}
}

Using method 2 (dedicated transform type):

TransformCanonicalMyParentTypeToParentType {
  id: ~ expression 'id'
}

TransformCanonicalMyParentTypeToMyTimedValueHistory {
  parent:    ~ expression '{ id: id }'
  value:     ~ expression 'timedValue'
  timestamp: ~ expression 'timestamp'
  unit:      ~ expression '{ id: unit }'
}

You would end up with two MyTimedValueHistory entries after data load:

[{
  value: 5,
  timestamp: "2018-09-01",
  unit: {id: "skittles"}
},
{
  value: 5,
  timestamp: "2018-09-02",
  unit: {id: "skittles"}
}]

#2

Note: this is currently doing the opposite for TimedCharacteristicHistory:
Deduplication happens in method 2 not in method 1.

See c3ShowType(Ann.Db), timedCharacteristicHistoryField


#3

Correct, as of 7.8.1 method 2 also deduplicates.


#4

@yaroslav just tested it out and you are right, method 2 also now deduplicates. They still appear to show slightly different behavior though - when running the example above method 1 ended up with a TimedValueHistory result of:

[{
  "parent": {
    "id": "ABC"
  },
  "timestamp": "2018-09-01T00:00:00.000",
  "value": 5
}]

while method 2 ended up with:

[{
  "parent": {
    "id": "ABC"
  },
  "timestamp": "2018-09-02T00:00:00.000",
  "value": 5
}]

So it appears method 2 will do deduplication but update the timestamp of the TimedValueHistory object, while method 1 will ignore the second object entirely


#5

This doesn’t sound like correct behavior - timestamp should not change. @trothwein - thoughts? Are we doing something wrong here?


#6

@scott.kruyswyk I’m a little confused. Is this for a TimedCharacteristic or a plain TimedValue. If it’s the latter, then I don’t see how method 2 would have done any deduping as that was only implemented for TimedCharacteristics.


#7

@trothwein I’m using a TimedValue that also mixes Double I get the same result with method 2 whether I call

CanonicalParentType.process([
  {
    "id": "ABC",
    "value": 5,
    "timestamp": "2018-09-01"
  },
  {
    "id": "ABC",
    "value": 5,
    "timestamp": "2018-09-02"
  }
])

or

CanonicalParentType.process([
  {
    "id": "ABC",
    "value": 5,
    "timestamp": "2018-09-01"
  }
])

CanonicalParentType.process([
  {
    "id": "ABC",
    "value": 5,
    "timestamp": "2018-09-02"
  }
])

I am using c3server 7.8.1.5699 - I can provide a package with a small repro if you’d like, I’m not allowed to attach zip files on community :slight_smile:


#8

@scott.kruyswyk Can you create a branch and a jira ticket with repro instructions?


#9

@trothwein nevermind - I noticed it was because I had my CanonicalParentType set up with its identifier field as id - since I was not doing any explicit override on the id field when transforming to MyTimedValueHistory, it was automatically setting the id of the TimedValueHistory type to id and thus only one entry was being created/updated.

Once I renamed the field on my CanonicalParentType then things worked as expected :slight_smile:. @yaroslav just an fyi that method 2 still does not deduplicate when using TimedValue/TimedValueHistory