Generate unique id in Canonical Transform


Is there a function (expression engine function) that generates a unique id string that can be used in the Canonical transform expressions?



If your transform is in Javascript you can try with UUID.createUUID() other use use md5 function e.g.

id:      ~ expression 'md5(concat(field1,field2,...))'


I want to avoid using javascript transform and leverage the existing transform expressions.
md5 function may not ensure unicity if I don’t have a changing fields.



Best practice is to construct the ID from the inbound fields (via concatenation or hashing). If you generate a random ID, then this transform is prone to creating duplicates if the file is ever reprocessed.

1 Like


If every single field is identical, what is the use case of creating multiple identical records in the system?

1 Like


Well, in fact, md5 would fit for my use case. I was initially focusing on looking for an equivalent expression engine function to UUID.createUUID() but you are right it is may not be useful in canonical transform use cases.



We have a situation in which two source systems are feeding to one canonical. One system provides columns that can be used to make a unique key. The other does not (it can provide two identical records, representing unique transactions, they key fields are simply not provided to us).

The following appears to work in the transform to conditionally make an id when able and generate a random one otherwise:

id: ~ expression "KEY_PART1 == undefined ? undefined : concat(KEY_PART1,KEY_PART2)"

In our case, the one file does not even contain the KEY_PART1 column, so we make id undefined which the type system then generates when it creates the record.