canonicalTransform ignoreConversionErrors


#1

We have a transform which uses this annotation: @canonicalTransform(ignoreConversionErrors=true) to allow for uncleaned data to attempt to be assigned to the destination value and just “whatever happens, happens”.

The SourceType.field1 : string is being transformed to EntityType.fieldX : double via this expression fieldX : ~ expression "exists(field1) ? field1: null" , we used to have a more complex expression but had to remove it due to an open issue, so the focus is on the canonicalTransform expectation.

Our understanding is that the typesystem will cast this value to the appropriate destination type, but we get the following error in our data load upload log
Error Msg: For expression: exists(field1 ) ? field1 : null; expecting result type to be double but was class java.lang.String. }

Are there are only certain primitives that this annotation is supposed to work with or something else?


#2

You can simply replace this expression with field1 as the function exists will true if the parameter is not null.


#3

Sigh.

2 platform bugs exist - attempting a transform in the form of exists(field1) && not(matches(field1,'[^0-9$.,]')) ? number(field1) : null fails as it attempts to prebuild the expression with the number function being given a null value. Second is the root cause underlying this post, and again why I stressed focusing on the annotation expectation. The logic you propose or as I have shown is irrelevant if the type system is going to attempt to cast the value to the destination type. The problem is that given any result of the transform that is invalid for the destination domain, exceptions are still being thrown and failing to persist the record. I have 2 zendesk tickets open for this already. So my question remains, as to what the expectation of the developer should be with ignoreConversionErrors=true, and perhaps does it differ based on the transform used (EEF vs JS), the type of destination value (reference, primitive), or even within primitives and auto-[un]boxing for float,int,double,etc.


#4

looking at the documentation of ignoreConversionErrors with c3ShowFunc(Ann.CanonicalTransform, 'ignoreConversionErrors') it seems that it should ignore conversion errors! Agree it’s confusing in your case.

Is fieldX a required field? may be the error you’re seeing is a side effect of this?

@garrynigel any ideas?


#5

@bachr @clowtown this is a bug, and the bug is that ignoreConversionErrors is not used in the current DI pipeline :slight_smile:

Also the feature should not differ based on the transform(JS, EEE, based on primitive type etc) it should behave exactly as expected that is ignore… “whatever happens happens”. Also if there are exceptions to the rules the document should be and will be updated.

However, in production, down stream this could have effects where the Db engine or analytics might have errors and is not recommended to be used. And it makes sense that’s why this annotation has not gained that much popularity.

Ignore conversion errors flag was added and mainly used for development purposes. transformCanonical api was added on Canonical type for developers to test their transforms in static/console. You could use it for quickly testing out the transform.

Currently work is being undertaken to consolidate all this in the Transform type.