Where does the `Canonical.process()` function comes from?


#1

I came across a test code that loads data from canonical and applies the transformation to generate the target instances. Something like:

var canonicalMT = CanonicalMyType.make({id: 'AAA', ...});
CanonicalMyType.process([canonicalMT]);
var mt  = MyType.fetch({filter: Filter.eq("id", "AAA")}).objs.first();
. . .

I tried to find the declaration/signature of process() from the console with something like c3ShowType(CanonicalMyType) or the parent type c3ShowType(Canonical), but did not find anything!

Where does the process() function in Canonical comes from?


#2

The declaration of process is hidden because process is a private function and should not be used to load data, the only interface to load and process files should be through the SourceFile type and the apis on it.


#3

What are the possible values for the source: TypeRef used when creating an instance of SourceFile?

Can I used a collection of lines for instance?


#4

Testing for data integration should primarily be using canonical test runner. If code in a test exists like this, it should be updated to use canonical test runner.

SourceFile entries are not to be manually created, the methods that create the metadata for source files are syncAll syncBatch and sync.
However this is a new feature introduced in 7.7 and the data integrator c3doc has to be updated on the way to introduce this new interface.

And to answer your question source field refers to the Source/Canonical type for which the file is loaded for.


#5

Thanks Garry for the clarification.

I’m not aware of the Canonical Test Runner! Do you have a sample on how to use it?


#6

The documentation on CanonicalTestRunner is well updated.
c3ShowType(CanonicalTestRunner).
There are many examples in base repository
e.g. crm/test/canonical thats the way you structure the test folder and then run the tests.
as

  var testPackage = MetadataStore.current().package("myTenant", true);
  var testPath = MetadataPath.make({repository: testPackage.repository, package: testPackage.package, category: MetadataFileCategory.OTHER, encodedSubPath: "test/canonical/folder1"});
  var res = CanonicalTestRunner.testPath(testPath);

#7

What are the impacts of testing transformations using Canonical.process() and Canonical.transformCanonical()?

My organization tests transformation logic using those methods because we can’t use CanonicalTestRunner.


#8

CanonicalTestRunner tests end to end processing for files, with validations, it basically is as close to a production load for a file, with both incremental and initial processing. CanonicalTestRunner is self contained and tag specific. if data is loaded into a facaded Persistable type, then you might have complications loading data, but otherwise CanonicalTestRunner should be figured out for an organization.

Having said that, transformCanonical only transforms the data and is going to be deprecated, Also FYI this was a toybox function created in the past and production does not use the same code path.

Canonical.process() is a private function, hence no guarantees on maintenance of that code, which follows another code path that is not equivalent to the production one.

However in 7.8.1, we are introducing Source.process() which will be exactly as that in production, so you could theoretically use it. But I would highly recommend CanonicalTestRunner.


#9

Thank you for the explanation.

if data is loaded into a facaded Persistable type, then you might have complications loading data, but otherwise CanonicalTestRunner should be figured out for an organization.

All of our Persistable types are facaded :stuck_out_tongue: