FileSourceSystem not mounted by default


#1

Following the s3-data-integration documentation of v7.8 I wanted to create a new FileSourceSystem bound to a dedicated s3 bucket:

  1. creating the FileSourceSystem
    FileSourceSystem.create({ id: "FS1", name: "FS1", typeIdent: "FILE", rootUrlOverride: "s3://my-bucket-1" });

  2. creating the FileSourceCollection
    FileSourceCollection.merge(FileSourceCollection.make({ id: "CanonicaFoo", sourceSystem: { id: 'FS1' }}));

  3. mapping the role/region to the targeted bucket:
    CloudIdentityMapping.upsertBatch([{
    prefix: "my-bucket-1",
    region: "eu-west-3",
    iamRoleArn: CloudIdentity.make().credentials().iamRoleArn,
    id: "my-bucket-1"
    }]);

  4. allowing s3:GetBucketList and s3:GetObject the CloudIdentity.make().credentials().iamRoleArn role in AWS for the my-bucket-1 bucket.

calling the following command will fail:
FileSourceCollection.get('CanonicaFoo').listFiles();

with the error:
Invalid or inaccessible URL s3://my-bucket-1/CanonicalFoo/ for s3

It seems that we are missing a mandatory step of mounting the s3 bucket. The previous command will work if we mount the s3 bucket with the following commands:

var c = FileSystem.fileSystemConfig();
c.s3.mounts.INTEGRATION='my-bucket-1';
c.setConfig(ConfigOverride.TAG);
c.evictFromCache();
// INTEGRATION is an arbitrary key name (to avoid conflicting with DATA_LOAD for instance

Is the mount step needed ? why we don’t automatically do the mount on all the buckets declared in the FileSystemSource instances?