Mount S3 bucket of a different AWS account

Is it possible to mount an S3 bucket that belongs to a different AWS account than C3?? if yes how?

Here are the steps we experimented in order to get the job done :

Let assume you already have an s3 bucket named my-bucket

  • Mount the new bucket in the FileSystem configuration:
var c = FileSystem.fileSystemConfig();
c.s3.mounts.ARBITRARY_NAME='my-bucket';
c.setConfig(ConfigOverride.TAG);
c.evictFromCache();
  • Get the IAM role of the c3 cluster and allow it to access your bucket in your aws console:
// to get the IAM role
CloudIdentity.make().credentials().iamRoleArn

The policy you add in AWS should look like:

{
    "Id": "Policy1538393595123",
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::my-bucket"
            ],
            "Principal": {
                "AWS": [
                    "arn:aws:iam::836860412345:role/the-role-name"
                ]
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::my-bucket/*"
            ],
            "Principal": {
                "AWS": [
                    "arn:aws:iam::836860412345:role/the-role-name"
                ]
            }
        }
    ]
}
  • Create the FileSourceSystem corresponding to this bucket:
FileSourceSystem.create({ id: "MyFSS", name: "MyFSS", typeIdent: "FILE", rootUrlOverride: "s3://my-bucket" });
  • Associate the previous FileSourceSystem to the FileSourceCollection instances that you want to integrate through your bucket:
FileSourceCollection.merge({ id: "CanonicalFoo", sourceSystem: { id: 'MyFSS' }));
FileSourceCollection.merge({ id: "CanonicalBar", sourceSystem: { id: 'MyFSS' }));

  • Check the inboxUrl() of your FileSourceCollection is correctly mounted on your bucket :
FileSourceCollection.get("CanonicalFoo").inboxUrl();
// should look like: s3://my-bucket/CanonicalFoo/inbox/
  • Make sure that your FileSourceCollection is listing the files:
FileSourceCollection.get("CanonicalFoo").listFiles();
2 Likes

@bachr @NabilKoroghli Please do not encourage people to update and override configs for FileSystem. by changing the config object itself.
We have provided methods in 7.9 to do the same.
Kindly read the document Setting up FileSystem for a User in c3docs.
especially this section

2. Change the default FileSystem on the environment,

FileSystem.azure().makeDefault();

3. Set a mount on the FileSystem.

If the user desires a different location, they can specify the alternate mount or path by setting:

FileSystem.azure().setMount(FileSystemMount.DATA_LOAD, "c3-local-env/dl/test/prod");

If you have specific needs for updating configs let us know we can add additional methods.

3 Likes