How to define a connection to an external DynamoDB datastore?

#1

The Data Virtualization topic in the documentation lists an available connector for DynamoDB, but SqlSourceSystem doesn’t support it, so how can we define an external type that persists to Dynamo?

1 Like

#2

@rohit.sureka Is there any support for external types on DynamoDB? There is no SqlSystem/SqlSourceCollection as is mentioned as I don’t believe there is jdbc support for it (is there?).

0 Likes

#3

A connector for DynamoDB does exist. It is a “low-level” connector, providing basic connectivity and API support. In 7.8/7.9, you will need to implement fetch and create/upsert APIs.

Here is a basic (hard-coded) example with plenty of opportunity for improvement. NOTE: password handling is omitted from this example. I leave it to you to implement create->put APIs.

DynamoDBHelper.c3typ - help type to facilitate integration with DynamoDB.

type DynamoDBHelper<T> {
  
  /**
   * Fetches data from DynamoDB and creates a FetchResult from the response
   * @param spec - C3 Fetch specification
   * @returns FetchResults
   */
  fetch: function(spec : FetchSpec) : FetchResult<T> js server
  
}

DynamoDBHelper.js - helper implementation. Lots of room for improvement.

var log = C3.logger("DynamoDBHelper");

/**
 * Fetches data from DynamoDB and creates a FetchResult from the response
 * @param spec - Fetch specification
 * @returns FetchResults
 */
function fetch(spec) {
  
  var typeName = c3CurrentAction().getTarget().getType();
  var tag = MetadataStore.tag(); 
  var typ = tag.readType(typeName); 
  var tableName = typ.extensions().schema.name;
  var partitionKeyField = typ.extensions().db.partitionKeyField;
  var rows = [];
  var scanSpec = null;
  
  if (spec.filter != undefined) {
	  scanSpec = constructDynamoDBScanSpec(spec);
  }
  
  var scanResults = DynamoDB.scan(tableName, scanSpec);
  
  if (scanResults.count > 0) {
	  rows = createC3TypeFromDynamoResults(typeName, scanResults.items);
  }
  
  return {
    objs : rows,
    count: rows.length,
    hasMore : false
  }
}

/**
 * Converts DynamoDB rows to a C3 Type
 * @param typeName - Name of the type to instantiate
 * @param dynamoItems - list of records from DynamoDB
 * @returns [] - array of records to be included in the FetchResult
 */
function createC3TypeFromDynamoResults(typeName, dynamoItems) {
	var rows = [];
	
	// long term, get list of fields from Type System
        // enhance to iterate through fieldTypes
	var fields = ["Id", "Brand", "Description", "Price", "ProductCategory", "Title", "BicycleType"];
	
	dynamoItems.forEach(function (item) {
		var o = c3Type(typeName).make();
		
		for (var j = 0; j < fields.length; j++) {
			
			if (item[fields[j]] != undefined ) {
				o = o.putField(fields[j], item[fields[j]]);
			}
		}
		
		rows.push(o);
		
	})
	
	return rows;
		
}

/**
 * Constructs a DynamoDBScanSpec from a FetchSpec.  For the moment it supports the most basic of queries. 
 * @param fetchSpec
 * @returns DynamoDBScanSpec
 */
function constructDynamoDBScanSpec(fetchSpec) {
	
	var vars = fetchSpec.filter.split("==");
	var filterExpression = vars[0].trim() + " = :val";
	var expressionAttributeValues = { ":val": vars[1].replace(/'/g, "") };
	var spec = DynamoDBScanSpec.make({filterExpression: filterExpression, expressionAttributeValues: expressionAttributeValues });

	return spec;
	
}

ProductCatalog.c3typ - Type that models the ProductCatalog DynamoDB table.

@db(partitionKeyField="id")
type ProductCatalog mixes DynamoDBHelper<ProductCatalog> schema name "ProductCatalog" {

  Id: int
  Brand: string
  Description: string
  Price: double
  ProductCategory: string
  Title: string
  BicycleType: string

}
0 Likes

#4

@scottk
What would be the parameters/fields inside DynamoDBSourceSystem type? It would be great if you could provide the definition of source system similar to cassandra (How to connect an external cassandra data store)

0 Likes

#5

In this example a SourceSystem was not used. DynamoDB is a bit different than Cassandra and relational databases where you do not specify a connect string. Access is managed by the IAM role in the AWS account.

1 Like