How to define a connection to an external DynamoDB datastore?


The Data Virtualization topic in the documentation lists an available connector for DynamoDB, but SqlSourceSystem doesn’t support it, so how can we define an external type that persists to Dynamo?

1 Like


@rohit.sureka Is there any support for external types on DynamoDB? There is no SqlSystem/SqlSourceCollection as is mentioned as I don’t believe there is jdbc support for it (is there?).



A connector for DynamoDB does exist. It is a “low-level” connector, providing basic connectivity and API support. In 7.8/7.9, you will need to implement fetch and create/upsert APIs.

Here is a basic (hard-coded) example with plenty of opportunity for improvement. NOTE: password handling is omitted from this example. I leave it to you to implement create->put APIs.

DynamoDBHelper.c3typ - help type to facilitate integration with DynamoDB.

type DynamoDBHelper<T> {
   * Fetches data from DynamoDB and creates a FetchResult from the response
   * @param spec - C3 Fetch specification
   * @returns FetchResults
  fetch: function(spec : FetchSpec) : FetchResult<T> js server

DynamoDBHelper.js - helper implementation. Lots of room for improvement.

var log = C3.logger("DynamoDBHelper");

 * Fetches data from DynamoDB and creates a FetchResult from the response
 * @param spec - Fetch specification
 * @returns FetchResults
function fetch(spec) {
  var typeName = c3CurrentAction().getTarget().getType();
  var tag = MetadataStore.tag(); 
  var typ = tag.readType(typeName); 
  var tableName = typ.extensions();
  var partitionKeyField = typ.extensions().db.partitionKeyField;
  var rows = [];
  var scanSpec = null;
  if (spec.filter != undefined) {
	  scanSpec = constructDynamoDBScanSpec(spec);
  var scanResults = DynamoDB.scan(tableName, scanSpec);
  if (scanResults.count > 0) {
	  rows = createC3TypeFromDynamoResults(typeName, scanResults.items);
  return {
    objs : rows,
    count: rows.length,
    hasMore : false

 * Converts DynamoDB rows to a C3 Type
 * @param typeName - Name of the type to instantiate
 * @param dynamoItems - list of records from DynamoDB
 * @returns [] - array of records to be included in the FetchResult
function createC3TypeFromDynamoResults(typeName, dynamoItems) {
	var rows = [];
	// long term, get list of fields from Type System
        // enhance to iterate through fieldTypes
	var fields = ["Id", "Brand", "Description", "Price", "ProductCategory", "Title", "BicycleType"];
	dynamoItems.forEach(function (item) {
		var o = c3Type(typeName).make();
		for (var j = 0; j < fields.length; j++) {
			if (item[fields[j]] != undefined ) {
				o = o.putField(fields[j], item[fields[j]]);
	return rows;

 * Constructs a DynamoDBScanSpec from a FetchSpec.  For the moment it supports the most basic of queries. 
 * @param fetchSpec
 * @returns DynamoDBScanSpec
function constructDynamoDBScanSpec(fetchSpec) {
	var vars = fetchSpec.filter.split("==");
	var filterExpression = vars[0].trim() + " = :val";
	var expressionAttributeValues = { ":val": vars[1].replace(/'/g, "") };
	var spec = DynamoDBScanSpec.make({filterExpression: filterExpression, expressionAttributeValues: expressionAttributeValues });

	return spec;

ProductCatalog.c3typ - Type that models the ProductCatalog DynamoDB table.

type ProductCatalog mixes DynamoDBHelper<ProductCatalog> schema name "ProductCatalog" {

  Id: int
  Brand: string
  Description: string
  Price: double
  ProductCategory: string
  Title: string
  BicycleType: string



What would be the parameters/fields inside DynamoDBSourceSystem type? It would be great if you could provide the definition of source system similar to cassandra (How to connect an external cassandra data store)



In this example a SourceSystem was not used. DynamoDB is a bit different than Cassandra and relational databases where you do not specify a connect string. Access is managed by the IAM role in the AWS account.

1 Like