Fetching from cassandra backed entity - Unsupported valueType=ReferenceType

#1

We have an entity type utilizing cassandra and have a field defined on it as ourComponent : OurComponentType
When we try to fetch we get this error: Columnar type Error : Unsupported valueType=ReferenceType for field=ourComponent in type=Columnar We understood reference fields to work with these, is it a limitation or does someone know of an annotation perhaps that enables this?

0 Likes

#2

References are used in a relational db (i.e postgres) to do joins. I don’t think it make sense to do a join if one of the types involves Cassandra.

0 Likes

#3

I understand the ramifications of trying to join non relational to relational on the whole, but we shouldn’t let the technology used to persist the entity data cloud the vision of applicable use cases. After all, who’s to say we don’t swap out the back end on a type in the next release. I’d like to know the support of this concept, as it does not fail to provision and does not fail to ingest data. It only fails on retrieval. Regardless of the entity backing, because I’m not supposed to know or care if I’m an end user. If there’s data structured into a type from a fetch from unstructured data - that can be related to other things of helpful nature in the platform, then there’s value and applicable use cases for this.

0 Likes

#4

@clowtown I agree this should be a provisioning time error.

This is currently not supported only for timeseries data points (aka types mixing in TimedDataPoint or TimeseriesDataPoint). The reason for this is, in order to heavily compress & optimize storage in cassandra for timeseries data points (for cost reasons), we convert the traditional data points into a columnar version of the data. Not having reference/arry/map fields allows us to heavily compress the time series data ~10-12x and provide significant cost savings from an overall product standpoint.

Any other cassandra type (that is not time series) should have support for any data structure that you want to store.

Even for time series we eventually want to allow storing any data structure that the user wants to store, but every time extra fields have been added on the lowest level time series type, they’ve brought up a higher level design question (we should avoid storing other meta information on telemetry - obviously unless its related and changing per data point).

You should raise a ticket with services to not allow / throw an error for such a field during provisioning

0 Likes

#5

thank you, i’ve opend a zendesk ticket from our side for this.

0 Likes