Sounds similar to Confluent Kafka Schema Registry and Kafka Connect.
The Schema Registry and Kafka Connect themselves are open-source, but some of
the datasource-specific adapters, and GUIs to manage it all, are not
open-source (see Confluent Enterprise Edition).
Note that the Schema Registry
Has anyone seen AWS Glue? I was wondering if there is something similar going
to be built into Spark Structured Streaming? I like the Data Catalog idea to
store and track any data source/destination. It profiles the data to derive the
scheme and data types. Also, it does some sort-of automated
