Re: Spark Structured Streaming and Kafka message schema evolution

Thanks Jungtaek. I have reasons for this. So I will bring it up in another thread Cheers, LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw * *Disclaimer:* Use it a

Re: Spark Structured Streaming and Kafka message schema evolution

If I understand correctly, SQL semantics are strict on column schema. Reading via Kafka data source doesn't require you to specify the schema as it provides the key and value as binary, but once you deserialize them, unless you keep the type as primitive (e.g. String), you'll need to specify the sc

Spark Structured Streaming and Kafka message schema evolution

This is just a query. In general Kafka-connect requires means to register that schema such that producers and consumers understand that. It also allows schema evolution, i.e. changes to metadata that identifies the structure of data sent via topic. When we stream a kafka topic into (Spark Structu