rdblue commented on pull request #1046: URL: https://github.com/apache/iceberg/pull/1046#issuecomment-635497067
Thanks for the context, @teabot. For our streaming systems, we aren't supporting the case you're talking about and each stream (Kafka topic) is getting its own Iceberg schema so that schema evolution works mostly the same way for a topic as it does for a table. That aligns well with streaming SQL use cases, which we hope will make it easier to use streaming systems. It doesn't help the problem of event ordering across event types, though, which are required to be in separate streams unless they use the union-as-struct convention. I think the trade-off is worth it. In most cases you don't need strict order guarantees across event types. Where you do, you can either join two streams with a common ordinal field or make the schema slightly more complex using the union-as-struct approach if you need to avoid the join. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
