getniz commented on issue #2101: URL: https://github.com/apache/hudi/issues/2101#issuecomment-698761326
@n3nash thanks for the response with details, 1 & 3 option I may not be able to consider as I need to build this layer as immediate target tables for further consumption in Reporting layer. If I use option 2, can I able to consume the topic and flatten the schema in deltastreamer with-out staging and then load directly to the immediate target layer using above Spark submit batch command. Also, I came to know that Hudi supports Confluent schema registry, in that case if I get the JSON schema from Source and register with Confluent registry can I achieve in flattening the file. Sorry, my questions may be silly sometimes please bear with me, I'm a learner here : ) Objective of what I'm trying to do is to consume Data from several topics in near real-time(all the topics data are formatted/structured) and push to DataLake using Hudi. If I stage and transform it, then I may end up eating time. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
