vinothchandar commented on issue #639: About global dedup, find some commit who keep inflight and still generate parquet file and fail dedup URL: https://github.com/apache/incubator-hudi/issues/639#issuecomment-483790014 Hi.. please leave us a note on the dev mailing list for faster response. https://hudi.apache.org/community.html > Did I missing some hudi config in writing pipeline? For Spark SQL, you need to push the path filter as documented here. https://hudi.apache.org/querying_data.html#spark-ro-view Getting duplicates typically means, this is missing.. > Why the commit is still inflight but parquet file can be generated? So, the sequence of events is . We write parquet files and then upon successful writing of all attempted parquet files, we actually make the commit as completed. (i.e not inflight anymore). So this is normal. This is done to prevent queries from reading partially written parquet files.. > In hudi, did the commit finish and then spark .save finish so there is no asynchronous problem? .save() internally calls hoodieWriteClient.commit(). > Is there any reader side config issue with hive output format ? since it's org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat OutputFormat is not really used anywhere by Hudi. So no. Hope that helps. I will start a thread on the mailing list and we can continue there, if you have more follow ups.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
