vinothchandar commented on issue #639: About global dedup, find some commit who 
keep inflight and still generate parquet file and fail dedup
URL: https://github.com/apache/incubator-hudi/issues/639#issuecomment-483790014
 
 
   Hi.. please leave us a note on the dev mailing list for faster response. 
   https://hudi.apache.org/community.html 
   
   > Did I missing some hudi config in writing pipeline?
   
   For Spark SQL, you need to push the path filter as documented here. 
https://hudi.apache.org/querying_data.html#spark-ro-view 
   Getting duplicates typically means, this  is missing..  
   
   > Why the commit is still inflight but parquet file can be generated?
   
   So, the sequence of events is . We write parquet files and then upon 
successful writing of all attempted parquet files, we actually make the commit 
as completed. (i.e not inflight anymore). So this is normal. This is done to 
prevent queries from reading partially written parquet files.. 
   
   >  In hudi, did the commit finish and then spark .save finish so there is no 
asynchronous problem?
   
   .save() internally calls hoodieWriteClient.commit(). 
   
   > Is there any reader side config issue with hive output format ? since it's 
org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat
   
   OutputFormat is not really used anywhere by Hudi. So no. 
   
   Hope that helps. I will start a thread on the mailing list and we can 
continue there, if you have more follow ups. 
   
   
   
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[email protected]


With regards,
Apache Git Services

Reply via email to