[ https://issues.apache.org/jira/browse/HUDI-69?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082773#comment-17082773 ]
Yanjia Gary Li edited comment on HUDI-69 at 4/14/20, 10:11 PM: --------------------------------------------------------------- After a closer look, I think Spark datasource support for realtime table needs: * Support hadoop.mapreduce.xxx apis. We use hadoop.mapred.RecordReader, but Spark sql use hadoop.mapreduce.RecordReader. We need to figure how to support both apis, or upgrade to mapreduce. * Implement the extension of ParquetInputFormat from Spark or a custom data source reader to handle merge. [https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala] * Use Datasource V2 to be the default data source. Please let me know what you guys think. was (Author: garyli1019): After a closer look, I think Spark datasource support for realtime table needs: * Refactoring HoodieRealtimeFormat and (file split, record reader). Decouple Hudi logic from the MapredParquetInputFormat. I think we can maintain the Hudi file split and path filtering in a central place, and able to be adopted by different query engines. With bootstrap support, the file format maintenance could be more complicated. I think this is very essential. * Implement the extension of ParquetInputFormat from Spark or a custom data source reader to handle merge. [https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala] * Use Datasource V2 to be the default data source. Please let me know what you guys think. > Support realtime view in Spark datasource #136 > ---------------------------------------------- > > Key: HUDI-69 > URL: https://issues.apache.org/jira/browse/HUDI-69 > Project: Apache Hudi (incubating) > Issue Type: New Feature > Components: Spark Integration > Reporter: Vinoth Chandar > Assignee: Yanjia Gary Li > Priority: Major > Fix For: 0.6.0 > > > https://github.com/uber/hudi/issues/136 -- This message was sent by Atlassian Jira (v8.3.4#803005)