Hi Cheng, I think that is one way you could approach it. I'm not really familiar enough with Hudi to know if that's the right way. I took a quick look at https://hudi.incubator.apache.org/concepts.html and I'm wondering if it would actually be cleaner to implement as a variant of HdfsTable. HdfsTable is used for any Hive filesystem-based table, not just HDFS - e.g. S3 or whatever. Hudi seems like it's similar Hive ACID in a lot of ways, which we're currently adding support for in that way.
Which Hudi features are you planning to implement? Copy-on-Write seems like it would be simpler to implement - it might only require changes in the frontend (i.e. java code). Merge-on-read probably requires backend support for merging the delta files with the base files. Write support also seems more complex than read support. Also another note - currently there are actually two catalog implementations that require their own table implementation, e.g. see fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java and fe/src/main/java/org/apache/impala/catalog/HBaseTable.java On Tue, Jul 16, 2019 at 9:55 AM FIXED-TERM Cheng Yuanbin (CR/PJ-AI-S1) < fixed-term.yuanbin.ch...@us.bosch.com> wrote: > Hi, > > Our team now is using Apache Hudi to migrate our data pipeline from batch > to incremental processing. > However, we find that the Apache Impala cannot pull the Hudi metadata from > the Hive. > Here is the issue: https://github.com/apache/incubator-hudi/issues/179 > Now I am trying to fix this issue. > > After reading some code related to the table object of the Impala, > currently, my thought is to implement a new HudiTable class and add it to > the fromMetastoreTable method in Table class. > Maybe only add some support methods in the current Table type can also > solve this issue? Not very familiar with the Impala source code. > Here is the Jira ticket for this issue: > https://issues.apache.org/jira/projects/HUDI/issues/HUDI-146 > > Do you have any idea about how to solve this issue? > > I appreciate any help! > > Best regards > > Yuanbin Cheng > CR/PJ-AI-S1 > > >
