Thanks Danny and Vinoth for your comments. I have created https://issues.apache.org/jira/browse/HUDI-2082 for tracking this.
Regards, Vinay Patil On Wed, Jun 16, 2021 at 12:53 PM Vinoth Chandar <vin...@apache.org> wrote: > +1 on this effort overall. It will be a little tricky, but doable. > > First thing is to see how we can replace raw usages of Spark APIs with > HoodieEngineContext. It will be cool if we can completely generify > DeltaStreamer. > but I suspect we need flink/spark specific modules ultimately > > On Wed, Jun 16, 2021 at 12:17 AM Danny Chan <danny0...@apache.org> wrote: > > > There was actually an issue here: > > https://issues.apache.org/jira/browse/HUDI-1872, maybe you can take it > and > > go on with the work ~ > > > > Best, > > Danny Chan > > > > Vinay Patil <vinay18.pa...@gmail.com> 于2021年6月11日周五 下午3:26写道： > > > > > Thank you Danny for your response. > > > > > > Can we have a JIRA story where all the refactoring is required for > > > Hudi-Flink code as well. > > > > > > I will create a task if we agree that a Flag will be helpful to choose > > > different runners > > > > > > Regards, > > > Vinay Patil > > > > > > > > > On Fri, Jun 11, 2021 at 12:34 PM Danny Chan <danny0...@apache.org> > > wrote: > > > > > > > Basically agree with that, but before that we may need some > refactoring > > > to > > > > the existing code: > > > > > > > > Move the HoodieFlinkStreamer from the hudi-flink module into the > > > > hudi-utilities to be together with the HoodieDeltaStreamer. > > > > We are planning to add separate flink compaction programs too, which > > has > > > > the same problem. > > > > > > > > Best, > > > > Danny Chan > > > > > > > > Vinay Patil <vinay18.pa...@gmail.com> 于2021年6月9日周三 下午3:42写道： > > > > > > > > > Hi Team, > > > > > > > > > > Currently, Hudi supports Flink as well Spark, there are two > different > > > > > classes > > > > > 1. HoodieDeltaStreamer > > > > > 2. FlinkHoodieDeltaStreamer > > > > > > > > > > Should we have a provision to pass the flag like --runner to choose > > > > between > > > > > Flink or Spark and have a single entry point class which will take > > all > > > > the > > > > > common configs. > > > > > > > > > > Based on the runner flag, we can call HoodieDeltaStreamer or > > > > > FlinkHoodieDeltaStreamer > > > > > > > > > > Thoughts? > > > > > > > > > > Regards, > > > > > Vinay Patil > > > > > > > > > > > > > > >
