Hi Nicholas, We haven't given enough thought to it. At a high level, it looks bit hard to generalise as we won't have any control over the arbitrary queries. In that case, the burden would be on the user to configure complex queries to have incremental nature. We would also need to provide a way to pass custom parameters to the arbitrary queries which would be defined by the user. It is feasible but needs significant design work. We will make note of this in our future work.
I hope that makes sense. Thanks, Rushiraj On Mon, Jan 6, 2020 at 2:40 PM 蒋晓峰 <[email protected]> wrote: > > Hi Purushotham, > About arbitrary queries (multi table complex queries), why support this > only in Bulk Mode? What concern about this? > Thanks, > Nicholas > > > At 2020-01-06 15:41:31, "Purushotham Pushpavanthar" <[email protected]> > wrote: > >Hi everyone, > > > >We are working on introducing JDBC Delta Streamer as > >one of the sources for HUDI. We've drafted initial version of design in > >RFC-14. > >Kindly review and let us know your thoughts. > > > >I'm initiating this thread to discuss few comments raised by Vinoth. > > > > 1. As discussed on the RFC page, we are going to support compound > > incremental columns. > > 2. About arbitrary queries (multi table complex queries), we are > > planning to support this only in Bulk Mode. > > > > > >[1] https://issues.apache.org/jira/browse/HUDI-251 > >[2] > >https://cwiki.apache.org/confluence/display/HUDI/RFC+-+14+%3A+JDBC+incremental+puller > > > >Regards, > >Purushotham Pushpavanth
