Incremental scan was not available with jdbc operator till now. +1 for adding that.
-Priyanka On Mon, May 9, 2016 at 8:56 AM, Mohit Jotwani <[email protected]> wrote: > +1 for incremental data. > > Regards, > Mohit > On 9 May 2016 19:59, "Yogi Devendra" <[email protected]> wrote: > > > +1 for incremental data fetching. > > for fetchDirection variable; it is better to get inputs from original > > author (if possible). > > > > ~ Yogi > > > > On 9 May 2016 at 19:04, Akshay Gore <[email protected]> wrote: > > > > > +1 for incremental data fetching. This is a must-have feature. > > > > > > -Akshay > > > On 09-May-2016 3:39 pm, "Sandeep Deshmukh" <[email protected]> > > > wrote: > > > > > > > Hi All, > > > > > > > > I am using JdbcPOJOInputOperator to ingest data from mysql to HDFS. I > > > > observed that once the existing data is ingested, newly added data > in > > > > mysql is not ingested. At the same time, if I add some data to mysql > > when > > > > the ingestion is still going on, the newly added data is also > ingested > > on > > > > HDFS. > > > > > > > > In the code, fetching data in batches in achieved using fetchSize > > > parameter > > > > that limits the number of tuples to fetch per result set and > pageNumber > > > is > > > > used internally to manage the offset calculation as ( fetchSize * > > > > pageNumber). The pageNumber is incremented per window. > > > > > > > > When the existing tuples are ingested, there is no further data > ingest > > > but > > > > the pageNumber variable is still incremented. This results is trying > to > > > > fetch data that is beyond the number of tuples in the > > table/queryresult. > > > > > > > > Changing offset calculations to tuples read so far will fix this > issue > > > and > > > > the operator can then be used to poll for newer data in the table. > > > > > > > > If you need to have a quick look at the code: > > https://github.com/apache/ > > > > incubator-apex-malhar/blob/master/library/src/main/java/ > > > > com/datatorrent/lib/db/jdbc/JdbcPOJOInputOperator.java > > > > > > > > Side observation: fetchDirection variable is unused in the code. Will > > > > remove it from the class. > > > > > > > > Would like get your thoughts on my observations. I will create a JIRA > > and > > > > open a PR based on inputs received on this thread. > > > > > > > > Regards, > > > > Sandeep > > > > > > > > > >
