Hi Vinoth, I will try to reproduce the error on docker cluster and keep you updated.
On Tue, Dec 24, 2019 at 11:23 PM Vinoth Chandar <vin...@apache.org> wrote: > Pratyaksh, > > If you are still having this issue, could you try reproducing this on the > docker setup > > https://hudi.apache.org/docker_demo.html#step-7--incremental-query-for-copy-on-write-table > similar to this and raise a JIRA. > Happy to look into it and get it fixed if needed > > Thanks > Vinoth > > On Tue, Dec 24, 2019 at 8:43 AM lamberken <lamber...@163.com> wrote: > > > > > > > Hi, @Pratyaksh Sharma > > > > > > The log4j-1.2.17.jar lib also needs to added to the classpath, for > example: > > java -cp > > > /path/to/hive-jdbc-2.3.1.jar:/path/to/log4j-1.2.17.jar:packaging/hudi-utilities-bundle/target/hudi-utilities-bundle-0.5.1-SNAPSHOT.jar > > org.apache.hudi.utilities.HiveIncrementalPuller --help > > > > > > best, > > lamber-ken > > > > At 2019-12-24 17:23:20, "Pratyaksh Sharma" <pratyaks...@gmail.com> > wrote: > > >Hi Vinoth, > > > > > >Sorry my bad, I did not realise earlier that spark is not needed for > this > > >class. I tried running it with the below command to get the mentioned > > >exception - > > > > > >Command - > > > > > >java -cp > > > > > >/path/to/hive-jdbc-2.3.1.jar:packaging/hudi-utilities-bundle/target/hudi-utilities-bundle-0.5.1-SNAPSHOT.jar > > >org.apache.hudi.utilities.HiveIncrementalPuller --help > > > > > >Exception - > > >Exception in thread "main" java.lang.NoClassDefFoundError: > > >org/apache/log4j/LogManager > > > at > > > > > >org.apache.hudi.utilities.HiveIncrementalPuller.<clinit>(HiveIncrementalPuller.java:64) > > >Caused by: java.lang.ClassNotFoundException: org.apache.log4j.LogManager > > > at java.net.URLClassLoader.findClass(URLClassLoader.java:382) > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > > > ... 1 more > > > > > >I was able to fix it by including the corresponding jar in the bundle. > > > > > >After fixing the above, still I am getting the NPE even though the > > template > > >is bundled in the jar. > > > > > >On Mon, Dec 23, 2019 at 10:45 PM Vinoth Chandar <vin...@apache.org> > > wrote: > > > > > >> Hi Pratyaksh, > > >> > > >> HveIncrementalPuller is just a java program. Does not need Spark, > since > > it > > >> just runs a HiveQL remotely.. > > >> > > >> On the error you specified, seems like it can't find the template? Can > > you > > >> see if the bundle does not have the template file.. May be this got > > broken > > >> during the bundling changes.. (since its no longer part of the > resources > > >> folder of the bundle module).. We should also probably be throwing a > > better > > >> error than NPE.. > > >> > > >> We can raise a JIRA, once you confirm. > > >> > > >> String templateContent = > > >> > > >> > > > FileIOUtils.readAsUTFString(this.getClass().getResourceAsStream("IncrementalPull.sqltemplate")); > > >> > > >> > > >> On Mon, Dec 23, 2019 at 6:02 AM Pratyaksh Sharma < > pratyaks...@gmail.com > > > > > >> wrote: > > >> > > >> > Hi, > > >> > > > >> > Can someone guide me or share some documentation regarding how to > use > > >> > HiveIncrementalPuller. I already went through the documentation on > > >> > https://hudi.apache.org/querying_data.html. I tried using this > puller > > >> > using > > >> > the below command and facing the given exception. > > >> > > > >> > Any leads are appreciated. > > >> > > > >> > Command - > > >> > spark-submit --name incremental-puller --queue etl --files > > >> > incremental_sql.txt --master yarn --deploy-mode cluster > > --driver-memory > > >> 4g > > >> > --executor-memory 4g --num-executors 2 --class > > >> > org.apache.hudi.utilities.HiveIncrementalPuller > > >> > hudi-utilities-bundle-0.5.1-SNAPSHOT.jar --hiveUrl > > >> > jdbc:hive2://HOST:PORT/ --hiveUser <user> --hivePass <pass> > > >> > --extractSQLFile incremental_sql.txt --sourceDb <source_db> > > --sourceTable > > >> > <src_table> --targetDb tmp --targetTable tempTable --fromCommitTime > 0 > > >> > --maxCommits 1 > > >> > > > >> > Error - > > >> > > > >> > java.lang.NullPointerException > > >> > at org.apache.hudi.common.util.FileIOUtils.copy(FileIOUtils.java:73) > > >> > at > > >> > > > >> > > > >> > > > org.apache.hudi.common.util.FileIOUtils.readAsUTFString(FileIOUtils.java:66) > > >> > at > > >> > > > >> > > > >> > > > org.apache.hudi.common.util.FileIOUtils.readAsUTFString(FileIOUtils.java:61) > > >> > at > > >> > > > >> > > > >> > > > org.apache.hudi.utilities.HiveIncrementalPuller.<init>(HiveIncrementalPuller.java:113) > > >> > at > > >> > > > >> > > > >> > > > org.apache.hudi.utilities.HiveIncrementalPuller.main(HiveIncrementalPuller.java:343) > > >> > > > >> > > >
