Hi Vinoth, I am able to reproduce this error on docker setup and have filed a jira - https://issues.apache.org/jira/browse/HUDI-484.
Steps to reproduce are mentioned in the jira description itself. On Thu, Dec 26, 2019 at 12:42 PM Pratyaksh Sharma <pratyaks...@gmail.com> wrote: > Hi Vinoth, > > I will try to reproduce the error on docker cluster and keep you updated. > > On Tue, Dec 24, 2019 at 11:23 PM Vinoth Chandar <vin...@apache.org> wrote: > >> Pratyaksh, >> >> If you are still having this issue, could you try reproducing this on the >> docker setup >> >> https://hudi.apache.org/docker_demo.html#step-7--incremental-query-for-copy-on-write-table >> similar to this and raise a JIRA. >> Happy to look into it and get it fixed if needed >> >> Thanks >> Vinoth >> >> On Tue, Dec 24, 2019 at 8:43 AM lamberken <lamber...@163.com> wrote: >> >> > >> > >> > Hi, @Pratyaksh Sharma >> > >> > >> > The log4j-1.2.17.jar lib also needs to added to the classpath, for >> example: >> > java -cp >> > >> /path/to/hive-jdbc-2.3.1.jar:/path/to/log4j-1.2.17.jar:packaging/hudi-utilities-bundle/target/hudi-utilities-bundle-0.5.1-SNAPSHOT.jar >> > org.apache.hudi.utilities.HiveIncrementalPuller --help >> > >> > >> > best, >> > lamber-ken >> > >> > At 2019-12-24 17:23:20, "Pratyaksh Sharma" <pratyaks...@gmail.com> >> wrote: >> > >Hi Vinoth, >> > > >> > >Sorry my bad, I did not realise earlier that spark is not needed for >> this >> > >class. I tried running it with the below command to get the mentioned >> > >exception - >> > > >> > >Command - >> > > >> > >java -cp >> > >> > >> >/path/to/hive-jdbc-2.3.1.jar:packaging/hudi-utilities-bundle/target/hudi-utilities-bundle-0.5.1-SNAPSHOT.jar >> > >org.apache.hudi.utilities.HiveIncrementalPuller --help >> > > >> > >Exception - >> > >Exception in thread "main" java.lang.NoClassDefFoundError: >> > >org/apache/log4j/LogManager >> > > at >> > >> > >> >org.apache.hudi.utilities.HiveIncrementalPuller.<clinit>(HiveIncrementalPuller.java:64) >> > >Caused by: java.lang.ClassNotFoundException: >> org.apache.log4j.LogManager >> > > at java.net.URLClassLoader.findClass(URLClassLoader.java:382) >> > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >> > > at >> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) >> > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >> > > ... 1 more >> > > >> > >I was able to fix it by including the corresponding jar in the bundle. >> > > >> > >After fixing the above, still I am getting the NPE even though the >> > template >> > >is bundled in the jar. >> > > >> > >On Mon, Dec 23, 2019 at 10:45 PM Vinoth Chandar <vin...@apache.org> >> > wrote: >> > > >> > >> Hi Pratyaksh, >> > >> >> > >> HveIncrementalPuller is just a java program. Does not need Spark, >> since >> > it >> > >> just runs a HiveQL remotely.. >> > >> >> > >> On the error you specified, seems like it can't find the template? >> Can >> > you >> > >> see if the bundle does not have the template file.. May be this got >> > broken >> > >> during the bundling changes.. (since its no longer part of the >> resources >> > >> folder of the bundle module).. We should also probably be throwing a >> > better >> > >> error than NPE.. >> > >> >> > >> We can raise a JIRA, once you confirm. >> > >> >> > >> String templateContent = >> > >> >> > >> >> > >> FileIOUtils.readAsUTFString(this.getClass().getResourceAsStream("IncrementalPull.sqltemplate")); >> > >> >> > >> >> > >> On Mon, Dec 23, 2019 at 6:02 AM Pratyaksh Sharma < >> pratyaks...@gmail.com >> > > >> > >> wrote: >> > >> >> > >> > Hi, >> > >> > >> > >> > Can someone guide me or share some documentation regarding how to >> use >> > >> > HiveIncrementalPuller. I already went through the documentation on >> > >> > https://hudi.apache.org/querying_data.html. I tried using this >> puller >> > >> > using >> > >> > the below command and facing the given exception. >> > >> > >> > >> > Any leads are appreciated. >> > >> > >> > >> > Command - >> > >> > spark-submit --name incremental-puller --queue etl --files >> > >> > incremental_sql.txt --master yarn --deploy-mode cluster >> > --driver-memory >> > >> 4g >> > >> > --executor-memory 4g --num-executors 2 --class >> > >> > org.apache.hudi.utilities.HiveIncrementalPuller >> > >> > hudi-utilities-bundle-0.5.1-SNAPSHOT.jar --hiveUrl >> > >> > jdbc:hive2://HOST:PORT/ --hiveUser <user> --hivePass <pass> >> > >> > --extractSQLFile incremental_sql.txt --sourceDb <source_db> >> > --sourceTable >> > >> > <src_table> --targetDb tmp --targetTable tempTable >> --fromCommitTime 0 >> > >> > --maxCommits 1 >> > >> > >> > >> > Error - >> > >> > >> > >> > java.lang.NullPointerException >> > >> > at >> org.apache.hudi.common.util.FileIOUtils.copy(FileIOUtils.java:73) >> > >> > at >> > >> > >> > >> > >> > >> >> > >> org.apache.hudi.common.util.FileIOUtils.readAsUTFString(FileIOUtils.java:66) >> > >> > at >> > >> > >> > >> > >> > >> >> > >> org.apache.hudi.common.util.FileIOUtils.readAsUTFString(FileIOUtils.java:61) >> > >> > at >> > >> > >> > >> > >> > >> >> > >> org.apache.hudi.utilities.HiveIncrementalPuller.<init>(HiveIncrementalPuller.java:113) >> > >> > at >> > >> > >> > >> > >> > >> >> > >> org.apache.hudi.utilities.HiveIncrementalPuller.main(HiveIncrementalPuller.java:343) >> > >> > >> > >> >> > >> >
