This does sound like a fair bit of pain. I am wondering if it makes sense to change the integ-test setup/docker demo to use incremental puller. Bunch of the packaging issues around jars, seem like regressions that the hudi-utilities is not a fat jar anymore?
if there are nt any takers, I can also try my hand at fixing this, once I get done with few things on my end. left a comment on HUDI-485 On Tue, Dec 31, 2019 at 4:19 PM lamberken <lamber...@163.com> wrote: > > > Hi @Pratyaksh Sharma, > > > Thanks for your detail stackstrace and reproduce steps. And your > suggestion is reasonable. > > > 1, For NPE issue, please tracking pr #1167 < > https://github.com/apache/incubator-hudi/pull/1167> > 2, For TTransportException issue, I have a question that can other > statements be executed except create statement? > > > best, > lamber-ken > > At 2019-12-30 23:11:17, "Pratyaksh Sharma" <pratyaks...@gmail.com> wrote: > >Thank you Lamberken, the above issue gets resolved with what you > suggested. > >However, still HiveIncrementalPuller is not working. > >Subsequently I found and fixed a bug raised here - > >https://issues.apache.org/jira/browse/HUDI-485. > > > >Currently I am facing the below exception when trying to run the create > >table statement on docker cluster. Any leads for solving this are welcome > - > > > >6811 [main] ERROR org.apache.hudi.utilities.HiveIncrementalPuller - > >Exception when executing SQL > > > >java.sql.SQLException: org.apache.thrift.transport.TTransportException > > > >at > > >org.apache.hive.jdbc.HiveStatement.waitForOperationToComplete(HiveStatement.java:399) > > > >at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:254) > > > >at > > >org.apache.hudi.utilities.HiveIncrementalPuller.executeStatement(HiveIncrementalPuller.java:233) > > > >at > > >org.apache.hudi.utilities.HiveIncrementalPuller.executeIncrementalSQL(HiveIncrementalPuller.java:200) > > > >at > > >org.apache.hudi.utilities.HiveIncrementalPuller.saveDelta(HiveIncrementalPuller.java:157) > > > >at > > >org.apache.hudi.utilities.HiveIncrementalPuller.main(HiveIncrementalPuller.java:345) > > > >Caused by: org.apache.thrift.transport.TTransportException > > > >at > > >org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) > > > >at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) > > > >at > > >org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:374) > > > >at > > >org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:451) > > > >at > org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:433) > > > >at > > >org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:38) > > > >at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) > > > >at > > >org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:425) > > > >at > > >org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:321) > > > >at > > >org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:225) > > > >at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77) > > > >at > > >org.apache.hive.service.rpc.thrift.TCLIService$Client.recv_GetOperationStatus(TCLIService.java:467) > > > >at > > >org.apache.hive.service.rpc.thrift.TCLIService$Client.GetOperationStatus(TCLIService.java:454) > > > >at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > >at > > >sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > > >at > > >sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > > >at java.lang.reflect.Method.invoke(Method.java:498) > > > >at > > >org.apache.hive.jdbc.HiveConnection$SynchronizedHandler.invoke(HiveConnection.java:1524) > > > >at com.sun.proxy.$Proxy5.GetOperationStatus(Unknown Source) > > > >at > > >org.apache.hive.jdbc.HiveStatement.waitForOperationToComplete(HiveStatement.java:367) > > > >... 5 more > > > >6812 [main] ERROR org.apache.hudi.utilities.HiveIncrementalPuller - Could > >not close the resultset opened > > > >java.sql.SQLException: org.apache.thrift.transport.TTransportException > > > >at > > >org.apache.hive.jdbc.HiveStatement.closeClientOperation(HiveStatement.java:214) > > > >at org.apache.hive.jdbc.HiveStatement.close(HiveStatement.java:231) > > > >at > > >org.apache.hudi.utilities.HiveIncrementalPuller.saveDelta(HiveIncrementalPuller.java:165) > > > >at > > >org.apache.hudi.utilities.HiveIncrementalPuller.main(HiveIncrementalPuller.java:345) > > > >Caused by: org.apache.thrift.transport.TTransportException > > > >at > > >org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) > > > >at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) > > > >at > > >org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:374) > > > >at > > >org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:451) > > > >at > org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:433) > > > >at > > >org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:38) > > > >at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) > > > >at > > >org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:425) > > > >at > > >org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:321) > > > >at > > >org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:225) > > > >at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77) > > > >at > > >org.apache.hive.service.rpc.thrift.TCLIService$Client.recv_CloseOperation(TCLIService.java:513) > > > >at > > >org.apache.hive.service.rpc.thrift.TCLIService$Client.CloseOperation(TCLIService.java:500) > > > >at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > >at > > >sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > > >at > > >sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > > >at java.lang.reflect.Method.invoke(Method.java:498) > > > >at > > >org.apache.hive.jdbc.HiveConnection$SynchronizedHandler.invoke(HiveConnection.java:1524) > > > >at com.sun.proxy.$Proxy5.CloseOperation(Unknown Source) > > > >at > > >org.apache.hive.jdbc.HiveStatement.closeClientOperation(HiveStatement.java:208) > > > >... 3 more > > > >Also the documentation does not mention the jars which need to be passed > >externally in classPath for executing above tool. We should upgrade the > >documentation to list down the jars so that it becomes easier for a new > >user to use this tool. I spent a lot of time adding all the jars > >incrementally. This jira (https://issues.apache.org/jira/browse/HUDI-486) > >tracks this. > > > >On Mon, Dec 30, 2019 at 5:35 PM lamberken <lamber...@163.com> wrote: > > > >> > >> > >> Hi @Pratyaksh Sharma > >> > >> > >> Thanks for your steps to reproduce this issue. Try to modify bellow > codes, > >> and test again. > >> > >> > >> org.apache.hudi.utilities.HiveIncrementalPuller#HiveIncrementalPuller / > >> --------------------------------- / String templateContent = > >> > FileIOUtils.readAsUTFString(this.getClass().getResourceAsStream("IncrementalPull.sqltemplate")); > >> Changed to > >> / --------------------------------- / String templateContent = > >> > FileIOUtils.readAsUTFString(this.getClass().getResourceAsStream("/IncrementalPull.sqltemplate")); > >> best, > >> lamber-ken > >> > >> > >> > >> > >> > >> At 2019-12-30 19:25:08, "Pratyaksh Sharma" <pratyaks...@gmail.com> > wrote: > >> >Hi Vinoth, > >> > > >> >I am able to reproduce this error on docker setup and have filed a > jira - > >> >https://issues.apache.org/jira/browse/HUDI-484. > >> > > >> >Steps to reproduce are mentioned in the jira description itself. > >> > > >> >On Thu, Dec 26, 2019 at 12:42 PM Pratyaksh Sharma < > pratyaks...@gmail.com> > >> >wrote: > >> > > >> >> Hi Vinoth, > >> >> > >> >> I will try to reproduce the error on docker cluster and keep you > >> updated. > >> >> > >> >> On Tue, Dec 24, 2019 at 11:23 PM Vinoth Chandar <vin...@apache.org> > >> wrote: > >> >> > >> >>> Pratyaksh, > >> >>> > >> >>> If you are still having this issue, could you try reproducing this > on > >> the > >> >>> docker setup > >> >>> > >> >>> > >> > https://hudi.apache.org/docker_demo.html#step-7--incremental-query-for-copy-on-write-table > >> >>> similar to this and raise a JIRA. > >> >>> Happy to look into it and get it fixed if needed > >> >>> > >> >>> Thanks > >> >>> Vinoth > >> >>> > >> >>> On Tue, Dec 24, 2019 at 8:43 AM lamberken <lamber...@163.com> > wrote: > >> >>> > >> >>> > > >> >>> > > >> >>> > Hi, @Pratyaksh Sharma > >> >>> > > >> >>> > > >> >>> > The log4j-1.2.17.jar lib also needs to added to the classpath, for > >> >>> example: > >> >>> > java -cp > >> >>> > > >> >>> > >> > /path/to/hive-jdbc-2.3.1.jar:/path/to/log4j-1.2.17.jar:packaging/hudi-utilities-bundle/target/hudi-utilities-bundle-0.5.1-SNAPSHOT.jar > >> >>> > org.apache.hudi.utilities.HiveIncrementalPuller --help > >> >>> > > >> >>> > > >> >>> > best, > >> >>> > lamber-ken > >> >>> > > >> >>> > At 2019-12-24 17:23:20, "Pratyaksh Sharma" <pratyaks...@gmail.com > > > >> >>> wrote: > >> >>> > >Hi Vinoth, > >> >>> > > > >> >>> > >Sorry my bad, I did not realise earlier that spark is not needed > for > >> >>> this > >> >>> > >class. I tried running it with the below command to get the > >> mentioned > >> >>> > >exception - > >> >>> > > > >> >>> > >Command - > >> >>> > > > >> >>> > >java -cp > >> >>> > > >> >>> > > >> >>> > >> > >/path/to/hive-jdbc-2.3.1.jar:packaging/hudi-utilities-bundle/target/hudi-utilities-bundle-0.5.1-SNAPSHOT.jar > >> >>> > >org.apache.hudi.utilities.HiveIncrementalPuller --help > >> >>> > > > >> >>> > >Exception - > >> >>> > >Exception in thread "main" java.lang.NoClassDefFoundError: > >> >>> > >org/apache/log4j/LogManager > >> >>> > > at > >> >>> > > >> >>> > > >> >>> > >> > >org.apache.hudi.utilities.HiveIncrementalPuller.<clinit>(HiveIncrementalPuller.java:64) > >> >>> > >Caused by: java.lang.ClassNotFoundException: > >> >>> org.apache.log4j.LogManager > >> >>> > > at > >> java.net.URLClassLoader.findClass(URLClassLoader.java:382) > >> >>> > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > >> >>> > > at > >> >>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) > >> >>> > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > >> >>> > > ... 1 more > >> >>> > > > >> >>> > >I was able to fix it by including the corresponding jar in the > >> bundle. > >> >>> > > > >> >>> > >After fixing the above, still I am getting the NPE even though > the > >> >>> > template > >> >>> > >is bundled in the jar. > >> >>> > > > >> >>> > >On Mon, Dec 23, 2019 at 10:45 PM Vinoth Chandar < > vin...@apache.org> > >> >>> > wrote: > >> >>> > > > >> >>> > >> Hi Pratyaksh, > >> >>> > >> > >> >>> > >> HveIncrementalPuller is just a java program. Does not need > Spark, > >> >>> since > >> >>> > it > >> >>> > >> just runs a HiveQL remotely.. > >> >>> > >> > >> >>> > >> On the error you specified, seems like it can't find the > template? > >> >>> Can > >> >>> > you > >> >>> > >> see if the bundle does not have the template file.. May be this > >> got > >> >>> > broken > >> >>> > >> during the bundling changes.. (since its no longer part of the > >> >>> resources > >> >>> > >> folder of the bundle module).. We should also probably be > >> throwing a > >> >>> > better > >> >>> > >> error than NPE.. > >> >>> > >> > >> >>> > >> We can raise a JIRA, once you confirm. > >> >>> > >> > >> >>> > >> String templateContent = > >> >>> > >> > >> >>> > >> > >> >>> > > >> >>> > >> > FileIOUtils.readAsUTFString(this.getClass().getResourceAsStream("IncrementalPull.sqltemplate")); > >> >>> > >> > >> >>> > >> > >> >>> > >> On Mon, Dec 23, 2019 at 6:02 AM Pratyaksh Sharma < > >> >>> pratyaks...@gmail.com > >> >>> > > > >> >>> > >> wrote: > >> >>> > >> > >> >>> > >> > Hi, > >> >>> > >> > > >> >>> > >> > Can someone guide me or share some documentation regarding > how > >> to > >> >>> use > >> >>> > >> > HiveIncrementalPuller. I already went through the > documentation > >> on > >> >>> > >> > https://hudi.apache.org/querying_data.html. I tried using > this > >> >>> puller > >> >>> > >> > using > >> >>> > >> > the below command and facing the given exception. > >> >>> > >> > > >> >>> > >> > Any leads are appreciated. > >> >>> > >> > > >> >>> > >> > Command - > >> >>> > >> > spark-submit --name incremental-puller --queue etl --files > >> >>> > >> > incremental_sql.txt --master yarn --deploy-mode cluster > >> >>> > --driver-memory > >> >>> > >> 4g > >> >>> > >> > --executor-memory 4g --num-executors 2 --class > >> >>> > >> > org.apache.hudi.utilities.HiveIncrementalPuller > >> >>> > >> > hudi-utilities-bundle-0.5.1-SNAPSHOT.jar --hiveUrl > >> >>> > >> > jdbc:hive2://HOST:PORT/ --hiveUser <user> --hivePass <pass> > >> >>> > >> > --extractSQLFile incremental_sql.txt --sourceDb <source_db> > >> >>> > --sourceTable > >> >>> > >> > <src_table> --targetDb tmp --targetTable tempTable > >> >>> --fromCommitTime 0 > >> >>> > >> > --maxCommits 1 > >> >>> > >> > > >> >>> > >> > Error - > >> >>> > >> > > >> >>> > >> > java.lang.NullPointerException > >> >>> > >> > at > >> >>> org.apache.hudi.common.util.FileIOUtils.copy(FileIOUtils.java:73) > >> >>> > >> > at > >> >>> > >> > > >> >>> > >> > > >> >>> > >> > >> >>> > > >> >>> > >> > org.apache.hudi.common.util.FileIOUtils.readAsUTFString(FileIOUtils.java:66) > >> >>> > >> > at > >> >>> > >> > > >> >>> > >> > > >> >>> > >> > >> >>> > > >> >>> > >> > org.apache.hudi.common.util.FileIOUtils.readAsUTFString(FileIOUtils.java:61) > >> >>> > >> > at > >> >>> > >> > > >> >>> > >> > > >> >>> > >> > >> >>> > > >> >>> > >> > org.apache.hudi.utilities.HiveIncrementalPuller.<init>(HiveIncrementalPuller.java:113) > >> >>> > >> > at > >> >>> > >> > > >> >>> > >> > > >> >>> > >> > >> >>> > > >> >>> > >> > org.apache.hudi.utilities.HiveIncrementalPuller.main(HiveIncrementalPuller.java:343) > >> >>> > >> > > >> >>> > >> > >> >>> > > >> >>> > >> >> > >> >
