[
https://issues.apache.org/jira/browse/NIFI-2575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess updated NIFI-2575:
-------------------------------
Description:
When configuring a HiveQL processor using the Zookeeper URL (e.g.
jdbc:hive2://ydavis-hdp-nifi-test-3.openstacklocal:2181,ydavis-hdp-nifi-test-1.openstacklocal:2181,ydavis-hdp-nifi-test-2.openstacklocal:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2),
it appears that the JDBC driver does not properly build the the uri in the
expected format. This is because HS2 is storing JDBC parameters in ZK
(https://issues.apache.org/jira/browse/HIVE-11581) and it is expecting the
driver to be able to parse and use those values to configure the connection.
However it appears the driver is expecting zookeeper to simply return the
host:port and subsequently building an invalid URI.
This problem has result in two variation of errors. The following was
experienced by [~mattyb149]
{noformat}
2016-08-15 12:45:12,918 INFO [Timer-Driven Process Thread-2]
org.apache.hive.jdbc.Utils Resolved authority:
hive.server2.authentication=KERBEROS;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=hdp-cluster-2-2.novalocal;hive.server2.thrift.port=10000;hive.server2.use.SSL=false;hive.server2.authentication.kerberos.principal=hive/_h...@hdf.com
2016-08-15 12:45:13,835 INFO [Timer-Driven Process Thread-2]
org.apache.hive.jdbc.HiveConnection Will try to open client transport with JDBC
Uri:
jdbc:hive2://hive.server2.authentication=KERBEROS;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=hdp-cluster-2-2.novalocal;hive.server2.thrift.port=10000;hive.server2.use.SSL=false;hive.server2.authentication.kerberos.principal=hive/_h...@hdf.com/default;principal=hive/_h...@hdf.com;serviceDiscoveryMode=zookeeper;zooKeeperNamespace=hiveserver2
2016-08-15 12:45:13,835 INFO [Timer-Driven Process Thread-2]
org.apache.hive.jdbc.HiveConnection Could not open client transport with JDBC
Uri:
jdbc:hive2://hive.server2.authentication=KERBEROS;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=hdp-cluster-2-2.novalocal;hive.server2.thrift.port=10000;hive.server2.use.SSL=false;hive.server2.authentication.kerberos.principal=hive/_h...@hdf.com/default;principal=hive/_h...@hdf.com;serviceDiscoveryMode=zookeeper;zooKeeperNamespace=hiveserver2
2016-08-15 12:45:13,836 INFO [Timer-Driven Process Thread-2]
o.a.c.f.imps.CuratorFrameworkImpl Starting
2016-08-15 12:45:14,064 INFO [Timer-Driven Process Thread-2-EventThread]
o.a.c.f.state.ConnectionStateManager State change: CONNECTED
2016-08-15 12:45:14,182 INFO [Curator-Framework-0]
o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting
2016-08-15 12:45:14,337 ERROR [Timer-Driven Process Thread-2]
o.a.nifi.processors.hive.SelectHiveQL
SelectHiveQL[id=7aaffd71-0156-1000-d962-8102c06b23df]
SelectHiveQL[id=7aaffd71-0156-1000-d962-8102c06b23df] failed to process due to
java.lang.reflect.UndeclaredThrowableException; rolling back session:
java.lang.reflect.UndeclaredThrowableException
2016-08-15 12:45:14,346 ERROR [Timer-Driven Process Thread-2]
o.a.nifi.processors.hive.SelectHiveQL
java.lang.reflect.UndeclaredThrowableException: null
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
~[na:na]
at
org.apache.nifi.dbcp.hive.HiveConnectionPool.getConnection(HiveConnectionPool.java:255)
~[na:na]
at sun.reflect.GeneratedMethodAccessor331.invoke(Unknown Source)
~[na:na]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[na:1.8.0_65]
at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_65]
at
org.apache.nifi.controller.service.StandardControllerServiceProvider$1.invoke(StandardControllerServiceProvider.java:174)
~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at com.sun.proxy.$Proxy81.getConnection(Unknown Source) ~[na:na]
at
org.apache.nifi.processors.hive.SelectHiveQL.onTrigger(SelectHiveQL.java:158)
~[na:na]
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
~[nifi-api-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1060)
[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127)
[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_65]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[na:1.8.0_65]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[na:1.8.0_65]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[na:1.8.0_65]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_65]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_65]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot create
PoolableConnectionFactory (Could not open client transport for any of the
Server URI's in ZooKeeper: Unable to read HiveServer2 uri from ZooKeeper)
at
org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1549)
~[na:na]
{noformat}
The following error was experienced by [~YolandaMDavis]
{noformat}
2016-08-15 19:22:27,338 INFO [Timer-Driven Process Thread-7]
org.apache.hive.jdbc.Utils Supplied authorities:
ydavis-hdp-nifi-test-3.openstacklocal:2181,ydavis-hdp-nifi-test-1.openstacklocal:2181,ydavis-hdp-nifi-test-2.openstacklocal:2181
2016-08-15 19:22:27,340 INFO [Timer-Driven Process Thread-7]
o.a.c.f.imps.CuratorFrameworkImpl Starting
2016-08-15 19:22:27,360 INFO [Timer-Driven Process Thread-7-EventThread]
o.a.c.f.state.ConnectionStateManager State change: CONNECTED
2016-08-15 19:22:27,365 INFO [Timer-Driven Process Thread-7]
o.a.hive.jdbc.ZooKeeperHiveClientHelper Selected HiveServer2 instance with uri:
hive.server2.authentication=NONE;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=ydavis-hdp-nifi-test-2.openstacklocal;hive.server2.thrift.port=10000;hive.server2.use.SSL=false
2016-08-15 19:22:27,365 INFO [Curator-Framework-0]
o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting
2016-08-15 19:22:27,371 INFO [Timer-Driven Process Thread-7]
org.apache.hive.jdbc.Utils Resolved authority:
hive.server2.authentication=NONE;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=ydavis-hdp-nifi-test-2.openstacklocal;hive.server2.thrift.port=10000;hive.server2.use.SSL=false
2016-08-15 19:22:27,374 INFO [Timer-Driven Process Thread-7]
org.apache.hive.jdbc.HiveConnection Will try to open client transport with JDBC
Uri:
jdbc:hive2://hive.server2.authentication=NONE;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=ydavis-hdp-nifi-test-2.openstacklocal;hive.server2.thrift.port=10000;hive.server2.use.SSL=false/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
2016-08-15 19:22:27,374 ERROR [Timer-Driven Process Thread-7]
o.apache.nifi.processors.hive.PutHiveQL
PutHiveQL[id=74c17a11-0156-1000-5d7e-40ed6aa18cbb]
PutHiveQL[id=74c17a11-0156-1000-5d7e-40ed6aa18cbb] failed to process due to
java.lang.NullPointerException; rolling back session:
java.lang.NullPointerException
2016-08-15 19:22:27,383 ERROR [Timer-Driven Process Thread-7]
o.apache.nifi.processors.hive.PutHiveQL
java.lang.NullPointerException: null
at org.apache.thrift.transport.TSocket.open(TSocket.java:170) ~[na:na]
at
org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:266)
~[na:na]
at
org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
~[na:na]
at
org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:204)
~[na:na]
at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:176)
~[na:na]
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105) ~[na:na]
at
org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)
~[na:na]
at
org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
~[na:na]
at
org.apache.commons.dbcp.BasicDataSource.validateConnectionFactory(BasicDataSource.java:1556)
~[na:na]
at
org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1545)
~[na:na]
at
org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1388)
~[na:na]
at
org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
~[na:na]
at
org.apache.nifi.dbcp.hive.HiveConnectionPool.getConnection(HiveConnectionPool.java:264)
~[na:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[na:1.8.0_101]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[na:1.8.0_101]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[na:1.8.0_101]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_101]
at
org.apache.nifi.controller.service.StandardControllerServiceProvider$1.invoke(StandardControllerServiceProvider.java:174)
~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at com.sun.proxy.$Proxy120.getConnection(Unknown Source) ~[na:na]
at
org.apache.nifi.processors.hive.PutHiveQL.onTrigger(PutHiveQL.java:152) ~[na:na]
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
~[nifi-api-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1060)
[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127)
[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_101]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[na:1.8.0_101]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[na:1.8.0_101]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[na:1.8.0_101]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_101]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_101]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
2016-08-15 19:22:27,384 ERROR [Timer-Driven Process Thread-7]
o.apache.nifi.processors.hive.PutHiveQL
PutHiveQL[id=74c17a11-0156-1000-5d7e-40ed6aa18cbb]
PutHiveQL[id=74c17a11-0156-1000-5d7e-40ed6aa18cbb] failed to process session
due to java.lang.NullPointerException: java.lang.NullPointerException
{noformat}
This problem appears to be resolved in the Apache 1.3 release which is not yet
available.
was:
When configuring a HiveQL processor using the Zookeeper URL (e.g.
jdbc:hive2://ydavis-hdp-nifi-test-3.openstacklocal:2181,ydavis-hdp-nifi-test-1.openstacklocal:2181,ydavis-hdp-nifi-test-2.openstacklocal:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2),
it appears that the JDBC driver does not properly build the the uri in the
expected format. This is because HS2 is storing JDBC parameters in ZK
(https://issues.apache.org/jira/browse/HIVE-11581) and it is expecting the
driver to be able to parse and use those values to configure the connection.
However it appears the driver is expecting zookeeper to simply return the
host:port and subsequently building an invalid URI.
This problem has result in two variation of errors. The following was
experienced by [~mattyb149]
{noformat}
2016-08-15 12:36:14,883 INFO [Timer-Driven Process Thread-5]
org.apache.hive.jdbc.HiveConnection Could not open client transport with JDBC
Uri:
jdbc:hive2://hdp-cluster-2-2.novalocal:10000/default;hive.server2.authentication=KERBEROS;hive.serve...;serviceDiscoveryMode=zookeeper;zooKeeperNamespace=hiveserver2
2016-08-15 12:36:16,019 INFO [Timer-Driven Process Thread-5]
o.a.c.f.imps.CuratorFrameworkImpl Starting
2016-08-15 12:36:16,054 INFO [Write-Ahead Local State Provider Maintenance]
org.wali.MinimalLockingWriteAheadLog
org.wali.MinimalLockingWriteAheadLog@434c179e checkpointed with 2 Records and 0
Swap Files in 1170 milliseconds (Stop-the-world time = 1159 milliseconds, Clear
Edit Logs time = 8 millis), max Transaction ID 5
2016-08-15 12:36:16,335 INFO [Timer-Driven Process Thread-5-EventThread]
o.a.c.f.state.ConnectionStateManager State change: CONNECTED
2016-08-15 12:36:16,368 INFO [pool-16-thread-1]
org.wali.MinimalLockingWriteAheadLog
org.wali.MinimalLockingWriteAheadLog@1caf77f6 checkpointed with 0 Records and 0
Swap Files in 1484 milliseconds (Stop-the-world time = 1438 milliseconds, Clear
Edit Logs time = 45 millis), max Transaction ID -1
2016-08-15 12:36:16,368 INFO [pool-16-thread-1]
o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile
Repository with 0 records in 1484 milliseconds
2016-08-15 12:36:16,457 INFO [Curator-Framework-0]
o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting
2016-08-15 12:36:16,588 ERROR [Timer-Driven Process Thread-5]
o.a.nifi.processors.hive.SelectHiveQL
SelectHiveQL[id=7aaffd71-0156-1000-d962-8102c06b23df]
SelectHiveQL[id=7aaffd71-0156-1000-d962-8102c06b23df] failed to process due to
java.lang.reflect.UndeclaredThrowableException; rolling back session:
java.lang.reflect.UndeclaredThrowableException
2016-08-15 12:36:16,596 ERROR [Timer-Driven Process Thread-5]
o.a.nifi.processors.hive.SelectHiveQL
java.lang.reflect.UndeclaredThrowableException: null
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
~[na:na]
at
org.apache.nifi.dbcp.hive.HiveConnectionPool.getConnection(HiveConnectionPool.java:255)
~[na:na]
at sun.reflect.GeneratedMethodAccessor331.invoke(Unknown Source)
~[na:na]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[na:1.8.0_65]
at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_65]
at
org.apache.nifi.controller.service.StandardControllerServiceProvider$1.invoke(StandardControllerServiceProvider.java:174)
~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at com.sun.proxy.$Proxy81.getConnection(Unknown Source) ~[na:na]
at
org.apache.nifi.processors.hive.SelectHiveQL.onTrigger(SelectHiveQL.java:158)
~[na:na]
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
~[nifi-api-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1060)
[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127)
[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_65]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[na:1.8.0_65]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[na:1.8.0_65]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[na:1.8.0_65]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_65]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_65]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot create
PoolableConnectionFactory (Could not open client transport for any of the
Server URI's in ZooKeeper: Unable to read HiveServer2 uri from ZooKeeper)
at
org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1549)
~[na:na]
at
org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1388)
~[na:na]
at
org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
~[na:na]
at
org.apache.nifi.dbcp.hive.HiveConnectionPool$1.run(HiveConnectionPool.java:258)
~[na:na]
at
org.apache.nifi.dbcp.hive.HiveConnectionPool$1.run(HiveConnectionPool.java:255)
~[na:na]
at java.security.AccessController.doPrivileged(Native Method)
~[na:1.8.0_65]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_65]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
~[na:na]
... 19 common frames omitted
Caused by: java.sql.SQLException: Could not open client transport for any of
the Server URI's in ZooKeeper: Unable to read HiveServer2 uri from ZooKeeper
{noformat}
The following error was experienced by [~YolandaMDavis]
{noformat}
2016-08-15 19:22:27,338 INFO [Timer-Driven Process Thread-7]
org.apache.hive.jdbc.Utils Supplied authorities:
ydavis-hdp-nifi-test-3.openstacklocal:2181,ydavis-hdp-nifi-test-1.openstacklocal:2181,ydavis-hdp-nifi-test-2.openstacklocal:2181
2016-08-15 19:22:27,340 INFO [Timer-Driven Process Thread-7]
o.a.c.f.imps.CuratorFrameworkImpl Starting
2016-08-15 19:22:27,360 INFO [Timer-Driven Process Thread-7-EventThread]
o.a.c.f.state.ConnectionStateManager State change: CONNECTED
2016-08-15 19:22:27,365 INFO [Timer-Driven Process Thread-7]
o.a.hive.jdbc.ZooKeeperHiveClientHelper Selected HiveServer2 instance with uri:
hive.server2.authentication=NONE;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=ydavis-hdp-nifi-test-2.openstacklocal;hive.server2.thrift.port=10000;hive.server2.use.SSL=false
2016-08-15 19:22:27,365 INFO [Curator-Framework-0]
o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting
2016-08-15 19:22:27,371 INFO [Timer-Driven Process Thread-7]
org.apache.hive.jdbc.Utils Resolved authority:
hive.server2.authentication=NONE;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=ydavis-hdp-nifi-test-2.openstacklocal;hive.server2.thrift.port=10000;hive.server2.use.SSL=false
2016-08-15 19:22:27,374 INFO [Timer-Driven Process Thread-7]
org.apache.hive.jdbc.HiveConnection Will try to open client transport with JDBC
Uri:
jdbc:hive2://hive.server2.authentication=NONE;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=ydavis-hdp-nifi-test-2.openstacklocal;hive.server2.thrift.port=10000;hive.server2.use.SSL=false/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
2016-08-15 19:22:27,374 ERROR [Timer-Driven Process Thread-7]
o.apache.nifi.processors.hive.PutHiveQL
PutHiveQL[id=74c17a11-0156-1000-5d7e-40ed6aa18cbb]
PutHiveQL[id=74c17a11-0156-1000-5d7e-40ed6aa18cbb] failed to process due to
java.lang.NullPointerException; rolling back session:
java.lang.NullPointerException
2016-08-15 19:22:27,383 ERROR [Timer-Driven Process Thread-7]
o.apache.nifi.processors.hive.PutHiveQL
java.lang.NullPointerException: null
at org.apache.thrift.transport.TSocket.open(TSocket.java:170) ~[na:na]
at
org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:266)
~[na:na]
at
org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
~[na:na]
at
org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:204)
~[na:na]
at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:176)
~[na:na]
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105) ~[na:na]
at
org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)
~[na:na]
at
org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
~[na:na]
at
org.apache.commons.dbcp.BasicDataSource.validateConnectionFactory(BasicDataSource.java:1556)
~[na:na]
at
org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1545)
~[na:na]
at
org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1388)
~[na:na]
at
org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
~[na:na]
at
org.apache.nifi.dbcp.hive.HiveConnectionPool.getConnection(HiveConnectionPool.java:264)
~[na:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[na:1.8.0_101]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[na:1.8.0_101]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[na:1.8.0_101]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_101]
at
org.apache.nifi.controller.service.StandardControllerServiceProvider$1.invoke(StandardControllerServiceProvider.java:174)
~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at com.sun.proxy.$Proxy120.getConnection(Unknown Source) ~[na:na]
at
org.apache.nifi.processors.hive.PutHiveQL.onTrigger(PutHiveQL.java:152) ~[na:na]
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
~[nifi-api-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1060)
[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127)
[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_101]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[na:1.8.0_101]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[na:1.8.0_101]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[na:1.8.0_101]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_101]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_101]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
2016-08-15 19:22:27,384 ERROR [Timer-Driven Process Thread-7]
o.apache.nifi.processors.hive.PutHiveQL
PutHiveQL[id=74c17a11-0156-1000-5d7e-40ed6aa18cbb]
PutHiveQL[id=74c17a11-0156-1000-5d7e-40ed6aa18cbb] failed to process session
due to java.lang.NullPointerException: java.lang.NullPointerException
{noformat}
This problem appears to be resolved in the Apache 1.3 release which is not yet
available.
> HiveQL Processors Fail due to invalid JDBC URI resolution when using
> Zookeeper URI
> ----------------------------------------------------------------------------------
>
> Key: NIFI-2575
> URL: https://issues.apache.org/jira/browse/NIFI-2575
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Reporter: Yolanda M. Davis
> Fix For: 1.0.0
>
>
> When configuring a HiveQL processor using the Zookeeper URL (e.g.
> jdbc:hive2://ydavis-hdp-nifi-test-3.openstacklocal:2181,ydavis-hdp-nifi-test-1.openstacklocal:2181,ydavis-hdp-nifi-test-2.openstacklocal:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2),
> it appears that the JDBC driver does not properly build the the uri in the
> expected format. This is because HS2 is storing JDBC parameters in ZK
> (https://issues.apache.org/jira/browse/HIVE-11581) and it is expecting the
> driver to be able to parse and use those values to configure the connection.
> However it appears the driver is expecting zookeeper to simply return the
> host:port and subsequently building an invalid URI.
> This problem has result in two variation of errors. The following was
> experienced by [~mattyb149]
> {noformat}
> 2016-08-15 12:45:12,918 INFO [Timer-Driven Process Thread-2]
> org.apache.hive.jdbc.Utils Resolved authority:
> hive.server2.authentication=KERBEROS;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=hdp-cluster-2-2.novalocal;hive.server2.thrift.port=10000;hive.server2.use.SSL=false;hive.server2.authentication.kerberos.principal=hive/_h...@hdf.com
> 2016-08-15 12:45:13,835 INFO [Timer-Driven Process Thread-2]
> org.apache.hive.jdbc.HiveConnection Will try to open client transport with
> JDBC Uri:
> jdbc:hive2://hive.server2.authentication=KERBEROS;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=hdp-cluster-2-2.novalocal;hive.server2.thrift.port=10000;hive.server2.use.SSL=false;hive.server2.authentication.kerberos.principal=hive/_h...@hdf.com/default;principal=hive/_h...@hdf.com;serviceDiscoveryMode=zookeeper;zooKeeperNamespace=hiveserver2
> 2016-08-15 12:45:13,835 INFO [Timer-Driven Process Thread-2]
> org.apache.hive.jdbc.HiveConnection Could not open client transport with JDBC
> Uri:
> jdbc:hive2://hive.server2.authentication=KERBEROS;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=hdp-cluster-2-2.novalocal;hive.server2.thrift.port=10000;hive.server2.use.SSL=false;hive.server2.authentication.kerberos.principal=hive/_h...@hdf.com/default;principal=hive/_h...@hdf.com;serviceDiscoveryMode=zookeeper;zooKeeperNamespace=hiveserver2
> 2016-08-15 12:45:13,836 INFO [Timer-Driven Process Thread-2]
> o.a.c.f.imps.CuratorFrameworkImpl Starting
> 2016-08-15 12:45:14,064 INFO [Timer-Driven Process Thread-2-EventThread]
> o.a.c.f.state.ConnectionStateManager State change: CONNECTED
> 2016-08-15 12:45:14,182 INFO [Curator-Framework-0]
> o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting
> 2016-08-15 12:45:14,337 ERROR [Timer-Driven Process Thread-2]
> o.a.nifi.processors.hive.SelectHiveQL
> SelectHiveQL[id=7aaffd71-0156-1000-d962-8102c06b23df]
> SelectHiveQL[id=7aaffd71-0156-1000-d962-8102c06b23df] failed to process due
> to java.lang.reflect.UndeclaredThrowableException; rolling back session:
> java.lang.reflect.UndeclaredThrowableException
> 2016-08-15 12:45:14,346 ERROR [Timer-Driven Process Thread-2]
> o.a.nifi.processors.hive.SelectHiveQL
> java.lang.reflect.UndeclaredThrowableException: null
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> ~[na:na]
> at
> org.apache.nifi.dbcp.hive.HiveConnectionPool.getConnection(HiveConnectionPool.java:255)
> ~[na:na]
> at sun.reflect.GeneratedMethodAccessor331.invoke(Unknown
> Source) ~[na:na]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.8.0_65]
> at java.lang.reflect.Method.invoke(Method.java:497)
> ~[na:1.8.0_65]
> at
> org.apache.nifi.controller.service.StandardControllerServiceProvider$1.invoke(StandardControllerServiceProvider.java:174)
> ~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at com.sun.proxy.$Proxy81.getConnection(Unknown Source) ~[na:na]
> at
> org.apache.nifi.processors.hive.SelectHiveQL.onTrigger(SelectHiveQL.java:158)
> ~[na:na]
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> ~[nifi-api-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1060)
> [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
> [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127)
> [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_65]
> at
> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_65]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> [na:1.8.0_65]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> [na:1.8.0_65]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_65]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_65]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
> Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot create
> PoolableConnectionFactory (Could not open client transport for any of the
> Server URI's in ZooKeeper: Unable to read HiveServer2 uri from ZooKeeper)
> at
> org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1549)
> ~[na:na]
> {noformat}
> The following error was experienced by [~YolandaMDavis]
> {noformat}
> 2016-08-15 19:22:27,338 INFO [Timer-Driven Process Thread-7]
> org.apache.hive.jdbc.Utils Supplied authorities:
> ydavis-hdp-nifi-test-3.openstacklocal:2181,ydavis-hdp-nifi-test-1.openstacklocal:2181,ydavis-hdp-nifi-test-2.openstacklocal:2181
> 2016-08-15 19:22:27,340 INFO [Timer-Driven Process Thread-7]
> o.a.c.f.imps.CuratorFrameworkImpl Starting
> 2016-08-15 19:22:27,360 INFO [Timer-Driven Process Thread-7-EventThread]
> o.a.c.f.state.ConnectionStateManager State change: CONNECTED
> 2016-08-15 19:22:27,365 INFO [Timer-Driven Process Thread-7]
> o.a.hive.jdbc.ZooKeeperHiveClientHelper Selected HiveServer2 instance with
> uri:
> hive.server2.authentication=NONE;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=ydavis-hdp-nifi-test-2.openstacklocal;hive.server2.thrift.port=10000;hive.server2.use.SSL=false
> 2016-08-15 19:22:27,365 INFO [Curator-Framework-0]
> o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting
> 2016-08-15 19:22:27,371 INFO [Timer-Driven Process Thread-7]
> org.apache.hive.jdbc.Utils Resolved authority:
> hive.server2.authentication=NONE;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=ydavis-hdp-nifi-test-2.openstacklocal;hive.server2.thrift.port=10000;hive.server2.use.SSL=false
> 2016-08-15 19:22:27,374 INFO [Timer-Driven Process Thread-7]
> org.apache.hive.jdbc.HiveConnection Will try to open client transport with
> JDBC Uri:
> jdbc:hive2://hive.server2.authentication=NONE;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=ydavis-hdp-nifi-test-2.openstacklocal;hive.server2.thrift.port=10000;hive.server2.use.SSL=false/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
> 2016-08-15 19:22:27,374 ERROR [Timer-Driven Process Thread-7]
> o.apache.nifi.processors.hive.PutHiveQL
> PutHiveQL[id=74c17a11-0156-1000-5d7e-40ed6aa18cbb]
> PutHiveQL[id=74c17a11-0156-1000-5d7e-40ed6aa18cbb] failed to process due to
> java.lang.NullPointerException; rolling back session:
> java.lang.NullPointerException
> 2016-08-15 19:22:27,383 ERROR [Timer-Driven Process Thread-7]
> o.apache.nifi.processors.hive.PutHiveQL
> java.lang.NullPointerException: null
> at org.apache.thrift.transport.TSocket.open(TSocket.java:170) ~[na:na]
> at
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:266)
> ~[na:na]
> at
> org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
> ~[na:na]
> at
> org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:204)
> ~[na:na]
> at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:176)
> ~[na:na]
> at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105) ~[na:na]
> at
> org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)
> ~[na:na]
> at
> org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
> ~[na:na]
> at
> org.apache.commons.dbcp.BasicDataSource.validateConnectionFactory(BasicDataSource.java:1556)
> ~[na:na]
> at
> org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1545)
> ~[na:na]
> at
> org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1388)
> ~[na:na]
> at
> org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
> ~[na:na]
> at
> org.apache.nifi.dbcp.hive.HiveConnectionPool.getConnection(HiveConnectionPool.java:264)
> ~[na:na]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[na:1.8.0_101]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[na:1.8.0_101]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.8.0_101]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_101]
> at
> org.apache.nifi.controller.service.StandardControllerServiceProvider$1.invoke(StandardControllerServiceProvider.java:174)
> ~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at com.sun.proxy.$Proxy120.getConnection(Unknown Source) ~[na:na]
> at
> org.apache.nifi.processors.hive.PutHiveQL.onTrigger(PutHiveQL.java:152)
> ~[na:na]
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> ~[nifi-api-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1060)
> [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
> [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127)
> [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_101]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> [na:1.8.0_101]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> [na:1.8.0_101]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> [na:1.8.0_101]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_101]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_101]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
> 2016-08-15 19:22:27,384 ERROR [Timer-Driven Process Thread-7]
> o.apache.nifi.processors.hive.PutHiveQL
> PutHiveQL[id=74c17a11-0156-1000-5d7e-40ed6aa18cbb]
> PutHiveQL[id=74c17a11-0156-1000-5d7e-40ed6aa18cbb] failed to process session
> due to java.lang.NullPointerException: java.lang.NullPointerException
> {noformat}
> This problem appears to be resolved in the Apache 1.3 release which is not
> yet available.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
