[
https://issues.apache.org/jira/browse/NIFI-2575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yolanda M. Davis updated NIFI-2575:
-----------------------------------
Summary: HiveQL Processors Fails due to invalid JDBC URI resolution when
using Zookeeper URI (was: HiveQL Processor Fails due to invalid JDBC URI
resolution when using Zookeeper URI)
> HiveQL Processors Fails due to invalid JDBC URI resolution when using
> Zookeeper URI
> -----------------------------------------------------------------------------------
>
> Key: NIFI-2575
> URL: https://issues.apache.org/jira/browse/NIFI-2575
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Reporter: Yolanda M. Davis
> Fix For: 1.0.0
>
>
> When configuring a HiveQL processor using the Zookeeper URL (e.g.
> jdbc:hive2://ydavis-hdp-nifi-test-3.openstacklocal:2181,ydavis-hdp-nifi-test-1.openstacklocal:2181,ydavis-hdp-nifi-test-2.openstacklocal:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2),
> it appears that the JDBC driver does not properly build the the uri in the
> expected format. This is because HS2 is storing JDBC parameters in ZK
> (https://issues.apache.org/jira/browse/HIVE-11581) and it is expecting the
> driver to be able to parse and use those values to configure the connection.
> However it appears the driver is expecting zookeeper to simply return the
> host:port and subsequently building an invalid URI.
> This problem has result in two variation of errors. The following was
> experienced by [~mattyb149]
> {noformat}
> 2016-08-15 12:36:14,883 INFO [Timer-Driven Process Thread-5]
> org.apache.hive.jdbc.HiveConnection Could not open client transport with JDBC
> Uri:
> jdbc:hive2://hdp-cluster-2-2.novalocal:10000/default;hive.server2.authentication=KERBEROS;hive.serve...;serviceDiscoveryMode=zookeeper;zooKeeperNamespace=hiveserver2
> 2016-08-15 12:36:16,019 INFO [Timer-Driven Process Thread-5]
> o.a.c.f.imps.CuratorFrameworkImpl Starting
> 2016-08-15 12:36:16,054 INFO [Write-Ahead Local State Provider Maintenance]
> org.wali.MinimalLockingWriteAheadLog
> org.wali.MinimalLockingWriteAheadLog@434c179e checkpointed with 2 Records and
> 0 Swap Files in 1170 milliseconds (Stop-the-world time = 1159 milliseconds,
> Clear Edit Logs time = 8 millis), max Transaction ID 5
> 2016-08-15 12:36:16,335 INFO [Timer-Driven Process Thread-5-EventThread]
> o.a.c.f.state.ConnectionStateManager State change: CONNECTED
> 2016-08-15 12:36:16,368 INFO [pool-16-thread-1]
> org.wali.MinimalLockingWriteAheadLog
> org.wali.MinimalLockingWriteAheadLog@1caf77f6 checkpointed with 0 Records and
> 0 Swap Files in 1484 milliseconds (Stop-the-world time = 1438 milliseconds,
> Clear Edit Logs time = 45 millis), max Transaction ID -1
> 2016-08-15 12:36:16,368 INFO [pool-16-thread-1]
> o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile
> Repository with 0 records in 1484 milliseconds
> 2016-08-15 12:36:16,457 INFO [Curator-Framework-0]
> o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting
> 2016-08-15 12:36:16,588 ERROR [Timer-Driven Process Thread-5]
> o.a.nifi.processors.hive.SelectHiveQL
> SelectHiveQL[id=7aaffd71-0156-1000-d962-8102c06b23df]
> SelectHiveQL[id=7aaffd71-0156-1000-d962-8102c06b23df] failed to process due
> to java.lang.reflect.UndeclaredThrowableException; rolling back session:
> java.lang.reflect.UndeclaredThrowableException
> 2016-08-15 12:36:16,596 ERROR [Timer-Driven Process Thread-5]
> o.a.nifi.processors.hive.SelectHiveQL
> java.lang.reflect.UndeclaredThrowableException: null
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> ~[na:na]
> at
> org.apache.nifi.dbcp.hive.HiveConnectionPool.getConnection(HiveConnectionPool.java:255)
> ~[na:na]
> at sun.reflect.GeneratedMethodAccessor331.invoke(Unknown Source)
> ~[na:na]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.8.0_65]
> at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_65]
> at
> org.apache.nifi.controller.service.StandardControllerServiceProvider$1.invoke(StandardControllerServiceProvider.java:174)
> ~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at com.sun.proxy.$Proxy81.getConnection(Unknown Source) ~[na:na]
> at
> org.apache.nifi.processors.hive.SelectHiveQL.onTrigger(SelectHiveQL.java:158)
> ~[na:na]
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> ~[nifi-api-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1060)
> [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
> [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127)
> [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_65]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> [na:1.8.0_65]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> [na:1.8.0_65]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> [na:1.8.0_65]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_65]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_65]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
> Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot create
> PoolableConnectionFactory (Could not open client transport for any of the
> Server URI's in ZooKeeper: Unable to read HiveServer2 uri from ZooKeeper)
> at
> org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1549)
> ~[na:na]
> at
> org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1388)
> ~[na:na]
> at
> org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
> ~[na:na]
> at
> org.apache.nifi.dbcp.hive.HiveConnectionPool$1.run(HiveConnectionPool.java:258)
> ~[na:na]
> at
> org.apache.nifi.dbcp.hive.HiveConnectionPool$1.run(HiveConnectionPool.java:255)
> ~[na:na]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[na:1.8.0_65]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_65]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
> ~[na:na]
> ... 19 common frames omitted
> Caused by: java.sql.SQLException: Could not open client transport for any of
> the Server URI's in ZooKeeper: Unable to read HiveServer2 uri from ZooKeeper
> {noformat}
> The following error was experienced by [~YolandaMDavis]
> {noformat}
> 2016-08-15 19:22:27,338 INFO [Timer-Driven Process Thread-7]
> org.apache.hive.jdbc.Utils Supplied authorities:
> ydavis-hdp-nifi-test-3.openstacklocal:2181,ydavis-hdp-nifi-test-1.openstacklocal:2181,ydavis-hdp-nifi-test-2.openstacklocal:2181
> 2016-08-15 19:22:27,340 INFO [Timer-Driven Process Thread-7]
> o.a.c.f.imps.CuratorFrameworkImpl Starting
> 2016-08-15 19:22:27,360 INFO [Timer-Driven Process Thread-7-EventThread]
> o.a.c.f.state.ConnectionStateManager State change: CONNECTED
> 2016-08-15 19:22:27,365 INFO [Timer-Driven Process Thread-7]
> o.a.hive.jdbc.ZooKeeperHiveClientHelper Selected HiveServer2 instance with
> uri:
> hive.server2.authentication=NONE;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=ydavis-hdp-nifi-test-2.openstacklocal;hive.server2.thrift.port=10000;hive.server2.use.SSL=false
> 2016-08-15 19:22:27,365 INFO [Curator-Framework-0]
> o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting
> 2016-08-15 19:22:27,371 INFO [Timer-Driven Process Thread-7]
> org.apache.hive.jdbc.Utils Resolved authority:
> hive.server2.authentication=NONE;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=ydavis-hdp-nifi-test-2.openstacklocal;hive.server2.thrift.port=10000;hive.server2.use.SSL=false
> 2016-08-15 19:22:27,374 INFO [Timer-Driven Process Thread-7]
> org.apache.hive.jdbc.HiveConnection Will try to open client transport with
> JDBC Uri:
> jdbc:hive2://hive.server2.authentication=NONE;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=ydavis-hdp-nifi-test-2.openstacklocal;hive.server2.thrift.port=10000;hive.server2.use.SSL=false/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
> 2016-08-15 19:22:27,374 ERROR [Timer-Driven Process Thread-7]
> o.apache.nifi.processors.hive.PutHiveQL
> PutHiveQL[id=74c17a11-0156-1000-5d7e-40ed6aa18cbb]
> PutHiveQL[id=74c17a11-0156-1000-5d7e-40ed6aa18cbb] failed to process due to
> java.lang.NullPointerException; rolling back session:
> java.lang.NullPointerException
> 2016-08-15 19:22:27,383 ERROR [Timer-Driven Process Thread-7]
> o.apache.nifi.processors.hive.PutHiveQL
> java.lang.NullPointerException: null
> at org.apache.thrift.transport.TSocket.open(TSocket.java:170) ~[na:na]
> at
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:266)
> ~[na:na]
> at
> org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
> ~[na:na]
> at
> org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:204)
> ~[na:na]
> at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:176)
> ~[na:na]
> at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105) ~[na:na]
> at
> org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)
> ~[na:na]
> at
> org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
> ~[na:na]
> at
> org.apache.commons.dbcp.BasicDataSource.validateConnectionFactory(BasicDataSource.java:1556)
> ~[na:na]
> at
> org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1545)
> ~[na:na]
> at
> org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1388)
> ~[na:na]
> at
> org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
> ~[na:na]
> at
> org.apache.nifi.dbcp.hive.HiveConnectionPool.getConnection(HiveConnectionPool.java:264)
> ~[na:na]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[na:1.8.0_101]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[na:1.8.0_101]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.8.0_101]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_101]
> at
> org.apache.nifi.controller.service.StandardControllerServiceProvider$1.invoke(StandardControllerServiceProvider.java:174)
> ~[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at com.sun.proxy.$Proxy120.getConnection(Unknown Source) ~[na:na]
> at
> org.apache.nifi.processors.hive.PutHiveQL.onTrigger(PutHiveQL.java:152)
> ~[na:na]
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> ~[nifi-api-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1060)
> [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
> [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127)
> [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_101]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> [na:1.8.0_101]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> [na:1.8.0_101]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> [na:1.8.0_101]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_101]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_101]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
> 2016-08-15 19:22:27,384 ERROR [Timer-Driven Process Thread-7]
> o.apache.nifi.processors.hive.PutHiveQL
> PutHiveQL[id=74c17a11-0156-1000-5d7e-40ed6aa18cbb]
> PutHiveQL[id=74c17a11-0156-1000-5d7e-40ed6aa18cbb] failed to process session
> due to java.lang.NullPointerException: java.lang.NullPointerException
> {noformat}
> This problem appears to be resolved in the Apache 1.3 release which is not
> yet available.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
