[GitHub] zeppelin issue #1157: [ZEPPELIN-1146] Zeppelin JDBC interpreter should work ...
Github user rja1 commented on the issue:
https://github.com/apache/zeppelin/pull/1157
Thanks for the tip @prabhjyotsingh and for your work, hive is working!
I'm running into an issue with jdbc(phoenix) now though, hoping you can
help. My interpreter config is listed below. Note that we don't have a
/hbase-secure dir in zookeeper, just hbase.
[zk: localhost:2181(CONNECTED) 1] ls /hbase
[replication, meta-region-server, rs, splitWAL, backup-masters, table-lock,
flush-table-proc, region-in-transition, online-snapshot, acl, master, running,
balancer, tokenauth, recovering-regions, draining, namespace, hbaseid, table]
Here's the notebook / error:
select * from USER_ACCOUNTS where USER_SEED = '1000'
Failed after attempts=1, exceptions:
Mon Jul 18 14:00:22 MDT 2016,
RpcRetryingCaller{globalStartTime=1468872022128, pause=100, retries=1},
org.apache.hadoop.hbase.MasterNotRunningException:
com.google.protobuf.ServiceException:
org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to
name01.hadoop.test.company.com/10.4.59.25:6 failed on local exception:
org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to
name01.hadoop.test.company.com/10.4.59.25:6 is closing. Call id=0,
waitTime=11
class org.apache.phoenix.exception.PhoenixIOException
org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:111)
org.apache.phoenix.query.ConnectionQueryServicesImpl.ensureTableCreated(ConnectionQueryServicesImpl.java:1064)
org.apache.phoenix.query.ConnectionQueryServicesImpl.createTable(ConnectionQueryServicesImpl.java:1370)
org.apache.phoenix.schema.MetaDataClient.createTableInternal(MetaDataClient.java:2116)
org.apache.phoenix.schema.MetaDataClient.createTable(MetaDataClient.java:828)
org.apache.phoenix.compile.CreateTableCompiler$2.execute(CreateTableCompiler.java:183)
org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:338)
org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:326)
org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:324)
org.apache.phoenix.jdbc.PhoenixStatement.executeUpdate(PhoenixStatement.java:1326)
org.apache.phoenix.query.ConnectionQueryServicesImpl$13.call(ConnectionQueryServicesImpl.java:2275)
org.apache.phoenix.query.ConnectionQueryServicesImpl$13.call(ConnectionQueryServicesImpl.java:2244)
org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:78)
org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:2244)
org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:233)
org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.createConnection(PhoenixEmbeddedDriver.java:135)
org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:202)
java.sql.DriverManager.getConnection(DriverManager.java:664)
java.sql.DriverManager.getConnection(DriverManager.java:208)
org.apache.zeppelin.jdbc.JDBCInterpreter.getConnection(JDBCInterpreter.java:226)
org.apache.zeppelin.jdbc.JDBCInterpreter.getStatement(JDBCInterpreter.java:237)
org.apache.zeppelin.jdbc.JDBCInterpreter.executeSql(JDBCInterpreter.java:296)
org.apache.zeppelin.jdbc.JDBCInterpreter.interpret(JDBCInterpreter.java:402)
org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:94)
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:383)
org.apache.zeppelin.scheduler.Job.run(Job.java:176)
org.apache.zeppelin.scheduler.ParallelScheduler$JobRunner.run(ParallelScheduler.java:162)
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
Interpreter Config:
"2BRGRRCBW": {
"id": "2BRGRRCBW",
"name": "jdbc",
"group": "jdbc",
"properties": {
"phoenix.user": "zeppelin",
"hive.url":
"jdbc:hive2://cms01.hadoop.test.company.com:1/default;principal\u003dhive/_h...@hadoop.test.company.com",
"default.driver": "org.postgresql.Driver",
"phoenix.driver": "org.apache.phoenix.jdbc.PhoenixDriver",
"hive.user": "hive",
"psql.password": "",
"psql.user": "phoenixuser",
[GitHub] zeppelin issue #1157: [ZEPPELIN-1146] Zeppelin JDBC interpreter should work ...
Github user jongyoul commented on the issue: https://github.com/apache/zeppelin/pull/1157 @prabhjyotsingh Thanks for checking it. I think we should fix all of flaky tests --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1157: [ZEPPELIN-1146] Zeppelin JDBC interpreter should work ...
Github user prabhjyotsingh commented on the issue: https://github.com/apache/zeppelin/pull/1157 @jongyoul Sure let me check. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1157: [ZEPPELIN-1146] Zeppelin JDBC interpreter should work ...
Github user jongyoul commented on the issue: https://github.com/apache/zeppelin/pull/1157 @prabhjyotsingh This PR is not passed. Why do you merge it? I think we need to pass the CI even though we have some flaky tests. Could you please check it again? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1157: [ZEPPELIN-1146] Zeppelin JDBC interpreter should work ...
Github user prabhjyotsingh commented on the issue: https://github.com/apache/zeppelin/pull/1157 You can either use %hive or %jdbc(hive), for the same. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1157: [ZEPPELIN-1146] Zeppelin JDBC interpreter should work ...
Github user rja1 commented on the issue: https://github.com/apache/zeppelin/pull/1157 Thanks @prabhjyotsingh. Novice question: how to you make the distinction in the notebook that you want jdbc hive vs jdbc postgresql, jdbc phoenix, etc.. If I run: %jdbc show tables; Zeppelin defaults to trying to connect to postgres (as that's the default driver, which makes sense). I'm guessing you're doing something like: %jdbc.hive? show tables; --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1157: [ZEPPELIN-1146] Zeppelin JDBC interpreter should work ...
Github user prabhjyotsingh commented on the issue:
https://github.com/apache/zeppelin/pull/1157
@rja1 Sure, attaching both screenshot and json. I have tested this for hive
and phoenix.
https://cloud.githubusercontent.com/assets/674497/16776893/11020f28-4885-11e6-878b-d715561efca3.png;>
```
{
"id": "2BQPN9U57",
"name": "jdbc",
"group": "jdbc",
"properties": {
"phoenix.user": "phoenixuser",
"hive.url":
"jdbc:hive2://prabhu-zeppelin-secure-1.novalocal:2181,prabhu-zeppelin-secure-2.novalocal:2181,prabhu-zeppelin-secure-4.novalocal:2181/;serviceDiscoveryMode\u003dzooKeeper;zooKeeperNamespace\u003dhiveserver2",
"psql.password": "",
"default.driver": "org.postgresql.Driver",
"phoenix.driver":
"org.apache.phoenix.jdbc.PhoenixDriver",
"hive.user": "hive",
"psql.user": "phoenixuser",
"psql.url": "jdbc:postgresql://localhost:5432/",
"default.user": "gpadmin",
"phoenix.hbase.client.retries.number": "1",
"phoenix.url":
"jdbc:phoenix:prabhu-zeppelin-secure-1.novalocal,prabhu-zeppelin-secure-2.novalocal,prabhu-zeppelin-secure-4.novalocal:/hbase-secure",
"tajo.url": "jdbc:tajo://localhost:26002/default",
"tajo.driver": "org.apache.tajo.jdbc.TajoDriver",
"psql.driver": "org.postgresql.Driver",
"default.password": "",
"zeppelin.interpreter.localRepo":
"/usr/hdp/current/zeppelin-server/local-repo/2BQPN9U57",
"zeppelin.jdbc.auth.type": "KERBEROS",
"hive.password": "",
"zeppelin.jdbc.concurrent.use": "true",
"hive.driver": "org.apache.hive.jdbc.HiveDriver",
"zeppelin.jdbc.keytab.location":
"/etc/security/keytabs/zeppelin.server.kerberos.keytab",
"common.max_count": "1000",
"phoenix.password": "",
"zeppelin.jdbc.principal":
"zeppelin/secu...@example.com",
"zeppelin.jdbc.concurrent.max_connection": "10",
"default.url": "jdbc:postgresql://localhost:5432/"
},
"interpreterGroup": [{
"class": "org.apache.zeppelin.jdbc.JDBCInterpreter",
"name": "sql"
}],
"dependencies": [{
"groupArtifactVersion":
"org.apache.hive:hive-jdbc:2.0.1",
"local": false
}, {
"groupArtifactVersion":
"org.apache.hadoop:hadoop-common:2.7.2",
"local": false
}, {
"groupArtifactVersion":
"org.apache.hive.shims:hive-shims-0.23:2.1.0",
"local": false
}, {
"groupArtifactVersion":
"org.apache.phoenix:phoenix-core:4.4.0-HBase-1.1",
"local": false
}],
"option": {
"remote": true,
"perNoteSession": false,
"perNoteProcess": false,
"isExistingProcess": false
}
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin issue #1157: [ZEPPELIN-1146] Zeppelin JDBC interpreter should work ...
Github user rja1 commented on the issue: https://github.com/apache/zeppelin/pull/1157 @prabhjyotsingh would you mind sharing your jdbc interpreter settings please? I must be missing something.. Thanks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1157: [ZEPPELIN-1146] Zeppelin JDBC interpreter should work ...
Github user prabhjyotsingh commented on the issue: https://github.com/apache/zeppelin/pull/1157 Yes fair point, let me do a quick test and do it right away. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1157: [ZEPPELIN-1146] Zeppelin JDBC interpreter should work ...
Github user rconline commented on the issue: https://github.com/apache/zeppelin/pull/1157 @jongyoul i think this suggestion makes sense. Its possible that other jdbc data sources may have security setup via ACL and don't need Kerberos. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1157: [ZEPPELIN-1146] Zeppelin JDBC interpreter should work ...
Github user jongyoul commented on the issue: https://github.com/apache/zeppelin/pull/1157 And It's not necessary to have hadoop dependencies in case not to depend on hadoop environment like mysql and postresql. I think we need different way to deal with it. I think it's better to change the scope of those dependencies as provided and Zeppelin handles it with reflection. In case that you use Hive, you already have a dependencies of hadoop-common thus it's OK, otherwise, JdbcInterpreter doesn't need to have hadoop-common. What do you think of my idea? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1157: [ZEPPELIN-1146] Zeppelin JDBC interpreter should work ...
Github user jongyoul commented on the issue: https://github.com/apache/zeppelin/pull/1157 Doesn't it become potential problem to set hadoop version as 2.7.2? Is there a way to fit the version of hadoop globally? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1157: [ZEPPELIN-1146] Zeppelin JDBC interpreter should work ...
Github user prabhjyotsingh commented on the issue: https://github.com/apache/zeppelin/pull/1157 Merging this if no more discussion. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1157: [ZEPPELIN-1146] Zeppelin JDBC interpreter should work ...
Github user rconlinehwx commented on the issue: https://github.com/apache/zeppelin/pull/1157 LGTM. @jongyoul Typically I use an ambari created 3 node cluster and then follow these instructions to setup my kerberos - https://docs.hortonworks.com/HDPDocuments/Ambari-2.2.1.1/bk_Ambari_Security_Guide/content/ch_configuring_amb_hdp_for_kerberos.html --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1157: [ZEPPELIN-1146] Zeppelin JDBC interpreter should work ...
Github user prabhjyotsingh commented on the issue: https://github.com/apache/zeppelin/pull/1157 @jongyoul i couldn't figure out a easier way, than having a actual kerberos environment, and then testing it there. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1157: [ZEPPELIN-1146] Zeppelin JDBC interpreter should work ...
Github user jongyoul commented on the issue: https://github.com/apache/zeppelin/pull/1157 @prabhjyotsingh Changes look good, but do you guide me to this PR with my local machine? Does it have easy way to test it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
