Are you able to write to Hive to an existing partition? (The stack trace shows
that it’s being created)
From: Federico D'Ambrosio <fedex...@gmail.com>
Reply-To: "dev@hive.apache.org" <dev@hive.apache.org>
Date: Monday, July 10, 2017 at 7:38 AM
To: "u...@hive.apache.org" <u...@hive.apache.org>, "dev@hive.apache.org"
<dev@hive.apache.org>
Subject: Non-local session path expected to be non-null trying to write on Hive
using storm-hive
Greetings,
I'mtrying to get a working dataflow stack on a 6 node cluster (2 masters + 4
slaves, no Kerberos) using Kafka (2.10_0.10), Storm (1.0.1) and Hive2 (1.2.1).
Storm is able to communicate with Kafka, but can't seemingly operate on Hive
(on master-1), even though it manages to connect to its metastore.
I thought originally it was a problem of permissions on either HDFS or the
local filesystem, but even though I set 777 permissions on /tmp/hive, there's
still this issue.
In core-site.xml:
* hadoop.proxyuser.hcat.group
· hadoop.proxyuser.hcat.hosts
* hadoop.proxyuser.hdfs.groups
* hadoop.proxyuser.hdfs.hosts
* hadoop.proxyuser.hive.groups
* hadoop.proxyuser.hive.hosts
* hadoop.proxyuser.root.groups
* hadoop.proxyuser.root.hosts
are all set to '*'.
Hive2, as far as I see is correctly set to work with transactions, being the
target table with transactional=true, stored as orc and bucketed. In the
hive-site.xml:
* hive.compactor.worker.threads = 1
* hive.compactor.initiator.on = true
* hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManager
I get a Nullpointer Exception, you may find the stack trace among the attached
files.
From what I can gather, the NullpointerException is thrown in the following
method inside SessionState:
1. public static Path getHDFSSessionPath(Configuration conf) {
2. SessionState ss = SessionState.get();
3. if (ss == null) {
4. String sessionPathString = conf.get(HDFS_SESSION_PATH_KEY);
5. Preconditions.checkNotNull(sessionPathString, "Conf non-local
session path expected to be non-null");
6. return new Path(sessionPathString);
7. }
8. Preconditions.checkNotNull(ss.hdfsSessionPath, "Non-local session path
expected to be non-null");
9. return ss.hdfsSessionPath;
10.}
Specifically, by:
1. Preconditions.checkNotNull(ss.hdfsSessionPath, "Non-local session path
expected to be non-null");
So, it seems to be an hdfs related issue, but I can't understand why it's
happening.
From what I gather, this occurs when Hive tries to retrieve the local path of
the session, which is stored in the _hive.local.session.path configuration
variable. The value of this variable is assigned each time a new Hive session
is created, and it is formed by merging the path for user temporary files
(hive.exec.local.scratchdir) to the session ID
(hive.session.id<http://hive.session.id>).
If indeed is a permissions issue, what should I look into to find the origin of
the issue?
Thanks for your help,
Federico
