[
https://issues.apache.org/jira/browse/HIVE-22041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alessandro Di Diego updated HIVE-22041:
---------------------------------------
Description:
It seems that in secure clusters, HiveServer2 creates an unbounded amount of
crc files related to the Delegation Token, e.g.:
{quote}{{# ls -latr .hive_hadoop_delegation_token*.tmp.crc | more}}
{{-rw-r--r-- 1 hive hive 12 Jul 22 18:41
.hive_hadoop_delegation_token7336761305415229297.tmp.crc}}
{{-rw-r--r-- 1 hive hive 12 Jul 22 18:41
.hive_hadoop_delegation_token2098027444797756507.tmp.crc}}
{{-rw-r--r-- 1 hive hive 12 Jul 22 18:41
.hive_hadoop_delegation_token570615377517838929.tmp.crc}}
{{-rw-r--r-- 1 hive hive 12 Jul 22 18:41
.hive_hadoop_delegation_token2806157469169711507.tmp.crc}}
{{-rw-r--r-- 1 hive hive 12 Jul 22 18:41
.hive_hadoop_delegation_token4743780849236152782.tmp.crc}}
{{-rw-r--r-- 1 hive hive 12 Jul 22 18:41
.hive_hadoop_delegation_token4779276962989484605.tmp.crc}}
{{-rw-r--r-- 1 hive hive 12 Jul 22 18:41
.hive_hadoop_delegation_token614278562135964419.tmp.crc}}
{{-rw-r--r-- 1 hive hive 12 Jul 22 18:41
.hive_hadoop_delegation_token5163699580858054526.tmp.crc}}
{{-rw-r--r-- 1 hive hive 12 Jul 22 18:41
.hive_hadoop_delegation_token6764054090932425303.tmp.crc}}
{{-rw-r--r-- 1 hive hive 12 Jul 22 18:42
.hive_hadoop_delegation_token6389909203128084522.tmp.crc}}
{{-rw-r--r-- 1 hive hive 12 Jul 22 18:42
.hive_hadoop_delegation_token7810958545651446366.tmp.crc}}
{{[...]}}
{quote}
this could quickly fill the inode table of the filesystem (usually the one
mounted on /tmp).
I've experienced it in Hive 1.1.0 (CDH 5.16.2), but I think the same issue is
present in the current master branch, since here:
[https://github.com/apache/hive/blob/d1343a69e6be7a312b7c0bb2aeebefaa40535b65/ql/src/java/org/apache/hadoop/hive/ql/exec/SecureCmdDoAs.java#L71]
the crc file gets created, but only the token file gets deleted:
[https://github.com/apache/hive/blob/d1343a69e6be7a312b7c0bb2aeebefaa40535b65/ql/src/java/org/apache/hadoop/hive/ql/exec/SecureCmdDoAs.java#L80]
HIVE-13883 greatly relates to the issue; I think that the same logic could be
used here to fix this, although I wonder if the Credentials class should also
expose some kind of delete method that cleans every file it has produced...
was:
It seems that in secure clusters, HiveServer2 creates an unbounded amount of
crc files related to the Delegation Token, e.g.:
{quote}{{# ls -latr .hive_hadoop_delegation_token*.tmp.crc | more}}
{{-rw-r--r-- 1 hive hive 12 Jul 22 18:41
.hive_hadoop_delegation_token7336761305415229297.tmp.crc}}
{{-rw-r--r-- 1 hive hive 12 Jul 22 18:41
.hive_hadoop_delegation_token2098027444797756507.tmp.crc}}
{{-rw-r--r-- 1 hive hive 12 Jul 22 18:41
.hive_hadoop_delegation_token570615377517838929.tmp.crc}}
{{-rw-r--r-- 1 hive hive 12 Jul 22 18:41
.hive_hadoop_delegation_token2806157469169711507.tmp.crc}}
{{-rw-r--r-- 1 hive hive 12 Jul 22 18:41
.hive_hadoop_delegation_token4743780849236152782.tmp.crc}}
{{-rw-r--r-- 1 hive hive 12 Jul 22 18:41
.hive_hadoop_delegation_token4779276962989484605.tmp.crc}}
{{-rw-r--r-- 1 hive hive 12 Jul 22 18:41
.hive_hadoop_delegation_token614278562135964419.tmp.crc}}
{{-rw-r--r-- 1 hive hive 12 Jul 22 18:41
.hive_hadoop_delegation_token5163699580858054526.tmp.crc}}
{{-rw-r--r-- 1 hive hive 12 Jul 22 18:41
.hive_hadoop_delegation_token6764054090932425303.tmp.crc}}
{{-rw-r--r-- 1 hive hive 12 Jul 22 18:42
.hive_hadoop_delegation_token6389909203128084522.tmp.crc}}
{{-rw-r--r-- 1 hive hive 12 Jul 22 18:42
.hive_hadoop_delegation_token7810958545651446366.tmp.crc}}
{{[...]}}
{quote}
this could quickly fill the inode table of the filesystem (usually the one
mounted on /tmp).
I've experienced it in Hive 1.1.0 (CDH 5.16.2), but I think the same issue is
present in the current master branch, since here:
[https://github.com/apache/hive/blob/d1343a69e6be7a312b7c0bb2aeebefaa40535b65/ql/src/java/org/apache/hadoop/hive/ql/exec/SecureCmdDoAs.java#L71]
the crc file gets created, but only the token file gets deleted:
[https://github.com/apache/hive/blob/d1343a69e6be7a312b7c0bb2aeebefaa40535b65/ql/src/java/org/apache/hadoop/hive/ql/exec/SecureCmdDoAs.java#L80]
HIVE-13883 is greatly related to the issue, I think that the same logic could
be used to fix this.
Although, I wonder if the Credentials class should expose some delete method
that also deletes the crc file...
> HiveServer2 creates Delegation Token crc files and never deletes them
> ---------------------------------------------------------------------
>
> Key: HIVE-22041
> URL: https://issues.apache.org/jira/browse/HIVE-22041
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
> Affects Versions: 1.1.0
> Reporter: Alessandro Di Diego
> Priority: Minor
>
> It seems that in secure clusters, HiveServer2 creates an unbounded amount of
> crc files related to the Delegation Token, e.g.:
> {quote}{{# ls -latr .hive_hadoop_delegation_token*.tmp.crc | more}}
> {{-rw-r--r-- 1 hive hive 12 Jul 22 18:41
> .hive_hadoop_delegation_token7336761305415229297.tmp.crc}}
> {{-rw-r--r-- 1 hive hive 12 Jul 22 18:41
> .hive_hadoop_delegation_token2098027444797756507.tmp.crc}}
> {{-rw-r--r-- 1 hive hive 12 Jul 22 18:41
> .hive_hadoop_delegation_token570615377517838929.tmp.crc}}
> {{-rw-r--r-- 1 hive hive 12 Jul 22 18:41
> .hive_hadoop_delegation_token2806157469169711507.tmp.crc}}
> {{-rw-r--r-- 1 hive hive 12 Jul 22 18:41
> .hive_hadoop_delegation_token4743780849236152782.tmp.crc}}
> {{-rw-r--r-- 1 hive hive 12 Jul 22 18:41
> .hive_hadoop_delegation_token4779276962989484605.tmp.crc}}
> {{-rw-r--r-- 1 hive hive 12 Jul 22 18:41
> .hive_hadoop_delegation_token614278562135964419.tmp.crc}}
> {{-rw-r--r-- 1 hive hive 12 Jul 22 18:41
> .hive_hadoop_delegation_token5163699580858054526.tmp.crc}}
> {{-rw-r--r-- 1 hive hive 12 Jul 22 18:41
> .hive_hadoop_delegation_token6764054090932425303.tmp.crc}}
> {{-rw-r--r-- 1 hive hive 12 Jul 22 18:42
> .hive_hadoop_delegation_token6389909203128084522.tmp.crc}}
> {{-rw-r--r-- 1 hive hive 12 Jul 22 18:42
> .hive_hadoop_delegation_token7810958545651446366.tmp.crc}}
> {{[...]}}
> {quote}
> this could quickly fill the inode table of the filesystem (usually the one
> mounted on /tmp).
> I've experienced it in Hive 1.1.0 (CDH 5.16.2), but I think the same issue is
> present in the current master branch, since here:
> [https://github.com/apache/hive/blob/d1343a69e6be7a312b7c0bb2aeebefaa40535b65/ql/src/java/org/apache/hadoop/hive/ql/exec/SecureCmdDoAs.java#L71]
> the crc file gets created, but only the token file gets deleted:
> [https://github.com/apache/hive/blob/d1343a69e6be7a312b7c0bb2aeebefaa40535b65/ql/src/java/org/apache/hadoop/hive/ql/exec/SecureCmdDoAs.java#L80]
> HIVE-13883 greatly relates to the issue; I think that the same logic could be
> used here to fix this, although I wonder if the Credentials class should also
> expose some kind of delete method that cleans every file it has produced...
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
