[jira] [Comment Edited] (SPARK-11265) YarnClient can't get tokens to talk to Hive 1.2.1 in a secure cluster
[
https://issues.apache.org/jira/browse/SPARK-11265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14974239#comment-14974239
]
Steve Loughran edited comment on SPARK-11265 at 10/26/15 2:08 PM:
--
What's changed?
The spark code uses reflection to get the method
{{("org.apache.hadoop.hive.ql.metadata.Hive#get")}}, then invokes it with a
single argument: {{hive = hiveClass.getMethod("get").invoke(null,
hiveConf.asInstanceOf[Object])}}
Hive 0.13 has >1 method; it has, in order, {{get(HiveConf}}, {{get(HiveConf,
boolean)}}, and {{get()}}.
Hive 1.2.1 adds one new method {{get(Configuration c, Class clazz)}}
*before* the others, and now invoke is failing as the returned method doesn't
take a HiveConf.
What could have been happening here is that the {{Class.get()}} method was
returning the {{get(HiveConf}} method because it was first in the file, and on
1.2.1 the new method returned the new one, which didn't take a single
{{HiveConf}}, hence the stack trace
The fix, under all of it, is simply getting the method {{get(HiveConf.class)}},
and invoking it with the configuration created by reflection. That's all:
explicitly asking for a method that's always been there. The code probably
worked before just because nobody was looking at it.
was (Author: ste...@apache.org):
What's changed?
The spark code uses reflection to get the method
{{("org.apache.hadoop.hive.ql.metadata.Hive#get")}}, then invokes it with a
single argument: {{hive = hiveClass.getMethod("get").invoke(null,
hiveConf.asInstanceOf[Object])}}
Hive 0.13 has >1 method with this name, even in Hive 0.31.1; it has, in order,
{{get(HiveConf}}, {{get(HiveConf, boolean)}}, and {{get()}}.
Hive 1.2.1 adds one new method {{get(Configuration c, Class clazz)}}
*before* the others, and now invoke is failing as the returned method doesn't
take a HiveConf.
What could have been happening here is that the {{Class.get()}} method was
returning the {{get(HiveConf}} method because it was first in the file, and on
1.2.1 the new method returned the new one, which didn't take a single
{{HiveConf}}, hence the stack trace
The fix, under all of it, is simply getting the method {{get(HiveConf.class)}},
and invoking it with the configuration created by reflection. That's all:
explicitly asking for a method that's always been there. The code probably
worked before just because nobody was looking at it.
> YarnClient can't get tokens to talk to Hive 1.2.1 in a secure cluster
> -
>
> Key: SPARK-11265
> URL: https://issues.apache.org/jira/browse/SPARK-11265
> Project: Spark
> Issue Type: Bug
> Components: YARN
>Affects Versions: 1.5.1
> Environment: Kerberized Hadoop cluster
>Reporter: Steve Loughran
>
> As reported on the dev list, trying to run a YARN client which wants to talk
> to Hive in a Kerberized hadoop cluster fails.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-11265) YarnClient can't get tokens to talk to Hive 1.2.1 in a secure cluster
[
https://issues.apache.org/jira/browse/SPARK-11265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14974239#comment-14974239
]
Steve Loughran edited comment on SPARK-11265 at 10/26/15 2:08 PM:
--
What's changed?
The spark code uses reflection to get the method
{{("org.apache.hadoop.hive.ql.metadata.Hive#get")}}, then invokes it with a
single argument: {{hive = hiveClass.getMethod("get").invoke(null,
hiveConf.asInstanceOf[Object])}}
Hive 0.13 has >1 method with this name, even in Hive 0.31.1; it has, in order,
{{get(HiveConf}}, {{get(HiveConf, boolean)}}, and {{get()}}.
Hive 1.2.1 adds one new method {{get(Configuration c, Class clazz)}}
*before* the others, and now invoke is failing as the returned method doesn't
take a HiveConf.
What could have been happening here is that the {{Class.get()}} method was
returning the {{get(HiveConf}} method because it was first in the file, and on
1.2.1 the new method returned the new one, which didn't take a single
{{HiveConf}}, hence the stack trace
The fix, under all of it, is simply getting the method {{get(HiveConf.class)}},
and invoking it with the configuration created by reflection. That's all:
explicitly asking for a method that's always been there. The code probably
worked before just because nobody was looking at it.
was (Author: ste...@apache.org):
What's changed?
The spark code uses reflection to get the method
{{("org.apache.hadoop.hive.ql.metadata.Hive#get"), then invokes it with a
single argument: {{hive = hiveClass.getMethod("get").invoke(null,
hiveConf.asInstanceOf[Object])}}
Hive 0.13 has >1 method with this name, even in Hive 0.31.1; it has, in order,
{{get(HiveConf}}, {{get(HiveConf, boolean)}}, and {{get()}}.
Hive 1.2.1 adds one new method {{get(Configuration c, Class clazz)}}
*before* the others, and now invoke is failing as the returned method doesn't
take a HiveConf.
What could have been happening here is that the {{Class.get()}} method was
returning the {{get(HiveConf}} method because it was first in the file, and on
1.2.1 the new method returned the new one, which didn't take a single
{{HiveConf}}, hence the stack trace
The fix, under all of it, is simply getting the method {{get(HiveConf.class)}},
and invoking it with the configuration created by reflection. That's all:
explicitly asking for a method that's always been there. The code probably
worked before just because nobody was looking at it.
> YarnClient can't get tokens to talk to Hive 1.2.1 in a secure cluster
> -
>
> Key: SPARK-11265
> URL: https://issues.apache.org/jira/browse/SPARK-11265
> Project: Spark
> Issue Type: Bug
> Components: YARN
>Affects Versions: 1.5.1
> Environment: Kerberized Hadoop cluster
>Reporter: Steve Loughran
>
> As reported on the dev list, trying to run a YARN client which wants to talk
> to Hive in a Kerberized hadoop cluster fails.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
