[jira] [Commented] (HIVE-10100) Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0
[
https://issues.apache.org/jira/browse/HIVE-10100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15362914#comment-15362914
]
Prasanth Jayachandran commented on HIVE-10100:
--
The test failures look related. I will look at it shortly.
> Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0
> --
>
> Key: HIVE-10100
> URL: https://issues.apache.org/jira/browse/HIVE-10100
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Gunther Hagleitner
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-10100.1.patch, HIVE-10100.2.patch,
> HIVE-10100.3.patch, yarn_bin.patch
>
>
> HADOOP-11257 adds a warning to stdout
> {noformat}
> WARNING: Use "yarn jar" to launch YARN applications.
> {noformat}
> which will cause issues if untreated with folks that programatically parse
> stdout for query results (i.e.: CLI, silent mode, etc).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10100) Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0
[
https://issues.apache.org/jira/browse/HIVE-10100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360201#comment-15360201
]
Hive QA commented on HIVE-10100:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12815803/HIVE-10100.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 28 failed/errored test(s), 10287 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_merge_stats_orc
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_archive_excludeHadoop20
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_fetch_aggregation
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nonmr_fetch
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_analyze
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook_use_metadata
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorized_parquet
org.apache.hadoop.hive.cli.TestContribCliDriver.testCliDriver_dboutput
org.apache.hadoop.hive.cli.TestContribNegativeCliDriver.testNegativeCliDriver_case_with_row_sequence
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_join_hash
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_join_hash
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1
org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapPlan1
org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapPlan2
org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan1
org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan2
org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan3
org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan4
org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan5
org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan6
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/349/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/349/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-349/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 28 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12815803 - PreCommit-HIVE-MASTER-Build
> Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0
> --
>
> Key: HIVE-10100
> URL: https://issues.apache.org/jira/browse/HIVE-10100
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Gunther Hagleitner
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-10100.1.patch, HIVE-10100.2.patch,
> HIVE-10100.3.patch, yarn_bin.patch
>
>
> HADOOP-11257 adds a warning to stdout
> {noformat}
> WARNING: Use "yarn jar" to launch YARN applications.
> {noformat}
> which will cause issues if untreated with folks that programatically parse
> stdout for query results (i.e.: CLI, silent mode, etc).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10100) Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0
[
https://issues.apache.org/jira/browse/HIVE-10100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353371#comment-15353371
]
Gunther Hagleitner commented on HIVE-10100:
---
[~sseth] we can ask, but the warning has been around for a long time. My guess
eventually the idea is to get rid of it. So might as well change these places.
> Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0
> --
>
> Key: HIVE-10100
> URL: https://issues.apache.org/jira/browse/HIVE-10100
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Gunther Hagleitner
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-10100.1.patch, HIVE-10100.2.patch, yarn_bin.patch
>
>
> HADOOP-11257 adds a warning to stdout
> {noformat}
> WARNING: Use "yarn jar" to launch YARN applications.
> {noformat}
> which will cause issues if untreated with folks that programatically parse
> stdout for query results (i.e.: CLI, silent mode, etc).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10100) Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0
[
https://issues.apache.org/jira/browse/HIVE-10100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15352322#comment-15352322
]
Siddharth Seth commented on HIVE-10100:
---
Why is this critical? - due to the confusion caused by the error message
printed.
It may be worth asking for a way to suppress this specific message rather than
trying to change Hive code all over the place.
'yarn jar metastore' <- what does this mean ?
> Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0
> --
>
> Key: HIVE-10100
> URL: https://issues.apache.org/jira/browse/HIVE-10100
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Gunther Hagleitner
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-10100.1.patch, HIVE-10100.2.patch, yarn_bin.patch
>
>
> HADOOP-11257 adds a warning to stdout
> {noformat}
> WARNING: Use "yarn jar" to launch YARN applications.
> {noformat}
> which will cause issues if untreated with folks that programatically parse
> stdout for query results (i.e.: CLI, silent mode, etc).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10100) Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0
[
https://issues.apache.org/jira/browse/HIVE-10100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15352319#comment-15352319
]
Siddharth Seth commented on HIVE-10100:
---
hadoop jar from within Java code (instead of using the programatic API) ? - I'm
sure there's a good reason to do this :)
> Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0
> --
>
> Key: HIVE-10100
> URL: https://issues.apache.org/jira/browse/HIVE-10100
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Gunther Hagleitner
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-10100.1.patch, HIVE-10100.2.patch, yarn_bin.patch
>
>
> HADOOP-11257 adds a warning to stdout
> {noformat}
> WARNING: Use "yarn jar" to launch YARN applications.
> {noformat}
> which will cause issues if untreated with folks that programatically parse
> stdout for query results (i.e.: CLI, silent mode, etc).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10100) Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0
[
https://issues.apache.org/jira/browse/HIVE-10100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15352141#comment-15352141
]
Siddharth Seth commented on HIVE-10100:
---
YARN_USER_CLASSPATH needs to be set in a couple more scripts - beeline, hplsql,
hwi.
Other than that, looks good to me.
Don't really know why hadoop jar was deprecated in favor of yarn jar, and what
the difference between the two is.
> Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0
> --
>
> Key: HIVE-10100
> URL: https://issues.apache.org/jira/browse/HIVE-10100
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Gunther Hagleitner
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-10100.1.patch, HIVE-10100.2.patch
>
>
> HADOOP-11257 adds a warning to stdout
> {noformat}
> WARNING: Use "yarn jar" to launch YARN applications.
> {noformat}
> which will cause issues if untreated with folks that programatically parse
> stdout for query results (i.e.: CLI, silent mode, etc).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10100) Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0
[
https://issues.apache.org/jira/browse/HIVE-10100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15351931#comment-15351931
]
Prasanth Jayachandran commented on HIVE-10100:
--
bq.For HADOOP_CLASSPATH_FIRST and YARN_CLASSPATH_FIRST - should we use the
value set in HADOOP_CLASSPATH_FIRST or the one used in YARN_CLASSPATH_FIRST ?
I think we need this to be always true for proper initialization of log4j2 and
jline.
> Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0
> --
>
> Key: HIVE-10100
> URL: https://issues.apache.org/jira/browse/HIVE-10100
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Gunther Hagleitner
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-10100.1.patch
>
>
> HADOOP-11257 adds a warning to stdout
> {noformat}
> WARNING: Use "yarn jar" to launch YARN applications.
> {noformat}
> which will cause issues if untreated with folks that programatically parse
> stdout for query results (i.e.: CLI, silent mode, etc).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10100) Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0
[
https://issues.apache.org/jira/browse/HIVE-10100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15351929#comment-15351929
]
Prasanth Jayachandran commented on HIVE-10100:
--
bq. Why is this snippet used in some files, but not others ?
Some scripts explicitly modifies the variables after hive script sets it up to
append service specific info. Example: beeline seems to be doing some ordering
of classpath and metastore appends its own java opts which might not be
required otherwise.
I can't find YARN_CLASSPATH in yarn script. I think after some version
YARN_CLASSPATH is renamed to YARN_USER_CLASSPATH. So I added HADOOP_CLASSPATH
to YARN_USER_CLASSPATH in the next patch.
Yes. Hive script is slow for 2 reasons
1) Automatic initialization of logging (when trying to run hadoop or hbase
commands entire classpath is searched to look for log4j2.properties file)
2) hbase mapredcp takes time
There is some info here HIVE-12497. Regardless we should bring down the time.
> Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0
> --
>
> Key: HIVE-10100
> URL: https://issues.apache.org/jira/browse/HIVE-10100
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Gunther Hagleitner
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-10100.1.patch
>
>
> HADOOP-11257 adds a warning to stdout
> {noformat}
> WARNING: Use "yarn jar" to launch YARN applications.
> {noformat}
> which will cause issues if untreated with folks that programatically parse
> stdout for query results (i.e.: CLI, silent mode, etc).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10100) Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0
[
https://issues.apache.org/jira/browse/HIVE-10100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15347520#comment-15347520
]
Prasanth Jayachandran commented on HIVE-10100:
--
[~gopalv]/[~sseth] Can someone please review this patch?
> Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0
> --
>
> Key: HIVE-10100
> URL: https://issues.apache.org/jira/browse/HIVE-10100
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.3.0, 2.2.0
>Reporter: Gunther Hagleitner
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-10100.1.patch
>
>
> HADOOP-11257 adds a warning to stdout
> {noformat}
> WARNING: Use "yarn jar" to launch YARN applications.
> {noformat}
> which will cause issues if untreated with folks that programatically parse
> stdout for query results (i.e.: CLI, silent mode, etc).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10100) Warning yarn jar instead of hadoop jar in hadoop 2.7.0
[
https://issues.apache.org/jira/browse/HIVE-10100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14384348#comment-14384348
]
Gunther Hagleitner commented on HIVE-10100:
---
Thank you [~cnauroth]! Lowering priority. Will keep open to switch from hadoop
jar to yarn jar to get rid of the warning over time.
Warning yarn jar instead of hadoop jar in hadoop 2.7.0
--
Key: HIVE-10100
URL: https://issues.apache.org/jira/browse/HIVE-10100
Project: Hive
Issue Type: Bug
Reporter: Gunther Hagleitner
Priority: Critical
HADOOP-11257 adds a warning to stdout
{noformat}
WARNING: Use yarn jar to launch YARN applications.
{noformat}
which will cause issues if untreated with folks that programatically parse
stdout for query results (i.e.: CLI, silent mode, etc).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
