[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[
https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16160763#comment-16160763
]
Hadoop QA commented on HADOOP-13335:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
31s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
7s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
9s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
11s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 45s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HADOOP-13335 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12817244/HADOOP-13335.05.trunk.patch
|
| Optional Tests | asflicense mvnsite unit shellcheck shelldocs |
| uname | Linux dfb63748b806 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 722ee84 |
| shellcheck | v0.4.6 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13232/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13232/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add an option to suppress the 'use yarn jar' warning or remove it
> -
>
> Key: HADOOP-13335
> URL: https://issues.apache.org/jira/browse/HADOOP-13335
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 2.7.0
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HADOOP-13335.01.patch, HADOOP-13335.02_branch-2.patch,
> HADOOP-13335.02.patch, HADOOP-13335.03_branch-2.patch, HADOOP-13335.03.patch,
> HADOOP-13335.04.patch, HADOOP-13335.05.branch-2.patch,
> HADOOP-13335.05.trunk.patch
>
>
> https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation'
> warning for 'hadoop jar'.
> hadoop jar is used for a lot more that starting jobs. As an example - hive
> uses it to start all it's services (HiveServer2, the hive client, beeline
> etc).
> Using 'yarn jar' for to start these services / tools doesn't make a lot of
> sense - there's no relation to yarn other than requiring the classpath to
> include yarn libraries.
> I'd propose reverting the changes where this message is printed if YARN
> variables are set (leave it in the help message), or adding a mechanism which
> would allow users to suppress this WARNING.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[
https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946250#comment-15946250
]
Hadoop QA commented on HADOOP-13335:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
13s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
11s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
11s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 2s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-13335 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12817244/HADOOP-13335.05.trunk.patch
|
| Optional Tests | asflicense mvnsite unit shellcheck shelldocs |
| uname | Linux 2ee09b321099 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 063b513 |
| shellcheck | v0.4.5 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11986/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11986/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add an option to suppress the 'use yarn jar' warning or remove it
> -
>
> Key: HADOOP-13335
> URL: https://issues.apache.org/jira/browse/HADOOP-13335
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 2.7.0
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HADOOP-13335.01.patch, HADOOP-13335.02_branch-2.patch,
> HADOOP-13335.02.patch, HADOOP-13335.03_branch-2.patch, HADOOP-13335.03.patch,
> HADOOP-13335.04.patch, HADOOP-13335.05.branch-2.patch,
> HADOOP-13335.05.trunk.patch
>
>
> https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation'
> warning for 'hadoop jar'.
> hadoop jar is used for a lot more that starting jobs. As an example - hive
> uses it to start all it's services (HiveServer2, the hive client, beeline
> etc).
> Using 'yarn jar' for to start these services / tools doesn't make a lot of
> sense - there's no relation to yarn other than requiring the classpath to
> include yarn libraries.
> I'd propose reverting the changes where this message is printed if YARN
> variables are set (leave it in the help message), or adding a mechanism which
> would allow users to suppress this WARNING.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15375792#comment-15375792 ] Siddharth Seth commented on HADOOP-13335: - bq. a) this doesn't solve the user confusion problem. The message does more harm than good, and gets worse once users are pointed to yarn jar, and then told that YARN_* properties are deprecated in trunk. bq. b) there is already a way to disable the messages. By unsetting variables. yarn jar and hadoop jar not usable on the same shell. This is not a hive only problem. Please reconsider your -1, or suggest an alternate solution to solve this. I've already provided quite a few options. > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HADOOP-13335.01.patch, HADOOP-13335.02.patch, > HADOOP-13335.02_branch-2.patch, HADOOP-13335.03.patch, > HADOOP-13335.03_branch-2.patch, HADOOP-13335.04.patch, > HADOOP-13335.05.branch-2.patch, HADOOP-13335.05.trunk.patch > > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15371741#comment-15371741 ] Allen Wittenauer commented on HADOOP-13335: --- -1 : a) this doesn't solve the user confusion problem. b) there is already a way to disable the messages. > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HADOOP-13335.01.patch, HADOOP-13335.02.patch, > HADOOP-13335.02_branch-2.patch, HADOOP-13335.03.patch, > HADOOP-13335.03_branch-2.patch, HADOOP-13335.04.patch, > HADOOP-13335.05.branch-2.patch, HADOOP-13335.05.trunk.patch > > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[
https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15371620#comment-15371620
]
Hadoop QA commented on HADOOP-13335:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
28s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
13s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
8s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
40s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12817244/HADOOP-13335.05.trunk.patch
|
| JIRA Issue | HADOOP-13335 |
| Optional Tests | asflicense mvnsite unit shellcheck shelldocs |
| uname | Linux 7d405b27606a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 0fd3980 |
| shellcheck | v0.4.4 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9961/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9961/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add an option to suppress the 'use yarn jar' warning or remove it
> -
>
> Key: HADOOP-13335
> URL: https://issues.apache.org/jira/browse/HADOOP-13335
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 2.7.0
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HADOOP-13335.01.patch, HADOOP-13335.02.patch,
> HADOOP-13335.02_branch-2.patch, HADOOP-13335.03.patch,
> HADOOP-13335.03_branch-2.patch, HADOOP-13335.04.patch,
> HADOOP-13335.05.branch-2.patch, HADOOP-13335.05.trunk.patch
>
>
> https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation'
> warning for 'hadoop jar'.
> hadoop jar is used for a lot more that starting jobs. As an example - hive
> uses it to start all it's services (HiveServer2, the hive client, beeline
> etc).
> Using 'yarn jar' for to start these services / tools doesn't make a lot of
> sense - there's no relation to yarn other than requiring the classpath to
> include yarn libraries.
> I'd propose reverting the changes where this message is printed if YARN
> variables are set (leave it in the help message), or adding a mechanism which
> would allow users to suppress this WARNING.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366778#comment-15366778 ] Siddharth Seth commented on HADOOP-13335: - bq. I'm pretty much going to block anything that removes the warnings. So we can keep this open, but it isn't going to go anywhere. On what technical grounds ? Printing a warning like this on branch-2 can be considered incompatible. That's ignoring the bit that the warning itself doesn't necessarily make sense. In terms of deprecation - I already see the following in trunk. "hadoop_deprecate_envvar YARN_OPTS HADOOP_OPTS" - What exactly are we trying to achieve here, between the deprecation and the push for yarn jar via this message. > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HADOOP-13335.01.patch, HADOOP-13335.02.patch, > HADOOP-13335.02_branch-2.patch, HADOOP-13335.03.patch, > HADOOP-13335.03_branch-2.patch, HADOOP-13335.04.patch > > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366738#comment-15366738 ] Allen Wittenauer commented on HADOOP-13335: --- There's a resolution here: bq. In terms of Hive (for 2.x) - yes, an unset works I'm pretty much going to block anything that removes the warnings. So we can keep this open, but it isn't going to go anywhere. > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HADOOP-13335.01.patch, HADOOP-13335.02.patch, > HADOOP-13335.02_branch-2.patch, HADOOP-13335.03.patch, > HADOOP-13335.03_branch-2.patch, HADOOP-13335.04.patch > > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366571#comment-15366571 ] Siddharth Seth commented on HADOOP-13335: - bq. It's not and I'm still not sure why you seem to insist this is true when both Vinod and I have said that there isn't any intent on it going away anytime remotely soon. Anytime soon - no. From your comments, it does look like at least you think this is a candidate for 4.x removal. In terms of Hive (for 2.x) - yes, an unset works - for the most part anyway. However, this warning is not required - and should not be seen by other systems / users. Unsetting variables is a way to suppress the message, however it isn't an optimal or efficient way to do this. Is this deprecation, what's the public API? There's multiple options here. 1. Get rid of the warning altogether - which I think is the correct thing to do. (Haven't heard a good reason to keep it) 2. Add a variable to deprecate this specific warning. 3. Add an option to deprecate all such warnings (and apparently there's a lot of these in trunk). > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HADOOP-13335.01.patch, HADOOP-13335.02.patch, > HADOOP-13335.02_branch-2.patch, HADOOP-13335.03.patch, > HADOOP-13335.03_branch-2.patch, HADOOP-13335.04.patch > > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366048#comment-15366048 ] Allen Wittenauer commented on HADOOP-13335: --- bq. . If the intent is to deprecate the (yarn jar) command It's not and I'm still not sure why you seem to insist this is true when both Vinod and I have said that there isn't any intent on it going away anytime remotely soon. As I said before, if you don't want to use 'yarn jar', fine. But there is already a way to disable the message in branch-2 which is the only version Hive currently supports anyway. When Hive starts to work with trunk/3.x, feel free to file JIRAs regarding the new functionality. As for this one, I'm leaning towards closing it as Won't Fix given there is a way to suppress the message. > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HADOOP-13335.01.patch, HADOOP-13335.02.patch, > HADOOP-13335.02_branch-2.patch, HADOOP-13335.03.patch, > HADOOP-13335.03_branch-2.patch, HADOOP-13335.04.patch > > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365652#comment-15365652 ] Siddharth Seth commented on HADOOP-13335: - bq. There have been a ton of removals all over the place in trunk already. S3, hftp, ... lots and lots of places. Bear in mind that the last opportunity Hadoop had to remove content was over five years ago. Good to know. bq. Once again: they do not. I'm not sure how many more and different ways I can state this point. The rest of that discussion is pretty much moot since the first opportunity to make them function the same will be 4.x, easily years off. And once again - the only difference is the additional YARN_* environment variables added on. You seem to agree that having these additional variables is unnecessary and confusing, and they should eventually be removed (maybe in 4.x). If the intent is to deprecate the (yarn jar) command - I don't see the point of pointing users towards this when they invoke hadoop jar. Another way of looking at this: if someone has set the YARN_* options - they would do this intentionally, and would be aware of the yarn jar command. If they use hadoop jar after this point for something else - they know exactly what they're doing, and hadoop should not try pointing them back at yarn jar. bq. yarn-config.sh and hadoop-config.sh in a way that should be mostly conflict-free Right there is another problem. "Mostly conflict-free" is not a great API, and adds more confusion. bq. Deprecation warnings have always been added to the output throughout the history of Hadoop. Expect a lot more to show up in trunk. Is this particular case supposed to be a deprecation warning ? Most system will also have a mechanism to suppress deprecation messages; we should add such an option to the scripts in trunk. Is there some place that the public API for the hadoop scripts is documented ? The only place I have seen references along with documentation is within the scripts itself, and in 'usage' output. > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HADOOP-13335.01.patch, HADOOP-13335.02.patch, > HADOOP-13335.02_branch-2.patch, HADOOP-13335.03.patch, > HADOOP-13335.03_branch-2.patch, HADOOP-13335.04.patch > > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365418#comment-15365418 ] Allen Wittenauer commented on HADOOP-13335: --- bq. It is going to be hard to remove either of these. I don't know when something that has been deprecated in a previous release has actually been removed. There have been a ton of removals all over the place in trunk already. S3, hftp, ... lots and lots of places. Bear in mind that the last opportunity Hadoop had to remove content was over five years ago. bq. If hadoop jar and yarn jar should behave the same Once again: they do not. I'm not sure how many more and different ways I can state this point. The rest of that discussion is pretty much moot since the first opportunity to make them function the same will be 4.x, easily years off. bq. This is arguable - but printing a new warning in the 2.7.0 release can be considered to be an incompatible change. Deprecation warnings have always been added to the output throughout the history of Hadoop. Expect a lot more to show up in trunk. > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HADOOP-13335.01.patch, HADOOP-13335.02.patch, > HADOOP-13335.02_branch-2.patch, HADOOP-13335.03.patch, > HADOOP-13335.03_branch-2.patch, HADOOP-13335.04.patch > > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15364973#comment-15364973 ] Siddharth Seth commented on HADOOP-13335: - bq. It's going to be hard to take yarn jar or hadoop jar away. It's doubtful they will ever get removed. That said, we can at least make them act and work the same way. To me, that's the ultimate goal and it's pretty close to what happens in trunk: 1. yarn command sucks in yarn-env.sh, hadoop-env.sh, yarn-config.sh and hadoop-config.sh in a way that should be mostly conflict-free. (non-yarn commands do not pull in yarn-x.sh, obviously) 2. If YARN_OPTS is defined, yarn x (jar, rmadmin, etc) will use it but throw a deprecation warning. 3. Otherwise use HADOOP_OPTS It is going to be hard to remove either of these. I don't know when something that has been deprecated in a previous release has actually been removed. If hadoop jar and yarn jar should behave the same - and 1) 'yarn jar' is not the preferred usage, or 2) 'yarn jar' behaves as an alias - I'd be in favor of removing this warnings altogether. Don't encourage users to use yarn jar over hadoop jar/ don't advertise yarn jar. In terms of removing the warning - I'm all for it, and is the preferred approach to 'fix' this jar (that's the first suggestion in the description). This is arguable - but printing a new warning in the 2.7.0 release can be considered to be an incompatible change. Other than being an annoyance to users - Hive silent mode is broken by this, since it was not written to work with output from the hadoop command. Lets keep the change / detailed recommendation in the 'help' output - and remove it from hadoop jar invocation. bq. Very long term (post-3.x), it would probably be better if hive called hadoop-config.sh and/or hadoop-functions.sh directly. This would bypass the middleman and give much better control. I'd be very interested to hear what sort of holes we have in the functionality here that makes this hard/impossible. Off the top, I suspect we need to make one big function of the series of function calls in hadoop-config.sh, but would love to hear your insight on this. I don't know enough about the internals of the scripts to have an educated opinion on this. hadoop scripts, required environment variables, how they interact seems quite complicated. Setting HADOOP_CLIENT_OPTS is apparently the way to change the hiveclient heap size. I would expect hive to control this independently, and not depend on variables exported by hadoop scripts. One possible usage is for Hadoop to provide basic information - CLASSPATH, configs. Products like Hive build on top of this information, rather than trying to use hadoop scripts, and define their own mechanism for users to specify various environment variables. hadoop jar is obviously useful for custom tools, which want a simple way to execute on a hadoop cluster. > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HADOOP-13335.01.patch, HADOOP-13335.02.patch, > HADOOP-13335.02_branch-2.patch, HADOOP-13335.03.patch, > HADOOP-13335.03_branch-2.patch, HADOOP-13335.04.patch > > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15364885#comment-15364885 ] Allen Wittenauer commented on HADOOP-13335: --- bq. we are well past the point where we can remove "hadoop jar" or "yarn jar" anytime in the future. Totally agree and that was never really on the table. Long term, hadoop jar == yarn jar functionality-wise, just not in the 2.x/3.x timeframe. bq. How about we simply remove the warnings in "hadoop jar" (kind of how we 'undeprecated' the mapred API) and move forward? Because users don't understand that YARN_OPTS and HADOOP_OPTS are separate in branch-2. This whole situation started a while back because users were changing/setting YARN_OPTS and didn't understand why hadoop jar blah didn't pick up their changes when they were running YARN jobs. > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HADOOP-13335.01.patch, HADOOP-13335.02.patch, > HADOOP-13335.02_branch-2.patch, HADOOP-13335.03.patch, > HADOOP-13335.03_branch-2.patch, HADOOP-13335.04.patch > > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15364862#comment-15364862 ] Vinod Kumar Vavilapalli commented on HADOOP-13335: -- Folks, we are well past the point where we can remove "hadoop jar" or "yarn jar" anytime in the future. Our users know about them, and I'd not even think about deprecating / removing them in 3.x / 4.x or any up-version for that matter. How about we simply remove the warnings in "hadoop jar" (kind of how we 'undeprecated' the mapred API) and move forward? > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HADOOP-13335.01.patch, HADOOP-13335.02.patch, > HADOOP-13335.02_branch-2.patch, HADOOP-13335.03.patch, > HADOOP-13335.03_branch-2.patch, HADOOP-13335.04.patch > > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15364457#comment-15364457 ] Allen Wittenauer commented on HADOOP-13335: --- bq. Not yet - until someone feels the need to have hdfs used as a separate command to do something other than filesystem operations. (Really hope there's no hdfs jar command) Not on my watch. :) It's pretty clear from this JIRA just how much damage the extra YARN stuff has caused. (and this is just the tip of the iceberg...) Plus, between HADOOP-11485, HADOOP-12930, and a handful of other features, there are much better ways for 1st, 2nd, and 3rd parties to integrate extra stuff. \[1] bq. I don't see the difference between hadoop jar and yarn jar other than a different set of variables being set and respected by the different commands. That's correct. But those extra vars make a world of difference . In branch-2, HADOOP\_OPTS and YARN\_OPTS don't cross. Ever. This effectively makes the hadoop, hdfs, and mapred entry points configured differently than yarn. bq. Stepping back - should the YARN\_\* parameters exist?, and should yarn jar exist? If I understand you correctly, I think you're trying to get rid of some of this. That's absolutely correct: none of this should exist and I've worked really hard on either removing or hiding the complexity going forward. But this only gets easier in trunk. It's way too late and way too hard to fix this mess in branch-2. bq. If 'yarn jar' is something that we think is confusing, or something we potentially want to get rid off - I'd say it's better to not print any warning at all - and leave hadoop jar as is? It's going to be hard to take yarn jar or hadoop jar away. It's doubtful they will ever get removed. That said, we can at least make them act and work the same way. To me, that's the ultimate goal and it's pretty close to what happens in trunk: 1. yarn command sucks in yarn-env.sh, hadoop-env.sh, yarn-config.sh and hadoop-config.sh in a way that should be mostly conflict-free. (non-yarn commands do not pull in yarn-x.sh, obviously) 2. If YARN\_OPTS is defined, yarn x (jar, rmadmin, etc) will use it but throw a deprecation warning. 3. Otherwise use HADOOP\_OPTS As folks migrate to a release based on trunk, this extra fluff will go away at least configuration-wise. Eventually, when we can remove support for all of these deprecated vars, this will reduce code complexity and gives us (effectively) one code path to test. But bear in mind that we're years away before YARN\_OPTS and friends disappear. It's been 5+ years since our last trunk release. I'll likely be dead by the time 4.x comes out and these useless YARN\_\* vars can get culled officially. But the work has to start now. bq. The hive binary could unset YARN_OPTS / YARN_CLIENT_OPTS - and leave them intact for the session/shell from where the hive binary was invoked. Which, again, if hive wants to stick to using hadoop jar, this would be my advice. \[2] Just keep in mind this also means that any user settings that they might have wanted to apply to their YARN environment will not kick in. It's no different than what is happening today, but it may not reflect what users want. Thus we're back to why the warnings went in. \[1] I'm not saying the community would do this, but let's use hive as an example here of how much more powerful things are in trunk. With HADOOP-12930, it's now possible for hive to add a 'hadoop hive' command or a (even more outrageous!) 'hdfs hivefs' command. Rather than integrate *outside* the framework, one could integrate *inside* and pull exactly the information required. This will hopefully put 'hadoop jar' and 'yarn jar' on life support. We'll keep them around but it's going to be much more desirable to just integrate directly. The new 'mapred streaming' command is a great example here: why make users call hadoop jar with some weird version number when it's now trivial to dynamically add commands at build or configure time? \[2] Very long term (post-3.x), it would probably be better if hive called hadoop-config.sh and/or hadoop-functions.sh directly. This would bypass the middleman and give much better control. I'd be very interested to hear what sort of holes we have in the functionality here that makes this hard/impossible. Off the top, I suspect we need to make one big function of the series of function calls in hadoop-config.sh, but would love to hear your insight on this. > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth >
[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363835#comment-15363835 ] Siddharth Seth commented on HADOOP-13335: - bq. There are no HDFS_* vars that change how the client operates. ... Not yet - until someone feels the need to have hdfs used as a separate command to do something other than filesystem operations. (Really hope there's no hdfs jar command) bq. I'm not sure there is more clarity required. If a user wants/needs YARN_*, then they need to use 'yarn jar'. Again - I don't see the difference between hadoop jar and yarn jar other than a different set of variables being set and respected by the different commands. Stepping back - should the YARN_* parameters exist?, and should yarn jar exist? If I understand you correctly, I think you're trying to get rid of some of this. If 'yarn jar' is something that we think is confusing, or something we potentially want to get rid off - I'd say it's better to not print any warning at all - and leave hadoop jar as is? The hive binary could unset YARN_OPTS / YARN_CLIENT_OPTS - and leave them intact for the session/shell from where the hive binary was invoked. That said, the hive jira linked to this one proposes moving to using 'yarn jar' - and I believe this is mainly because of the WARNING. It would be good for users / downstream projects to know how this should be handled. Will yarn jar survive or not. cc [~vinodkv] since this discussion has quite a bit to do with yarn, and to potentially get more context on yarn jar and the YARN_* variables. > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HADOOP-13335.01.patch, HADOOP-13335.02.patch, > HADOOP-13335.02_branch-2.patch, HADOOP-13335.03.patch, > HADOOP-13335.03_branch-2.patch, HADOOP-13335.04.patch > > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363621#comment-15363621 ] Allen Wittenauer commented on HADOOP-13335: --- bq. It's adding some YARN specific variables - which no existing scripts would have setup. The YARN\_\* vars were introduced when YARN was committed to Hadoop. So they are very, very old at this point in time. In any case, users do know about them and use them. bq. What happens when someone wants to talk to both yarn and hdfs? hdfs-config.sh is not going to get invoked anywhere. There are no HDFS\_\* vars that change how the client operates. Everything there is solely server-side. So there is zero impact that hdfs-config.sh isn't processed in this context. bq. yarn jar is just confusing - and doesn't seem to be providing a lot of utility. Totally agree, which is why I'm trying to eliminate all the extra vars. As I said above, we can't do that for a long time though. 2.x is missing the infrastructure. 3.x has the infrastructure and we can finally mark them as deprecated. Which, again, means the first opportunity to make this work in a sane way isn't until 4.x. :( bq. Hopefully we can have more clarity on what yarn jar means, and how it should be handled in trunk. I'm not sure there is more clarity required. If a user wants/needs YARN\_\*, then they need to use 'yarn jar'. If they don't want to use that, then they have the *option* to use 'hadoop jar' or 'yarn yar'. Hadoop has decided to give users a warning that YARN\_\* opts are not supported or utilized when 'hadoop jar' is executed as they likely intended to use 'yarn jar' instead. Adding a var here to turn this message off when it's already possible to turn this message off via other but similar means is excessive. So I guess I'm technically -1 on branch-2 too. > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HADOOP-13335.01.patch, HADOOP-13335.02.patch, > HADOOP-13335.02_branch-2.patch, HADOOP-13335.03.patch, > HADOOP-13335.03_branch-2.patch, HADOOP-13335.04.patch > > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363564#comment-15363564 ] Siddharth Seth commented on HADOOP-13335: - bq. If anything wants to talk to yarn in any way/shape/form and use hadoop's bootstrap code, they really, really, really should be using yarn jar. Otherwise there is a risk that some things may get mis-configured. We won't be able to clean this up until 4.x. (3.x finally deprecates a lot of this craziness from the project split back that happened in 0.22...) I'm still not clear about the utility of yarn jar. It's adding some YARN specific variables - which no existing scripts would have setup. Other than that - it appears to behave the same as hadoop jar. What happens when someone wants to talk to both yarn and hdfs? hdfs-config.sh is not going to get invoked anywhere. For trunk, it may make sense to separate the 'jar' sub-command instead of clubbing it along with service specific sub-commands (appplications, top, rmadmin, nodemanager, etc) under the 'yarn' command. yarn jar is just confusing - and doesn't seem to be providing a lot of utility. For this specific case - Hive could unset the YARN options in it's main script. Will have to check about the innovations of 'hadoop jar' that happen from within a running jvm. Lets get this patch into branch-2 at least. Hopefully we can have more clarity on what yarn jar means, and how it should be handled in trunk. > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HADOOP-13335.01.patch, HADOOP-13335.02.patch, > HADOOP-13335.02_branch-2.patch, HADOOP-13335.03.patch, > HADOOP-13335.03_branch-2.patch, HADOOP-13335.04.patch > > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363520#comment-15363520 ] Allen Wittenauer commented on HADOOP-13335: --- I'm pretty much -1 on changing this in trunk. The warning is there for a reason. If 3rd parties want to explicitly go against that advice, then they just need to unset the given YARN env vars prior to launching hadoop jar. Unsetting those prior to launching hadoop jar has no impact since they aren't referenced at all in that code path. Therefore, no functionality is lost. This is also much simpler than adding Yet Another Configuration Variable and puts the onus on the caller to do whatever they want to do without putting more code maintenance on Hadoop. > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HADOOP-13335.01.patch, HADOOP-13335.02.patch, > HADOOP-13335.02_branch-2.patch, HADOOP-13335.03.patch, > HADOOP-13335.03_branch-2.patch, HADOOP-13335.04.patch > > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363358#comment-15363358 ] Siddharth Seth commented on HADOOP-13335: - [~aw] - when you get a chance, does this look good to go in ? > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HADOOP-13335.01.patch, HADOOP-13335.02.patch, > HADOOP-13335.02_branch-2.patch, HADOOP-13335.03.patch, > HADOOP-13335.03_branch-2.patch, HADOOP-13335.04.patch > > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[
https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363310#comment-15363310
]
Hadoop QA commented on HADOOP-13335:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
14s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
8s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
51s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 54s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12816308/HADOOP-13335.04.patch
|
| JIRA Issue | HADOOP-13335 |
| Optional Tests | asflicense mvnsite unit shellcheck shelldocs |
| uname | Linux 9c2444d14b40 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 9560f25 |
| shellcheck | v0.4.4 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9926/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9926/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add an option to suppress the 'use yarn jar' warning or remove it
> -
>
> Key: HADOOP-13335
> URL: https://issues.apache.org/jira/browse/HADOOP-13335
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 2.7.0
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HADOOP-13335.01.patch, HADOOP-13335.02.patch,
> HADOOP-13335.02_branch-2.patch, HADOOP-13335.03.patch,
> HADOOP-13335.03_branch-2.patch, HADOOP-13335.04.patch
>
>
> https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation'
> warning for 'hadoop jar'.
> hadoop jar is used for a lot more that starting jobs. As an example - hive
> uses it to start all it's services (HiveServer2, the hive client, beeline
> etc).
> Using 'yarn jar' for to start these services / tools doesn't make a lot of
> sense - there's no relation to yarn other than requiring the classpath to
> include yarn libraries.
> I'd propose reverting the changes where this message is printed if YARN
> variables are set (leave it in the help message), or adding a mechanism which
> would allow users to suppress this WARNING.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[
https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363222#comment-15363222
]
Hadoop QA commented on HADOOP-13335:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s{color}
| {color:red} HADOOP-13335 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12816299/HADOOP-13335.03.patch
|
| JIRA Issue | HADOOP-13335 |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9925/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add an option to suppress the 'use yarn jar' warning or remove it
> -
>
> Key: HADOOP-13335
> URL: https://issues.apache.org/jira/browse/HADOOP-13335
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 2.7.0
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HADOOP-13335.01.patch, HADOOP-13335.02.patch,
> HADOOP-13335.02_branch-2.patch, HADOOP-13335.03.patch,
> HADOOP-13335.03_branch-2.patch
>
>
> https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation'
> warning for 'hadoop jar'.
> hadoop jar is used for a lot more that starting jobs. As an example - hive
> uses it to start all it's services (HiveServer2, the hive client, beeline
> etc).
> Using 'yarn jar' for to start these services / tools doesn't make a lot of
> sense - there's no relation to yarn other than requiring the classpath to
> include yarn libraries.
> I'd propose reverting the changes where this message is printed if YARN
> variables are set (leave it in the help message), or adding a mechanism which
> would allow users to suppress this WARNING.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363172#comment-15363172 ] Allen Wittenauer commented on HADOOP-13335: --- bq. The problem comes in when YARN_CLIENT_OPTS / YARN_OPTS are defined on a shell where hive is invoked Unset them form the environment prior to launch. Those env vars aren't being used outside of the yarn script. > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth > Attachments: HADOOP-13335.01.patch, HADOOP-13335.02.patch, > HADOOP-13335.02_branch-2.patch > > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363164#comment-15363164 ] Allen Wittenauer commented on HADOOP-13335: --- As to the patch: * for trunk, hadoop_error should be used instead of redirection * all env vars should start with either HADOOP_, YARN_, HDFS_, or MAPRED_. > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth > Attachments: HADOOP-13335.01.patch > > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363161#comment-15363161 ] Siddharth Seth commented on HADOOP-13335: - bq. But hive doesn't call yarn, it uses hadoop. hadoop does not import yarn-config.sh, which is where yarn configuration parameters are expected to be located. YARN_USER_CLASSPATH is not handled or likely defined at all. So none of those settings will be defined. Hive does not use 'yarn' at the moment (and hopefully will not need to). The problem comes in when YARN_CLIENT_OPTS / YARN_OPTS are defined on a shell where hive is invoked (for beeline / the cli for example). That ends up displaying the 'use yarn jar' message which is confusing. > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth > Attachments: HADOOP-13335.01.patch > > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363150#comment-15363150 ] Allen Wittenauer commented on HADOOP-13335: --- bq. If I'm not mistaken yarn-config.sh imports hadoop-config.sh - so parameters like HADOOP_USER_CLASSPATH_FIRST etc will be respected. YARN_CLASSPATH_FIRST is handled after building the hadoop classpath. But hive doesn't call yarn, it uses hadoop. hadoop does not import yarn-config.sh, which is where yarn configuration parameters are expected to be located. YARN\_USER\_CLASSPATH is not handled or likely defined *at all*. So none of those settings will be defined. > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth > Attachments: HADOOP-13335.01.patch > > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[
https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363122#comment-15363122
]
Hadoop QA commented on HADOOP-13335:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s{color}
| {color:red} HADOOP-13335 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12816280/HADOOP-13335.01.patch
|
| JIRA Issue | HADOOP-13335 |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9924/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add an option to suppress the 'use yarn jar' warning or remove it
> -
>
> Key: HADOOP-13335
> URL: https://issues.apache.org/jira/browse/HADOOP-13335
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 2.7.0
>Reporter: Siddharth Seth
> Attachments: HADOOP-13335.01.patch
>
>
> https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation'
> warning for 'hadoop jar'.
> hadoop jar is used for a lot more that starting jobs. As an example - hive
> uses it to start all it's services (HiveServer2, the hive client, beeline
> etc).
> Using 'yarn jar' for to start these services / tools doesn't make a lot of
> sense - there's no relation to yarn other than requiring the classpath to
> include yarn libraries.
> I'd propose reverting the changes where this message is printed if YARN
> variables are set (leave it in the help message), or adding a mechanism which
> would allow users to suppress this WARNING.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15362984#comment-15362984 ] Allen Wittenauer commented on HADOOP-13335: --- hadoop jar does not include any of the following yarn content that is normally initialized during the yarn-config.sh bootstrap: * anything defined in yarn-env.sh * some YARN_*_DIR * some YARN_*_OPT and YARN_*_OPTS * YARN_*_CLASSPATH If anything wants to talk to yarn in any way/shape/form and use hadoop's bootstrap code, they really, really, really should be using yarn jar. Otherwise there is a risk that some things may get mis-configured. We won't be able to clean this up until 4.x. (3.x finally deprecates a lot of this craziness from the project split back that happened in 0.22...) > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15362949#comment-15362949 ] Siddharth Seth commented on HADOOP-13335: - [~aw] - what does yarn jar mean (as compared to Hadoop jar) - is it's main intent to provide the yarn libraries in the classpath ? Will upload a patch a little later to optionally disable the warning. > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360609#comment-15360609 ] Allen Wittenauer commented on HADOOP-13335: --- bq. hive uses it to start all it's services (HiveServer2, the hive client, beeline etc). ... almost/all? of which interact with yarn, which means they should be using yarn jar. > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
