[jira] [Updated] (HADOOP-13212) Provide an option to set the socket buffers in S3AFileSystem
[ https://issues.apache.org/jira/browse/HADOOP-13212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HADOOP-13212: -- Status: Patch Available (was: Open) > Provide an option to set the socket buffers in S3AFileSystem > > > Key: HADOOP-13212 > URL: https://issues.apache.org/jira/browse/HADOOP-13212 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HADOOP-13212-branch-2-001.patch > > > It should be possible to provide hints about send/receive buffers to > AmazonS3Client via ClientConfiguration. It would be good to expose these > parameters in S3AFileSystem for perf. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-13212) Provide an option to set the socket buffers in S3AFileSystem
[ https://issues.apache.org/jira/browse/HADOOP-13212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan reassigned HADOOP-13212: - Assignee: Rajesh Balamohan > Provide an option to set the socket buffers in S3AFileSystem > > > Key: HADOOP-13212 > URL: https://issues.apache.org/jira/browse/HADOOP-13212 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HADOOP-13212-branch-2-001.patch > > > It should be possible to provide hints about send/receive buffers to > AmazonS3Client via ClientConfiguration. It would be good to expose these > parameters in S3AFileSystem for perf. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13212) Provide an option to set the socket buffers in S3AFileSystem
[ https://issues.apache.org/jira/browse/HADOOP-13212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HADOOP-13212: -- Attachment: HADOOP-13212-branch-2-001.patch HttpClient internally would set it to 8192 when these options were not available. Having this patch enables to change it at runtime for performance benchmarks and experiments. > Provide an option to set the socket buffers in S3AFileSystem > > > Key: HADOOP-13212 > URL: https://issues.apache.org/jira/browse/HADOOP-13212 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Rajesh Balamohan >Priority: Minor > Attachments: HADOOP-13212-branch-2-001.patch > > > It should be possible to provide hints about send/receive buffers to > AmazonS3Client via ClientConfiguration. It would be good to expose these > parameters in S3AFileSystem for perf. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13352) Make X-FRAME-OPTIONS configurable in HttpServer2
[ https://issues.apache.org/jira/browse/HADOOP-13352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HADOOP-13352: -- Attachment: HADOOP-13352.001.patch Adds config capability for X-FRAME-OPTIONS support in HttpServer2. > Make X-FRAME-OPTIONS configurable in HttpServer2 > > > Key: HADOOP-13352 > URL: https://issues.apache.org/jira/browse/HADOOP-13352 > Project: Hadoop Common > Issue Type: Bug > Components: net, security >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: 2.9.0 > > Attachments: HADOOP-13352.001.patch > > > In HADOOP-12964 we introduced support for X-FRAME-OPTIONS in HttpServer2. > This JIRA makes it configurable. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13353) LdapGroupsMapping getPassward shouldn't return null when IOException throws
[
https://issues.apache.org/jira/browse/HADOOP-13353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15367165#comment-15367165
]
Hadoop QA commented on HADOOP-13353:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 17m 2s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 52m 2s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.http.TestHttpServerLifecycle |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12816745/HADOOP-13353.patch |
| JIRA Issue | HADOOP-13353 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux bbe91d4b5a84 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 9d46a49 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9940/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9940/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9940/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> LdapGroupsMapping getPassward shouldn't return null when IOException throws
> ---
>
> Key: HADOOP-13353
> URL: https://issues.apache.or
[jira] [Updated] (HADOOP-13353) LdapGroupsMapping getPassward shouldn't return null when IOException throws
[ https://issues.apache.org/jira/browse/HADOOP-13353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhaohao Liang updated HADOOP-13353: --- Status: Patch Available (was: Open) > LdapGroupsMapping getPassward shouldn't return null when IOException throws > --- > > Key: HADOOP-13353 > URL: https://issues.apache.org/jira/browse/HADOOP-13353 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.6.0 >Reporter: Zhaohao Liang > Attachments: HADOOP-13353.patch > > > When IOException throws in getPassword(), getPassword() return null String, > this will cause setConf() throws java.lang.NullPointerException when check > isEmpty() on null string. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13353) LdapGroupsMapping getPassward shouldn't return null when IOException throws
[ https://issues.apache.org/jira/browse/HADOOP-13353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhaohao Liang updated HADOOP-13353: --- Attachment: HADOOP-13353.patch > LdapGroupsMapping getPassward shouldn't return null when IOException throws > --- > > Key: HADOOP-13353 > URL: https://issues.apache.org/jira/browse/HADOOP-13353 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.6.0 >Reporter: Zhaohao Liang > Attachments: HADOOP-13353.patch > > > When IOException throws in getPassword(), getPassword() return null String, > this will cause setConf() throws java.lang.NullPointerException when check > isEmpty() on null string. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13353) LdapGroupsMapping getPassward shouldn't return null when IOException throws
Zhaohao Liang created HADOOP-13353: -- Summary: LdapGroupsMapping getPassward shouldn't return null when IOException throws Key: HADOOP-13353 URL: https://issues.apache.org/jira/browse/HADOOP-13353 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 2.6.0 Reporter: Zhaohao Liang When IOException throws in getPassword(), getPassword() return null String, this will cause setConf() throws java.lang.NullPointerException when check isEmpty() on null string. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13351) TestDFSClientSocketSize buffer size tests are flaky
[
https://issues.apache.org/jira/browse/HADOOP-13351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15367098#comment-15367098
]
Mingliang Liu edited comment on HADOOP-13351 at 7/8/16 2:12 AM:
{quote}
the OS declining to set a large send buffer size for a socket.
{quote}
Thanks [~fabbri] for the explanation. In {{StandardSocketOptions}}, an attempt
to set the socket send buffer to larger than its maximum size causes it to be
set to its maximum size (refer to [link
here|https://docs.oracle.com/javase/7/docs/api/java/net/StandardSocketOptions.html#SO_SNDBUF]).
I think the value for {{Socket}} would work in the same way. This way, a
larger value like 512KB would not cause problems for set operation. However,
the test may still fail since the maximum size, as the default size, is system
dependent.
That being said, testing the specified value is indeed tricky. To make the test
assertion covers this case, how about comparing the hinted buffer size? I see
very limited chance that the test fails, but it does confirm that, the config
key works as expected.
Sample code as following:
{code:java}
@Test
public void testSpecifiedSendBufferSize() throws IOException {
conf.setInt(DFS_CLIENT_SOCKET_SEND_BUFFER_SIZE_KEY, 256 * 1024); // a big
hint
final Socket s1 = createSocket();
conf.setInt(DFS_CLIENT_SOCKET_SEND_BUFFER_SIZE_KEY, 1024); // a
small hint
final Socket s2 = createSocket();
LOG.info("The real big buffer size is {}, small buffer size is {}",
s1.getSendBufferSize(), s2.getSendBufferSize());
assertTrue(s1.getSendBufferSize() > s2.getSendBufferSize());
}
{code}
What's your thought?
was (Author: liuml07):
{quote}
the OS declining to set a large send buffer size for a socket.
{quote}
Thanks [~fabbri] for the explanation. In {{StandardSocketOptions}}, an attempt
to set the socket send buffer to larger than its maximum size causes it to be
set to its maximum size (refer to [link
here|https://docs.oracle.com/javase/7/docs/api/java/net/StandardSocketOptions.html#SO_SNDBUF]).
I think the value for {{Socket}} would work in the same way for socket. This
way, a larger value like 512KB would not cause problems for set operation.
However, the test may still fail as the maximum size, as the default size, is
system dependent.
That being said, testing the specified value is indeed tricky. To make the test
assertion covers this case, how about comparing the hinted buffer size? I see
very limited chance that the test fails, but it does confirm that, the config
key works as expected.
Sample code as following:
{code:java}
@Test
public void testSpecifiedSendBufferSize() throws IOException {
conf.setInt(DFS_CLIENT_SOCKET_SEND_BUFFER_SIZE_KEY, 256 * 1024);
final Socket s1 = createSocket();
conf.setInt(DFS_CLIENT_SOCKET_SEND_BUFFER_SIZE_KEY, 1024);
final Socket s2 = createSocket();
LOG.info("The real big buffer size is {}, small buffer size is {}",
s1.getSendBufferSize(), s2.getSendBufferSize());
assertTrue(s1.getSendBufferSize() > s2.getSendBufferSize());
}
{code}
What's your thought?
> TestDFSClientSocketSize buffer size tests are flaky
> ---
>
> Key: HADOOP-13351
> URL: https://issues.apache.org/jira/browse/HADOOP-13351
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Aaron Fabbri
>Assignee: Aaron Fabbri
> Attachments: HADOOP-13551.001.patch
>
>
> {{TestDFSClientSocketSize}} has two tests that assert that a value that was
> set via {{java.net.Socket#setSendBufferSize}} is equal to the value
> subsequently returned by {{java.net.Socket#getSendBufferSize}}.
> These tests are flaky when we run them. The occasionally fail.
> This is expected behavior, actually, because
> {{Socket#setSendBufferSize()}}[is only a
> hint|https://docs.oracle.com/javase/7/docs/api/java/net/Socket.html#setSendBufferSize(int)].
> (Similar to how the underlying libc {{setsockopt(SO_SNDBUF)}} works).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13351) TestDFSClientSocketSize buffer size tests are flaky
[
https://issues.apache.org/jira/browse/HADOOP-13351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15367098#comment-15367098
]
Mingliang Liu commented on HADOOP-13351:
{quote}
the OS declining to set a large send buffer size for a socket.
{quote}
Thanks [~fabbri] for the explanation. In {{StandardSocketOptions}}, an attempt
to set the socket send buffer to larger than its maximum size causes it to be
set to its maximum size (refer to [link
here|https://docs.oracle.com/javase/7/docs/api/java/net/StandardSocketOptions.html#SO_SNDBUF]).
I think the value for {{Socket}} would work in the same way for socket. This
way, a larger value like 512KB would not cause problems for set operation.
However, the test may still fail as the maximum size, as the default size, is
system dependent.
That being said, testing the specified value is indeed tricky. To make the test
assertion covers this case, how about comparing the hinted buffer size? I see
very limited chance that the test fails, but it does confirm that, the config
key works as expected.
Sample code as following:
{code:java}
@Test
public void testSpecifiedSendBufferSize() throws IOException {
conf.setInt(DFS_CLIENT_SOCKET_SEND_BUFFER_SIZE_KEY, 256 * 1024);
final Socket s1 = createSocket();
conf.setInt(DFS_CLIENT_SOCKET_SEND_BUFFER_SIZE_KEY, 1024);
final Socket s2 = createSocket();
LOG.info("The real big buffer size is {}, small buffer size is {}",
s1.getSendBufferSize(), s2.getSendBufferSize());
assertTrue(s1.getSendBufferSize() > s2.getSendBufferSize());
}
{code}
What's your thought?
> TestDFSClientSocketSize buffer size tests are flaky
> ---
>
> Key: HADOOP-13351
> URL: https://issues.apache.org/jira/browse/HADOOP-13351
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Aaron Fabbri
>Assignee: Aaron Fabbri
> Attachments: HADOOP-13551.001.patch
>
>
> {{TestDFSClientSocketSize}} has two tests that assert that a value that was
> set via {{java.net.Socket#setSendBufferSize}} is equal to the value
> subsequently returned by {{java.net.Socket#getSendBufferSize}}.
> These tests are flaky when we run them. The occasionally fail.
> This is expected behavior, actually, because
> {{Socket#setSendBufferSize()}}[is only a
> hint|https://docs.oracle.com/javase/7/docs/api/java/net/Socket.html#setSendBufferSize(int)].
> (Similar to how the underlying libc {{setsockopt(SO_SNDBUF)}} works).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work started] (HADOOP-13289) Remove unused variables in TestFairCallQueue
[
https://issues.apache.org/jira/browse/HADOOP-13289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HADOOP-13289 started by Ye Zhou.
> Remove unused variables in TestFairCallQueue
>
>
> Key: HADOOP-13289
> URL: https://issues.apache.org/jira/browse/HADOOP-13289
> Project: Hadoop Common
> Issue Type: Improvement
> Components: test
>Reporter: Konstantin Shvachko
>Assignee: Ye Zhou
> Labels: newbie
>
> # Remove unused member {{alwaysZeroScheduler}} and related initialization in
> {{TestFairCallQueue}}
> # Remove unused local vriable {{sched}} in
> {{testOfferSucceedsWhenScheduledLowPriority()}}
> And propagate to applicable release branches.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13352) Make X-FRAME-OPTIONS configurable in HttpServer2
Anu Engineer created HADOOP-13352: - Summary: Make X-FRAME-OPTIONS configurable in HttpServer2 Key: HADOOP-13352 URL: https://issues.apache.org/jira/browse/HADOOP-13352 Project: Hadoop Common Issue Type: Bug Components: net, security Reporter: Anu Engineer Assignee: Anu Engineer Fix For: 2.9.0 In HADOOP-12964 we introduced support for X-FRAME-OPTIONS in HttpServer2. This JIRA makes it configurable. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13351) TestDFSClientSocketSize buffer size tests are flaky
[
https://issues.apache.org/jira/browse/HADOOP-13351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366982#comment-15366982
]
Aaron Fabbri commented on HADOOP-13351:
---
Thank you for review [~liuml07]. I do not have ability to assign JIRA issues
here.. Looking into that.
On the code, I understand what you are saying. My thinking here is that we
should at least be getting a minimal send buffer, so let's test that. Since
the socket option is only a hint, I'm concerned increasing it too much could
add another cause of test flakiness: the OS declining to set a large send
buffer size for a socket. So I'd personally lean towards keeping it as is in
the patch or removing the two test cases.
> TestDFSClientSocketSize buffer size tests are flaky
> ---
>
> Key: HADOOP-13351
> URL: https://issues.apache.org/jira/browse/HADOOP-13351
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Aaron Fabbri
>Assignee: Aaron Fabbri
> Attachments: HADOOP-13551.001.patch
>
>
> {{TestDFSClientSocketSize}} has two tests that assert that a value that was
> set via {{java.net.Socket#setSendBufferSize}} is equal to the value
> subsequently returned by {{java.net.Socket#getSendBufferSize}}.
> These tests are flaky when we run them. The occasionally fail.
> This is expected behavior, actually, because
> {{Socket#setSendBufferSize()}}[is only a
> hint|https://docs.oracle.com/javase/7/docs/api/java/net/Socket.html#setSendBufferSize(int)].
> (Similar to how the underlying libc {{setsockopt(SO_SNDBUF)}} works).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13351) TestDFSClientSocketSize buffer size tests are flaky
[
https://issues.apache.org/jira/browse/HADOOP-13351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HADOOP-13351:
-
Assignee: Aaron Fabbri
> TestDFSClientSocketSize buffer size tests are flaky
> ---
>
> Key: HADOOP-13351
> URL: https://issues.apache.org/jira/browse/HADOOP-13351
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Aaron Fabbri
>Assignee: Aaron Fabbri
> Attachments: HADOOP-13551.001.patch
>
>
> {{TestDFSClientSocketSize}} has two tests that assert that a value that was
> set via {{java.net.Socket#setSendBufferSize}} is equal to the value
> subsequently returned by {{java.net.Socket#getSendBufferSize}}.
> These tests are flaky when we run them. The occasionally fail.
> This is expected behavior, actually, because
> {{Socket#setSendBufferSize()}}[is only a
> hint|https://docs.oracle.com/javase/7/docs/api/java/net/Socket.html#setSendBufferSize(int)].
> (Similar to how the underlying libc {{setsockopt(SO_SNDBUF)}} works).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13351) TestDFSClientSocketSize buffer size tests are flaky
[
https://issues.apache.org/jira/browse/HADOOP-13351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366971#comment-15366971
]
Mingliang Liu edited comment on HADOOP-13351 at 7/7/16 11:46 PM:
-
The v1 patch is also the best effort I can figure out so far. Thanks for the
patch.
{code}
void testSpecifiedSendBufferSize() throws IOException {
final int mySendBufferSize = 64 * 1024; // 64 KB
{code}
To make the test covers the specified value, I'd suggest we change
{{mySendBufferSize}} to be a different value (say, 512KB). The default value is
128KB >= 64KB. As a result, the test can pass without the following the
statement which sets the specific value:
{code}
conf.setInt(DFS_CLIENT_SOCKET_SEND_BUFFER_SIZE_KEY, mySendBufferSize);
{code}
was (Author: liuml07):
The v1 patch is also the best effort I can figure out so far. Thanks for the
patch.
{code}
void testSpecifiedSendBufferSize() throws IOException {
final int mySendBufferSize = 64 * 1024; // 64 KB
{code}
To make the test covers the specified value, I'd suggest we change
{{mySendBufferSize}} to be a different value (say, 32KB). The default value is
128KB >= 64KB. As a result, the test can pass without the following the
statement which sets the specific value:
{code}
conf.setInt(DFS_CLIENT_SOCKET_SEND_BUFFER_SIZE_KEY, mySendBufferSize);
{code}
> TestDFSClientSocketSize buffer size tests are flaky
> ---
>
> Key: HADOOP-13351
> URL: https://issues.apache.org/jira/browse/HADOOP-13351
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Aaron Fabbri
> Attachments: HADOOP-13551.001.patch
>
>
> {{TestDFSClientSocketSize}} has two tests that assert that a value that was
> set via {{java.net.Socket#setSendBufferSize}} is equal to the value
> subsequently returned by {{java.net.Socket#getSendBufferSize}}.
> These tests are flaky when we run them. The occasionally fail.
> This is expected behavior, actually, because
> {{Socket#setSendBufferSize()}}[is only a
> hint|https://docs.oracle.com/javase/7/docs/api/java/net/Socket.html#setSendBufferSize(int)].
> (Similar to how the underlying libc {{setsockopt(SO_SNDBUF)}} works).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13351) TestDFSClientSocketSize buffer size tests are flaky
[
https://issues.apache.org/jira/browse/HADOOP-13351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366971#comment-15366971
]
Mingliang Liu commented on HADOOP-13351:
The v1 patch is also the best effort I can figure out so far. Thanks for the
patch.
{code}
void testSpecifiedSendBufferSize() throws IOException {
final int mySendBufferSize = 64 * 1024; // 64 KB
{code}
To make the test covers the specified value, I'd suggest we change
{{mySendBufferSize}} to be a different value (say, 32KB). The default value is
128KB >= 64KB. As a result, the test can pass without the following the
statement which sets the specific value:
{code}
conf.setInt(DFS_CLIENT_SOCKET_SEND_BUFFER_SIZE_KEY, mySendBufferSize);
{code}
> TestDFSClientSocketSize buffer size tests are flaky
> ---
>
> Key: HADOOP-13351
> URL: https://issues.apache.org/jira/browse/HADOOP-13351
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Aaron Fabbri
> Attachments: HADOOP-13551.001.patch
>
>
> {{TestDFSClientSocketSize}} has two tests that assert that a value that was
> set via {{java.net.Socket#setSendBufferSize}} is equal to the value
> subsequently returned by {{java.net.Socket#getSendBufferSize}}.
> These tests are flaky when we run them. The occasionally fail.
> This is expected behavior, actually, because
> {{Socket#setSendBufferSize()}}[is only a
> hint|https://docs.oracle.com/javase/7/docs/api/java/net/Socket.html#setSendBufferSize(int)].
> (Similar to how the underlying libc {{setsockopt(SO_SNDBUF)}} works).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13351) TestDFSClientSocketSize buffer size tests are flaky
[
https://issues.apache.org/jira/browse/HADOOP-13351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron Fabbri updated HADOOP-13351:
--
Attachment: HADOOP-13551.001.patch
> TestDFSClientSocketSize buffer size tests are flaky
> ---
>
> Key: HADOOP-13351
> URL: https://issues.apache.org/jira/browse/HADOOP-13351
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Aaron Fabbri
> Attachments: HADOOP-13551.001.patch
>
>
> {{TestDFSClientSocketSize}} has two tests that assert that a value that was
> set via {{java.net.Socket#setSendBufferSize}} is equal to the value
> subsequently returned by {{java.net.Socket#getSendBufferSize}}.
> These tests are flaky when we run them. The occasionally fail.
> This is expected behavior, actually, because
> {{Socket#setSendBufferSize()}}[is only a
> hint|https://docs.oracle.com/javase/7/docs/api/java/net/Socket.html#setSendBufferSize(int)].
> (Similar to how the underlying libc {{setsockopt(SO_SNDBUF)}} works).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13351) TestDFSClientSocketSize buffer size tests are flaky
[
https://issues.apache.org/jira/browse/HADOOP-13351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366965#comment-15366965
]
Mingliang Liu commented on HADOOP-13351:
Thanks [~fabbri] for working on this. I can not assign this to you, so please
pick it up.
> TestDFSClientSocketSize buffer size tests are flaky
> ---
>
> Key: HADOOP-13351
> URL: https://issues.apache.org/jira/browse/HADOOP-13351
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Aaron Fabbri
>
> {{TestDFSClientSocketSize}} has two tests that assert that a value that was
> set via {{java.net.Socket#setSendBufferSize}} is equal to the value
> subsequently returned by {{java.net.Socket#getSendBufferSize}}.
> These tests are flaky when we run them. The occasionally fail.
> This is expected behavior, actually, because
> {{Socket#setSendBufferSize()}}[is only a
> hint|https://docs.oracle.com/javase/7/docs/api/java/net/Socket.html#setSendBufferSize(int)].
> (Similar to how the underlying libc {{setsockopt(SO_SNDBUF)}} works).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13351) TestDFSClientSocketSize buffer size tests are flaky
[
https://issues.apache.org/jira/browse/HADOOP-13351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-13351:
---
Assignee: (was: Mingliang Liu)
> TestDFSClientSocketSize buffer size tests are flaky
> ---
>
> Key: HADOOP-13351
> URL: https://issues.apache.org/jira/browse/HADOOP-13351
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Aaron Fabbri
>
> {{TestDFSClientSocketSize}} has two tests that assert that a value that was
> set via {{java.net.Socket#setSendBufferSize}} is equal to the value
> subsequently returned by {{java.net.Socket#getSendBufferSize}}.
> These tests are flaky when we run them. The occasionally fail.
> This is expected behavior, actually, because
> {{Socket#setSendBufferSize()}}[is only a
> hint|https://docs.oracle.com/javase/7/docs/api/java/net/Socket.html#setSendBufferSize(int)].
> (Similar to how the underlying libc {{setsockopt(SO_SNDBUF)}} works).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13350) Additional fix to LICENSE and NOTICE
[ https://issues.apache.org/jira/browse/HADOOP-13350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366940#comment-15366940 ] Andrew Wang commented on HADOOP-13350: -- My bad, I forgot that we branched that already. Committed there as well, thanks for catching this Akira. > Additional fix to LICENSE and NOTICE > > > Key: HADOOP-13350 > URL: https://issues.apache.org/jira/browse/HADOOP-13350 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0, 3.0.0-alpha1 >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Blocker > Fix For: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1 > > Attachments: HADOOP-13350-branch2.6+2.7.01.patch, > HADOOP-13350.01.patch > > > Fix up LICENSE and NOTICE after HADOOP-12893. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13351) TestDFSClientSocketSize buffer size tests are flaky
[
https://issues.apache.org/jira/browse/HADOOP-13351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366937#comment-15366937
]
Aaron Fabbri commented on HADOOP-13351:
---
[~liuml07] I'm already working on this.. Attaching patch.
> TestDFSClientSocketSize buffer size tests are flaky
> ---
>
> Key: HADOOP-13351
> URL: https://issues.apache.org/jira/browse/HADOOP-13351
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Aaron Fabbri
>Assignee: Mingliang Liu
>
> {{TestDFSClientSocketSize}} has two tests that assert that a value that was
> set via {{java.net.Socket#setSendBufferSize}} is equal to the value
> subsequently returned by {{java.net.Socket#getSendBufferSize}}.
> These tests are flaky when we run them. The occasionally fail.
> This is expected behavior, actually, because
> {{Socket#setSendBufferSize()}}[is only a
> hint|https://docs.oracle.com/javase/7/docs/api/java/net/Socket.html#setSendBufferSize(int)].
> (Similar to how the underlying libc {{setsockopt(SO_SNDBUF)}} works).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13351) TestDFSClientSocketSize buffer size tests are flaky
[
https://issues.apache.org/jira/browse/HADOOP-13351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366935#comment-15366935
]
Aaron Fabbri commented on HADOOP-13351:
---
Example error message (from [~xiaochen]):
{noformat}
Send buffer size should be the default value. expected:<131072> but was:<131071>
{noformat}
Stacktrace
{noformat}
java.lang.AssertionError: Send buffer size should be the default value.
expected:<131072> but was:<131071>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at
org.apache.hadoop.hdfs.TestDFSClientSocketSize.testDefaultSendBufferSize(TestDFSClientSocketSize.java:53)
{noformat}
> TestDFSClientSocketSize buffer size tests are flaky
> ---
>
> Key: HADOOP-13351
> URL: https://issues.apache.org/jira/browse/HADOOP-13351
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Aaron Fabbri
>Assignee: Mingliang Liu
>
> {{TestDFSClientSocketSize}} has two tests that assert that a value that was
> set via {{java.net.Socket#setSendBufferSize}} is equal to the value
> subsequently returned by {{java.net.Socket#getSendBufferSize}}.
> These tests are flaky when we run them. The occasionally fail.
> This is expected behavior, actually, because
> {{Socket#setSendBufferSize()}}[is only a
> hint|https://docs.oracle.com/javase/7/docs/api/java/net/Socket.html#setSendBufferSize(int)].
> (Similar to how the underlying libc {{setsockopt(SO_SNDBUF)}} works).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13351) TestDFSClientSocketSize buffer size tests are flaky
[
https://issues.apache.org/jira/browse/HADOOP-13351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366930#comment-15366930
]
Mingliang Liu commented on HADOOP-13351:
Thanks for reporting this, [~fabbri]. I can remember my adding this UT. I
assign this to myself. Feel free to grab it if you have a plan to fix.
> TestDFSClientSocketSize buffer size tests are flaky
> ---
>
> Key: HADOOP-13351
> URL: https://issues.apache.org/jira/browse/HADOOP-13351
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Aaron Fabbri
>Assignee: Mingliang Liu
>
> {{TestDFSClientSocketSize}} has two tests that assert that a value that was
> set via {{java.net.Socket#setSendBufferSize}} is equal to the value
> subsequently returned by {{java.net.Socket#getSendBufferSize}}.
> These tests are flaky when we run them. The occasionally fail.
> This is expected behavior, actually, because
> {{Socket#setSendBufferSize()}}[is only a
> hint|https://docs.oracle.com/javase/7/docs/api/java/net/Socket.html#setSendBufferSize(int)].
> (Similar to how the underlying libc {{setsockopt(SO_SNDBUF)}} works).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HADOOP-5052) Add an example for computing exact digits of Pi
[ https://issues.apache.org/jira/browse/HADOOP-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HADOOP-5052: Comment: was deleted (was: Very useful example visit here for more information. (http://bit.ly/1WQKTZ6)) > Add an example for computing exact digits of Pi > --- > > Key: HADOOP-5052 > URL: https://issues.apache.org/jira/browse/HADOOP-5052 > Project: Hadoop Common > Issue Type: New Feature >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Fix For: 0.21.0 > > Attachments: 5052_20090115_0.18.patch, 5052_20090117_0.18.patch, > 5052_20090127.patch, 5052_20090203.patch, 5052_20090211.patch, > 5052_20090219.patch > > > It would be useful to add an example showing how to use Hadoop to do > scientific computing. We should add an example for computing exact digits of > Pi. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-13351) TestDFSClientSocketSize buffer size tests are flaky
[
https://issues.apache.org/jira/browse/HADOOP-13351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu reassigned HADOOP-13351:
--
Assignee: Mingliang Liu
> TestDFSClientSocketSize buffer size tests are flaky
> ---
>
> Key: HADOOP-13351
> URL: https://issues.apache.org/jira/browse/HADOOP-13351
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Aaron Fabbri
>Assignee: Mingliang Liu
>
> {{TestDFSClientSocketSize}} has two tests that assert that a value that was
> set via {{java.net.Socket#setSendBufferSize}} is equal to the value
> subsequently returned by {{java.net.Socket#getSendBufferSize}}.
> These tests are flaky when we run them. The occasionally fail.
> This is expected behavior, actually, because
> {{Socket#setSendBufferSize()}}[is only a
> hint|https://docs.oracle.com/javase/7/docs/api/java/net/Socket.html#setSendBufferSize(int)].
> (Similar to how the underlying libc {{setsockopt(SO_SNDBUF)}} works).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13351) TestDFSClientSocketSize buffer size tests are flaky
Aaron Fabbri created HADOOP-13351:
-
Summary: TestDFSClientSocketSize buffer size tests are flaky
Key: HADOOP-13351
URL: https://issues.apache.org/jira/browse/HADOOP-13351
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.8.0, 3.0.0-alpha1
Reporter: Aaron Fabbri
{{TestDFSClientSocketSize}} has two tests that assert that a value that was set
via {{java.net.Socket#setSendBufferSize}} is equal to the value subsequently
returned by {{java.net.Socket#getSendBufferSize}}.
These tests are flaky when we run them. The occasionally fail.
This is expected behavior, actually, because {{Socket#setSendBufferSize()}}[is
only a
hint|https://docs.oracle.com/javase/7/docs/api/java/net/Socket.html#setSendBufferSize(int)].
(Similar to how the underlying libc {{setsockopt(SO_SNDBUF)}} works).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13320) Fix arguments check in documentation for WordCount v2.0
[
https://issues.apache.org/jira/browse/HADOOP-13320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HADOOP-13320:
---
Resolution: Fixed
Fix Version/s: 2.8.0
Status: Resolved (was: Patch Available)
> Fix arguments check in documentation for WordCount v2.0
> ---
>
> Key: HADOOP-13320
> URL: https://issues.apache.org/jira/browse/HADOOP-13320
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
>Reporter: niccolo becchi
>Assignee: niccolo becchi
>Priority: Minor
> Fix For: 2.8.0
>
>
> This issue is affecting the documentation page, so the code is not covered by
> any tests. It's actually visible on the page:
> https://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html#Example:_WordCount_v2.0
> On the Example: WordCount v2.0
> The actual arguments check is wrong, as it's never printing the message of
> the correct usage. So, running the code with no parameters, as in the
> following example:
> {code}
> yarn jar /var/tmp/WordCount.jar task0.WordCount2
> {code}
>
> I have got the following exception message in output:
> {code}
> Exception in thread "main" java.lang.IndexOutOfBoundsException: Index: 0,
> Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at task0.WordCount2.main(WordCount2.java:131)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {code}
> Intead than the expected friendly message:
> {code}
> Usage: wordcount [-skip skipPatternFile]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13320) Fix arguments check in documentation for WordCount v2.0
[
https://issues.apache.org/jira/browse/HADOOP-13320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366911#comment-15366911
]
Hudson commented on HADOOP-13320:
-
FAILURE: Integrated in Hadoop-trunk-Commit #10065 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/10065/])
HADOOP-13320. Fix arguments check in documentation for WordCount v2.0.
(aajisaka: rev 9d46a49c746b9e1ef552dbb10d1e22f87db68c76)
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/MapReduceTutorial.md
> Fix arguments check in documentation for WordCount v2.0
> ---
>
> Key: HADOOP-13320
> URL: https://issues.apache.org/jira/browse/HADOOP-13320
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
>Reporter: niccolo becchi
>Assignee: niccolo becchi
>Priority: Minor
>
> This issue is affecting the documentation page, so the code is not covered by
> any tests. It's actually visible on the page:
> https://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html#Example:_WordCount_v2.0
> On the Example: WordCount v2.0
> The actual arguments check is wrong, as it's never printing the message of
> the correct usage. So, running the code with no parameters, as in the
> following example:
> {code}
> yarn jar /var/tmp/WordCount.jar task0.WordCount2
> {code}
>
> I have got the following exception message in output:
> {code}
> Exception in thread "main" java.lang.IndexOutOfBoundsException: Index: 0,
> Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at task0.WordCount2.main(WordCount2.java:131)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {code}
> Intead than the expected friendly message:
> {code}
> Usage: wordcount [-skip skipPatternFile]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13320) Fix arguments check in documentation for WordCount v2.0
[
https://issues.apache.org/jira/browse/HADOOP-13320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HADOOP-13320:
---
Assignee: niccolo becchi
> Fix arguments check in documentation for WordCount v2.0
> ---
>
> Key: HADOOP-13320
> URL: https://issues.apache.org/jira/browse/HADOOP-13320
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
>Reporter: niccolo becchi
>Assignee: niccolo becchi
>Priority: Minor
>
> This issue is affecting the documentation page, so the code is not covered by
> any tests. It's actually visible on the page:
> https://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html#Example:_WordCount_v2.0
> On the Example: WordCount v2.0
> The actual arguments check is wrong, as it's never printing the message of
> the correct usage. So, running the code with no parameters, as in the
> following example:
> {code}
> yarn jar /var/tmp/WordCount.jar task0.WordCount2
> {code}
>
> I have got the following exception message in output:
> {code}
> Exception in thread "main" java.lang.IndexOutOfBoundsException: Index: 0,
> Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at task0.WordCount2.main(WordCount2.java:131)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {code}
> Intead than the expected friendly message:
> {code}
> Usage: wordcount [-skip skipPatternFile]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13320) Fix arguments check in documentation for WordCount v2.0
[
https://issues.apache.org/jira/browse/HADOOP-13320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366893#comment-15366893
]
Akira Ajisaka commented on HADOOP-13320:
Committed this to trunk, branch-2, and branch-2.8. Thanks [~niccolo.becchi] for
the contribution and thanks [~templedf] for the review.
> Fix arguments check in documentation for WordCount v2.0
> ---
>
> Key: HADOOP-13320
> URL: https://issues.apache.org/jira/browse/HADOOP-13320
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
>Reporter: niccolo becchi
>Priority: Minor
>
> This issue is affecting the documentation page, so the code is not covered by
> any tests. It's actually visible on the page:
> https://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html#Example:_WordCount_v2.0
> On the Example: WordCount v2.0
> The actual arguments check is wrong, as it's never printing the message of
> the correct usage. So, running the code with no parameters, as in the
> following example:
> {code}
> yarn jar /var/tmp/WordCount.jar task0.WordCount2
> {code}
>
> I have got the following exception message in output:
> {code}
> Exception in thread "main" java.lang.IndexOutOfBoundsException: Index: 0,
> Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at task0.WordCount2.main(WordCount2.java:131)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {code}
> Intead than the expected friendly message:
> {code}
> Usage: wordcount [-skip skipPatternFile]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13320) Fix arguments check in documentation for WordCount v2.0
[
https://issues.apache.org/jira/browse/HADOOP-13320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366892#comment-15366892
]
ASF GitHub Bot commented on HADOOP-13320:
-
Github user asfgit closed the pull request at:
https://github.com/apache/hadoop/pull/108
> Fix arguments check in documentation for WordCount v2.0
> ---
>
> Key: HADOOP-13320
> URL: https://issues.apache.org/jira/browse/HADOOP-13320
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
>Reporter: niccolo becchi
>Priority: Minor
>
> This issue is affecting the documentation page, so the code is not covered by
> any tests. It's actually visible on the page:
> https://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html#Example:_WordCount_v2.0
> On the Example: WordCount v2.0
> The actual arguments check is wrong, as it's never printing the message of
> the correct usage. So, running the code with no parameters, as in the
> following example:
> {code}
> yarn jar /var/tmp/WordCount.jar task0.WordCount2
> {code}
>
> I have got the following exception message in output:
> {code}
> Exception in thread "main" java.lang.IndexOutOfBoundsException: Index: 0,
> Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at task0.WordCount2.main(WordCount2.java:131)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {code}
> Intead than the expected friendly message:
> {code}
> Usage: wordcount [-skip skipPatternFile]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13320) Fix arguments check in documentation for WordCount v2.0
[
https://issues.apache.org/jira/browse/HADOOP-13320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HADOOP-13320:
---
Priority: Minor (was: Trivial)
Hadoop Flags: Reviewed
> Fix arguments check in documentation for WordCount v2.0
> ---
>
> Key: HADOOP-13320
> URL: https://issues.apache.org/jira/browse/HADOOP-13320
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
>Reporter: niccolo becchi
>Priority: Minor
>
> This issue is affecting the documentation page, so the code is not covered by
> any tests. It's actually visible on the page:
> https://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html#Example:_WordCount_v2.0
> On the Example: WordCount v2.0
> The actual arguments check is wrong, as it's never printing the message of
> the correct usage. So, running the code with no parameters, as in the
> following example:
> {code}
> yarn jar /var/tmp/WordCount.jar task0.WordCount2
> {code}
>
> I have got the following exception message in output:
> {code}
> Exception in thread "main" java.lang.IndexOutOfBoundsException: Index: 0,
> Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at task0.WordCount2.main(WordCount2.java:131)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {code}
> Intead than the expected friendly message:
> {code}
> Usage: wordcount [-skip skipPatternFile]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13320) Fix arguments check in documentation for WordCount v2.0
[
https://issues.apache.org/jira/browse/HADOOP-13320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366873#comment-15366873
]
Akira Ajisaka commented on HADOOP-13320:
+1
> Fix arguments check in documentation for WordCount v2.0
> ---
>
> Key: HADOOP-13320
> URL: https://issues.apache.org/jira/browse/HADOOP-13320
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
>Reporter: niccolo becchi
>Priority: Trivial
>
> This issue is affecting the documentation page, so the code is not covered by
> any tests. It's actually visible on the page:
> https://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html#Example:_WordCount_v2.0
> On the Example: WordCount v2.0
> The actual arguments check is wrong, as it's never printing the message of
> the correct usage. So, running the code with no parameters, as in the
> following example:
> {code}
> yarn jar /var/tmp/WordCount.jar task0.WordCount2
> {code}
>
> I have got the following exception message in output:
> {code}
> Exception in thread "main" java.lang.IndexOutOfBoundsException: Index: 0,
> Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at task0.WordCount2.main(WordCount2.java:131)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {code}
> Intead than the expected friendly message:
> {code}
> Usage: wordcount [-skip skipPatternFile]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13283) Support reset operation for new global storage statistics and per FS storage stats
[
https://issues.apache.org/jira/browse/HADOOP-13283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366871#comment-15366871
]
Mingliang Liu commented on HADOOP-13283:
Thanks for the review and support, [~jnp]. Thanks for the review [~hitesh].
> Support reset operation for new global storage statistics and per FS storage
> stats
> --
>
> Key: HADOOP-13283
> URL: https://issues.apache.org/jira/browse/HADOOP-13283
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Fix For: 2.8.0
>
> Attachments: HADOOP-13283.000.patch, HADOOP-13283.001.patch,
> HADOOP-13283.002.patch, HADOOP-13283.003.patch, HADOOP-13283.004.patch
>
>
> Applications may reuse the file system object across jobs and its storage
> statistics should be reset. Specially the {{FileSystem.Statistics}} supports
> reset and [HADOOP-13032] needs to keep that use case valid.
> This jira is for supporting reset operations for storage statistics.
> Thanks [~hitesh] for reporting this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13283) Support reset operation for new global storage statistics and per FS storage stats
[
https://issues.apache.org/jira/browse/HADOOP-13283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey updated HADOOP-13283:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
I have committed this to trunk, branch-2 and branch-2.8. Thanks Mingliang.
> Support reset operation for new global storage statistics and per FS storage
> stats
> --
>
> Key: HADOOP-13283
> URL: https://issues.apache.org/jira/browse/HADOOP-13283
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Fix For: 2.8.0
>
> Attachments: HADOOP-13283.000.patch, HADOOP-13283.001.patch,
> HADOOP-13283.002.patch, HADOOP-13283.003.patch, HADOOP-13283.004.patch
>
>
> Applications may reuse the file system object across jobs and its storage
> statistics should be reset. Specially the {{FileSystem.Statistics}} supports
> reset and [HADOOP-13032] needs to keep that use case valid.
> This jira is for supporting reset operations for storage statistics.
> Thanks [~hitesh] for reporting this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13350) Additional fix to LICENSE and NOTICE
[ https://issues.apache.org/jira/browse/HADOOP-13350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366861#comment-15366861 ] Akira Ajisaka commented on HADOOP-13350: Hi [~andrew.wang], would you commit it to branch-2.7.3 as well? > Additional fix to LICENSE and NOTICE > > > Key: HADOOP-13350 > URL: https://issues.apache.org/jira/browse/HADOOP-13350 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0, 3.0.0-alpha1 >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Blocker > Fix For: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1 > > Attachments: HADOOP-13350-branch2.6+2.7.01.patch, > HADOOP-13350.01.patch > > > Fix up LICENSE and NOTICE after HADOOP-12893. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13349) HADOOP_CLASSPATH vs HADOOP_USER_CLASSPATH
[ https://issues.apache.org/jira/browse/HADOOP-13349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366826#comment-15366826 ] Hudson commented on HADOOP-13349: - SUCCESS: Integrated in Hadoop-trunk-Commit #10063 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10063/]) HADOOP-13349. HADOOP_CLASSPATH vs HADOOP_USER_CLASSPATH (aw) (aw: rev a0035661c10119f134a6a686c658c098eb2e2ad8) * hadoop-yarn-project/hadoop-yarn/bin/yarn-config.sh * hadoop-common-project/hadoop-common/src/site/markdown/UnixShellGuide.md * hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh > HADOOP_CLASSPATH vs HADOOP_USER_CLASSPATH > - > > Key: HADOOP-13349 > URL: https://issues.apache.org/jira/browse/HADOOP-13349 > Project: Hadoop Common > Issue Type: Bug > Components: scripts >Affects Versions: 3.0.0-alpha1 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Blocker > Fix For: 3.0.0-alpha1 > > Attachments: HADOOP-13349.00.patch > > > There seems to be some confusion in HADOOP_CLASSPATH vs. > HADOOP_USER_CLASSPATH in the code and documentation, especially since > HADOOP_USER_CLASSPATH doesn't appear to be wired up. This needs to to get > fixed ASAP. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13348) delete spurious 'master' branch
[ https://issues.apache.org/jira/browse/HADOOP-13348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366809#comment-15366809 ] Andrew Wang commented on HADOOP-13348: -- Filed INFRA-12223 > delete spurious 'master' branch > --- > > Key: HADOOP-13348 > URL: https://issues.apache.org/jira/browse/HADOOP-13348 > Project: Hadoop Common > Issue Type: Task > Components: build >Reporter: Sean Busbey > > Right now the git repo has a branch named 'master' in addition to our 'trunk' > branch. Since 'master' is the common-place name of the 'most recent' branch > in git repositories, this is misleading to new folks. > It looks like the branch is from ~11 months ago. We should remove it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13348) delete spurious 'master' branch
[
https://issues.apache.org/jira/browse/HADOOP-13348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366794#comment-15366794
]
Andrew Wang commented on HADOOP-13348:
--
So I tried this, and it looks like the branch is protected by the INFRA hook:
{noformat}
-> % git push origin :master
remote: Rewinding refs/heads/master is forbidden.
remote:
To https://git-wip-us.apache.org/repos/asf/hadoop.git
! [remote rejected] master (pre-receive hook declined)
error: failed to push some refs to
'https://git-wip-us.apache.org/repos/asf/hadoop.git'
{noformat}
I'll file an INFRA JIRA about this.
> delete spurious 'master' branch
> ---
>
> Key: HADOOP-13348
> URL: https://issues.apache.org/jira/browse/HADOOP-13348
> Project: Hadoop Common
> Issue Type: Task
> Components: build
>Reporter: Sean Busbey
>
> Right now the git repo has a branch named 'master' in addition to our 'trunk'
> branch. Since 'master' is the common-place name of the 'most recent' branch
> in git repositories, this is misleading to new folks.
> It looks like the branch is from ~11 months ago. We should remove it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11361) Fix a race condition in MetricsSourceAdapter.updateJmxCache
[
https://issues.apache.org/jira/browse/HADOOP-11361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366789#comment-15366789
]
Yongjun Zhang commented on HADOOP-11361:
Hi [~vinayrpet] and [~ozawa],
Thanks for the discussion.
First of all, I tend to agree with Tsuyoshi on the AND/OR analysis, however, I
saw that all calls to {{getMetrics}} method passed true to {{all}} parameter,
so this change won't impact the current final result. But we certainly can
address this issue, either as a separate jira, or a side effect of the fix here.
Thanks [~vinayrpet] for the explanation of the race condition. Sorry I did
miss the "return" statement when making the earlier comment.
It looks to me that the acquirement of {{lastRecs}} and the {{updateAttrCache}}
should be protected in a same {{synchronized(this)}} block, to avoid this race
condition. And indeed the two threads Vinay described intend to have their own
builder, their own lastRecs. Say, we can break down the {{getMetrics}} method
into two parts,
{code}
void getMetricsPart1(MetricsCollectorImpl builder,
boolean all) {
builder.setRecordFilter(recordFilter).setMetricFilter(metricFilter);
synchronized(this) {
if (lastRecs == null || jmxCacheTS == 0) {
all = true; // Get all the metrics to populate the sink caches
}
}
try {
source.getMetrics(builder, all);
} catch (Exception e) {
LOG.error("Error getting metrics from source "+ name, e);
}
for (MetricsRecordBuilderImpl rb : builder) {
for (MetricsTag t : injectedTags) {
rb.add(t);
}
}
}
Iterable getMetricsPart2(MetricsCollectorImpl builder) {
lastRecs = builder.getRecords();
return lastRecs;
}
{code}
and change
{code}
if (getAllMetrics) {
MetricsCollectorImpl builder = new MetricsCollectorImpl();
getMetrics(builder, true);
}
synchronized(this) {
updateAttrCache();
if (getAllMetrics) {
updateInfoCache();
}
jmxCacheTS = Time.now();
lastRecs = null; // in case regular interval update is not running
lastRecsCleared = true;
}
{code}
to
{code}
MetricsCollectorImpl builder = NULL;
if (getAllMetrics) {
builder = new MetricsCollectorImpl();
getMetricsPart1(builder, true);
}
synchronized(this) {
if (getAllMetrics) {
getMetricsPart2(builder);
}
updateAttrCache();
if (getAllMetrics) {
updateInfoCache();
}
jmxCacheTS = Time.now();
lastRecs = null; // in case regular interval update is not running
lastRecsCleared = true;
}
{code}
This is just to throw an idea to see what's the best solution. The method names
can be adjusted.
What do you guys think?
Thanks.
> Fix a race condition in MetricsSourceAdapter.updateJmxCache
> ---
>
> Key: HADOOP-11361
> URL: https://issues.apache.org/jira/browse/HADOOP-11361
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.4.1, 2.5.1, 2.6.0
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: HADOOP-111361-003.patch, HADOOP-11361-002.patch,
> HADOOP-11361-004.patch, HADOOP-11361.patch, HDFS-7487.patch
>
>
> {noformat}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.updateAttrCache(MetricsSourceAdapter.java:247)
> at
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.updateJmxCache(MetricsSourceAdapter.java:177)
> at
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getAttribute(MetricsSourceAdapter.java:102)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11460) Deprecate shell vars
[ https://issues.apache.org/jira/browse/HADOOP-11460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366786#comment-15366786 ] Allen Wittenauer commented on HADOOP-11460: --- FYI: updated releasenote to reflect the fix in HADOOP-13349. HADOOP_USER_CLASSPATH was going to happen but didn't. > Deprecate shell vars > > > Key: HADOOP-11460 > URL: https://issues.apache.org/jira/browse/HADOOP-11460 > Project: Hadoop Common > Issue Type: Improvement > Components: scripts >Affects Versions: 3.0.0-alpha1 >Reporter: Allen Wittenauer >Assignee: John Smith > Labels: scripts, shell > Fix For: 3.0.0-alpha1 > > Attachments: HADOOP-11460-00.patch, HADOOP-11460-01.patch, > HADOOP-11460-02.patch, HADOOP-11460-03.patch, HADOOP-11460-04.patch > > > It is a very common shell pattern in 3.x to effectively replace sub-project > specific vars with generics. We should have a function that does this > replacement and provides a warning to the end user that the old shell var is > deprecated. Additionally, we should use this shell function to deprecate the > shell vars that are holdovers already. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-11460) Deprecate shell vars
[ https://issues.apache.org/jira/browse/HADOOP-11460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11460: -- Release Note: | Old | New | |: |: | | HADOOP\_HDFS\_LOG\_DIR | HADOOP\_LOG\_DIR | | HADOOP\_HDFS\_LOGFILE | HADOOP\_LOGFILE | | HADOOP\_HDFS\_NICENESS | HADOOP\_NICENESS | | HADOOP\_HDFS\_STOP\_TIMEOUT | HADOOP\_STOP\_TIMEOUT | | HADOOP\_HDFS\_PID\_DIR | HADOOP\_PID\_DIR | | HADOOP\_HDFS\_ROOT\_LOGGER | HADOOP\_ROOT\_LOGGER | | HADOOP\_HDFS\_IDENT\_STRING | HADOOP\_IDENT\_STRING | | HADOOP\_MAPRED\_LOG\_DIR | HADOOP\_LOG\_DIR | | HADOOP\_MAPRED\_LOGFILE | HADOOP\_LOGFILE | | HADOOP\_MAPRED\_NICENESS | HADOOP\_NICENESS | | HADOOP\_MAPRED\_STOP\_TIMEOUT | HADOOP\_STOP\_TIMEOUT | | HADOOP\_MAPRED\_PID\_DIR | HADOOP\_PID\_DIR | | HADOOP\_MAPRED\_ROOT\_LOGGER | HADOOP\_ROOT\_LOGGER | | HADOOP\_MAPRED\_IDENT\_STRING | HADOOP\_IDENT\_STRING | | YARN\_CONF\_DIR | HADOOP\_CONF\_DIR | | YARN\_LOG\_DIR | HADOOP\_LOG\_DIR | | YARN\_LOGFILE | HADOOP\_LOGFILE | | YARN\_NICENESS | HADOOP\_NICENESS | | YARN\_STOP\_TIMEOUT | HADOOP\_STOP\_TIMEOUT | | YARN\_PID\_DIR | HADOOP\_PID\_DIR | | YARN\_ROOT\_LOGGER | HADOOP\_ROOT\_LOGGER | | YARN\_IDENT\_STRING | HADOOP\_IDENT\_STRING | | YARN\_OPTS | HADOOP\_OPTS | | YARN\_SLAVES | HADOOP\_SLAVES | | YARN\_USER\_CLASSPATH | HADOOP\_CLASSPATH | | YARN\_USER\_CLASSPATH\_FIRST | HADOOP\_USER\_CLASSPATH\_FIRST | | KMS\_CONFIG | HADOOP\_CONF\_DIR | | KMS\_LOG | HADOOP\_LOG\_DIR | was: | Old | New | |: |: | | HADOOP\_HDFS\_LOG\_DIR | HADOOP\_LOG\_DIR | | HADOOP\_HDFS\_LOGFILE | HADOOP\_LOGFILE | | HADOOP\_HDFS\_NICENESS | HADOOP\_NICENESS | | HADOOP\_HDFS\_STOP\_TIMEOUT | HADOOP\_STOP\_TIMEOUT | | HADOOP\_HDFS\_PID\_DIR | HADOOP\_PID\_DIR | | HADOOP\_HDFS\_ROOT\_LOGGER | HADOOP\_ROOT\_LOGGER | | HADOOP\_HDFS\_IDENT\_STRING | HADOOP\_IDENT\_STRING | | HADOOP\_MAPRED\_LOG\_DIR | HADOOP\_LOG\_DIR | | HADOOP\_MAPRED\_LOGFILE | HADOOP\_LOGFILE | | HADOOP\_MAPRED\_NICENESS | HADOOP\_NICENESS | | HADOOP\_MAPRED\_STOP\_TIMEOUT | HADOOP\_STOP\_TIMEOUT | | HADOOP\_MAPRED\_PID\_DIR | HADOOP\_PID\_DIR | | HADOOP\_MAPRED\_ROOT\_LOGGER | HADOOP\_ROOT\_LOGGER | | HADOOP\_MAPRED\_IDENT\_STRING | HADOOP\_IDENT\_STRING | | YARN\_CONF\_DIR | HADOOP\_CONF\_DIR | | YARN\_LOG\_DIR | HADOOP\_LOG\_DIR | | YARN\_LOGFILE | HADOOP\_LOGFILE | | YARN\_NICENESS | HADOOP\_NICENESS | | YARN\_STOP\_TIMEOUT | HADOOP\_STOP\_TIMEOUT | | YARN\_PID\_DIR | HADOOP\_PID\_DIR | | YARN\_ROOT\_LOGGER | HADOOP\_ROOT\_LOGGER | | YARN\_IDENT\_STRING | HADOOP\_IDENT\_STRING | | YARN\_OPTS | HADOOP\_OPTS | | YARN\_SLAVES | HADOOP\_SLAVES | | YARN\_USER\_CLASSPATH | HADOOP\_USER\_CLASSPATH | | YARN\_USER\_CLASSPATH\_FIRST | HADOOP\_USER\_CLASSPATH\_FIRST | | KMS\_CONFIG | HADOOP\_CONF\_DIR | | KMS\_LOG | HADOOP\_LOG\_DIR | > Deprecate shell vars > > > Key: HADOOP-11460 > URL: https://issues.apache.org/jira/browse/HADOOP-11460 > Project: Hadoop Common > Issue Type: Improvement > Components: scripts >Affects Versions: 3.0.0-alpha1 >Reporter: Allen Wittenauer >Assignee: John Smith > Labels: scripts, shell > Fix For: 3.0.0-alpha1 > > Attachments: HADOOP-11460-00.patch, HADOOP-11460-01.patch, > HADOOP-11460-02.patch, HADOOP-11460-03.patch, HADOOP-11460-04.patch > > > It is a very common shell pattern in 3.x to effectively replace sub-project > specific vars with generics. We should have a function that does this > replacement and provides a warning to the end user that the old shell var is > deprecated. Additionally, we should use this shell function to deprecate the > shell vars that are holdovers already. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13349) HADOOP_CLASSPATH vs HADOOP_USER_CLASSPATH
[ https://issues.apache.org/jira/browse/HADOOP-13349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-13349: -- Resolution: Fixed Fix Version/s: 3.0.0-alpha1 Status: Resolved (was: Patch Available) > HADOOP_CLASSPATH vs HADOOP_USER_CLASSPATH > - > > Key: HADOOP-13349 > URL: https://issues.apache.org/jira/browse/HADOOP-13349 > Project: Hadoop Common > Issue Type: Bug > Components: scripts >Affects Versions: 3.0.0-alpha1 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Blocker > Fix For: 3.0.0-alpha1 > > Attachments: HADOOP-13349.00.patch > > > There seems to be some confusion in HADOOP_CLASSPATH vs. > HADOOP_USER_CLASSPATH in the code and documentation, especially since > HADOOP_USER_CLASSPATH doesn't appear to be wired up. This needs to to get > fixed ASAP. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13349) HADOOP_CLASSPATH vs HADOOP_USER_CLASSPATH
[ https://issues.apache.org/jira/browse/HADOOP-13349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366780#comment-15366780 ] Allen Wittenauer commented on HADOOP-13349: --- Close enough. :) Thanks. Committing this. > HADOOP_CLASSPATH vs HADOOP_USER_CLASSPATH > - > > Key: HADOOP-13349 > URL: https://issues.apache.org/jira/browse/HADOOP-13349 > Project: Hadoop Common > Issue Type: Bug > Components: scripts >Affects Versions: 3.0.0-alpha1 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Blocker > Attachments: HADOOP-13349.00.patch > > > There seems to be some confusion in HADOOP_CLASSPATH vs. > HADOOP_USER_CLASSPATH in the code and documentation, especially since > HADOOP_USER_CLASSPATH doesn't appear to be wired up. This needs to to get > fixed ASAP. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366778#comment-15366778 ] Siddharth Seth commented on HADOOP-13335: - bq. I'm pretty much going to block anything that removes the warnings. So we can keep this open, but it isn't going to go anywhere. On what technical grounds ? Printing a warning like this on branch-2 can be considered incompatible. That's ignoring the bit that the warning itself doesn't necessarily make sense. In terms of deprecation - I already see the following in trunk. "hadoop_deprecate_envvar YARN_OPTS HADOOP_OPTS" - What exactly are we trying to achieve here, between the deprecation and the push for yarn jar via this message. > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HADOOP-13335.01.patch, HADOOP-13335.02.patch, > HADOOP-13335.02_branch-2.patch, HADOOP-13335.03.patch, > HADOOP-13335.03_branch-2.patch, HADOOP-13335.04.patch > > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11993) maven enforcer plugin to ban java 8 incompatible dependencies
[ https://issues.apache.org/jira/browse/HADOOP-11993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366755#comment-15366755 ] Hudson commented on HADOOP-11993: - SUCCESS: Integrated in Hadoop-trunk-Commit #10061 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10061/]) HADOOP-11993. maven enforcer plugin to ban java 8 incompatible (ozawa: rev 0185de07676dc2f330d48453217222ad0d1cb3cf) * hadoop-project/pom.xml > maven enforcer plugin to ban java 8 incompatible dependencies > - > > Key: HADOOP-11993 > URL: https://issues.apache.org/jira/browse/HADOOP-11993 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 2.7.0 >Reporter: Steve Loughran >Assignee: Tsuyoshi Ozawa >Priority: Minor > Fix For: 3.0.0-alpha1 > > Attachments: HADOOP-11993.001.patch, HADOOP-11993.002.patch > > > It's possible to use maven-enforcer to ban dependencies; this can be used to > reject those known to be incompatible with Java 8 > [example|https://gist.github.com/HiJon89/65e34552c18e5ac9fd31] > If we set maven enforcer to do this checking, it can ensure that the 2.7+ > codebase isn't pulling in any incompatible binaries. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366738#comment-15366738 ] Allen Wittenauer commented on HADOOP-13335: --- There's a resolution here: bq. In terms of Hive (for 2.x) - yes, an unset works I'm pretty much going to block anything that removes the warnings. So we can keep this open, but it isn't going to go anywhere. > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HADOOP-13335.01.patch, HADOOP-13335.02.patch, > HADOOP-13335.02_branch-2.patch, HADOOP-13335.03.patch, > HADOOP-13335.03_branch-2.patch, HADOOP-13335.04.patch > > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth reopened HADOOP-13335: - [~aw] - lets try coming to a conclusion here. Re-opening. Kindly do not close the ticket till there's a resolution. > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HADOOP-13335.01.patch, HADOOP-13335.02.patch, > HADOOP-13335.02_branch-2.patch, HADOOP-13335.03.patch, > HADOOP-13335.03_branch-2.patch, HADOOP-13335.04.patch > > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11993) maven enforcer plugin to ban java 8 incompatible dependencies
[ https://issues.apache.org/jira/browse/HADOOP-11993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366730#comment-15366730 ] Tsuyoshi Ozawa commented on HADOOP-11993: - Committed this to trunk. Thanks [~ajisakaa] for review, thanks [~ste...@apache.org] for reporting, and thanks [~busbey] for the comment!. > maven enforcer plugin to ban java 8 incompatible dependencies > - > > Key: HADOOP-11993 > URL: https://issues.apache.org/jira/browse/HADOOP-11993 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 2.7.0 >Reporter: Steve Loughran >Assignee: Tsuyoshi Ozawa >Priority: Minor > Fix For: 3.0.0-alpha1 > > Attachments: HADOOP-11993.001.patch, HADOOP-11993.002.patch > > > It's possible to use maven-enforcer to ban dependencies; this can be used to > reject those known to be incompatible with Java 8 > [example|https://gist.github.com/HiJon89/65e34552c18e5ac9fd31] > If we set maven enforcer to do this checking, it can ensure that the 2.7+ > codebase isn't pulling in any incompatible binaries. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-11993) maven enforcer plugin to ban java 8 incompatible dependencies
[ https://issues.apache.org/jira/browse/HADOOP-11993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated HADOOP-11993: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-alpha1 Status: Resolved (was: Patch Available) > maven enforcer plugin to ban java 8 incompatible dependencies > - > > Key: HADOOP-11993 > URL: https://issues.apache.org/jira/browse/HADOOP-11993 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 2.7.0 >Reporter: Steve Loughran >Assignee: Tsuyoshi Ozawa >Priority: Minor > Fix For: 3.0.0-alpha1 > > Attachments: HADOOP-11993.001.patch, HADOOP-11993.002.patch > > > It's possible to use maven-enforcer to ban dependencies; this can be used to > reject those known to be incompatible with Java 8 > [example|https://gist.github.com/HiJon89/65e34552c18e5ac9fd31] > If we set maven enforcer to do this checking, it can ensure that the 2.7+ > codebase isn't pulling in any incompatible binaries. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11993) maven enforcer plugin to ban java 8 incompatible dependencies
[ https://issues.apache.org/jira/browse/HADOOP-11993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366725#comment-15366725 ] Tsuyoshi Ozawa commented on HADOOP-11993: - Checking this in. > maven enforcer plugin to ban java 8 incompatible dependencies > - > > Key: HADOOP-11993 > URL: https://issues.apache.org/jira/browse/HADOOP-11993 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 2.7.0 >Reporter: Steve Loughran >Assignee: Tsuyoshi Ozawa >Priority: Minor > Attachments: HADOOP-11993.001.patch, HADOOP-11993.002.patch > > > It's possible to use maven-enforcer to ban dependencies; this can be used to > reject those known to be incompatible with Java 8 > [example|https://gist.github.com/HiJon89/65e34552c18e5ac9fd31] > If we set maven enforcer to do this checking, it can ensure that the 2.7+ > codebase isn't pulling in any incompatible binaries. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-13335: -- Resolution: Won't Fix Status: Resolved (was: Patch Available) Closing this as won't fix. > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HADOOP-13335.01.patch, HADOOP-13335.02.patch, > HADOOP-13335.02_branch-2.patch, HADOOP-13335.03.patch, > HADOOP-13335.03_branch-2.patch, HADOOP-13335.04.patch > > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366571#comment-15366571 ] Siddharth Seth commented on HADOOP-13335: - bq. It's not and I'm still not sure why you seem to insist this is true when both Vinod and I have said that there isn't any intent on it going away anytime remotely soon. Anytime soon - no. From your comments, it does look like at least you think this is a candidate for 4.x removal. In terms of Hive (for 2.x) - yes, an unset works - for the most part anyway. However, this warning is not required - and should not be seen by other systems / users. Unsetting variables is a way to suppress the message, however it isn't an optimal or efficient way to do this. Is this deprecation, what's the public API? There's multiple options here. 1. Get rid of the warning altogether - which I think is the correct thing to do. (Haven't heard a good reason to keep it) 2. Add a variable to deprecate this specific warning. 3. Add an option to deprecate all such warnings (and apparently there's a lot of these in trunk). > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HADOOP-13335.01.patch, HADOOP-13335.02.patch, > HADOOP-13335.02_branch-2.patch, HADOOP-13335.03.patch, > HADOOP-13335.03_branch-2.patch, HADOOP-13335.04.patch > > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13346) DelegationTokenAuthenticationHandler writes via closed writer
[
https://issues.apache.org/jira/browse/HADOOP-13346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366525#comment-15366525
]
Xiao Chen commented on HADOOP-13346:
Thanks for reporting and fixing the issue [~gchanan], LGTM overall. Could you
fix up the checkstyle? +1 (non-binding) after that.
> DelegationTokenAuthenticationHandler writes via closed writer
> -
>
> Key: HADOOP-13346
> URL: https://issues.apache.org/jira/browse/HADOOP-13346
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
>Priority: Minor
> Attachments: HADOOP-13346.patch
>
>
> By default, jackson's ObjectMapper closes the writer after writing, so in the
> following code
> {code}
> ObjectMapper jsonMapper = new ObjectMapper();
> jsonMapper.writeValue(writer, map);
> writer.write(ENTER);
> {code}
> (https://github.com/apache/hadoop/blob/8a9d293dd60f6d51e1574e412d40746ba8175fe1/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticationHandler.java#L280-L282)
> writer.write actually writes to a closed stream. This doesn't seem to cause
> a problem with the version of jetty that hadoop uses (those just ignore
> closes), but causes problems on later verisons of jetty -- I hit this on
> jetty 8 while implementing SOLR-9200.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13350) Additional fix to LICENSE and NOTICE
[ https://issues.apache.org/jira/browse/HADOOP-13350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366505#comment-15366505 ] Hudson commented on HADOOP-13350: - SUCCESS: Integrated in Hadoop-trunk-Commit #10059 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10059/]) HADOOP-13350. Additional fix to LICENSE and NOTICE. Contributed by Xiao (wang: rev 0e0f45f0ec3d8279610e8af25f60f4abb668d55a) * LICENSE.txt * NOTICE.txt > Additional fix to LICENSE and NOTICE > > > Key: HADOOP-13350 > URL: https://issues.apache.org/jira/browse/HADOOP-13350 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0, 3.0.0-alpha1 >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Blocker > Fix For: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1 > > Attachments: HADOOP-13350-branch2.6+2.7.01.patch, > HADOOP-13350.01.patch > > > Fix up LICENSE and NOTICE after HADOOP-12893. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13350) Additional fix to LICENSE and NOTICE
[ https://issues.apache.org/jira/browse/HADOOP-13350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HADOOP-13350: --- Target Version/s: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1 (was: 2.8.0, 3.0.0-alpha1) > Additional fix to LICENSE and NOTICE > > > Key: HADOOP-13350 > URL: https://issues.apache.org/jira/browse/HADOOP-13350 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0, 3.0.0-alpha1 >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Blocker > Fix For: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1 > > Attachments: HADOOP-13350-branch2.6+2.7.01.patch, > HADOOP-13350.01.patch > > > Fix up LICENSE and NOTICE after HADOOP-12893. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13350) Additional fix to LICENSE and NOTICE
[ https://issues.apache.org/jira/browse/HADOOP-13350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366483#comment-15366483 ] Xiao Chen commented on HADOOP-13350: Thanks you, Andrew and Akira! > Additional fix to LICENSE and NOTICE > > > Key: HADOOP-13350 > URL: https://issues.apache.org/jira/browse/HADOOP-13350 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0, 3.0.0-alpha1 >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Blocker > Fix For: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1 > > Attachments: HADOOP-13350-branch2.6+2.7.01.patch, > HADOOP-13350.01.patch > > > Fix up LICENSE and NOTICE after HADOOP-12893. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13346) DelegationTokenAuthenticationHandler writes via closed writer
[
https://issues.apache.org/jira/browse/HADOOP-13346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HADOOP-13346:
-
Assignee: Gregory Chanan
> DelegationTokenAuthenticationHandler writes via closed writer
> -
>
> Key: HADOOP-13346
> URL: https://issues.apache.org/jira/browse/HADOOP-13346
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
>Priority: Minor
> Attachments: HADOOP-13346.patch
>
>
> By default, jackson's ObjectMapper closes the writer after writing, so in the
> following code
> {code}
> ObjectMapper jsonMapper = new ObjectMapper();
> jsonMapper.writeValue(writer, map);
> writer.write(ENTER);
> {code}
> (https://github.com/apache/hadoop/blob/8a9d293dd60f6d51e1574e412d40746ba8175fe1/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticationHandler.java#L280-L282)
> writer.write actually writes to a closed stream. This doesn't seem to cause
> a problem with the version of jetty that hadoop uses (those just ignore
> closes), but causes problems on later verisons of jetty -- I hit this on
> jetty 8 while implementing SOLR-9200.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13350) Additional fix to LICENSE and NOTICE
[ https://issues.apache.org/jira/browse/HADOOP-13350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HADOOP-13350: - Resolution: Fixed Fix Version/s: 3.0.0-alpha1 2.6.5 2.7.3 2.8.0 Status: Resolved (was: Patch Available) Committed very carefully to the relevant branches. Thanks for working on this Xiao! > Additional fix to LICENSE and NOTICE > > > Key: HADOOP-13350 > URL: https://issues.apache.org/jira/browse/HADOOP-13350 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0, 3.0.0-alpha1 >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Blocker > Fix For: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1 > > Attachments: HADOOP-13350-branch2.6+2.7.01.patch, > HADOOP-13350.01.patch > > > Fix up LICENSE and NOTICE after HADOOP-12893. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13350) Additional fix to LICENSE and NOTICE
[ https://issues.apache.org/jira/browse/HADOOP-13350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366476#comment-15366476 ] Akira Ajisaka commented on HADOOP-13350: +1, thanks [~xiaochen] and [~andrew.wang] for the great work. > Additional fix to LICENSE and NOTICE > > > Key: HADOOP-13350 > URL: https://issues.apache.org/jira/browse/HADOOP-13350 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0, 3.0.0-alpha1 >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Blocker > Attachments: HADOOP-13350-branch2.6+2.7.01.patch, > HADOOP-13350.01.patch > > > Fix up LICENSE and NOTICE after HADOOP-12893. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13350) Additional fix to LICENSE and NOTICE
[ https://issues.apache.org/jira/browse/HADOOP-13350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366472#comment-15366472 ] Andrew Wang commented on HADOOP-13350: -- LGTM thanks Xiao, let's get this stuff in. > Additional fix to LICENSE and NOTICE > > > Key: HADOOP-13350 > URL: https://issues.apache.org/jira/browse/HADOOP-13350 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0, 3.0.0-alpha1 >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Blocker > Attachments: HADOOP-13350-branch2.6+2.7.01.patch, > HADOOP-13350.01.patch > > > Fix up LICENSE and NOTICE after HADOOP-12893. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13350) Additional fix to LICENSE and NOTICE
[
https://issues.apache.org/jira/browse/HADOOP-13350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366467#comment-15366467
]
Hadoop QA commented on HADOOP-13350:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 0m 48s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12816673/HADOOP-13350-branch2.6%2B2.7.01.patch
|
| JIRA Issue | HADOOP-13350 |
| Optional Tests | asflicense |
| uname | Linux fb4047d2e140 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / b8f93cd |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9939/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Additional fix to LICENSE and NOTICE
>
>
> Key: HADOOP-13350
> URL: https://issues.apache.org/jira/browse/HADOOP-13350
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Blocker
> Attachments: HADOOP-13350-branch2.6+2.7.01.patch,
> HADOOP-13350.01.patch
>
>
> Fix up LICENSE and NOTICE after HADOOP-12893.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12893) Verify LICENSE.txt and NOTICE.txt
[ https://issues.apache.org/jira/browse/HADOOP-12893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366464#comment-15366464 ] Xiao Chen commented on HADOOP-12893: Thanks Andrew for the suggestion. I created HADOOP-13350 and attached the addendum patches there. > Verify LICENSE.txt and NOTICE.txt > - > > Key: HADOOP-12893 > URL: https://issues.apache.org/jira/browse/HADOOP-12893 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1 >Reporter: Allen Wittenauer >Assignee: Xiao Chen >Priority: Blocker > Fix For: 2.7.3, 2.6.5 > > Attachments: HADOOP-12893-addendum-branch-2.7.01.patch, > HADOOP-12893.002.patch, HADOOP-12893.003.patch, HADOOP-12893.004.patch, > HADOOP-12893.005.patch, HADOOP-12893.006.patch, HADOOP-12893.007.patch, > HADOOP-12893.008.patch, HADOOP-12893.009.patch, HADOOP-12893.01.patch, > HADOOP-12893.011.patch, HADOOP-12893.012.patch, HADOOP-12893.10.patch, > HADOOP-12893.addendum-trunk.01.patch, HADOOP-12893.branch-2.01.patch, > HADOOP-12893.branch-2.6.01.patch, HADOOP-12893.branch-2.7.01.patch, > HADOOP-12893.branch-2.7.02.patch, HADOOP-12893.branch-2.7.3.01.patch > > > We have many bundled dependencies in both the source and the binary artifacts > that are not in LICENSE.txt and NOTICE.txt. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13350) Additional fix to LICENSE and NOTICE
[ https://issues.apache.org/jira/browse/HADOOP-13350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366462#comment-15366462 ] Xiao Chen commented on HADOOP-13350: According to the spreadsheet (https://docs.google.com/spreadsheets/d/1HL2b4PSdQMZDVJmum1GIKrteFr2oainApTLiJTPnfd4/edit#gid=0), jcip is only in 2.8.0 and 3.0. So attached a different patch for 2.6 and 2.7. > Additional fix to LICENSE and NOTICE > > > Key: HADOOP-13350 > URL: https://issues.apache.org/jira/browse/HADOOP-13350 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0, 3.0.0-alpha1 >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Blocker > Attachments: HADOOP-13350-branch2.6+2.7.01.patch, > HADOOP-13350.01.patch > > > Fix up LICENSE and NOTICE after HADOOP-12893. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13350) Additional fix to LICENSE and NOTICE
[ https://issues.apache.org/jira/browse/HADOOP-13350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HADOOP-13350: --- Attachment: HADOOP-13350-branch2.6+2.7.01.patch > Additional fix to LICENSE and NOTICE > > > Key: HADOOP-13350 > URL: https://issues.apache.org/jira/browse/HADOOP-13350 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0, 3.0.0-alpha1 >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Blocker > Attachments: HADOOP-13350-branch2.6+2.7.01.patch, > HADOOP-13350.01.patch > > > Fix up LICENSE and NOTICE after HADOOP-12893. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13350) Additional fix to LICENSE and NOTICE
[ https://issues.apache.org/jira/browse/HADOOP-13350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HADOOP-13350: --- Status: Patch Available (was: Open) > Additional fix to LICENSE and NOTICE > > > Key: HADOOP-13350 > URL: https://issues.apache.org/jira/browse/HADOOP-13350 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0, 3.0.0-alpha1 >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Blocker > Attachments: HADOOP-13350.01.patch > > > Fix up LICENSE and NOTICE after HADOOP-12893. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13350) Additional fix to LICENSE and NOTICE
[ https://issues.apache.org/jira/browse/HADOOP-13350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HADOOP-13350: --- Attachment: HADOOP-13350.01.patch > Additional fix to LICENSE and NOTICE > > > Key: HADOOP-13350 > URL: https://issues.apache.org/jira/browse/HADOOP-13350 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0, 3.0.0-alpha1 >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Blocker > Attachments: HADOOP-13350.01.patch > > > Fix up LICENSE and NOTICE after HADOOP-12893. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13350) Additional fix to LICENSE and NOTICE
Xiao Chen created HADOOP-13350: -- Summary: Additional fix to LICENSE and NOTICE Key: HADOOP-13350 URL: https://issues.apache.org/jira/browse/HADOOP-13350 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.8.0, 3.0.0-alpha1 Reporter: Xiao Chen Assignee: Xiao Chen Priority: Blocker Fix up LICENSE and NOTICE after HADOOP-12893. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12893) Verify LICENSE.txt and NOTICE.txt
[ https://issues.apache.org/jira/browse/HADOOP-12893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366441#comment-15366441 ] Andrew Wang commented on HADOOP-12893: -- Filed NOTICE questions over at LEGAL-262. > Verify LICENSE.txt and NOTICE.txt > - > > Key: HADOOP-12893 > URL: https://issues.apache.org/jira/browse/HADOOP-12893 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1 >Reporter: Allen Wittenauer >Assignee: Xiao Chen >Priority: Blocker > Fix For: 2.7.3, 2.6.5 > > Attachments: HADOOP-12893-addendum-branch-2.7.01.patch, > HADOOP-12893.002.patch, HADOOP-12893.003.patch, HADOOP-12893.004.patch, > HADOOP-12893.005.patch, HADOOP-12893.006.patch, HADOOP-12893.007.patch, > HADOOP-12893.008.patch, HADOOP-12893.009.patch, HADOOP-12893.01.patch, > HADOOP-12893.011.patch, HADOOP-12893.012.patch, HADOOP-12893.10.patch, > HADOOP-12893.addendum-trunk.01.patch, HADOOP-12893.branch-2.01.patch, > HADOOP-12893.branch-2.6.01.patch, HADOOP-12893.branch-2.7.01.patch, > HADOOP-12893.branch-2.7.02.patch, HADOOP-12893.branch-2.7.3.01.patch > > > We have many bundled dependencies in both the source and the binary artifacts > that are not in LICENSE.txt and NOTICE.txt. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-10465) Fix use of generics within SortedMapWritable
[ https://issues.apache.org/jira/browse/HADOOP-10465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366436#comment-15366436 ] Akira Ajisaka edited comment on HADOOP-10465 at 7/7/16 5:21 PM: This breaks HADOOP-13338. Hive does not compile against 2.8.0-SNAPSHOT. Given this is an incompatible change, I'm +1 for reverting it from branch-2 and branch-2.8. I'll revert it in Jul. 9 PDT if there are no objections. was (Author: ajisakaa): This breaks HADOOP-13338. Hive does not compile against 2.8.0-SNAPSHOT. Given this is an incompatible change, I'm +1 for reverting branch-2 and branch-2.8. I'll revert it in Jul. 9 PDT if there are no objections. > Fix use of generics within SortedMapWritable > > > Key: HADOOP-10465 > URL: https://issues.apache.org/jira/browse/HADOOP-10465 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.0.0-alpha1 >Reporter: Bertrand Dechoux >Assignee: Bertrand Dechoux >Priority: Minor > Labels: BB2015-05-TBR > Fix For: 2.8.0 > > Attachments: HADOOP-10465-3.patch, HADOOP-10465-4.patch > > > SortedMapWritable could use Generics the right way so that there is no > warning within the implementation and more important for the consumer. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-10465) Fix use of generics within SortedMapWritable
[ https://issues.apache.org/jira/browse/HADOOP-10465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366436#comment-15366436 ] Akira Ajisaka commented on HADOOP-10465: This breaks HADOOP-13338. Hive does not compile against 2.8.0-SNAPSHOT. Given this is an incompatible change, I'm +1 for reverting branch-2 and branch-2.8. I'll revert it in Jul. 9 PDT if there are no objections. > Fix use of generics within SortedMapWritable > > > Key: HADOOP-10465 > URL: https://issues.apache.org/jira/browse/HADOOP-10465 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.0.0-alpha1 >Reporter: Bertrand Dechoux >Assignee: Bertrand Dechoux >Priority: Minor > Labels: BB2015-05-TBR > Fix For: 2.8.0 > > Attachments: HADOOP-10465-3.patch, HADOOP-10465-4.patch > > > SortedMapWritable could use Generics the right way so that there is no > warning within the implementation and more important for the consumer. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12893) Verify LICENSE.txt and NOTICE.txt
[ https://issues.apache.org/jira/browse/HADOOP-12893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366431#comment-15366431 ] Andrew Wang commented on HADOOP-12893: -- In the interest of "just getting it done", I'm okay committing this for now and then fixing it later based on LEGAL input. Xiao, do you mind filing a new JIRA with this addendum patch? I'll +1 and commit it. If these changes also apply to the branch-2's, LMK via the target versions. I'll also file the LEGAL JIRA and link it here, since this seems like our umbrella JIRA for L&N changes. > Verify LICENSE.txt and NOTICE.txt > - > > Key: HADOOP-12893 > URL: https://issues.apache.org/jira/browse/HADOOP-12893 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1 >Reporter: Allen Wittenauer >Assignee: Xiao Chen >Priority: Blocker > Fix For: 2.7.3, 2.6.5 > > Attachments: HADOOP-12893-addendum-branch-2.7.01.patch, > HADOOP-12893.002.patch, HADOOP-12893.003.patch, HADOOP-12893.004.patch, > HADOOP-12893.005.patch, HADOOP-12893.006.patch, HADOOP-12893.007.patch, > HADOOP-12893.008.patch, HADOOP-12893.009.patch, HADOOP-12893.01.patch, > HADOOP-12893.011.patch, HADOOP-12893.012.patch, HADOOP-12893.10.patch, > HADOOP-12893.addendum-trunk.01.patch, HADOOP-12893.branch-2.01.patch, > HADOOP-12893.branch-2.6.01.patch, HADOOP-12893.branch-2.7.01.patch, > HADOOP-12893.branch-2.7.02.patch, HADOOP-12893.branch-2.7.3.01.patch > > > We have many bundled dependencies in both the source and the binary artifacts > that are not in LICENSE.txt and NOTICE.txt. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13349) HADOOP_CLASSPATH vs HADOOP_USER_CLASSPATH
[ https://issues.apache.org/jira/browse/HADOOP-13349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366416#comment-15366416 ] Andrew Wang commented on HADOOP-13349: -- So IIUC, the confusion is because we have an option for HADOOP_USER_CLASSPATH_FIRST, but what that does is put the user's HADOOP_CLASSPATH (or YARN_USER_CLASSPATH) first. There is no HADOOP_USER_CLASSPATH, despite the naming of HADOOP_USER_CLASSPATH_FIRST. Assuming my interpretation is correct, I'm +1 since these are basically doc changes. Thanks for finding and fixing this Allen! > HADOOP_CLASSPATH vs HADOOP_USER_CLASSPATH > - > > Key: HADOOP-13349 > URL: https://issues.apache.org/jira/browse/HADOOP-13349 > Project: Hadoop Common > Issue Type: Bug > Components: scripts >Affects Versions: 3.0.0-alpha1 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Blocker > Attachments: HADOOP-13349.00.patch > > > There seems to be some confusion in HADOOP_CLASSPATH vs. > HADOOP_USER_CLASSPATH in the code and documentation, especially since > HADOOP_USER_CLASSPATH doesn't appear to be wired up. This needs to to get > fixed ASAP. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11993) maven enforcer plugin to ban java 8 incompatible dependencies
[ https://issues.apache.org/jira/browse/HADOOP-11993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366395#comment-15366395 ] Akira Ajisaka commented on HADOOP-11993: +1, thanks Tsuyoshi. > maven enforcer plugin to ban java 8 incompatible dependencies > - > > Key: HADOOP-11993 > URL: https://issues.apache.org/jira/browse/HADOOP-11993 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 2.7.0 >Reporter: Steve Loughran >Assignee: Tsuyoshi Ozawa >Priority: Minor > Attachments: HADOOP-11993.001.patch, HADOOP-11993.002.patch > > > It's possible to use maven-enforcer to ban dependencies; this can be used to > reject those known to be incompatible with Java 8 > [example|https://gist.github.com/HiJon89/65e34552c18e5ac9fd31] > If we set maven enforcer to do this checking, it can ensure that the 2.7+ > codebase isn't pulling in any incompatible binaries. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13135) Encounter response code 500 when accessing /metrics endpoint
[
https://issues.apache.org/jira/browse/HADOOP-13135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HADOOP-13135:
Description:
When accessing /metrics endpoint on hbase master through hadoop 2.7.1, I got:
{code}
HTTP ERROR 500
Problem accessing /metrics. Reason:
INTERNAL_SERVER_ERROR
Caused by:
java.lang.NullPointerException
at
org.apache.hadoop.http.HttpServer2.isInstrumentationAccessAllowed(HttpServer2.java:1029)
at
org.apache.hadoop.metrics.MetricsServlet.doGet(MetricsServlet.java:109)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at
org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:113)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
{code}
[~ajisakaa] suggested that code 500 should be 404 (NOT FOUND).
was:
When accessing /metrics endpoint on hbase master through hadoop 2.7.1, I got:
{code}
HTTP ERROR 500
Problem accessing /metrics. Reason:
INTERNAL_SERVER_ERROR
Caused by:
java.lang.NullPointerException
at
org.apache.hadoop.http.HttpServer2.isInstrumentationAccessAllowed(HttpServer2.java:1029)
at
org.apache.hadoop.metrics.MetricsServlet.doGet(MetricsServlet.java:109)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at
org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:113)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
{code}
[~ajisakaa] suggested that code 500 should be 404 (NOT FOUND).
> Encounter response code 500 when accessing /metrics endpoint
>
>
> Key: HADOOP-13135
> URL: https://issues.apache.org/jira/browse/HADOOP-13135
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: Ted Yu
>
> When accessing /metrics endpoint on hbase master through hadoop 2.7.1, I got:
> {code}
> HTTP ERROR 500
> Problem accessing /metrics. Reason:
> INTERNAL_SERVER_ERROR
> Caused by:
> java.lang.NullPointerException
> at
> org.apache.hadoop.http.HttpServer2.isInstrumentationAccessAllowed(HttpServer2.java:1029)
> at
> org.apache.hadoop.metrics.MetricsServlet.doGet(MetricsServlet.java:109)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at
> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
> at
> org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:113)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> {code}
> [~ajisakaa] suggested that code 500 should be 404 (NOT FOUND).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13349) HADOOP_CLASSPATH vs HADOOP_USER_CLASSPATH
[
https://issues.apache.org/jira/browse/HADOOP-13349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366282#comment-15366282
]
Hadoop QA commented on HADOOP-13349:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
47s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
13s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
8s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
43s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
27s{color} | {color:green} hadoop-yarn in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 57s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12816648/HADOOP-13349.00.patch
|
| JIRA Issue | HADOOP-13349 |
| Optional Tests | asflicense mvnsite unit shellcheck shelldocs |
| uname | Linux 255f652687ed 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / a3f93be |
| shellcheck | v0.4.4 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9937/testReport/ |
| modules | C: hadoop-common-project/hadoop-common
hadoop-yarn-project/hadoop-yarn U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9937/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> HADOOP_CLASSPATH vs HADOOP_USER_CLASSPATH
> -
>
> Key: HADOOP-13349
> URL: https://issues.apache.org/jira/browse/HADOOP-13349
> Project: Hadoop Common
> Issue Type: Bug
> Components: scripts
>Affects Versions: 3.0.0-alpha1
>Reporter: Allen Wittenauer
>Assignee: Allen Wittenauer
>Priority: Blocker
> Attachments: HADOOP-13349.00.patch
>
>
> There seems to be some confusion in HADOOP_CLASSPATH vs.
> HADOOP_USER_CLASSPATH in the code and documentation, especially since
> HADOOP_USER_CLASSPATH doesn't appear to be wired up. This needs to to get
> fixed ASAP.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13349) HADOOP_CLASSPATH vs HADOOP_USER_CLASSPATH
[ https://issues.apache.org/jira/browse/HADOOP-13349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366239#comment-15366239 ] Allen Wittenauer edited comment on HADOOP-13349 at 7/7/16 3:20 PM: --- -00: * change all instances of HADOOP\_USER\_CLASSPATH to HADOOP\_CLASSPATH Hooray! Almost all of these are documentation changes. The fact is, HADOOP\_USER\_CLASSPATH is a better name. But since Windows isn't changing, it just makes sense to go back to HADOOP\_CLASSPATH so that documentation is similar. If anyone thinks HADOOP\_USER\_CLASSPATH really is the way to go forward, it's no big deal to change it. This patch becomes more of a code change than a documentation change, however. It's worth pointing out even the unit tests were written to HADOOP\_CLASSPATH. :( was (Author: aw): -00: * change all instances of HADOOP\_USER\_CLASSPATH to HADOOP\_CLASSPATH Hooray! Almost all of these are documentation changes. The fact is, HADOOP\_USER\_CLASSPATH is a better name. But since Windows isn't changing, it just makes sense to go back to HADOOP\_CLASSPATH so that documentation is similar. If anything thinks HADOOP\_USER\_CLASSPATH really is the way to go forward, it's no big deal to change it. This patch becomes more of a code change than a documentation change, however. It's worth pointing out even the unit tests were written to HADOOP\_CLASSPATH. :( > HADOOP_CLASSPATH vs HADOOP_USER_CLASSPATH > - > > Key: HADOOP-13349 > URL: https://issues.apache.org/jira/browse/HADOOP-13349 > Project: Hadoop Common > Issue Type: Bug > Components: scripts >Affects Versions: 3.0.0-alpha1 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Blocker > Attachments: HADOOP-13349.00.patch > > > There seems to be some confusion in HADOOP_CLASSPATH vs. > HADOOP_USER_CLASSPATH in the code and documentation, especially since > HADOOP_USER_CLASSPATH doesn't appear to be wired up. This needs to to get > fixed ASAP. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13349) HADOOP_CLASSPATH vs HADOOP_USER_CLASSPATH
[ https://issues.apache.org/jira/browse/HADOOP-13349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-13349: -- Attachment: HADOOP-13349.00.patch > HADOOP_CLASSPATH vs HADOOP_USER_CLASSPATH > - > > Key: HADOOP-13349 > URL: https://issues.apache.org/jira/browse/HADOOP-13349 > Project: Hadoop Common > Issue Type: Bug > Components: scripts >Affects Versions: 3.0.0-alpha1 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Blocker > Attachments: HADOOP-13349.00.patch > > > There seems to be some confusion in HADOOP_CLASSPATH vs. > HADOOP_USER_CLASSPATH in the code and documentation, especially since > HADOOP_USER_CLASSPATH doesn't appear to be wired up. This needs to to get > fixed ASAP. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13349) HADOOP_CLASSPATH vs HADOOP_USER_CLASSPATH
[ https://issues.apache.org/jira/browse/HADOOP-13349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-13349: -- Attachment: (was: HADOOP-13342.00.patch) > HADOOP_CLASSPATH vs HADOOP_USER_CLASSPATH > - > > Key: HADOOP-13349 > URL: https://issues.apache.org/jira/browse/HADOOP-13349 > Project: Hadoop Common > Issue Type: Bug > Components: scripts >Affects Versions: 3.0.0-alpha1 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Blocker > Attachments: HADOOP-13349.00.patch > > > There seems to be some confusion in HADOOP_CLASSPATH vs. > HADOOP_USER_CLASSPATH in the code and documentation, especially since > HADOOP_USER_CLASSPATH doesn't appear to be wired up. This needs to to get > fixed ASAP. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HADOOP-13349) HADOOP_CLASSPATH vs HADOOP_USER_CLASSPATH
[
https://issues.apache.org/jira/browse/HADOOP-13349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HADOOP-13349:
--
Comment: was deleted
(was: | (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s{color}
| {color:red} HADOOP-13349 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12816644/HADOOP-13342.00.patch
|
| JIRA Issue | HADOOP-13349 |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9936/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
)
> HADOOP_CLASSPATH vs HADOOP_USER_CLASSPATH
> -
>
> Key: HADOOP-13349
> URL: https://issues.apache.org/jira/browse/HADOOP-13349
> Project: Hadoop Common
> Issue Type: Bug
> Components: scripts
>Affects Versions: 3.0.0-alpha1
>Reporter: Allen Wittenauer
>Assignee: Allen Wittenauer
>Priority: Blocker
> Attachments: HADOOP-13349.00.patch
>
>
> There seems to be some confusion in HADOOP_CLASSPATH vs.
> HADOOP_USER_CLASSPATH in the code and documentation, especially since
> HADOOP_USER_CLASSPATH doesn't appear to be wired up. This needs to to get
> fixed ASAP.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13349) HADOOP_CLASSPATH vs HADOOP_USER_CLASSPATH
[ https://issues.apache.org/jira/browse/HADOOP-13349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-13349: -- Status: Patch Available (was: Open) > HADOOP_CLASSPATH vs HADOOP_USER_CLASSPATH > - > > Key: HADOOP-13349 > URL: https://issues.apache.org/jira/browse/HADOOP-13349 > Project: Hadoop Common > Issue Type: Bug > Components: scripts >Affects Versions: 3.0.0-alpha1 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Blocker > Attachments: HADOOP-13342.00.patch > > > There seems to be some confusion in HADOOP_CLASSPATH vs. > HADOOP_USER_CLASSPATH in the code and documentation, especially since > HADOOP_USER_CLASSPATH doesn't appear to be wired up. This needs to to get > fixed ASAP. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13349) HADOOP_CLASSPATH vs HADOOP_USER_CLASSPATH
[
https://issues.apache.org/jira/browse/HADOOP-13349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366249#comment-15366249
]
Hadoop QA commented on HADOOP-13349:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s{color}
| {color:red} HADOOP-13349 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12816644/HADOOP-13342.00.patch
|
| JIRA Issue | HADOOP-13349 |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9936/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> HADOOP_CLASSPATH vs HADOOP_USER_CLASSPATH
> -
>
> Key: HADOOP-13349
> URL: https://issues.apache.org/jira/browse/HADOOP-13349
> Project: Hadoop Common
> Issue Type: Bug
> Components: scripts
>Affects Versions: 3.0.0-alpha1
>Reporter: Allen Wittenauer
>Assignee: Allen Wittenauer
>Priority: Blocker
> Attachments: HADOOP-13342.00.patch
>
>
> There seems to be some confusion in HADOOP_CLASSPATH vs.
> HADOOP_USER_CLASSPATH in the code and documentation, especially since
> HADOOP_USER_CLASSPATH doesn't appear to be wired up. This needs to to get
> fixed ASAP.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13349) HADOOP_CLASSPATH vs HADOOP_USER_CLASSPATH
[ https://issues.apache.org/jira/browse/HADOOP-13349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-13349: -- Attachment: HADOOP-13342.00.patch -00: * change all instances of HADOOP\_USER\_CLASSPATH to HADOOP\_CLASSPATH Hooray! Almost all of these are documentation changes. The fact is, HADOOP\_USER\_CLASSPATH is a better name. But since Windows isn't changing, it just makes sense to go back to HADOOP\_CLASSPATH so that documentation is similar. If anything thinks HADOOP\_USER\_CLASSPATH really is the way to go forward, it's no big deal to change it. This patch becomes more of a code change than a documentation change, however. It's worth pointing out even the unit tests were written to HADOOP\_CLASSPATH. :( > HADOOP_CLASSPATH vs HADOOP_USER_CLASSPATH > - > > Key: HADOOP-13349 > URL: https://issues.apache.org/jira/browse/HADOOP-13349 > Project: Hadoop Common > Issue Type: Bug > Components: scripts >Affects Versions: 3.0.0-alpha1 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Blocker > Attachments: HADOOP-13342.00.patch > > > There seems to be some confusion in HADOOP_CLASSPATH vs. > HADOOP_USER_CLASSPATH in the code and documentation, especially since > HADOOP_USER_CLASSPATH doesn't appear to be wired up. This needs to to get > fixed ASAP. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11361) Fix a race condition in MetricsSourceAdapter.updateJmxCache
[
https://issues.apache.org/jira/browse/HADOOP-11361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366173#comment-15366173
]
Tsuyoshi Ozawa commented on HADOOP-11361:
-
Thank you for explanation and reviews.I took a look more deeper. I think we
should check overall semantics of MetricsSourceAdapter instead of doing
workaround.
At first, I suspect that {{getMetrics}} has a semantics bug. A following
condition check whether infoCache should be updated.
{code}
if (lastRecs == null && jmxCacheTS == 0) {
all = true; // Get all the metrics to populate the sink caches
}
{code}
{{infoCache}} should be updated in following cases:
1. After updateAttrCache is called. It is expressed as lastRecs is null.
2. Before initialization is done - before calling {{updateJmxCache}}. It is
expressed as {{jmxCacheTS == 0}}.
I think these condition should be connected with {{OR}} not {{AND}}, so it can
be fixed as follows:
{code}
if (lastRecs == null || jmxCacheTS == 0) {
all = true; // Get all the metrics to populate the sink caches
}
{code}
What do you think?
Next, the NPE related problem:
{quote}
Race condition is there between two threads calling updateJmxCache() at same
time.
{quote}
You're right. v3 patch fixed the race condition, but it introduced deadlock
between JMXJsonServlet and ResourceManager's MetricSystem as Jason mentioned on
HADOOP-12594:
{quote}
The timer thread has the MetricsSystemImpl lock and is trying to grab the
MetricsSourceAdapter lock. In the meantime the JMX thread has the
MetricsSourceAdapter lock and is trying to grab the MetricsSystemImpl lock. The
locking order isn't consistent so we deadlocked.
{quote}
Brahma's solution is a bit tricky, so please let me confirm for a while.
> Fix a race condition in MetricsSourceAdapter.updateJmxCache
> ---
>
> Key: HADOOP-11361
> URL: https://issues.apache.org/jira/browse/HADOOP-11361
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.4.1, 2.5.1, 2.6.0
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: HADOOP-111361-003.patch, HADOOP-11361-002.patch,
> HADOOP-11361-004.patch, HADOOP-11361.patch, HDFS-7487.patch
>
>
> {noformat}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.updateAttrCache(MetricsSourceAdapter.java:247)
> at
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.updateJmxCache(MetricsSourceAdapter.java:177)
> at
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getAttribute(MetricsSourceAdapter.java:102)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13340) Compress Hadoop Archive output
[ https://issues.apache.org/jira/browse/HADOOP-13340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366181#comment-15366181 ] Jason Lowe commented on HADOOP-13340: - Yes a splittable codec could be used to accomplish something simmilar, but again the splits won't necessarily occur on the file boundaries -- depending upon the codec block size a significant amount of data may need to be decompressed and thrown away before arriving at the original file data within the codec block. (Note that a larger codec block size could compress multiple original small files and get a better overall compression ratio, so there's a tradeoff.) As I mentioned above, if the intent is to compress the original files on file boundaries when adding them to the har then IMHO the problem is the original files should have been compressed in the first place before trying to do the har. Otherwise those intending to consume the original files will find compressed data in the har rather than original file data and will need to know that they need a codec to get back to the original file contents. If the purpose of this request is to provide transparent compression within the har then that will need a splittable codec or reset the codec on file boundaries and set flags in the har (in a backwards-compatible manner) to indicate how compression was performed so the resulting input stream can compensate for how the data is laid out within the har. > Compress Hadoop Archive output > -- > > Key: HADOOP-13340 > URL: https://issues.apache.org/jira/browse/HADOOP-13340 > Project: Hadoop Common > Issue Type: New Feature > Components: tools >Affects Versions: 2.5.0 >Reporter: Duc Le Tu > Labels: features, performance > > Why Hadoop Archive tool cannot compress output like other map-reduce job? > I used some options like -D mapreduce.output.fileoutputformat.compress=true > -D > mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.GzipCodec > but it's not work. Did I wrong somewhere? > If not, please support option for compress output of Hadoop Archive tool, > it's very neccessary for data retention for everyone (small files problem and > compress data). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13342) ISAL download is breaking the Dockerfile
[ https://issues.apache.org/jira/browse/HADOOP-13342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366169#comment-15366169 ] Kai Zheng commented on HADOOP-13342: Hi [~aw], Thanks for your thoughts! bq. It's also worth pointing out that the PowerPC version of the Dockerfile will also not be including ISAL. Do you see any error on the platform? Did you mean the ISA-L library package couldn't be installed or it won't run? In your view, why is it being different when come to the ISA-L library, from the existing ones such as OpenSSL, Snappy and etc.? You mentioned a long term solution, do you have any idea besides the way to install it in the dockerfile? Thanks again. > ISAL download is breaking the Dockerfile > > > Key: HADOOP-13342 > URL: https://issues.apache.org/jira/browse/HADOOP-13342 > Project: Hadoop Common > Issue Type: Bug > Components: build >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Blocker > Fix For: 3.0.0-alpha1 > > Attachments: HADOOP-13342.00.patch > > > http://http.us.debian.org/debian/pool/main/libi/libisal/libisal2_2.15.0-2_amd64.deb > is returning a 404. We need to replace or remove this hack to prevent this > from happening in the future. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13342) ISAL download is breaking the Dockerfile
[ https://issues.apache.org/jira/browse/HADOOP-13342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366155#comment-15366155 ] Kai Zheng commented on HADOOP-13342: Thanks [~raviprak] for the clarifying, and your suggestion to bring it back. I will think about this and figure out a better way to do this, how to download and update new ISA-L library revision. > ISAL download is breaking the Dockerfile > > > Key: HADOOP-13342 > URL: https://issues.apache.org/jira/browse/HADOOP-13342 > Project: Hadoop Common > Issue Type: Bug > Components: build >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Blocker > Fix For: 3.0.0-alpha1 > > Attachments: HADOOP-13342.00.patch > > > http://http.us.debian.org/debian/pool/main/libi/libisal/libisal2_2.15.0-2_amd64.deb > is returning a 404. We need to replace or remove this hack to prevent this > from happening in the future. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13349) HADOOP_CLASSPATH vs HADOOP_USER_CLASSPATH
[ https://issues.apache.org/jira/browse/HADOOP-13349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366153#comment-15366153 ] Allen Wittenauer commented on HADOOP-13349: --- Oh, right: this was also made worse by YARN-1429, which added YARN_USER_CLASSPATH as an analog to HADOOP_CLASSPATH. > HADOOP_CLASSPATH vs HADOOP_USER_CLASSPATH > - > > Key: HADOOP-13349 > URL: https://issues.apache.org/jira/browse/HADOOP-13349 > Project: Hadoop Common > Issue Type: Bug > Components: scripts >Affects Versions: 3.0.0-alpha1 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Blocker > > There seems to be some confusion in HADOOP_CLASSPATH vs. > HADOOP_USER_CLASSPATH in the code and documentation, especially since > HADOOP_USER_CLASSPATH doesn't appear to be wired up. This needs to to get > fixed ASAP. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13349) HADOOP_CLASSPATH vs HADOOP_USER_CLASSPATH
[ https://issues.apache.org/jira/browse/HADOOP-13349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366141#comment-15366141 ] Allen Wittenauer commented on HADOOP-13349: --- IIRC, the intent was that HADOOP_USER_CLASSPATH would replace HADOOP_CLASSPATH since it was ambiguous as to what HADOOP_CLASSPATH represented. (Ironic!) However, it looks like that work was never completed and forward ports from branch-2 just made things worse. We're going to need to untangle this before 3.0.0-alpha1 for sure since that's a core user-facing API. CC: [~andrew.wang]. > HADOOP_CLASSPATH vs HADOOP_USER_CLASSPATH > - > > Key: HADOOP-13349 > URL: https://issues.apache.org/jira/browse/HADOOP-13349 > Project: Hadoop Common > Issue Type: Bug > Components: scripts >Affects Versions: 3.0.0-alpha1 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Blocker > > There seems to be some confusion in HADOOP_CLASSPATH vs. > HADOOP_USER_CLASSPATH in the code and documentation, especially since > HADOOP_USER_CLASSPATH doesn't appear to be wired up. This needs to to get > fixed ASAP. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13349) HADOOP_CLASSPATH vs HADOOP_USER_CLASSPATH
Allen Wittenauer created HADOOP-13349: - Summary: HADOOP_CLASSPATH vs HADOOP_USER_CLASSPATH Key: HADOOP-13349 URL: https://issues.apache.org/jira/browse/HADOOP-13349 Project: Hadoop Common Issue Type: Bug Components: scripts Affects Versions: 3.0.0-alpha1 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Priority: Blocker There seems to be some confusion in HADOOP_CLASSPATH vs. HADOOP_USER_CLASSPATH in the code and documentation, especially since HADOOP_USER_CLASSPATH doesn't appear to be wired up. This needs to to get fixed ASAP. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11540) Raw Reed-Solomon coder using Intel ISA-L library
[ https://issues.apache.org/jira/browse/HADOOP-11540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366090#comment-15366090 ] Kai Zheng commented on HADOOP-11540: Hello [~atm], Thanks for your detailed review and the nice comments! They all look good and I will update the patch accordingly. The idea of combining encoder and decoder sounds good and doable now because we have moved the real implementation codes to the common utility places. It will also be easier to configure the coder now, and we thus won't need the coder factory any more. I will consider this and do it separately as you suggested. Thanks again! > Raw Reed-Solomon coder using Intel ISA-L library > > > Key: HADOOP-11540 > URL: https://issues.apache.org/jira/browse/HADOOP-11540 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: HDFS-7285 >Reporter: Zhe Zhang >Assignee: Kai Zheng > Labels: hdfs-ec-3.0-must-do > Attachments: HADOOP-11540-initial.patch, HADOOP-11540-v1.patch, > HADOOP-11540-v10.patch, HADOOP-11540-v11.patch, HADOOP-11540-v2.patch, > HADOOP-11540-v4.patch, HADOOP-11540-v5.patch, HADOOP-11540-v6.patch, > HADOOP-11540-v7.patch, HADOOP-11540-v8.patch, HADOOP-11540-v9.patch, > HADOOP-11540-with-11996-codes.patch, Native Erasure Coder Performance - Intel > ISAL-v1.pdf > > > This is to provide RS codec implementation using Intel ISA-L library for > encoding and decoding. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13329) Dockerfile doesn't work on Linux/ppc
[ https://issues.apache.org/jira/browse/HADOOP-13329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366051#comment-15366051 ] Allen Wittenauer commented on HADOOP-13329: --- bq. We could have it copy the correct Dockerfile and the hadoop-env-checks script to some place under target or patchprocess and build it from there. Actually, that's a terrible idea. But there's got to be a way to make this work. :/ > Dockerfile doesn't work on Linux/ppc > > > Key: HADOOP-13329 > URL: https://issues.apache.org/jira/browse/HADOOP-13329 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 3.0.0-alpha1 >Reporter: Allen Wittenauer >Assignee: Amir Sanjar > Attachments: HADOOP-13329.patch > > > We need to rework how the Dockerfile is built to support both Linux/x86 and > Linux/PowerPC. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13335) Add an option to suppress the 'use yarn jar' warning or remove it
[ https://issues.apache.org/jira/browse/HADOOP-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366048#comment-15366048 ] Allen Wittenauer commented on HADOOP-13335: --- bq. . If the intent is to deprecate the (yarn jar) command It's not and I'm still not sure why you seem to insist this is true when both Vinod and I have said that there isn't any intent on it going away anytime remotely soon. As I said before, if you don't want to use 'yarn jar', fine. But there is already a way to disable the message in branch-2 which is the only version Hive currently supports anyway. When Hive starts to work with trunk/3.x, feel free to file JIRAs regarding the new functionality. As for this one, I'm leaning towards closing it as Won't Fix given there is a way to suppress the message. > Add an option to suppress the 'use yarn jar' warning or remove it > - > > Key: HADOOP-13335 > URL: https://issues.apache.org/jira/browse/HADOOP-13335 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HADOOP-13335.01.patch, HADOOP-13335.02.patch, > HADOOP-13335.02_branch-2.patch, HADOOP-13335.03.patch, > HADOOP-13335.03_branch-2.patch, HADOOP-13335.04.patch > > > https://issues.apache.org/jira/browse/HADOOP-11257 added a 'deprecation' > warning for 'hadoop jar'. > hadoop jar is used for a lot more that starting jobs. As an example - hive > uses it to start all it's services (HiveServer2, the hive client, beeline > etc). > Using 'yarn jar' for to start these services / tools doesn't make a lot of > sense - there's no relation to yarn other than requiring the classpath to > include yarn libraries. > I'd propose reverting the changes where this message is printed if YARN > variables are set (leave it in the help message), or adding a mechanism which > would allow users to suppress this WARNING. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13340) Compress Hadoop Archive output
[ https://issues.apache.org/jira/browse/HADOOP-13340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15365886#comment-15365886 ] Duc Le Tu commented on HADOOP-13340: I see some project support seekable and splittable compress codec like GZinga (https://github.com/eBay/GZinga), can we use it in this problem? > Compress Hadoop Archive output > -- > > Key: HADOOP-13340 > URL: https://issues.apache.org/jira/browse/HADOOP-13340 > Project: Hadoop Common > Issue Type: New Feature > Components: tools >Affects Versions: 2.5.0 >Reporter: Duc Le Tu > Labels: features, performance > > Why Hadoop Archive tool cannot compress output like other map-reduce job? > I used some options like -D mapreduce.output.fileoutputformat.compress=true > -D > mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.GzipCodec > but it's not work. Did I wrong somewhere? > If not, please support option for compress output of Hadoop Archive tool, > it's very neccessary for data retention for everyone (small files problem and > compress data). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-5052) Add an example for computing exact digits of Pi
[ https://issues.apache.org/jira/browse/HADOOP-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15365799#comment-15365799 ] Barack Obama commented on HADOOP-5052: -- Very useful example visit here for more information. (http://bit.ly/1WQKTZ6) > Add an example for computing exact digits of Pi > --- > > Key: HADOOP-5052 > URL: https://issues.apache.org/jira/browse/HADOOP-5052 > Project: Hadoop Common > Issue Type: New Feature >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Fix For: 0.21.0 > > Attachments: 5052_20090115_0.18.patch, 5052_20090117_0.18.patch, > 5052_20090127.patch, 5052_20090203.patch, 5052_20090211.patch, > 5052_20090219.patch > > > It would be useful to add an example showing how to use Hadoop to do > scientific computing. We should add an example for computing exact digits of > Pi. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11993) maven enforcer plugin to ban java 8 incompatible dependencies
[
https://issues.apache.org/jira/browse/HADOOP-11993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15365752#comment-15365752
]
Hadoop QA commented on HADOOP-11993:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
6s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 9m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12816585/HADOOP-11993.002.patch
|
| JIRA Issue | HADOOP-11993 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml |
| uname | Linux 5bfa20c6f39e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / a3f93be |
| Default Java | 1.8.0_91 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9935/testReport/ |
| modules | C: hadoop-project U: hadoop-project |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9935/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> maven enforcer plugin to ban java 8 incompatible dependencies
> -
>
> Key: HADOOP-11993
> URL: https://issues.apache.org/jira/browse/HADOOP-11993
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 2.7.0
>Reporter: Steve Loughran
>Assignee: Tsuyoshi Ozawa
>Priority: Minor
> Attachments: HADOOP-11993.001.patch, HADOOP-11993.002.patch
>
>
> It's possible to use maven-enforcer to ban dependencies; this can be used to
> reject those known to be incompatible with Java 8
> [example|https://gist.github.com/HiJon89/65e34552c18e5ac9fd31]
> If we set maven enforcer to do this checking, it can ensure that the 2.7+
> codebase isn't pulling in any incompatible binaries.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11993) maven enforcer plugin to ban java 8 incompatible dependencies
[ https://issues.apache.org/jira/browse/HADOOP-11993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated HADOOP-11993: Attachment: HADOOP-11993.002.patch > maven enforcer plugin to ban java 8 incompatible dependencies > - > > Key: HADOOP-11993 > URL: https://issues.apache.org/jira/browse/HADOOP-11993 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 2.7.0 >Reporter: Steve Loughran >Assignee: Tsuyoshi Ozawa >Priority: Minor > Attachments: HADOOP-11993.001.patch, HADOOP-11993.002.patch > > > It's possible to use maven-enforcer to ban dependencies; this can be used to > reject those known to be incompatible with Java 8 > [example|https://gist.github.com/HiJon89/65e34552c18e5ac9fd31] > If we set maven enforcer to do this checking, it can ensure that the 2.7+ > codebase isn't pulling in any incompatible binaries. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11993) maven enforcer plugin to ban java 8 incompatible dependencies
[ https://issues.apache.org/jira/browse/HADOOP-11993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15365735#comment-15365735 ] Tsuyoshi Ozawa commented on HADOOP-11993: - [~ajisakaa] sorry, I attached wrong patch. Uploading correct one. > maven enforcer plugin to ban java 8 incompatible dependencies > - > > Key: HADOOP-11993 > URL: https://issues.apache.org/jira/browse/HADOOP-11993 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 2.7.0 >Reporter: Steve Loughran >Assignee: Tsuyoshi Ozawa >Priority: Minor > Attachments: HADOOP-11993.001.patch > > > It's possible to use maven-enforcer to ban dependencies; this can be used to > reject those known to be incompatible with Java 8 > [example|https://gist.github.com/HiJon89/65e34552c18e5ac9fd31] > If we set maven enforcer to do this checking, it can ensure that the 2.7+ > codebase isn't pulling in any incompatible binaries. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
