[jira] [Commented] (HADOOP-10101) Update guava dependency to the latest version
[ https://issues.apache.org/jira/browse/HADOOP-10101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14306817#comment-14306817 ] Tsuyoshi OZAWA commented on HADOOP-10101: - [~ste...@apache.org] I checked Javac warnings: these warnings look not related to the patch. I also checked findbugs are not related. Could you take a look at 009 patch? Update guava dependency to the latest version - Key: HADOOP-10101 URL: https://issues.apache.org/jira/browse/HADOOP-10101 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.2.0, 2.6.0 Reporter: Rakesh R Assignee: Vinayakumar B Attachments: HADOOP-10101-002.patch, HADOOP-10101-004.patch, HADOOP-10101-005.patch, HADOOP-10101-006.patch, HADOOP-10101-007.patch, HADOOP-10101-008.patch, HADOOP-10101-009.patch, HADOOP-10101.patch, HADOOP-10101.patch The existing guava version is 11.0.2 which is quite old. This -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11543) Improve help message for hadoop/yarn command
[ https://issues.apache.org/jira/browse/HADOOP-11543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA updated HADOOP-11543: Fix Version/s: 2.7.0 Improve help message for hadoop/yarn command Key: HADOOP-11543 URL: https://issues.apache.org/jira/browse/HADOOP-11543 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 2.6.0 Reporter: Jagadesh Kiran N Assignee: Brahma Reddy Battula Priority: Trivial Fix For: 2.7.0 Attachments: HADOOP-11543-001.patch, HADOOP-11543-002.patch, HADOOP-11543-002.patch, HADOOP-11543-branch-2.001.patch, HADOOP-11543-branch-2.002, HADOOP-11543.patch, OrinYarncommnad.png, YARN-3128-01.patch, YARN-3128.patch Pls check the snapshot attached -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11542) Raw Reed-Solomon coder in pure Java
[ https://issues.apache.org/jira/browse/HADOOP-11542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11542: --- Attachment: HADOOP-11542-v1.patch Updated the patch sync-ing with other relevant patches. More tests added and passed. Raw Reed-Solomon coder in pure Java --- Key: HADOOP-11542 URL: https://issues.apache.org/jira/browse/HADOOP-11542 Project: Hadoop Common Issue Type: Sub-task Affects Versions: HDFS-EC Reporter: Kai Zheng Assignee: Kai Zheng Attachments: HADOOP-11542-v1.patch, HDFS-7418-v1.patch, HDFS-7418-v2.patch This will implement RS coder by porting existing codes in HDFS-RAID in the new codec and coder framework, which could be useful in case native support isn't available or convenient in some environments or platforms. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11542) Raw Reed-Solomon coder in pure Java
[ https://issues.apache.org/jira/browse/HADOOP-11542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11542: --- Attachment: (was: HDFS-7418-v2.patch) Raw Reed-Solomon coder in pure Java --- Key: HADOOP-11542 URL: https://issues.apache.org/jira/browse/HADOOP-11542 Project: Hadoop Common Issue Type: Sub-task Affects Versions: HDFS-EC Reporter: Kai Zheng Assignee: Kai Zheng Attachments: HADOOP-11542-v1.patch This will implement RS coder by porting existing codes in HDFS-RAID in the new codec and coder framework, which could be useful in case native support isn't available or convenient in some environments or platforms. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11542) Raw Reed-Solomon coder in pure Java
[ https://issues.apache.org/jira/browse/HADOOP-11542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11542: --- Attachment: (was: HDFS-7418-v1.patch) Raw Reed-Solomon coder in pure Java --- Key: HADOOP-11542 URL: https://issues.apache.org/jira/browse/HADOOP-11542 Project: Hadoop Common Issue Type: Sub-task Affects Versions: HDFS-EC Reporter: Kai Zheng Assignee: Kai Zheng Attachments: HADOOP-11542-v1.patch This will implement RS coder by porting existing codes in HDFS-RAID in the new codec and coder framework, which could be useful in case native support isn't available or convenient in some environments or platforms. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11554) Expose HadoopKerberosName as a hadoop subcommand
[ https://issues.apache.org/jira/browse/HADOOP-11554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11554: -- Attachment: HADOOP-11554.patch -00: * Initial patch Expose HadoopKerberosName as a hadoop subcommand Key: HADOOP-11554 URL: https://issues.apache.org/jira/browse/HADOOP-11554 Project: Hadoop Common Issue Type: New Feature Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Attachments: HADOOP-11554.patch HadoopKerberosName has been around as a secret hack for quite a while. We should clean up the output and make it official by exposing it via the hadoop command. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11543) Improve help message for hadoop/yarn command
[ https://issues.apache.org/jira/browse/HADOOP-11543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308602#comment-14308602 ] Tsuyoshi OZAWA commented on HADOOP-11543: - +1, I'll commit it shortly with minor fixes(indentation). Improve help message for hadoop/yarn command Key: HADOOP-11543 URL: https://issues.apache.org/jira/browse/HADOOP-11543 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 2.6.0 Reporter: Jagadesh Kiran N Assignee: Brahma Reddy Battula Priority: Trivial Fix For: 2.7.0 Attachments: HADOOP-11543-001.patch, HADOOP-11543-002.patch, HADOOP-11543-002.patch, HADOOP-11543-branch-2-003.patch, HADOOP-11543-branch-2.001.patch, HADOOP-11543-branch-2.002, HADOOP-11543.patch, OrinYarncommnad.png, YARN-3128-01.patch, YARN-3128.patch Pls check the snapshot attached -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11543) Improve help message for hadoop/yarn command
[ https://issues.apache.org/jira/browse/HADOOP-11543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308611#comment-14308611 ] Tsuyoshi OZAWA commented on HADOOP-11543: - Ouch, I found one missing point: --loglevel option. Revert the commit for branch-2. [~aw], In branch-2, I found print_usage prints --log-level option as usage, but given option is not referred from anywhere. Is it possible to remove it? Improve help message for hadoop/yarn command Key: HADOOP-11543 URL: https://issues.apache.org/jira/browse/HADOOP-11543 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 2.6.0 Reporter: Jagadesh Kiran N Assignee: Brahma Reddy Battula Priority: Trivial Fix For: 2.7.0 Attachments: HADOOP-11543-001.patch, HADOOP-11543-002.patch, HADOOP-11543-002.patch, HADOOP-11543-branch-2-003.patch, HADOOP-11543-branch-2.001.patch, HADOOP-11543-branch-2.002, HADOOP-11543.patch, OrinYarncommnad.png, YARN-3128-01.patch, YARN-3128.patch Pls check the snapshot attached -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11543) Improve help message for hadoop/yarn command
[ https://issues.apache.org/jira/browse/HADOOP-11543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308695#comment-14308695 ] Brahma Reddy Battula commented on HADOOP-11543: --- AFAIK ./yarn --loglevel INFO logs Unrecognized option: --loglevel Improve help message for hadoop/yarn command Key: HADOOP-11543 URL: https://issues.apache.org/jira/browse/HADOOP-11543 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 2.6.0 Reporter: Jagadesh Kiran N Assignee: Brahma Reddy Battula Priority: Trivial Fix For: 2.7.0 Attachments: HADOOP-11543-001.patch, HADOOP-11543-002.patch, HADOOP-11543-002.patch, HADOOP-11543-branch-2-003.patch, HADOOP-11543-branch-2.001.patch, HADOOP-11543-branch-2.002, HADOOP-11543.patch, OrinYarncommnad.png, YARN-3128-01.patch, YARN-3128.patch Pls check the snapshot attached -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11551) Let nightly jenkins jobs run the tool of HADOOP-11045 and include the result in the job report
[
https://issues.apache.org/jira/browse/HADOOP-11551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308723#comment-14308723
]
Tsuyoshi OZAWA commented on HADOOP-11551:
-
{code}
For pre-commit test, the result would be not consistent because it involves
uncommitted patches. I was thinking about adding it only for committed code,
the nightly jobs run on trunk.
{code}
Make sense. One question: if we add it for committed code, where should we
print the lists of flaky tests?
Let nightly jenkins jobs run the tool of HADOOP-11045 and include the result
in the job report
--
Key: HADOOP-11551
URL: https://issues.apache.org/jira/browse/HADOOP-11551
Project: Hadoop Common
Issue Type: Bug
Components: build, tools
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
This jira is to propose running the tool created with HADOOP-11045 at the end
of jenkins test job - I am thinking about trunk jobs currently - and report
the results at the job report. This way when we look at test failure, we can
tell the failure pattern, and whether failed test is likely a flaky test or
not.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11551) Let nightly jenkins jobs run the tool of HADOOP-11045 and include the result in the job report
[ https://issues.apache.org/jira/browse/HADOOP-11551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308629#comment-14308629 ] Yongjun Zhang commented on HADOOP-11551: Hi Tsuyoshi, Thanks for your quick feedback and support! For pre-commit test, the result would be not consistent because it involves uncommitted patches. I was thinking about adding it only for committed code, the nightly jobs run on trunk. Developer certainly can run on pre-commit jobs themselves. What do you think? Let nightly jenkins jobs run the tool of HADOOP-11045 and include the result in the job report -- Key: HADOOP-11551 URL: https://issues.apache.org/jira/browse/HADOOP-11551 Project: Hadoop Common Issue Type: Bug Components: build, tools Reporter: Yongjun Zhang This jira is to propose running the tool created with HADOOP-11045 at the end of jenkins test job - I am thinking about trunk jobs currently - and report the results at the job report. This way when we look at test failure, we can tell the failure pattern, and whether failed test is likely a flaky test or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11543) Improve help message for hadoop/yarn command
[ https://issues.apache.org/jira/browse/HADOOP-11543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308631#comment-14308631 ] Allen Wittenauer commented on HADOOP-11543: --- loglevel is handled in hadoop-config.sh generically for all the shell scripts in both trunk and branch-2. Improve help message for hadoop/yarn command Key: HADOOP-11543 URL: https://issues.apache.org/jira/browse/HADOOP-11543 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 2.6.0 Reporter: Jagadesh Kiran N Assignee: Brahma Reddy Battula Priority: Trivial Fix For: 2.7.0 Attachments: HADOOP-11543-001.patch, HADOOP-11543-002.patch, HADOOP-11543-002.patch, HADOOP-11543-branch-2-003.patch, HADOOP-11543-branch-2.001.patch, HADOOP-11543-branch-2.002, HADOOP-11543.patch, OrinYarncommnad.png, YARN-3128-01.patch, YARN-3128.patch Pls check the snapshot attached -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11554) Expose HadoopKerberosName as a hadoop subcommand
[ https://issues.apache.org/jira/browse/HADOOP-11554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308669#comment-14308669 ] Chris Nauroth commented on HADOOP-11554: Thanks, Allen. This is a good idea. Would you please add the subcommand in hadoop.cmd too for consistency? I can volunteer to test on Windows. Let's add it to the documentation in CommandsManual.apt.vm. It's probably also worth a brief mention in SecureMode.apt.vm under the section Mapping from Kerberos principal to OS user account. Expose HadoopKerberosName as a hadoop subcommand Key: HADOOP-11554 URL: https://issues.apache.org/jira/browse/HADOOP-11554 Project: Hadoop Common Issue Type: New Feature Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Attachments: HADOOP-11554.patch HadoopKerberosName has been around as a secret hack for quite a while. We should clean up the output and make it official by exposing it via the hadoop command. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11535) TableMapping related tests failed due to 'correct' resolving for test hostname
[
https://issues.apache.org/jira/browse/HADOOP-11535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308682#comment-14308682
]
Kai Zheng commented on HADOOP-11535:
Hmmm, looks like it's not the case. Actually whatever bad or invalid hostname
is used, it can resolve to the fixed IP address in my network, as follows.
{noformat}
nslookup abc.invalid.test
Server: 10.248.2.5
Address:10.248.2.5#53
Non-authoritative answer:
Name: abc.invalid.test
Address: 220.250.64.228
{noformat}
So I thought in the tests we should not use hostname, instead use IP address,
to avoid such bad situation at all. I'm not sure if it's the typical or not,
but Java as nslookup does return non-authoritative answer when resolving a
domain name. Using domain name isn't necessary for the tests.
[~ste...@apache.org], do you agree we can change the test, not using domain
name like 'a.b.c', instead using IP address like '1.2.3.4' ? Thanks.
TableMapping related tests failed due to 'correct' resolving for test hostname
--
Key: HADOOP-11535
URL: https://issues.apache.org/jira/browse/HADOOP-11535
Project: Hadoop Common
Issue Type: Test
Reporter: Kai Zheng
Assignee: Kai Zheng
Priority: Minor
When mvn test in my environment, it reported the following.
{noformat}
Failed tests:
TestTableMapping.testClearingCachedMappings:144 expected:/[rack1] but
was:/[default-rack]
TestTableMapping.testTableCaching:79 expected:/[rack1] but
was:/[default-rack]
TestTableMapping.testResolve:56 expected:/[rack1] but
was:/[default-rack]
{noformat}
It's caused by the good resolving for the 'bad test' hostname 'a.b.c' as
follows.
{noformat}
[drankye@zkdesk hadoop-common-project]$ ping a.b.c
PING a.b.c (220.250.64.228) 56(84) bytes of data.
{noformat}
I understand it may happen in just my local environment, and document this
just in case others also meet this. We may use even worse hostname than
'a.b.c' to avoid such situation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11535) TableMapping related tests failed due to 'correct' resolving for test hostname
[
https://issues.apache.org/jira/browse/HADOOP-11535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kai Zheng updated HADOOP-11535:
---
Attachment: HADOOP-11535-v1.patch
Uploaded a patch, using IP directly.
TableMapping related tests failed due to 'correct' resolving for test hostname
--
Key: HADOOP-11535
URL: https://issues.apache.org/jira/browse/HADOOP-11535
Project: Hadoop Common
Issue Type: Test
Reporter: Kai Zheng
Assignee: Kai Zheng
Priority: Minor
Attachments: HADOOP-11535-v1.patch
When mvn test in my environment, it reported the following.
{noformat}
Failed tests:
TestTableMapping.testClearingCachedMappings:144 expected:/[rack1] but
was:/[default-rack]
TestTableMapping.testTableCaching:79 expected:/[rack1] but
was:/[default-rack]
TestTableMapping.testResolve:56 expected:/[rack1] but
was:/[default-rack]
{noformat}
It's caused by the good resolving for the 'bad test' hostname 'a.b.c' as
follows.
{noformat}
[drankye@zkdesk hadoop-common-project]$ ping a.b.c
PING a.b.c (220.250.64.228) 56(84) bytes of data.
{noformat}
I understand it may happen in just my local environment, and document this
just in case others also meet this. We may use even worse hostname than
'a.b.c' to avoid such situation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11535) TableMapping related tests failed due to 'correct' resolving for test hostname
[
https://issues.apache.org/jira/browse/HADOOP-11535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kai Zheng updated HADOOP-11535:
---
Status: Patch Available (was: Open)
I submit the patch any way in case it's liked.
TableMapping related tests failed due to 'correct' resolving for test hostname
--
Key: HADOOP-11535
URL: https://issues.apache.org/jira/browse/HADOOP-11535
Project: Hadoop Common
Issue Type: Test
Reporter: Kai Zheng
Assignee: Kai Zheng
Priority: Minor
Attachments: HADOOP-11535-v1.patch
When mvn test in my environment, it reported the following.
{noformat}
Failed tests:
TestTableMapping.testClearingCachedMappings:144 expected:/[rack1] but
was:/[default-rack]
TestTableMapping.testTableCaching:79 expected:/[rack1] but
was:/[default-rack]
TestTableMapping.testResolve:56 expected:/[rack1] but
was:/[default-rack]
{noformat}
It's caused by the good resolving for the 'bad test' hostname 'a.b.c' as
follows.
{noformat}
[drankye@zkdesk hadoop-common-project]$ ping a.b.c
PING a.b.c (220.250.64.228) 56(84) bytes of data.
{noformat}
I understand it may happen in just my local environment, and document this
just in case others also meet this. We may use even worse hostname than
'a.b.c' to avoid such situation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11543) Improve help message for hadoop/yarn command
[
https://issues.apache.org/jira/browse/HADOOP-11543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308557#comment-14308557
]
Hadoop QA commented on HADOOP-11543:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12696950/HADOOP-11543-branch-2-003.patch
against trunk revision af3aadf.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/5611//console
This message is automatically generated.
Improve help message for hadoop/yarn command
Key: HADOOP-11543
URL: https://issues.apache.org/jira/browse/HADOOP-11543
Project: Hadoop Common
Issue Type: Improvement
Components: scripts
Affects Versions: 2.6.0
Reporter: Jagadesh Kiran N
Assignee: Brahma Reddy Battula
Priority: Trivial
Fix For: 2.7.0
Attachments: HADOOP-11543-001.patch, HADOOP-11543-002.patch,
HADOOP-11543-002.patch, HADOOP-11543-branch-2-003.patch,
HADOOP-11543-branch-2.001.patch, HADOOP-11543-branch-2.002,
HADOOP-11543.patch, OrinYarncommnad.png, YARN-3128-01.patch, YARN-3128.patch
Pls check the snapshot attached
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11543) Improve help message for hadoop/yarn command
[ https://issues.apache.org/jira/browse/HADOOP-11543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308587#comment-14308587 ] Hudson commented on HADOOP-11543: - FAILURE: Integrated in Hadoop-trunk-Commit #7035 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7035/]) Move HADOOP-11543 from BUG-FIX to IMPROVEMENT in CHANGES.txt. (ozawa: rev 9d910691183ef7190248922e13f384832a5611f9) * hadoop-common-project/hadoop-common/CHANGES.txt Improve help message for hadoop/yarn command Key: HADOOP-11543 URL: https://issues.apache.org/jira/browse/HADOOP-11543 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 2.6.0 Reporter: Jagadesh Kiran N Assignee: Brahma Reddy Battula Priority: Trivial Fix For: 2.7.0 Attachments: HADOOP-11543-001.patch, HADOOP-11543-002.patch, HADOOP-11543-002.patch, HADOOP-11543-branch-2-003.patch, HADOOP-11543-branch-2.001.patch, HADOOP-11543-branch-2.002, HADOOP-11543.patch, OrinYarncommnad.png, YARN-3128-01.patch, YARN-3128.patch Pls check the snapshot attached -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11554) Expose HadoopKerberosName as a hadoop subcommand
[ https://issues.apache.org/jira/browse/HADOOP-11554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11554: -- Status: Patch Available (was: Open) Expose HadoopKerberosName as a hadoop subcommand Key: HADOOP-11554 URL: https://issues.apache.org/jira/browse/HADOOP-11554 Project: Hadoop Common Issue Type: New Feature Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Attachments: HADOOP-11554.patch HadoopKerberosName has been around as a secret hack for quite a while. We should clean up the output and make it official by exposing it via the hadoop command. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11485) Pluggable shell integration
[
https://issues.apache.org/jira/browse/HADOOP-11485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308773#comment-14308773
]
Hadoop QA commented on HADOOP-11485:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12696956/HADOOP-11485-03.patch
against trunk revision af3aadf.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-assemblies hadoop-common-project/hadoop-common
hadoop-hdfs-project/hadoop-hdfs.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/5612//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/5612//console
This message is automatically generated.
Pluggable shell integration
---
Key: HADOOP-11485
URL: https://issues.apache.org/jira/browse/HADOOP-11485
Project: Hadoop Common
Issue Type: New Feature
Components: scripts
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Allen Wittenauer
Labels: scripts, shell
Attachments: HADOOP-11485-00.patch, HADOOP-11485-01.patch,
HADOOP-11485-02.patch, HADOOP-11485-03.patch
It would be useful to provide a way for core and non-core Hadoop components
to plug into the shell infrastructure. This would allow us to pull the HDFS,
MapReduce, and YARN shell functions out of hadoop-functions.sh.
Additionally, it should let 3rd parties such as HBase influence things like
classpaths at runtime.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11551) Let nightly jenkins jobs run the tool of HADOOP-11045 and include the result in the job report
[ https://issues.apache.org/jira/browse/HADOOP-11551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308597#comment-14308597 ] Yongjun Zhang commented on HADOOP-11551: Hi [~ozawa] and [~ajisakaa], Thanks a lot for your earlier help on HADOOP-11045. I wonder if you could comment on the proposal here? If we agree to push this forward, we might need help from infra folks to get it in. Thanks. Let nightly jenkins jobs run the tool of HADOOP-11045 and include the result in the job report -- Key: HADOOP-11551 URL: https://issues.apache.org/jira/browse/HADOOP-11551 Project: Hadoop Common Issue Type: Bug Components: build, tools Reporter: Yongjun Zhang This jira is to propose running the tool created with HADOOP-11045 at the end of jenkins test job - I am thinking about trunk jobs currently - and report the results at the job report. This way when we look at test failure, we can tell the failure pattern, and whether failed test is likely a flaky test or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11551) Let nightly jenkins jobs run the tool of HADOOP-11045 and include the result in the job report
[ https://issues.apache.org/jira/browse/HADOOP-11551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308623#comment-14308623 ] Tsuyoshi OZAWA commented on HADOOP-11551: - [~yzhangal], +1 for this change. With this tool, we can reduce time and effort to confirm whether test failures are flaky and comment it on JIRA manually. I thought it's straightforward to call the python script from dev-support/test-patch.sh. About infra, we need to check the version of python which the servers for Jenkins CI have. Let nightly jenkins jobs run the tool of HADOOP-11045 and include the result in the job report -- Key: HADOOP-11551 URL: https://issues.apache.org/jira/browse/HADOOP-11551 Project: Hadoop Common Issue Type: Bug Components: build, tools Reporter: Yongjun Zhang This jira is to propose running the tool created with HADOOP-11045 at the end of jenkins test job - I am thinking about trunk jobs currently - and report the results at the job report. This way when we look at test failure, we can tell the failure pattern, and whether failed test is likely a flaky test or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11554) Expose HadoopKerberosName as a hadoop subcommand
[
https://issues.apache.org/jira/browse/HADOOP-11554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308643#comment-14308643
]
Hadoop QA commented on HADOOP-11554:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12696962/HADOOP-11554.patch
against trunk revision 9d91069.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:red}-1 eclipse:eclipse{color}. The patch failed to build with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/5613//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/5613//console
This message is automatically generated.
Expose HadoopKerberosName as a hadoop subcommand
Key: HADOOP-11554
URL: https://issues.apache.org/jira/browse/HADOOP-11554
Project: Hadoop Common
Issue Type: New Feature
Components: scripts
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Attachments: HADOOP-11554.patch
HadoopKerberosName has been around as a secret hack for quite a while. We
should clean up the output and make it official by exposing it via the hadoop
command.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11485) Pluggable shell integration
[
https://issues.apache.org/jira/browse/HADOOP-11485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308655#comment-14308655
]
Chris Nauroth commented on HADOOP-11485:
I think v03 has some issues with usage of {{hadoop_translate_cygwin_path}} in
shellprofile.d/yarn:
{code}
hadoop_translate_cygwin_path ${yld}
{code}
Inside {{hadoop_translate_cygwin_path}}, the argument is part of an {{eval}}
for setting (possibly changing) the variable value, so instead of expanding the
variable's value at the call site, we just need to pass through the variable
name:
{code}
hadoop_translate_cygwin_path yld
{code}
Also, I think I see a typo, defining variable {{yhd}}, but then trying to use
{{ygd}}:
{code}
local yhd=$HADOOP_YARN_HOME
hadoop_translate_cygwin_path ${ygd}
{code}
Pluggable shell integration
---
Key: HADOOP-11485
URL: https://issues.apache.org/jira/browse/HADOOP-11485
Project: Hadoop Common
Issue Type: New Feature
Components: scripts
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Allen Wittenauer
Labels: scripts, shell
Attachments: HADOOP-11485-00.patch, HADOOP-11485-01.patch,
HADOOP-11485-02.patch, HADOOP-11485-03.patch
It would be useful to provide a way for core and non-core Hadoop components
to plug into the shell infrastructure. This would allow us to pull the HDFS,
MapReduce, and YARN shell functions out of hadoop-functions.sh.
Additionally, it should let 3rd parties such as HBase influence things like
classpaths at runtime.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11543) Improve help message for hadoop/yarn command
[
https://issues.apache.org/jira/browse/HADOOP-11543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308674#comment-14308674
]
Tsuyoshi OZAWA commented on HADOOP-11543:
-
[~aw] Thank you for the clarification. Make sense. I think it's useful to
print \[--loglevel loglevel\] as usage in both trunk and branch-2, but it's
missing on trunk. My suggestion is that we should revert trunk change and add
\[--loglevel loglevel\] to usage in both hadoop and yarn command. Does this
make sense to you?
The message will be like this on both trunk and branch-2:
{code}
$ bin/hadoop
Usage: hadoop [--config confdir] [--loglevel loglevel] [COMMAND | CLASSNAME]
CLASSNAMErun the class named CLASSNAME
or
where COMMAND is one of:
archive -archiveName NAME -p parent path src* dest
{code}
Improve help message for hadoop/yarn command
Key: HADOOP-11543
URL: https://issues.apache.org/jira/browse/HADOOP-11543
Project: Hadoop Common
Issue Type: Improvement
Components: scripts
Affects Versions: 2.6.0
Reporter: Jagadesh Kiran N
Assignee: Brahma Reddy Battula
Priority: Trivial
Fix For: 2.7.0
Attachments: HADOOP-11543-001.patch, HADOOP-11543-002.patch,
HADOOP-11543-002.patch, HADOOP-11543-branch-2-003.patch,
HADOOP-11543-branch-2.001.patch, HADOOP-11543-branch-2.002,
HADOOP-11543.patch, OrinYarncommnad.png, YARN-3128-01.patch, YARN-3128.patch
Pls check the snapshot attached
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11543) Improve help message for hadoop/yarn command
[ https://issues.apache.org/jira/browse/HADOOP-11543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308706#comment-14308706 ] Brahma Reddy Battula commented on HADOOP-11543: --- I feel, _loglevel_ might not required here since it is not handled in hadoop and yarn scripts( please check above error for same)...please Let me know your opinion.. Improve help message for hadoop/yarn command Key: HADOOP-11543 URL: https://issues.apache.org/jira/browse/HADOOP-11543 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 2.6.0 Reporter: Jagadesh Kiran N Assignee: Brahma Reddy Battula Priority: Trivial Fix For: 2.7.0 Attachments: HADOOP-11543-001.patch, HADOOP-11543-002.patch, HADOOP-11543-002.patch, HADOOP-11543-branch-2-003.patch, HADOOP-11543-branch-2.001.patch, HADOOP-11543-branch-2.002, HADOOP-11543.patch, OrinYarncommnad.png, YARN-3128-01.patch, YARN-3128.patch Pls check the snapshot attached -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11485) Pluggable shell integration
[ https://issues.apache.org/jira/browse/HADOOP-11485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11485: -- Status: Open (was: Patch Available) Pluggable shell integration --- Key: HADOOP-11485 URL: https://issues.apache.org/jira/browse/HADOOP-11485 Project: Hadoop Common Issue Type: New Feature Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: scripts, shell Attachments: HADOOP-11485-00.patch, HADOOP-11485-01.patch, HADOOP-11485-02.patch It would be useful to provide a way for core and non-core Hadoop components to plug into the shell infrastructure. This would allow us to pull the HDFS, MapReduce, and YARN shell functions out of hadoop-functions.sh. Additionally, it should let 3rd parties such as HBase influence things like classpaths at runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11485) Pluggable shell integration
[ https://issues.apache.org/jira/browse/HADOOP-11485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11485: -- Status: Patch Available (was: Open) Pluggable shell integration --- Key: HADOOP-11485 URL: https://issues.apache.org/jira/browse/HADOOP-11485 Project: Hadoop Common Issue Type: New Feature Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: scripts, shell Attachments: HADOOP-11485-00.patch, HADOOP-11485-01.patch, HADOOP-11485-02.patch, HADOOP-11485-03.patch It would be useful to provide a way for core and non-core Hadoop components to plug into the shell infrastructure. This would allow us to pull the HDFS, MapReduce, and YARN shell functions out of hadoop-functions.sh. Additionally, it should let 3rd parties such as HBase influence things like classpaths at runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11485) Pluggable shell integration
[ https://issues.apache.org/jira/browse/HADOOP-11485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11485: -- Attachment: HADOOP-11485-03.patch -03: * Rebase * Fix some minor problems with the example * Remove some deprecated vars now that HADOOP-11460 is committed * Fix some shellcheck errors (woops) Pluggable shell integration --- Key: HADOOP-11485 URL: https://issues.apache.org/jira/browse/HADOOP-11485 Project: Hadoop Common Issue Type: New Feature Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: scripts, shell Attachments: HADOOP-11485-00.patch, HADOOP-11485-01.patch, HADOOP-11485-02.patch, HADOOP-11485-03.patch It would be useful to provide a way for core and non-core Hadoop components to plug into the shell infrastructure. This would allow us to pull the HDFS, MapReduce, and YARN shell functions out of hadoop-functions.sh. Additionally, it should let 3rd parties such as HBase influence things like classpaths at runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-10513) ReflectionUtils::CONSTRUCTOR_CACHE leaks class loaders
[
https://issues.apache.org/jira/browse/HADOOP-10513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307238#comment-14307238
]
Lukáš Karas commented on HADOOP-10513:
--
as a workaround, method can be called with reflection...
Method clearCacheMethod =
ReflectionUtils.class.getDeclaredMethod(clearCache);
if (clearCacheMethod != null) {
clearCacheMethod.setAccessible(true) ;
clearCacheMethod.invoke(null) ;
}
ReflectionUtils::CONSTRUCTOR_CACHE leaks class loaders
--
Key: HADOOP-10513
URL: https://issues.apache.org/jira/browse/HADOOP-10513
Project: Hadoop Common
Issue Type: Bug
Components: util
Affects Versions: 2.4.0
Reporter: Gopal V
Labels: leak, permgen
Attachments: class-loader-leak.png
Any code which uses custom class-loaders needs to be aware of the
ReflectionUtils::CONSTRUCTOR_CACHE
{code}
/**
* Cache of constructors for each class. Pins the classes so they
* can't be garbage collected until ReflectionUtils can be collected.
*/
private static final MapClass?, Constructor? CONSTRUCTOR_CACHE =
new ConcurrentHashMapClass?, Constructor?();
{code}
This is not a problem when using only 1 AppClassLoader.
But in cases where the application uses multiple classloaders (to isolate
UDFs), this holds onto class references and their associated class loaders by
ref ( that leaks all the statics).
The clear method for this cache is unfortunately marked only for testing.
{code}
// methods to support testing
static void clearCache() {
CONSTRUCTOR_CACHE.clear();
}
{code}
The cache shows up as the only reference for the class loaders.
!class-loader-leak.png!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-9645) KerberosAuthenticator NPEs on connect error
[
https://issues.apache.org/jira/browse/HADOOP-9645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307217#comment-14307217
]
Rakesh R commented on HADOOP-9645:
--
It looks like this is duplicate of HDFS-3980. Can we mark as duplicate ?
KerberosAuthenticator NPEs on connect error
---
Key: HADOOP-9645
URL: https://issues.apache.org/jira/browse/HADOOP-9645
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 2.0.5-alpha
Reporter: Daryn Sharp
Priority: Critical
A NPE occurs if there's a kerberos error during initial connect. In this
case, the NN was using a HTTP service principal with a stale kvno. It causes
webhdfs to fail in a non-user friendly manner by masking the real error from
the user.
{noformat}
java.lang.RuntimeException: java.lang.NullPointerException
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1241)
at
sun.net.www.protocol.http.HttpURLConnection.getHeaderField(HttpURLConnection.java:2713)
at
java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:477)
at
org.apache.hadoop.security.authentication.client.KerberosAuthenticator.isNegotiate(KerberosAuthenticator.java:164)
at
org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:140)
at
org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:217)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.openHttpUrlConnection(WebHdfsFileSystem.java:364)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-7947) Validate XMLs if a relevant tool is available, when using scripts
[
https://issues.apache.org/jira/browse/HADOOP-7947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HADOOP-7947:
-
Assignee: Kengo Seki
Validate XMLs if a relevant tool is available, when using scripts
-
Key: HADOOP-7947
URL: https://issues.apache.org/jira/browse/HADOOP-7947
Project: Hadoop Common
Issue Type: Wish
Components: scripts
Affects Versions: 2.7.0
Reporter: Harsh J
Assignee: Kengo Seki
Labels: newbie
Attachments: HADOOP-7947.001.patch
Given that we are locked down to using only XML for configuration and most of
the administrators need to manage it by themselves (unless a tool that
manages for you is used), it would be good to also validate the provided
config XML (*-site.xml) files with a tool like {{xmllint}} or maybe Xerces
somehow, when running a command or (at least) when starting up daemons.
We should use this only if a relevant tool is available, and optionally be
silent if the env. requests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11545) [ hadoop credential ] ArrayIndexOutOfBoundsException thrown when we list -provider
[
https://issues.apache.org/jira/browse/HADOOP-11545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsuyoshi OZAWA updated HADOOP-11545:
Component/s: security
[ hadoop credential ] ArrayIndexOutOfBoundsException thrown when we list
-provider
---
Key: HADOOP-11545
URL: https://issues.apache.org/jira/browse/HADOOP-11545
Project: Hadoop Common
Issue Type: Bug
Components: security
Affects Versions: 2.6.0
Reporter: Brahma Reddy Battula
Assignee: Brahma Reddy Battula
Scenario:
Please run the following command . dn't give the provider path.
{noformat}
[hdfs@host194 bin]$ ./hadoop credential list -provider
java.lang.ArrayIndexOutOfBoundsException: 2
at
org.apache.hadoop.security.alias.CredentialShell.init(CredentialShell.java:117)
at
org.apache.hadoop.security.alias.CredentialShell.run(CredentialShell.java:63)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at
org.apache.hadoop.security.alias.CredentialShell.main(CredentialShell.java:427)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11545) ArrayIndexOutOfBoundsException is thrown with hadoop credential list -provider
[
https://issues.apache.org/jira/browse/HADOOP-11545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsuyoshi OZAWA updated HADOOP-11545:
Summary: ArrayIndexOutOfBoundsException is thrown with hadoop credential
list -provider (was: [ hadoop credential ] ArrayIndexOutOfBoundsException
thrown when we list -provider)
ArrayIndexOutOfBoundsException is thrown with hadoop credential list -provider
--
Key: HADOOP-11545
URL: https://issues.apache.org/jira/browse/HADOOP-11545
Project: Hadoop Common
Issue Type: Bug
Components: security
Affects Versions: 2.6.0
Reporter: Brahma Reddy Battula
Assignee: Brahma Reddy Battula
Scenario:
Please run the following command . dn't give the provider path.
{noformat}
[hdfs@host194 bin]$ ./hadoop credential list -provider
java.lang.ArrayIndexOutOfBoundsException: 2
at
org.apache.hadoop.security.alias.CredentialShell.init(CredentialShell.java:117)
at
org.apache.hadoop.security.alias.CredentialShell.run(CredentialShell.java:63)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at
org.apache.hadoop.security.alias.CredentialShell.main(CredentialShell.java:427)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-6964) Allow compact property description in xml
[
https://issues.apache.org/jira/browse/HADOOP-6964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307448#comment-14307448
]
Allen Wittenauer commented on HADOOP-6964:
--
bq. abort with RuntimeException at the time
This would be disastrous during a 'refresh' or 'renew' or
whatever-the-HDFS-team-decides-to-call-it-this-week operation. A running
daemon would bite the dust!
bq. sometimes it's hard to debug the wrong configurations.
Yup, agree. Luckily, HADOOP-7947 (which [~sekikn] is also working on!) covers
the validation part. It isn't at runtime, however.
Allow compact property description in xml
-
Key: HADOOP-6964
URL: https://issues.apache.org/jira/browse/HADOOP-6964
Project: Hadoop Common
Issue Type: Improvement
Components: conf
Reporter: Owen O'Malley
Assignee: Kengo Seki
Labels: newbie
Attachments: HADOOP-6964.patch
We should allow users to use the more compact form of xml elements. For
example, we could allow:
{noformat}
property name=mapred.local.dir value=/disk/0/mapred,/disk/1/mapred/
{noformat}
The old format would also be supported.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-6964) Allow compact property description in xml
[
https://issues.apache.org/jira/browse/HADOOP-6964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307451#comment-14307451
]
Allen Wittenauer commented on HADOOP-6964:
--
Also, I'm +1 barring any further comments from the other folks watching. I'll
commit this EOD California time otherwise.
Allow compact property description in xml
-
Key: HADOOP-6964
URL: https://issues.apache.org/jira/browse/HADOOP-6964
Project: Hadoop Common
Issue Type: Improvement
Components: conf
Reporter: Owen O'Malley
Assignee: Kengo Seki
Labels: newbie
Attachments: HADOOP-6964.patch
We should allow users to use the more compact form of xml elements. For
example, we could allow:
{noformat}
property name=mapred.local.dir value=/disk/0/mapred,/disk/1/mapred/
{noformat}
The old format would also be supported.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-7713) dfs -count -q should label output column
[
https://issues.apache.org/jira/browse/HADOOP-7713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HADOOP-7713:
-
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
I've committed this to trunk.
Thanks!
dfs -count -q should label output column
Key: HADOOP-7713
URL: https://issues.apache.org/jira/browse/HADOOP-7713
Project: Hadoop Common
Issue Type: Improvement
Reporter: Nigel Daley
Assignee: Jonathan Allen
Priority: Trivial
Labels: newbie
Fix For: 3.0.0
Attachments: HADOOP-7713-11.patch, HADOOP-7713.patch,
HADOOP-7713.patch, HADOOP-7713.patch, HADOOP-7713.patch, HADOOP-7713.patch,
HADOOP-7713.patch, HADOOP-7713.patch, HADOOP-7713.patch, HADOOP-7713.patch,
HADOOP-7713.patch
These commands should label the output columns:
{code}
hadoop dfs -count dir...dir
hadoop dfs -count -q dir...dir
{code}
Current output of the 2nd command above:
{code}
% hadoop dfs -count -q /user/foo /tmp
none inf 9569 9493 6372553322
hdfs://nn1.bar.com/user/foo
none inf 101 2689 209349812906
hdfs://nn1.bar.com/tmp
{code}
It is not obvious what these columns mean.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-7713) dfs -count -q should label output column
[
https://issues.apache.org/jira/browse/HADOOP-7713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307446#comment-14307446
]
Hudson commented on HADOOP-7713:
FAILURE: Integrated in Hadoop-trunk-Commit #7020 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/7020/])
HADOOP-7713. dfs -count -q should label output column (Jonathan Allen via aw)
(aw: rev fb06c0083799fa5ca514447ba6b63ce564272805)
* hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml
*
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/shell/TestCount.java
* hadoop-common-project/hadoop-common/src/site/apt/FileSystemShell.apt.vm
* hadoop-common-project/hadoop-common/src/test/resources/testConf.xml
*
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestContentSummary.java
* hadoop-common-project/hadoop-common/CHANGES.txt
*
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Count.java
*
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ContentSummary.java
dfs -count -q should label output column
Key: HADOOP-7713
URL: https://issues.apache.org/jira/browse/HADOOP-7713
Project: Hadoop Common
Issue Type: Improvement
Reporter: Nigel Daley
Assignee: Jonathan Allen
Priority: Trivial
Labels: newbie
Fix For: 3.0.0
Attachments: HADOOP-7713-11.patch, HADOOP-7713.patch,
HADOOP-7713.patch, HADOOP-7713.patch, HADOOP-7713.patch, HADOOP-7713.patch,
HADOOP-7713.patch, HADOOP-7713.patch, HADOOP-7713.patch, HADOOP-7713.patch,
HADOOP-7713.patch
These commands should label the output columns:
{code}
hadoop dfs -count dir...dir
hadoop dfs -count -q dir...dir
{code}
Current output of the 2nd command above:
{code}
% hadoop dfs -count -q /user/foo /tmp
none inf 9569 9493 6372553322
hdfs://nn1.bar.com/user/foo
none inf 101 2689 209349812906
hdfs://nn1.bar.com/tmp
{code}
It is not obvious what these columns mean.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-10623) Provide a utility to be able inspect the config as seen by a hadoop client / daemon
[
https://issues.apache.org/jira/browse/HADOOP-10623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307981#comment-14307981
]
Hadoop QA commented on HADOOP-10623:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12654156/HADOOP-10623.v04.patch
against trunk revision 276485e.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/5607//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/5607//console
This message is automatically generated.
Provide a utility to be able inspect the config as seen by a hadoop client /
daemon
Key: HADOOP-10623
URL: https://issues.apache.org/jira/browse/HADOOP-10623
Project: Hadoop Common
Issue Type: New Feature
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Attachments: HADOOP-10623.v01.patch, HADOOP-10623.v02.patch,
HADOOP-10623.v03.patch, HADOOP-10623.v04.patch
To ease debugging of config issues it is convenient to be able to generate a
config as seen by the job client or a hadoop daemon
{noformat}
]$ hadoop org.apache.hadoop.util.ConfigTool -help
Usage: ConfigTool [ -xml | -json ] [ -loadDefaults ] [ resource1... ]
if resource contains '/', load from local filesystem
otherwise, load from the classpath
Generic options supported are
-conf configuration file specify an application configuration file
-D property=valueuse value for given property
-fs local|namenode:port specify a namenode
-jt local|jobtracker:portspecify a job tracker
-files comma separated list of filesspecify comma separated files to be
copied to the map reduce cluster
-libjars comma separated list of jarsspecify comma separated jar files
to include in the classpath.
-archives comma separated list of archivesspecify comma separated
archives to be unarchived on the compute machines.
The general command line syntax is
bin/hadoop command [genericOptions] [commandOptions]
{noformat}
{noformat}
$ hadoop org.apache.hadoop.util.ConfigTool -Dmy.test.conf=val mapred-site.xml
./hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/etc/hadoop/core-site.xml | python
-mjson.tool
{
properties: [
{
isFinal: false,
key: mapreduce.framework.name,
resource: mapred-site.xml,
value: yarn
},
{
isFinal: false,
key: mapreduce.client.genericoptionsparser.used,
resource: programatically,
value: true
},
{
isFinal: false,
key: my.test.conf,
resource: from command line,
value: val
},
{
isFinal: false,
key: from.file.key,
resource:
hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/etc/hadoop/core-site.xml,
value: from.file.val
},
{
isFinal: false,
key: mapreduce.shuffle.port,
resource: mapred-site.xml,
value: ${my.mapreduce.shuffle.port}
}
]
}
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-10623) Provide a utility to be able inspect the config as seen by a hadoop client / daemon
[
https://issues.apache.org/jira/browse/HADOOP-10623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HADOOP-10623:
--
Status: Open (was: Patch Available)
Cancelling patch as it no longer applies.
Provide a utility to be able inspect the config as seen by a hadoop client /
daemon
Key: HADOOP-10623
URL: https://issues.apache.org/jira/browse/HADOOP-10623
Project: Hadoop Common
Issue Type: New Feature
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Attachments: HADOOP-10623.v01.patch, HADOOP-10623.v02.patch,
HADOOP-10623.v03.patch, HADOOP-10623.v04.patch
To ease debugging of config issues it is convenient to be able to generate a
config as seen by the job client or a hadoop daemon
{noformat}
]$ hadoop org.apache.hadoop.util.ConfigTool -help
Usage: ConfigTool [ -xml | -json ] [ -loadDefaults ] [ resource1... ]
if resource contains '/', load from local filesystem
otherwise, load from the classpath
Generic options supported are
-conf configuration file specify an application configuration file
-D property=valueuse value for given property
-fs local|namenode:port specify a namenode
-jt local|jobtracker:portspecify a job tracker
-files comma separated list of filesspecify comma separated files to be
copied to the map reduce cluster
-libjars comma separated list of jarsspecify comma separated jar files
to include in the classpath.
-archives comma separated list of archivesspecify comma separated
archives to be unarchived on the compute machines.
The general command line syntax is
bin/hadoop command [genericOptions] [commandOptions]
{noformat}
{noformat}
$ hadoop org.apache.hadoop.util.ConfigTool -Dmy.test.conf=val mapred-site.xml
./hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/etc/hadoop/core-site.xml | python
-mjson.tool
{
properties: [
{
isFinal: false,
key: mapreduce.framework.name,
resource: mapred-site.xml,
value: yarn
},
{
isFinal: false,
key: mapreduce.client.genericoptionsparser.used,
resource: programatically,
value: true
},
{
isFinal: false,
key: my.test.conf,
resource: from command line,
value: val
},
{
isFinal: false,
key: from.file.key,
resource:
hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/etc/hadoop/core-site.xml,
value: from.file.val
},
{
isFinal: false,
key: mapreduce.shuffle.port,
resource: mapred-site.xml,
value: ${my.mapreduce.shuffle.port}
}
]
}
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11541) Raw XOR coder
[ https://issues.apache.org/jira/browse/HADOOP-11541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11541: --- Attachment: (was: HDFS-7417-v3.patch) Raw XOR coder - Key: HADOOP-11541 URL: https://issues.apache.org/jira/browse/HADOOP-11541 Project: Hadoop Common Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Zheng Fix For: HDFS-EC This will implement XOR codes by porting the codes from HDFS-RAID. The coder in the algorithm is needed by some high level codecs like LRC. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11541) Raw XOR coder
[ https://issues.apache.org/jira/browse/HADOOP-11541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11541: --- Attachment: (was: HDFS-7417-v1.patch) Raw XOR coder - Key: HADOOP-11541 URL: https://issues.apache.org/jira/browse/HADOOP-11541 Project: Hadoop Common Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Zheng Fix For: HDFS-EC This will implement XOR codes by porting the codes from HDFS-RAID. The coder in the algorithm is needed by some high level codecs like LRC. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11542) Raw Reed-Solomon coder in pure Java
[ https://issues.apache.org/jira/browse/HADOOP-11542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11542: --- Attachment: HADOOP-11542-v2.patch Updated the patch cleaning up unrelevant codes. Raw Reed-Solomon coder in pure Java --- Key: HADOOP-11542 URL: https://issues.apache.org/jira/browse/HADOOP-11542 Project: Hadoop Common Issue Type: Sub-task Affects Versions: HDFS-EC Reporter: Kai Zheng Assignee: Kai Zheng Attachments: HADOOP-11542-v1.patch, HADOOP-11542-v2.patch This will implement RS coder by porting existing codes in HDFS-RAID in the new codec and coder framework, which could be useful in case native support isn't available or convenient in some environments or platforms. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-9317) User cannot specify a kerberos keytab for commands
[
https://issues.apache.org/jira/browse/HADOOP-9317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HADOOP-9317:
-
Status: Open (was: Patch Available)
bq. So your suggestion won't work because concurrent launches issuing the
kinit will still result in the race condition where one process may be issuing
the kinit while another is trying to run hadoop commands.
If you look at the sample script I wrote, we should be using a different
credential cache per invocation, thus removing the race condition.
In any case, cancelling the patch since it no longer applies.
User cannot specify a kerberos keytab for commands
--
Key: HADOOP-9317
URL: https://issues.apache.org/jira/browse/HADOOP-9317
Project: Hadoop Common
Issue Type: Bug
Components: security
Affects Versions: 2.0.0-alpha, 0.23.0, 3.0.0
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Priority: Critical
Attachments: HADOOP-9317.branch-23.patch,
HADOOP-9317.branch-23.patch, HADOOP-9317.patch, HADOOP-9317.patch,
HADOOP-9317.patch, HADOOP-9317.patch
{{UserGroupInformation}} only allows kerberos users to be logged in via the
ticket cache when running hadoop commands. {{UGI}} allows a keytab to be
used, but it's only exposed programatically. This forces keytab-based users
running hadoop commands to periodically issue a kinit from the keytab. A
race condition exists during the kinit when the ticket cache is deleted and
re-created. Hadoop commands will fail when the ticket cache does not
momentarily exist.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-10101) Update guava dependency to the latest version
[ https://issues.apache.org/jira/browse/HADOOP-10101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307784#comment-14307784 ] Tsuyoshi OZAWA commented on HADOOP-10101: - Rakesh's patch was merged on HDFS-7709. Update guava dependency to the latest version - Key: HADOOP-10101 URL: https://issues.apache.org/jira/browse/HADOOP-10101 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.2.0, 2.6.0 Reporter: Rakesh R Assignee: Vinayakumar B Attachments: HADOOP-10101-002.patch, HADOOP-10101-004.patch, HADOOP-10101-005.patch, HADOOP-10101-006.patch, HADOOP-10101-007.patch, HADOOP-10101-008.patch, HADOOP-10101-009.patch, HADOOP-10101.patch, HADOOP-10101.patch The existing guava version is 11.0.2 which is quite old. This -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11549) flaky test detection tool failed to handle special control characters in test result
[
https://issues.apache.org/jira/browse/HADOOP-11549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307815#comment-14307815
]
Yongjun Zhang commented on HADOOP-11549:
Thanks [~ozawa]!
flaky test detection tool failed to handle special control characters in test
result
Key: HADOOP-11549
URL: https://issues.apache.org/jira/browse/HADOOP-11549
Project: Hadoop Common
Issue Type: Bug
Components: tools
Affects Versions: 2.7.0
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
Fix For: 2.7.0
Attachments: HADOOP-11549.001.patch
When running the tool from HADOOP-11045 on latest Hadoop-hdfs-trunk job, I'm
seeing a problem here:
{code}
[yzhang@localhost jenkinsftf]$ ./determine-flaky-tests-hadoop.py -j
Hadoop-hdfs-trunk
Recently FAILED builds in url:
https://builds.apache.org/job/Hadoop-hdfs-trunk
THERE ARE 5 builds (out of 6) that have failed tests in the past 14 days,
as listed below:
===https://builds.apache.org/job/Hadoop-hdfs-trunk/2026/testReport
(2015-02-04 03:30:00)
Could not open testReport, check
https://builds.apache.org/job/Hadoop-hdfs-trunk/2026/Console for why it was
reported failed
...
{code}
I saw that the testReport can actually be opened from browser. After looking,
I found that HDFS-7287 fix added the following test code:
{code}
//Create a directory whose name should be escaped in XML
Path invalidXMLDir = new Path(/dirContainingInvalidXMLChar\uhere);
hdfs.mkdirs(invalidXMLDir);
...
{code}
And the output from this code caused the tool to choke.
I found a solution here and I'm attaching a patch.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11542) Raw Reed-Solomon coder in pure Java
[
https://issues.apache.org/jira/browse/HADOOP-11542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14306827#comment-14306827
]
Hadoop QA commented on HADOOP-11542:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12696693/HADOOP-11542-v1.patch
against trunk revision ec950be.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/5601//console
This message is automatically generated.
Raw Reed-Solomon coder in pure Java
---
Key: HADOOP-11542
URL: https://issues.apache.org/jira/browse/HADOOP-11542
Project: Hadoop Common
Issue Type: Sub-task
Affects Versions: HDFS-EC
Reporter: Kai Zheng
Assignee: Kai Zheng
Attachments: HADOOP-11542-v1.patch
This will implement RS coder by porting existing codes in HDFS-RAID in the
new codec and coder framework, which could be useful in case native support
isn't available or convenient in some environments or platforms.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11541) Raw XOR coder
[ https://issues.apache.org/jira/browse/HADOOP-11541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11541: --- Attachment: HADOOP-11541-v1.patch Uploaded the patch with added and passed tests. Raw XOR coder - Key: HADOOP-11541 URL: https://issues.apache.org/jira/browse/HADOOP-11541 Project: Hadoop Common Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Zheng Fix For: HDFS-EC Attachments: HADOOP-11541-v1.patch This will implement XOR codes by porting the codes from HDFS-RAID. The coder in the algorithm is needed by some high level codecs like LRC. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11541) Raw XOR coder
[ https://issues.apache.org/jira/browse/HADOOP-11541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11541: --- Attachment: (was: HDFS-7417-v2.patch) Raw XOR coder - Key: HADOOP-11541 URL: https://issues.apache.org/jira/browse/HADOOP-11541 Project: Hadoop Common Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Zheng Fix For: HDFS-EC This will implement XOR codes by porting the codes from HDFS-RAID. The coder in the algorithm is needed by some high level codecs like LRC. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11542) Raw Reed-Solomon coder in pure Java
[
https://issues.apache.org/jira/browse/HADOOP-11542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14306847#comment-14306847
]
Hadoop QA commented on HADOOP-11542:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12696703/HADOOP-11542-v2.patch
against trunk revision ec950be.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/5602//console
This message is automatically generated.
Raw Reed-Solomon coder in pure Java
---
Key: HADOOP-11542
URL: https://issues.apache.org/jira/browse/HADOOP-11542
Project: Hadoop Common
Issue Type: Sub-task
Affects Versions: HDFS-EC
Reporter: Kai Zheng
Assignee: Kai Zheng
Attachments: HADOOP-11542-v1.patch, HADOOP-11542-v2.patch
This will implement RS coder by porting existing codes in HDFS-RAID in the
new codec and coder framework, which could be useful in case native support
isn't available or convenient in some environments or platforms.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11541) Raw XOR coder
[
https://issues.apache.org/jira/browse/HADOOP-11541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14306848#comment-14306848
]
Hadoop QA commented on HADOOP-11541:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12696700/HADOOP-11541-v1.patch
against trunk revision ec950be.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/5603//console
This message is automatically generated.
Raw XOR coder
-
Key: HADOOP-11541
URL: https://issues.apache.org/jira/browse/HADOOP-11541
Project: Hadoop Common
Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Kai Zheng
Fix For: HDFS-EC
Attachments: HADOOP-11541-v1.patch
This will implement XOR codes by porting the codes from HDFS-RAID. The coder
in the algorithm is needed by some high level codecs like LRC.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-10101) Update guava dependency to the latest version
[ https://issues.apache.org/jira/browse/HADOOP-10101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14306854#comment-14306854 ] Rakesh R commented on HADOOP-10101: --- bq. I also checked findbugs are not related. I've attached patch to fix HDFS-7709, kindly review it. Thanks! Update guava dependency to the latest version - Key: HADOOP-10101 URL: https://issues.apache.org/jira/browse/HADOOP-10101 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.2.0, 2.6.0 Reporter: Rakesh R Assignee: Vinayakumar B Attachments: HADOOP-10101-002.patch, HADOOP-10101-004.patch, HADOOP-10101-005.patch, HADOOP-10101-006.patch, HADOOP-10101-007.patch, HADOOP-10101-008.patch, HADOOP-10101-009.patch, HADOOP-10101.patch, HADOOP-10101.patch The existing guava version is 11.0.2 which is quite old. This -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-11550) Ehance ErasureCoder API to support complex cases requiring multiple steps for encoding/decoding of a block group
Kai Zheng created HADOOP-11550: -- Summary: Ehance ErasureCoder API to support complex cases requiring multiple steps for encoding/decoding of a block group Key: HADOOP-11550 URL: https://issues.apache.org/jira/browse/HADOOP-11550 Project: Hadoop Common Issue Type: Improvement Reporter: Kai Zheng Assignee: Kai Zheng In HDFS-7662, *ErasureCoder* API is defined to support encoding/decoding of a block group considering the most simple but often case that only one coding step is involved. This is to ehance the API to support complex cases requiring multiple steps for encoding/decoding of a block group. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-10062) race condition in MetricsSystemImpl#publishMetricsNow that causes incorrect results
[
https://issues.apache.org/jira/browse/HADOOP-10062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307263#comment-14307263
]
Hudson commented on HADOOP-10062:
-
SUCCESS: Integrated in Hadoop-Hdfs-trunk #2027 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2027/])
HADOOP-10062. race condition in MetricsSystemImpl#publishMetricsNow that causes
incorrect results. (Contributed by Sangjin Lee) (junping_du: rev
0b567f424673b5cea1c3bc23b2bd268ef6b7625f)
*
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestMetricsSystemImpl.java
*
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java
* hadoop-common-project/hadoop-common/CHANGES.txt
race condition in MetricsSystemImpl#publishMetricsNow that causes incorrect
results
---
Key: HADOOP-10062
URL: https://issues.apache.org/jira/browse/HADOOP-10062
Project: Hadoop Common
Issue Type: Bug
Components: metrics
Affects Versions: 3.0.0
Environment: CentOS 6.4, Oracle JDK 1.6.0_31, JDK1.7.0_45
Reporter: Shinichi Yamashita
Assignee: Sangjin Lee
Fix For: 3.0.0, 2.7.0
Attachments: HADOOP-10062-failed.txt, HADOOP-10062-success.txt,
HADOOP-10062.003.patch, HADOOP-10062.patch, HADOOP-10062.patch
TestMetricsSystemInpl#testMultiThreadedPublish failed with Metrics not
collected
{code}
Running org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl
Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.688 sec
FAILURE! - in org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl
testMultiThreadedPublish(org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl)
Time elapsed: 0.056 sec FAILURE!
java.lang.AssertionError: Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Passed
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at
org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl.testMultiThreadedPublish(TestMetricsSystemImpl.java:232)
Results :
Failed tests:
TestMetricsSystemImpl.testMultiThreadedPublish:232 Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Passed
Tests run: 6, Failures: 1, Errors: 0, Skipped: 0
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11547) hadoop-common native compilation fails on Windows due to missing support for __attribute__ declaration.
[
https://issues.apache.org/jira/browse/HADOOP-11547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307271#comment-14307271
]
Hudson commented on HADOOP-11547:
-
SUCCESS: Integrated in Hadoop-Hdfs-trunk #2027 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2027/])
HADOOP-11547. hadoop-common native compilation fails on Windows due to missing
support for __attribute__ declaration. Contributed by Chris Nauroth. (cnauroth:
rev a35e86c6701ffec263966bab31bd9368923c7e80)
* hadoop-common-project/hadoop-common/src/main/native/src/exception.h
* hadoop-common-project/hadoop-common/CHANGES.txt
hadoop-common native compilation fails on Windows due to missing support for
__attribute__ declaration.
---
Key: HADOOP-11547
URL: https://issues.apache.org/jira/browse/HADOOP-11547
Project: Hadoop Common
Issue Type: Bug
Components: native
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Fix For: 2.7.0
Attachments: HADOOP-11547.001.patch, HADOOP-11547.002.patch,
HADOOP-11547.003.patch
HADOOP-11403 made a change to include exception.h in NativeIO.c. This header
includes use of the non-standard gcc {{\_\_attribute\_\_}} declaration, and
thus fails compilation on Windows.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-10976) moving the source code of hadoop-tools docs to the directory under hadoop-tools
[ https://issues.apache.org/jira/browse/HADOOP-10976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307275#comment-14307275 ] Hudson commented on HADOOP-10976: - SUCCESS: Integrated in Hadoop-Hdfs-trunk #2027 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2027/]) HADOOP-10976. moving the source code of hadoop-tools docs to the directory under hadoop-tools (Masatake Iwasaki via aw) (aw: rev 9112f093cde0b8242324aa5267c9beb21c38bf6b) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/apt/HadoopStreaming.apt.vm * hadoop-tools/hadoop-distcp/src/site/resources/css/site.css * hadoop-tools/hadoop-streaming/src/site/resources/css/site.css * hadoop-tools/hadoop-distcp/src/site/markdown/DistCp.md.vm * hadoop-project/src/site/site.xml * hadoop-tools/hadoop-archives/src/site/resources/css/site.css * hadoop-tools/hadoop-archives/src/site/markdown/HadoopArchives.md.vm * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/HadoopArchives.md.vm * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-tools/hadoop-streaming/src/site/apt/HadoopStreaming.apt.vm * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/DistCp.md.vm moving the source code of hadoop-tools docs to the directory under hadoop-tools --- Key: HADOOP-10976 URL: https://issues.apache.org/jira/browse/HADOOP-10976 Project: Hadoop Common Issue Type: Improvement Components: documentation Reporter: Masatake Iwasaki Assignee: Masatake Iwasaki Priority: Minor Fix For: 3.0.0 Attachments: HADOOP-10976-0.patch, HADOOP-10976-1.patch, HADOOP-10976-3.patch, HADOOP-10976-4.patch Some of the doc files of hadoop-tools are placed in the mapreduce project. It should be moved for the ease of maintenance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11543) Improve help message for hadoop/yarn command
[ https://issues.apache.org/jira/browse/HADOOP-11543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307265#comment-14307265 ] Hudson commented on HADOOP-11543: - SUCCESS: Integrated in Hadoop-Hdfs-trunk #2027 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2027/]) HADOOP-11543. Improve help message for hadoop/yarn command. Contributed by Brahma Reddy Battula. (ozawa: rev ec950be1735a877675c1c9493846e5b960f373c4) * hadoop-yarn-project/hadoop-yarn/bin/yarn * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-common-project/hadoop-common/src/main/bin/hadoop Improve help message for hadoop/yarn command Key: HADOOP-11543 URL: https://issues.apache.org/jira/browse/HADOOP-11543 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 2.6.0 Reporter: Jagadesh Kiran N Assignee: Brahma Reddy Battula Priority: Trivial Fix For: 2.7.0 Attachments: HADOOP-11543-001.patch, HADOOP-11543-002.patch, HADOOP-11543-002.patch, HADOOP-11543-branch-2.001.patch, HADOOP-11543-branch-2.002, HADOOP-11543.patch, OrinYarncommnad.png, YARN-3128-01.patch, YARN-3128.patch Pls check the snapshot attached -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11548) checknative should display a nicer error message when openssl support is not compiled in
[
https://issues.apache.org/jira/browse/HADOOP-11548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307269#comment-14307269
]
Hudson commented on HADOOP-11548:
-
SUCCESS: Integrated in Hadoop-Hdfs-trunk #2027 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2027/])
HADOOP-11548. checknative should display a nicer error message when openssl
support is not compiled in. Contributed by Anu Engineer. (cnauroth: rev
064e0770a0f32e7f32a7ba16c71394d7e75a3eb3)
* hadoop-common-project/hadoop-common/src/CMakeLists.txt
* hadoop-common-project/hadoop-common/CHANGES.txt
checknative should display a nicer error message when openssl support is not
compiled in
Key: HADOOP-11548
URL: https://issues.apache.org/jira/browse/HADOOP-11548
Project: Hadoop Common
Issue Type: Bug
Components: build, native
Affects Versions: 2.6.0
Reporter: Colin Patrick McCabe
Assignee: Anu Engineer
Fix For: 2.7.0
Attachments: HDFS-7520.001.patch
checknative should display a nicer error message when openssl support is not
compiled in. Currently, it displays this:
{code}
[cmccabe@keter hadoop]$ hadoop checknative
14/12/12 14:08:43 INFO bzip2.Bzip2Factory: Successfully loaded initialized
native-bzip2 library system-native
14/12/12 14:08:43 INFO zlib.ZlibFactory: Successfully loaded initialized
native-zlib library
Native library checking:
hadoop: true /usr/lib/hadoop/lib/native/libhadoop.so.1.0.0
zlib:true /lib64/libz.so.1
snappy: true /usr/lib64/libsnappy.so.1
lz4: true revision:99
bzip2: true /lib64/libbz2.so.1
openssl: false org.apache.hadoop.crypto.OpensslCipher.initIDs()V
{code}
Instead, we should display something like this, if openssl is not supported
by the current build:
{code}
openssl: false Hadoop was built without openssl support.
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11547) hadoop-common native compilation fails on Windows due to missing support for __attribute__ declaration.
[
https://issues.apache.org/jira/browse/HADOOP-11547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307348#comment-14307348
]
Hudson commented on HADOOP-11547:
-
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #96 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/96/])
HADOOP-11547. hadoop-common native compilation fails on Windows due to missing
support for __attribute__ declaration. Contributed by Chris Nauroth. (cnauroth:
rev a35e86c6701ffec263966bab31bd9368923c7e80)
* hadoop-common-project/hadoop-common/src/main/native/src/exception.h
* hadoop-common-project/hadoop-common/CHANGES.txt
hadoop-common native compilation fails on Windows due to missing support for
__attribute__ declaration.
---
Key: HADOOP-11547
URL: https://issues.apache.org/jira/browse/HADOOP-11547
Project: Hadoop Common
Issue Type: Bug
Components: native
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Fix For: 2.7.0
Attachments: HADOOP-11547.001.patch, HADOOP-11547.002.patch,
HADOOP-11547.003.patch
HADOOP-11403 made a change to include exception.h in NativeIO.c. This header
includes use of the non-standard gcc {{\_\_attribute\_\_}} declaration, and
thus fails compilation on Windows.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11549) flaky test detection tool failed to handle special control characters in test result
[
https://issues.apache.org/jira/browse/HADOOP-11549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307273#comment-14307273
]
Hudson commented on HADOOP-11549:
-
SUCCESS: Integrated in Hadoop-Hdfs-trunk #2027 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2027/])
HADOOP-11549. flaky test detection tool failed to handle special control
characters in test result. Contributed by Yongjun Zhang. (aajisaka: rev
5f4ef2d13fa3fa3d4d315902e55c7775ef08b3d2)
* hadoop-common-project/hadoop-common/CHANGES.txt
* dev-support/determine-flaky-tests-hadoop.py
flaky test detection tool failed to handle special control characters in test
result
Key: HADOOP-11549
URL: https://issues.apache.org/jira/browse/HADOOP-11549
Project: Hadoop Common
Issue Type: Bug
Components: tools
Affects Versions: 2.7.0
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
Fix For: 2.7.0
Attachments: HADOOP-11549.001.patch
When running the tool from HADOOP-11045 on latest Hadoop-hdfs-trunk job, I'm
seeing a problem here:
{code}
[yzhang@localhost jenkinsftf]$ ./determine-flaky-tests-hadoop.py -j
Hadoop-hdfs-trunk
Recently FAILED builds in url:
https://builds.apache.org/job/Hadoop-hdfs-trunk
THERE ARE 5 builds (out of 6) that have failed tests in the past 14 days,
as listed below:
===https://builds.apache.org/job/Hadoop-hdfs-trunk/2026/testReport
(2015-02-04 03:30:00)
Could not open testReport, check
https://builds.apache.org/job/Hadoop-hdfs-trunk/2026/Console for why it was
reported failed
...
{code}
I saw that the testReport can actually be opened from browser. After looking,
I found that HDFS-7287 fix added the following test code:
{code}
//Create a directory whose name should be escaped in XML
Path invalidXMLDir = new Path(/dirContainingInvalidXMLChar\uhere);
hdfs.mkdirs(invalidXMLDir);
...
{code}
And the output from this code caused the tool to choke.
I found a solution here and I'm attaching a patch.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11492) Bump up curator version to 2.7.1
[ https://issues.apache.org/jira/browse/HADOOP-11492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307272#comment-14307272 ] Hudson commented on HADOOP-11492: - SUCCESS: Integrated in Hadoop-Hdfs-trunk #2027 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2027/]) HADOOP-11492. Bump up curator version to 2.7.1. (Arun Suresh and Karthik Kambatla via kasha) (kasha: rev 55235d28c3641ff5f1c05c53dd7271cafc131926) * hadoop-project/pom.xml * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java * hadoop-common-project/hadoop-common/CHANGES.txt Bump up curator version to 2.7.1 Key: HADOOP-11492 URL: https://issues.apache.org/jira/browse/HADOOP-11492 Project: Hadoop Common Issue Type: Task Affects Versions: 2.6.0 Reporter: Karthik Kambatla Assignee: Arun Suresh Fix For: 2.7.0 Attachments: hadoop-11492-1.patch, hadoop-11492-2.patch, hadoop-11492-3.patch, hadoop-11492-3.patch Curator 2.7.1 got released recently and contains CURATOR-111 that YARN-2716 requires. PS: Filing a common JIRA so folks from other sub-projects also notice this change and shout out if there are any reservations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11544) Remove unused configuration keys for tracing
[ https://issues.apache.org/jira/browse/HADOOP-11544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307266#comment-14307266 ] Hudson commented on HADOOP-11544: - SUCCESS: Integrated in Hadoop-Hdfs-trunk #2027 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2027/]) HADOOP-11544. Remove unused configuration keys for tracing. Contributed by Masatake Iwasaki. (aajisaka: rev 42548f4dc2b2e8ace891b970c2c6712c02bb43ea) * hadoop-common-project/hadoop-common/src/main/resources/core-default.xml * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java * hadoop-common-project/hadoop-common/CHANGES.txt Remove unused configuration keys for tracing Key: HADOOP-11544 URL: https://issues.apache.org/jira/browse/HADOOP-11544 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.7.0 Reporter: Masatake Iwasaki Assignee: Masatake Iwasaki Priority: Trivial Fix For: 2.7.0 Attachments: HADOOP-11544.001.patch, HADOOP-11544.002.patch CommonConfigurationKeys.HADOOP_TRACE_SAMPLER* are no longer used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11346) Rewrite sls/rumen to use new shell framework
[ https://issues.apache.org/jira/browse/HADOOP-11346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307281#comment-14307281 ] Hudson commented on HADOOP-11346: - SUCCESS: Integrated in Hadoop-Hdfs-trunk #2027 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2027/]) HADOOP-11346. Rewrite sls/rumen to use new shell framework (John Smith via aw) (aw: rev f990e9d229d3b83e2f2ce5b1921e2d3e7d318dca) * hadoop-tools/hadoop-sls/src/main/bin/rumen2sls.sh * hadoop-tools/hadoop-sls/src/main/bin/slsrun.sh * hadoop-common-project/hadoop-common/CHANGES.txt Rewrite sls/rumen to use new shell framework Key: HADOOP-11346 URL: https://issues.apache.org/jira/browse/HADOOP-11346 Project: Hadoop Common Issue Type: Improvement Components: scripts, tools Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: John Smith Fix For: 3.0.0 Attachments: HADOOP-11346-01.patch, HADOOP-11346-02.patch, HADOOP-11346.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11546) Checkstyle failing: Unable to instantiate DoubleCheckedLockingCheck
[
https://issues.apache.org/jira/browse/HADOOP-11546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307268#comment-14307268
]
Hudson commented on HADOOP-11546:
-
SUCCESS: Integrated in Hadoop-Hdfs-trunk #2027 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2027/])
HADOOP-11546. Checkstyle failing: Unable to instantiate
DoubleCheckedLockingCheck. (ozawa) (ozawa: rev
ba58da29f49da15f7eb6300f996e0ae7ba9d0d1e)
* hadoop-common-project/hadoop-common/CHANGES.txt
* hadoop-tools/hadoop-azure/src/config/checkstyle.xml
* hadoop-common-project/hadoop-common/dev-support/checkstyle.xml
* hadoop-hdfs-project/hadoop-hdfs/dev-support/checkstyle.xml
Checkstyle failing: Unable to instantiate DoubleCheckedLockingCheck
---
Key: HADOOP-11546
URL: https://issues.apache.org/jira/browse/HADOOP-11546
Project: Hadoop Common
Issue Type: Bug
Components: build
Environment: Jenkins HDFS builds; java 8
Reporter: Steve Loughran
Assignee: Tsuyoshi OZAWA
Fix For: 2.7.0
Attachments: HADOOP-11546.1.patch
HDFS builds are failing in jenkins
{code}
Failed to execute goal
org.apache.maven.plugins:maven-checkstyle-plugin:2.12.1:checkstyle
(default-cli) on project hadoop-hdfs: An error has occurred in Checkstyle
report generation. Failed during checkstyle configuration: cannot initialize
module TreeWalker - Unable to instantiate DoubleCheckedLocking: Unable to
instantiate DoubleCheckedLockingCheck -
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11460) Deprecate shell vars
[ https://issues.apache.org/jira/browse/HADOOP-11460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307280#comment-14307280 ] Hudson commented on HADOOP-11460: - SUCCESS: Integrated in Hadoop-Hdfs-trunk #2027 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2027/]) HADOOP-11460. Deprecate shell vars (John Smith via aw) (aw: rev 43d5caef5e63b3c96737cb642c1e8f31ec8c9a8c) * hadoop-yarn-project/hadoop-yarn/bin/stop-yarn.sh * hadoop-common-project/hadoop-kms/src/main/conf/kms-env.sh * hadoop-common-project/hadoop-kms/src/main/libexec/kms-config.sh * hadoop-yarn-project/hadoop-yarn/bin/start-yarn.sh * hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh * hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh * hadoop-mapreduce-project/conf/mapred-env.sh * hadoop-mapreduce-project/bin/mapred-config.sh * hadoop-yarn-project/hadoop-yarn/conf/yarn-env.sh * hadoop-yarn-project/hadoop-yarn/bin/yarn * hadoop-yarn-project/hadoop-yarn/bin/yarn-config.sh * hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs-config.sh * hadoop-common-project/hadoop-common/CHANGES.txt Deprecate shell vars Key: HADOOP-11460 URL: https://issues.apache.org/jira/browse/HADOOP-11460 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: John Smith Labels: scripts, shell Fix For: 3.0.0 Attachments: HADOOP-11460-00.patch, HADOOP-11460-01.patch, HADOOP-11460-02.patch, HADOOP-11460-03.patch, HADOOP-11460-04.patch It is a very common shell pattern in 3.x to effectively replace sub-project specific vars with generics. We should have a function that does this replacement and provides a warning to the end user that the old shell var is deprecated. Additionally, we should use this shell function to deprecate the shell vars that are holdovers already. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11544) Remove unused configuration keys for tracing
[ https://issues.apache.org/jira/browse/HADOOP-11544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307297#comment-14307297 ] Hudson commented on HADOOP-11544: - SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #92 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/92/]) HADOOP-11544. Remove unused configuration keys for tracing. Contributed by Masatake Iwasaki. (aajisaka: rev 42548f4dc2b2e8ace891b970c2c6712c02bb43ea) * hadoop-common-project/hadoop-common/src/main/resources/core-default.xml * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java * hadoop-common-project/hadoop-common/CHANGES.txt Remove unused configuration keys for tracing Key: HADOOP-11544 URL: https://issues.apache.org/jira/browse/HADOOP-11544 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.7.0 Reporter: Masatake Iwasaki Assignee: Masatake Iwasaki Priority: Trivial Fix For: 2.7.0 Attachments: HADOOP-11544.001.patch, HADOOP-11544.002.patch CommonConfigurationKeys.HADOOP_TRACE_SAMPLER* are no longer used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11492) Bump up curator version to 2.7.1
[ https://issues.apache.org/jira/browse/HADOOP-11492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307303#comment-14307303 ] Hudson commented on HADOOP-11492: - SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #92 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/92/]) HADOOP-11492. Bump up curator version to 2.7.1. (Arun Suresh and Karthik Kambatla via kasha) (kasha: rev 55235d28c3641ff5f1c05c53dd7271cafc131926) * hadoop-project/pom.xml * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java Bump up curator version to 2.7.1 Key: HADOOP-11492 URL: https://issues.apache.org/jira/browse/HADOOP-11492 Project: Hadoop Common Issue Type: Task Affects Versions: 2.6.0 Reporter: Karthik Kambatla Assignee: Arun Suresh Fix For: 2.7.0 Attachments: hadoop-11492-1.patch, hadoop-11492-2.patch, hadoop-11492-3.patch, hadoop-11492-3.patch Curator 2.7.1 got released recently and contains CURATOR-111 that YARN-2716 requires. PS: Filing a common JIRA so folks from other sub-projects also notice this change and shout out if there are any reservations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11546) Checkstyle failing: Unable to instantiate DoubleCheckedLockingCheck
[
https://issues.apache.org/jira/browse/HADOOP-11546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307299#comment-14307299
]
Hudson commented on HADOOP-11546:
-
SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #92 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/92/])
HADOOP-11546. Checkstyle failing: Unable to instantiate
DoubleCheckedLockingCheck. (ozawa) (ozawa: rev
ba58da29f49da15f7eb6300f996e0ae7ba9d0d1e)
* hadoop-common-project/hadoop-common/dev-support/checkstyle.xml
* hadoop-tools/hadoop-azure/src/config/checkstyle.xml
* hadoop-hdfs-project/hadoop-hdfs/dev-support/checkstyle.xml
* hadoop-common-project/hadoop-common/CHANGES.txt
Checkstyle failing: Unable to instantiate DoubleCheckedLockingCheck
---
Key: HADOOP-11546
URL: https://issues.apache.org/jira/browse/HADOOP-11546
Project: Hadoop Common
Issue Type: Bug
Components: build
Environment: Jenkins HDFS builds; java 8
Reporter: Steve Loughran
Assignee: Tsuyoshi OZAWA
Fix For: 2.7.0
Attachments: HADOOP-11546.1.patch
HDFS builds are failing in jenkins
{code}
Failed to execute goal
org.apache.maven.plugins:maven-checkstyle-plugin:2.12.1:checkstyle
(default-cli) on project hadoop-hdfs: An error has occurred in Checkstyle
report generation. Failed during checkstyle configuration: cannot initialize
module TreeWalker - Unable to instantiate DoubleCheckedLocking: Unable to
instantiate DoubleCheckedLockingCheck -
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11346) Rewrite sls/rumen to use new shell framework
[ https://issues.apache.org/jira/browse/HADOOP-11346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307312#comment-14307312 ] Hudson commented on HADOOP-11346: - SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #92 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/92/]) HADOOP-11346. Rewrite sls/rumen to use new shell framework (John Smith via aw) (aw: rev f990e9d229d3b83e2f2ce5b1921e2d3e7d318dca) * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-tools/hadoop-sls/src/main/bin/rumen2sls.sh * hadoop-tools/hadoop-sls/src/main/bin/slsrun.sh Rewrite sls/rumen to use new shell framework Key: HADOOP-11346 URL: https://issues.apache.org/jira/browse/HADOOP-11346 Project: Hadoop Common Issue Type: Improvement Components: scripts, tools Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: John Smith Fix For: 3.0.0 Attachments: HADOOP-11346-01.patch, HADOOP-11346-02.patch, HADOOP-11346.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11548) checknative should display a nicer error message when openssl support is not compiled in
[
https://issues.apache.org/jira/browse/HADOOP-11548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307300#comment-14307300
]
Hudson commented on HADOOP-11548:
-
SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #92 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/92/])
HADOOP-11548. checknative should display a nicer error message when openssl
support is not compiled in. Contributed by Anu Engineer. (cnauroth: rev
064e0770a0f32e7f32a7ba16c71394d7e75a3eb3)
* hadoop-common-project/hadoop-common/src/CMakeLists.txt
* hadoop-common-project/hadoop-common/CHANGES.txt
checknative should display a nicer error message when openssl support is not
compiled in
Key: HADOOP-11548
URL: https://issues.apache.org/jira/browse/HADOOP-11548
Project: Hadoop Common
Issue Type: Bug
Components: build, native
Affects Versions: 2.6.0
Reporter: Colin Patrick McCabe
Assignee: Anu Engineer
Fix For: 2.7.0
Attachments: HDFS-7520.001.patch
checknative should display a nicer error message when openssl support is not
compiled in. Currently, it displays this:
{code}
[cmccabe@keter hadoop]$ hadoop checknative
14/12/12 14:08:43 INFO bzip2.Bzip2Factory: Successfully loaded initialized
native-bzip2 library system-native
14/12/12 14:08:43 INFO zlib.ZlibFactory: Successfully loaded initialized
native-zlib library
Native library checking:
hadoop: true /usr/lib/hadoop/lib/native/libhadoop.so.1.0.0
zlib:true /lib64/libz.so.1
snappy: true /usr/lib64/libsnappy.so.1
lz4: true revision:99
bzip2: true /lib64/libbz2.so.1
openssl: false org.apache.hadoop.crypto.OpensslCipher.initIDs()V
{code}
Instead, we should display something like this, if openssl is not supported
by the current build:
{code}
openssl: false Hadoop was built without openssl support.
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11549) flaky test detection tool failed to handle special control characters in test result
[
https://issues.apache.org/jira/browse/HADOOP-11549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307304#comment-14307304
]
Hudson commented on HADOOP-11549:
-
SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #92 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/92/])
HADOOP-11549. flaky test detection tool failed to handle special control
characters in test result. Contributed by Yongjun Zhang. (aajisaka: rev
5f4ef2d13fa3fa3d4d315902e55c7775ef08b3d2)
* hadoop-common-project/hadoop-common/CHANGES.txt
* dev-support/determine-flaky-tests-hadoop.py
flaky test detection tool failed to handle special control characters in test
result
Key: HADOOP-11549
URL: https://issues.apache.org/jira/browse/HADOOP-11549
Project: Hadoop Common
Issue Type: Bug
Components: tools
Affects Versions: 2.7.0
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
Fix For: 2.7.0
Attachments: HADOOP-11549.001.patch
When running the tool from HADOOP-11045 on latest Hadoop-hdfs-trunk job, I'm
seeing a problem here:
{code}
[yzhang@localhost jenkinsftf]$ ./determine-flaky-tests-hadoop.py -j
Hadoop-hdfs-trunk
Recently FAILED builds in url:
https://builds.apache.org/job/Hadoop-hdfs-trunk
THERE ARE 5 builds (out of 6) that have failed tests in the past 14 days,
as listed below:
===https://builds.apache.org/job/Hadoop-hdfs-trunk/2026/testReport
(2015-02-04 03:30:00)
Could not open testReport, check
https://builds.apache.org/job/Hadoop-hdfs-trunk/2026/Console for why it was
reported failed
...
{code}
I saw that the testReport can actually be opened from browser. After looking,
I found that HDFS-7287 fix added the following test code:
{code}
//Create a directory whose name should be escaped in XML
Path invalidXMLDir = new Path(/dirContainingInvalidXMLChar\uhere);
hdfs.mkdirs(invalidXMLDir);
...
{code}
And the output from this code caused the tool to choke.
I found a solution here and I'm attaching a patch.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11460) Deprecate shell vars
[ https://issues.apache.org/jira/browse/HADOOP-11460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307311#comment-14307311 ] Hudson commented on HADOOP-11460: - SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #92 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/92/]) HADOOP-11460. Deprecate shell vars (John Smith via aw) (aw: rev 43d5caef5e63b3c96737cb642c1e8f31ec8c9a8c) * hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh * hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh * hadoop-yarn-project/hadoop-yarn/bin/yarn-config.sh * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-mapreduce-project/bin/mapred-config.sh * hadoop-common-project/hadoop-kms/src/main/libexec/kms-config.sh * hadoop-yarn-project/hadoop-yarn/bin/stop-yarn.sh * hadoop-mapreduce-project/conf/mapred-env.sh * hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs-config.sh * hadoop-common-project/hadoop-kms/src/main/conf/kms-env.sh * hadoop-yarn-project/hadoop-yarn/conf/yarn-env.sh * hadoop-yarn-project/hadoop-yarn/bin/start-yarn.sh * hadoop-yarn-project/hadoop-yarn/bin/yarn Deprecate shell vars Key: HADOOP-11460 URL: https://issues.apache.org/jira/browse/HADOOP-11460 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: John Smith Labels: scripts, shell Fix For: 3.0.0 Attachments: HADOOP-11460-00.patch, HADOOP-11460-01.patch, HADOOP-11460-02.patch, HADOOP-11460-03.patch, HADOOP-11460-04.patch It is a very common shell pattern in 3.x to effectively replace sub-project specific vars with generics. We should have a function that does this replacement and provides a warning to the end user that the old shell var is deprecated. Additionally, we should use this shell function to deprecate the shell vars that are holdovers already. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11547) hadoop-common native compilation fails on Windows due to missing support for __attribute__ declaration.
[
https://issues.apache.org/jira/browse/HADOOP-11547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307302#comment-14307302
]
Hudson commented on HADOOP-11547:
-
SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #92 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/92/])
HADOOP-11547. hadoop-common native compilation fails on Windows due to missing
support for __attribute__ declaration. Contributed by Chris Nauroth. (cnauroth:
rev a35e86c6701ffec263966bab31bd9368923c7e80)
* hadoop-common-project/hadoop-common/CHANGES.txt
* hadoop-common-project/hadoop-common/src/main/native/src/exception.h
hadoop-common native compilation fails on Windows due to missing support for
__attribute__ declaration.
---
Key: HADOOP-11547
URL: https://issues.apache.org/jira/browse/HADOOP-11547
Project: Hadoop Common
Issue Type: Bug
Components: native
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Fix For: 2.7.0
Attachments: HADOOP-11547.001.patch, HADOOP-11547.002.patch,
HADOOP-11547.003.patch
HADOOP-11403 made a change to include exception.h in NativeIO.c. This header
includes use of the non-standard gcc {{\_\_attribute\_\_}} declaration, and
thus fails compilation on Windows.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-10062) race condition in MetricsSystemImpl#publishMetricsNow that causes incorrect results
[
https://issues.apache.org/jira/browse/HADOOP-10062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307294#comment-14307294
]
Hudson commented on HADOOP-10062:
-
SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #92 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/92/])
HADOOP-10062. race condition in MetricsSystemImpl#publishMetricsNow that causes
incorrect results. (Contributed by Sangjin Lee) (junping_du: rev
0b567f424673b5cea1c3bc23b2bd268ef6b7625f)
*
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java
* hadoop-common-project/hadoop-common/CHANGES.txt
*
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestMetricsSystemImpl.java
race condition in MetricsSystemImpl#publishMetricsNow that causes incorrect
results
---
Key: HADOOP-10062
URL: https://issues.apache.org/jira/browse/HADOOP-10062
Project: Hadoop Common
Issue Type: Bug
Components: metrics
Affects Versions: 3.0.0
Environment: CentOS 6.4, Oracle JDK 1.6.0_31, JDK1.7.0_45
Reporter: Shinichi Yamashita
Assignee: Sangjin Lee
Fix For: 3.0.0, 2.7.0
Attachments: HADOOP-10062-failed.txt, HADOOP-10062-success.txt,
HADOOP-10062.003.patch, HADOOP-10062.patch, HADOOP-10062.patch
TestMetricsSystemInpl#testMultiThreadedPublish failed with Metrics not
collected
{code}
Running org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl
Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.688 sec
FAILURE! - in org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl
testMultiThreadedPublish(org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl)
Time elapsed: 0.056 sec FAILURE!
java.lang.AssertionError: Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Passed
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at
org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl.testMultiThreadedPublish(TestMetricsSystemImpl.java:232)
Results :
Failed tests:
TestMetricsSystemImpl.testMultiThreadedPublish:232 Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Passed
Tests run: 6, Failures: 1, Errors: 0, Skipped: 0
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11543) Improve help message for hadoop/yarn command
[ https://issues.apache.org/jira/browse/HADOOP-11543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307343#comment-14307343 ] Hudson commented on HADOOP-11543: - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #96 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/96/]) HADOOP-11543. Improve help message for hadoop/yarn command. Contributed by Brahma Reddy Battula. (ozawa: rev ec950be1735a877675c1c9493846e5b960f373c4) * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/bin/yarn * hadoop-common-project/hadoop-common/src/main/bin/hadoop Improve help message for hadoop/yarn command Key: HADOOP-11543 URL: https://issues.apache.org/jira/browse/HADOOP-11543 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 2.6.0 Reporter: Jagadesh Kiran N Assignee: Brahma Reddy Battula Priority: Trivial Fix For: 2.7.0 Attachments: HADOOP-11543-001.patch, HADOOP-11543-002.patch, HADOOP-11543-002.patch, HADOOP-11543-branch-2.001.patch, HADOOP-11543-branch-2.002, HADOOP-11543.patch, OrinYarncommnad.png, YARN-3128-01.patch, YARN-3128.patch Pls check the snapshot attached -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11548) checknative should display a nicer error message when openssl support is not compiled in
[
https://issues.apache.org/jira/browse/HADOOP-11548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307346#comment-14307346
]
Hudson commented on HADOOP-11548:
-
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #96 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/96/])
HADOOP-11548. checknative should display a nicer error message when openssl
support is not compiled in. Contributed by Anu Engineer. (cnauroth: rev
064e0770a0f32e7f32a7ba16c71394d7e75a3eb3)
* hadoop-common-project/hadoop-common/src/CMakeLists.txt
* hadoop-common-project/hadoop-common/CHANGES.txt
checknative should display a nicer error message when openssl support is not
compiled in
Key: HADOOP-11548
URL: https://issues.apache.org/jira/browse/HADOOP-11548
Project: Hadoop Common
Issue Type: Bug
Components: build, native
Affects Versions: 2.6.0
Reporter: Colin Patrick McCabe
Assignee: Anu Engineer
Fix For: 2.7.0
Attachments: HDFS-7520.001.patch
checknative should display a nicer error message when openssl support is not
compiled in. Currently, it displays this:
{code}
[cmccabe@keter hadoop]$ hadoop checknative
14/12/12 14:08:43 INFO bzip2.Bzip2Factory: Successfully loaded initialized
native-bzip2 library system-native
14/12/12 14:08:43 INFO zlib.ZlibFactory: Successfully loaded initialized
native-zlib library
Native library checking:
hadoop: true /usr/lib/hadoop/lib/native/libhadoop.so.1.0.0
zlib:true /lib64/libz.so.1
snappy: true /usr/lib64/libsnappy.so.1
lz4: true revision:99
bzip2: true /lib64/libbz2.so.1
openssl: false org.apache.hadoop.crypto.OpensslCipher.initIDs()V
{code}
Instead, we should display something like this, if openssl is not supported
by the current build:
{code}
openssl: false Hadoop was built without openssl support.
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11546) Checkstyle failing: Unable to instantiate DoubleCheckedLockingCheck
[
https://issues.apache.org/jira/browse/HADOOP-11546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307345#comment-14307345
]
Hudson commented on HADOOP-11546:
-
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #96 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/96/])
HADOOP-11546. Checkstyle failing: Unable to instantiate
DoubleCheckedLockingCheck. (ozawa) (ozawa: rev
ba58da29f49da15f7eb6300f996e0ae7ba9d0d1e)
* hadoop-hdfs-project/hadoop-hdfs/dev-support/checkstyle.xml
* hadoop-common-project/hadoop-common/dev-support/checkstyle.xml
* hadoop-tools/hadoop-azure/src/config/checkstyle.xml
* hadoop-common-project/hadoop-common/CHANGES.txt
Checkstyle failing: Unable to instantiate DoubleCheckedLockingCheck
---
Key: HADOOP-11546
URL: https://issues.apache.org/jira/browse/HADOOP-11546
Project: Hadoop Common
Issue Type: Bug
Components: build
Environment: Jenkins HDFS builds; java 8
Reporter: Steve Loughran
Assignee: Tsuyoshi OZAWA
Fix For: 2.7.0
Attachments: HADOOP-11546.1.patch
HDFS builds are failing in jenkins
{code}
Failed to execute goal
org.apache.maven.plugins:maven-checkstyle-plugin:2.12.1:checkstyle
(default-cli) on project hadoop-hdfs: An error has occurred in Checkstyle
report generation. Failed during checkstyle configuration: cannot initialize
module TreeWalker - Unable to instantiate DoubleCheckedLocking: Unable to
instantiate DoubleCheckedLockingCheck -
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-10062) race condition in MetricsSystemImpl#publishMetricsNow that causes incorrect results
[
https://issues.apache.org/jira/browse/HADOOP-10062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307341#comment-14307341
]
Hudson commented on HADOOP-10062:
-
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #96 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/96/])
HADOOP-10062. race condition in MetricsSystemImpl#publishMetricsNow that causes
incorrect results. (Contributed by Sangjin Lee) (junping_du: rev
0b567f424673b5cea1c3bc23b2bd268ef6b7625f)
*
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java
* hadoop-common-project/hadoop-common/CHANGES.txt
*
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestMetricsSystemImpl.java
race condition in MetricsSystemImpl#publishMetricsNow that causes incorrect
results
---
Key: HADOOP-10062
URL: https://issues.apache.org/jira/browse/HADOOP-10062
Project: Hadoop Common
Issue Type: Bug
Components: metrics
Affects Versions: 3.0.0
Environment: CentOS 6.4, Oracle JDK 1.6.0_31, JDK1.7.0_45
Reporter: Shinichi Yamashita
Assignee: Sangjin Lee
Fix For: 3.0.0, 2.7.0
Attachments: HADOOP-10062-failed.txt, HADOOP-10062-success.txt,
HADOOP-10062.003.patch, HADOOP-10062.patch, HADOOP-10062.patch
TestMetricsSystemInpl#testMultiThreadedPublish failed with Metrics not
collected
{code}
Running org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl
Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.688 sec
FAILURE! - in org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl
testMultiThreadedPublish(org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl)
Time elapsed: 0.056 sec FAILURE!
java.lang.AssertionError: Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Passed
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at
org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl.testMultiThreadedPublish(TestMetricsSystemImpl.java:232)
Results :
Failed tests:
TestMetricsSystemImpl.testMultiThreadedPublish:232 Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Passed
Tests run: 6, Failures: 1, Errors: 0, Skipped: 0
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11549) flaky test detection tool failed to handle special control characters in test result
[
https://issues.apache.org/jira/browse/HADOOP-11549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307350#comment-14307350
]
Hudson commented on HADOOP-11549:
-
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #96 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/96/])
HADOOP-11549. flaky test detection tool failed to handle special control
characters in test result. Contributed by Yongjun Zhang. (aajisaka: rev
5f4ef2d13fa3fa3d4d315902e55c7775ef08b3d2)
* dev-support/determine-flaky-tests-hadoop.py
* hadoop-common-project/hadoop-common/CHANGES.txt
flaky test detection tool failed to handle special control characters in test
result
Key: HADOOP-11549
URL: https://issues.apache.org/jira/browse/HADOOP-11549
Project: Hadoop Common
Issue Type: Bug
Components: tools
Affects Versions: 2.7.0
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
Fix For: 2.7.0
Attachments: HADOOP-11549.001.patch
When running the tool from HADOOP-11045 on latest Hadoop-hdfs-trunk job, I'm
seeing a problem here:
{code}
[yzhang@localhost jenkinsftf]$ ./determine-flaky-tests-hadoop.py -j
Hadoop-hdfs-trunk
Recently FAILED builds in url:
https://builds.apache.org/job/Hadoop-hdfs-trunk
THERE ARE 5 builds (out of 6) that have failed tests in the past 14 days,
as listed below:
===https://builds.apache.org/job/Hadoop-hdfs-trunk/2026/testReport
(2015-02-04 03:30:00)
Could not open testReport, check
https://builds.apache.org/job/Hadoop-hdfs-trunk/2026/Console for why it was
reported failed
...
{code}
I saw that the testReport can actually be opened from browser. After looking,
I found that HDFS-7287 fix added the following test code:
{code}
//Create a directory whose name should be escaped in XML
Path invalidXMLDir = new Path(/dirContainingInvalidXMLChar\uhere);
hdfs.mkdirs(invalidXMLDir);
...
{code}
And the output from this code caused the tool to choke.
I found a solution here and I'm attaching a patch.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-10976) moving the source code of hadoop-tools docs to the directory under hadoop-tools
[ https://issues.apache.org/jira/browse/HADOOP-10976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307352#comment-14307352 ] Hudson commented on HADOOP-10976: - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #96 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/96/]) HADOOP-10976. moving the source code of hadoop-tools docs to the directory under hadoop-tools (Masatake Iwasaki via aw) (aw: rev 9112f093cde0b8242324aa5267c9beb21c38bf6b) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/DistCp.md.vm * hadoop-project/src/site/site.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/HadoopArchives.md.vm * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-tools/hadoop-archives/src/site/markdown/HadoopArchives.md.vm * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/apt/HadoopStreaming.apt.vm * hadoop-tools/hadoop-distcp/src/site/markdown/DistCp.md.vm * hadoop-tools/hadoop-streaming/src/site/resources/css/site.css * hadoop-tools/hadoop-streaming/src/site/apt/HadoopStreaming.apt.vm * hadoop-tools/hadoop-distcp/src/site/resources/css/site.css * hadoop-tools/hadoop-archives/src/site/resources/css/site.css moving the source code of hadoop-tools docs to the directory under hadoop-tools --- Key: HADOOP-10976 URL: https://issues.apache.org/jira/browse/HADOOP-10976 Project: Hadoop Common Issue Type: Improvement Components: documentation Reporter: Masatake Iwasaki Assignee: Masatake Iwasaki Priority: Minor Fix For: 3.0.0 Attachments: HADOOP-10976-0.patch, HADOOP-10976-1.patch, HADOOP-10976-3.patch, HADOOP-10976-4.patch Some of the doc files of hadoop-tools are placed in the mapreduce project. It should be moved for the ease of maintenance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11492) Bump up curator version to 2.7.1
[ https://issues.apache.org/jira/browse/HADOOP-11492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307349#comment-14307349 ] Hudson commented on HADOOP-11492: - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #96 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/96/]) HADOOP-11492. Bump up curator version to 2.7.1. (Arun Suresh and Karthik Kambatla via kasha) (kasha: rev 55235d28c3641ff5f1c05c53dd7271cafc131926) * hadoop-project/pom.xml * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java Bump up curator version to 2.7.1 Key: HADOOP-11492 URL: https://issues.apache.org/jira/browse/HADOOP-11492 Project: Hadoop Common Issue Type: Task Affects Versions: 2.6.0 Reporter: Karthik Kambatla Assignee: Arun Suresh Fix For: 2.7.0 Attachments: hadoop-11492-1.patch, hadoop-11492-2.patch, hadoop-11492-3.patch, hadoop-11492-3.patch Curator 2.7.1 got released recently and contains CURATOR-111 that YARN-2716 requires. PS: Filing a common JIRA so folks from other sub-projects also notice this change and shout out if there are any reservations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11463) Replace method-local TransferManager object with S3AFileSystem#transfers
[ https://issues.apache.org/jira/browse/HADOOP-11463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307354#comment-14307354 ] Hudson commented on HADOOP-11463: - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #96 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/96/]) HADOOP-11463 Replace method-local TransferManager object with S3AFileSystem#transfers. (Ted Yu via stevel) (stevel: rev 4e7ad4f0a88bd36bc91db6b1bd311d7f5c6bebee) * hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java * hadoop-common-project/hadoop-common/CHANGES.txt Replace method-local TransferManager object with S3AFileSystem#transfers Key: HADOOP-11463 URL: https://issues.apache.org/jira/browse/HADOOP-11463 Project: Hadoop Common Issue Type: Task Components: fs/s3 Affects Versions: 2.7.0 Reporter: Ted Yu Assignee: Ted Yu Fix For: 2.7.0 Attachments: hadoop-11463-001.patch, hadoop-11463-002.patch, hadoop-11463-003.patch This is continuation of HADOOP-11446. The following changes are made according to Thomas Demoor's comments: 1. Replace method-local TransferManager object with S3AFileSystem#transfers 2. Do not shutdown TransferManager after purging existing multipart file - otherwise the current transfer is unable to proceed 3. Shutdown TransferManager instance in the close method of S3AFileSystem -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11346) Rewrite sls/rumen to use new shell framework
[ https://issues.apache.org/jira/browse/HADOOP-11346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307359#comment-14307359 ] Hudson commented on HADOOP-11346: - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #96 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/96/]) HADOOP-11346. Rewrite sls/rumen to use new shell framework (John Smith via aw) (aw: rev f990e9d229d3b83e2f2ce5b1921e2d3e7d318dca) * hadoop-tools/hadoop-sls/src/main/bin/rumen2sls.sh * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-tools/hadoop-sls/src/main/bin/slsrun.sh Rewrite sls/rumen to use new shell framework Key: HADOOP-11346 URL: https://issues.apache.org/jira/browse/HADOOP-11346 Project: Hadoop Common Issue Type: Improvement Components: scripts, tools Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: John Smith Fix For: 3.0.0 Attachments: HADOOP-11346-01.patch, HADOOP-11346-02.patch, HADOOP-11346.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11460) Deprecate shell vars
[ https://issues.apache.org/jira/browse/HADOOP-11460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307358#comment-14307358 ] Hudson commented on HADOOP-11460: - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #96 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/96/]) HADOOP-11460. Deprecate shell vars (John Smith via aw) (aw: rev 43d5caef5e63b3c96737cb642c1e8f31ec8c9a8c) * hadoop-mapreduce-project/conf/mapred-env.sh * hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh * hadoop-yarn-project/hadoop-yarn/bin/yarn * hadoop-yarn-project/hadoop-yarn/bin/stop-yarn.sh * hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh * hadoop-yarn-project/hadoop-yarn/bin/start-yarn.sh * hadoop-common-project/hadoop-kms/src/main/conf/kms-env.sh * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-mapreduce-project/bin/mapred-config.sh * hadoop-yarn-project/hadoop-yarn/conf/yarn-env.sh * hadoop-yarn-project/hadoop-yarn/bin/yarn-config.sh * hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs-config.sh * hadoop-common-project/hadoop-kms/src/main/libexec/kms-config.sh Deprecate shell vars Key: HADOOP-11460 URL: https://issues.apache.org/jira/browse/HADOOP-11460 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: John Smith Labels: scripts, shell Fix For: 3.0.0 Attachments: HADOOP-11460-00.patch, HADOOP-11460-01.patch, HADOOP-11460-02.patch, HADOOP-11460-03.patch, HADOOP-11460-04.patch It is a very common shell pattern in 3.x to effectively replace sub-project specific vars with generics. We should have a function that does this replacement and provides a warning to the end user that the old shell var is deprecated. Additionally, we should use this shell function to deprecate the shell vars that are holdovers already. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11346) Rewrite sls/rumen to use new shell framework
[ https://issues.apache.org/jira/browse/HADOOP-11346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307400#comment-14307400 ] Hudson commented on HADOOP-11346: - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2046 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2046/]) HADOOP-11346. Rewrite sls/rumen to use new shell framework (John Smith via aw) (aw: rev f990e9d229d3b83e2f2ce5b1921e2d3e7d318dca) * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-tools/hadoop-sls/src/main/bin/rumen2sls.sh * hadoop-tools/hadoop-sls/src/main/bin/slsrun.sh Rewrite sls/rumen to use new shell framework Key: HADOOP-11346 URL: https://issues.apache.org/jira/browse/HADOOP-11346 Project: Hadoop Common Issue Type: Improvement Components: scripts, tools Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: John Smith Fix For: 3.0.0 Attachments: HADOOP-11346-01.patch, HADOOP-11346-02.patch, HADOOP-11346.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11492) Bump up curator version to 2.7.1
[ https://issues.apache.org/jira/browse/HADOOP-11492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307390#comment-14307390 ] Hudson commented on HADOOP-11492: - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2046 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2046/]) HADOOP-11492. Bump up curator version to 2.7.1. (Arun Suresh and Karthik Kambatla via kasha) (kasha: rev 55235d28c3641ff5f1c05c53dd7271cafc131926) * hadoop-project/pom.xml * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/ZKDelegationTokenSecretManager.java * hadoop-common-project/hadoop-common/CHANGES.txt Bump up curator version to 2.7.1 Key: HADOOP-11492 URL: https://issues.apache.org/jira/browse/HADOOP-11492 Project: Hadoop Common Issue Type: Task Affects Versions: 2.6.0 Reporter: Karthik Kambatla Assignee: Arun Suresh Fix For: 2.7.0 Attachments: hadoop-11492-1.patch, hadoop-11492-2.patch, hadoop-11492-3.patch, hadoop-11492-3.patch Curator 2.7.1 got released recently and contains CURATOR-111 that YARN-2716 requires. PS: Filing a common JIRA so folks from other sub-projects also notice this change and shout out if there are any reservations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11547) hadoop-common native compilation fails on Windows due to missing support for __attribute__ declaration.
[
https://issues.apache.org/jira/browse/HADOOP-11547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307389#comment-14307389
]
Hudson commented on HADOOP-11547:
-
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2046 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2046/])
HADOOP-11547. hadoop-common native compilation fails on Windows due to missing
support for __attribute__ declaration. Contributed by Chris Nauroth. (cnauroth:
rev a35e86c6701ffec263966bab31bd9368923c7e80)
* hadoop-common-project/hadoop-common/src/main/native/src/exception.h
* hadoop-common-project/hadoop-common/CHANGES.txt
hadoop-common native compilation fails on Windows due to missing support for
__attribute__ declaration.
---
Key: HADOOP-11547
URL: https://issues.apache.org/jira/browse/HADOOP-11547
Project: Hadoop Common
Issue Type: Bug
Components: native
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Fix For: 2.7.0
Attachments: HADOOP-11547.001.patch, HADOOP-11547.002.patch,
HADOOP-11547.003.patch
HADOOP-11403 made a change to include exception.h in NativeIO.c. This header
includes use of the non-standard gcc {{\_\_attribute\_\_}} declaration, and
thus fails compilation on Windows.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11549) flaky test detection tool failed to handle special control characters in test result
[
https://issues.apache.org/jira/browse/HADOOP-11549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307391#comment-14307391
]
Hudson commented on HADOOP-11549:
-
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2046 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2046/])
HADOOP-11549. flaky test detection tool failed to handle special control
characters in test result. Contributed by Yongjun Zhang. (aajisaka: rev
5f4ef2d13fa3fa3d4d315902e55c7775ef08b3d2)
* hadoop-common-project/hadoop-common/CHANGES.txt
* dev-support/determine-flaky-tests-hadoop.py
flaky test detection tool failed to handle special control characters in test
result
Key: HADOOP-11549
URL: https://issues.apache.org/jira/browse/HADOOP-11549
Project: Hadoop Common
Issue Type: Bug
Components: tools
Affects Versions: 2.7.0
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
Fix For: 2.7.0
Attachments: HADOOP-11549.001.patch
When running the tool from HADOOP-11045 on latest Hadoop-hdfs-trunk job, I'm
seeing a problem here:
{code}
[yzhang@localhost jenkinsftf]$ ./determine-flaky-tests-hadoop.py -j
Hadoop-hdfs-trunk
Recently FAILED builds in url:
https://builds.apache.org/job/Hadoop-hdfs-trunk
THERE ARE 5 builds (out of 6) that have failed tests in the past 14 days,
as listed below:
===https://builds.apache.org/job/Hadoop-hdfs-trunk/2026/testReport
(2015-02-04 03:30:00)
Could not open testReport, check
https://builds.apache.org/job/Hadoop-hdfs-trunk/2026/Console for why it was
reported failed
...
{code}
I saw that the testReport can actually be opened from browser. After looking,
I found that HDFS-7287 fix added the following test code:
{code}
//Create a directory whose name should be escaped in XML
Path invalidXMLDir = new Path(/dirContainingInvalidXMLChar\uhere);
hdfs.mkdirs(invalidXMLDir);
...
{code}
And the output from this code caused the tool to choke.
I found a solution here and I'm attaching a patch.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11460) Deprecate shell vars
[ https://issues.apache.org/jira/browse/HADOOP-11460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307399#comment-14307399 ] Hudson commented on HADOOP-11460: - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2046 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2046/]) HADOOP-11460. Deprecate shell vars (John Smith via aw) (aw: rev 43d5caef5e63b3c96737cb642c1e8f31ec8c9a8c) * hadoop-yarn-project/hadoop-yarn/bin/yarn * hadoop-common-project/hadoop-kms/src/main/libexec/kms-config.sh * hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh * hadoop-yarn-project/hadoop-yarn/conf/yarn-env.sh * hadoop-common-project/hadoop-kms/src/main/conf/kms-env.sh * hadoop-yarn-project/hadoop-yarn/bin/stop-yarn.sh * hadoop-mapreduce-project/conf/mapred-env.sh * hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs-config.sh * hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh * hadoop-mapreduce-project/bin/mapred-config.sh * hadoop-yarn-project/hadoop-yarn/bin/start-yarn.sh * hadoop-yarn-project/hadoop-yarn/bin/yarn-config.sh * hadoop-common-project/hadoop-common/CHANGES.txt Deprecate shell vars Key: HADOOP-11460 URL: https://issues.apache.org/jira/browse/HADOOP-11460 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: John Smith Labels: scripts, shell Fix For: 3.0.0 Attachments: HADOOP-11460-00.patch, HADOOP-11460-01.patch, HADOOP-11460-02.patch, HADOOP-11460-03.patch, HADOOP-11460-04.patch It is a very common shell pattern in 3.x to effectively replace sub-project specific vars with generics. We should have a function that does this replacement and provides a warning to the end user that the old shell var is deprecated. Additionally, we should use this shell function to deprecate the shell vars that are holdovers already. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-10062) race condition in MetricsSystemImpl#publishMetricsNow that causes incorrect results
[
https://issues.apache.org/jira/browse/HADOOP-10062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307382#comment-14307382
]
Hudson commented on HADOOP-10062:
-
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2046 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2046/])
HADOOP-10062. race condition in MetricsSystemImpl#publishMetricsNow that causes
incorrect results. (Contributed by Sangjin Lee) (junping_du: rev
0b567f424673b5cea1c3bc23b2bd268ef6b7625f)
*
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestMetricsSystemImpl.java
*
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java
* hadoop-common-project/hadoop-common/CHANGES.txt
race condition in MetricsSystemImpl#publishMetricsNow that causes incorrect
results
---
Key: HADOOP-10062
URL: https://issues.apache.org/jira/browse/HADOOP-10062
Project: Hadoop Common
Issue Type: Bug
Components: metrics
Affects Versions: 3.0.0
Environment: CentOS 6.4, Oracle JDK 1.6.0_31, JDK1.7.0_45
Reporter: Shinichi Yamashita
Assignee: Sangjin Lee
Fix For: 3.0.0, 2.7.0
Attachments: HADOOP-10062-failed.txt, HADOOP-10062-success.txt,
HADOOP-10062.003.patch, HADOOP-10062.patch, HADOOP-10062.patch
TestMetricsSystemInpl#testMultiThreadedPublish failed with Metrics not
collected
{code}
Running org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl
Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.688 sec
FAILURE! - in org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl
testMultiThreadedPublish(org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl)
Time elapsed: 0.056 sec FAILURE!
java.lang.AssertionError: Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Passed
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at
org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl.testMultiThreadedPublish(TestMetricsSystemImpl.java:232)
Results :
Failed tests:
TestMetricsSystemImpl.testMultiThreadedPublish:232 Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Passed
Tests run: 6, Failures: 1, Errors: 0, Skipped: 0
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-10976) moving the source code of hadoop-tools docs to the directory under hadoop-tools
[ https://issues.apache.org/jira/browse/HADOOP-10976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307393#comment-14307393 ] Hudson commented on HADOOP-10976: - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2046 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2046/]) HADOOP-10976. moving the source code of hadoop-tools docs to the directory under hadoop-tools (Masatake Iwasaki via aw) (aw: rev 9112f093cde0b8242324aa5267c9beb21c38bf6b) * hadoop-tools/hadoop-archives/src/site/resources/css/site.css * hadoop-tools/hadoop-distcp/src/site/resources/css/site.css * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/apt/HadoopStreaming.apt.vm * hadoop-tools/hadoop-distcp/src/site/markdown/DistCp.md.vm * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/DistCp.md.vm * hadoop-tools/hadoop-archives/src/site/markdown/HadoopArchives.md.vm * hadoop-tools/hadoop-streaming/src/site/apt/HadoopStreaming.apt.vm * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-tools/hadoop-streaming/src/site/resources/css/site.css * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/HadoopArchives.md.vm * hadoop-project/src/site/site.xml moving the source code of hadoop-tools docs to the directory under hadoop-tools --- Key: HADOOP-10976 URL: https://issues.apache.org/jira/browse/HADOOP-10976 Project: Hadoop Common Issue Type: Improvement Components: documentation Reporter: Masatake Iwasaki Assignee: Masatake Iwasaki Priority: Minor Fix For: 3.0.0 Attachments: HADOOP-10976-0.patch, HADOOP-10976-1.patch, HADOOP-10976-3.patch, HADOOP-10976-4.patch Some of the doc files of hadoop-tools are placed in the mapreduce project. It should be moved for the ease of maintenance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11546) Checkstyle failing: Unable to instantiate DoubleCheckedLockingCheck
[
https://issues.apache.org/jira/browse/HADOOP-11546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307386#comment-14307386
]
Hudson commented on HADOOP-11546:
-
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2046 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2046/])
HADOOP-11546. Checkstyle failing: Unable to instantiate
DoubleCheckedLockingCheck. (ozawa) (ozawa: rev
ba58da29f49da15f7eb6300f996e0ae7ba9d0d1e)
* hadoop-common-project/hadoop-common/CHANGES.txt
* hadoop-tools/hadoop-azure/src/config/checkstyle.xml
* hadoop-hdfs-project/hadoop-hdfs/dev-support/checkstyle.xml
* hadoop-common-project/hadoop-common/dev-support/checkstyle.xml
Checkstyle failing: Unable to instantiate DoubleCheckedLockingCheck
---
Key: HADOOP-11546
URL: https://issues.apache.org/jira/browse/HADOOP-11546
Project: Hadoop Common
Issue Type: Bug
Components: build
Environment: Jenkins HDFS builds; java 8
Reporter: Steve Loughran
Assignee: Tsuyoshi OZAWA
Fix For: 2.7.0
Attachments: HADOOP-11546.1.patch
HDFS builds are failing in jenkins
{code}
Failed to execute goal
org.apache.maven.plugins:maven-checkstyle-plugin:2.12.1:checkstyle
(default-cli) on project hadoop-hdfs: An error has occurred in Checkstyle
report generation. Failed during checkstyle configuration: cannot initialize
module TreeWalker - Unable to instantiate DoubleCheckedLocking: Unable to
instantiate DoubleCheckedLockingCheck -
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11543) Improve help message for hadoop/yarn command
[ https://issues.apache.org/jira/browse/HADOOP-11543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307384#comment-14307384 ] Hudson commented on HADOOP-11543: - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2046 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2046/]) HADOOP-11543. Improve help message for hadoop/yarn command. Contributed by Brahma Reddy Battula. (ozawa: rev ec950be1735a877675c1c9493846e5b960f373c4) * hadoop-yarn-project/hadoop-yarn/bin/yarn * hadoop-common-project/hadoop-common/src/main/bin/hadoop * hadoop-common-project/hadoop-common/CHANGES.txt Improve help message for hadoop/yarn command Key: HADOOP-11543 URL: https://issues.apache.org/jira/browse/HADOOP-11543 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 2.6.0 Reporter: Jagadesh Kiran N Assignee: Brahma Reddy Battula Priority: Trivial Fix For: 2.7.0 Attachments: HADOOP-11543-001.patch, HADOOP-11543-002.patch, HADOOP-11543-002.patch, HADOOP-11543-branch-2.001.patch, HADOOP-11543-branch-2.002, HADOOP-11543.patch, OrinYarncommnad.png, YARN-3128-01.patch, YARN-3128.patch Pls check the snapshot attached -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11548) checknative should display a nicer error message when openssl support is not compiled in
[
https://issues.apache.org/jira/browse/HADOOP-11548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307387#comment-14307387
]
Hudson commented on HADOOP-11548:
-
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2046 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2046/])
HADOOP-11548. checknative should display a nicer error message when openssl
support is not compiled in. Contributed by Anu Engineer. (cnauroth: rev
064e0770a0f32e7f32a7ba16c71394d7e75a3eb3)
* hadoop-common-project/hadoop-common/CHANGES.txt
* hadoop-common-project/hadoop-common/src/CMakeLists.txt
checknative should display a nicer error message when openssl support is not
compiled in
Key: HADOOP-11548
URL: https://issues.apache.org/jira/browse/HADOOP-11548
Project: Hadoop Common
Issue Type: Bug
Components: build, native
Affects Versions: 2.6.0
Reporter: Colin Patrick McCabe
Assignee: Anu Engineer
Fix For: 2.7.0
Attachments: HDFS-7520.001.patch
checknative should display a nicer error message when openssl support is not
compiled in. Currently, it displays this:
{code}
[cmccabe@keter hadoop]$ hadoop checknative
14/12/12 14:08:43 INFO bzip2.Bzip2Factory: Successfully loaded initialized
native-bzip2 library system-native
14/12/12 14:08:43 INFO zlib.ZlibFactory: Successfully loaded initialized
native-zlib library
Native library checking:
hadoop: true /usr/lib/hadoop/lib/native/libhadoop.so.1.0.0
zlib:true /lib64/libz.so.1
snappy: true /usr/lib64/libsnappy.so.1
lz4: true revision:99
bzip2: true /lib64/libbz2.so.1
openssl: false org.apache.hadoop.crypto.OpensslCipher.initIDs()V
{code}
Instead, we should display something like this, if openssl is not supported
by the current build:
{code}
openssl: false Hadoop was built without openssl support.
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11463) Replace method-local TransferManager object with S3AFileSystem#transfers
[ https://issues.apache.org/jira/browse/HADOOP-11463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307395#comment-14307395 ] Hudson commented on HADOOP-11463: - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2046 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2046/]) HADOOP-11463 Replace method-local TransferManager object with S3AFileSystem#transfers. (Ted Yu via stevel) (stevel: rev 4e7ad4f0a88bd36bc91db6b1bd311d7f5c6bebee) * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java Replace method-local TransferManager object with S3AFileSystem#transfers Key: HADOOP-11463 URL: https://issues.apache.org/jira/browse/HADOOP-11463 Project: Hadoop Common Issue Type: Task Components: fs/s3 Affects Versions: 2.7.0 Reporter: Ted Yu Assignee: Ted Yu Fix For: 2.7.0 Attachments: hadoop-11463-001.patch, hadoop-11463-002.patch, hadoop-11463-003.patch This is continuation of HADOOP-11446. The following changes are made according to Thomas Demoor's comments: 1. Replace method-local TransferManager object with S3AFileSystem#transfers 2. Do not shutdown TransferManager after purging existing multipart file - otherwise the current transfer is unable to proceed 3. Shutdown TransferManager instance in the close method of S3AFileSystem -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-9087) Queue size metric for metric sinks isn't actually maintained
[
https://issues.apache.org/jira/browse/HADOOP-9087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308174#comment-14308174
]
Hadoop QA commented on HADOOP-9087:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12554769/HADOOP-9087.patch
against trunk revision e2ee2ff.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/5608//console
This message is automatically generated.
Queue size metric for metric sinks isn't actually maintained
Key: HADOOP-9087
URL: https://issues.apache.org/jira/browse/HADOOP-9087
Project: Hadoop Common
Issue Type: Bug
Reporter: Mostafa Elhemali
Priority: Minor
Attachments: HADOOP-9087.patch
Came across this while looking at the code for the metrics system: the qsize
gauge in the MetricsSinkAdapter is not updated/maintained so it will always
be zero (there's a compiler warning about it actually).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-11551) Let nightly jenkins jobs run the tool of HADOOP-11045 and include the result in the job report
Yongjun Zhang created HADOOP-11551: -- Summary: Let nightly jenkins jobs run the tool of HADOOP-11045 and include the result in the job report Key: HADOOP-11551 URL: https://issues.apache.org/jira/browse/HADOOP-11551 Project: Hadoop Common Issue Type: Bug Components: build, tools Reporter: Yongjun Zhang This jira is to propose running the tool created with HADOOP-11045 at the end of jenkins test job - I am thinking about trunk jobs currently - and report the results at the job report. This way when we look at test failure, we can tell the failure pattern, and whether failed test is likely a flaky test or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-10392) Use FileSystem#makeQualified(Path) instead of Path#makeQualified(FileSystem)
[ https://issues.apache.org/jira/browse/HADOOP-10392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308153#comment-14308153 ] Akira AJISAKA commented on HADOOP-10392: These failures are not related to the patch. MAPREDUCE-6223, MAPREDUCE-6234, and MAPREDUCE-6233 will fix them. Use FileSystem#makeQualified(Path) instead of Path#makeQualified(FileSystem) Key: HADOOP-10392 URL: https://issues.apache.org/jira/browse/HADOOP-10392 Project: Hadoop Common Issue Type: Sub-task Components: fs Affects Versions: 2.3.0 Reporter: Akira AJISAKA Assignee: Akira AJISAKA Priority: Minor Labels: newbie Attachments: HADOOP-10392.2.patch, HADOOP-10392.3.patch, HADOOP-10392.4.patch, HADOOP-10392.4.patch, HADOOP-10392.5.patch, HADOOP-10392.6.patch, HADOOP-10392.7.patch, HADOOP-10392.7.patch, HADOOP-10392.patch There're some methods calling Path.makeQualified(FileSystem), which causes javac warning. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11485) Pluggable shell integration
[
https://issues.apache.org/jira/browse/HADOOP-11485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308135#comment-14308135
]
Chris Nauroth commented on HADOOP-11485:
Hi, [~aw]. Thank you for doing this. This looks great.
I took it for a test run with a concrete use case that I mentioned on another
jira: adding hadoop-azure to the classpath at distro time. Everything worked
as advertised in the example. I read through that and then dropped the
following file at shellprofile.d/hadoop_azure:
{code}
hadoop_add_profile hadoop_azure
function _hadoop_azure_hadoop_classpath
{
hadoop_add_classpath
${HADOOP_PREFIX}/share/hadoop/tools/lib/hadoop-azure-3.0.0-SNAPSHOT.jar after
hadoop_add_classpath
${HADOOP_PREFIX}/share/hadoop/tools/lib/microsoft-windowsazure-storage-sdk-0.6.0.jar
after
}
{code}
With that in place, I can run fs shell commands that access paths in Azure
Storage. This is easier than the existing options for trying to achieve this
kind of customization.
I have only 2 meta-comments, which don't necessarily need to be addressed in
the scope of this jira:
# With the introduction of shell profiles, we essentially have a new public API
contract. We'll need to maintain compatibility so that we don't break existing
shell profiles during routine code maintenance. (i.e. {{hadoop_add_classpath}}
can't be renamed within a major version). Maybe we need a commenting
convention, similar to the interface visibility and stability annotations that
we use in Java, to help guide developers.
# I think we'll also eventually need user-facing documentation on
hadoop.apache.org to describe the new functionality in the shell rewrite. For
shell profile implementors, something like API docs would be helpful to let
people know about useful things like {{hadoop_debug}}.
I think the current patch needs a small rebase. After that, +1.
Pluggable shell integration
---
Key: HADOOP-11485
URL: https://issues.apache.org/jira/browse/HADOOP-11485
Project: Hadoop Common
Issue Type: New Feature
Components: scripts
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Allen Wittenauer
Labels: scripts, shell
Attachments: HADOOP-11485-00.patch, HADOOP-11485-01.patch,
HADOOP-11485-02.patch
It would be useful to provide a way for core and non-core Hadoop components
to plug into the shell infrastructure. This would allow us to pull the HDFS,
MapReduce, and YARN shell functions out of hadoop-functions.sh.
Additionally, it should let 3rd parties such as HBase influence things like
classpaths at runtime.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-6964) Allow compact property description in xml
[
https://issues.apache.org/jira/browse/HADOOP-6964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308102#comment-14308102
]
Akira AJISAKA commented on HADOOP-6964:
---
bq. validate multiple definition of attributes and elements and abort with
RuntimeException
I'm thinking it would be sufficient to output warning message as follows:
{code:title=Configuration.java}
if (name.equals(field.getTagName()) field.hasChildNodes())
if (attr != null) {
LOG.warn(Detected multiple definitions. Property name: + attr
+
will be overwritten.);
}
attr = StringInterner.weakIntern(
((Text)field.getFirstChild()).getData().trim());
{code}
However, I'm okay with doing this by a separate jira. +1 for the patch, thanks
Kengo and Allen.
Allow compact property description in xml
-
Key: HADOOP-6964
URL: https://issues.apache.org/jira/browse/HADOOP-6964
Project: Hadoop Common
Issue Type: Improvement
Components: conf
Reporter: Owen O'Malley
Assignee: Kengo Seki
Labels: newbie
Attachments: HADOOP-6964.patch
We should allow users to use the more compact form of xml elements. For
example, we could allow:
{noformat}
property name=mapred.local.dir value=/disk/0/mapred,/disk/1/mapred/
{noformat}
The old format would also be supported.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
