[jira] [Commented] (HADOOP-11813) releasedocmaker.py should use today's date instead of unreleased
[
https://issues.apache.org/jira/browse/HADOOP-11813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504483#comment-14504483
]
Darrell Taylor commented on HADOOP-11813:
-
There were no tests around this originally, do some need to be written?
Manual steps taken to test this were :
{code}
cd hadoop-common-project/hadoop-common/src/site/markdown/release/
python ../../../../../../dev-support/releasedocmaker.py --version 3.0.0 --index
{code}
Then confirmed that files were produced in
hadoop-common-project/hadoop-common/src/site/markdown/release/3.0.0-SNAPSHOT/.
The CHANGES.3.0.0-SNAPSHOT.md file contained the work 'Unreleased' in this
case. The same command was run again with the --usetoday switch and the
CHANGES.3.0.0-SNAPSHOT.md included today's date instead of 'Unreleased'.
This was also checked through the maven build by running
{code}
mvn pre-site -DskipTests -Preleasedocs
{code}
The CHANGES.3.0.0-SNAPSHOT.md file included today's date instead of
'Unlreleased'
releasedocmaker.py should use today's date instead of unreleased
Key: HADOOP-11813
URL: https://issues.apache.org/jira/browse/HADOOP-11813
Project: Hadoop Common
Issue Type: Task
Components: build
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Darrell Taylor
Priority: Minor
Labels: newbie
Attachments: HADOOP-11813.patch
After discussing with a few folks, it'd be more convenient if releasedocmaker
used the current date rather than unreleased when processing a version that
JIRA hasn't declared released.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-11853) fix test case TestCoordRerunXCommand.testCoordRerunDateNeg
Ryota Egashira created HADOOP-11853: --- Summary: fix test case TestCoordRerunXCommand.testCoordRerunDateNeg Key: HADOOP-11853 URL: https://issues.apache.org/jira/browse/HADOOP-11853 Project: Hadoop Common Issue Type: Bug Reporter: Ryota Egashira -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11704) DelegationTokenAuthenticationFilter must pass ipaddress instead of hostname to ProxyUsers#authorize()
[ https://issues.apache.org/jira/browse/HADOOP-11704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-11704: - Resolution: Fixed Status: Resolved (was: Patch Available) DelegationTokenAuthenticationFilter must pass ipaddress instead of hostname to ProxyUsers#authorize() - Key: HADOOP-11704 URL: https://issues.apache.org/jira/browse/HADOOP-11704 Project: Hadoop Common Issue Type: Bug Reporter: Anubhav Dhoot Assignee: Anubhav Dhoot Fix For: 2.8.0 Attachments: HADOOP-11704.001.patch, HADOOP-11704.002.patch DelegationTokenAuthenticationHandler and DelegationTokenAuthenticationFilter are using ServletRequest#getRemoteHost which can send an address if possible. It should use getRemoteAddr instead -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11704) DelegationTokenAuthenticationFilter must pass ipaddress instead of hostname to ProxyUsers#authorize()
[ https://issues.apache.org/jira/browse/HADOOP-11704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505458#comment-14505458 ] Arun Suresh commented on HADOOP-11704: -- Committed to trunk and branch-2 DelegationTokenAuthenticationFilter must pass ipaddress instead of hostname to ProxyUsers#authorize() - Key: HADOOP-11704 URL: https://issues.apache.org/jira/browse/HADOOP-11704 Project: Hadoop Common Issue Type: Bug Reporter: Anubhav Dhoot Assignee: Anubhav Dhoot Fix For: 2.8.0 Attachments: HADOOP-11704.001.patch, HADOOP-11704.002.patch DelegationTokenAuthenticationHandler and DelegationTokenAuthenticationFilter are using ServletRequest#getRemoteHost which can send an address if possible. It should use getRemoteAddr instead -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11590) Update sbin commands and documentation to use new --slaves option
[
https://issues.apache.org/jira/browse/HADOOP-11590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505548#comment-14505548
]
Hadoop QA commented on HADOOP-11590:
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 00s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12726963/HADOOP-11590-02.patch |
| Optional Tests | site shellcheck |
| git revision | HADOOP-11590 / a8c2df7 |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/fake//console |
This message was automatically generated.
Update sbin commands and documentation to use new --slaves option
-
Key: HADOOP-11590
URL: https://issues.apache.org/jira/browse/HADOOP-11590
Project: Hadoop Common
Issue Type: Bug
Components: documentation, scripts
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Allen Wittenauer
Priority: Blocker
Attachments: HADOOP-11590-00.patch, HADOOP-11590-01.patch,
HADOOP-11590-02.patch
With HADOOP-11565 now committed, we need to remove usages of yarn-daemons.sh
and hadoop-daemons.sh from the start and stop scripts, converting them to use
the new --slaves option. Additionally, the documentation should be updated
to reflect these new command options.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-10290) Surefire steals focus on MacOS
[
https://issues.apache.org/jira/browse/HADOOP-10290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505594#comment-14505594
]
André Kelpe commented on HADOOP-10290:
--
I am using JAVA_TOOL_OPTIONS on my systems since that works across all the
different build tools (maven, sbt, gradle) and gets picked up by the JVM
directly:
https://docs.oracle.com/javase/7/docs/webnotes/tsg/TSG-VM/html/envvars.html
Surefire steals focus on MacOS
--
Key: HADOOP-10290
URL: https://issues.apache.org/jira/browse/HADOOP-10290
Project: Hadoop Common
Issue Type: Bug
Components: build
Reporter: Laurent Goujon
Attachments: hadoop-10290.patch, hadoop-10290.patch
When running tests on MacOS X, surefire plugin keeps stealing focus from
current application.
This can be avoided by adding {noformat}-Djava.awt.headless=true{noformat} to
the surefire commandline
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11820) aw jira testing, ignore
[ https://issues.apache.org/jira/browse/HADOOP-11820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505513#comment-14505513 ] Hadoop QA commented on HADOOP-11820: (!) A patch to test-patch or smart-apply-patch has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/fake//console in case of problems. aw jira testing, ignore --- Key: HADOOP-11820 URL: https://issues.apache.org/jira/browse/HADOOP-11820 Project: Hadoop Common Issue Type: Task Reporter: Allen Wittenauer Attachments: HADOOP-11746-14.patch, HADOOP-11746-16.patch, HADOOP-11746-17.patch, HADOOP-11746-20.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11820) aw jira testing, ignore
[ https://issues.apache.org/jira/browse/HADOOP-11820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11820: -- Attachment: HADOOP-11746-20.patch aw jira testing, ignore --- Key: HADOOP-11820 URL: https://issues.apache.org/jira/browse/HADOOP-11820 Project: Hadoop Common Issue Type: Task Reporter: Allen Wittenauer Attachments: HADOOP-11746-14.patch, HADOOP-11746-16.patch, HADOOP-11746-17.patch, HADOOP-11746-20.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11855) Add -Djava.awt.headless=true to pom.xml entries
[ https://issues.apache.org/jira/browse/HADOOP-11855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505551#comment-14505551 ] Steve Loughran commented on HADOOP-11855: - Isn't this the same as HADOOP-10290 ... the one you closed as WONTFIX? Add -Djava.awt.headless=true to pom.xml entries --- Key: HADOOP-11855 URL: https://issues.apache.org/jira/browse/HADOOP-11855 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 3.0.0 Environment: Mac OS X Reporter: Allen Wittenauer Priority: Minor Labels: newbie On OS X, certain profiles cause stealing of the focus, including tests, site, and javadoc creation. In order to prevent this, -Djava.awt.headless=true should be added to the argument lines for those subsystems. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11843) Make setting up the build environment easier
[ https://issues.apache.org/jira/browse/HADOOP-11843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505445#comment-14505445 ] Allen Wittenauer commented on HADOOP-11843: --- bq. I could copy the file into the ~/hadoop directory I guess but that feels a little bit dirty. Any recommendations on what to do here? That's exactly what I've been doing + using the newer test-patch.sh (HADOOP-11746) which supports HTTP URLs to test currently uploaded patches. Make setting up the build environment easier Key: HADOOP-11843 URL: https://issues.apache.org/jira/browse/HADOOP-11843 Project: Hadoop Common Issue Type: New Feature Reporter: Niels Basjes Assignee: Niels Basjes Attachments: HADOOP-11843-2015-04-17-1612.patch, HADOOP-11843-2015-04-17-2226.patch, HADOOP-11843-2015-04-17-2308.patch, HADOOP-11843-2015-04-19-2206.patch, HADOOP-11843-2015-04-19-2232.patch ( As discussed with [~aw] ) In AVRO-1537 a docker based solution was created to setup all the tools for doing a full build. This enables much easier reproduction of any issues and getting up and running for new developers. This issue is to 'copy/port' that setup into the hadoop project in preparation for the bug squash. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-10290) Surefire steals focus on MacOS
[
https://issues.apache.org/jira/browse/HADOOP-10290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505575#comment-14505575
]
Sean Busbey commented on HADOOP-10290:
--
FWIW, I fix this across projects by setting a {{~/.mavenrc}} with MAVEN_OPTS
that includes {{-Dapple.awt.UIElement=true}} ([ref
pdf|https://developer.apple.com/library/mac/documentation/Java/Reference/Java_PropertiesRef/Java_PropertiesRef.pdf]).
Since this is a build-box issue, why not just document it in our dev
guidelines?
Surefire steals focus on MacOS
--
Key: HADOOP-10290
URL: https://issues.apache.org/jira/browse/HADOOP-10290
Project: Hadoop Common
Issue Type: Bug
Components: build
Reporter: Laurent Goujon
Attachments: hadoop-10290.patch, hadoop-10290.patch
When running tests on MacOS X, surefire plugin keeps stealing focus from
current application.
This can be avoided by adding {noformat}-Djava.awt.headless=true{noformat} to
the surefire commandline
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11827) Speed-up distcp buildListing() using threadpool
[
https://issues.apache.org/jira/browse/HADOOP-11827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505612#comment-14505612
]
Hadoop QA commented on HADOOP-11827:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12726960/HADOOP-11827-04.patch
against trunk revision 997408e.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-tools/hadoop-distcp.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/6140//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/6140//console
This message is automatically generated.
Speed-up distcp buildListing() using threadpool
---
Key: HADOOP-11827
URL: https://issues.apache.org/jira/browse/HADOOP-11827
Project: Hadoop Common
Issue Type: Improvement
Components: tools/distcp
Affects Versions: 2.7.0, 2.7.1
Reporter: Zoran Dimitrijevic
Assignee: Zoran Dimitrijevic
Attachments: HADOOP-11827-02.patch, HADOOP-11827-03.patch,
HADOOP-11827-04.patch, HADOOP-11827.patch
Original Estimate: 24h
Remaining Estimate: 24h
For very large source trees on s3 distcp is taking long time to build file
listing (client code, before starting mappers). For a dataset I used (1.5M
files, 50K dirs) it was taking 65 minutes before my fix in HADOOP-11785 and
36 minutes after the fix).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-11857) Fix CommandFormat#commandFormat java doc annotation
Xiaoyu Yao created HADOOP-11857:
---
Summary: Fix CommandFormat#commandFormat java doc annotation
Key: HADOOP-11857
URL: https://issues.apache.org/jira/browse/HADOOP-11857
Project: Hadoop Common
Issue Type: Improvement
Reporter: Xiaoyu Yao
Priority: Trivial
The first parameter name does not match with the javadoc.
{code}
/**
* @deprecated use replacement since name is an unused parameter
* @param name of command, but never used
* @param min see replacement
* @param max see replacement
* @param possibleOpt see replacement
* @see #CommandFormat(int, int, String...)
*/
@Deprecated
public CommandFormat(String n, int min, int max, String ... possibleOpt) {
this(min, max, possibleOpt);
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Reopened] (HADOOP-10290) Surefire steals focus on MacOS
[
https://issues.apache.org/jira/browse/HADOOP-10290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer reopened HADOOP-10290:
---
JDK7 is not fixing this for me. Re-opening.
Surefire steals focus on MacOS
--
Key: HADOOP-10290
URL: https://issues.apache.org/jira/browse/HADOOP-10290
Project: Hadoop Common
Issue Type: Bug
Components: build
Reporter: Laurent Goujon
Attachments: hadoop-10290.patch, hadoop-10290.patch
When running tests on MacOS X, surefire plugin keeps stealing focus from
current application.
This can be avoided by adding {noformat}-Djava.awt.headless=true{noformat} to
the surefire commandline
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11855) Add -Djava.awt.headless=true to pom.xml entries
[ https://issues.apache.org/jira/browse/HADOOP-11855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505557#comment-14505557 ] Allen Wittenauer commented on HADOOP-11855: --- Yes, and clearly JDK7 doesn't fix this. :( I'll re-open that one I guess. Add -Djava.awt.headless=true to pom.xml entries --- Key: HADOOP-11855 URL: https://issues.apache.org/jira/browse/HADOOP-11855 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 3.0.0 Environment: Mac OS X Reporter: Allen Wittenauer Priority: Minor Labels: newbie On OS X, certain profiles cause stealing of the focus, including tests, site, and javadoc creation. In order to prevent this, -Djava.awt.headless=true should be added to the argument lines for those subsystems. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh
[
https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HADOOP-11746:
--
Attachment: HADOOP-11746-21.patch
-21:
A one line change in the site test:
{code}
+ add_jira_table +1 site There were no new javadoc warning messages.
---
+ add_jira_table +1 site Site still builds.
{code}
...
[~cnauroth], does your +1 still stand? I'm pretty much ready to commit this.
Promise. lol
rewrite test-patch.sh
-
Key: HADOOP-11746
URL: https://issues.apache.org/jira/browse/HADOOP-11746
Project: Hadoop Common
Issue Type: Test
Components: build, test
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Allen Wittenauer
Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch,
HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch,
HADOOP-11746-05.patch, HADOOP-11746-06.patch, HADOOP-11746-07.patch,
HADOOP-11746-09.patch, HADOOP-11746-10.patch, HADOOP-11746-11.patch,
HADOOP-11746-12.patch, HADOOP-11746-13.patch, HADOOP-11746-14.patch,
HADOOP-11746-15.patch, HADOOP-11746-16.patch, HADOOP-11746-17.patch,
HADOOP-11746-18.patch, HADOOP-11746-19.patch, HADOOP-11746-20.patch,
HADOOP-11746-21.patch
This code is bad and you should feel bad.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11854) Fix Typos in all the projects
[ https://issues.apache.org/jira/browse/HADOOP-11854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang updated HADOOP-11854: Attachment: HADOOP-11854.suggestions.001.patch Here's a list of the spelling errors I've run across. It's very likely that someone will have to merge several batches of such errors found. Fix Typos in all the projects - Key: HADOOP-11854 URL: https://issues.apache.org/jira/browse/HADOOP-11854 Project: Hadoop Common Issue Type: Bug Reporter: Brahma Reddy Battula Assignee: Brahma Reddy Battula Priority: Minor Attachments: HADOOP-11854.suggestions.001.patch Recently I had seen, there are so many jira's for fixing the typo's ( Keep on accumulating more ). Hence I want to plan in proper manner such that everything will be addressed.. I am thinking, we can fix project level ( at most package level)... My intention to avoid the number of jira's on typo's...One more suggestion to reviewer's is please dn't commit for class level try to check project level ( atmost package level) if any such typo's present... Please correct me If I am wrong.. I will close this jira.. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-11856) ArrayIndexOutOfBoundsException in MiniDFSCluster.
surendra singh lilhore created HADOOP-11856:
---
Summary: ArrayIndexOutOfBoundsException in MiniDFSCluster.
Key: HADOOP-11856
URL: https://issues.apache.org/jira/browse/HADOOP-11856
Project: Hadoop Common
Issue Type: Bug
Components: test
Affects Versions: 2.6.0
Reporter: surendra singh lilhore
Assignee: surendra singh lilhore
I want to create MiniDFSCluster with 2 datanode and for each datanode I want to
set different number of StorageTypes, but in this case I am getting
ArrayIndexOutOfBoundsException.
My cluster schema is like this.
{code}
final MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf)
.numDataNodes(2)
.storageTypes(new StorageType[][] {{
StorageType.DISK, StorageType.ARCHIVE },{ StorageType.DISK } })
.build();
{code}
*Exception* :
{code}
java.lang.ArrayIndexOutOfBoundsException: 1
at
org.apache.hadoop.hdfs.MiniDFSCluster.makeDataNodeDirs(MiniDFSCluster.java:1218)
at
org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1402)
at
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:832)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11704) ProxyUsers expects ipaddress due to MachineList but callers send in remote host
[ https://issues.apache.org/jira/browse/HADOOP-11704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505413#comment-14505413 ] Arun Suresh commented on HADOOP-11704: -- +1 Thanks for the fix [~adhoot] Will be committing this shortly.. ProxyUsers expects ipaddress due to MachineList but callers send in remote host Key: HADOOP-11704 URL: https://issues.apache.org/jira/browse/HADOOP-11704 Project: Hadoop Common Issue Type: Bug Reporter: Anubhav Dhoot Assignee: Anubhav Dhoot Attachments: HADOOP-11704.001.patch, HADOOP-11704.002.patch DelegationTokenAuthenticationHandler and DelegationTokenAuthenticationFilter are using ServletRequest#getRemoteHost which can send an address if possible. It should use getRemoteAddr instead -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11704) ProxyUsers expects ipaddress due to MachineList but callers send in remote host
[ https://issues.apache.org/jira/browse/HADOOP-11704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-11704: - Fix Version/s: 2.8.0 ProxyUsers expects ipaddress due to MachineList but callers send in remote host Key: HADOOP-11704 URL: https://issues.apache.org/jira/browse/HADOOP-11704 Project: Hadoop Common Issue Type: Bug Reporter: Anubhav Dhoot Assignee: Anubhav Dhoot Fix For: 2.8.0 Attachments: HADOOP-11704.001.patch, HADOOP-11704.002.patch DelegationTokenAuthenticationHandler and DelegationTokenAuthenticationFilter are using ServletRequest#getRemoteHost which can send an address if possible. It should use getRemoteAddr instead -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11704) DelegationTokenAuthenticationFilter must pass ipaddress instead of hostname to ProxyUsers#authorize()
[ https://issues.apache.org/jira/browse/HADOOP-11704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-11704: - Summary: DelegationTokenAuthenticationFilter must pass ipaddress instead of hostname to ProxyUsers#authorize() (was: ProxyUsers expects ipaddress due to MachineList but callers send in remote host ) DelegationTokenAuthenticationFilter must pass ipaddress instead of hostname to ProxyUsers#authorize() - Key: HADOOP-11704 URL: https://issues.apache.org/jira/browse/HADOOP-11704 Project: Hadoop Common Issue Type: Bug Reporter: Anubhav Dhoot Assignee: Anubhav Dhoot Fix For: 2.8.0 Attachments: HADOOP-11704.001.patch, HADOOP-11704.002.patch DelegationTokenAuthenticationHandler and DelegationTokenAuthenticationFilter are using ServletRequest#getRemoteHost which can send an address if possible. It should use getRemoteAddr instead -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11704) DelegationTokenAuthenticationFilter must pass ipaddress instead of hostname to ProxyUsers#authorize()
[ https://issues.apache.org/jira/browse/HADOOP-11704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505489#comment-14505489 ] Hudson commented on HADOOP-11704: - FAILURE: Integrated in Hadoop-trunk-Commit #7625 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7625/]) HADOOP-11704. DelegationTokenAuthenticationFilter must pass ipaddress instead of hostname to ProxyUsers#authorize (Anubhav Dhoot via asuresh) (Arun Suresh: rev 424a00daa069bf2049014fd46ad152ec5fc77ac8) * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticationFilter.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/token/delegation/web/TestWebDelegationToken.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticationHandler.java DelegationTokenAuthenticationFilter must pass ipaddress instead of hostname to ProxyUsers#authorize() - Key: HADOOP-11704 URL: https://issues.apache.org/jira/browse/HADOOP-11704 Project: Hadoop Common Issue Type: Bug Reporter: Anubhav Dhoot Assignee: Anubhav Dhoot Fix For: 2.8.0 Attachments: HADOOP-11704.001.patch, HADOOP-11704.002.patch DelegationTokenAuthenticationHandler and DelegationTokenAuthenticationFilter are using ServletRequest#getRemoteHost which can send an address if possible. It should use getRemoteAddr instead -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11820) aw jira testing, ignore
[
https://issues.apache.org/jira/browse/HADOOP-11820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505514#comment-14505514
]
Hadoop QA commented on HADOOP-11820:
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | reexec | 0m 00s | dev-support patch detected. |
| {color:blue}0{color} | pre-patch | 0m 00s | Pre-patch trunk compilation is
healthy. |
| {color:red}-1{color} | @author | 0m 00s | The patch appears to contain 13
@author tags which the Hadoop community has agreed to not allow in code
contributions. |
| {color:green}+1{color} | whitespace | 0m 00s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | release audit | 0m 12s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | shellcheck | 0m 03s | The applied patch generated 2
new shellcheck (v0.3.6) issues (total was 189, now 47). |
| | | 0m 27s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12726955/HADOOP-11746-20.patch |
| Optional Tests | shellcheck |
| git revision | trunk / f47a576 |
| shellcheck |
https://builds.apache.org/job/PreCommit-HADOOP-Build/fake/artifact/patchprocess/diffpatchshellcheck.txt
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/fake//console |
This message was automatically generated.
aw jira testing, ignore
---
Key: HADOOP-11820
URL: https://issues.apache.org/jira/browse/HADOOP-11820
Project: Hadoop Common
Issue Type: Task
Reporter: Allen Wittenauer
Attachments: HADOOP-11746-14.patch, HADOOP-11746-16.patch,
HADOOP-11746-17.patch, HADOOP-11746-20.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11590) Update sbin commands and documentation to use new --slaves option
[
https://issues.apache.org/jira/browse/HADOOP-11590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505515#comment-14505515
]
Hadoop QA commented on HADOOP-11590:
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 00s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12701772/HADOOP-11590-01.patch |
| Optional Tests | site shellcheck |
| git revision | HADOOP-11590 / a8c2df7 |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/fake//console |
This message was automatically generated.
Update sbin commands and documentation to use new --slaves option
-
Key: HADOOP-11590
URL: https://issues.apache.org/jira/browse/HADOOP-11590
Project: Hadoop Common
Issue Type: Bug
Components: documentation, scripts
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Allen Wittenauer
Priority: Blocker
Attachments: HADOOP-11590-00.patch, HADOOP-11590-01.patch
With HADOOP-11565 now committed, we need to remove usages of yarn-daemons.sh
and hadoop-daemons.sh from the start and stop scripts, converting them to use
the new --slaves option. Additionally, the documentation should be updated
to reflect these new command options.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11820) aw jira testing, ignore
[ https://issues.apache.org/jira/browse/HADOOP-11820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11820: -- Attachment: HADOOP-11590-02.patch aw jira testing, ignore --- Key: HADOOP-11820 URL: https://issues.apache.org/jira/browse/HADOOP-11820 Project: Hadoop Common Issue Type: Task Reporter: Allen Wittenauer Attachments: HADOOP-11590-02.patch, HADOOP-11746-14.patch, HADOOP-11746-16.patch, HADOOP-11746-17.patch, HADOOP-11746-20.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11827) Speed-up distcp buildListing() using threadpool
[ https://issues.apache.org/jira/browse/HADOOP-11827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash updated HADOOP-11827: -- Attachment: HADOOP-11827-04.patch Thanks for your contribution Zoran. I agree with all your points. I've made some small changes to the test code (to get rid of silly warnings) I'll commit this end-of-day if there are no objections. Speed-up distcp buildListing() using threadpool --- Key: HADOOP-11827 URL: https://issues.apache.org/jira/browse/HADOOP-11827 Project: Hadoop Common Issue Type: Improvement Components: tools/distcp Affects Versions: 2.7.0, 2.7.1 Reporter: Zoran Dimitrijevic Assignee: Zoran Dimitrijevic Attachments: HADOOP-11827-02.patch, HADOOP-11827-03.patch, HADOOP-11827-04.patch, HADOOP-11827.patch Original Estimate: 24h Remaining Estimate: 24h For very large source trees on s3 distcp is taking long time to build file listing (client code, before starting mappers). For a dataset I used (1.5M files, 50K dirs) it was taking 65 minutes before my fix in HADOOP-11785 and 36 minutes after the fix). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11820) aw jira testing, ignore
[
https://issues.apache.org/jira/browse/HADOOP-11820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505585#comment-14505585
]
Hadoop QA commented on HADOOP-11820:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12726955/HADOOP-11746-20.patch
against trunk revision 997408e.
{color:red}-1 @author{color}. The patch appears to contain 13 @author tags
which the Hadoop community has agreed to not allow in code contributions.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/6138//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/6138//console
This message is automatically generated.
aw jira testing, ignore
---
Key: HADOOP-11820
URL: https://issues.apache.org/jira/browse/HADOOP-11820
Project: Hadoop Common
Issue Type: Task
Reporter: Allen Wittenauer
Attachments: HADOOP-11590-02.patch, HADOOP-11746-14.patch,
HADOOP-11746-16.patch, HADOOP-11746-17.patch, HADOOP-11746-20.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11746) rewrite test-patch.sh
[ https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505598#comment-14505598 ] Chris Nauroth commented on HADOOP-11746: I checked the diff since my last review (-17), and everything looks good to me. I agree with the decision to maintain the existing behavior of -1 when the existing trunk build is already broken, and I agree with changing those failure code paths to exit with a non-zero code. +1 for patch -21. Thanks for your work on this, Allen. The new functionality is great. Now hurry up and commit before you get more ideas! :-) Sean, thank you for helping with the code review. rewrite test-patch.sh - Key: HADOOP-11746 URL: https://issues.apache.org/jira/browse/HADOOP-11746 Project: Hadoop Common Issue Type: Test Components: build, test Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch, HADOOP-11746-05.patch, HADOOP-11746-06.patch, HADOOP-11746-07.patch, HADOOP-11746-09.patch, HADOOP-11746-10.patch, HADOOP-11746-11.patch, HADOOP-11746-12.patch, HADOOP-11746-13.patch, HADOOP-11746-14.patch, HADOOP-11746-15.patch, HADOOP-11746-16.patch, HADOOP-11746-17.patch, HADOOP-11746-18.patch, HADOOP-11746-19.patch, HADOOP-11746-20.patch, HADOOP-11746-21.patch This code is bad and you should feel bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (HADOOP-11590) Update sbin commands and documentation to use new --slaves option
[
https://issues.apache.org/jira/browse/HADOOP-11590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HADOOP-11590:
--
Comment: was deleted
(was: \\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 00s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12726963/HADOOP-11590-02.patch |
| Optional Tests | site shellcheck |
| git revision | HADOOP-11590 / a8c2df7 |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/fake//console |
This message was automatically generated.)
Update sbin commands and documentation to use new --slaves option
-
Key: HADOOP-11590
URL: https://issues.apache.org/jira/browse/HADOOP-11590
Project: Hadoop Common
Issue Type: Bug
Components: documentation, scripts
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Allen Wittenauer
Priority: Blocker
Attachments: HADOOP-11590-00.patch, HADOOP-11590-01.patch,
HADOOP-11590-02.patch
With HADOOP-11565 now committed, we need to remove usages of yarn-daemons.sh
and hadoop-daemons.sh from the start and stop scripts, converting them to use
the new --slaves option. Additionally, the documentation should be updated
to reflect these new command options.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11745) Incorporate ShellCheck static analysis into Jenkins pre-commit builds.
[ https://issues.apache.org/jira/browse/HADOOP-11745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505506#comment-14505506 ] Allen Wittenauer commented on HADOOP-11745: --- Even though HADOOP-11746 has the fix, someone needs to follow up and get shellcheck actually installed on the Jenkins nodes. Incorporate ShellCheck static analysis into Jenkins pre-commit builds. -- Key: HADOOP-11745 URL: https://issues.apache.org/jira/browse/HADOOP-11745 Project: Hadoop Common Issue Type: Improvement Components: build, scripts Reporter: Chris Nauroth Assignee: Allen Wittenauer Priority: Minor During the shell script rewrite on trunk, we've been using ShellCheck as a static analysis tool to catch common errors. We can incorporate this directly into Jenkins pre-commit builds. Jenkins can reply with a -1 on shell script patches that introduce new ShellCheck warnings. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HADOOP-11855) Add -Djava.awt.headless=true to pom.xml entries
[ https://issues.apache.org/jira/browse/HADOOP-11855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-11855. --- Resolution: Duplicate Add -Djava.awt.headless=true to pom.xml entries --- Key: HADOOP-11855 URL: https://issues.apache.org/jira/browse/HADOOP-11855 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 3.0.0 Environment: Mac OS X Reporter: Allen Wittenauer Priority: Minor Labels: newbie On OS X, certain profiles cause stealing of the focus, including tests, site, and javadoc creation. In order to prevent this, -Djava.awt.headless=true should be added to the argument lines for those subsystems. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (HADOOP-11590) Update sbin commands and documentation to use new --slaves option
[
https://issues.apache.org/jira/browse/HADOOP-11590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HADOOP-11590:
--
Comment: was deleted
(was: \\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 00s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12701772/HADOOP-11590-01.patch |
| Optional Tests | site shellcheck |
| git revision | HADOOP-11590 / a8c2df7 |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/fake//console |
This message was automatically generated.)
Update sbin commands and documentation to use new --slaves option
-
Key: HADOOP-11590
URL: https://issues.apache.org/jira/browse/HADOOP-11590
Project: Hadoop Common
Issue Type: Bug
Components: documentation, scripts
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Allen Wittenauer
Priority: Blocker
Attachments: HADOOP-11590-00.patch, HADOOP-11590-01.patch
With HADOOP-11565 now committed, we need to remove usages of yarn-daemons.sh
and hadoop-daemons.sh from the start and stop scripts, converting them to use
the new --slaves option. Additionally, the documentation should be updated
to reflect these new command options.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-11855) Add -Djava.awt.headless=true to pom.xml entries
Allen Wittenauer created HADOOP-11855: - Summary: Add -Djava.awt.headless=true to pom.xml entries Key: HADOOP-11855 URL: https://issues.apache.org/jira/browse/HADOOP-11855 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 3.0.0 Environment: Mac OS X Reporter: Allen Wittenauer Priority: Minor On OS X, certain profiles cause stealing of the focus, including tests, site, and javadoc creation. In order to prevent this, -Djava.awt.headless=true should be added to the argument lines for those subsystems. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11590) Update sbin commands and documentation to use new --slaves option
[ https://issues.apache.org/jira/browse/HADOOP-11590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11590: -- Attachment: HADOOP-11590-02.patch Update sbin commands and documentation to use new --slaves option - Key: HADOOP-11590 URL: https://issues.apache.org/jira/browse/HADOOP-11590 Project: Hadoop Common Issue Type: Bug Components: documentation, scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Priority: Blocker Attachments: HADOOP-11590-00.patch, HADOOP-11590-01.patch, HADOOP-11590-02.patch With HADOOP-11565 now committed, we need to remove usages of yarn-daemons.sh and hadoop-daemons.sh from the start and stop scripts, converting them to use the new --slaves option. Additionally, the documentation should be updated to reflect these new command options. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11850) Typos in hadoop-common java docs
[ https://issues.apache.org/jira/browse/HADOOP-11850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504903#comment-14504903 ] surendra singh lilhore commented on HADOOP-11850: - Thanks [~jghoman] for review. Fixed review comments and attached new patch. Please review Typos in hadoop-common java docs Key: HADOOP-11850 URL: https://issues.apache.org/jira/browse/HADOOP-11850 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.6.0 Reporter: surendra singh lilhore Assignee: surendra singh lilhore Priority: Minor Attachments: HADOOP-11850.patch, HADOOP-11850_1.patch This jira will fix the typo in hdfs-common project -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-8728) Display (fs -text) shouldn't hard-depend on Writable serialized sequence files.
[ https://issues.apache.org/jira/browse/HADOOP-8728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504527#comment-14504527 ] Harsh J commented on HADOOP-8728: - Patch still applies. Display (fs -text) shouldn't hard-depend on Writable serialized sequence files. --- Key: HADOOP-8728 URL: https://issues.apache.org/jira/browse/HADOOP-8728 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 2.6.0 Reporter: Harsh J Priority: Minor Attachments: HADOOP-8728.patch The Display command (fs -text) currently reads only Writable-based SequenceFiles. This isn't necessary to do, and prevents reading non-Writable-based serialization in SequenceFiles from the shell. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-8728) Display (fs -text) shouldn't hard-depend on Writable serialized sequence files.
[ https://issues.apache.org/jira/browse/HADOOP-8728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated HADOOP-8728: Target Version/s: 2.8.0 Display (fs -text) shouldn't hard-depend on Writable serialized sequence files. --- Key: HADOOP-8728 URL: https://issues.apache.org/jira/browse/HADOOP-8728 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 2.6.0 Reporter: Harsh J Priority: Minor Attachments: HADOOP-8728.patch The Display command (fs -text) currently reads only Writable-based SequenceFiles. This isn't necessary to do, and prevents reading non-Writable-based serialization in SequenceFiles from the shell. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-8728) Display (fs -text) shouldn't hard-depend on Writable serialized sequence files.
[
https://issues.apache.org/jira/browse/HADOOP-8728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504609#comment-14504609
]
Hadoop QA commented on HADOOP-8728:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12542448/HADOOP-8728.patch
against trunk revision d52de61.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/6136//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/6136//console
This message is automatically generated.
Display (fs -text) shouldn't hard-depend on Writable serialized sequence
files.
---
Key: HADOOP-8728
URL: https://issues.apache.org/jira/browse/HADOOP-8728
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 2.6.0
Reporter: Harsh J
Priority: Minor
Attachments: HADOOP-8728.patch
The Display command (fs -text) currently reads only Writable-based
SequenceFiles. This isn't necessary to do, and prevents reading
non-Writable-based serialization in SequenceFiles from the shell.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11704) ProxyUsers expects ipaddress due to MachineList but callers send in remote host
[ https://issues.apache.org/jira/browse/HADOOP-11704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anubhav Dhoot updated HADOOP-11704: --- Attachment: HADOOP-11704.002.patch Attaching unit tests to validate fix ProxyUsers expects ipaddress due to MachineList but callers send in remote host Key: HADOOP-11704 URL: https://issues.apache.org/jira/browse/HADOOP-11704 Project: Hadoop Common Issue Type: Bug Reporter: Anubhav Dhoot Assignee: Anubhav Dhoot Attachments: HADOOP-11704.001.patch, HADOOP-11704.002.patch DelegationTokenAuthenticationHandler and DelegationTokenAuthenticationFilter are using ServletRequest#getRemoteHost which can send an address if possible. It should use getRemoteAddr instead -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11704) ProxyUsers expects ipaddress due to MachineList but callers send in remote host
[
https://issues.apache.org/jira/browse/HADOOP-11704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504684#comment-14504684
]
Hadoop QA commented on HADOOP-11704:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12726826/HADOOP-11704.002.patch
against trunk revision d52de61.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/6137//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/6137//console
This message is automatically generated.
ProxyUsers expects ipaddress due to MachineList but callers send in remote
host
Key: HADOOP-11704
URL: https://issues.apache.org/jira/browse/HADOOP-11704
Project: Hadoop Common
Issue Type: Bug
Reporter: Anubhav Dhoot
Assignee: Anubhav Dhoot
Attachments: HADOOP-11704.001.patch, HADOOP-11704.002.patch
DelegationTokenAuthenticationHandler and DelegationTokenAuthenticationFilter
are using ServletRequest#getRemoteHost which can send an address if possible.
It should use getRemoteAddr instead
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-8728) Display (fs -text) shouldn't hard-depend on Writable serialized sequence files.
[ https://issues.apache.org/jira/browse/HADOOP-8728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated HADOOP-8728: Affects Version/s: (was: 2.0.0-alpha) 2.6.0 Display (fs -text) shouldn't hard-depend on Writable serialized sequence files. --- Key: HADOOP-8728 URL: https://issues.apache.org/jira/browse/HADOOP-8728 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 2.6.0 Reporter: Harsh J Priority: Minor Attachments: HADOOP-8728.patch The Display command (fs -text) currently reads only Writable-based SequenceFiles. This isn't necessary to do, and prevents reading non-Writable-based serialization in SequenceFiles from the shell. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11843) Make setting up the build environment easier
[ https://issues.apache.org/jira/browse/HADOOP-11843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504659#comment-14504659 ] Niels Basjes commented on HADOOP-11843: --- I don't have a Mac so I'm unable to develop/test this boot2docker/NFS solution. At home I run everything as Windows-7(VirtualBox(CentOS-6.6(Docker( HADOOP-11843 and I run IntelliJ inside the same CentOS which performs fine. At work I have a system that runs CentOS natively on the hardware which also performs fine. Although being a real pain I propose I simply add some documentation in the BUILDING.TXT pointing to these workarounds for the time these problems exist. Make setting up the build environment easier Key: HADOOP-11843 URL: https://issues.apache.org/jira/browse/HADOOP-11843 Project: Hadoop Common Issue Type: New Feature Reporter: Niels Basjes Assignee: Niels Basjes Attachments: HADOOP-11843-2015-04-17-1612.patch, HADOOP-11843-2015-04-17-2226.patch, HADOOP-11843-2015-04-17-2308.patch, HADOOP-11843-2015-04-19-2206.patch, HADOOP-11843-2015-04-19-2232.patch ( As discussed with [~aw] ) In AVRO-1537 a docker based solution was created to setup all the tools for doing a full build. This enables much easier reproduction of any issues and getting up and running for new developers. This issue is to 'copy/port' that setup into the hadoop project in preparation for the bug squash. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11850) Typos in hadoop-common java docs
[ https://issues.apache.org/jira/browse/HADOOP-11850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] surendra singh lilhore updated HADOOP-11850: Attachment: HADOOP-11850_1.patch Typos in hadoop-common java docs Key: HADOOP-11850 URL: https://issues.apache.org/jira/browse/HADOOP-11850 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.6.0 Reporter: surendra singh lilhore Assignee: surendra singh lilhore Priority: Minor Attachments: HADOOP-11850.patch, HADOOP-11850_1.patch This jira will fix the typo in hdfs-common project -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11828) Implement the Hitchhiker erasure coding algorithm
[ https://issues.apache.org/jira/browse/HADOOP-11828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jack liuquan updated HADOOP-11828: -- Attachment: HADOOP-11828-hitchhikerXOR-V3.zip Implement the Hitchhiker erasure coding algorithm - Key: HADOOP-11828 URL: https://issues.apache.org/jira/browse/HADOOP-11828 Project: Hadoop Common Issue Type: Sub-task Reporter: Zhe Zhang Assignee: jack liuquan Attachments: 7715-hitchhikerXOR-v2-testcode.patch, 7715-hitchhikerXOR-v2.patch, HADOOP-11828-hitchhikerXOR-V3.zip, HDFS-7715-hhxor-decoder.patch, HDFS-7715-hhxor-encoder.patch [Hitchhiker | http://www.eecs.berkeley.edu/~nihar/publications/Hitchhiker_SIGCOMM14.pdf] is a new erasure coding algorithm developed as a research project at UC Berkeley. It has been shown to reduce network traffic and disk I/O by 25%-45% during data reconstruction. This JIRA aims to introduce Hitchhiker to the HDFS-EC framework, as one of the pluggable codec algorithms. The existing implementation is based on HDFS-RAID. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11828) Implement the Hitchhiker erasure coding algorithm
[ https://issues.apache.org/jira/browse/HADOOP-11828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jack liuquan updated HADOOP-11828: -- Attachment: HADOOP-11828-hitchhikerXOR-V3.patch Implement the Hitchhiker erasure coding algorithm - Key: HADOOP-11828 URL: https://issues.apache.org/jira/browse/HADOOP-11828 Project: Hadoop Common Issue Type: Sub-task Reporter: Zhe Zhang Assignee: jack liuquan Attachments: 7715-hitchhikerXOR-v2-testcode.patch, 7715-hitchhikerXOR-v2.patch, HADOOP-11828-hitchhikerXOR-V3.patch, HDFS-7715-hhxor-decoder.patch, HDFS-7715-hhxor-encoder.patch [Hitchhiker | http://www.eecs.berkeley.edu/~nihar/publications/Hitchhiker_SIGCOMM14.pdf] is a new erasure coding algorithm developed as a research project at UC Berkeley. It has been shown to reduce network traffic and disk I/O by 25%-45% during data reconstruction. This JIRA aims to introduce Hitchhiker to the HDFS-EC framework, as one of the pluggable codec algorithms. The existing implementation is based on HDFS-RAID. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11828) Implement the Hitchhiker erasure coding algorithm
[ https://issues.apache.org/jira/browse/HADOOP-11828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jack liuquan updated HADOOP-11828: -- Attachment: (was: HADOOP-11828-hitchhikerXOR-V3.zip) Implement the Hitchhiker erasure coding algorithm - Key: HADOOP-11828 URL: https://issues.apache.org/jira/browse/HADOOP-11828 Project: Hadoop Common Issue Type: Sub-task Reporter: Zhe Zhang Assignee: jack liuquan Attachments: 7715-hitchhikerXOR-v2-testcode.patch, 7715-hitchhikerXOR-v2.patch, HADOOP-11828-hitchhikerXOR-V3.patch, HDFS-7715-hhxor-decoder.patch, HDFS-7715-hhxor-encoder.patch [Hitchhiker | http://www.eecs.berkeley.edu/~nihar/publications/Hitchhiker_SIGCOMM14.pdf] is a new erasure coding algorithm developed as a research project at UC Berkeley. It has been shown to reduce network traffic and disk I/O by 25%-45% during data reconstruction. This JIRA aims to introduce Hitchhiker to the HDFS-EC framework, as one of the pluggable codec algorithms. The existing implementation is based on HDFS-RAID. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-11854) Fix Typos in all the projects
Brahma Reddy Battula created HADOOP-11854: - Summary: Fix Typos in all the projects Key: HADOOP-11854 URL: https://issues.apache.org/jira/browse/HADOOP-11854 Project: Hadoop Common Issue Type: Bug Reporter: Brahma Reddy Battula Assignee: Brahma Reddy Battula Priority: Minor Recently I had seen, there are so many jira's for fixing the typo's ( Keep on accumulating more ). Hence I want to plan in proper manner such that everything will be addressed.. I am thinking, we can fix project level ( at most package level)... My intention to avoid the number of jira's on typo's...One more suggestion to reviewer's is please dn't commit for class level try to check project level ( atmost package level) if any such typo's present... Please correct me If I am wrong.. I will close this jira.. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11828) Implement the Hitchhiker erasure coding algorithm
[ https://issues.apache.org/jira/browse/HADOOP-11828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505125#comment-14505125 ] jack liuquan commented on HADOOP-11828: --- Hi kai, I have uploaded a new patch. I think you are right, maybe the hitchhiker in ErasureCoder layer is better. So in the new patch, I move hitchhiker-XOR to the ErasureCoder layer. Please review the codes, Thanks a lot! Implement the Hitchhiker erasure coding algorithm - Key: HADOOP-11828 URL: https://issues.apache.org/jira/browse/HADOOP-11828 Project: Hadoop Common Issue Type: Sub-task Reporter: Zhe Zhang Assignee: jack liuquan Attachments: 7715-hitchhikerXOR-v2-testcode.patch, 7715-hitchhikerXOR-v2.patch, HADOOP-11828-hitchhikerXOR-V3.patch, HDFS-7715-hhxor-decoder.patch, HDFS-7715-hhxor-encoder.patch [Hitchhiker | http://www.eecs.berkeley.edu/~nihar/publications/Hitchhiker_SIGCOMM14.pdf] is a new erasure coding algorithm developed as a research project at UC Berkeley. It has been shown to reduce network traffic and disk I/O by 25%-45% during data reconstruction. This JIRA aims to introduce Hitchhiker to the HDFS-EC framework, as one of the pluggable codec algorithms. The existing implementation is based on HDFS-RAID. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11850) Typos in hadoop-common java docs
[
https://issues.apache.org/jira/browse/HADOOP-11850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505150#comment-14505150
]
Brahma Reddy Battula commented on HADOOP-11850:
---
[~surendrasingh] thanks for working on this...Patch overall ,LGTM,
One comment:
Can you please fix the following one also...
*{{Class :}}*
org.apache.hadoop.io.compress.bzip2.CBZip2InputStream, *line num: 1222*
{code}
* Initializes the {@link #tt} array.
*
* This method is called when the required length of the array is known.
* I don't initialize it at construction time to avoid unneccessary
* memory allocation when compressing small files.
*/
{code}
Typos in hadoop-common java docs
Key: HADOOP-11850
URL: https://issues.apache.org/jira/browse/HADOOP-11850
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: surendra singh lilhore
Assignee: surendra singh lilhore
Priority: Minor
Attachments: HADOOP-11850.patch, HADOOP-11850_1.patch
This jira will fix the typo in hdfs-common project
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11850) Typos in hadoop-common java docs
[ https://issues.apache.org/jira/browse/HADOOP-11850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HADOOP-11850: -- Issue Type: Sub-task (was: Bug) Parent: HADOOP-11854 Typos in hadoop-common java docs Key: HADOOP-11850 URL: https://issues.apache.org/jira/browse/HADOOP-11850 Project: Hadoop Common Issue Type: Sub-task Affects Versions: 2.6.0 Reporter: surendra singh lilhore Assignee: surendra singh lilhore Priority: Minor Attachments: HADOOP-11850.patch, HADOOP-11850_1.patch This jira will fix the typo in hdfs-common project -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11852) Disable symlinks in trunk
[ https://issues.apache.org/jira/browse/HADOOP-11852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505694#comment-14505694 ] Andrew Wang commented on HADOOP-11852: -- I ran mvn eclipse:eclipse locally and it worked okay, not sure why test-patch barfed. Disable symlinks in trunk - Key: HADOOP-11852 URL: https://issues.apache.org/jira/browse/HADOOP-11852 Project: Hadoop Common Issue Type: Sub-task Affects Versions: 3.0.0 Reporter: Andrew Wang Assignee: Andrew Wang Attachments: hadoop-11852.001.patch In HADOOP-10020 and HADOOP-10162 we disabled symlinks in branch-2. Since there's currently no plan to finish this work, let's disable it in trunk too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11746) rewrite test-patch.sh
[ https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505755#comment-14505755 ] Hudson commented on HADOOP-11746: - FAILURE: Integrated in Hadoop-trunk-Commit #7627 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7627/]) HADOOP-11746. rewrite test-patch.sh (aw) (aw: rev 73ddb6b4f825be1d06fd1d2be86a4bea241e7aa0) * dev-support/test-patch.d/checkstyle.sh * dev-support/shelldocs.py * hadoop-common-project/hadoop-common/CHANGES.txt * dev-support/test-patch.sh * dev-support/test-patch.d/shellcheck.sh * dev-support/test-patch.d/whitespace.sh rewrite test-patch.sh - Key: HADOOP-11746 URL: https://issues.apache.org/jira/browse/HADOOP-11746 Project: Hadoop Common Issue Type: Test Components: build, test Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Fix For: 2.8.0 Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch, HADOOP-11746-05.patch, HADOOP-11746-06.patch, HADOOP-11746-07.patch, HADOOP-11746-09.patch, HADOOP-11746-10.patch, HADOOP-11746-11.patch, HADOOP-11746-12.patch, HADOOP-11746-13.patch, HADOOP-11746-14.patch, HADOOP-11746-15.patch, HADOOP-11746-16.patch, HADOOP-11746-17.patch, HADOOP-11746-18.patch, HADOOP-11746-19.patch, HADOOP-11746-20.patch, HADOOP-11746-21.branch-2.patch, HADOOP-11746-21.patch This code is bad and you should feel bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11746) rewrite test-patch.sh
[ https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505798#comment-14505798 ] Allen Wittenauer commented on HADOOP-11746: --- FYI, HDFS-8200 @ https://builds.apache.org/job/PreCommit-HDFS-Build/10335/console appears to be the first jenkins run with the new test-patch.sh . rewrite test-patch.sh - Key: HADOOP-11746 URL: https://issues.apache.org/jira/browse/HADOOP-11746 Project: Hadoop Common Issue Type: Test Components: build, test Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Fix For: 2.8.0 Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch, HADOOP-11746-05.patch, HADOOP-11746-06.patch, HADOOP-11746-07.patch, HADOOP-11746-09.patch, HADOOP-11746-10.patch, HADOOP-11746-11.patch, HADOOP-11746-12.patch, HADOOP-11746-13.patch, HADOOP-11746-14.patch, HADOOP-11746-15.patch, HADOOP-11746-16.patch, HADOOP-11746-17.patch, HADOOP-11746-18.patch, HADOOP-11746-19.patch, HADOOP-11746-20.patch, HADOOP-11746-21.branch-2.patch, HADOOP-11746-21.patch This code is bad and you should feel bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-7154) Should set MALLOC_ARENA_MAX in hadoop-config.sh
[ https://issues.apache.org/jira/browse/HADOOP-7154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505800#comment-14505800 ] Lari Hotari commented on HADOOP-7154: - A note about MALLOC_ARENA_MAX: MALLOC_ARENA_MAX is broken on glibc 2.15 (like Ubuntu 10.04) . The fix was made for 2.16 and backported to 2.15 . MALLOC_ARENA_MAX doesn't work on Ubuntu 10.04 because of [this bug|https://sourceware.org/bugzilla/show_bug.cgi?id=13071]. The same bug seems to be reported to Redhat as https://bugzilla.redhat.com/show_bug.cgi?id=799327 . Other reports: https://sourceware.org/bugzilla/show_bug.cgi?id=13137 , https://sourceware.org/bugzilla/show_bug.cgi?id=13754 , https://sourceware.org/bugzilla/show_bug.cgi?id=11261 . This is the commit to glibc fixing the bug: https://github.com/bminor/glibc/commit/41b81892f11fe1353123e892158b53de73863d62 (backport for 2.15 is https://github.com/bminor/glibc/commit/7cf8e20d03a43b1375e90d381a16caa2686e4fdf ). Should set MALLOC_ARENA_MAX in hadoop-config.sh --- Key: HADOOP-7154 URL: https://issues.apache.org/jira/browse/HADOOP-7154 Project: Hadoop Common Issue Type: Improvement Components: scripts Affects Versions: 0.22.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Minor Fix For: 1.0.4, 0.22.0 Attachments: hadoop-7154.txt New versions of glibc present in RHEL6 include a new arena allocator design. In several clusters we've seen this new allocator cause huge amounts of virtual memory to be used, since when multiple threads perform allocations, they each get their own memory arena. On a 64-bit system, these arenas are 64M mappings, and the maximum number of arenas is 8 times the number of cores. We've observed a DN process using 14GB of vmem for only 300M of resident set. This causes all kinds of nasty issues for obvious reasons. Setting MALLOC_ARENA_MAX to a low number will restrict the number of memory arenas and bound the virtual memory, with no noticeable downside in performance - we've been recommending MALLOC_ARENA_MAX=4. We should set this in hadoop-env.sh to avoid this issue as RHEL6 becomes more and more common. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11846) TestCertificateUtil.testCorruptPEM failing on Jenkins JDK8
[
https://issues.apache.org/jira/browse/HADOOP-11846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-11846:
Resolution: Fixed
Fix Version/s: 2.8.0
Status: Resolved (was: Patch Available)
+1, applied!
TestCertificateUtil.testCorruptPEM failing on Jenkins JDK8
--
Key: HADOOP-11846
URL: https://issues.apache.org/jira/browse/HADOOP-11846
Project: Hadoop Common
Issue Type: Bug
Components: build, security
Affects Versions: 3.0.0
Environment: Java 8
Reporter: Steve Loughran
Assignee: Larry McCay
Fix For: 2.8.0
Attachments: HADOOP-11846-1.patch
Jenkins is failing on {{TestCertificateUtil.testCorruptPEM}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11812) Implement listLocatedStatus for ViewFileSystem to speed up split calculation
[ https://issues.apache.org/jira/browse/HADOOP-11812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505823#comment-14505823 ] Hudson commented on HADOOP-11812: - SUCCESS: Integrated in Hadoop-trunk-Commit #7628 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7628/]) HADOOP-11812. Implement listLocatedStatus for ViewFileSystem to speed up split calculation (gera) (gera: rev 6d2cf9fbbd02482315a091ab07af26e40cc5134f) * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/LocatedFileStatus.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ChRootedFileSystem.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFsLocatedFileStatus.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestChRootedFileSystem.java Implement listLocatedStatus for ViewFileSystem to speed up split calculation Key: HADOOP-11812 URL: https://issues.apache.org/jira/browse/HADOOP-11812 Project: Hadoop Common Issue Type: Improvement Components: fs Affects Versions: 2.7.0 Reporter: Gera Shegalov Assignee: Gera Shegalov Priority: Blocker Labels: performance Fix For: 2.8.0 Attachments: HADOOP-11812.001.patch, HADOOP-11812.002.patch, HADOOP-11812.003.patch, HADOOP-11812.004.patch, HADOOP-11812.005.patch ViewFileSystem is currently not taking advantage of MAPREDUCE-1981. This causes several x of RPC overhead and added latency. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11746) rewrite test-patch.sh
[ https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505710#comment-14505710 ] Chris Nauroth commented on HADOOP-11746: I didn't realize trunk test-patch.sh had diverged so much from branch-2. Is this basically just copying the new code over to branch-2? If so, then +1 for the branch-2 patch. If not, then can you point out specific bits that would need review? Thanks! rewrite test-patch.sh - Key: HADOOP-11746 URL: https://issues.apache.org/jira/browse/HADOOP-11746 Project: Hadoop Common Issue Type: Test Components: build, test Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch, HADOOP-11746-05.patch, HADOOP-11746-06.patch, HADOOP-11746-07.patch, HADOOP-11746-09.patch, HADOOP-11746-10.patch, HADOOP-11746-11.patch, HADOOP-11746-12.patch, HADOOP-11746-13.patch, HADOOP-11746-14.patch, HADOOP-11746-15.patch, HADOOP-11746-16.patch, HADOOP-11746-17.patch, HADOOP-11746-18.patch, HADOOP-11746-19.patch, HADOOP-11746-20.patch, HADOOP-11746-21.branch-2.patch, HADOOP-11746-21.patch This code is bad and you should feel bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11590) Update sbin commands and documentation to use new --slaves option
[
https://issues.apache.org/jira/browse/HADOOP-11590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505929#comment-14505929
]
Hadoop QA commented on HADOOP-11590:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12726963/HADOOP-11590-02.patch
against trunk revision 997408e.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation patch that doesn't require tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The following test timeouts occurred in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestDFSClientFailover
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/6142//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/6142//console
This message is automatically generated.
Update sbin commands and documentation to use new --slaves option
-
Key: HADOOP-11590
URL: https://issues.apache.org/jira/browse/HADOOP-11590
Project: Hadoop Common
Issue Type: Bug
Components: documentation, scripts
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Allen Wittenauer
Priority: Blocker
Attachments: HADOOP-11590-00.patch, HADOOP-11590-01.patch,
HADOOP-11590-02.patch
With HADOOP-11565 now committed, we need to remove usages of yarn-daemons.sh
and hadoop-daemons.sh from the start and stop scripts, converting them to use
the new --slaves option. Additionally, the documentation should be updated
to reflect these new command options.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh
[ https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11746: -- Release Note: * test-patch.sh now has new output that is different than the previous versions * test-patch.sh is now pluggable via the test-patch.d directory, with checkstyle and shellcheck tests included * JIRA comments now use much more markup to improve readability * test-patch.sh now supports either a file name, a URL, or a JIRA issue as input in developer mode * If part of the patch testing code is changed, test-patch.sh will now attempt to re-executing itself using the new version. * Some logic to try and reduce the amount of unnecessary tests. For example, patches that only modify markdown should not run the Java compilation tests. * Plugins for checkstyle, shellcheck, and whitespace now execute as necessary. * New test code for mvn site * A breakdown of the times needed to execute certain blocks as well as a total runtime is now reported to assist in fixing long running tests and optimize the entire process. * Several new options * --resetrepo will put test-patch.sh in destructive mode, similar to a normal Jenkins run * --testlist allows one to provide a comma delimited list of test subsystems to forcibly execute * --modulelist to provide a comma delimited list of module tests to execute in addition to the ones that are automatically detected * --offline mode to attempt to stop connecting to the Internet for certain operations * test-patch.sh now defaults to the POSIX equivalents on Solaris and Illumos-based operating systems * shelldocs.py may be used to generate test-patch.sh API information * FindBugs output is now listed on the JIRA comment * lots of general code cleanup, including attempts to remove any local state files to reduce potential race conditions * Some logic to determine if a patch is for a given major branch using several strategies as well as a particular git ref (using git+ref as part of the name). * Some logic to determine if a patch references a particular JIRA issue. * Unit tests are only flagged as necessary with native or Java code, since Hadoop has no framework in place yet for other types of unit tests. * test-patch now exits with a failure status if problems arise trying to do git checkouts. Previously the exit code was success. was: * test-patch.sh now has new output that is different than the previous versions * test-patch.sh is now pluggable via the test-patch.d directory, with checkstyle and shellcheck tests included * JIRA comments now use much more markup to improve readability * test-patch.sh now supports either a file name, a URL, or a JIRA issue as input in developer mode * If part of the patch testing code is changed, test-patch.sh will now attempt to re-executing itself using the new version. * Some logic to try and reduce the amount of unnecessary tests. For example, patches that only modify markdown should not run the Java compilation tests. * Plugins for checkstyle, shellcheck, and whitespace now execute as necessary. * New test code for mvn site * A breakdown of the times needed to execute certain blocks as well as a total runtime is now reported to assist in fixing long running tests and optimize the entire process. rewrite test-patch.sh - Key: HADOOP-11746 URL: https://issues.apache.org/jira/browse/HADOOP-11746 Project: Hadoop Common Issue Type: Test Components: build, test Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch, HADOOP-11746-05.patch, HADOOP-11746-06.patch, HADOOP-11746-07.patch, HADOOP-11746-09.patch, HADOOP-11746-10.patch, HADOOP-11746-11.patch, HADOOP-11746-12.patch, HADOOP-11746-13.patch, HADOOP-11746-14.patch, HADOOP-11746-15.patch, HADOOP-11746-16.patch, HADOOP-11746-17.patch, HADOOP-11746-18.patch, HADOOP-11746-19.patch, HADOOP-11746-20.patch, HADOOP-11746-21.branch-2.patch, HADOOP-11746-21.patch This code is bad and you should feel bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11812) Implement listLocatedStatus for ViewFileSystem to speed up split calculation
[ https://issues.apache.org/jira/browse/HADOOP-11812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gera Shegalov updated HADOOP-11812: --- Resolution: Fixed Fix Version/s: 2.8.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks [~laurentgo] and [~cnauroth] for review! Committed to trunk and branch-2. Implement listLocatedStatus for ViewFileSystem to speed up split calculation Key: HADOOP-11812 URL: https://issues.apache.org/jira/browse/HADOOP-11812 Project: Hadoop Common Issue Type: Improvement Components: fs Affects Versions: 2.7.0 Reporter: Gera Shegalov Assignee: Gera Shegalov Priority: Blocker Labels: performance Fix For: 2.8.0 Attachments: HADOOP-11812.001.patch, HADOOP-11812.002.patch, HADOOP-11812.003.patch, HADOOP-11812.004.patch, HADOOP-11812.005.patch ViewFileSystem is currently not taking advantage of MAPREDUCE-1981. This causes several x of RPC overhead and added latency. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11746) rewrite test-patch.sh
[ https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505810#comment-14505810 ] Allen Wittenauer commented on HADOOP-11746: --- Nope, MAPREDUCE-6324 @ https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5426/console was first by a few minutes. Both have entered into mvn test phase. Interesting to note that git clean removed quite a bit of gunk from the git repo on the HDFS test. Hmm.. rewrite test-patch.sh - Key: HADOOP-11746 URL: https://issues.apache.org/jira/browse/HADOOP-11746 Project: Hadoop Common Issue Type: Test Components: build, test Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Fix For: 2.8.0 Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch, HADOOP-11746-05.patch, HADOOP-11746-06.patch, HADOOP-11746-07.patch, HADOOP-11746-09.patch, HADOOP-11746-10.patch, HADOOP-11746-11.patch, HADOOP-11746-12.patch, HADOOP-11746-13.patch, HADOOP-11746-14.patch, HADOOP-11746-15.patch, HADOOP-11746-16.patch, HADOOP-11746-17.patch, HADOOP-11746-18.patch, HADOOP-11746-19.patch, HADOOP-11746-20.patch, HADOOP-11746-21.branch-2.patch, HADOOP-11746-21.patch This code is bad and you should feel bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11858) [JDK8] Set minimum version of Hadoop 3 to JDK 8
[ https://issues.apache.org/jira/browse/HADOOP-11858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated HADOOP-11858: --- Attachment: HADOOP-11858.001.patch [JDK8] Set minimum version of Hadoop 3 to JDK 8 --- Key: HADOOP-11858 URL: https://issues.apache.org/jira/browse/HADOOP-11858 Project: Hadoop Common Issue Type: Sub-task Components: build Affects Versions: 3.0.0 Reporter: Robert Kanter Assignee: Robert Kanter Attachments: HADOOP-11858.001.patch Set minimum version of trunk to JDK 8 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-11859) PseudoAuthenticationHandler fails with httpcomponents v4.4
Eugene Koifman created HADOOP-11859: --- Summary: PseudoAuthenticationHandler fails with httpcomponents v4.4 Key: HADOOP-11859 URL: https://issues.apache.org/jira/browse/HADOOP-11859 Project: Hadoop Common Issue Type: Bug Reporter: Eugene Koifman Assignee: Eugene Koifman This shows in the context of WebHCat and Hive (which recently moved to httpcomponents:httpclient:4.4) but could happen in other places. URLEncodedUtils.parse(String, Charset) which is called from PseudoAuthenticationHandler.getUserName() with the 1st argument produced by HttpServletRequest.getQueryString(). The later returns NULL if there is no query string in the URL. in httpcoponents:httpclient:4.2.5 parse() gracefully handles first argument being NULL, but in 4.4 it NPEs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11746) rewrite test-patch.sh
[
https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505677#comment-14505677
]
Hadoop QA commented on HADOOP-11746:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12726970/HADOOP-11746-21.patch
against trunk revision 997408e.
{color:red}-1 @author{color}. The patch appears to contain 13 @author tags
which the Hadoop community has agreed to not allow in code contributions.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/6141//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/6141//console
This message is automatically generated.
rewrite test-patch.sh
-
Key: HADOOP-11746
URL: https://issues.apache.org/jira/browse/HADOOP-11746
Project: Hadoop Common
Issue Type: Test
Components: build, test
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Allen Wittenauer
Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch,
HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch,
HADOOP-11746-05.patch, HADOOP-11746-06.patch, HADOOP-11746-07.patch,
HADOOP-11746-09.patch, HADOOP-11746-10.patch, HADOOP-11746-11.patch,
HADOOP-11746-12.patch, HADOOP-11746-13.patch, HADOOP-11746-14.patch,
HADOOP-11746-15.patch, HADOOP-11746-16.patch, HADOOP-11746-17.patch,
HADOOP-11746-18.patch, HADOOP-11746-19.patch, HADOOP-11746-20.patch,
HADOOP-11746-21.branch-2.patch, HADOOP-11746-21.patch
This code is bad and you should feel bad.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11846) TestCertificateUtil.testCorruptPEM failing on Jenkins JDK8
[
https://issues.apache.org/jira/browse/HADOOP-11846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505848#comment-14505848
]
Hudson commented on HADOOP-11846:
-
SUCCESS: Integrated in Hadoop-trunk-Commit #7629 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/7629/])
HADOOP-11846 TestCertificateUtil.testCorruptPEM failing on Jenkins JDK8. (Larry
McCay via stevel) (stevel: rev 2c1469036863c593a1bcba68f7c68b46f2a8e169)
*
hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestCertificateUtil.java
* hadoop-common-project/hadoop-common/CHANGES.txt
TestCertificateUtil.testCorruptPEM failing on Jenkins JDK8
--
Key: HADOOP-11846
URL: https://issues.apache.org/jira/browse/HADOOP-11846
Project: Hadoop Common
Issue Type: Bug
Components: build, security
Affects Versions: 3.0.0
Environment: Java 8
Reporter: Steve Loughran
Assignee: Larry McCay
Fix For: 2.8.0
Attachments: HADOOP-11846-1.patch
Jenkins is failing on {{TestCertificateUtil.testCorruptPEM}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh
[ https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11746: -- Attachment: HADOOP-11746-21.branch-2.patch branch-2 version. rewrite test-patch.sh - Key: HADOOP-11746 URL: https://issues.apache.org/jira/browse/HADOOP-11746 Project: Hadoop Common Issue Type: Test Components: build, test Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch, HADOOP-11746-05.patch, HADOOP-11746-06.patch, HADOOP-11746-07.patch, HADOOP-11746-09.patch, HADOOP-11746-10.patch, HADOOP-11746-11.patch, HADOOP-11746-12.patch, HADOOP-11746-13.patch, HADOOP-11746-14.patch, HADOOP-11746-15.patch, HADOOP-11746-16.patch, HADOOP-11746-17.patch, HADOOP-11746-18.patch, HADOOP-11746-19.patch, HADOOP-11746-20.patch, HADOOP-11746-21.branch-2.patch, HADOOP-11746-21.patch This code is bad and you should feel bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-11858) [JDK8] Set minimum version of Hadoop 3 to JDK 8
Robert Kanter created HADOOP-11858: -- Summary: [JDK8] Set minimum version of Hadoop 3 to JDK 8 Key: HADOOP-11858 URL: https://issues.apache.org/jira/browse/HADOOP-11858 Project: Hadoop Common Issue Type: Sub-task Components: build Affects Versions: 3.0.0 Reporter: Robert Kanter Assignee: Robert Kanter Set minimum version of trunk to JDK 8 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11827) Speed-up distcp buildListing() using threadpool
[ https://issues.apache.org/jira/browse/HADOOP-11827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505741#comment-14505741 ] Zoran Dimitrijevic commented on HADOOP-11827: - LGTM++ Speed-up distcp buildListing() using threadpool --- Key: HADOOP-11827 URL: https://issues.apache.org/jira/browse/HADOOP-11827 Project: Hadoop Common Issue Type: Improvement Components: tools/distcp Affects Versions: 2.7.0, 2.7.1 Reporter: Zoran Dimitrijevic Assignee: Zoran Dimitrijevic Attachments: HADOOP-11827-02.patch, HADOOP-11827-03.patch, HADOOP-11827-04.patch, HADOOP-11827.patch Original Estimate: 24h Remaining Estimate: 24h For very large source trees on s3 distcp is taking long time to build file listing (client code, before starting mappers). For a dataset I used (1.5M files, 50K dirs) it was taking 65 minutes before my fix in HADOOP-11785 and 36 minutes after the fix). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11859) PseudoAuthenticationHandler fails with httpcomponents v4.4
[ https://issues.apache.org/jira/browse/HADOOP-11859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HADOOP-11859: Status: Patch Available (was: Open) PseudoAuthenticationHandler fails with httpcomponents v4.4 -- Key: HADOOP-11859 URL: https://issues.apache.org/jira/browse/HADOOP-11859 Project: Hadoop Common Issue Type: Bug Reporter: Eugene Koifman Assignee: Eugene Koifman Attachments: HADOOP-11859.patch This shows in the context of WebHCat and Hive (which recently moved to httpcomponents:httpclient:4.4) but could happen in other places. URLEncodedUtils.parse(String, Charset) which is called from PseudoAuthenticationHandler.getUserName() with the 1st argument produced by HttpServletRequest.getQueryString(). The later returns NULL if there is no query string in the URL. in httpcoponents:httpclient:4.2.5 parse() gracefully handles first argument being NULL, but in 4.4 it NPEs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11859) PseudoAuthenticationHandler fails with httpcomponents v4.4
[ https://issues.apache.org/jira/browse/HADOOP-11859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HADOOP-11859: Attachment: HADOOP-11859.patch PseudoAuthenticationHandler fails with httpcomponents v4.4 -- Key: HADOOP-11859 URL: https://issues.apache.org/jira/browse/HADOOP-11859 Project: Hadoop Common Issue Type: Bug Reporter: Eugene Koifman Assignee: Eugene Koifman Attachments: HADOOP-11859.patch This shows in the context of WebHCat and Hive (which recently moved to httpcomponents:httpclient:4.4) but could happen in other places. URLEncodedUtils.parse(String, Charset) which is called from PseudoAuthenticationHandler.getUserName() with the 1st argument produced by HttpServletRequest.getQueryString(). The later returns NULL if there is no query string in the URL. in httpcoponents:httpclient:4.2.5 parse() gracefully handles first argument being NULL, but in 4.4 it NPEs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-11860) Enhance erasure coder allowing to read least required inputs in decoding of block group
Kai Zheng created HADOOP-11860:
--
Summary: Enhance erasure coder allowing to read least required
inputs in decoding of block group
Key: HADOOP-11860
URL: https://issues.apache.org/jira/browse/HADOOP-11860
Project: Hadoop Common
Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Kai Zheng
Based on the work in HADOOP-11847, this will follow on and improve in the
{{ErasureCoder}} layer accordingly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11820) aw jira testing, ignore
[
https://issues.apache.org/jira/browse/HADOOP-11820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505963#comment-14505963
]
Hadoop QA commented on HADOOP-11820:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12726959/HADOOP-11590-02.patch
against trunk revision 997408e.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.ha.TestDFSZKFailoverController
org.apache.hadoop.hdfs.server.namenode.TestFileTruncate
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/6139//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/6139//console
This message is automatically generated.
aw jira testing, ignore
---
Key: HADOOP-11820
URL: https://issues.apache.org/jira/browse/HADOOP-11820
Project: Hadoop Common
Issue Type: Task
Reporter: Allen Wittenauer
Attachments: HADOOP-11590-02.patch, HADOOP-11746-14.patch,
HADOOP-11746-16.patch, HADOOP-11746-17.patch, HADOOP-11746-20.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11861) test-patch script always fails when option --build-native=false is specified
[
https://issues.apache.org/jira/browse/HADOOP-11861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505969#comment-14505969
]
Anu Engineer commented on HADOOP-11861:
---
[~aw] would you like to take a look at this to see if this is related to the
new changes to build ?
test-patch script always fails when option --build-native=false is
specified
---
Key: HADOOP-11861
URL: https://issues.apache.org/jira/browse/HADOOP-11861
Project: Hadoop Common
Issue Type: Bug
Components: build
Affects Versions: 2.8.0
Reporter: Anu Engineer
if you specify --build-native=false like
{code}
./dev-support/test-patch.sh --build-native=false
~/workspaces/patches/hdfs-8211.001.patch
{code}
mvn fails with invalid lifecycle error.
Here are the steps to repro :
1) run any patch with --buid-native=false option
2) Open up /tmp/hadoop-test-patch/tmp-patch/patchJavacWarnings.txt to see
the failure reason.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11848) Incorrect arguments to sizeof in DomainSocket.c
[ https://issues.apache.org/jira/browse/HADOOP-11848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14506008#comment-14506008 ] Colin Patrick McCabe commented on HADOOP-11848: --- I agree, this was not what was intended. Malcolm, do you want to post a patch for this? Incorrect arguments to sizeof in DomainSocket.c --- Key: HADOOP-11848 URL: https://issues.apache.org/jira/browse/HADOOP-11848 Project: Hadoop Common Issue Type: Bug Components: native Affects Versions: 2.6.0 Reporter: Malcolm Kavalsky Assignee: Malcolm Kavalsky Original Estimate: 24h Remaining Estimate: 24h Length of buffer to be zeroed using sizeof , should not use the address of the structure rather the structure itself. DomainSocket.c line 156 Replace current: memset(addr,0,sizeof,(addr)); With: memset(addr, 0, sizeof(addr)); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11827) Speed-up distcp buildListing() using threadpool
[ https://issues.apache.org/jira/browse/HADOOP-11827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash updated HADOOP-11827: -- Resolution: Fixed Fix Version/s: 2.8.0 Status: Resolved (was: Patch Available) Thanks Zoran! I've committed this to trunk and branch-2. It should be released with 2.8.0 Speed-up distcp buildListing() using threadpool --- Key: HADOOP-11827 URL: https://issues.apache.org/jira/browse/HADOOP-11827 Project: Hadoop Common Issue Type: Improvement Components: tools/distcp Affects Versions: 2.7.0, 2.7.1 Reporter: Zoran Dimitrijevic Assignee: Zoran Dimitrijevic Fix For: 2.8.0 Attachments: HADOOP-11827-02.patch, HADOOP-11827-03.patch, HADOOP-11827-04.patch, HADOOP-11827.patch Original Estimate: 24h Remaining Estimate: 24h For very large source trees on s3 distcp is taking long time to build file listing (client code, before starting mappers). For a dataset I used (1.5M files, 50K dirs) it was taking 65 minutes before my fix in HADOOP-11785 and 36 minutes after the fix). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HADOOP-11829) Improve the vector size of Bloom Filter from int to long, and storage from memory to disk
[ https://issues.apache.org/jira/browse/HADOOP-11829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hongbo Xu resolved HADOOP-11829. Resolution: Invalid Improve the vector size of Bloom Filter from int to long, and storage from memory to disk - Key: HADOOP-11829 URL: https://issues.apache.org/jira/browse/HADOOP-11829 Project: Hadoop Common Issue Type: Improvement Components: util Reporter: Hongbo Xu Assignee: Hongbo Xu Priority: Minor Original Estimate: 168h Remaining Estimate: 168h org.apache.hadoop.util.bloom.BloomFilter(int vectorSize, int nbHash, int hashType) This filter almost can insert 900 million objects, when False Positives Probability is 0.0001, and it needs 2.1G RAM. In My project, I needs established a filter which capacity is 2 billion, and it needs 4.7G RAM, the vector size is 38340233509, out the range of int, and I does not have so much RAM to do this, so I rebuild a big bloom filter which vector size type is long, and split the bit data to some files on disk, then distribute files to work node, and the performance is very good. I think I can contribute this code to Hadoop Common, and a 128-bit Hash function (MurmurHash) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11862) Add support key share across KMS instances for KMS HA
[ https://issues.apache.org/jira/browse/HADOOP-11862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dengxiumao updated HADOOP-11862: Description: The patch [HADOOP-11620|https://issues.apache.org/jira/browse/HADOOP-11620] only supports specification of multiple hostnames in the kms key provider uri. it means that it support config as: bq. property namehadoop.security.key.provider.path/name valuekms://http@[HOSTNAME1];[HOSTNAME2]:16000/kms/value /property but HA is still not available, keys can not share across KMS instances, if one of KMS instances goes down, Encrypted files, which encrypted by the keys in the KMS, can not be read. Add support key share across KMS instances for KMS HA - Key: HADOOP-11862 URL: https://issues.apache.org/jira/browse/HADOOP-11862 Project: Hadoop Common Issue Type: Improvement Components: kms Affects Versions: 2.6.0 Reporter: dengxiumao Labels: kms, transparentenc The patch [HADOOP-11620|https://issues.apache.org/jira/browse/HADOOP-11620] only supports specification of multiple hostnames in the kms key provider uri. it means that it support config as: bq. property namehadoop.security.key.provider.path/name valuekms://http@[HOSTNAME1];[HOSTNAME2]:16000/kms/value /property but HA is still not available, keys can not share across KMS instances, if one of KMS instances goes down, Encrypted files, which encrypted by the keys in the KMS, can not be read. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11862) Add support key share across KMS instances for KMS HA
[
https://issues.apache.org/jira/browse/HADOOP-11862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
dengxiumao updated HADOOP-11862:
Labels: kms (was: kms transparentenc)
Add support key share across KMS instances for KMS HA
-
Key: HADOOP-11862
URL: https://issues.apache.org/jira/browse/HADOOP-11862
Project: Hadoop Common
Issue Type: Improvement
Components: kms
Affects Versions: 2.6.0
Reporter: dengxiumao
Labels: kms
The patch [HADOOP-11620|https://issues.apache.org/jira/browse/HADOOP-11620]
only supports specification of multiple hostnames in the kms key provider
uri. it means that it support config as:
{quote}
property
namehadoop.security.key.provider.path/name
valuekms://http@[HOSTNAME1];[HOSTNAME2]:16000/kms/value
/property
{quote}
but HA is still not available, keys can not share across KMS instances, if
one of KMS instances goes down, Encrypted files, which encrypted by the keys
in the KMS, can not be read.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-11862) Add support key share across KMS instances for KMS HA
dengxiumao created HADOOP-11862: --- Summary: Add support key share across KMS instances for KMS HA Key: HADOOP-11862 URL: https://issues.apache.org/jira/browse/HADOOP-11862 Project: Hadoop Common Issue Type: Improvement Components: kms Affects Versions: 2.6.0 Reporter: dengxiumao -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11862) Add support key share across KMS instances for KMS HA
[
https://issues.apache.org/jira/browse/HADOOP-11862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
dengxiumao updated HADOOP-11862:
External issue ID: 11620
External issue URL: https://issues.apache.org/jira/browse/HADOOP-11620
Add support key share across KMS instances for KMS HA
-
Key: HADOOP-11862
URL: https://issues.apache.org/jira/browse/HADOOP-11862
Project: Hadoop Common
Issue Type: Improvement
Components: kms
Affects Versions: 2.6.0
Reporter: dengxiumao
Labels: kms, transparentenc
The patch [HADOOP-11620|https://issues.apache.org/jira/browse/HADOOP-11620]
only supports specification of multiple hostnames in the kms key provider
uri. it means that it support config as:
{quote}
property
namehadoop.security.key.provider.path/name
valuekms://http@[HOSTNAME1];[HOSTNAME2]:16000/kms/value
/property
{quote}
but HA is still not available, keys can not share across KMS instances, if
one of KMS instances goes down, Encrypted files, which encrypted by the keys
in the KMS, can not be read.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11627) Remove io.native.lib.available from trunk
[
https://issues.apache.org/jira/browse/HADOOP-11627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinayakumar B updated HADOOP-11627:
---
Attachment: HADOOP-11627-009.patch
Attached the one-line updated patch
Remove io.native.lib.available from trunk
-
Key: HADOOP-11627
URL: https://issues.apache.org/jira/browse/HADOOP-11627
Project: Hadoop Common
Issue Type: Improvement
Affects Versions: 3.0.0
Reporter: Akira AJISAKA
Assignee: Brahma Reddy Battula
Attachments: HADOOP-11627-002.patch, HADOOP-11627-003.patch,
HADOOP-11627-004.patch, HADOOP-11627-005.patch, HADOOP-11627-006.patch,
HADOOP-11627-007.patch, HADOOP-11627-008.patch, HADOOP-11627-009.patch,
HADOOP-11627.patch
According to the discussion in HADOOP-8642, we should remove
{{io.native.lib.available}} from trunk, and always use native libraries if
they exist.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11858) [JDK8] Set minimum version of Hadoop 3 to JDK 8
[ https://issues.apache.org/jira/browse/HADOOP-11858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14506206#comment-14506206 ] Vinod Kumar Vavilapalli commented on HADOOP-11858: -- Actually I didn't see any conclusion on this on the previous thread on the dev lists. There were proposals and counter points, but we never made a decision. We should get consensus first, given this has implications on how branch-2 evolves. I propose we reignite this on the dev lists before moving ahead. [JDK8] Set minimum version of Hadoop 3 to JDK 8 --- Key: HADOOP-11858 URL: https://issues.apache.org/jira/browse/HADOOP-11858 Project: Hadoop Common Issue Type: Sub-task Components: build Affects Versions: 3.0.0 Reporter: Robert Kanter Assignee: Robert Kanter Attachments: HADOOP-11858.001.patch, HADOOP-11858.002.patch Set minimum version of trunk to JDK 8 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11857) Fix CommandFormat#commandFormat java doc annotation
[
https://issues.apache.org/jira/browse/HADOOP-11857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14506276#comment-14506276
]
J.Andreina commented on HADOOP-11857:
-
Thanks [~xyao] for raising this issue .Can i assign this issue to me,if you
have not started working on this.
Fix CommandFormat#commandFormat java doc annotation
---
Key: HADOOP-11857
URL: https://issues.apache.org/jira/browse/HADOOP-11857
Project: Hadoop Common
Issue Type: Improvement
Reporter: Xiaoyu Yao
Priority: Trivial
Labels: newbie
The first parameter name does not match with the javadoc.
{code}
/**
* @deprecated use replacement since name is an unused parameter
* @param name of command, but never used
* @param min see replacement
* @param max see replacement
* @param possibleOpt see replacement
* @see #CommandFormat(int, int, String...)
*/
@Deprecated
public CommandFormat(String n, int min, int max, String ... possibleOpt) {
this(min, max, possibleOpt);
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11862) Add support key share across KMS instances for KMS HA
[
https://issues.apache.org/jira/browse/HADOOP-11862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
dengxiumao updated HADOOP-11862:
Description:
The patch [HADOOP-11620|https://issues.apache.org/jira/browse/HADOOP-11620]
only supports specification of multiple hostnames in the kms key provider uri.
it means that it support config as:
{quote}
property
namehadoop.security.key.provider.path/name
valuekms://http@[HOSTNAME1];[HOSTNAME2]:16000/kms/value
/property
{quote}
but HA is still not available, keys can not share across KMS instances, if one
of KMS instances goes down, Encrypted files, which encrypted by the keys in the
KMS, can not be read.
was:
The patch [HADOOP-11620|https://issues.apache.org/jira/browse/HADOOP-11620]
only supports specification of multiple hostnames in the kms key provider uri.
it means that it support config as:
bq.
property
namehadoop.security.key.provider.path/name
valuekms://http@[HOSTNAME1];[HOSTNAME2]:16000/kms/value
/property
but HA is still not available, keys can not share across KMS instances, if one
of KMS instances goes down, Encrypted files, which encrypted by the keys in the
KMS, can not be read.
Add support key share across KMS instances for KMS HA
-
Key: HADOOP-11862
URL: https://issues.apache.org/jira/browse/HADOOP-11862
Project: Hadoop Common
Issue Type: Improvement
Components: kms
Affects Versions: 2.6.0
Reporter: dengxiumao
Labels: kms, transparentenc
The patch [HADOOP-11620|https://issues.apache.org/jira/browse/HADOOP-11620]
only supports specification of multiple hostnames in the kms key provider
uri. it means that it support config as:
{quote}
property
namehadoop.security.key.provider.path/name
valuekms://http@[HOSTNAME1];[HOSTNAME2]:16000/kms/value
/property
{quote}
but HA is still not available, keys can not share across KMS instances, if
one of KMS instances goes down, Encrypted files, which encrypted by the keys
in the KMS, can not be read.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11862) Add support key share across KMS instances for KMS HA
[
https://issues.apache.org/jira/browse/HADOOP-11862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
dengxiumao updated HADOOP-11862:
External issue ID: HADOOP-11620 (was: 11620)
Add support key share across KMS instances for KMS HA
-
Key: HADOOP-11862
URL: https://issues.apache.org/jira/browse/HADOOP-11862
Project: Hadoop Common
Issue Type: Improvement
Components: kms
Affects Versions: 2.6.0
Reporter: dengxiumao
Labels: kms, transparentenc
The patch [HADOOP-11620|https://issues.apache.org/jira/browse/HADOOP-11620]
only supports specification of multiple hostnames in the kms key provider
uri. it means that it support config as:
{quote}
property
namehadoop.security.key.provider.path/name
valuekms://http@[HOSTNAME1];[HOSTNAME2]:16000/kms/value
/property
{quote}
but HA is still not available, keys can not share across KMS instances, if
one of KMS instances goes down, Encrypted files, which encrypted by the keys
in the KMS, can not be read.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11848) Incorrect arguments to sizeof in DomainSocket.c
[ https://issues.apache.org/jira/browse/HADOOP-11848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14506480#comment-14506480 ] Chris Nauroth commented on HADOOP-11848: Hi [~malcolm kavalsky]. Is your remote git://git.apache.org/hadoop-common.git? If so, then I noticed the same thing earlier today. We probably need to follow up on that with Apache Infrastructure. Meanwhile though, I think you can make progress by using a different remote: https://git-wip-us.apache.org/repos/asf/hadoop.git. This is the remote where we actually push committed patches. git.apache.org is a mirror of that, and then the github copy is a mirror of git.apache.org. Incorrect arguments to sizeof in DomainSocket.c --- Key: HADOOP-11848 URL: https://issues.apache.org/jira/browse/HADOOP-11848 Project: Hadoop Common Issue Type: Bug Components: native Affects Versions: 2.6.0 Reporter: Malcolm Kavalsky Assignee: Malcolm Kavalsky Original Estimate: 24h Remaining Estimate: 24h Length of buffer to be zeroed using sizeof , should not use the address of the structure rather the structure itself. DomainSocket.c line 156 Replace current: memset(addr,0,sizeof,(addr)); With: memset(addr, 0, sizeof(addr)); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11857) Fix CommandFormat#commandFormat java doc annotation
[
https://issues.apache.org/jira/browse/HADOOP-11857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14506399#comment-14506399
]
Xiaoyu Yao commented on HADOOP-11857:
-
[~andreina], I assigned the issue to you. Feel free to post your patch for fix.
Fix CommandFormat#commandFormat java doc annotation
---
Key: HADOOP-11857
URL: https://issues.apache.org/jira/browse/HADOOP-11857
Project: Hadoop Common
Issue Type: Improvement
Reporter: Xiaoyu Yao
Assignee: J.Andreina
Priority: Trivial
Labels: newbie
The first parameter name does not match with the javadoc.
{code}
/**
* @deprecated use replacement since name is an unused parameter
* @param name of command, but never used
* @param min see replacement
* @param max see replacement
* @param possibleOpt see replacement
* @see #CommandFormat(int, int, String...)
*/
@Deprecated
public CommandFormat(String n, int min, int max, String ... possibleOpt) {
this(min, max, possibleOpt);
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11857) Fix CommandFormat#commandFormat java doc annotation
[
https://issues.apache.org/jira/browse/HADOOP-11857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao updated HADOOP-11857:
Assignee: J.Andreina
Fix CommandFormat#commandFormat java doc annotation
---
Key: HADOOP-11857
URL: https://issues.apache.org/jira/browse/HADOOP-11857
Project: Hadoop Common
Issue Type: Improvement
Reporter: Xiaoyu Yao
Assignee: J.Andreina
Priority: Trivial
Labels: newbie
The first parameter name does not match with the javadoc.
{code}
/**
* @deprecated use replacement since name is an unused parameter
* @param name of command, but never used
* @param min see replacement
* @param max see replacement
* @param possibleOpt see replacement
* @see #CommandFormat(int, int, String...)
*/
@Deprecated
public CommandFormat(String n, int min, int max, String ... possibleOpt) {
this(min, max, possibleOpt);
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11848) Incorrect arguments to sizeof in DomainSocket.c
[ https://issues.apache.org/jira/browse/HADOOP-11848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14506452#comment-14506452 ] Malcolm Kavalsky commented on HADOOP-11848: --- I wanted to send in a patch for this yesterday, but am having trouble with git : $ git remote update Fetching origin fatal: read error: Connection reset by peer error: Could not fetch origin So, perhaps you could patch this ? Incorrect arguments to sizeof in DomainSocket.c --- Key: HADOOP-11848 URL: https://issues.apache.org/jira/browse/HADOOP-11848 Project: Hadoop Common Issue Type: Bug Components: native Affects Versions: 2.6.0 Reporter: Malcolm Kavalsky Assignee: Malcolm Kavalsky Original Estimate: 24h Remaining Estimate: 24h Length of buffer to be zeroed using sizeof , should not use the address of the structure rather the structure itself. DomainSocket.c line 156 Replace current: memset(addr,0,sizeof,(addr)); With: memset(addr, 0, sizeof(addr)); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11847) Enhance raw coder allowing to read least required inputs in decoding
[
https://issues.apache.org/jira/browse/HADOOP-11847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505944#comment-14505944
]
Kai Zheng commented on HADOOP-11847:
HADOOP-11860 was opened to follow on and improve accordingly in the
{{ErasureCoder}} layer.
Enhance raw coder allowing to read least required inputs in decoding
Key: HADOOP-11847
URL: https://issues.apache.org/jira/browse/HADOOP-11847
Project: Hadoop Common
Issue Type: Sub-task
Components: io
Reporter: Kai Zheng
Assignee: Kai Zheng
Attachments: HADOOP-11847-v1.patch
This is to enhance raw erasure coder to allow only reading least required
inputs while decoding. It will also refine and document the relevant APIs for
better understanding and usage. When using least required inputs, it may add
computating overhead but will possiblly outperform overall since less network
traffic and disk IO are involved.
This is something planned to do but just got reminded by [~zhz]' s question
raised in HDFS-7678, also copied here:
bq.Kai Zheng I have a question about decoding: in a (6+3) schema, if block #2
is missing, and I want to repair it with blocks 0, 1, 3, 4, 5, 8, how should
I construct the inputs to RawErasureDecoder#decode?
With this work, hopefully the answer to above question would be obvious.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11861) test-patch script always fails when option --build-native=false is specified
[
https://issues.apache.org/jira/browse/HADOOP-11861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505973#comment-14505973
]
Chris Nauroth commented on HADOOP-11861:
On this line, {{NATIVE_PROFILE}} would be undefined.
{code}
echo_and_redirect ${PATCH_DIR}/patchJavacWarnings.txt ${MVN} clean test
-DskipTests -D${PROJECT_NAME}PatchProcess ${NATIVE_PROFILE} -Ptest-patch
{code}
It seems like we end up passing an extra empty string argument to the mvn call,
and mvn interprets that as an extra goal to run, which of course doesn't exist.
{code}
/opt/local/share/java/maven3/bin/mvn clean test -DskipTests
-DhadoopPatchProcess -Ptest-patch
/tmp/hadoop-test-patch/2621/patchJavacWarnings.txt 21
{code}
test-patch script always fails when option --build-native=false is
specified
---
Key: HADOOP-11861
URL: https://issues.apache.org/jira/browse/HADOOP-11861
Project: Hadoop Common
Issue Type: Bug
Components: build
Affects Versions: 2.8.0
Reporter: Anu Engineer
if you specify --build-native=false like
{code}
./dev-support/test-patch.sh --build-native=false
~/workspaces/patches/hdfs-8211.001.patch
{code}
mvn fails with invalid lifecycle error.
Here are the steps to repro :
1) run any patch with --buid-native=false option
2) Open up /tmp/hadoop-test-patch/tmp-patch/patchJavacWarnings.txt to see
the failure reason.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11846) TestCertificateUtil.testCorruptPEM failing on Jenkins JDK8
[
https://issues.apache.org/jira/browse/HADOOP-11846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505977#comment-14505977
]
Larry McCay commented on HADOOP-11846:
--
Thanks, Steve!
TestCertificateUtil.testCorruptPEM failing on Jenkins JDK8
--
Key: HADOOP-11846
URL: https://issues.apache.org/jira/browse/HADOOP-11846
Project: Hadoop Common
Issue Type: Bug
Components: build, security
Affects Versions: 3.0.0
Environment: Java 8
Reporter: Steve Loughran
Assignee: Larry McCay
Fix For: 2.8.0
Attachments: HADOOP-11846-1.patch
Jenkins is failing on {{TestCertificateUtil.testCorruptPEM}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11858) [JDK8] Set minimum version of Hadoop 3 to JDK 8
[ https://issues.apache.org/jira/browse/HADOOP-11858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14506092#comment-14506092 ] Andrew Wang commented on HADOOP-11858: -- One comment, we should update the BUILDING.txt instructions too. Otherwise +1. We of course also need to do the related Jenkins-side work, which is where the real fun lies. [JDK8] Set minimum version of Hadoop 3 to JDK 8 --- Key: HADOOP-11858 URL: https://issues.apache.org/jira/browse/HADOOP-11858 Project: Hadoop Common Issue Type: Sub-task Components: build Affects Versions: 3.0.0 Reporter: Robert Kanter Assignee: Robert Kanter Attachments: HADOOP-11858.001.patch Set minimum version of trunk to JDK 8 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11847) Enhance raw coder allowing to read least required inputs in decoding
[
https://issues.apache.org/jira/browse/HADOOP-11847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14506111#comment-14506111
]
Zhe Zhang commented on HADOOP-11847:
Thanks Kai for the patch. Please find my review below:
# We try to decode all null slots in the input arrays. I'm not sure if this
will cause unnecessary computation.
# Could you explain this change? Shouldn't the first argument be
{{numDataUnits}}?
{code}
- xorRawDecoder.initialize(getNumDataUnits(), 1, getChunkSize());
+ xorRawDecoder.initialize(getNumDataUnits() + getNumParityUnits() - 1,
+ 1, getChunkSize());
{code}
# {{checkParameters}} goes through the input arrays once, and the {{badCount}}
makes another pass. Can we just assert {{badCount + erasedIndexes.length ==
numDataUnits}}?
# {{ensureWhenUseXXX}} needs some Javadoc. Maybe also add a better explanation
than {{// Lazy on demand}}?
# These variable names look confusing: {{decodingDirectBufferInputs}} vs.
{{decodingDirectBuffersForInput}}, and {{decodingDirectBufferOutputs}} vs.
{{decodingDirectBuffersForOutput}}
# Is {{decodingByteArrayBuffersForInput}} always filled with zero bytes? I
don't see where it's filled with actual data
Enhance raw coder allowing to read least required inputs in decoding
Key: HADOOP-11847
URL: https://issues.apache.org/jira/browse/HADOOP-11847
Project: Hadoop Common
Issue Type: Sub-task
Components: io
Reporter: Kai Zheng
Assignee: Kai Zheng
Attachments: HADOOP-11847-v1.patch
This is to enhance raw erasure coder to allow only reading least required
inputs while decoding. It will also refine and document the relevant APIs for
better understanding and usage. When using least required inputs, it may add
computating overhead but will possiblly outperform overall since less network
traffic and disk IO are involved.
This is something planned to do but just got reminded by [~zhz]' s question
raised in HDFS-7678, also copied here:
bq.Kai Zheng I have a question about decoding: in a (6+3) schema, if block #2
is missing, and I want to repair it with blocks 0, 1, 3, 4, 5, 8, how should
I construct the inputs to RawErasureDecoder#decode?
With this work, hopefully the answer to above question would be obvious.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11859) PseudoAuthenticationHandler fails with httpcomponents v4.4
[
https://issues.apache.org/jira/browse/HADOOP-11859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505993#comment-14505993
]
Hadoop QA commented on HADOOP-11859:
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 26s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | javac | 7m 25s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 31s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 5m 26s | There were no new checkstyle
issues. |
| {color:green}+1{color} | install | 1m 31s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 0m 36s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | common tests | 5m 18s | Tests passed in
hadoop-auth. |
| | | 45m 11s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12727011/HADOOP-11859.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 2c14690 |
| hadoop-auth test log |
https://builds.apache.org/job/PreCommit-HADOOP-Build/6143/artifact/patchprocess/testrun_hadoop-auth.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/6143/testReport/ |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/6143//console |
This message was automatically generated.
PseudoAuthenticationHandler fails with httpcomponents v4.4
--
Key: HADOOP-11859
URL: https://issues.apache.org/jira/browse/HADOOP-11859
Project: Hadoop Common
Issue Type: Bug
Reporter: Eugene Koifman
Assignee: Eugene Koifman
Attachments: HADOOP-11859.patch
This shows in the context of WebHCat and Hive (which recently moved to
httpcomponents:httpclient:4.4) but could happen in other places.
URLEncodedUtils.parse(String, Charset) which is called from
PseudoAuthenticationHandler.getUserName() with the 1st argument produced by
HttpServletRequest.getQueryString().
The later returns NULL if there is no query string in the URL.
in httpcoponents:httpclient:4.2.5 parse() gracefully handles first argument
being NULL, but in 4.4 it NPEs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11858) [JDK8] Set minimum version of Hadoop 3 to JDK 8
[ https://issues.apache.org/jira/browse/HADOOP-11858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated HADOOP-11858: --- Attachment: HADOOP-11858.002.patch 002 patch also updates BUILDING.txt [JDK8] Set minimum version of Hadoop 3 to JDK 8 --- Key: HADOOP-11858 URL: https://issues.apache.org/jira/browse/HADOOP-11858 Project: Hadoop Common Issue Type: Sub-task Components: build Affects Versions: 3.0.0 Reporter: Robert Kanter Assignee: Robert Kanter Attachments: HADOOP-11858.001.patch, HADOOP-11858.002.patch Set minimum version of trunk to JDK 8 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-11861) test-patch script always fails when option --build-native=false is specified
Anu Engineer created HADOOP-11861:
-
Summary: test-patch script always fails when option
--build-native=false is specified
Key: HADOOP-11861
URL: https://issues.apache.org/jira/browse/HADOOP-11861
Project: Hadoop Common
Issue Type: Bug
Components: build
Affects Versions: 2.8.0
Reporter: Anu Engineer
if you specify --build-native=false like
{code}
./dev-support/test-patch.sh --build-native=false
~/workspaces/patches/hdfs-8211.001.patch
{code}
mvn fails with invalid lifecycle error.
Here are the steps to repro :
1) run any patch with --buid-native=false option
2) Open up /tmp/hadoop-test-patch/tmp-patch/patchJavacWarnings.txt to see
the failure reason.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11827) Speed-up distcp buildListing() using threadpool
[ https://issues.apache.org/jira/browse/HADOOP-11827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14506048#comment-14506048 ] Hudson commented on HADOOP-11827: - FAILURE: Integrated in Hadoop-trunk-Commit #7630 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7630/]) HADOOP-11827. Speed-up distcp buildListing() using threadpool (Zoran Dimitrijevic via raviprak) (raviprak: rev cfba355052df15f8eb6cc9b8e90e2d8492bec7d7) * hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpConstants.java * hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestCopyListing.java * hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestIntegration.java * hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/WorkRequestProcessor.java * hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/WorkReport.java * hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyListing.java * hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/OptionsParser.java * hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/WorkRequest.java * hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/util/TestProducerConsumer.java * hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptionSwitch.java * hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/ProducerConsumer.java * hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java Speed-up distcp buildListing() using threadpool --- Key: HADOOP-11827 URL: https://issues.apache.org/jira/browse/HADOOP-11827 Project: Hadoop Common Issue Type: Improvement Components: tools/distcp Affects Versions: 2.7.0, 2.7.1 Reporter: Zoran Dimitrijevic Assignee: Zoran Dimitrijevic Fix For: 2.8.0 Attachments: HADOOP-11827-02.patch, HADOOP-11827-03.patch, HADOOP-11827-04.patch, HADOOP-11827.patch Original Estimate: 24h Remaining Estimate: 24h For very large source trees on s3 distcp is taking long time to build file listing (client code, before starting mappers). For a dataset I used (1.5M files, 50K dirs) it was taking 65 minutes before my fix in HADOOP-11785 and 36 minutes after the fix). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11843) Make setting up the build environment easier
[
https://issues.apache.org/jira/browse/HADOOP-11843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505280#comment-14505280
]
Arpit Agarwal commented on HADOOP-11843:
This is a great idea. Got the container started on Mac with the following steps.
{code}
$ brew install docker boot2docker
$ boot2docker init -m 4096
$ boot2docker start
$ $(boot2docker shellinit)
$ ./start-build-env.sh
{code}
As Chris already pointed out the performance with boot2docker makes it
impractical. For now we can go with Neal's recommendation of a linux VM. I
often build in Centos/Ubuntu VMs on OS X and the performance is not
significantly worse than building natively.
Make setting up the build environment easier
Key: HADOOP-11843
URL: https://issues.apache.org/jira/browse/HADOOP-11843
Project: Hadoop Common
Issue Type: New Feature
Reporter: Niels Basjes
Assignee: Niels Basjes
Attachments: HADOOP-11843-2015-04-17-1612.patch,
HADOOP-11843-2015-04-17-2226.patch, HADOOP-11843-2015-04-17-2308.patch,
HADOOP-11843-2015-04-19-2206.patch, HADOOP-11843-2015-04-19-2232.patch
( As discussed with [~aw] )
In AVRO-1537 a docker based solution was created to setup all the tools for
doing a full build. This enables much easier reproduction of any issues and
getting up and running for new developers.
This issue is to 'copy/port' that setup into the hadoop project in
preparation for the bug squash.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HADOOP-11843) Make setting up the build environment easier
[
https://issues.apache.org/jira/browse/HADOOP-11843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505280#comment-14505280
]
Arpit Agarwal edited comment on HADOOP-11843 at 4/21/15 5:08 PM:
-
This is a great idea. Got the container started on Mac with the following steps.
{code}
$ brew install docker boot2docker
$ boot2docker init -m 4096
$ boot2docker start
$ $(boot2docker shellinit)
$ ./start-build-env.sh
{code}
As Chris already pointed out the performance with boot2docker makes it
impractical. For now we can go with Niels's recommendation of a linux VM. I
often build in Centos/Ubuntu VMs on OS X and the performance is not
significantly worse than building natively.
was (Author: arpitagarwal):
This is a great idea. Got the container started on Mac with the following steps.
{code}
$ brew install docker boot2docker
$ boot2docker init -m 4096
$ boot2docker start
$ $(boot2docker shellinit)
$ ./start-build-env.sh
{code}
As Chris already pointed out the performance with boot2docker makes it
impractical. For now we can go with Neal's recommendation of a linux VM. I
often build in Centos/Ubuntu VMs on OS X and the performance is not
significantly worse than building natively.
Make setting up the build environment easier
Key: HADOOP-11843
URL: https://issues.apache.org/jira/browse/HADOOP-11843
Project: Hadoop Common
Issue Type: New Feature
Reporter: Niels Basjes
Assignee: Niels Basjes
Attachments: HADOOP-11843-2015-04-17-1612.patch,
HADOOP-11843-2015-04-17-2226.patch, HADOOP-11843-2015-04-17-2308.patch,
HADOOP-11843-2015-04-19-2206.patch, HADOOP-11843-2015-04-19-2232.patch
( As discussed with [~aw] )
In AVRO-1537 a docker based solution was created to setup all the tools for
doing a full build. This enables much easier reproduction of any issues and
getting up and running for new developers.
This issue is to 'copy/port' that setup into the hadoop project in
preparation for the bug squash.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
