[jira] [Commented] (HADOOP-8397) NPE thrown when IPC layer gets an EOF reading a response
[
https://issues.apache.org/jira/browse/HADOOP-8397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13274110#comment-13274110
]
Todd Lipcon commented on HADOOP-8397:
-
I think we just need to check for a null result, and throw EOFException in that
case.
> NPE thrown when IPC layer gets an EOF reading a response
>
>
> Key: HADOOP-8397
> URL: https://issues.apache.org/jira/browse/HADOOP-8397
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 2.0.0
>Reporter: Todd Lipcon
>Priority: Critical
>
> When making a call on an IPC connection where the other end has shut down, I
> see the following exception:
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:852)
> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:781)
> from the lines:
> {code}
> RpcResponseHeaderProto response =
> RpcResponseHeaderProto.parseDelimitedFrom(in);
> int callId = response.getCallId();
> {code}
> This is because parseDelimitedFrom() returns null in the case that the next
> thing to be read on the stream is an EOF.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8397) NPE thrown when IPC layer gets an EOF reading a response
Todd Lipcon created HADOOP-8397:
---
Summary: NPE thrown when IPC layer gets an EOF reading a response
Key: HADOOP-8397
URL: https://issues.apache.org/jira/browse/HADOOP-8397
Project: Hadoop Common
Issue Type: Bug
Components: ipc
Affects Versions: 2.0.0
Reporter: Todd Lipcon
Priority: Critical
When making a call on an IPC connection where the other end has shut down, I
see the following exception:
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:852)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:781)
from the lines:
{code}
RpcResponseHeaderProto response =
RpcResponseHeaderProto.parseDelimitedFrom(in);
int callId = response.getCallId();
{code}
This is because parseDelimitedFrom() returns null in the case that the next
thing to be read on the stream is an EOF.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8276) Auto-HA: add config for java options to pass to zkfc daemon
[ https://issues.apache.org/jira/browse/HADOOP-8276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon resolved HADOOP-8276. - Resolution: Fixed Fix Version/s: Auto Failover (HDFS-3042) Hadoop Flags: Reviewed > Auto-HA: add config for java options to pass to zkfc daemon > --- > > Key: HADOOP-8276 > URL: https://issues.apache.org/jira/browse/HADOOP-8276 > Project: Hadoop Common > Issue Type: Improvement > Components: scripts >Affects Versions: Auto Failover (HDFS-3042) >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Minor > Fix For: Auto Failover (HDFS-3042) > > Attachments: hadoop-8276.txt > > > Currently the zkfc daemon is started without any ability to specify java > options for it. We should be add a flag so heap size, etc can be specified. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8395) Text shell command unnecessarily demands that a SequenceFile's key class be WritableComparable
[ https://issues.apache.org/jira/browse/HADOOP-8395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13274011#comment-13274011 ] Hudson commented on HADOOP-8395: Integrated in Hadoop-Mapreduce-trunk #1077 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1077/]) HADOOP-8395. Text shell command unnecessarily demands that a SequenceFile's key class be WritableComparable (harsh) (Revision 1337449) Result = SUCCESS harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337449 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Display.java > Text shell command unnecessarily demands that a SequenceFile's key class be > WritableComparable > -- > > Key: HADOOP-8395 > URL: https://issues.apache.org/jira/browse/HADOOP-8395 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 2.0.0 >Reporter: Harsh J >Assignee: Harsh J >Priority: Trivial > Labels: shell > Fix For: 3.0.0 > > Attachments: HADOOP-8395.patch > > > Text from Display set of Shell commands (hadoop fs -text), has a strict > subclass check for a sequence-file-header loaded key class to be a subclass > of WritableComparable. > The sequence file writer itself has no such checks (one can create sequence > files with just plain writable keys, comparable is needed for sequence file's > sorter alone, which not all of them use always), and hence its not reasonable > for Text command to carry it either. > We should relax the check and simply just check for "Writable", not > "WritableComparable". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8113) Correction to BUILDING.txt: HDFS needs ProtocolBuffer, too (not just MapReduce)
[
https://issues.apache.org/jira/browse/HADOOP-8113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13274009#comment-13274009
]
Hudson commented on HADOOP-8113:
Integrated in Hadoop-Mapreduce-trunk #1077 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1077/])
Add HADOOP-8113 to CHANGES.txt (Revision 1337415)
Result = SUCCESS
szetszwo :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337415
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
> Correction to BUILDING.txt: HDFS needs ProtocolBuffer, too (not just
> MapReduce)
> ---
>
> Key: HADOOP-8113
> URL: https://issues.apache.org/jira/browse/HADOOP-8113
> Project: Hadoop Common
> Issue Type: Improvement
> Components: documentation
>Affects Versions: 0.23.2
>Reporter: Eugene Koontz
>Assignee: Eugene Koontz
>Priority: Trivial
> Fix For: 2.0.0
>
> Attachments: HADOOP-8113.patch
>
>
> Currently BUILDING.txt states:
> {quote}
> ProtocolBuffer 2.4.1+ (for MapReduce)
> {quote}
> But HDFS needs ProtocolBuffer too:
> {code}
> hadoop-common/hadoop-hdfs-project$ find . -name "*.proto" | wc -l
> 11
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8366) Use ProtoBuf for RpcResponseHeader
[ https://issues.apache.org/jira/browse/HADOOP-8366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13274006#comment-13274006 ] Hudson commented on HADOOP-8366: Integrated in Hadoop-Mapreduce-trunk #1077 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1077/]) Move HADOOP-8285 and HADOOP-8366 to 2.0.0 in CHANGES.txt. (Revision 1337431) HADOOP-8366 Use ProtoBuf for RpcResponseHeader (sanjay radia) (Revision 1337283) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337431 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt sradia : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337283 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Status.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/proto/RpcPayloadHeader.proto > Use ProtoBuf for RpcResponseHeader > -- > > Key: HADOOP-8366 > URL: https://issues.apache.org/jira/browse/HADOOP-8366 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Sanjay Radia >Assignee: Sanjay Radia >Priority: Blocker > Fix For: 2.0.0 > > Attachments: hadoop-8366-1.patch, hadoop-8366-2.patch, > hadoop-8366-3.patch, hadoop-8366-4.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8285) Use ProtoBuf for RpcPayLoadHeader
[ https://issues.apache.org/jira/browse/HADOOP-8285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13274007#comment-13274007 ] Hudson commented on HADOOP-8285: Integrated in Hadoop-Mapreduce-trunk #1077 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1077/]) Move HADOOP-8285 and HADOOP-8366 to 2.0.0 in CHANGES.txt. (Revision 1337431) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337431 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt > Use ProtoBuf for RpcPayLoadHeader > - > > Key: HADOOP-8285 > URL: https://issues.apache.org/jira/browse/HADOOP-8285 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Fix For: 2.0.0 > > Attachments: hadoop-8285-1-common.patch, hadoop-8285-1.patch, > hadoop-8285-2-common.patch, hadoop-8285-2.patch, hadoop-8285-3-common.patch, > hadoop-8285-3.patch, hadoop-8285-4-common.patch, hadoop-8285-4.patch, > hadoop-8285-5-common.patch, hadoop-8285-5.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8224) Don't hardcode hdfs.audit.logger in the scripts
[ https://issues.apache.org/jira/browse/HADOOP-8224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13274008#comment-13274008 ] Hudson commented on HADOOP-8224: Integrated in Hadoop-Mapreduce-trunk #1077 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1077/]) HADOOP-8224. Don't hardcode hdfs.audit.logger in the scripts. Contributed by Tomohiko Kinebuchi (Revision 1337339) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337339 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemon.sh * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh > Don't hardcode hdfs.audit.logger in the scripts > --- > > Key: HADOOP-8224 > URL: https://issues.apache.org/jira/browse/HADOOP-8224 > Project: Hadoop Common > Issue Type: Improvement > Components: conf >Affects Versions: 2.0.0 >Reporter: Eli Collins >Assignee: Tomohiko Kinebuchi > Fix For: 2.0.0 > > Attachments: HADOOP-8224.txt, HADOOP-8224.txt, hadoop-8224.txt > > > The HADOOP_*OPTS defined for HDFS in hadoop-env.sh hard-code the > hdfs.audit.logger (is explicitly set via "-Dhdfs.audit.logger=INFO,RFAAUDIT") > so it's not overridable. Let's allow someone to override it as we do the > other parameters by introducing HADOOP_AUDIT_LOGGER. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8316) Audit logging should be disabled by default
[ https://issues.apache.org/jira/browse/HADOOP-8316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13274005#comment-13274005 ] Hudson commented on HADOOP-8316: Integrated in Hadoop-Mapreduce-trunk #1077 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1077/]) HADOOP-8316. Audit logging should be disabled by default. Contributed by Eli Collins (Revision 1337334) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337334 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/log4j.properties > Audit logging should be disabled by default > --- > > Key: HADOOP-8316 > URL: https://issues.apache.org/jira/browse/HADOOP-8316 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Affects Versions: 2.0.0 >Reporter: Eli Collins >Assignee: Eli Collins > Fix For: 2.0.0 > > Attachments: hadoop-8316.txt > > > HADOOP-7633 made hdfs, mr and security audit logging on by default (INFO > level) in log4j.properties used for the packages, this then got copied over > to the non-packaging log4j.properties in HADOOP-8216 (which made them > consistent). > Seems like we should keep with the v1.x setting which is disabled (WARNING > level) by default. There's a performance overhead to audit logging, and > HADOOP-7633 provided not rationale (just "We should add the audit logs as > part of default confs") as to why they were enabled for the packages. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8353) hadoop-daemon.sh and yarn-daemon.sh can be misleading on stop
[ https://issues.apache.org/jira/browse/HADOOP-8353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13274004#comment-13274004 ] Hudson commented on HADOOP-8353: Integrated in Hadoop-Mapreduce-trunk #1077 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1077/]) HADOOP-8353. hadoop-daemon.sh and yarn-daemon.sh can be misleading on stop. Contributed by Roman Shaposhnik. (Revision 1337251) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337251 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemon.sh * /hadoop/common/trunk/hadoop-mapreduce-project/bin/mr-jobhistory-daemon.sh * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh > hadoop-daemon.sh and yarn-daemon.sh can be misleading on stop > - > > Key: HADOOP-8353 > URL: https://issues.apache.org/jira/browse/HADOOP-8353 > Project: Hadoop Common > Issue Type: Improvement > Components: scripts >Affects Versions: 0.23.1 >Reporter: Roman Shaposhnik >Assignee: Roman Shaposhnik > Fix For: 2.0.0 > > Attachments: HADOOP-8353-2.patch.txt, HADOOP-8353.patch.txt > > > The way that stop actions is implemented is a simple SIGTERM sent to the JVM. > There's a time delay between when the action is called and when the process > actually exists. This can be misleading to the callers of the *-daemon.sh > scripts since they expect stop action to return when process is actually > stopped. > I suggest we augment the stop action with a time-delay check for the process > status and a SIGKILL once the delay has expired. > I understand that sending SIGKILL is a measure of last resort and is > generally frowned upon among init.d script writers, but the excuse we have > for Hadoop is that it is engineered to be a fault tolerant system and thus > there's not danger of putting system into an incontinent state by a violent > SIGKILL. Of course, the time delay will be long enough to make SIGKILL event > a rare condition. > Finally, there's always an option of an exponential back-off type of solution > if we decide that SIGKILL timeout is short. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6801) io.sort.mb and io.sort.factor were renamed and moved to mapreduce but are still in CommonConfigurationKeysPublic.java and used in SequenceFile.java
[ https://issues.apache.org/jira/browse/HADOOP-6801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13273999#comment-13273999 ] Hadoop QA commented on HADOOP-6801: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12526633/HADOOP-6801.r5.diff against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.fs.viewfs.TestViewFsTrash +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/991//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/991//console This message is automatically generated. > io.sort.mb and io.sort.factor were renamed and moved to mapreduce but are > still in CommonConfigurationKeysPublic.java and used in SequenceFile.java > --- > > Key: HADOOP-6801 > URL: https://issues.apache.org/jira/browse/HADOOP-6801 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 0.22.0 >Reporter: Erik Steffl >Assignee: Harsh J >Priority: Minor > Attachments: HADOOP-6801.r1.diff, HADOOP-6801.r2.diff, > HADOOP-6801.r3.diff, HADOOP-6801.r4.diff, HADOOP-6801.r5.diff > > > Following configuration keys in CommonConfigurationKeysPublic.java (former > CommonConfigurationKeys.java): > public static final String IO_SORT_MB_KEY = "io.sort.mb"; > public static final String IO_SORT_FACTOR_KEY = "io.sort.factor"; > are partially moved: > - they were renamed to mapreduce.task.io.sort.mb and > mapreduce.task.io.sort.factor respectively > - they were moved to mapreduce project, documented in mapred-default.xml > However: > - they are still listed in CommonConfigurationKeysPublic.java as quoted > above > - strings "io.sort.mb" and "io.sort.factor" are used in SequenceFile.java > in Hadoop Common project > Not sure what the solution is, these constants should probably be removed > from CommonConfigurationKeysPublic.java but I am not sure what's the best > solution for SequenceFile.java. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6801) io.sort.mb and io.sort.factor were renamed and moved to mapreduce but are still in CommonConfigurationKeysPublic.java and used in SequenceFile.java
[ https://issues.apache.org/jira/browse/HADOOP-6801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13273992#comment-13273992 ] Hadoop QA commented on HADOOP-6801: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12526631/HADOOP-6801.r4.diff against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 javadoc. The javadoc tool appears to have generated 8 warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.fs.viewfs.TestViewFsTrash +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/990//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/990//console This message is automatically generated. > io.sort.mb and io.sort.factor were renamed and moved to mapreduce but are > still in CommonConfigurationKeysPublic.java and used in SequenceFile.java > --- > > Key: HADOOP-6801 > URL: https://issues.apache.org/jira/browse/HADOOP-6801 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 0.22.0 >Reporter: Erik Steffl >Assignee: Harsh J >Priority: Minor > Attachments: HADOOP-6801.r1.diff, HADOOP-6801.r2.diff, > HADOOP-6801.r3.diff, HADOOP-6801.r4.diff, HADOOP-6801.r5.diff > > > Following configuration keys in CommonConfigurationKeysPublic.java (former > CommonConfigurationKeys.java): > public static final String IO_SORT_MB_KEY = "io.sort.mb"; > public static final String IO_SORT_FACTOR_KEY = "io.sort.factor"; > are partially moved: > - they were renamed to mapreduce.task.io.sort.mb and > mapreduce.task.io.sort.factor respectively > - they were moved to mapreduce project, documented in mapred-default.xml > However: > - they are still listed in CommonConfigurationKeysPublic.java as quoted > above > - strings "io.sort.mb" and "io.sort.factor" are used in SequenceFile.java > in Hadoop Common project > Not sure what the solution is, these constants should probably be removed > from CommonConfigurationKeysPublic.java but I am not sure what's the best > solution for SequenceFile.java. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8395) Text shell command unnecessarily demands that a SequenceFile's key class be WritableComparable
[ https://issues.apache.org/jira/browse/HADOOP-8395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13273989#comment-13273989 ] Hudson commented on HADOOP-8395: Integrated in Hadoop-Hdfs-trunk #1041 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1041/]) HADOOP-8395. Text shell command unnecessarily demands that a SequenceFile's key class be WritableComparable (harsh) (Revision 1337449) Result = FAILURE harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337449 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Display.java > Text shell command unnecessarily demands that a SequenceFile's key class be > WritableComparable > -- > > Key: HADOOP-8395 > URL: https://issues.apache.org/jira/browse/HADOOP-8395 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 2.0.0 >Reporter: Harsh J >Assignee: Harsh J >Priority: Trivial > Labels: shell > Fix For: 3.0.0 > > Attachments: HADOOP-8395.patch > > > Text from Display set of Shell commands (hadoop fs -text), has a strict > subclass check for a sequence-file-header loaded key class to be a subclass > of WritableComparable. > The sequence file writer itself has no such checks (one can create sequence > files with just plain writable keys, comparable is needed for sequence file's > sorter alone, which not all of them use always), and hence its not reasonable > for Text command to carry it either. > We should relax the check and simply just check for "Writable", not > "WritableComparable". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8113) Correction to BUILDING.txt: HDFS needs ProtocolBuffer, too (not just MapReduce)
[
https://issues.apache.org/jira/browse/HADOOP-8113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13273987#comment-13273987
]
Hudson commented on HADOOP-8113:
Integrated in Hadoop-Hdfs-trunk #1041 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1041/])
Add HADOOP-8113 to CHANGES.txt (Revision 1337415)
Result = FAILURE
szetszwo :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337415
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
> Correction to BUILDING.txt: HDFS needs ProtocolBuffer, too (not just
> MapReduce)
> ---
>
> Key: HADOOP-8113
> URL: https://issues.apache.org/jira/browse/HADOOP-8113
> Project: Hadoop Common
> Issue Type: Improvement
> Components: documentation
>Affects Versions: 0.23.2
>Reporter: Eugene Koontz
>Assignee: Eugene Koontz
>Priority: Trivial
> Fix For: 2.0.0
>
> Attachments: HADOOP-8113.patch
>
>
> Currently BUILDING.txt states:
> {quote}
> ProtocolBuffer 2.4.1+ (for MapReduce)
> {quote}
> But HDFS needs ProtocolBuffer too:
> {code}
> hadoop-common/hadoop-hdfs-project$ find . -name "*.proto" | wc -l
> 11
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8285) Use ProtoBuf for RpcPayLoadHeader
[ https://issues.apache.org/jira/browse/HADOOP-8285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13273985#comment-13273985 ] Hudson commented on HADOOP-8285: Integrated in Hadoop-Hdfs-trunk #1041 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1041/]) Move HADOOP-8285 and HADOOP-8366 to 2.0.0 in CHANGES.txt. (Revision 1337431) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337431 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt > Use ProtoBuf for RpcPayLoadHeader > - > > Key: HADOOP-8285 > URL: https://issues.apache.org/jira/browse/HADOOP-8285 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Fix For: 2.0.0 > > Attachments: hadoop-8285-1-common.patch, hadoop-8285-1.patch, > hadoop-8285-2-common.patch, hadoop-8285-2.patch, hadoop-8285-3-common.patch, > hadoop-8285-3.patch, hadoop-8285-4-common.patch, hadoop-8285-4.patch, > hadoop-8285-5-common.patch, hadoop-8285-5.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8224) Don't hardcode hdfs.audit.logger in the scripts
[ https://issues.apache.org/jira/browse/HADOOP-8224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13273986#comment-13273986 ] Hudson commented on HADOOP-8224: Integrated in Hadoop-Hdfs-trunk #1041 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1041/]) HADOOP-8224. Don't hardcode hdfs.audit.logger in the scripts. Contributed by Tomohiko Kinebuchi (Revision 1337339) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337339 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemon.sh * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh > Don't hardcode hdfs.audit.logger in the scripts > --- > > Key: HADOOP-8224 > URL: https://issues.apache.org/jira/browse/HADOOP-8224 > Project: Hadoop Common > Issue Type: Improvement > Components: conf >Affects Versions: 2.0.0 >Reporter: Eli Collins >Assignee: Tomohiko Kinebuchi > Fix For: 2.0.0 > > Attachments: HADOOP-8224.txt, HADOOP-8224.txt, hadoop-8224.txt > > > The HADOOP_*OPTS defined for HDFS in hadoop-env.sh hard-code the > hdfs.audit.logger (is explicitly set via "-Dhdfs.audit.logger=INFO,RFAAUDIT") > so it's not overridable. Let's allow someone to override it as we do the > other parameters by introducing HADOOP_AUDIT_LOGGER. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8353) hadoop-daemon.sh and yarn-daemon.sh can be misleading on stop
[ https://issues.apache.org/jira/browse/HADOOP-8353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13273982#comment-13273982 ] Hudson commented on HADOOP-8353: Integrated in Hadoop-Hdfs-trunk #1041 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1041/]) HADOOP-8353. hadoop-daemon.sh and yarn-daemon.sh can be misleading on stop. Contributed by Roman Shaposhnik. (Revision 1337251) Result = FAILURE atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337251 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemon.sh * /hadoop/common/trunk/hadoop-mapreduce-project/bin/mr-jobhistory-daemon.sh * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh > hadoop-daemon.sh and yarn-daemon.sh can be misleading on stop > - > > Key: HADOOP-8353 > URL: https://issues.apache.org/jira/browse/HADOOP-8353 > Project: Hadoop Common > Issue Type: Improvement > Components: scripts >Affects Versions: 0.23.1 >Reporter: Roman Shaposhnik >Assignee: Roman Shaposhnik > Fix For: 2.0.0 > > Attachments: HADOOP-8353-2.patch.txt, HADOOP-8353.patch.txt > > > The way that stop actions is implemented is a simple SIGTERM sent to the JVM. > There's a time delay between when the action is called and when the process > actually exists. This can be misleading to the callers of the *-daemon.sh > scripts since they expect stop action to return when process is actually > stopped. > I suggest we augment the stop action with a time-delay check for the process > status and a SIGKILL once the delay has expired. > I understand that sending SIGKILL is a measure of last resort and is > generally frowned upon among init.d script writers, but the excuse we have > for Hadoop is that it is engineered to be a fault tolerant system and thus > there's not danger of putting system into an incontinent state by a violent > SIGKILL. Of course, the time delay will be long enough to make SIGKILL event > a rare condition. > Finally, there's always an option of an exponential back-off type of solution > if we decide that SIGKILL timeout is short. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8366) Use ProtoBuf for RpcResponseHeader
[ https://issues.apache.org/jira/browse/HADOOP-8366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13273984#comment-13273984 ] Hudson commented on HADOOP-8366: Integrated in Hadoop-Hdfs-trunk #1041 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1041/]) Move HADOOP-8285 and HADOOP-8366 to 2.0.0 in CHANGES.txt. (Revision 1337431) HADOOP-8366 Use ProtoBuf for RpcResponseHeader (sanjay radia) (Revision 1337283) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337431 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt sradia : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337283 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Status.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/proto/RpcPayloadHeader.proto > Use ProtoBuf for RpcResponseHeader > -- > > Key: HADOOP-8366 > URL: https://issues.apache.org/jira/browse/HADOOP-8366 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Sanjay Radia >Assignee: Sanjay Radia >Priority: Blocker > Fix For: 2.0.0 > > Attachments: hadoop-8366-1.patch, hadoop-8366-2.patch, > hadoop-8366-3.patch, hadoop-8366-4.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8316) Audit logging should be disabled by default
[ https://issues.apache.org/jira/browse/HADOOP-8316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13273983#comment-13273983 ] Hudson commented on HADOOP-8316: Integrated in Hadoop-Hdfs-trunk #1041 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1041/]) HADOOP-8316. Audit logging should be disabled by default. Contributed by Eli Collins (Revision 1337334) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337334 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/log4j.properties > Audit logging should be disabled by default > --- > > Key: HADOOP-8316 > URL: https://issues.apache.org/jira/browse/HADOOP-8316 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Affects Versions: 2.0.0 >Reporter: Eli Collins >Assignee: Eli Collins > Fix For: 2.0.0 > > Attachments: hadoop-8316.txt > > > HADOOP-7633 made hdfs, mr and security audit logging on by default (INFO > level) in log4j.properties used for the packages, this then got copied over > to the non-packaging log4j.properties in HADOOP-8216 (which made them > consistent). > Seems like we should keep with the v1.x setting which is disabled (WARNING > level) by default. There's a performance overhead to audit logging, and > HADOOP-7633 provided not rationale (just "We should add the audit logs as > part of default confs") as to why they were enabled for the packages. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-6801) io.sort.mb and io.sort.factor were renamed and moved to mapreduce but are still in CommonConfigurationKeysPublic.java and used in SequenceFile.java
[
https://issues.apache.org/jira/browse/HADOOP-6801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J updated HADOOP-6801:
Attachment: HADOOP-6801.r5.diff
The javac warnings further, were cause of a bad link to SequenceFile.Sorter
(inner class)
{code}
[WARNING]
/Users/harshchouraria/Work/code/apache/hadoop/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java:142:
warning - Tag @link: reference not found: SequenceFile.Sorter{code}
In this new patch I've fixed the javac warns and javadoc warns both.
> io.sort.mb and io.sort.factor were renamed and moved to mapreduce but are
> still in CommonConfigurationKeysPublic.java and used in SequenceFile.java
> ---
>
> Key: HADOOP-6801
> URL: https://issues.apache.org/jira/browse/HADOOP-6801
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 0.22.0
>Reporter: Erik Steffl
>Assignee: Harsh J
>Priority: Minor
> Attachments: HADOOP-6801.r1.diff, HADOOP-6801.r2.diff,
> HADOOP-6801.r3.diff, HADOOP-6801.r4.diff, HADOOP-6801.r5.diff
>
>
> Following configuration keys in CommonConfigurationKeysPublic.java (former
> CommonConfigurationKeys.java):
> public static final String IO_SORT_MB_KEY = "io.sort.mb";
> public static final String IO_SORT_FACTOR_KEY = "io.sort.factor";
> are partially moved:
> - they were renamed to mapreduce.task.io.sort.mb and
> mapreduce.task.io.sort.factor respectively
> - they were moved to mapreduce project, documented in mapred-default.xml
> However:
> - they are still listed in CommonConfigurationKeysPublic.java as quoted
> above
> - strings "io.sort.mb" and "io.sort.factor" are used in SequenceFile.java
> in Hadoop Common project
> Not sure what the solution is, these constants should probably be removed
> from CommonConfigurationKeysPublic.java but I am not sure what's the best
> solution for SequenceFile.java.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-6801) io.sort.mb and io.sort.factor were renamed and moved to mapreduce but are still in CommonConfigurationKeysPublic.java and used in SequenceFile.java
[ https://issues.apache.org/jira/browse/HADOOP-6801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated HADOOP-6801: Attachment: HADOOP-6801.r4.diff The javac warning was cause of the graceful config deprecation code added, which referred to deprecated keys a few times. I've suppressed that particular constructor to not warn for now in this new patch. > io.sort.mb and io.sort.factor were renamed and moved to mapreduce but are > still in CommonConfigurationKeysPublic.java and used in SequenceFile.java > --- > > Key: HADOOP-6801 > URL: https://issues.apache.org/jira/browse/HADOOP-6801 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 0.22.0 >Reporter: Erik Steffl >Assignee: Harsh J >Priority: Minor > Attachments: HADOOP-6801.r1.diff, HADOOP-6801.r2.diff, > HADOOP-6801.r3.diff, HADOOP-6801.r4.diff > > > Following configuration keys in CommonConfigurationKeysPublic.java (former > CommonConfigurationKeys.java): > public static final String IO_SORT_MB_KEY = "io.sort.mb"; > public static final String IO_SORT_FACTOR_KEY = "io.sort.factor"; > are partially moved: > - they were renamed to mapreduce.task.io.sort.mb and > mapreduce.task.io.sort.factor respectively > - they were moved to mapreduce project, documented in mapred-default.xml > However: > - they are still listed in CommonConfigurationKeysPublic.java as quoted > above > - strings "io.sort.mb" and "io.sort.factor" are used in SequenceFile.java > in Hadoop Common project > Not sure what the solution is, these constants should probably be removed > from CommonConfigurationKeysPublic.java but I am not sure what's the best > solution for SequenceFile.java. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8317) Update maven-assembly-plugin to 2.3 - fix build on FreeBSD
[ https://issues.apache.org/jira/browse/HADOOP-8317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13273970#comment-13273970 ] Hudson commented on HADOOP-8317: Integrated in Hadoop-Hdfs-0.23-Build #254 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/254/]) svn merge -c 1332775 FIXES: HADOOP-8317. Update maven-assembly-plugin to 2.3 - fix build on FreeBSD (Radim Kolar via bobby) (Revision 1337209) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1337209 Files : * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-project/pom.xml * /hadoop/common/branches/branch-0.23/pom.xml > Update maven-assembly-plugin to 2.3 - fix build on FreeBSD > -- > > Key: HADOOP-8317 > URL: https://issues.apache.org/jira/browse/HADOOP-8317 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 0.23.3, 2.0.0 > Environment: FreeBSD 8.2, AMD 64, OPENJDK 6, ZFS >Reporter: Radim Kolar > Fix For: 0.23.3, 2.0.0, 3.0.0 > > Attachments: assembly-plugin-update.txt > > > There is bug in hadoop-assembly plugin which makes builds fail on FreeBSD > because its chmod do not understand nonstgandard linux parameters. Unless you > do mvn clean before every build it fails with: > [INFO] --- maven-assembly-plugin:2.2.1:single (dist) @ hadoop-common --- > [WARNING] The following patterns were never triggered in this artifact > exclusion filter: > o 'org.apache.ant:*:jar' > o 'jdiff:jdiff:jar' > [INFO] Copying files to > /usr/local/jboss/.jenkins/jobs/Hadoop-0.23/workspace/hadoop-common-project/hadoop-common/target/hadoop-common-0.23.3-SNAPSHOT > [WARNING] --- > [WARNING] Standard error: > [WARNING] --- > [WARNING] > [WARNING] --- > [WARNING] Standard output: > [WARNING] --- > [WARNING] chmod: > /usr/local/jboss/.jenkins/jobs/Hadoop-0.23/workspace/hadoop-common-project/hadoop-common/target/hadoop-common-0.23.3-SNAPSHOT/share/hadoop/common/lib/hadoop-auth-0.23.3-SNAPSHOT.jar: > Inappropriate file type or format > [WARNING] --- > mojoFailed org.apache.maven.plugins:maven-assembly-plugin:2.2.1(dist) > projectFailed org.apache.hadoop:hadoop-common:0.23.3-SNAPSHOT > sessionEnded -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6801) io.sort.mb and io.sort.factor were renamed and moved to mapreduce but are still in CommonConfigurationKeysPublic.java and used in SequenceFile.java
[ https://issues.apache.org/jira/browse/HADOOP-6801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13273967#comment-13273967 ] Hadoop QA commented on HADOOP-6801: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12526628/HADOOP-6801.r3.diff against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. -1 javac. The applied patch generated 1938 javac compiler warnings (more than the trunk's current 1934 warnings). -1 javadoc. The javadoc tool appears to have generated 8 warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/989//testReport/ Javac warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/989//artifact/trunk/trunk/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/989//console This message is automatically generated. > io.sort.mb and io.sort.factor were renamed and moved to mapreduce but are > still in CommonConfigurationKeysPublic.java and used in SequenceFile.java > --- > > Key: HADOOP-6801 > URL: https://issues.apache.org/jira/browse/HADOOP-6801 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 0.22.0 >Reporter: Erik Steffl >Assignee: Harsh J >Priority: Minor > Attachments: HADOOP-6801.r1.diff, HADOOP-6801.r2.diff, > HADOOP-6801.r3.diff > > > Following configuration keys in CommonConfigurationKeysPublic.java (former > CommonConfigurationKeys.java): > public static final String IO_SORT_MB_KEY = "io.sort.mb"; > public static final String IO_SORT_FACTOR_KEY = "io.sort.factor"; > are partially moved: > - they were renamed to mapreduce.task.io.sort.mb and > mapreduce.task.io.sort.factor respectively > - they were moved to mapreduce project, documented in mapred-default.xml > However: > - they are still listed in CommonConfigurationKeysPublic.java as quoted > above > - strings "io.sort.mb" and "io.sort.factor" are used in SequenceFile.java > in Hadoop Common project > Not sure what the solution is, these constants should probably be removed > from CommonConfigurationKeysPublic.java but I am not sure what's the best > solution for SequenceFile.java. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8358) Config-related WARN for dfs.web.ugi can be avoided.
[
https://issues.apache.org/jira/browse/HADOOP-8358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13273966#comment-13273966
]
Harsh J commented on HADOOP-8358:
-
Any further comments on the patch? Its quite trivial a change and helps remove
unnecessary WARN noise.
> Config-related WARN for dfs.web.ugi can be avoided.
> ---
>
> Key: HADOOP-8358
> URL: https://issues.apache.org/jira/browse/HADOOP-8358
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
>Affects Versions: 2.0.0
>Reporter: Harsh J
>Assignee: Harsh J
>Priority: Trivial
> Attachments: HADOOP-8358.patch
>
>
> {code}
> 2012-05-04 11:55:13,367 WARN org.apache.hadoop.http.lib.StaticUserWebFilter:
> dfs.web.ugi should not be used. Instead, use hadoop.http.staticuser.user.
> {code}
> Looks easy to fix, and we should avoid using old config params that we
> ourselves deprecated.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8297) Writable javadocs don't carry default constructor
[ https://issues.apache.org/jira/browse/HADOOP-8297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13273964#comment-13273964 ] Harsh J commented on HADOOP-8297: - Given that this is a trivial patch that just fixes javadocs for developers, if no one has further comments on the helpful changes introduced here I will commit this in by monday. > Writable javadocs don't carry default constructor > - > > Key: HADOOP-8297 > URL: https://issues.apache.org/jira/browse/HADOOP-8297 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Affects Versions: 2.0.0 >Reporter: Harsh J >Assignee: Harsh J >Priority: Trivial > Attachments: HADOOP-8297.patch > > > The Writable API docs have a custom writable example but doesn't carry a > default constructor in it. Apparently a default constructor is required and > hence the example ought to carry it for benefit of the reader/paster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7659) fs -getmerge isn't guaranteed to work well over non-HDFS filesystems
[
https://issues.apache.org/jira/browse/HADOOP-7659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J updated HADOOP-7659:
Target Version/s: 3.0.0
Fix Version/s: (was: 0.24.0)
Given that this is a trivial change that merely adds a helpful javadoc note, if
no one has any comments on this, am going to commit it in by monday.
> fs -getmerge isn't guaranteed to work well over non-HDFS filesystems
>
>
> Key: HADOOP-7659
> URL: https://issues.apache.org/jira/browse/HADOOP-7659
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 0.20.204.0
>Reporter: Harsh J
>Assignee: Harsh J
>Priority: Minor
> Attachments: HADOOP-7659.patch
>
>
> When you use {{fs -getmerge}} with HDFS, you are guaranteed file list sorting
> (part-0, part-1, onwards). When you use the same with other FSes we
> bundle, the ordering of listing is not guaranteed at all. This is cause of
> http://download.oracle.com/javase/6/docs/api/java/io/File.html#list() which
> we use internally for native file listing.
> This should either be documented as a known issue on -getmerge help
> pages/mans, or a consistent ordering (similar to HDFS) must be applied atop
> the listing. I suspect the latter only makes it worthy for what we include -
> while other FSes out there still have to deal with this issue. Perhaps we
> need a recommendation doc note added to our API?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-6801) io.sort.mb and io.sort.factor were renamed and moved to mapreduce but are still in CommonConfigurationKeysPublic.java and used in SequenceFile.java
[ https://issues.apache.org/jira/browse/HADOOP-6801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated HADOOP-6801: Attachment: HADOOP-6801.r3.diff Updated patch. - Improved docs. - Added more javadoc links to right places. - Added deprecated and new property tests for graceful change. > io.sort.mb and io.sort.factor were renamed and moved to mapreduce but are > still in CommonConfigurationKeysPublic.java and used in SequenceFile.java > --- > > Key: HADOOP-6801 > URL: https://issues.apache.org/jira/browse/HADOOP-6801 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 0.22.0 >Reporter: Erik Steffl >Assignee: Harsh J >Priority: Minor > Attachments: HADOOP-6801.r1.diff, HADOOP-6801.r2.diff, > HADOOP-6801.r3.diff > > > Following configuration keys in CommonConfigurationKeysPublic.java (former > CommonConfigurationKeys.java): > public static final String IO_SORT_MB_KEY = "io.sort.mb"; > public static final String IO_SORT_FACTOR_KEY = "io.sort.factor"; > are partially moved: > - they were renamed to mapreduce.task.io.sort.mb and > mapreduce.task.io.sort.factor respectively > - they were moved to mapreduce project, documented in mapred-default.xml > However: > - they are still listed in CommonConfigurationKeysPublic.java as quoted > above > - strings "io.sort.mb" and "io.sort.factor" are used in SequenceFile.java > in Hadoop Common project > Not sure what the solution is, these constants should probably be removed > from CommonConfigurationKeysPublic.java but I am not sure what's the best > solution for SequenceFile.java. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8396) DataStreamer, OutOfMemoryError, unable to create new native thread
[ https://issues.apache.org/jira/browse/HADOOP-8396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G resolved HADOOP-8396. - Resolution: Invalid Marking it as Invalid. > DataStreamer, OutOfMemoryError, unable to create new native thread > -- > > Key: HADOOP-8396 > URL: https://issues.apache.org/jira/browse/HADOOP-8396 > Project: Hadoop Common > Issue Type: Bug > Components: io >Affects Versions: 1.0.2 > Environment: Ubuntu 64bit, 4GB of RAM, Core Duo processors, commodity > hardware. >Reporter: Catalin Alexandru Zamfir >Priority: Blocker > Labels: DataStreamer, I/O, OutOfMemoryError, ResponseProcessor, > hadoop,, leak, memory, rpc, > > We're trying to write about 1 few billion records, via "Avro". When we got > this error, that's unrelated to our code: > 10725984 [Main] INFO net.gameloft.RnD.Hadoop.App - ## At: 2:58:43.290 # > Written: 52100 records > Exception in thread "DataStreamer for file > /Streams/Cubed/Stuff/objGame/aRandomGame/objType/aRandomType/2012/05/11/20/29/Shard.avro > block blk_3254486396346586049_75838" java.lang.OutOfMemoryError: unable to > create new native thread > at java.lang.Thread.start0(Native Method) > at java.lang.Thread.start(Thread.java:657) > at > org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:612) > at > org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:184) > at org.apache.hadoop.ipc.Client.getConnection(Client.java:1202) > at org.apache.hadoop.ipc.Client.call(Client.java:1046) > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225) > at $Proxy8.getProtocolVersion(Unknown Source) > at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396) > at > org.apache.hadoop.hdfs.DFSClient.createClientDatanodeProtocolProxy(DFSClient.java:160) > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:3117) > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2200(DFSClient.java:2586) > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2790) > 10746169 [Main] INFO net.gameloft.RnD.Hadoop.App - ## At: 2:59:03.474 # > Written: 52200 records > Exception in thread "ResponseProcessor for block > blk_4201760269657070412_73948" java.lang.OutOfMemoryError > at sun.misc.Unsafe.allocateMemory(Native Method) > at java.nio.DirectByteBuffer.(DirectByteBuffer.java:117) > at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:305) > at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:75) > at sun.nio.ch.IOUtil.read(IOUtil.java:223) > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:254) > at > org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:55) > at > org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142) > at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155) > at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128) > at java.io.DataInputStream.readFully(DataInputStream.java:195) > at java.io.DataInputStream.readLong(DataInputStream.java:416) > at > org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:124) > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2964) > # > # There is insufficient memory for the Java Runtime Environment to continue. > # Native memory allocation (malloc) failed to allocate 32 bytes for intptr_t > in > /build/buildd/openjdk-6-6b23~pre11/build/openjdk/hotspot/src/share/vm/runtime/deoptimization.cpp > [thread 1587264368 also had an error] > [thread 309168 also had an error] > [thread 1820371824 also had an error] > [thread 1343454064 also had an error] > [thread 1345444720 also had an error] > # An error report file with more information is saved as: > # [thread 1345444720 also had an error] > [thread -1091290256 also had an error] > [thread 678165360 also had an error] > [thread 678497136 also had an error] > [thread 675511152 also had an error] > [thread 1385937776 also had an error] > [thread 911969136 also had an error] > [thread -1086207120 also had an error] > [thread -1088251024 also had an error] > [thread -1088914576 also had an error] > [thread -1086870672 also had an error] > [thread 441797488 also had an error][thread 445778800 also had an error] > [thread 440400752 also had an error] > [thread 444119920 also had an error][thread 1151298416 also had an error] > [thread 443124592 also had an error] > [thread 1152625520 also had an
[jira] [Commented] (HADOOP-1381) The distance between sync blocks in SequenceFiles should be configurable rather than hard coded to 2000 bytes
[ https://issues.apache.org/jira/browse/HADOOP-1381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13273911#comment-13273911 ] Harsh J commented on HADOOP-1381: - Any further comments on the addition? > The distance between sync blocks in SequenceFiles should be configurable > rather than hard coded to 2000 bytes > - > > Key: HADOOP-1381 > URL: https://issues.apache.org/jira/browse/HADOOP-1381 > Project: Hadoop Common > Issue Type: Improvement > Components: io >Affects Versions: 0.22.0 >Reporter: Owen O'Malley >Assignee: Harsh J > Attachments: HADOOP-1381.r1.diff, HADOOP-1381.r2.diff, > HADOOP-1381.r3.diff, HADOOP-1381.r4.diff, HADOOP-1381.r5.diff, > HADOOP-1381.r5.diff > > > Currently SequenceFiles put in sync blocks every 2000 bytes. It would be much > better if it was configurable with a much higher default (1mb or so?). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8323) Revert HADOOP-7940 and improve javadocs and test for Text.clear()
[ https://issues.apache.org/jira/browse/HADOOP-8323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13273909#comment-13273909 ] Harsh J commented on HADOOP-8323: - Any further comments on the test and javadocs? If not, given that it is a trivial non-breaking addition I shall commit it in by Monday. > Revert HADOOP-7940 and improve javadocs and test for Text.clear() > - > > Key: HADOOP-8323 > URL: https://issues.apache.org/jira/browse/HADOOP-8323 > Project: Hadoop Common > Issue Type: Bug > Components: io >Affects Versions: 2.0.0 >Reporter: Harsh J >Assignee: Harsh J >Priority: Critical > Labels: performance > Attachments: HADOOP-8323.patch, HADOOP-8323.patch, HADOOP-8323.patch > > > Per [~jdonofrio]'s comments on HADOOP-7940, we should revert it as it has > caused a performance regression (for scenarios where Text is reused, popular > in MR). > The clear() works as intended, as the API also offers a current length API. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8396) DataStreamer, OutOfMemoryError, unable to create new native thread
[ https://issues.apache.org/jira/browse/HADOOP-8396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13273905#comment-13273905 ] Catalin Alexandru Zamfir commented on HADOOP-8396: -- Mark it as invalid please. We've found the culprit. The base writer was opening files but not closing them. It assumed that FileSystem.create (path, false) would append. A misinterpretation of the docs. We found "FileSystem.append" which does exactly what must be done in case a file exists. We were explicitly flushing and closing the streams. When we added a check to see if the file existed or not and opened a FSDataOutputStream via append, the number of threads and consumed memory kept well between 167 and 247, whenever our 'flushing & closing" scheme went in. > DataStreamer, OutOfMemoryError, unable to create new native thread > -- > > Key: HADOOP-8396 > URL: https://issues.apache.org/jira/browse/HADOOP-8396 > Project: Hadoop Common > Issue Type: Bug > Components: io >Affects Versions: 1.0.2 > Environment: Ubuntu 64bit, 4GB of RAM, Core Duo processors, commodity > hardware. >Reporter: Catalin Alexandru Zamfir >Priority: Blocker > Labels: DataStreamer, I/O, OutOfMemoryError, ResponseProcessor, > hadoop,, leak, memory, rpc, > > We're trying to write about 1 few billion records, via "Avro". When we got > this error, that's unrelated to our code: > 10725984 [Main] INFO net.gameloft.RnD.Hadoop.App - ## At: 2:58:43.290 # > Written: 52100 records > Exception in thread "DataStreamer for file > /Streams/Cubed/Stuff/objGame/aRandomGame/objType/aRandomType/2012/05/11/20/29/Shard.avro > block blk_3254486396346586049_75838" java.lang.OutOfMemoryError: unable to > create new native thread > at java.lang.Thread.start0(Native Method) > at java.lang.Thread.start(Thread.java:657) > at > org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:612) > at > org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:184) > at org.apache.hadoop.ipc.Client.getConnection(Client.java:1202) > at org.apache.hadoop.ipc.Client.call(Client.java:1046) > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225) > at $Proxy8.getProtocolVersion(Unknown Source) > at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396) > at > org.apache.hadoop.hdfs.DFSClient.createClientDatanodeProtocolProxy(DFSClient.java:160) > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:3117) > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2200(DFSClient.java:2586) > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2790) > 10746169 [Main] INFO net.gameloft.RnD.Hadoop.App - ## At: 2:59:03.474 # > Written: 52200 records > Exception in thread "ResponseProcessor for block > blk_4201760269657070412_73948" java.lang.OutOfMemoryError > at sun.misc.Unsafe.allocateMemory(Native Method) > at java.nio.DirectByteBuffer.(DirectByteBuffer.java:117) > at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:305) > at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:75) > at sun.nio.ch.IOUtil.read(IOUtil.java:223) > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:254) > at > org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:55) > at > org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142) > at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155) > at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128) > at java.io.DataInputStream.readFully(DataInputStream.java:195) > at java.io.DataInputStream.readLong(DataInputStream.java:416) > at > org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:124) > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2964) > # > # There is insufficient memory for the Java Runtime Environment to continue. > # Native memory allocation (malloc) failed to allocate 32 bytes for intptr_t > in > /build/buildd/openjdk-6-6b23~pre11/build/openjdk/hotspot/src/share/vm/runtime/deoptimization.cpp > [thread 1587264368 also had an error] > [thread 309168 also had an error] > [thread 1820371824 also had an error] > [thread 1343454064 also had an error] > [thread 1345444720 also had an error] > # An error report file with more information is saved as: > # [thread 1345444720 also had an error] > [thread -1091290256 also had an error] > [thread 678165360 also had
[jira] [Commented] (HADOOP-8396) DataStreamer, OutOfMemoryError, unable to create new native thread
[ https://issues.apache.org/jira/browse/HADOOP-8396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13273895#comment-13273895 ] Harsh J commented on HADOOP-8396: - It is also more likely that you're hitting a ulimit -n limit rather than an actual memory fill up. Hitting that limit will cause the JVM to exit with the same exception. > DataStreamer, OutOfMemoryError, unable to create new native thread > -- > > Key: HADOOP-8396 > URL: https://issues.apache.org/jira/browse/HADOOP-8396 > Project: Hadoop Common > Issue Type: Bug > Components: io >Affects Versions: 1.0.2 > Environment: Ubuntu 64bit, 4GB of RAM, Core Duo processors, commodity > hardware. >Reporter: Catalin Alexandru Zamfir >Priority: Blocker > Labels: DataStreamer, I/O, OutOfMemoryError, ResponseProcessor, > hadoop,, leak, memory, rpc, > > We're trying to write about 1 few billion records, via "Avro". When we got > this error, that's unrelated to our code: > 10725984 [Main] INFO net.gameloft.RnD.Hadoop.App - ## At: 2:58:43.290 # > Written: 52100 records > Exception in thread "DataStreamer for file > /Streams/Cubed/Stuff/objGame/aRandomGame/objType/aRandomType/2012/05/11/20/29/Shard.avro > block blk_3254486396346586049_75838" java.lang.OutOfMemoryError: unable to > create new native thread > at java.lang.Thread.start0(Native Method) > at java.lang.Thread.start(Thread.java:657) > at > org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:612) > at > org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:184) > at org.apache.hadoop.ipc.Client.getConnection(Client.java:1202) > at org.apache.hadoop.ipc.Client.call(Client.java:1046) > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225) > at $Proxy8.getProtocolVersion(Unknown Source) > at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396) > at > org.apache.hadoop.hdfs.DFSClient.createClientDatanodeProtocolProxy(DFSClient.java:160) > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:3117) > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2200(DFSClient.java:2586) > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2790) > 10746169 [Main] INFO net.gameloft.RnD.Hadoop.App - ## At: 2:59:03.474 # > Written: 52200 records > Exception in thread "ResponseProcessor for block > blk_4201760269657070412_73948" java.lang.OutOfMemoryError > at sun.misc.Unsafe.allocateMemory(Native Method) > at java.nio.DirectByteBuffer.(DirectByteBuffer.java:117) > at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:305) > at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:75) > at sun.nio.ch.IOUtil.read(IOUtil.java:223) > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:254) > at > org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:55) > at > org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142) > at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155) > at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128) > at java.io.DataInputStream.readFully(DataInputStream.java:195) > at java.io.DataInputStream.readLong(DataInputStream.java:416) > at > org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:124) > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2964) > # > # There is insufficient memory for the Java Runtime Environment to continue. > # Native memory allocation (malloc) failed to allocate 32 bytes for intptr_t > in > /build/buildd/openjdk-6-6b23~pre11/build/openjdk/hotspot/src/share/vm/runtime/deoptimization.cpp > [thread 1587264368 also had an error] > [thread 309168 also had an error] > [thread 1820371824 also had an error] > [thread 1343454064 also had an error] > [thread 1345444720 also had an error] > # An error report file with more information is saved as: > # [thread 1345444720 also had an error] > [thread -1091290256 also had an error] > [thread 678165360 also had an error] > [thread 678497136 also had an error] > [thread 675511152 also had an error] > [thread 1385937776 also had an error] > [thread 911969136 also had an error] > [thread -1086207120 also had an error] > [thread -1088251024 also had an error] > [thread -1088914576 also had an error] > [thread -1086870672 also had an error] > [thread 441797488 also had an error][thread 445778800 also had an error] > [thread 440400752 also had an
[jira] [Commented] (HADOOP-8396) DataStreamer, OutOfMemoryError, unable to create new native thread
[
https://issues.apache.org/jira/browse/HADOOP-8396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13273883#comment-13273883
]
Uma Maheswara Rao G commented on HADOOP-8396:
-
{quote}
nd I've checked, after writing a few million records, executing a "reader"
class on that data, returns the data, meaning Hadoop got to write these to the
HDFS, but watching "htop" the number of threads and memory for this code kept
increasing and only when writing started. We're writing from one single thread
(main).
{quote}
You are not closing the file from application once your data write completed?
When you open the stream to DFS, user only will start writing the data on that
stream. Whatever data user writes, DataStreamer will writes to DNs. User only
will will know, whether stream can be closed or not. Hadoop Client can not
assume that user will not write any more data on that stream. Because, it is
still possible that user can write some more data on that stream if stream
still opens. Once that is closed automatically Streamer threads will exit. I am
not sure you are about this lines or some other. Please correct if I understand
ur point wrongly.
> DataStreamer, OutOfMemoryError, unable to create new native thread
> --
>
> Key: HADOOP-8396
> URL: https://issues.apache.org/jira/browse/HADOOP-8396
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
>Affects Versions: 1.0.2
> Environment: Ubuntu 64bit, 4GB of RAM, Core Duo processors, commodity
> hardware.
>Reporter: Catalin Alexandru Zamfir
>Priority: Blocker
> Labels: DataStreamer, I/O, OutOfMemoryError, ResponseProcessor,
> hadoop,, leak, memory, rpc,
>
> We're trying to write about 1 few billion records, via "Avro". When we got
> this error, that's unrelated to our code:
> 10725984 [Main] INFO net.gameloft.RnD.Hadoop.App - ## At: 2:58:43.290 #
> Written: 52100 records
> Exception in thread "DataStreamer for file
> /Streams/Cubed/Stuff/objGame/aRandomGame/objType/aRandomType/2012/05/11/20/29/Shard.avro
> block blk_3254486396346586049_75838" java.lang.OutOfMemoryError: unable to
> create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:657)
> at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:612)
> at
> org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:184)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1202)
> at org.apache.hadoop.ipc.Client.call(Client.java:1046)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
> at $Proxy8.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
> at
> org.apache.hadoop.hdfs.DFSClient.createClientDatanodeProtocolProxy(DFSClient.java:160)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:3117)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2200(DFSClient.java:2586)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2790)
> 10746169 [Main] INFO net.gameloft.RnD.Hadoop.App - ## At: 2:59:03.474 #
> Written: 52200 records
> Exception in thread "ResponseProcessor for block
> blk_4201760269657070412_73948" java.lang.OutOfMemoryError
> at sun.misc.Unsafe.allocateMemory(Native Method)
> at java.nio.DirectByteBuffer.(DirectByteBuffer.java:117)
> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:305)
> at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:75)
> at sun.nio.ch.IOUtil.read(IOUtil.java:223)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:254)
> at
> org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:55)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
> at java.io.DataInputStream.readFully(DataInputStream.java:195)
> at java.io.DataInputStream.readLong(DataInputStream.java:416)
> at
> org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:124)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2964)
> #
> # There is insufficient memory for the Java Runtime Environment to continue.
> # Native memory allocation (malloc) failed to allocate 32 bytes for intptr_t
> in
> /build/buildd/openjdk-6-6b23~pre11/build/openjdk/hotspot/src/share/
[jira] [Commented] (HADOOP-8396) DataStreamer, OutOfMemoryError, unable to create new native thread
[ https://issues.apache.org/jira/browse/HADOOP-8396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13273881#comment-13273881 ] Catalin Alexandru Zamfir commented on HADOOP-8396: -- Reading this article: http://blog.egilh.com/2006/06/2811aspx.html given that the latest JVM allocates about 1M per thread, means that the 3.000/4.000 threads are consistent to the node I'm running this on which has about 3GB available for "user-space". In theory I could reduce the stack size for native threads via -Xss, but that would only increase the number of threads, without actually resolving the problem. I think the problem is that Hadoop should let go of native threads that have already written their data to the HDFS. And I've checked, after writing a few million records, executing a "reader" class on that data, returns the data, meaning Hadoop got to write these to the HDFS, but watching "htop" the number of threads and memory for this code kept increasing and only when writing started. We're writing from one single thread (main). Hadoop should let go of native threads or instruct the JVM to loose these threads once it knows it's written the corresponding data. > DataStreamer, OutOfMemoryError, unable to create new native thread > -- > > Key: HADOOP-8396 > URL: https://issues.apache.org/jira/browse/HADOOP-8396 > Project: Hadoop Common > Issue Type: Bug > Components: io >Affects Versions: 1.0.2 > Environment: Ubuntu 64bit, 4GB of RAM, Core Duo processors, commodity > hardware. >Reporter: Catalin Alexandru Zamfir >Priority: Blocker > Labels: DataStreamer, I/O, OutOfMemoryError, ResponseProcessor, > hadoop,, leak, memory, rpc, > > We're trying to write about 1 few billion records, via "Avro". When we got > this error, that's unrelated to our code: > 10725984 [Main] INFO net.gameloft.RnD.Hadoop.App - ## At: 2:58:43.290 # > Written: 52100 records > Exception in thread "DataStreamer for file > /Streams/Cubed/Stuff/objGame/aRandomGame/objType/aRandomType/2012/05/11/20/29/Shard.avro > block blk_3254486396346586049_75838" java.lang.OutOfMemoryError: unable to > create new native thread > at java.lang.Thread.start0(Native Method) > at java.lang.Thread.start(Thread.java:657) > at > org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:612) > at > org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:184) > at org.apache.hadoop.ipc.Client.getConnection(Client.java:1202) > at org.apache.hadoop.ipc.Client.call(Client.java:1046) > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225) > at $Proxy8.getProtocolVersion(Unknown Source) > at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396) > at > org.apache.hadoop.hdfs.DFSClient.createClientDatanodeProtocolProxy(DFSClient.java:160) > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:3117) > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2200(DFSClient.java:2586) > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2790) > 10746169 [Main] INFO net.gameloft.RnD.Hadoop.App - ## At: 2:59:03.474 # > Written: 52200 records > Exception in thread "ResponseProcessor for block > blk_4201760269657070412_73948" java.lang.OutOfMemoryError > at sun.misc.Unsafe.allocateMemory(Native Method) > at java.nio.DirectByteBuffer.(DirectByteBuffer.java:117) > at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:305) > at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:75) > at sun.nio.ch.IOUtil.read(IOUtil.java:223) > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:254) > at > org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:55) > at > org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142) > at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155) > at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128) > at java.io.DataInputStream.readFully(DataInputStream.java:195) > at java.io.DataInputStream.readLong(DataInputStream.java:416) > at > org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:124) > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2964) > # > # There is insufficient memory for the Java Runtime Environment to continue. > # Native memory allocation (malloc) failed to allocate 32 bytes for intptr_t > in > /build/buildd/openjdk-6-6b23~pre11/build/openjdk
[jira] [Commented] (HADOOP-8396) DataStreamer, OutOfMemoryError, unable to create new native thread
[ https://issues.apache.org/jira/browse/HADOOP-8396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13273873#comment-13273873 ] Catalin Alexandru Zamfir commented on HADOOP-8396: -- I monitored the code with htop. Over time it grew from: "Tasks: 35, 147 thr, 1 running" to "Tasks: 36, 3475 thr, 1 running". Once I killed it via "kill" the number of "thr" kept dropping to the same 147. So it seems it's a direct relation between the fact that the application allocates a big number of, I guess, native threads but does not kill them when done. Also, I'm calling Runtime.getRuntime ().gc () every time the count of open streams is bigger than 5. Also, I'm explicitly flushing and closing these streams before removing them and before running the getRuntime ().gc () method. I'm not using any specific "GC" strategy, but the default one defined by openjdk-6. There are no other GC parameters on the command line when the program is run. I'll be profiling the code today, using jmap and post the results here. The output of cat /proc/sys/kernel/threads-max is: 48056 Still, the code only gets to about 3/4/5.000 "thr" (threads) in htop, reaches about 500M written records (or less by a few millions) and dies. Here's a dump of jmap -histo:live pid: num #instances #bytes class name -- 1: 1303640 96984920 [B 2:976162 69580696 [C 3:648949 31149552 java.nio.HeapByteBuffer 4:647505 31080240 java.nio.HeapCharBuffer 5:533222 12797328 java.util.HashMap$Entry 6:481595 11558280 java.lang.String 7: 80864556064 [I 8: 298053901240 9:1770602832960 java.lang.Long 10: 298052388304 11: 588632354520 sun.misc.FloatingDecimal 12: 12097168 [Lorg.h2.util.CacheObject; 13: 506742041760 Using "watch jmap -histo:live" to see reactions i get this on the CLI, every few million records: OpenJDK Server VM warning: GC locker is held; pre-dump GC was skipped Also, I see the "[B" class name, alternating between 20MB and 90MB, with both of them growing constantly over time. Also, after running the code for 15 minutes, "Eden space" and "PS Old Generation" started growing like crazy. "Eden space" started with an acceptable 25MB while "PS Old Generation" something small also (10/25MB, can't remember). Heap Configuration: MinHeapFreeRatio = 40 MaxHeapFreeRatio = 70 MaxHeapSize = 792723456 (756.0MB) NewSize = 1048576 (1.0MB) MaxNewSize = 4294901760 (4095.9375MB) OldSize = 4194304 (4.0MB) NewRatio = 2 SurvivorRatio= 8 PermSize = 16777216 (16.0MB) MaxPermSize = 134217728 (128.0MB) Heap Usage: PS Young Generation Eden Space: capacity = 260177920 (248.125MB) used = 104665056 (99.81637573242188MB) free = 155512864 (148.30862426757812MB) 40.22826225991814% used >From Space: capacity = 1441792 (1.375MB) used = 1409864 (1.3445510864257812MB) free = 31928 (0.03044891357421875MB) 97.78553355823864% used To Space: capacity = 1966080 (1.875MB) used = 0 (0.0MB) free = 1966080 (1.875MB) 0.0% used PS Old Generation capacity = 528482304 (504.0MB) used = 31693784 (30.225547790527344MB) free = 496788520 (473.77445220947266MB) 5.9971324981205045% used PS Perm Generation capacity = 16777216 (16.0MB) used = 13510752 (12.884857177734375MB) free = 3266464 (3.115142822265625MB) 80.53035736083984% used Hope this ton of information helps you. > DataStreamer, OutOfMemoryError, unable to create new native thread > -- > > Key: HADOOP-8396 > URL: https://issues.apache.org/jira/browse/HADOOP-8396 > Project: Hadoop Common > Issue Type: Bug > Components: io >Affects Versions: 1.0.2 > Environment: Ubuntu 64bit, 4GB of RAM, Core Duo processors, commodity > hardware. >Reporter: Catalin Alexandru Zamfir >Priority: Blocker > Labels: DataStreamer, I/O, OutOfMemoryError, ResponseProcessor, > hadoop,, leak, memory, rpc, > > We're trying to write about 1 few billion records, via "Avro". When we got > this error, that's unrelated to our code: > 10725984 [Main] INFO net.gameloft.RnD.Hadoop.App - ## At: 2:58:43.290 # > Written: 52100 records > Exception in thread "DataStreamer for file > /Streams/Cubed/Stuff/objGame/aRandomGame/objType/aRandomType/2012/05/11/20/29/Shard.avro > block blk_3254486396346586049_75838" java.lang.OutOfMemoryError: unable to > create new native thread > at java.lang.Thread.st
