[jira] [Commented] (HADOOP-8151) Error handling in snappy decompressor throws invalid exceptions
[
https://issues.apache.org/jira/browse/HADOOP-8151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13269392#comment-13269392
]
Matt Foley commented on HADOOP-8151:
Leaving Jira open pending code review and commit of fix for trunk and branch-2.
> Error handling in snappy decompressor throws invalid exceptions
> ---
>
> Key: HADOOP-8151
> URL: https://issues.apache.org/jira/browse/HADOOP-8151
> Project: Hadoop Common
> Issue Type: Bug
> Components: io, native
>Affects Versions: 1.0.2, 0.24.0
>Reporter: Todd Lipcon
>Assignee: Matt Foley
> Fix For: 1.0.3
>
> Attachments: HADOOP-8151-branch-1.0.patch, HADOOP-8151.patch
>
>
> SnappyDecompressor.c has the following code in a few places:
> {code}
> THROW(env, "Ljava/lang/InternalError", "Could not decompress data. Buffer
> length is too small.");
> {code}
> this is incorrect, though, since the THROW macro doesn't need the "L" before
> the class name. This results in a ClassNotFoundException for
> Ljava.lang.InternalError being thrown, instead of the intended exception.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8323) Revert HADOOP-7940 and improve javadocs and test for Text.clear()
[ https://issues.apache.org/jira/browse/HADOOP-8323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13269385#comment-13269385 ] Harsh J commented on HADOOP-8323: - bq. -1 javadoc. The javadoc tool appears to have generated 2 warning messages. This is HADOOP-8359, not this patch. > Revert HADOOP-7940 and improve javadocs and test for Text.clear() > - > > Key: HADOOP-8323 > URL: https://issues.apache.org/jira/browse/HADOOP-8323 > Project: Hadoop Common > Issue Type: Bug > Components: io >Affects Versions: 2.0.0 >Reporter: Harsh J >Assignee: Harsh J >Priority: Critical > Labels: performance > Attachments: HADOOP-8323.patch, HADOOP-8323.patch, HADOOP-8323.patch > > > Per [~jdonofrio]'s comments on HADOOP-7940, we should revert it as it has > caused a performance regression (for scenarios where Text is reused, popular > in MR). > The clear() works as intended, as the API also offers a current length API. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-1381) The distance between sync blocks in SequenceFiles should be configurable rather than hard coded to 2000 bytes
[ https://issues.apache.org/jira/browse/HADOOP-1381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13269383#comment-13269383 ] Harsh J commented on HADOOP-1381: - bq. -1 javadoc. The javadoc tool appears to have generated 2 warning messages. This is HADOOP-8359, not this patch. > The distance between sync blocks in SequenceFiles should be configurable > rather than hard coded to 2000 bytes > - > > Key: HADOOP-1381 > URL: https://issues.apache.org/jira/browse/HADOOP-1381 > Project: Hadoop Common > Issue Type: Improvement > Components: io >Affects Versions: 0.22.0 >Reporter: Owen O'Malley >Assignee: Harsh J > Attachments: HADOOP-1381.r1.diff, HADOOP-1381.r2.diff, > HADOOP-1381.r3.diff, HADOOP-1381.r4.diff, HADOOP-1381.r5.diff, > HADOOP-1381.r5.diff > > > Currently SequenceFiles put in sync blocks every 2000 bytes. It would be much > better if it was configurable with a much higher default (1mb or so?). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8230) Enable sync by default and disable append
[ https://issues.apache.org/jira/browse/HADOOP-8230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13269360#comment-13269360 ] Eli Collins commented on HADOOP-8230: - Suresh, Sanjay, the rationale is discussed in HDFS-3120. In short: # We shouldn't provide a flag that enables append because we know append has data loss issues # HBase and other programs have data loss when running against a default Hadoop 1.x install The rationale is pretty clear - this prevents data loss. Per my earlier comment, I'm open to having an option to disable durable sync if you think that use case is important, but what is that use case? Given that the sync code path is well tested and debugged, why would you want to run with a buggy sync implementation? > Enable sync by default and disable append > - > > Key: HADOOP-8230 > URL: https://issues.apache.org/jira/browse/HADOOP-8230 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 1.0.0 >Reporter: Eli Collins >Assignee: Eli Collins > Fix For: 1.1.0 > > Attachments: hadoop-8230.txt > > > Per HDFS-3120 for 1.x let's: > - Always enable the sync path, which is currently only enabled if > dfs.support.append is set > - Remove the dfs.support.append configuration option. We'll keep the code > paths though in case we ever fix append on branch-1, in which case we can add > the config option back -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8166) Remove JDK 1.5 dependency from building forrest docs
[ https://issues.apache.org/jira/browse/HADOOP-8166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-8166: --- Target Version/s: 1.1.0 (was: 1.0.3) Not complete in time for 1.0.3. Please consider continuing the contribution in 1.1. > Remove JDK 1.5 dependency from building forrest docs > > > Key: HADOOP-8166 > URL: https://issues.apache.org/jira/browse/HADOOP-8166 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Affects Versions: 0.20.203.0, 0.20.204.0, 0.20.205.0, 1.0.0, 1.0.1 >Reporter: Mark Butler > Attachments: forrest.patch, hadoop-8166.txt > > > Currently Hadoop requires both JDK 1.6 and JDK 1.5. JDK 1.5 is a requirement > of Forrest. It is easy to remove the latter requirement by turning off > forrest.validate.sitemap and forrest.validate.skins.stylesheets. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8307) The task-controller is not packaged in the tarball
[ https://issues.apache.org/jira/browse/HADOOP-8307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-8307: --- Target Version/s: (was: 1.0.3) > The task-controller is not packaged in the tarball > -- > > Key: HADOOP-8307 > URL: https://issues.apache.org/jira/browse/HADOOP-8307 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 1.0.3 >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Attachments: hadoop-8307.patch > > > Ant in some situations, puts artifacts such as task-controller into the > build/hadoop-*/ directory before the "package" target deletes it to start > over. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8151) Error handling in snappy decompressor throws invalid exceptions
[
https://issues.apache.org/jira/browse/HADOOP-8151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Foley updated HADOOP-8151:
---
Target Version/s: 1.0.3, 2.0.0 (was: 2.0.0, 1.0.3)
Fix Version/s: 1.0.3
> Error handling in snappy decompressor throws invalid exceptions
> ---
>
> Key: HADOOP-8151
> URL: https://issues.apache.org/jira/browse/HADOOP-8151
> Project: Hadoop Common
> Issue Type: Bug
> Components: io, native
>Affects Versions: 1.0.2, 0.24.0
>Reporter: Todd Lipcon
>Assignee: Matt Foley
> Fix For: 1.0.3
>
> Attachments: HADOOP-8151-branch-1.0.patch, HADOOP-8151.patch
>
>
> SnappyDecompressor.c has the following code in a few places:
> {code}
> THROW(env, "Ljava/lang/InternalError", "Could not decompress data. Buffer
> length is too small.");
> {code}
> this is incorrect, though, since the THROW macro doesn't need the "L" before
> the class name. This results in a ClassNotFoundException for
> Ljava.lang.InternalError being thrown, instead of the intended exception.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8151) Error handling in snappy decompressor throws invalid exceptions
[
https://issues.apache.org/jira/browse/HADOOP-8151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13269333#comment-13269333
]
Hadoop QA commented on HADOOP-8151:
---
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12525796/HADOOP-8151.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
-1 javadoc. The javadoc tool appears to have generated 2 warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed unit tests in
hadoop-common-project/hadoop-common.
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/946//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/946//console
This message is automatically generated.
> Error handling in snappy decompressor throws invalid exceptions
> ---
>
> Key: HADOOP-8151
> URL: https://issues.apache.org/jira/browse/HADOOP-8151
> Project: Hadoop Common
> Issue Type: Bug
> Components: io, native
>Affects Versions: 1.0.2, 0.24.0
>Reporter: Todd Lipcon
>Assignee: Matt Foley
> Attachments: HADOOP-8151-branch-1.0.patch, HADOOP-8151.patch
>
>
> SnappyDecompressor.c has the following code in a few places:
> {code}
> THROW(env, "Ljava/lang/InternalError", "Could not decompress data. Buffer
> length is too small.");
> {code}
> this is incorrect, though, since the THROW macro doesn't need the "L" before
> the class name. This results in a ClassNotFoundException for
> Ljava.lang.InternalError being thrown, instead of the intended exception.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8151) Error handling in snappy decompressor throws invalid exceptions
[
https://issues.apache.org/jira/browse/HADOOP-8151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Foley updated HADOOP-8151:
---
Target Version/s: 1.0.3, 2.0.0 (was: 3.0.0, 2.0.0, 1.0.3)
Status: Patch Available (was: Open)
> Error handling in snappy decompressor throws invalid exceptions
> ---
>
> Key: HADOOP-8151
> URL: https://issues.apache.org/jira/browse/HADOOP-8151
> Project: Hadoop Common
> Issue Type: Bug
> Components: io, native
>Affects Versions: 1.0.2, 0.24.0
>Reporter: Todd Lipcon
>Assignee: Matt Foley
> Attachments: HADOOP-8151-branch-1.0.patch, HADOOP-8151.patch
>
>
> SnappyDecompressor.c has the following code in a few places:
> {code}
> THROW(env, "Ljava/lang/InternalError", "Could not decompress data. Buffer
> length is too small.");
> {code}
> this is incorrect, though, since the THROW macro doesn't need the "L" before
> the class name. This results in a ClassNotFoundException for
> Ljava.lang.InternalError being thrown, instead of the intended exception.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8151) Error handling in snappy decompressor throws invalid exceptions
[
https://issues.apache.org/jira/browse/HADOOP-8151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Foley updated HADOOP-8151:
---
Attachment: HADOOP-8151.patch
Uploaded patch for trunk. Should also work in 0.23 and 2.0. As Todd noted,
includes fix for lz4 too.
> Error handling in snappy decompressor throws invalid exceptions
> ---
>
> Key: HADOOP-8151
> URL: https://issues.apache.org/jira/browse/HADOOP-8151
> Project: Hadoop Common
> Issue Type: Bug
> Components: io, native
>Affects Versions: 1.0.2, 0.24.0
>Reporter: Todd Lipcon
>Assignee: Matt Foley
> Attachments: HADOOP-8151-branch-1.0.patch, HADOOP-8151.patch
>
>
> SnappyDecompressor.c has the following code in a few places:
> {code}
> THROW(env, "Ljava/lang/InternalError", "Could not decompress data. Buffer
> length is too small.");
> {code}
> this is incorrect, though, since the THROW macro doesn't need the "L" before
> the class name. This results in a ClassNotFoundException for
> Ljava.lang.InternalError being thrown, instead of the intended exception.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8136) Enhance hadoop to use a newer version (0.8.1) of the jets3t library
[ https://issues.apache.org/jira/browse/HADOOP-8136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-8136: --- Target Version/s: 1.1.0 (was: 1.0.3) Not resolved in time for 1.0.3. Please consider continuing the contribution in 1.1.0. Thank you. > Enhance hadoop to use a newer version (0.8.1) of the jets3t library > --- > > Key: HADOOP-8136 > URL: https://issues.apache.org/jira/browse/HADOOP-8136 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 1.0.0, 0.22.0, 0.23.3 > Environment: Ubuntu 11.04, 64 bit, JDK 1.6.0_30 >Reporter: Jagane Sundar > Attachments: HADOOP-8136-0-for_branch_1_0.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > Hadoop is built against, and includes, an older version of the Jets3t library > - version 0.6.1. > The current version of the Jets3t library(as of March 2012) is 0.8.1. This > new version includes many improvements such as support for "Requester-Pays" > buckets. > Since hadoop includes a copy of the version 0.6.1 jets3t library, and since > this version ends up early in the CLASSPATH, any Map Reduce application that > wants to use the jets3t library ends up getting version 0.6.1 of the jets3t > library. The MR application fails, usually with an error stating that the > method signature of some method in the Jets3t library does not match. > It would be useful to enhance Jets3tNativeFileSystemStore.java to use the API > published by the 0.8.1 version of the jets3t library. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (HADOOP-8136) Enhance hadoop to use a newer version (0.8.1) of the jets3t library
[ https://issues.apache.org/jira/browse/HADOOP-8136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232297#comment-13232297 ] Matt Foley edited comment on HADOOP-8136 at 5/7/12 1:19 AM: I am working on the patch for trunk, Matt. Running into mvn difficulties. Mvn is picking up the old version of jets3t. Once I fix that, I should have the trunk patch ready. Hopefully later today. Jagane was (Author: jagane): I am working on the patch for trunk, Matt. Running into mvn difficulties. Mvn is picking up the old version of jets3t. Once I fix that, I should have the trunk patch ready. Hopefully later today. Jagane - Reply message - From: "Matt Foley (Commented) (JIRA)" To: "Jagane Sundar" Subject: [jira] [Commented] (HADOOP-8136) Enhance hadoop to use a newer version (0.8.1) of the jets3t library Date: Sun, Mar 18, 2012 1:20 am [ https://issues.apache.org/jira/browse/HADOOP-8136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232215#comment-13232215 ] Matt Foley commented on HADOOP-8136: Jagane, please attach patch(es) for trunk, and enable auto-test via "Patch Available", to assure no side effects of the version upgrade. Thanks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira > Enhance hadoop to use a newer version (0.8.1) of the jets3t library > --- > > Key: HADOOP-8136 > URL: https://issues.apache.org/jira/browse/HADOOP-8136 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 1.0.0, 0.22.0, 0.23.3 > Environment: Ubuntu 11.04, 64 bit, JDK 1.6.0_30 >Reporter: Jagane Sundar > Attachments: HADOOP-8136-0-for_branch_1_0.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > Hadoop is built against, and includes, an older version of the Jets3t library > - version 0.6.1. > The current version of the Jets3t library(as of March 2012) is 0.8.1. This > new version includes many improvements such as support for "Requester-Pays" > buckets. > Since hadoop includes a copy of the version 0.6.1 jets3t library, and since > this version ends up early in the CLASSPATH, any Map Reduce application that > wants to use the jets3t library ends up getting version 0.6.1 of the jets3t > library. The MR application fails, usually with an error stating that the > method signature of some method in the Jets3t library does not match. > It would be useful to enhance Jets3tNativeFileSystemStore.java to use the API > published by the 0.8.1 version of the jets3t library. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7381) FindBugs OutOfMemoryError
[ https://issues.apache.org/jira/browse/HADOOP-7381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-7381: --- Fix Version/s: 1.0.3 > FindBugs OutOfMemoryError > - > > Key: HADOOP-7381 > URL: https://issues.apache.org/jira/browse/HADOOP-7381 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 0.20.205.0 > Environment: FindBugs 1.3.9, ant 1.8.2, RHEL6, Jenkins 1.414 in > Tomcat 7.0.14, Sun Java HotSpot(TM) 64-Bit Server VM (build 14.2-b01, mixed > mode) >Reporter: Joep Rottinghuis >Assignee: Joep Rottinghuis > Fix For: 1.0.3 > > Attachments: hadoop-7381.patch > > > When running the findbugs target from Jenkins, I get an OutOfMemory error. > The "effort" in FindBugs is set to Max which ends up using a lot of memory to > go through all the classes. The jvmargs passed to FindBugs is hardcoded to > 512 MB max. > We can leave the default to 512M, as long as we pass this as an ant parameter > which can be overwritten in individual cases through -D, or in the > build.properties file (either basedir, or user's home directory). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7381) FindBugs OutOfMemoryError
[ https://issues.apache.org/jira/browse/HADOOP-7381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-7381: --- Resolution: Fixed Status: Resolved (was: Patch Available) Committed to branch-1 and branch-1.0. Thanks, Joep! > FindBugs OutOfMemoryError > - > > Key: HADOOP-7381 > URL: https://issues.apache.org/jira/browse/HADOOP-7381 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 0.20.205.0 > Environment: FindBugs 1.3.9, ant 1.8.2, RHEL6, Jenkins 1.414 in > Tomcat 7.0.14, Sun Java HotSpot(TM) 64-Bit Server VM (build 14.2-b01, mixed > mode) >Reporter: Joep Rottinghuis >Assignee: Joep Rottinghuis > Attachments: hadoop-7381.patch > > > When running the findbugs target from Jenkins, I get an OutOfMemory error. > The "effort" in FindBugs is set to Max which ends up using a lot of memory to > go through all the classes. The jvmargs passed to FindBugs is hardcoded to > 512 MB max. > We can leave the default to 512M, as long as we pass this as an ant parameter > which can be overwritten in individual cases through -D, or in the > build.properties file (either basedir, or user's home directory). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8230) Enable sync by default and disable append
[
https://issues.apache.org/jira/browse/HADOOP-8230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13269311#comment-13269311
]
Todd Lipcon commented on HADOOP-8230:
-
bq. This was turned on in the 20 originally and then we had to turn it off due
bugs.
Then several us spent many months fixing those bugs, and we haven't seen any
since.
{quote}
Given that the default in Hadoop 1 is off, why not leave it off and give a way
to turn it on.
The current default is off and I don't see a reason to change that default in
1.1.
There are installations have that are using the current default.
Can you please explain the reason the make this change?
{quote}
It's a pain for HBase users to have to manually flip this, and risk data loss
if they don't. Changing the default also means we have fewer code paths to
maintain for the average user.
> Enable sync by default and disable append
> -
>
> Key: HADOOP-8230
> URL: https://issues.apache.org/jira/browse/HADOOP-8230
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 1.0.0
>Reporter: Eli Collins
>Assignee: Eli Collins
> Fix For: 1.1.0
>
> Attachments: hadoop-8230.txt
>
>
> Per HDFS-3120 for 1.x let's:
> - Always enable the sync path, which is currently only enabled if
> dfs.support.append is set
> - Remove the dfs.support.append configuration option. We'll keep the code
> paths though in case we ever fix append on branch-1, in which case we can add
> the config option back
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7211) Security uses proprietary Sun APIs
[
https://issues.apache.org/jira/browse/HADOOP-7211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Foley updated HADOOP-7211:
---
Target Version/s: (was: 2.0.0, 1.0.3)
Resolved as dup, so removing Target Versions.
> Security uses proprietary Sun APIs
> --
>
> Key: HADOOP-7211
> URL: https://issues.apache.org/jira/browse/HADOOP-7211
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Reporter: Eli Collins
>Assignee: Luke Lu
>
> The security code uses the KrbException, Credentials, and PrincipalName
> classes from sun.security.krb5 and Krb5Util from sun.security.jgss.krb5.
> These may disappear in future Java releases. Also Hadoop does not compile
> using jdks that do not support them, for example with the following IBM JDK.
> {noformat}
> $ /home/eli/toolchain/java-x86_64-60/bin/java -version
> java version "1.6.0"
> Java(TM) SE Runtime Environment (build pxa6460sr9fp1-20110208_03(SR9 FP1))
> IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 Linux amd64-64
> jvmxa6460sr9-20110203_74623 (JIT enabled, AOT enabled)
> J9VM - 20110203_074623
> JIT - r9_20101028_17488ifx3
> GC - 20101027_AA)
> JCL - 20110203_01
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (HADOOP-7211) Security uses proprietary Sun APIs
[
https://issues.apache.org/jira/browse/HADOOP-7211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13248095#comment-13248095
]
Matt Foley edited comment on HADOOP-7211 at 5/6/12 9:48 PM:
On HADOOP-6941, my aim was to make the hadoop code base compile with IBM's JDK,
and ensure that there is no regression with Oracle's. I had done testing of
HDFS/MR with security enabled. I didn't do the tests that would go through the
SecurityUtil code path though (argh) - fixed in HADOOP-8251.
was (Author: devaraj):
On HADOOP-6941, my aim was to make the hadoop code base compile with IBM's
JDK, and ensure that there is no regression with Oracle's. I had done testing
of HDFS/MR with security enabled. I didn't do the tests that would go through
the SecurityUtil code path though (argh) [fixed in HADOOP-8251].
> Security uses proprietary Sun APIs
> --
>
> Key: HADOOP-7211
> URL: https://issues.apache.org/jira/browse/HADOOP-7211
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Reporter: Eli Collins
>Assignee: Luke Lu
>
> The security code uses the KrbException, Credentials, and PrincipalName
> classes from sun.security.krb5 and Krb5Util from sun.security.jgss.krb5.
> These may disappear in future Java releases. Also Hadoop does not compile
> using jdks that do not support them, for example with the following IBM JDK.
> {noformat}
> $ /home/eli/toolchain/java-x86_64-60/bin/java -version
> java version "1.6.0"
> Java(TM) SE Runtime Environment (build pxa6460sr9fp1-20110208_03(SR9 FP1))
> IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 Linux amd64-64
> jvmxa6460sr9-20110203_74623 (JIT enabled, AOT enabled)
> J9VM - 20110203_074623
> JIT - r9_20101028_17488ifx3
> GC - 20101027_AA)
> JCL - 20110203_01
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HADOOP-8364) Rationalize the way architecture-specific sub-components are built with ant in branch-1
[ https://issues.apache.org/jira/browse/HADOOP-8364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley reassigned HADOOP-8364: -- Assignee: Giridharan Kesavan > Rationalize the way architecture-specific sub-components are built with ant > in branch-1 > --- > > Key: HADOOP-8364 > URL: https://issues.apache.org/jira/browse/HADOOP-8364 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Matt Foley >Assignee: Giridharan Kesavan > > Three different compile flags, compile.native, compile.c++, and > compile.libhdfs, turn on or off different architecture-specific subcomponent > builds, but they are generally all off or all on and there's no evident need > for three different ways to do things. Also, in build.xml, jsvc and > task-controller are included in targets "package" and "bin-package" as > sub-ant tasks, while librecordio is included as a simple dependency. We > should work through these and get them done in one understandable way. > This is a matter of maintainability and understandability, and therefore > robustness under future changes in build.xml. No substantial change in > functionality is proposed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8307) The task-controller is not packaged in the tarball
[ https://issues.apache.org/jira/browse/HADOOP-8307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13269287#comment-13269287 ] Matt Foley commented on HADOOP-8307: The issue identified in the title of this bug was resolved by fixing the integration error in [MAPREDUCE-3377 change #1306744|https://issues.apache.org/jira/browse/MAPREDUCE-3377?focusedCommentId=13266972&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13266972]. So this patch isn't required. Owen says it should be done anyway, to correctly include task-controller in architecture-specific builds. But Matt feels the staging directory should in fact start out empty, even for a cumulative build of 64 and 32 bit architecture-specific environments, and that the build process should copy in all required subcomponents. Suggest resolving this as part of HADOOP-8364. > The task-controller is not packaged in the tarball > -- > > Key: HADOOP-8307 > URL: https://issues.apache.org/jira/browse/HADOOP-8307 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 1.0.3 >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Attachments: hadoop-8307.patch > > > Ant in some situations, puts artifacts such as task-controller into the > build/hadoop-*/ directory before the "package" target deletes it to start > over. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8364) Rationalize the way architecture-specific sub-components are built with ant in branch-1
[ https://issues.apache.org/jira/browse/HADOOP-8364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13269285#comment-13269285 ] Matt Foley commented on HADOOP-8364: While we're at it, the result of doing a 64-bit and 32-bit build in either order should be made the same; ie order-independent, for the cumulative aspects of build. See further, HADOOP-8307 suggested patch. > Rationalize the way architecture-specific sub-components are built with ant > in branch-1 > --- > > Key: HADOOP-8364 > URL: https://issues.apache.org/jira/browse/HADOOP-8364 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Matt Foley > > Three different compile flags, compile.native, compile.c++, and > compile.libhdfs, turn on or off different architecture-specific subcomponent > builds, but they are generally all off or all on and there's no evident need > for three different ways to do things. Also, in build.xml, jsvc and > task-controller are included in targets "package" and "bin-package" as > sub-ant tasks, while librecordio is included as a simple dependency. We > should work through these and get them done in one understandable way. > This is a matter of maintainability and understandability, and therefore > robustness under future changes in build.xml. No substantial change in > functionality is proposed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8230) Enable sync by default and disable append
[ https://issues.apache.org/jira/browse/HADOOP-8230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13269286#comment-13269286 ] Sanjay Radia commented on HADOOP-8230: -- > Couldn't the same be said for any new feature? Given that sync was fixed > prior to the 1.0 release, > I don't see why this should be considered an incompatible change. This was turned on in the 20 originally and then we had to turn it off due bugs. Given that the default in Hadoop 1 is off, why not leave it off and give a way to turn it on. The current default is off and I don't see a reason to change that default in 1.1. There are installations have that are using the current default. Can you please explain the reason the make this change? > Enable sync by default and disable append > - > > Key: HADOOP-8230 > URL: https://issues.apache.org/jira/browse/HADOOP-8230 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 1.0.0 >Reporter: Eli Collins >Assignee: Eli Collins > Fix For: 1.1.0 > > Attachments: hadoop-8230.txt > > > Per HDFS-3120 for 1.x let's: > - Always enable the sync path, which is currently only enabled if > dfs.support.append is set > - Remove the dfs.support.append configuration option. We'll keep the code > paths though in case we ever fix append on branch-1, in which case we can add > the config option back -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8230) Enable sync by default and disable append
[ https://issues.apache.org/jira/browse/HADOOP-8230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13269283#comment-13269283 ] Suresh Srinivas commented on HADOOP-8230: - Adding dfs.support.sync flag is along the lines of my previous comments. I am reluctantly okay with enabling it by default. This should be a blocker on 1.1. It might be easy to revert this patch, and add the new flag, as lot of paths to be enabled by the new flag are removed in this patch. > Enable sync by default and disable append > - > > Key: HADOOP-8230 > URL: https://issues.apache.org/jira/browse/HADOOP-8230 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 1.0.0 >Reporter: Eli Collins >Assignee: Eli Collins > Fix For: 1.1.0 > > Attachments: hadoop-8230.txt > > > Per HDFS-3120 for 1.x let's: > - Always enable the sync path, which is currently only enabled if > dfs.support.append is set > - Remove the dfs.support.append configuration option. We'll keep the code > paths though in case we ever fix append on branch-1, in which case we can add > the config option back -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8364) Rationalize the way architecture-specific sub-components are built with ant in branch-1
Matt Foley created HADOOP-8364: -- Summary: Rationalize the way architecture-specific sub-components are built with ant in branch-1 Key: HADOOP-8364 URL: https://issues.apache.org/jira/browse/HADOOP-8364 Project: Hadoop Common Issue Type: Improvement Reporter: Matt Foley Three different compile flags, compile.native, compile.c++, and compile.libhdfs, turn on or off different architecture-specific subcomponent builds, but they are generally all off or all on and there's no evident need for three different ways to do things. Also, in build.xml, jsvc and task-controller are included in targets "package" and "bin-package" as sub-ant tasks, while librecordio is included as a simple dependency. We should work through these and get them done in one understandable way. This is a matter of maintainability and understandability, and therefore robustness under future changes in build.xml. No substantial change in functionality is proposed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8190) Eclipse plugin fails to access remote cluster
[ https://issues.apache.org/jira/browse/HADOOP-8190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HADOOP-8190: --- Target Version/s: 1.1.0 (was: 1.0.3, 1.0.1) Fix Version/s: (was: 1.0.3) No patch submitted in time for 1.0. Please consider contributing on the 1.1 branch (branch-1). Thank you. > Eclipse plugin fails to access remote cluster > - > > Key: HADOOP-8190 > URL: https://issues.apache.org/jira/browse/HADOOP-8190 > Project: Hadoop Common > Issue Type: Bug > Components: contrib/eclipse-plugin >Affects Versions: 0.20.205.0 > Environment: Windows and Linux (all) >Reporter: Ambud Sharma >Priority: Critical > Labels: gsoc2012 > Original Estimate: 12h > Remaining Estimate: 12h > > Eclipse plugin fails to access remote file system. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8323) Revert HADOOP-7940 and improve javadocs and test for Text.clear()
[ https://issues.apache.org/jira/browse/HADOOP-8323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13269221#comment-13269221 ] Hudson commented on HADOOP-8323: Integrated in Hadoop-Mapreduce-trunk #1071 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1071/]) HADOOP-8323. Revert HADOOP-7940, cause it may cause a performance regression. (harsh) (Revision 1334603) Result = SUCCESS harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1334603 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/Text.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestText.java > Revert HADOOP-7940 and improve javadocs and test for Text.clear() > - > > Key: HADOOP-8323 > URL: https://issues.apache.org/jira/browse/HADOOP-8323 > Project: Hadoop Common > Issue Type: Bug > Components: io >Affects Versions: 2.0.0 >Reporter: Harsh J >Assignee: Harsh J >Priority: Critical > Labels: performance > Attachments: HADOOP-8323.patch, HADOOP-8323.patch, HADOOP-8323.patch > > > Per [~jdonofrio]'s comments on HADOOP-7940, we should revert it as it has > caused a performance regression (for scenarios where Text is reused, popular > in MR). > The clear() works as intended, as the API also offers a current length API. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7940) method clear() in org.apache.hadoop.io.Text does not work
[
https://issues.apache.org/jira/browse/HADOOP-7940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13269220#comment-13269220
]
Hudson commented on HADOOP-7940:
Integrated in Hadoop-Mapreduce-trunk #1071 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1071/])
HADOOP-8323. Revert HADOOP-7940, cause it may cause a performance
regression. (harsh) (Revision 1334603)
Result = SUCCESS
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1334603
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/Text.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestText.java
> method clear() in org.apache.hadoop.io.Text does not work
> -
>
> Key: HADOOP-7940
> URL: https://issues.apache.org/jira/browse/HADOOP-7940
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
>Affects Versions: 0.20.205.0
> Environment: Ubuntu, hadoop cloudera CDH3U2, Oracle SUN JDK 6U30
>Reporter: Aaron,
>Assignee: Csaba Miklos
> Labels: patch
> Fix For: 2.0.0
>
> Attachments: HADOOP-7940.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> LineReader reader = new LineReader(in, 4096);
> ...
> Text text = new Text();
> while((reader.readLine(text)) > 0) {
> ...
> text.clear();
> }
> }
> Even the clear() method is called each time, some bytes are still not filled
> as zero.
> So, when reader.readLine(text) is called in a loop, some bytes are dirty
> which was from last call.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8323) Revert HADOOP-7940 and improve javadocs and test for Text.clear()
[ https://issues.apache.org/jira/browse/HADOOP-8323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13269209#comment-13269209 ] Hudson commented on HADOOP-8323: Integrated in Hadoop-Hdfs-trunk #1036 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1036/]) HADOOP-8323. Revert HADOOP-7940, cause it may cause a performance regression. (harsh) (Revision 1334603) Result = FAILURE harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1334603 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/Text.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestText.java > Revert HADOOP-7940 and improve javadocs and test for Text.clear() > - > > Key: HADOOP-8323 > URL: https://issues.apache.org/jira/browse/HADOOP-8323 > Project: Hadoop Common > Issue Type: Bug > Components: io >Affects Versions: 2.0.0 >Reporter: Harsh J >Assignee: Harsh J >Priority: Critical > Labels: performance > Attachments: HADOOP-8323.patch, HADOOP-8323.patch, HADOOP-8323.patch > > > Per [~jdonofrio]'s comments on HADOOP-7940, we should revert it as it has > caused a performance regression (for scenarios where Text is reused, popular > in MR). > The clear() works as intended, as the API also offers a current length API. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7940) method clear() in org.apache.hadoop.io.Text does not work
[
https://issues.apache.org/jira/browse/HADOOP-7940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13269208#comment-13269208
]
Hudson commented on HADOOP-7940:
Integrated in Hadoop-Hdfs-trunk #1036 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1036/])
HADOOP-8323. Revert HADOOP-7940, cause it may cause a performance
regression. (harsh) (Revision 1334603)
Result = FAILURE
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1334603
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/Text.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestText.java
> method clear() in org.apache.hadoop.io.Text does not work
> -
>
> Key: HADOOP-7940
> URL: https://issues.apache.org/jira/browse/HADOOP-7940
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
>Affects Versions: 0.20.205.0
> Environment: Ubuntu, hadoop cloudera CDH3U2, Oracle SUN JDK 6U30
>Reporter: Aaron,
>Assignee: Csaba Miklos
> Labels: patch
> Fix For: 2.0.0
>
> Attachments: HADOOP-7940.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> LineReader reader = new LineReader(in, 4096);
> ...
> Text text = new Text();
> while((reader.readLine(text)) > 0) {
> ...
> text.clear();
> }
> }
> Even the clear() method is called each time, some bytes are still not filled
> as zero.
> So, when reader.readLine(text) is called in a loop, some bytes are dirty
> which was from last call.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8323) Revert HADOOP-7940 and improve javadocs and test for Text.clear()
[ https://issues.apache.org/jira/browse/HADOOP-8323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13269201#comment-13269201 ] Hadoop QA commented on HADOOP-8323: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12525770/HADOOP-8323.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. -1 javadoc. The javadoc tool appears to have generated 2 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/945//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/945//console This message is automatically generated. > Revert HADOOP-7940 and improve javadocs and test for Text.clear() > - > > Key: HADOOP-8323 > URL: https://issues.apache.org/jira/browse/HADOOP-8323 > Project: Hadoop Common > Issue Type: Bug > Components: io >Affects Versions: 2.0.0 >Reporter: Harsh J >Assignee: Harsh J >Priority: Critical > Labels: performance > Attachments: HADOOP-8323.patch, HADOOP-8323.patch, HADOOP-8323.patch > > > Per [~jdonofrio]'s comments on HADOOP-7940, we should revert it as it has > caused a performance regression (for scenarios where Text is reused, popular > in MR). > The clear() works as intended, as the API also offers a current length API. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8323) Revert HADOOP-7940 and improve javadocs and test for Text.clear()
[ https://issues.apache.org/jira/browse/HADOOP-8323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated HADOOP-8323: Target Version/s: 2.0.0, 3.0.0 (was: 3.0.0, 2.0.0) Status: Patch Available (was: Open) > Revert HADOOP-7940 and improve javadocs and test for Text.clear() > - > > Key: HADOOP-8323 > URL: https://issues.apache.org/jira/browse/HADOOP-8323 > Project: Hadoop Common > Issue Type: Bug > Components: io >Affects Versions: 2.0.0 >Reporter: Harsh J >Assignee: Harsh J >Priority: Critical > Labels: performance > Attachments: HADOOP-8323.patch, HADOOP-8323.patch, HADOOP-8323.patch > > > Per [~jdonofrio]'s comments on HADOOP-7940, we should revert it as it has > caused a performance regression (for scenarios where Text is reused, popular > in MR). > The clear() works as intended, as the API also offers a current length API. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7940) method clear() in org.apache.hadoop.io.Text does not work
[
https://issues.apache.org/jira/browse/HADOOP-7940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13269194#comment-13269194
]
Hudson commented on HADOOP-7940:
Integrated in Hadoop-Mapreduce-trunk-Commit #2211 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2211/])
HADOOP-8323. Revert HADOOP-7940, cause it may cause a performance
regression. (harsh) (Revision 1334603)
Result = ABORTED
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1334603
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/Text.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestText.java
> method clear() in org.apache.hadoop.io.Text does not work
> -
>
> Key: HADOOP-7940
> URL: https://issues.apache.org/jira/browse/HADOOP-7940
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
>Affects Versions: 0.20.205.0
> Environment: Ubuntu, hadoop cloudera CDH3U2, Oracle SUN JDK 6U30
>Reporter: Aaron,
>Assignee: Csaba Miklos
> Labels: patch
> Fix For: 2.0.0
>
> Attachments: HADOOP-7940.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> LineReader reader = new LineReader(in, 4096);
> ...
> Text text = new Text();
> while((reader.readLine(text)) > 0) {
> ...
> text.clear();
> }
> }
> Even the clear() method is called each time, some bytes are still not filled
> as zero.
> So, when reader.readLine(text) is called in a loop, some bytes are dirty
> which was from last call.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8323) Revert HADOOP-7940 and improve javadocs and test for Text.clear()
[ https://issues.apache.org/jira/browse/HADOOP-8323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13269195#comment-13269195 ] Hudson commented on HADOOP-8323: Integrated in Hadoop-Mapreduce-trunk-Commit #2211 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2211/]) HADOOP-8323. Revert HADOOP-7940, cause it may cause a performance regression. (harsh) (Revision 1334603) Result = ABORTED harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1334603 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/Text.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestText.java > Revert HADOOP-7940 and improve javadocs and test for Text.clear() > - > > Key: HADOOP-8323 > URL: https://issues.apache.org/jira/browse/HADOOP-8323 > Project: Hadoop Common > Issue Type: Bug > Components: io >Affects Versions: 2.0.0 >Reporter: Harsh J >Assignee: Harsh J >Priority: Critical > Labels: performance > Attachments: HADOOP-8323.patch, HADOOP-8323.patch, HADOOP-8323.patch > > > Per [~jdonofrio]'s comments on HADOOP-7940, we should revert it as it has > caused a performance regression (for scenarios where Text is reused, popular > in MR). > The clear() works as intended, as the API also offers a current length API. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8323) Revert HADOOP-7940 and improve javadocs and test for Text.clear()
[
https://issues.apache.org/jira/browse/HADOOP-8323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J updated HADOOP-8323:
Attachment: HADOOP-8323.patch
New patch that just adds a test + javadoc note.
{{mvn clean test -Dtest=TestText}} passes.
{{mvn javadoc:javadoc}} throws up only HADOOP-8359's two warnings.
> Revert HADOOP-7940 and improve javadocs and test for Text.clear()
> -
>
> Key: HADOOP-8323
> URL: https://issues.apache.org/jira/browse/HADOOP-8323
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
>Affects Versions: 2.0.0
>Reporter: Harsh J
>Assignee: Harsh J
>Priority: Critical
> Labels: performance
> Attachments: HADOOP-8323.patch, HADOOP-8323.patch, HADOOP-8323.patch
>
>
> Per [~jdonofrio]'s comments on HADOOP-7940, we should revert it as it has
> caused a performance regression (for scenarios where Text is reused, popular
> in MR).
> The clear() works as intended, as the API also offers a current length API.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8323) Revert HADOOP-7940 and improve javadocs and test for Text.clear()
[ https://issues.apache.org/jira/browse/HADOOP-8323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated HADOOP-8323: Target Version/s: 2.0.0, 3.0.0 (was: 3.0.0, 2.0.0) Status: Open (was: Patch Available) (Successfully reverted from trunk and branch-2. Javadoc and new test patch to follow) > Revert HADOOP-7940 and improve javadocs and test for Text.clear() > - > > Key: HADOOP-8323 > URL: https://issues.apache.org/jira/browse/HADOOP-8323 > Project: Hadoop Common > Issue Type: Bug > Components: io >Affects Versions: 2.0.0 >Reporter: Harsh J >Assignee: Harsh J >Priority: Critical > Labels: performance > Attachments: HADOOP-8323.patch, HADOOP-8323.patch > > > Per [~jdonofrio]'s comments on HADOOP-7940, we should revert it as it has > caused a performance regression (for scenarios where Text is reused, popular > in MR). > The clear() works as intended, as the API also offers a current length API. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7940) method clear() in org.apache.hadoop.io.Text does not work
[
https://issues.apache.org/jira/browse/HADOOP-7940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13269189#comment-13269189
]
Harsh J commented on HADOOP-7940:
-
*Note:* This was reverted by HADOOP-8323. Please see that ticket for more
information.
> method clear() in org.apache.hadoop.io.Text does not work
> -
>
> Key: HADOOP-7940
> URL: https://issues.apache.org/jira/browse/HADOOP-7940
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
>Affects Versions: 0.20.205.0
> Environment: Ubuntu, hadoop cloudera CDH3U2, Oracle SUN JDK 6U30
>Reporter: Aaron,
>Assignee: Csaba Miklos
> Labels: patch
> Fix For: 2.0.0
>
> Attachments: HADOOP-7940.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> LineReader reader = new LineReader(in, 4096);
> ...
> Text text = new Text();
> while((reader.readLine(text)) > 0) {
> ...
> text.clear();
> }
> }
> Even the clear() method is called each time, some bytes are still not filled
> as zero.
> So, when reader.readLine(text) is called in a loop, some bytes are dirty
> which was from last call.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8323) Revert HADOOP-7940 and improve javadocs and test for Text.clear()
[ https://issues.apache.org/jira/browse/HADOOP-8323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13269187#comment-13269187 ] Harsh J commented on HADOOP-8323: - Reverting at the moment. Will attach new patch after that. > Revert HADOOP-7940 and improve javadocs and test for Text.clear() > - > > Key: HADOOP-8323 > URL: https://issues.apache.org/jira/browse/HADOOP-8323 > Project: Hadoop Common > Issue Type: Bug > Components: io >Affects Versions: 2.0.0 >Reporter: Harsh J >Assignee: Harsh J >Priority: Critical > Labels: performance > Attachments: HADOOP-8323.patch, HADOOP-8323.patch > > > Per [~jdonofrio]'s comments on HADOOP-7940, we should revert it as it has > caused a performance regression (for scenarios where Text is reused, popular > in MR). > The clear() works as intended, as the API also offers a current length API. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7940) method clear() in org.apache.hadoop.io.Text does not work
[
https://issues.apache.org/jira/browse/HADOOP-7940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13269185#comment-13269185
]
Hudson commented on HADOOP-7940:
Integrated in Hadoop-Common-trunk-Commit #2193 (See
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2193/])
HADOOP-8323. Revert HADOOP-7940, cause it may cause a performance
regression. (harsh) (Revision 1334603)
Result = SUCCESS
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1334603
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/Text.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestText.java
> method clear() in org.apache.hadoop.io.Text does not work
> -
>
> Key: HADOOP-7940
> URL: https://issues.apache.org/jira/browse/HADOOP-7940
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
>Affects Versions: 0.20.205.0
> Environment: Ubuntu, hadoop cloudera CDH3U2, Oracle SUN JDK 6U30
>Reporter: Aaron,
>Assignee: Csaba Miklos
> Labels: patch
> Fix For: 2.0.0
>
> Attachments: HADOOP-7940.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> LineReader reader = new LineReader(in, 4096);
> ...
> Text text = new Text();
> while((reader.readLine(text)) > 0) {
> ...
> text.clear();
> }
> }
> Even the clear() method is called each time, some bytes are still not filled
> as zero.
> So, when reader.readLine(text) is called in a loop, some bytes are dirty
> which was from last call.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8323) Revert HADOOP-7940 and improve javadocs and test for Text.clear()
[ https://issues.apache.org/jira/browse/HADOOP-8323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13269186#comment-13269186 ] Hudson commented on HADOOP-8323: Integrated in Hadoop-Common-trunk-Commit #2193 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2193/]) HADOOP-8323. Revert HADOOP-7940, cause it may cause a performance regression. (harsh) (Revision 1334603) Result = SUCCESS harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1334603 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/Text.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestText.java > Revert HADOOP-7940 and improve javadocs and test for Text.clear() > - > > Key: HADOOP-8323 > URL: https://issues.apache.org/jira/browse/HADOOP-8323 > Project: Hadoop Common > Issue Type: Bug > Components: io >Affects Versions: 2.0.0 >Reporter: Harsh J >Assignee: Harsh J >Priority: Critical > Labels: performance > Attachments: HADOOP-8323.patch, HADOOP-8323.patch > > > Per [~jdonofrio]'s comments on HADOOP-7940, we should revert it as it has > caused a performance regression (for scenarios where Text is reused, popular > in MR). > The clear() works as intended, as the API also offers a current length API. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7940) method clear() in org.apache.hadoop.io.Text does not work
[
https://issues.apache.org/jira/browse/HADOOP-7940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13269183#comment-13269183
]
Hudson commented on HADOOP-7940:
Integrated in Hadoop-Hdfs-trunk-Commit #2268 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2268/])
HADOOP-8323. Revert HADOOP-7940, cause it may cause a performance
regression. (harsh) (Revision 1334603)
Result = SUCCESS
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1334603
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/Text.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestText.java
> method clear() in org.apache.hadoop.io.Text does not work
> -
>
> Key: HADOOP-7940
> URL: https://issues.apache.org/jira/browse/HADOOP-7940
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
>Affects Versions: 0.20.205.0
> Environment: Ubuntu, hadoop cloudera CDH3U2, Oracle SUN JDK 6U30
>Reporter: Aaron,
>Assignee: Csaba Miklos
> Labels: patch
> Fix For: 2.0.0
>
> Attachments: HADOOP-7940.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> LineReader reader = new LineReader(in, 4096);
> ...
> Text text = new Text();
> while((reader.readLine(text)) > 0) {
> ...
> text.clear();
> }
> }
> Even the clear() method is called each time, some bytes are still not filled
> as zero.
> So, when reader.readLine(text) is called in a loop, some bytes are dirty
> which was from last call.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8323) Revert HADOOP-7940 and improve javadocs and test for Text.clear()
[ https://issues.apache.org/jira/browse/HADOOP-8323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13269184#comment-13269184 ] Hudson commented on HADOOP-8323: Integrated in Hadoop-Hdfs-trunk-Commit #2268 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2268/]) HADOOP-8323. Revert HADOOP-7940, cause it may cause a performance regression. (harsh) (Revision 1334603) Result = SUCCESS harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1334603 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/Text.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestText.java > Revert HADOOP-7940 and improve javadocs and test for Text.clear() > - > > Key: HADOOP-8323 > URL: https://issues.apache.org/jira/browse/HADOOP-8323 > Project: Hadoop Common > Issue Type: Bug > Components: io >Affects Versions: 2.0.0 >Reporter: Harsh J >Assignee: Harsh J >Priority: Critical > Labels: performance > Attachments: HADOOP-8323.patch, HADOOP-8323.patch > > > Per [~jdonofrio]'s comments on HADOOP-7940, we should revert it as it has > caused a performance regression (for scenarios where Text is reused, popular > in MR). > The clear() works as intended, as the API also offers a current length API. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-1381) The distance between sync blocks in SequenceFiles should be configurable rather than hard coded to 2000 bytes
[ https://issues.apache.org/jira/browse/HADOOP-1381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13269152#comment-13269152 ] Hadoop QA commented on HADOOP-1381: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12525761/HADOOP-1381.r5.diff against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. -1 javadoc. The javadoc tool appears to have generated 2 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/944//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/944//console This message is automatically generated. > The distance between sync blocks in SequenceFiles should be configurable > rather than hard coded to 2000 bytes > - > > Key: HADOOP-1381 > URL: https://issues.apache.org/jira/browse/HADOOP-1381 > Project: Hadoop Common > Issue Type: Improvement > Components: io >Affects Versions: 0.22.0 >Reporter: Owen O'Malley >Assignee: Harsh J > Attachments: HADOOP-1381.r1.diff, HADOOP-1381.r2.diff, > HADOOP-1381.r3.diff, HADOOP-1381.r4.diff, HADOOP-1381.r5.diff, > HADOOP-1381.r5.diff > > > Currently SequenceFiles put in sync blocks every 2000 bytes. It would be much > better if it was configurable with a much higher default (1mb or so?). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
