[jira] [Commented] (HADOOP-10048) LocalDirAllocator should avoid holding locks while accessing the filesystem
[ https://issues.apache.org/jira/browse/HADOOP-10048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15197090#comment-15197090 ] Yi Li commented on HADOOP-10048: It seems the failed test is not related with this patch. Any suggestions? > LocalDirAllocator should avoid holding locks while accessing the filesystem > --- > > Key: HADOOP-10048 > URL: https://issues.apache.org/jira/browse/HADOOP-10048 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.3.0 >Reporter: Jason Lowe >Assignee: Jason Lowe > Labels: BB2015-05-TBR > Attachments: HADOOP-10048.003.patch, HADOOP-10048.patch, > HADOOP-10048.trunk.patch > > > As noted in MAPREDUCE-5584 and HADOOP-7016, LocalDirAllocator can be a > bottleneck for multithreaded setups like the ShuffleHandler. We should > consider moving to a lockless design or minimizing the critical sections to a > very small amount of time that does not involve I/O operations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12926) lz4.c does not detect 64-bit mode properly
[ https://issues.apache.org/jira/browse/HADOOP-12926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Burlison updated HADOOP-12926: --- Status: Patch Available (was: Open) > lz4.c does not detect 64-bit mode properly > -- > > Key: HADOOP-12926 > URL: https://issues.apache.org/jira/browse/HADOOP-12926 > Project: Hadoop Common > Issue Type: Sub-task > Components: native >Affects Versions: 2.7.2 > Environment: All >Reporter: Alan Burlison >Assignee: Alan Burlison > Attachments: HADOOP-12926.001.patch > > > lz4.c should check to see if the _LP64 macro is defined to correctly detect > when it is being built in 64-bit mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12926) lz4.c does not detect 64-bit mode properly
[ https://issues.apache.org/jira/browse/HADOOP-12926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Burlison updated HADOOP-12926: --- Attachment: HADOOP-12926.001.patch > lz4.c does not detect 64-bit mode properly > -- > > Key: HADOOP-12926 > URL: https://issues.apache.org/jira/browse/HADOOP-12926 > Project: Hadoop Common > Issue Type: Sub-task > Components: native >Affects Versions: 2.7.2 > Environment: All >Reporter: Alan Burlison >Assignee: Alan Burlison > Attachments: HADOOP-12926.001.patch > > > lz4.c should check to see if the _LP64 macro is defined to correctly detect > when it is being built in 64-bit mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12925) Checks for SPARC architecture need to include 64-bit SPARC
[
https://issues.apache.org/jira/browse/HADOOP-12925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alan Burlison updated HADOOP-12925:
---
Status: Patch Available (was: Open)
> Checks for SPARC architecture need to include 64-bit SPARC
> --
>
> Key: HADOOP-12925
> URL: https://issues.apache.org/jira/browse/HADOOP-12925
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: conf
>Affects Versions: 2.7.2
> Environment: 64-bit SPARC
>Reporter: Alan Burlison
>Assignee: Alan Burlison
> Attachments: HADOOP-12925.001.patch
>
>
> FastByteComparisons.java and NativeCrc32.java check for the SPARC platform by
> comparing the os.arch property against "sparc". That doesn't detect 64-bit
> SPARC ("sparcv9"), the test should be "startsWith", not "equals"
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12727) Minor cleanups needed for CMake 3.X
[
https://issues.apache.org/jira/browse/HADOOP-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alan Burlison updated HADOOP-12727:
---
Status: Patch Available (was: Open)
> Minor cleanups needed for CMake 3.X
> ---
>
> Key: HADOOP-12727
> URL: https://issues.apache.org/jira/browse/HADOOP-12727
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: native
>Affects Versions: 2.7.1
>Reporter: Alan Burlison
>Assignee: Alan Burlison
>Priority: Minor
> Attachments: HADOOP-12727.001.patch
>
>
> On switching from CMake 2.8.6 to 3.3.2 a couple of minor issues popped up:
> \\
> \\
> * There's a syntax error in
> {{hadoop-common-project/hadoop-common/src/CMakeLists.txt}} that generates a
> warning in 3.X
> * {{CMAKE_SHARED_LINKER_FLAGS}} is being incorrectly set in
> {{hadoop-common-project/hadoop-common/HadoopCommon.cmake}} - despite the name
> it contains the flags passed to {{ar}} not to the linker. 2.8.6 ignores the
> incorrect flags, 3.3.2 doesn't and building static libraries fails as a
> result. See http://public.kitware.com/pipermail/cmake/2016-January/062447.html
> Patch to follow.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12925) Checks for SPARC architecture need to include 64-bit SPARC
[
https://issues.apache.org/jira/browse/HADOOP-12925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alan Burlison updated HADOOP-12925:
---
Attachment: HADOOP-12925.001.patch
> Checks for SPARC architecture need to include 64-bit SPARC
> --
>
> Key: HADOOP-12925
> URL: https://issues.apache.org/jira/browse/HADOOP-12925
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: conf
>Affects Versions: 2.7.2
> Environment: 64-bit SPARC
>Reporter: Alan Burlison
>Assignee: Alan Burlison
> Attachments: HADOOP-12925.001.patch
>
>
> FastByteComparisons.java and NativeCrc32.java check for the SPARC platform by
> comparing the os.arch property against "sparc". That doesn't detect 64-bit
> SPARC ("sparcv9"), the test should be "startsWith", not "equals"
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12727) Minor cleanups needed for CMake 3.X
[
https://issues.apache.org/jira/browse/HADOOP-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alan Burlison updated HADOOP-12727:
---
Status: Open (was: Patch Available)
> Minor cleanups needed for CMake 3.X
> ---
>
> Key: HADOOP-12727
> URL: https://issues.apache.org/jira/browse/HADOOP-12727
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: native
>Affects Versions: 2.7.1
>Reporter: Alan Burlison
>Assignee: Alan Burlison
>Priority: Minor
> Attachments: HADOOP-12727.001.patch
>
>
> On switching from CMake 2.8.6 to 3.3.2 a couple of minor issues popped up:
> \\
> \\
> * There's a syntax error in
> {{hadoop-common-project/hadoop-common/src/CMakeLists.txt}} that generates a
> warning in 3.X
> * {{CMAKE_SHARED_LINKER_FLAGS}} is being incorrectly set in
> {{hadoop-common-project/hadoop-common/HadoopCommon.cmake}} - despite the name
> it contains the flags passed to {{ar}} not to the linker. 2.8.6 ignores the
> incorrect flags, 3.3.2 doesn't and building static libraries fails as a
> result. See http://public.kitware.com/pipermail/cmake/2016-January/062447.html
> Patch to follow.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12727) Minor cleanups needed for CMake 3.X
[
https://issues.apache.org/jira/browse/HADOOP-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alan Burlison updated HADOOP-12727:
---
Attachment: HADOOP-12727.001.patch
> Minor cleanups needed for CMake 3.X
> ---
>
> Key: HADOOP-12727
> URL: https://issues.apache.org/jira/browse/HADOOP-12727
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: native
>Affects Versions: 2.7.1
>Reporter: Alan Burlison
>Assignee: Alan Burlison
>Priority: Minor
> Attachments: HADOOP-12727.001.patch
>
>
> On switching from CMake 2.8.6 to 3.3.2 a couple of minor issues popped up:
> \\
> \\
> * There's a syntax error in
> {{hadoop-common-project/hadoop-common/src/CMakeLists.txt}} that generates a
> warning in 3.X
> * {{CMAKE_SHARED_LINKER_FLAGS}} is being incorrectly set in
> {{hadoop-common-project/hadoop-common/HadoopCommon.cmake}} - despite the name
> it contains the flags passed to {{ar}} not to the linker. 2.8.6 ignores the
> incorrect flags, 3.3.2 doesn't and building static libraries fails as a
> result. See http://public.kitware.com/pipermail/cmake/2016-January/062447.html
> Patch to follow.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12727) Minor cleanups needed for CMake 3.X
[
https://issues.apache.org/jira/browse/HADOOP-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alan Burlison updated HADOOP-12727:
---
Status: Patch Available (was: Open)
> Minor cleanups needed for CMake 3.X
> ---
>
> Key: HADOOP-12727
> URL: https://issues.apache.org/jira/browse/HADOOP-12727
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: native
>Affects Versions: 2.7.1
>Reporter: Alan Burlison
>Assignee: Alan Burlison
>Priority: Minor
> Attachments: HADOOP-12727.001.patch
>
>
> On switching from CMake 2.8.6 to 3.3.2 a couple of minor issues popped up:
> \\
> \\
> * There's a syntax error in
> {{hadoop-common-project/hadoop-common/src/CMakeLists.txt}} that generates a
> warning in 3.X
> * {{CMAKE_SHARED_LINKER_FLAGS}} is being incorrectly set in
> {{hadoop-common-project/hadoop-common/HadoopCommon.cmake}} - despite the name
> it contains the flags passed to {{ar}} not to the linker. 2.8.6 ignores the
> incorrect flags, 3.3.2 doesn't and building static libraries fails as a
> result. See http://public.kitware.com/pipermail/cmake/2016-January/062447.html
> Patch to follow.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12924) Add default coder key for creating raw coders
[ https://issues.apache.org/jira/browse/HADOOP-12924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15196905#comment-15196905 ] Rui Li commented on HADOOP-12924: - Thanks Zhe for the review and comments. Yes I think the concern is valid. If the raw coders are not compatible with each other, we may have to encode it into the policy. And don't allow users to configure it separately. But I'm not sure if that will make it difficult for users to try new coders like ISA-L. Of course let's wait for Kai's inputs. > Add default coder key for creating raw coders > - > > Key: HADOOP-12924 > URL: https://issues.apache.org/jira/browse/HADOOP-12924 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Rui Li >Assignee: Rui Li >Priority: Minor > Attachments: HADOOP-12924.1.patch > > > As suggested > [here|https://issues.apache.org/jira/browse/HADOOP-12826?focusedCommentId=15194402=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15194402]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12924) Add default coder key for creating raw coders
[
https://issues.apache.org/jira/browse/HADOOP-12924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15196885#comment-15196885
]
Hadoop QA commented on HADOOP-12924:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m 27s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
51s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 53s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 41s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 57s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
35s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s
{color} | {color:green} trunk passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 39s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 34s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 34s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 21s
{color} | {color:red} hadoop-common-project/hadoop-common: patch generated 2
new + 77 unchanged - 0 fixed = 79 total (was 77) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 54s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed with JDK v1.8.0_74 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 6m 24s {color}
| {color:red} hadoop-common in the patch failed with JDK v1.8.0_74. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 10s
{color} | {color:green} hadoop-common in the patch passed with JDK v1.7.0_95.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 69m 10s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_74 Failed junit tests | hadoop.net.TestDNS |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12793682/HADOOP-12924.1.patch |
| JIRA Issue | HADOOP-12924 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 9360da597ae3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT
[jira] [Commented] (HADOOP-12924) Add default coder key for creating raw coders
[
https://issues.apache.org/jira/browse/HADOOP-12924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15196858#comment-15196858
]
Zhe Zhang commented on HADOOP-12924:
Thanks Rui. Patch itself LGTM.
However I realized some issues in the current code. Pinging [~drankye] for
discussion.
Kai mentioned that new RS Java coder and legacy RS coder from HDFS-RAID are not
interoperable. However, they map to the same EC policy at inode level. E.g. if
a client writes a file using HDFS-RAID coder, we will record EC policy RS-6-3.
Then if another client tries to decode the file with new coder it will fail.
These 2 clients could have different {{IO_ERASURECODE_CODEC_RS_RAWCODER_KEY}}
configured. IIUC, they should be treated as 2 different policies rather than 2
different implementations?
> Add default coder key for creating raw coders
> -
>
> Key: HADOOP-12924
> URL: https://issues.apache.org/jira/browse/HADOOP-12924
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Rui Li
>Assignee: Rui Li
>Priority: Minor
> Attachments: HADOOP-12924.1.patch
>
>
> As suggested
> [here|https://issues.apache.org/jira/browse/HADOOP-12826?focusedCommentId=15194402=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15194402].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
