[jira] [Commented] (HDFS-1245) Plugable block id generation
[ https://issues.apache.org/jira/browse/HDFS-1245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551855#comment-13551855 ] Konstantin Boudnik commented on HDFS-1245: -- +1 patch looks good. I guess it was meant to be marked against trunk, right? Currently it 'Affects 0.22'. The refactoring seems to be handy enough to be incorporated into branch-2, does it not? > Plugable block id generation > - > > Key: HDFS-1245 > URL: https://issues.apache.org/jira/browse/HDFS-1245 > Project: Hadoop HDFS > Issue Type: New Feature > Components: namenode >Affects Versions: 0.22.0 >Reporter: Dmytro Molkov >Assignee: Konstantin Shvachko > Attachments: blockIdGenerator.patch > > > The idea is to have a way to easily create block id generation engines that > may fit a certain purpose. One of them could be HDFS-898 started by > Konstantin, but potentially others. > We chatted with Dhruba about this for a while and came up with the following > approach: > There should be a BlockIDGenerator interface that has following methods: > void blockAdded(Block) > void blockRemoved(Block) > Block nextBlock() > First two methods are needed for block generation engines that hold a certain > state. During the restart, when namenode reads the fsimage it will notify > generator about all the blocks it reads from the image and during runtime > namenode will notify the generator about block removals on file deletion. > The instance of the generator will also have a reference to the block > registry, the interface that BlockManager implements. The only method there > is __blockExists(Block)__, so that the current random block id generation can > be implemented, since it needs to check with the block manager if the id is > already present. > What does the community think about this proposal? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4375) Use token request messages defined in hadoop common
[
https://issues.apache.org/jira/browse/HDFS-4375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551816#comment-13551816
]
Hadoop QA commented on HDFS-4375:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12564547/HDFS-4375.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/3830//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3830//console
This message is automatically generated.
> Use token request messages defined in hadoop common
> ---
>
> Key: HDFS-4375
> URL: https://issues.apache.org/jira/browse/HDFS-4375
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode, security
>Affects Versions: 2.0.2-alpha
>Reporter: Suresh Srinivas
>Assignee: Suresh Srinivas
> Attachments: HDFS-4375.patch, HDFS-4375.patch, HDFS-4375.patch
>
>
> HDFS changes related to HADOOP-9192 to reuse the protobuf messages defined in
> common.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4350) Make enabling of stale marking on read and write paths independent
[
https://issues.apache.org/jira/browse/HDFS-4350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551807#comment-13551807
]
Hadoop QA commented on HDFS-4350:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12564544/hdfs-4350-4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/3829//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3829//console
This message is automatically generated.
> Make enabling of stale marking on read and write paths independent
> --
>
> Key: HDFS-4350
> URL: https://issues.apache.org/jira/browse/HDFS-4350
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Andrew Wang
>Assignee: Andrew Wang
> Attachments: hdfs-4350-1.patch, hdfs-4350-2.patch, hdfs-4350-3.patch,
> hdfs-4350-4.patch
>
>
> Marking of datanodes as stale for the read and write path was introduced in
> HDFS-3703 and HDFS-3912 respectively. This is enabled using two new keys,
> {{DFS_NAMENODE_CHECK_STALE_DATANODE_KEY}} and
> {{DFS_NAMENODE_AVOID_STALE_DATANODE_FOR_WRITE_KEY}}. However, there currently
> exists a dependency, since you cannot enable write marking without also
> enabling read marking, since the first key enables both checking of staleness
> and read marking.
> I propose renaming the first key to
> {{DFS_NAMENODE_AVOID_STALE_DATANODE_FOR_READ_KEY}}, and make checking enabled
> if either of the keys are set. This will allow read and write marking to be
> enabled independently.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1245) Plugable block id generation
[
https://issues.apache.org/jira/browse/HDFS-1245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551805#comment-13551805
]
Hadoop QA commented on HDFS-1245:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12564540/blockIdGenerator.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/3827//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3827//console
This message is automatically generated.
> Plugable block id generation
> -
>
> Key: HDFS-1245
> URL: https://issues.apache.org/jira/browse/HDFS-1245
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: namenode
>Affects Versions: 0.22.0
>Reporter: Dmytro Molkov
>Assignee: Konstantin Shvachko
> Attachments: blockIdGenerator.patch
>
>
> The idea is to have a way to easily create block id generation engines that
> may fit a certain purpose. One of them could be HDFS-898 started by
> Konstantin, but potentially others.
> We chatted with Dhruba about this for a while and came up with the following
> approach:
> There should be a BlockIDGenerator interface that has following methods:
> void blockAdded(Block)
> void blockRemoved(Block)
> Block nextBlock()
> First two methods are needed for block generation engines that hold a certain
> state. During the restart, when namenode reads the fsimage it will notify
> generator about all the blocks it reads from the image and during runtime
> namenode will notify the generator about block removals on file deletion.
> The instance of the generator will also have a reference to the block
> registry, the interface that BlockManager implements. The only method there
> is __blockExists(Block)__, so that the current random block id generation can
> be implemented, since it needs to check with the block manager if the id is
> already present.
> What does the community think about this proposal?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4391) TestDataTransferKeepalive fails when tests are executed in a certain order.
[ https://issues.apache.org/jira/browse/HDFS-4391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551798#comment-13551798 ] Suresh Srinivas commented on HDFS-4391: --- The test order also changes when we move to JDK7. > TestDataTransferKeepalive fails when tests are executed in a certain order. > --- > > Key: HDFS-4391 > URL: https://issues.apache.org/jira/browse/HDFS-4391 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 3.0.0 >Reporter: Konstantin Shvachko > > TestDataTransferKeepalive if testKeepaliveTimeouts() is executed AFTER > testManyClosedSocketsInCache(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4375) Use token request messages defined in hadoop common
[ https://issues.apache.org/jira/browse/HDFS-4375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-4375: -- Attachment: HDFS-4375.patch > Use token request messages defined in hadoop common > --- > > Key: HDFS-4375 > URL: https://issues.apache.org/jira/browse/HDFS-4375 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode, security >Affects Versions: 2.0.2-alpha >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Attachments: HDFS-4375.patch, HDFS-4375.patch, HDFS-4375.patch > > > HDFS changes related to HADOOP-9192 to reuse the protobuf messages defined in > common. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4375) Use token request messages defined in hadoop common
[ https://issues.apache.org/jira/browse/HDFS-4375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-4375: -- Attachment: (was: HDFS-4375.patch) > Use token request messages defined in hadoop common > --- > > Key: HDFS-4375 > URL: https://issues.apache.org/jira/browse/HDFS-4375 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode, security >Affects Versions: 2.0.2-alpha >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Attachments: HDFS-4375.patch, HDFS-4375.patch, HDFS-4375.patch > > > HDFS changes related to HADOOP-9192 to reuse the protobuf messages defined in > common. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4350) Make enabling of stale marking on read and write paths independent
[
https://issues.apache.org/jira/browse/HDFS-4350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-4350:
--
Attachment: hdfs-4350-4.patch
Switched the config strings in javadoc over to {{@link}}, based on Jing's
advice.
> Make enabling of stale marking on read and write paths independent
> --
>
> Key: HDFS-4350
> URL: https://issues.apache.org/jira/browse/HDFS-4350
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Andrew Wang
>Assignee: Andrew Wang
> Attachments: hdfs-4350-1.patch, hdfs-4350-2.patch, hdfs-4350-3.patch,
> hdfs-4350-4.patch
>
>
> Marking of datanodes as stale for the read and write path was introduced in
> HDFS-3703 and HDFS-3912 respectively. This is enabled using two new keys,
> {{DFS_NAMENODE_CHECK_STALE_DATANODE_KEY}} and
> {{DFS_NAMENODE_AVOID_STALE_DATANODE_FOR_WRITE_KEY}}. However, there currently
> exists a dependency, since you cannot enable write marking without also
> enabling read marking, since the first key enables both checking of staleness
> and read marking.
> I propose renaming the first key to
> {{DFS_NAMENODE_AVOID_STALE_DATANODE_FOR_READ_KEY}}, and make checking enabled
> if either of the keys are set. This will allow read and write marking to be
> enabled independently.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4375) Use token request messages defined in hadoop common
[
https://issues.apache.org/jira/browse/HDFS-4375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551776#comment-13551776
]
Hadoop QA commented on HDFS-4375:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12564543/HDFS-4375.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3828//console
This message is automatically generated.
> Use token request messages defined in hadoop common
> ---
>
> Key: HDFS-4375
> URL: https://issues.apache.org/jira/browse/HDFS-4375
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode, security
>Affects Versions: 2.0.2-alpha
>Reporter: Suresh Srinivas
>Assignee: Suresh Srinivas
> Attachments: HDFS-4375.patch, HDFS-4375.patch, HDFS-4375.patch
>
>
> HDFS changes related to HADOOP-9192 to reuse the protobuf messages defined in
> common.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4375) Use token request messages defined in hadoop common
[ https://issues.apache.org/jira/browse/HDFS-4375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-4375: -- Attachment: HDFS-4375.patch Rebased patch. > Use token request messages defined in hadoop common > --- > > Key: HDFS-4375 > URL: https://issues.apache.org/jira/browse/HDFS-4375 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode, security >Affects Versions: 2.0.2-alpha >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Attachments: HDFS-4375.patch, HDFS-4375.patch, HDFS-4375.patch > > > HDFS changes related to HADOOP-9192 to reuse the protobuf messages defined in > common. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1245) Plugable block id generation
[ https://issues.apache.org/jira/browse/HDFS-1245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-1245: -- Assignee: Konstantin Shvachko (was: Dmytro Molkov) Target Version/s: 2.0.3-alpha Affects Version/s: 0.22.0 Status: Patch Available (was: Open) > Plugable block id generation > - > > Key: HDFS-1245 > URL: https://issues.apache.org/jira/browse/HDFS-1245 > Project: Hadoop HDFS > Issue Type: New Feature > Components: namenode >Affects Versions: 0.22.0 >Reporter: Dmytro Molkov >Assignee: Konstantin Shvachko > Attachments: blockIdGenerator.patch > > > The idea is to have a way to easily create block id generation engines that > may fit a certain purpose. One of them could be HDFS-898 started by > Konstantin, but potentially others. > We chatted with Dhruba about this for a while and came up with the following > approach: > There should be a BlockIDGenerator interface that has following methods: > void blockAdded(Block) > void blockRemoved(Block) > Block nextBlock() > First two methods are needed for block generation engines that hold a certain > state. During the restart, when namenode reads the fsimage it will notify > generator about all the blocks it reads from the image and during runtime > namenode will notify the generator about block removals on file deletion. > The instance of the generator will also have a reference to the block > registry, the interface that BlockManager implements. The only method there > is __blockExists(Block)__, so that the current random block id generation can > be implemented, since it needs to check with the block manager if the id is > already present. > What does the community think about this proposal? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1245) Plugable block id generation
[
https://issues.apache.org/jira/browse/HDFS-1245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko updated HDFS-1245:
--
Attachment: blockIdGenerator.patch
Here is the patch that introduces {{RandomBlockIdGenerator}}. No big changes so
far. The generator is instantiated in {{format()}} method and after loading the
image during regular startup.
I assume that the generator can be chosen during file system formatting, and
then instantiated (always the same for the file system instance) as indicated
in fsimage or VERSION. The latter is not implemented. As I said there is no
need for that logic until other generators appear.
> Plugable block id generation
> -
>
> Key: HDFS-1245
> URL: https://issues.apache.org/jira/browse/HDFS-1245
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: namenode
>Reporter: Dmytro Molkov
>Assignee: Dmytro Molkov
> Attachments: blockIdGenerator.patch
>
>
> The idea is to have a way to easily create block id generation engines that
> may fit a certain purpose. One of them could be HDFS-898 started by
> Konstantin, but potentially others.
> We chatted with Dhruba about this for a while and came up with the following
> approach:
> There should be a BlockIDGenerator interface that has following methods:
> void blockAdded(Block)
> void blockRemoved(Block)
> Block nextBlock()
> First two methods are needed for block generation engines that hold a certain
> state. During the restart, when namenode reads the fsimage it will notify
> generator about all the blocks it reads from the image and during runtime
> namenode will notify the generator about block removals on file deletion.
> The instance of the generator will also have a reference to the block
> registry, the interface that BlockManager implements. The only method there
> is __blockExists(Block)__, so that the current random block id generation can
> be implemented, since it needs to check with the block manager if the id is
> already present.
> What does the community think about this proposal?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1245) Plugable block id generation
[
https://issues.apache.org/jira/browse/HDFS-1245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551757#comment-13551757
]
Konstantin Shvachko commented on HDFS-1245:
---
I would like to resurrect this jira.
Now that we have {{GenerationStamp}} and {{INodeId}} generators, it would be
logical to wrap block id generation into a class.
I plan to introduce BlockIdGenerator, which generates random ids.
Making it pluggable will depend on whether anybody wants to build another
generator.
It may make sense to implement Dhruba's suggestion in HDFS-898 to let people
use random generator on old clusters, and format new ones with the sequential
block id generator. Old clusters can also go through the procedure I described
in HDFS-898, which can be achieved by one-time image processing to find the
biggest hole between existing random block ids, and then switch to sequential
generation within that hole.
> Plugable block id generation
> -
>
> Key: HDFS-1245
> URL: https://issues.apache.org/jira/browse/HDFS-1245
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: namenode
>Reporter: Dmytro Molkov
>Assignee: Dmytro Molkov
>
> The idea is to have a way to easily create block id generation engines that
> may fit a certain purpose. One of them could be HDFS-898 started by
> Konstantin, but potentially others.
> We chatted with Dhruba about this for a while and came up with the following
> approach:
> There should be a BlockIDGenerator interface that has following methods:
> void blockAdded(Block)
> void blockRemoved(Block)
> Block nextBlock()
> First two methods are needed for block generation engines that hold a certain
> state. During the restart, when namenode reads the fsimage it will notify
> generator about all the blocks it reads from the image and during runtime
> namenode will notify the generator about block removals on file deletion.
> The instance of the generator will also have a reference to the block
> registry, the interface that BlockManager implements. The only method there
> is __blockExists(Block)__, so that the current random block id generation can
> be implemented, since it needs to check with the block manager if the id is
> already present.
> What does the community think about this proposal?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-347) DFS read performance suboptimal when client co-located on nodes with data
[
https://issues.apache.org/jira/browse/HDFS-347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551755#comment-13551755
]
Todd Lipcon commented on HDFS-347:
--
I believe all of the component pieces have now been committed to the HDFS-347
branch. I ran a number of benchmarks yesterday on the branch in progress, and
just re-confirmed the results from the code committed in SVN. Here's a report
of the benchmarks and results:
h1. Benchmarks
To validate the branch, I ran a series of before/after benchmarks, specifically
focused on random-read. In particular, I ran benchmarks based on
TestParallelRead, which has different variants which run the same workload
through the different read paths.
On the trunk ("before") branch, I ran TestParallelRead (normal read path) and
TestParallelLocalRead (read path based on HDFS-2246). On the HDFS-347 branch, I
ran TestParallelRead (normal read path) and TestParallelShortCircuitRead (new
short-circuit path).
I made the following modifications to the test cases to act as a better
benchmark:
1) Modified to 0% PROPORTION_NON_READ:
Without this modification, I found that both the 'before' and 'after' tests
became lock-bound, since the 'seek-and-read' workload holds a lock on the
DFSInputStream. So, this obscured the actual performance differences between
the data paths.
2) Modified to 30,000 iterations
Simply jacked up the number of iterations to get more reproducible results and
ensure that the JIT had plenty of time to kick in (the benchmarks ran for
~50seconds each with this change instead of only ~5sec)
3) Added a variation which has two target blocks
I had a thought that there could potentially be a regression for workloads
which frequently switch back and forth between two different blocks of the same
file. This variation is the same test, but with the DFS Block Size set to
128KB, so that the 256KB test file is split into two equal sized blocks. This
causes a good percentage of the random reads to span block boundaries, and
ensures that the various caches in the code work OK even when moving between
different blocks.
h2. Comparing non-local read
When the new code path is disabled, or when the DN is not local, we continue to
use the existing code path. We expect that this code path's performance should
be unaffected.
Results:
|| Test || #Threads || #Files|| Trunk MB/sec || HDFS-347 MB/sec ||
|| TestParallelRead | 4 | 1 | 428.4 | 423.0 |
|| TestParallelRead | 16 | 1 | 669.5 | 651.1 |
|| TestParallelRead | 8 | 2 | 603.4 | 582.7 |
|| TestParallelRead 2-blocks | 4 | 1 | 354.0 | 345.9 |
|| TestParallelRead 2-blocks | 16 | 1 | 534.9 | 520.0 |
|| TestParallelRead 2-blocks | 8 | 2 | 483.1 | 460.8 |
The above numbers seem to show a 2-4% regression, but I think it's within the
noise on my machine (other software was running, etc). Colin also has one or
two ideas for micro-optimizations which might win back a couple percent here
and there, if it's not just noise.
To put this in perspective, here are results for the same test against branch-1:
|| Test || #Threads || #Files|| Branch-1 ||
|| TestParallelRead | 4 | 1 | 229.7 |
|| TestParallelRead | 16 | 1 | 264.4 |
|| TestParallelRead | 8 | 2 | 260.1 |
(so trunk is 2-3x as fast as branch-1)
h2. Comparing local read
Here we expect the performance to be as good or better than the old (HDFS-2246)
implementation. Results:
|| Test || #Threads || #Files|| Trunk MB/sec || HDFS-347 MB/sec ||
|| TestParallelLocalRead | 4 | 1 | 901.4 | 1033.6 |
|| TestParallelLocalRead | 16 | 1 | 1079.8 | 1203.9 |
|| TestParallelLocalRead | 8 | 2 | 1087.4 | 1274.0 |
|| TestParallelLocalRead 2-blocks | 4 | 1 | 856.6 | 919.2 |
|| TestParallelLocalRead 2-blocks | 16 | 1 | 1045.8 | 1137.0 |
|| TestParallelLocalRead 2-blocks | 8 | 2 | 966.7 | 1392.9 |
The result shows that the new implementation is indeed between 10% and 44%
faster than the HDFS-2246 implementation. We're theorizing that the reason is
because the old implementation would cache block paths, but not open file
descriptors. So, because every positional read creates a new BlockReader, it
would have to issue new {{open()}} syscalls, even if the location was cached.
h2. Comparing sequential read
I used the BenchmarkThroughput tool, configured to write a 1GB file, and then
read it back 100 times. This ensures that it's in buffer cache, so that we're
benchmarking CPU overhead (since the actual disk access didn't change in the
patch, and we're looking for a potential regression in CPU resource usage). I
recorded the MB/sec rate for the short-circuit before and short-circuit after,
and then loaded the data into R and ran a T-test:
{code}
> d.before <- read.table(file="/tmp/before-patch.txt")
> d.after <- read.table(file="/tmp/after-patch.txt")
> t.test(d.before, d.after)
> d.before <- read.table(file="/tmp/before-patch.txt")
> d.after <- read.table(file="/tmp
[jira] [Commented] (HDFS-4387) test_libhdfs_threaded SEGV on OpenJDK 7
[
https://issues.apache.org/jira/browse/HDFS-4387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551751#comment-13551751
]
Hadoop QA commented on HDFS-4387:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12564522/01.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/3826//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3826//console
This message is automatically generated.
> test_libhdfs_threaded SEGV on OpenJDK 7
> ---
>
> Key: HDFS-4387
> URL: https://issues.apache.org/jira/browse/HDFS-4387
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: libhdfs
>Affects Versions: 3.0.0
>Reporter: Andy Isaacson
>Priority: Minor
> Attachments: 01.patch
>
>
> Building and running tests on OpenJDK 7 on Ubuntu 12.10 fails with {{mvn test
> -Pnative}}. The output is hard to decipher but the underlying issue is that
> {{test_libhdfs_native}} segfaults at startup.
> {noformat}
> (gdb) run
> Starting program:
> /mnt/trunk/hadoop-hdfs-project/hadoop-hdfs/target/native/test_libhdfs_threaded
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> Program received signal SIGSEGV, Segmentation fault.
> 0x7739a897 in attachJNIThread (name=0x0, is_daemon=is_daemon@entry=0
> '\000', group=0x0) at thread.c:768
> 768 thread.c: No such file or directory.
> (gdb) where
> #0 0x7739a897 in attachJNIThread (name=0x0,
> is_daemon=is_daemon@entry=0 '\000', group=0x0) at thread.c:768
> #1 0x77395020 in attachCurrentThread (is_daemon=0, args=0x0,
> penv=0x7fffddb8) at jni.c:1454
> #2 Jam_AttachCurrentThread (vm=, penv=0x7fffddb8,
> args=0x0) at jni.c:1466
> #3 0x77bcf979 in getGlobalJNIEnv () at
> /mnt/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:527
> #4 getJNIEnv () at
> /mnt/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:585
> #5 0x00402512 in nmdCreate (conf=conf@entry=0x7fffdeb0) at
> /mnt/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/native_mini_dfs.c:49
> #6 0x004016e1 in main () at
> /mnt/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/test_libhdfs_threaded.c:283
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4391) TestDataTransferKeepalive fails when tests are executed in a certain order.
[
https://issues.apache.org/jira/browse/HDFS-4391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551741#comment-13551741
]
Konstantin Shvachko commented on HDFS-4391:
---
JUnit does not guarantee the order of tests execution. So it does not always
fail. If {{testManyClosedSocketsInCache()}} is executed last the test passes.
But if goes first the next test {{testKeepaliveTimeouts()}} fails with the
assertion:
{code}
Running org.apache.hadoop.hdfs.TestDataTransferKeepalive
Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 9.687 sec <<<
FAILURE!
testKeepaliveTimeouts(org.apache.hadoop.hdfs.TestDataTransferKeepalive) Time
elapsed: 702 sec <<< FAILURE!
java.lang.AssertionError: expected:<0> but was:<3>
at org.junit.Assert.fail(Assert.java:91)
at org.junit.Assert.failNotEquals(Assert.java:645)
at org.junit.Assert.assertEquals(Assert.java:126)
at org.junit.Assert.assertEquals(Assert.java:470)
at org.junit.Assert.assertEquals(Assert.java:454)
at
org.apache.hadoop.hdfs.TestDataTransferKeepalive.testKeepaliveTimeouts(TestDataTransferKeepalive.java:95)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.FailOnTimeout$1.run(FailOnTimeout.java:28)
{code}
I can reproduce it in eclipse debugger just by moving
{{testManyClosedSocketsInCache()}} ahead of other test cases in the code.
> TestDataTransferKeepalive fails when tests are executed in a certain order.
> ---
>
> Key: HDFS-4391
> URL: https://issues.apache.org/jira/browse/HDFS-4391
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
>Reporter: Konstantin Shvachko
>
> TestDataTransferKeepalive if testKeepaliveTimeouts() is executed AFTER
> testManyClosedSocketsInCache().
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4391) TestDataTransferKeepalive fails when tests are executed in a certain order.
Konstantin Shvachko created HDFS-4391: - Summary: TestDataTransferKeepalive fails when tests are executed in a certain order. Key: HDFS-4391 URL: https://issues.apache.org/jira/browse/HDFS-4391 Project: Hadoop HDFS Issue Type: Bug Components: test Affects Versions: 3.0.0 Reporter: Konstantin Shvachko TestDataTransferKeepalive if testKeepaliveTimeouts() is executed AFTER testManyClosedSocketsInCache(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-4390) Bypass UNIX domain socket unit tests when they cannot be run
[ https://issues.apache.org/jira/browse/HDFS-4390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon resolved HDFS-4390. --- Resolution: Fixed Hadoop Flags: Reviewed Committed to branch > Bypass UNIX domain socket unit tests when they cannot be run > > > Key: HDFS-4390 > URL: https://issues.apache.org/jira/browse/HDFS-4390 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > Attachments: _06.patch > > > Testing revealed that the existing mechanisms for bypassing UNIX domain > socket-related tests when they are not available are inadequate. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4390) Bypass UNIX domain socket unit tests when they cannot be run
[ https://issues.apache.org/jira/browse/HDFS-4390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551727#comment-13551727 ] Todd Lipcon commented on HDFS-4390: --- Looks good, thanks for fixing this. +1 > Bypass UNIX domain socket unit tests when they cannot be run > > > Key: HDFS-4390 > URL: https://issues.apache.org/jira/browse/HDFS-4390 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > Attachments: _06.patch > > > Testing revealed that the existing mechanisms for bypassing UNIX domain > socket-related tests when they are not available are inadequate. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4390) Bypass UNIX domain socket unit tests when they cannot be run
[ https://issues.apache.org/jira/browse/HDFS-4390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-4390: --- Attachment: _06.patch > Bypass UNIX domain socket unit tests when they cannot be run > > > Key: HDFS-4390 > URL: https://issues.apache.org/jira/browse/HDFS-4390 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > Attachments: _06.patch > > > Testing revealed that the existing mechanisms for bypassing UNIX domain > socket-related tests when they are not available are inadequate. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4390) Bypass UNIX domain socket units when they cannot be run
Colin Patrick McCabe created HDFS-4390: -- Summary: Bypass UNIX domain socket units when they cannot be run Key: HDFS-4390 URL: https://issues.apache.org/jira/browse/HDFS-4390 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor Attachments: _06.patch Testing revealed that the existing mechanisms for bypassing UNIX domain socket-related tests when they are not available are inadequate. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4390) Bypass UNIX domain socket unit tests when they cannot be run
[ https://issues.apache.org/jira/browse/HDFS-4390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-4390: --- Summary: Bypass UNIX domain socket unit tests when they cannot be run (was: Bypass UNIX domain socket units when they cannot be run) > Bypass UNIX domain socket unit tests when they cannot be run > > > Key: HDFS-4390 > URL: https://issues.apache.org/jira/browse/HDFS-4390 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > Attachments: _06.patch > > > Testing revealed that the existing mechanisms for bypassing UNIX domain > socket-related tests when they are not available are inadequate. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4261) TestBalancerWithNodeGroup times out
[ https://issues.apache.org/jira/browse/HDFS-4261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551704#comment-13551704 ] Junping Du commented on HDFS-4261: -- I see. That make sense. Thanks ATM and Suresh.:) > TestBalancerWithNodeGroup times out > --- > > Key: HDFS-4261 > URL: https://issues.apache.org/jira/browse/HDFS-4261 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer >Affects Versions: 1.0.4, 1.1.1, 2.0.2-alpha >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Junping Du > Fix For: 3.0.0 > > Attachments: HDFS-4261-branch-1.patch, HDFS-4261-branch-1-v2.patch, > HDFS-4261.patch, HDFS-4261-v2.patch, HDFS-4261-v3.patch, HDFS-4261-v4.patch, > HDFS-4261-v5.patch, HDFS-4261-v6.patch, HDFS-4261-v7.patch, > HDFS-4261-v8.patch, jstack-mac-18567, jstack-win-5488, > org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup-output.txt.mac, > > org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup-output.txt.win, > test-balancer-with-node-group-timeout.txt > > > When I manually ran TestBalancerWithNodeGroup, it always timed out in my > machine. Looking at the Jerkins report [build > #3573|https://builds.apache.org/job/PreCommit-HDFS-Build/3573//testReport/org.apache.hadoop.hdfs.server.balancer/], > TestBalancerWithNodeGroup somehow was skipped so that the problem was not > detected. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-4388) DomainSocket should throw AsynchronousCloseException when appropriate
[
https://issues.apache.org/jira/browse/HDFS-4388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon resolved HDFS-4388.
---
Resolution: Fixed
Hadoop Flags: Reviewed
Committed to branch. Thanks.
> DomainSocket should throw AsynchronousCloseException when appropriate
> -
>
> Key: HDFS-4388
> URL: https://issues.apache.org/jira/browse/HDFS-4388
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
>Priority: Trivial
> Attachments: _05a.patch
>
>
> {{DomainSocket}} should throw {{AsynchronousCloseException}} when appropriate
> (i.e., when an {{accept}} or other blocking operation is interrupted by a
> concurrent close.) This is nicer than throwing a generic {{IOException}} or
> {{SocketException}}.
> Similarly, we should well throw {{ClosedChannelException}} when an operation
> is attempted on a closed {{DomainSocket}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4388) DomainSocket should throw AsynchronousCloseException when appropriate
[
https://issues.apache.org/jira/browse/HDFS-4388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551685#comment-13551685
]
Todd Lipcon commented on HDFS-4388:
---
+1, looks like a good fix.
> DomainSocket should throw AsynchronousCloseException when appropriate
> -
>
> Key: HDFS-4388
> URL: https://issues.apache.org/jira/browse/HDFS-4388
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
>Priority: Trivial
> Attachments: _05a.patch
>
>
> {{DomainSocket}} should throw {{AsynchronousCloseException}} when appropriate
> (i.e., when an {{accept}} or other blocking operation is interrupted by a
> concurrent close.) This is nicer than throwing a generic {{IOException}} or
> {{SocketException}}.
> Similarly, we should well throw {{ClosedChannelException}} when an operation
> is attempted on a closed {{DomainSocket}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4387) test_libhdfs_threaded SEGV on OpenJDK 7
[
https://issues.apache.org/jira/browse/HDFS-4387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HDFS-4387:
---
Status: Patch Available (was: Open)
> test_libhdfs_threaded SEGV on OpenJDK 7
> ---
>
> Key: HDFS-4387
> URL: https://issues.apache.org/jira/browse/HDFS-4387
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: libhdfs
>Affects Versions: 3.0.0
>Reporter: Andy Isaacson
>Priority: Minor
> Attachments: 01.patch
>
>
> Building and running tests on OpenJDK 7 on Ubuntu 12.10 fails with {{mvn test
> -Pnative}}. The output is hard to decipher but the underlying issue is that
> {{test_libhdfs_native}} segfaults at startup.
> {noformat}
> (gdb) run
> Starting program:
> /mnt/trunk/hadoop-hdfs-project/hadoop-hdfs/target/native/test_libhdfs_threaded
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> Program received signal SIGSEGV, Segmentation fault.
> 0x7739a897 in attachJNIThread (name=0x0, is_daemon=is_daemon@entry=0
> '\000', group=0x0) at thread.c:768
> 768 thread.c: No such file or directory.
> (gdb) where
> #0 0x7739a897 in attachJNIThread (name=0x0,
> is_daemon=is_daemon@entry=0 '\000', group=0x0) at thread.c:768
> #1 0x77395020 in attachCurrentThread (is_daemon=0, args=0x0,
> penv=0x7fffddb8) at jni.c:1454
> #2 Jam_AttachCurrentThread (vm=, penv=0x7fffddb8,
> args=0x0) at jni.c:1466
> #3 0x77bcf979 in getGlobalJNIEnv () at
> /mnt/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:527
> #4 getJNIEnv () at
> /mnt/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:585
> #5 0x00402512 in nmdCreate (conf=conf@entry=0x7fffdeb0) at
> /mnt/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/native_mini_dfs.c:49
> #6 0x004016e1 in main () at
> /mnt/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/test_libhdfs_threaded.c:283
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4387) test_libhdfs_threaded SEGV on OpenJDK 7
[
https://issues.apache.org/jira/browse/HDFS-4387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HDFS-4387:
---
Attachment: 01.patch
> test_libhdfs_threaded SEGV on OpenJDK 7
> ---
>
> Key: HDFS-4387
> URL: https://issues.apache.org/jira/browse/HDFS-4387
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: libhdfs
>Affects Versions: 3.0.0
>Reporter: Andy Isaacson
>Priority: Minor
> Attachments: 01.patch
>
>
> Building and running tests on OpenJDK 7 on Ubuntu 12.10 fails with {{mvn test
> -Pnative}}. The output is hard to decipher but the underlying issue is that
> {{test_libhdfs_native}} segfaults at startup.
> {noformat}
> (gdb) run
> Starting program:
> /mnt/trunk/hadoop-hdfs-project/hadoop-hdfs/target/native/test_libhdfs_threaded
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> Program received signal SIGSEGV, Segmentation fault.
> 0x7739a897 in attachJNIThread (name=0x0, is_daemon=is_daemon@entry=0
> '\000', group=0x0) at thread.c:768
> 768 thread.c: No such file or directory.
> (gdb) where
> #0 0x7739a897 in attachJNIThread (name=0x0,
> is_daemon=is_daemon@entry=0 '\000', group=0x0) at thread.c:768
> #1 0x77395020 in attachCurrentThread (is_daemon=0, args=0x0,
> penv=0x7fffddb8) at jni.c:1454
> #2 Jam_AttachCurrentThread (vm=, penv=0x7fffddb8,
> args=0x0) at jni.c:1466
> #3 0x77bcf979 in getGlobalJNIEnv () at
> /mnt/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:527
> #4 getJNIEnv () at
> /mnt/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:585
> #5 0x00402512 in nmdCreate (conf=conf@entry=0x7fffdeb0) at
> /mnt/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/native_mini_dfs.c:49
> #6 0x004016e1 in main () at
> /mnt/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/test_libhdfs_threaded.c:283
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4387) test_libhdfs_threaded SEGV on OpenJDK 7
[
https://issues.apache.org/jira/browse/HDFS-4387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551665#comment-13551665
]
Colin Patrick McCabe commented on HDFS-4387:
even after making this change, JamVM crashes in nmdShutdown. This is a simple
function that does nothing but call {{MiniDFSCluster#shutdown}}.
{code}
2013-01-11 15:46:06,003 INFO hdfs.MiniDFSCluster
(MiniDFSCluster.java:shutdown(1411)) - Shutting down the Mini HDFS Cluster
2013-01-11 15:46:06,004 INFO hdfs.MiniDFSCluster
(MiniDFSCluster.java:shutdownDataNodes(1438)) - Shutting down DataNode 0
2013-01-11 15:46:06,004 WARN datanode.DirectoryScanner
(DirectoryScanner.java:shutdown(289)) - DirectoryScanner: shutdown has been
called
2013-01-11 15:46:06,015 INFO mortbay.log (Slf4jLog.java:info(67)) - Stopped
SelectChannelConnector@localhost:0
2013-01-11 15:46:06,016 INFO ipc.Server (Server.java:stop(2096)) - Stopping
server on 39373
2013-01-11 15:46:06,018 INFO ipc.Server (Server.java:run(829)) - Stopping IPC
Server Responder
2013-01-11 15:46:06,018 INFO ipc.Server (Server.java:run(687)) - Stopping IPC
Server listener on 39373
Segmentation fault (core dumped)
{code}
I really have no idea how to fix this, or even if it can be fixed. It looks
like something that shutdown itself is doing, rather than something in the C
code. Perhaps JamVM can't handle threads shutting down or the log source being
closed?
> test_libhdfs_threaded SEGV on OpenJDK 7
> ---
>
> Key: HDFS-4387
> URL: https://issues.apache.org/jira/browse/HDFS-4387
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: libhdfs
>Affects Versions: 3.0.0
>Reporter: Andy Isaacson
>Priority: Minor
>
> Building and running tests on OpenJDK 7 on Ubuntu 12.10 fails with {{mvn test
> -Pnative}}. The output is hard to decipher but the underlying issue is that
> {{test_libhdfs_native}} segfaults at startup.
> {noformat}
> (gdb) run
> Starting program:
> /mnt/trunk/hadoop-hdfs-project/hadoop-hdfs/target/native/test_libhdfs_threaded
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> Program received signal SIGSEGV, Segmentation fault.
> 0x7739a897 in attachJNIThread (name=0x0, is_daemon=is_daemon@entry=0
> '\000', group=0x0) at thread.c:768
> 768 thread.c: No such file or directory.
> (gdb) where
> #0 0x7739a897 in attachJNIThread (name=0x0,
> is_daemon=is_daemon@entry=0 '\000', group=0x0) at thread.c:768
> #1 0x77395020 in attachCurrentThread (is_daemon=0, args=0x0,
> penv=0x7fffddb8) at jni.c:1454
> #2 Jam_AttachCurrentThread (vm=, penv=0x7fffddb8,
> args=0x0) at jni.c:1466
> #3 0x77bcf979 in getGlobalJNIEnv () at
> /mnt/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:527
> #4 getJNIEnv () at
> /mnt/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:585
> #5 0x00402512 in nmdCreate (conf=conf@entry=0x7fffdeb0) at
> /mnt/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/native_mini_dfs.c:49
> #6 0x004016e1 in main () at
> /mnt/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/test_libhdfs_threaded.c:283
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-4356) BlockReaderLocal should use passed file descriptors rather than paths
[
https://issues.apache.org/jira/browse/HDFS-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon resolved HDFS-4356.
---
Resolution: Fixed
Hadoop Flags: Reviewed
> BlockReaderLocal should use passed file descriptors rather than paths
> -
>
> Key: HDFS-4356
> URL: https://issues.apache.org/jira/browse/HDFS-4356
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, hdfs-client, performance
>Affects Versions: 2.0.3-alpha
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: 04b-cumulative.patch, _04b.patch, _04c.patch,
> 04-cumulative.patch, 04d-cumulative.patch, _04e.patch, 04f-cumulative.patch,
> _04f.patch, 04g-cumulative.patch, _04g.patch
>
>
> {{BlockReaderLocal}} should use file descriptors passed over UNIX domain
> sockets rather than paths. We also need some configuration options for these
> UNIX domain sockets.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4356) BlockReaderLocal should use passed file descriptors rather than paths
[
https://issues.apache.org/jira/browse/HDFS-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551659#comment-13551659
]
Todd Lipcon commented on HDFS-4356:
---
+1, will commit to branch momentarily.
> BlockReaderLocal should use passed file descriptors rather than paths
> -
>
> Key: HDFS-4356
> URL: https://issues.apache.org/jira/browse/HDFS-4356
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, hdfs-client, performance
>Affects Versions: 2.0.3-alpha
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: 04b-cumulative.patch, _04b.patch, _04c.patch,
> 04-cumulative.patch, 04d-cumulative.patch, _04e.patch, 04f-cumulative.patch,
> _04f.patch, 04g-cumulative.patch, _04g.patch
>
>
> {{BlockReaderLocal}} should use file descriptors passed over UNIX domain
> sockets rather than paths. We also need some configuration options for these
> UNIX domain sockets.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4387) test_libhdfs_threaded SEGV on OpenJDK 7
[
https://issues.apache.org/jira/browse/HDFS-4387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551618#comment-13551618
]
Colin Patrick McCabe commented on HDFS-4387:
After some investigation, it looks like this is a result of you using the
{{libjvm.so}} from {{JamVM}}.
Although this isn't documented anywhere, some examination of the JamVM source
(as well as empirical tests) shows that JamVM crashes if you call
{{JNI_GetCreatedJavaVMs}} prior to calling {{JNI_CreateJavaVM}}. Since we
currently do this in {{libhdfs}}, we get the crash.
There is an argument to be made that {{libhdfs}} should not try to see if a VM
is already running before we create one. Since such a VM, presumably created
by another shared library or the application itself, would probably not have
the correct {{CLASSPATH}}, it would be unlikely to work anyway. Still, it's
unfortunate that we (apparently) have to take this step.
It looks like JamVM is installed by default on Ubuntu 12.10, even on the server
spin-- for whatever reason. So we probably do need to fix this. It would also
be nice if we avoided using JamVM by default.
> test_libhdfs_threaded SEGV on OpenJDK 7
> ---
>
> Key: HDFS-4387
> URL: https://issues.apache.org/jira/browse/HDFS-4387
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: libhdfs
>Affects Versions: 3.0.0
>Reporter: Andy Isaacson
>Priority: Minor
>
> Building and running tests on OpenJDK 7 on Ubuntu 12.10 fails with {{mvn test
> -Pnative}}. The output is hard to decipher but the underlying issue is that
> {{test_libhdfs_native}} segfaults at startup.
> {noformat}
> (gdb) run
> Starting program:
> /mnt/trunk/hadoop-hdfs-project/hadoop-hdfs/target/native/test_libhdfs_threaded
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> Program received signal SIGSEGV, Segmentation fault.
> 0x7739a897 in attachJNIThread (name=0x0, is_daemon=is_daemon@entry=0
> '\000', group=0x0) at thread.c:768
> 768 thread.c: No such file or directory.
> (gdb) where
> #0 0x7739a897 in attachJNIThread (name=0x0,
> is_daemon=is_daemon@entry=0 '\000', group=0x0) at thread.c:768
> #1 0x77395020 in attachCurrentThread (is_daemon=0, args=0x0,
> penv=0x7fffddb8) at jni.c:1454
> #2 Jam_AttachCurrentThread (vm=, penv=0x7fffddb8,
> args=0x0) at jni.c:1466
> #3 0x77bcf979 in getGlobalJNIEnv () at
> /mnt/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:527
> #4 getJNIEnv () at
> /mnt/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:585
> #5 0x00402512 in nmdCreate (conf=conf@entry=0x7fffdeb0) at
> /mnt/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/native_mini_dfs.c:49
> #6 0x004016e1 in main () at
> /mnt/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/test_libhdfs_threaded.c:283
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4389) Non-HA DFSClients do not attempt reconnects
[
https://issues.apache.org/jira/browse/HDFS-4389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551602#comment-13551602
]
Todd Lipcon commented on HDFS-4389:
---
Hey Daryn. I vaguely remember this being a conscious decision at some point,
but maybe I made that up. Two thoughts that might be relevant:
1) TestPersistBlocks seems to have started failing much more often recently,
but I don't have evidence for this. Any chance something else might have caused
a regression here?
2) In the old code, which retried over the restart, wouldn't it end up just
hitting a SafeModeException and then failing at that point, when the NN was
restarted? Given that the NN usually takes 30+seconds to leave safemode after
starting, any retrying clients would probably hit that and fail anyway, no?
> Non-HA DFSClients do not attempt reconnects
> ---
>
> Key: HDFS-4389
> URL: https://issues.apache.org/jira/browse/HDFS-4389
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, hdfs-client
>Affects Versions: 2.0.0-alpha, 3.0.0
>Reporter: Daryn Sharp
>Priority: Critical
>
> The HA retry policy implementation appears to have broken non-HA
> {{DFSClient}} connect retries. The ipc
> {{Client.Connection#handleConnectionFailure}} used to perform 45 connection
> attempts, but now it consults a retry policy. For non-HA proxies, the policy
> does not handle {{ConnectException}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4274) BlockPoolSliceScanner does not close verification log during shutdown
[
https://issues.apache.org/jira/browse/HDFS-4274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551590#comment-13551590
]
Hudson commented on HDFS-4274:
--
Integrated in Hadoop-trunk-Commit #3221 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/3221/])
Update CHANGES.txt to reflect HDFS-4274 merge. (Revision 1432316)
Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1432316
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> BlockPoolSliceScanner does not close verification log during shutdown
> -
>
> Key: HDFS-4274
> URL: https://issues.apache.org/jira/browse/HDFS-4274
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.0.0-alpha, trunk-win
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Minor
> Fix For: 2.0.3-alpha, trunk-win
>
> Attachments: HDFS-4274.1.patch, HDFS-4274.2.patch
>
>
> {{BlockPoolSliceScanner}} holds open a handle to a verification log. This
> file is not getting closed during process shutdown.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4356) BlockReaderLocal should use passed file descriptors rather than paths
[
https://issues.apache.org/jira/browse/HDFS-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HDFS-4356:
---
Attachment: _04g.patch
> BlockReaderLocal should use passed file descriptors rather than paths
> -
>
> Key: HDFS-4356
> URL: https://issues.apache.org/jira/browse/HDFS-4356
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, hdfs-client, performance
>Affects Versions: 2.0.3-alpha
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: 04b-cumulative.patch, _04b.patch, _04c.patch,
> 04-cumulative.patch, 04d-cumulative.patch, _04e.patch, 04f-cumulative.patch,
> _04f.patch, 04g-cumulative.patch, _04g.patch
>
>
> {{BlockReaderLocal}} should use file descriptors passed over UNIX domain
> sockets rather than paths. We also need some configuration options for these
> UNIX domain sockets.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4274) BlockPoolSliceScanner does not close verification log during shutdown
[
https://issues.apache.org/jira/browse/HDFS-4274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Collins updated HDFS-4274:
--
Priority: Minor (was: Major)
Target Version/s: (was: 3.0.0, trunk-win)
Affects Version/s: (was: 3.0.0)
2.0.0-alpha
Fix Version/s: (was: 3.0.0)
trunk-win
2.0.3-alpha
Merged and updated trunk CHANGES.txt
> BlockPoolSliceScanner does not close verification log during shutdown
> -
>
> Key: HDFS-4274
> URL: https://issues.apache.org/jira/browse/HDFS-4274
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.0.0-alpha, trunk-win
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Minor
> Fix For: 2.0.3-alpha, trunk-win
>
> Attachments: HDFS-4274.1.patch, HDFS-4274.2.patch
>
>
> {{BlockPoolSliceScanner}} holds open a handle to a verification log. This
> file is not getting closed during process shutdown.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4288) NN accepts incremental BR as IBR in safemode
[
https://issues.apache.org/jira/browse/HDFS-4288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551583#comment-13551583
]
Daryn Sharp commented on HDFS-4288:
---
{{TestPersistBlocks}} failure is unrelated to this jira - I was very paranoid
in debugging the failure. The problem is caused by the new retry IPC retry
policies introduced by HA. I filed HDFS-4389.
> NN accepts incremental BR as IBR in safemode
>
>
> Key: HDFS-4288
> URL: https://issues.apache.org/jira/browse/HDFS-4288
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
>Priority: Critical
> Attachments: HDFS-4288.branch-23.patch, HDFS-4288.patch
>
>
> If a DN is ready to send an incremental BR and the NN goes down, the DN will
> repeatedly try to reconnect. The NN will then process the DN's incremental
> BR as an initial BR. The NN now thinks the DN has only a few blocks, and
> will ignore all subsequent BRs from that DN until out of safemode -- which it
> may never do because of all the "missing" blocks on the affected DNs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4389) Non-HA DFSClients do not attempt reconnects
[
https://issues.apache.org/jira/browse/HDFS-4389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551581#comment-13551581
]
Daryn Sharp commented on HDFS-4389:
---
The problem was discovered while debugging random failures in
{{TestPersistBlocks}}. The {{TestRestartDfsWithFlush}} does the following:
# open a stream
# write 5 blocks
# flush
# wait for at least 1 block to be finalized, record size
# bounce the NN
# ensure file is at least as big as before bounce
# write 5 more blocks <- race condition blows up here
# close stream
# ensure all data is there
The problem occurs when {{DFSOutputStream.DataStreamer}} needs to call
{{DFSClient#addBlock}} while the NN is down. It receives a
{{ConnectException}} from the IPC layer, which isn't handled so it stores it
away and shuts down the stream. The additional write to the stream after the
NN restart throws the stored connect exception.
The end result is streams cannot survive a NN restart or network interruption
that lasts longer than the time it takes to write a block. The issue is
probably general to all client methods.
> Non-HA DFSClients do not attempt reconnects
> ---
>
> Key: HDFS-4389
> URL: https://issues.apache.org/jira/browse/HDFS-4389
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, hdfs-client
>Affects Versions: 2.0.0-alpha, 3.0.0
>Reporter: Daryn Sharp
>Priority: Critical
>
> The HA retry policy implementation appears to have broken non-HA
> {{DFSClient}} connect retries. The ipc
> {{Client.Connection#handleConnectionFailure}} used to perform 45 connection
> attempts, but now it consults a retry policy. For non-HA proxies, the policy
> does not handle {{ConnectException}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4356) BlockReaderLocal should use passed file descriptors rather than paths
[
https://issues.apache.org/jira/browse/HDFS-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551564#comment-13551564
]
Colin Patrick McCabe commented on HDFS-4356:
update: After I ran the program and looked at the thread names, I found that
{{DaemonFactory}} was not successfully setting the thread name I wanted
(probably because there is a wrapper {{Runnable}} somewhere) So I created my
own {{ThreadFactory}} as you suggested.
> BlockReaderLocal should use passed file descriptors rather than paths
> -
>
> Key: HDFS-4356
> URL: https://issues.apache.org/jira/browse/HDFS-4356
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, hdfs-client, performance
>Affects Versions: 2.0.3-alpha
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: 04b-cumulative.patch, _04b.patch, _04c.patch,
> 04-cumulative.patch, 04d-cumulative.patch, _04e.patch, 04f-cumulative.patch,
> _04f.patch, 04g-cumulative.patch
>
>
> {{BlockReaderLocal}} should use file descriptors passed over UNIX domain
> sockets rather than paths. We also need some configuration options for these
> UNIX domain sockets.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4389) Non-HA DFSClients do not attempt reconnects
Daryn Sharp created HDFS-4389:
-
Summary: Non-HA DFSClients do not attempt reconnects
Key: HDFS-4389
URL: https://issues.apache.org/jira/browse/HDFS-4389
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, hdfs-client
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: Daryn Sharp
Priority: Critical
The HA retry policy implementation appears to have broken non-HA {{DFSClient}}
connect retries. The ipc {{Client.Connection#handleConnectionFailure}} used to
perform 45 connection attempts, but now it consults a retry policy. For non-HA
proxies, the policy does not handle {{ConnectException}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4356) BlockReaderLocal should use passed file descriptors rather than paths
[
https://issues.apache.org/jira/browse/HDFS-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HDFS-4356:
---
Attachment: _04f.patch
> BlockReaderLocal should use passed file descriptors rather than paths
> -
>
> Key: HDFS-4356
> URL: https://issues.apache.org/jira/browse/HDFS-4356
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, hdfs-client, performance
>Affects Versions: 2.0.3-alpha
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: 04b-cumulative.patch, _04b.patch, _04c.patch,
> 04-cumulative.patch, 04d-cumulative.patch, _04e.patch, 04f-cumulative.patch,
> _04f.patch, 04g-cumulative.patch
>
>
> {{BlockReaderLocal}} should use file descriptors passed over UNIX domain
> sockets rather than paths. We also need some configuration options for these
> UNIX domain sockets.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4356) BlockReaderLocal should use passed file descriptors rather than paths
[
https://issues.apache.org/jira/browse/HDFS-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551562#comment-13551562
]
Colin Patrick McCabe commented on HDFS-4356:
* added debug guard in {{BlockReaderLocal#close}}
* fix comment for {{FileInputStreamCache#closed}}
{{DaemonFactory}} has this code:
{code}
\@Override
public Thread newThread(Runnable runnable) {
return new Daemon(runnable);
}
{code}
... and {{Daemon}}'s constructor looks like this:
{code}
/** Construct a daemon thread. */
public Daemon(Runnable runnable) {
super(runnable);
this.runnable = runnable;
this.setName(((Object)runnable).toString());
}
{code}
So it will set the thread name appropriately.
> BlockReaderLocal should use passed file descriptors rather than paths
> -
>
> Key: HDFS-4356
> URL: https://issues.apache.org/jira/browse/HDFS-4356
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, hdfs-client, performance
>Affects Versions: 2.0.3-alpha
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: 04b-cumulative.patch, _04b.patch, _04c.patch,
> 04-cumulative.patch, 04d-cumulative.patch, _04e.patch, 04f-cumulative.patch,
> _04f.patch, 04g-cumulative.patch
>
>
> {{BlockReaderLocal}} should use file descriptors passed over UNIX domain
> sockets rather than paths. We also need some configuration options for these
> UNIX domain sockets.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4274) BlockPoolSliceScanner does not close verification log during shutdown
[
https://issues.apache.org/jira/browse/HDFS-4274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551493#comment-13551493
]
Suresh Srinivas commented on HDFS-4274:
---
Please merge it.
> BlockPoolSliceScanner does not close verification log during shutdown
> -
>
> Key: HDFS-4274
> URL: https://issues.apache.org/jira/browse/HDFS-4274
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.0.0, trunk-win
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Fix For: 3.0.0
>
> Attachments: HDFS-4274.1.patch, HDFS-4274.2.patch
>
>
> {{BlockPoolSliceScanner}} holds open a handle to a verification log. This
> file is not getting closed during process shutdown.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4261) TestBalancerWithNodeGroup times out
[ https://issues.apache.org/jira/browse/HDFS-4261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551484#comment-13551484 ] Aaron T. Myers commented on HDFS-4261: -- bq. I agree, when patches apply to earlier branches with trivial changes. However, when a significant rework is needed to port a patch back, I have been asking folks to create a separate jira, with an appropriate link between the jiras. That's totally reasonable. trunk/branch-2 are so close to each other that the back-port is almost always clean or trivial, but you're right that branch-1 back-ports often warrant a separate JIRA. > TestBalancerWithNodeGroup times out > --- > > Key: HDFS-4261 > URL: https://issues.apache.org/jira/browse/HDFS-4261 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer >Affects Versions: 1.0.4, 1.1.1, 2.0.2-alpha >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Junping Du > Fix For: 3.0.0 > > Attachments: HDFS-4261-branch-1.patch, HDFS-4261-branch-1-v2.patch, > HDFS-4261.patch, HDFS-4261-v2.patch, HDFS-4261-v3.patch, HDFS-4261-v4.patch, > HDFS-4261-v5.patch, HDFS-4261-v6.patch, HDFS-4261-v7.patch, > HDFS-4261-v8.patch, jstack-mac-18567, jstack-win-5488, > org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup-output.txt.mac, > > org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup-output.txt.win, > test-balancer-with-node-group-timeout.txt > > > When I manually ran TestBalancerWithNodeGroup, it always timed out in my > machine. Looking at the Jerkins report [build > #3573|https://builds.apache.org/job/PreCommit-HDFS-Build/3573//testReport/org.apache.hadoop.hdfs.server.balancer/], > TestBalancerWithNodeGroup somehow was skipped so that the problem was not > detected. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4261) TestBalancerWithNodeGroup times out
[ https://issues.apache.org/jira/browse/HDFS-4261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551482#comment-13551482 ] Suresh Srinivas commented on HDFS-4261: --- bq. I think we should use the same JIRAs as were used for the trunk work, and just change the "fix version" field appropriately when the change is committed to branch-2. I agree, when patches apply to earlier branches with trivial changes. However, when a significant rework is needed to port a patch back, I have been asking folks to create a separate jira, with an appropriate link between the jiras. > TestBalancerWithNodeGroup times out > --- > > Key: HDFS-4261 > URL: https://issues.apache.org/jira/browse/HDFS-4261 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer >Affects Versions: 1.0.4, 1.1.1, 2.0.2-alpha >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Junping Du > Fix For: 3.0.0 > > Attachments: HDFS-4261-branch-1.patch, HDFS-4261-branch-1-v2.patch, > HDFS-4261.patch, HDFS-4261-v2.patch, HDFS-4261-v3.patch, HDFS-4261-v4.patch, > HDFS-4261-v5.patch, HDFS-4261-v6.patch, HDFS-4261-v7.patch, > HDFS-4261-v8.patch, jstack-mac-18567, jstack-win-5488, > org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup-output.txt.mac, > > org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup-output.txt.win, > test-balancer-with-node-group-timeout.txt > > > When I manually ran TestBalancerWithNodeGroup, it always timed out in my > machine. Looking at the Jerkins report [build > #3573|https://builds.apache.org/job/PreCommit-HDFS-Build/3573//testReport/org.apache.hadoop.hdfs.server.balancer/], > TestBalancerWithNodeGroup somehow was skipped so that the problem was not > detected. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4261) TestBalancerWithNodeGroup times out
[ https://issues.apache.org/jira/browse/HDFS-4261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551475#comment-13551475 ] Aaron T. Myers commented on HDFS-4261: -- bq. BTW, shall we reuse JIRAs... I think we should use the same JIRAs as were used for the trunk work, and just change the "fix version" field appropriately when the change is committed to branch-2. (Not reusing the JIRAs is the reason why I wasn't aware of the branch-1 back-ports.) > TestBalancerWithNodeGroup times out > --- > > Key: HDFS-4261 > URL: https://issues.apache.org/jira/browse/HDFS-4261 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer >Affects Versions: 1.0.4, 1.1.1, 2.0.2-alpha >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Junping Du > Fix For: 3.0.0 > > Attachments: HDFS-4261-branch-1.patch, HDFS-4261-branch-1-v2.patch, > HDFS-4261.patch, HDFS-4261-v2.patch, HDFS-4261-v3.patch, HDFS-4261-v4.patch, > HDFS-4261-v5.patch, HDFS-4261-v6.patch, HDFS-4261-v7.patch, > HDFS-4261-v8.patch, jstack-mac-18567, jstack-win-5488, > org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup-output.txt.mac, > > org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup-output.txt.win, > test-balancer-with-node-group-timeout.txt > > > When I manually ran TestBalancerWithNodeGroup, it always timed out in my > machine. Looking at the Jerkins report [build > #3573|https://builds.apache.org/job/PreCommit-HDFS-Build/3573//testReport/org.apache.hadoop.hdfs.server.balancer/], > TestBalancerWithNodeGroup somehow was skipped so that the problem was not > detected. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4274) BlockPoolSliceScanner does not close verification log during shutdown
[
https://issues.apache.org/jira/browse/HDFS-4274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551462#comment-13551462
]
Eli Collins commented on HDFS-4274:
---
This like a nice fix to merge to branch-2, anyone objections?
> BlockPoolSliceScanner does not close verification log during shutdown
> -
>
> Key: HDFS-4274
> URL: https://issues.apache.org/jira/browse/HDFS-4274
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.0.0, trunk-win
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Fix For: 3.0.0
>
> Attachments: HDFS-4274.1.patch, HDFS-4274.2.patch
>
>
> {{BlockPoolSliceScanner}} holds open a handle to a verification log. This
> file is not getting closed during process shutdown.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4328) TestLargeBlock#testLargeBlockSize is timing out
[ https://issues.apache.org/jira/browse/HDFS-4328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551407#comment-13551407 ] Hudson commented on HDFS-4328: -- Integrated in Hadoop-trunk-Commit #3220 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3220/]) Update CHANGES.txt to move HDFS-4328. (Revision 1432227) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1432227 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > TestLargeBlock#testLargeBlockSize is timing out > --- > > Key: HDFS-4328 > URL: https://issues.apache.org/jira/browse/HDFS-4328 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 2.0.3-alpha >Reporter: Jason Lowe >Assignee: Chris Nauroth > Attachments: HDFS-4328.1.patch > > > For some time now TestLargeBlock#testLargeBlockSize has been timing out on > trunk. It is getting hung up during cluster shutdown, and after 15 minutes > surefire kills it and causes the build to fail since it exited uncleanly. > In addition to fixing the hang, we should consider adding a timeout parameter > to the @Test decorator for this test. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4328) TestLargeBlock#testLargeBlockSize is timing out
[ https://issues.apache.org/jira/browse/HDFS-4328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-4328: -- Target Version/s: (was: 3.0.0) Affects Version/s: (was: 3.0.0) 2.0.3-alpha I've merged this to branch-2 and updated trunks CHANGES.txt. > TestLargeBlock#testLargeBlockSize is timing out > --- > > Key: HDFS-4328 > URL: https://issues.apache.org/jira/browse/HDFS-4328 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 2.0.3-alpha >Reporter: Jason Lowe >Assignee: Chris Nauroth > Attachments: HDFS-4328.1.patch > > > For some time now TestLargeBlock#testLargeBlockSize has been timing out on > trunk. It is getting hung up during cluster shutdown, and after 15 minutes > surefire kills it and causes the build to fail since it exited uncleanly. > In addition to fixing the hang, we should consider adding a timeout parameter > to the @Test decorator for this test. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4385) Maven RAT plugin is not checking all source files
[ https://issues.apache.org/jira/browse/HDFS-4385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551395#comment-13551395 ] Eli Collins commented on HDFS-4385: --- +1 Looks great! > Maven RAT plugin is not checking all source files > - > > Key: HDFS-4385 > URL: https://issues.apache.org/jira/browse/HDFS-4385 > Project: Hadoop HDFS > Issue Type: Bug > Components: build >Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.5 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Critical > Attachments: HDFS-4385-branch-0.23.patch, > HDFS-4385-branch-0.23.patch, HDFS-4385-branch-0.23.patch, HDFS-4385.patch, > HDFS-4385.patch, HDFS-4385.patch, HDFS-4385-remove-branch23.sh, > HDFS-4385-remove.sh > > > HDFS side of HADOOP-9097 > Running 'mvn apache-rat:check' passes, but running RAT by hand (by > downloading the JAR) produces some warnings for Java files, amongst others. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4385) Maven RAT plugin is not checking all source files
[ https://issues.apache.org/jira/browse/HDFS-4385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated HDFS-4385: Attachment: HDFS-4385.patch HDFS-4385-branch-0.23.patch update patches to include tree.h license in LICENSE.txt > Maven RAT plugin is not checking all source files > - > > Key: HDFS-4385 > URL: https://issues.apache.org/jira/browse/HDFS-4385 > Project: Hadoop HDFS > Issue Type: Bug > Components: build >Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.5 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Critical > Attachments: HDFS-4385-branch-0.23.patch, > HDFS-4385-branch-0.23.patch, HDFS-4385-branch-0.23.patch, HDFS-4385.patch, > HDFS-4385.patch, HDFS-4385.patch, HDFS-4385-remove-branch23.sh, > HDFS-4385-remove.sh > > > HDFS side of HADOOP-9097 > Running 'mvn apache-rat:check' passes, but running RAT by hand (by > downloading the JAR) produces some warnings for Java files, amongst others. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4384) test_libhdfs_threaded gets SEGV if JNIEnv cannot be initialized
[ https://issues.apache.org/jira/browse/HDFS-4384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551392#comment-13551392 ] Hudson commented on HDFS-4384: -- Integrated in Hadoop-trunk-Commit #3219 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3219/]) HDFS-4384. test_libhdfs_threaded gets SEGV if JNIEnv cannot be initialized. Contributed by Colin Patrick McCabe (Revision 1432221) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1432221 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/native_mini_dfs.c > test_libhdfs_threaded gets SEGV if JNIEnv cannot be initialized > --- > > Key: HDFS-4384 > URL: https://issues.apache.org/jira/browse/HDFS-4384 > Project: Hadoop HDFS > Issue Type: Bug > Components: libhdfs >Affects Versions: 2.0.3-alpha >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > Fix For: 2.0.3-alpha > > Attachments: HDFS-4384.001.patch > > > test_libhdfs_threaded gets a SEGV if JNIEnv cannot be initialized. This is > incorrect; it should instead exit with a nice error message in this case. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4384) test_libhdfs_threaded gets SEGV if JNIEnv cannot be initialized
[ https://issues.apache.org/jira/browse/HDFS-4384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-4384: -- Resolution: Fixed Fix Version/s: 2.0.3-alpha Target Version/s: (was: 2.0.3-alpha) Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I've committed this and merged to branch-2. Thanks Colin! > test_libhdfs_threaded gets SEGV if JNIEnv cannot be initialized > --- > > Key: HDFS-4384 > URL: https://issues.apache.org/jira/browse/HDFS-4384 > Project: Hadoop HDFS > Issue Type: Bug > Components: libhdfs >Affects Versions: 2.0.3-alpha >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > Fix For: 2.0.3-alpha > > Attachments: HDFS-4384.001.patch > > > test_libhdfs_threaded gets a SEGV if JNIEnv cannot be initialized. This is > incorrect; it should instead exit with a nice error message in this case. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4384) test_libhdfs_threaded gets SEGV if JNIEnv cannot be initialized
[ https://issues.apache.org/jira/browse/HDFS-4384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551372#comment-13551372 ] Eli Collins commented on HDFS-4384: --- Makes sense. +1 > test_libhdfs_threaded gets SEGV if JNIEnv cannot be initialized > --- > > Key: HDFS-4384 > URL: https://issues.apache.org/jira/browse/HDFS-4384 > Project: Hadoop HDFS > Issue Type: Bug > Components: libhdfs >Affects Versions: 2.0.3-alpha >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > Attachments: HDFS-4384.001.patch > > > test_libhdfs_threaded gets a SEGV if JNIEnv cannot be initialized. This is > incorrect; it should instead exit with a nice error message in this case. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4256) Backport concatenation of files into a single file to branch-1
[ https://issues.apache.org/jira/browse/HDFS-4256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sanjay Radia updated HDFS-4256: --- Summary: Backport concatenation of files into a single file to branch-1 (was: Support for concatenation of files into a single file in branch-1) > Backport concatenation of files into a single file to branch-1 > -- > > Key: HDFS-4256 > URL: https://issues.apache.org/jira/browse/HDFS-4256 > Project: Hadoop HDFS > Issue Type: Test > Components: namenode >Affects Versions: 1.2.0 >Reporter: Suresh Srinivas >Assignee: Sanjay Radia > Attachments: HDFS-4256-2.patch, HDFS-4256.patch > > > HDFS-222 added support concatenation of multiple files in a directory into a > single file. This helps several use cases where writes can be parallelized > and several folks have expressed in this functionality. > This jira intends to make changes equivalent from HDFS-222 into branch-1 to > be made available release 1.2.0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4375) Use token request messages defined in hadoop common
[
https://issues.apache.org/jira/browse/HDFS-4375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551279#comment-13551279
]
Hadoop QA commented on HDFS-4375:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12563990/HDFS-4375.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3824//console
This message is automatically generated.
> Use token request messages defined in hadoop common
> ---
>
> Key: HDFS-4375
> URL: https://issues.apache.org/jira/browse/HDFS-4375
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode, security
>Affects Versions: 2.0.2-alpha
>Reporter: Suresh Srinivas
>Assignee: Suresh Srinivas
> Attachments: HDFS-4375.patch, HDFS-4375.patch
>
>
> HDFS changes related to HADOOP-9192 to reuse the protobuf messages defined in
> common.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4375) Use token request messages defined in hadoop common
[ https://issues.apache.org/jira/browse/HDFS-4375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-4375: -- Status: Patch Available (was: Open) > Use token request messages defined in hadoop common > --- > > Key: HDFS-4375 > URL: https://issues.apache.org/jira/browse/HDFS-4375 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode, security >Affects Versions: 2.0.2-alpha >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Attachments: HDFS-4375.patch, HDFS-4375.patch > > > HDFS changes related to HADOOP-9192 to reuse the protobuf messages defined in > common. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4385) Maven RAT plugin is not checking all source files
[
https://issues.apache.org/jira/browse/HDFS-4385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551273#comment-13551273
]
Hadoop QA commented on HDFS-4385:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12564422/HDFS-4385.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/3823//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3823//console
This message is automatically generated.
> Maven RAT plugin is not checking all source files
> -
>
> Key: HDFS-4385
> URL: https://issues.apache.org/jira/browse/HDFS-4385
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build
>Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.5
>Reporter: Thomas Graves
>Assignee: Thomas Graves
>Priority: Critical
> Attachments: HDFS-4385-branch-0.23.patch,
> HDFS-4385-branch-0.23.patch, HDFS-4385.patch, HDFS-4385.patch,
> HDFS-4385-remove-branch23.sh, HDFS-4385-remove.sh
>
>
> HDFS side of HADOOP-9097
> Running 'mvn apache-rat:check' passes, but running RAT by hand (by
> downloading the JAR) produces some warnings for Java files, amongst others.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4381) Document fsimage format details in FSImageFormat class javadoc
[ https://issues.apache.org/jira/browse/HDFS-4381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551272#comment-13551272 ] Hudson commented on HDFS-4381: -- Integrated in Hadoop-trunk-Commit #3217 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3217/]) HDFS-4381. Document fsimage format details in FSImageFormat class javadoc. Contributed by Jing Zhao. (Revision 1432149) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1432149 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java > Document fsimage format details in FSImageFormat class javadoc > --- > > Key: HDFS-4381 > URL: https://issues.apache.org/jira/browse/HDFS-4381 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.0.0-alpha >Reporter: Jing Zhao >Assignee: Jing Zhao > Fix For: 2.0.3-alpha > > Attachments: HDFS-4381.001.patch, HDFS-4381.002.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4381) Document fsimage format details in FSImageFormat class javadoc
[ https://issues.apache.org/jira/browse/HDFS-4381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-4381: -- Resolution: Fixed Fix Version/s: 2.0.3-alpha Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I committed the patch to both trunk and branch-2. When trunk is merged to branch HDFS-2802, the snapshots branch should also pick this change up. Thank you Jing! > Document fsimage format details in FSImageFormat class javadoc > --- > > Key: HDFS-4381 > URL: https://issues.apache.org/jira/browse/HDFS-4381 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.0.0-alpha >Reporter: Jing Zhao >Assignee: Jing Zhao > Fix For: 2.0.3-alpha > > Attachments: HDFS-4381.001.patch, HDFS-4381.002.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4381) Document fsimage format details in FSImageFormat class javadoc
[ https://issues.apache.org/jira/browse/HDFS-4381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-4381: -- Summary: Document fsimage format details in FSImageFormat class javadoc (was: Document fsimage format details to FSImageFormat class javadoc) > Document fsimage format details in FSImageFormat class javadoc > --- > > Key: HDFS-4381 > URL: https://issues.apache.org/jira/browse/HDFS-4381 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.0.0-alpha >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HDFS-4381.001.patch, HDFS-4381.002.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4381) Document fsimage format details to FSImageFormat class javadoc
[ https://issues.apache.org/jira/browse/HDFS-4381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-4381: -- Component/s: namenode Affects Version/s: (was: 3.0.0) 2.0.0-alpha Summary: Document fsimage format details to FSImageFormat class javadoc (was: Add javadoc for FSImageFormat with description about the FSImage format with/without localNameINodes) > Document fsimage format details to FSImageFormat class javadoc > --- > > Key: HDFS-4381 > URL: https://issues.apache.org/jira/browse/HDFS-4381 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.0.0-alpha >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HDFS-4381.001.patch, HDFS-4381.002.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4366) Block Replication Policy Implementation May Skip Higher-Priority Blocks for Lower-Priority Blocks
[
https://issues.apache.org/jira/browse/HDFS-4366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551224#comment-13551224
]
Hadoop QA commented on HDFS-4366:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12564409/HDFS-4366.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/3822//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3822//console
This message is automatically generated.
> Block Replication Policy Implementation May Skip Higher-Priority Blocks for
> Lower-Priority Blocks
> -
>
> Key: HDFS-4366
> URL: https://issues.apache.org/jira/browse/HDFS-4366
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0, 0.23.5
>Reporter: Derek Dagit
>Assignee: Derek Dagit
> Attachments: HDFS-4366.patch, HDFS-4366.patch,
> hdfs-4366-unittest.patch
>
>
> In certain cases, higher-priority under-replicated blocks can be skipped by
> the replication policy implementation. The current implementation maintains,
> for each priority level, an index into a list of blocks that are
> under-replicated. Together, the lists compose a priority queue (see note
> later about branch-0.23). In some cases when blocks are removed from a list,
> the caller (BlockManager) properly handles the index into the list from which
> it removed a block. In some other cases, the index remains stationary while
> the list changes. Whenever this happens, and the removed block happened to
> be at or before the index, the implementation will skip over a block when
> selecting blocks for replication work.
> In situations when entire racks are decommissioned, leading to many
> under-replicated blocks, loss of blocks can occur.
> Background: HDFS-1765
> This patch to trunk greatly improved the state of the replication policy
> implementation. Prior to the patch, the following details were true:
> * The block "priority queue" was no such thing: It was really set of
> trees that held blocks in natural ordering, that being by the blocks ID,
> which resulted in iterator walks over the blocks in pseudo-random order.
> * There was only a single index into an iteration over all of the
> blocks...
> * ... meaning the implementation was only successful in respecting
> priority levels on the first pass. Overall, the behavior was a
> round-robin-type scheduling of blocks.
> After the patch
> * A proper priority queue is implemented, preserving log n operations
> while iterating over blocks in the order added.
> * A separate index for each priority is key is kept...
> * ... allowing for processing of the highest priority blocks first
> regardless of which priority had last been processed.
> The change was suggested for branch-0.23 as well as trunk, but it does not
> appear to have been pulled in.
> The problem:
> Although the indices are now tracked in a better way, there is a
> synchronization issue since the indices are managed outside of methods to
> modify the contents of the queue.
> Removal of a block from a priority level without adjusting the index can mean
> that the index then points to the block after the block it originally pointed
> to. In the next round of scheduling for that priority level, the block
> originally pointed to by the index is skipped.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4385) Maven RAT plugin is not checking all source files
[ https://issues.apache.org/jira/browse/HDFS-4385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated HDFS-4385: Attachment: HDFS-4385.patch HDFS-4385-branch-0.23.patch fixing patches that I generated wrong. > Maven RAT plugin is not checking all source files > - > > Key: HDFS-4385 > URL: https://issues.apache.org/jira/browse/HDFS-4385 > Project: Hadoop HDFS > Issue Type: Bug > Components: build >Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.5 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Critical > Attachments: HDFS-4385-branch-0.23.patch, > HDFS-4385-branch-0.23.patch, HDFS-4385.patch, HDFS-4385.patch, > HDFS-4385-remove-branch23.sh, HDFS-4385-remove.sh > > > HDFS side of HADOOP-9097 > Running 'mvn apache-rat:check' passes, but running RAT by hand (by > downloading the JAR) produces some warnings for Java files, amongst others. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4366) Block Replication Policy Implementation May Skip Higher-Priority Blocks for Lower-Priority Blocks
[ https://issues.apache.org/jira/browse/HDFS-4366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Derek Dagit updated HDFS-4366: -- Attachment: HDFS-4366.patch A test timed out; probably transient. Uploading identical patch. > Block Replication Policy Implementation May Skip Higher-Priority Blocks for > Lower-Priority Blocks > - > > Key: HDFS-4366 > URL: https://issues.apache.org/jira/browse/HDFS-4366 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0, 0.23.5 >Reporter: Derek Dagit >Assignee: Derek Dagit > Attachments: HDFS-4366.patch, HDFS-4366.patch, > hdfs-4366-unittest.patch > > > In certain cases, higher-priority under-replicated blocks can be skipped by > the replication policy implementation. The current implementation maintains, > for each priority level, an index into a list of blocks that are > under-replicated. Together, the lists compose a priority queue (see note > later about branch-0.23). In some cases when blocks are removed from a list, > the caller (BlockManager) properly handles the index into the list from which > it removed a block. In some other cases, the index remains stationary while > the list changes. Whenever this happens, and the removed block happened to > be at or before the index, the implementation will skip over a block when > selecting blocks for replication work. > In situations when entire racks are decommissioned, leading to many > under-replicated blocks, loss of blocks can occur. > Background: HDFS-1765 > This patch to trunk greatly improved the state of the replication policy > implementation. Prior to the patch, the following details were true: > * The block "priority queue" was no such thing: It was really set of > trees that held blocks in natural ordering, that being by the blocks ID, > which resulted in iterator walks over the blocks in pseudo-random order. > * There was only a single index into an iteration over all of the > blocks... > * ... meaning the implementation was only successful in respecting > priority levels on the first pass. Overall, the behavior was a > round-robin-type scheduling of blocks. > After the patch > * A proper priority queue is implemented, preserving log n operations > while iterating over blocks in the order added. > * A separate index for each priority is key is kept... > * ... allowing for processing of the highest priority blocks first > regardless of which priority had last been processed. > The change was suggested for branch-0.23 as well as trunk, but it does not > appear to have been pulled in. > The problem: > Although the indices are now tracked in a better way, there is a > synchronization issue since the indices are managed outside of methods to > modify the contents of the queue. > Removal of a block from a priority level without adjusting the index can mean > that the index then points to the block after the block it originally pointed > to. In the next round of scheduling for that priority level, the block > originally pointed to by the index is skipped. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4328) TestLargeBlock#testLargeBlockSize is timing out
[ https://issues.apache.org/jira/browse/HDFS-4328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1355#comment-1355 ] Hudson commented on HDFS-4328: -- Integrated in Hadoop-Mapreduce-trunk #1310 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1310/]) HDFS-4328. TestLargeBlock#testLargeBlockSize is timing out. Contributed by Chris Nauroth. (Revision 1431867) Result = FAILURE atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1431867 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/DataTransferThrottler.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLargeBlock.java > TestLargeBlock#testLargeBlockSize is timing out > --- > > Key: HDFS-4328 > URL: https://issues.apache.org/jira/browse/HDFS-4328 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 3.0.0 >Reporter: Jason Lowe >Assignee: Chris Nauroth > Attachments: HDFS-4328.1.patch > > > For some time now TestLargeBlock#testLargeBlockSize has been timing out on > trunk. It is getting hung up during cluster shutdown, and after 15 minutes > surefire kills it and causes the build to fail since it exited uncleanly. > In addition to fixing the hang, we should consider adding a timeout parameter > to the @Test decorator for this test. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4377) Some trivial DN comment cleanup
[ https://issues.apache.org/jira/browse/HDFS-4377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551108#comment-13551108 ] Hudson commented on HDFS-4377: -- Integrated in Hadoop-Mapreduce-trunk #1310 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1310/]) HDFS-4377. Some trivial DN comment cleanup. Contributed by Eli Collins (Revision 1431753) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1431753 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java > Some trivial DN comment cleanup > --- > > Key: HDFS-4377 > URL: https://issues.apache.org/jira/browse/HDFS-4377 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.0-alpha >Reporter: Eli Collins >Assignee: Eli Collins >Priority: Trivial > Fix For: 2.0.3-alpha > > Attachments: hdfs-4377.txt, hdfs-4377.txt > > > DataStorage.java > - The "initilized" member is misspelled > - Comment what the storageID member is > DataNode.java > - Cleanup createNewStorageId comment (should mention the port is included and > is overly verbose) > BlockManager.java > - TreeSet in the comment should be TreeMap -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4367) GetDataEncryptionKeyResponseProto does not handle null response
[ https://issues.apache.org/jira/browse/HDFS-4367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551109#comment-13551109 ] Hudson commented on HDFS-4367: -- Integrated in Hadoop-Mapreduce-trunk #1310 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1310/]) HDFS-4367. GetDataEncryptionKeyResponseProto does not handle null response. Contributed by Suresh Srinivas. (Revision 1431459) Result = FAILURE suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1431459 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto > GetDataEncryptionKeyResponseProto does not handle null response > > > Key: HDFS-4367 > URL: https://issues.apache.org/jira/browse/HDFS-4367 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.0.2-alpha >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas >Priority: Blocker > Fix For: 2.0.3-alpha > > Attachments: HDFS-4367.patch > > > GetDataEncryptionKeyResponseProto member dataEncryptionKey should be optional > to handle null response. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4382) Fix typo MAX_NOT_CHANGED_INTERATIONS
[
https://issues.apache.org/jira/browse/HDFS-4382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551107#comment-13551107
]
Hudson commented on HDFS-4382:
--
Integrated in Hadoop-Mapreduce-trunk #1310 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1310/])
HDFS-4382. Fix typo MAX_NOT_CHANGED_INTERATIONS. Contributed by Ted Yu.
(Revision 1431726)
Result = FAILURE
suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1431726
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/NameNodeConnector.java
> Fix typo MAX_NOT_CHANGED_INTERATIONS
>
>
> Key: HDFS-4382
> URL: https://issues.apache.org/jira/browse/HDFS-4382
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 3.0.0
>
> Attachments: hdfs-4382-v1.txt
>
>
> Here is an example:
> {code}
> + if (notChangedIterations >= MAX_NOT_CHANGED_INTERATIONS) {
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4328) TestLargeBlock#testLargeBlockSize is timing out
[ https://issues.apache.org/jira/browse/HDFS-4328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551101#comment-13551101 ] Hudson commented on HDFS-4328: -- Integrated in Hadoop-Hdfs-trunk #1282 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1282/]) HDFS-4328. TestLargeBlock#testLargeBlockSize is timing out. Contributed by Chris Nauroth. (Revision 1431867) Result = FAILURE atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1431867 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/DataTransferThrottler.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLargeBlock.java > TestLargeBlock#testLargeBlockSize is timing out > --- > > Key: HDFS-4328 > URL: https://issues.apache.org/jira/browse/HDFS-4328 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 3.0.0 >Reporter: Jason Lowe >Assignee: Chris Nauroth > Attachments: HDFS-4328.1.patch > > > For some time now TestLargeBlock#testLargeBlockSize has been timing out on > trunk. It is getting hung up during cluster shutdown, and after 15 minutes > surefire kills it and causes the build to fail since it exited uncleanly. > In addition to fixing the hang, we should consider adding a timeout parameter > to the @Test decorator for this test. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4367) GetDataEncryptionKeyResponseProto does not handle null response
[ https://issues.apache.org/jira/browse/HDFS-4367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551099#comment-13551099 ] Hudson commented on HDFS-4367: -- Integrated in Hadoop-Hdfs-trunk #1282 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1282/]) HDFS-4367. GetDataEncryptionKeyResponseProto does not handle null response. Contributed by Suresh Srinivas. (Revision 1431459) Result = FAILURE suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1431459 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto > GetDataEncryptionKeyResponseProto does not handle null response > > > Key: HDFS-4367 > URL: https://issues.apache.org/jira/browse/HDFS-4367 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.0.2-alpha >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas >Priority: Blocker > Fix For: 2.0.3-alpha > > Attachments: HDFS-4367.patch > > > GetDataEncryptionKeyResponseProto member dataEncryptionKey should be optional > to handle null response. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4377) Some trivial DN comment cleanup
[ https://issues.apache.org/jira/browse/HDFS-4377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551098#comment-13551098 ] Hudson commented on HDFS-4377: -- Integrated in Hadoop-Hdfs-trunk #1282 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1282/]) HDFS-4377. Some trivial DN comment cleanup. Contributed by Eli Collins (Revision 1431753) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1431753 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java > Some trivial DN comment cleanup > --- > > Key: HDFS-4377 > URL: https://issues.apache.org/jira/browse/HDFS-4377 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.0-alpha >Reporter: Eli Collins >Assignee: Eli Collins >Priority: Trivial > Fix For: 2.0.3-alpha > > Attachments: hdfs-4377.txt, hdfs-4377.txt > > > DataStorage.java > - The "initilized" member is misspelled > - Comment what the storageID member is > DataNode.java > - Cleanup createNewStorageId comment (should mention the port is included and > is overly verbose) > BlockManager.java > - TreeSet in the comment should be TreeMap -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4382) Fix typo MAX_NOT_CHANGED_INTERATIONS
[
https://issues.apache.org/jira/browse/HDFS-4382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551097#comment-13551097
]
Hudson commented on HDFS-4382:
--
Integrated in Hadoop-Hdfs-trunk #1282 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1282/])
HDFS-4382. Fix typo MAX_NOT_CHANGED_INTERATIONS. Contributed by Ted Yu.
(Revision 1431726)
Result = FAILURE
suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1431726
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/NameNodeConnector.java
> Fix typo MAX_NOT_CHANGED_INTERATIONS
>
>
> Key: HDFS-4382
> URL: https://issues.apache.org/jira/browse/HDFS-4382
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 3.0.0
>
> Attachments: hdfs-4382-v1.txt
>
>
> Here is an example:
> {code}
> + if (notChangedIterations >= MAX_NOT_CHANGED_INTERATIONS) {
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2757) Cannot read a local block that's being written to when using the local read short circuit
[
https://issues.apache.org/jira/browse/HDFS-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551082#comment-13551082
]
Hudson commented on HDFS-2757:
--
Integrated in Hadoop-Hdfs-0.23-Build #491 (See
[https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/491/])
merge -r 1382408:1382409 for HDFS-2757 (Revision 1431506)
Result = FAILURE
kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1431506
Files :
* /hadoop/common/branches/branch-0.23
* /hadoop/common/branches/branch-0.23/hadoop-common-project
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-auth
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common
*
/hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs
*
/hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java
*
/hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-examples
* /hadoop/common/branches/branch-0.23/hadoop-project
* /hadoop/common/branches/branch-0.23/hadoop-project/src/site
*
/hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
*
/hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt
> Cannot read a local block that's being written to when using the local read
> short circuit
> -
>
> Key: HDFS-2757
> URL: https://issues.apache.org/jira/browse/HDFS-2757
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Jean-Daniel Cryans
>Assignee: Jean-Daniel Cryans
> Fix For: 1.2.0, 2.0.2-alpha, 0.23.6
>
> Attachments: hdfs-2757-b1.txt, HDFS-2757-branch-1.patch,
> HDFS-2757-branch-1-v2.patch, HDFS-2757-trunk.patch, hdfs-2757.txt
>
>
> When testing the tail'ing of a local file with the read short circuit on, I
> get:
> {noformat}
> 2012-01-06 00:17:31,598 WARN org.apache.hadoop.hdfs.DFSClient:
> BlockReaderLocal requested with incorrect offset: Offset 0 and length
> 8230400 don't match block blk_-2842916025951313698_454072 ( blockLen 124 )
> 2012-01-06 00:17:31,598 WARN org.apache.hadoop.hdfs.DFSClient:
> BlockReaderLocal: Removing blk_-2842916025951313698_454072 from cache because
> local file
> /export4/jdcryans/dfs/data/blocksBeingWritten/blk_-2842916025951313698 could
> not be opened.
> 2012-01-06 00:17:31,599 INFO org.apache.hadoop.hdfs.DFSClient: Failed to read
> block blk_-2842916025951313698_454072 on local machine java.io.IOException:
> Offset 0 and length 8230400 don't match block blk_-2842916025951313698_454072
> ( blockLen 124 )
> 2012-01-06 00:17:31,599 INFO org.apache.hadoop.hdfs.DFSClient: Try reading
> via the datanode on /10.4.13.38:51010
> java.io.EOFException:
> hdfs://sv4r11s38:9100/hbase-1/.logs/sv4r13s38,62023,1325808100311/sv4r13s38%2C62023%2C1325808100311.1325808100818,
> entryStart=7190409, pos=8230400, end=8230400, edit=5
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4261) TestBalancerWithNodeGroup times out
[
https://issues.apache.org/jira/browse/HDFS-4261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551064#comment-13551064
]
Junping Du commented on HDFS-4261:
--
To Aaron's previous comments:
{noformat}
Please make sure that whatever gets back-ported to branch-1 also gets
back-ported to branch-2. I was under the impression that all of this stuff was
only going to trunk
{noformat}
Thanks for reminder. We can start the backport work soon. BTW, shall we reuse
JIRAs in HADOOP-8817 (or in HADOOP-8468) or file new ones?
> TestBalancerWithNodeGroup times out
> ---
>
> Key: HDFS-4261
> URL: https://issues.apache.org/jira/browse/HDFS-4261
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer
>Affects Versions: 1.0.4, 1.1.1, 2.0.2-alpha
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Junping Du
> Fix For: 3.0.0
>
> Attachments: HDFS-4261-branch-1.patch, HDFS-4261-branch-1-v2.patch,
> HDFS-4261.patch, HDFS-4261-v2.patch, HDFS-4261-v3.patch, HDFS-4261-v4.patch,
> HDFS-4261-v5.patch, HDFS-4261-v6.patch, HDFS-4261-v7.patch,
> HDFS-4261-v8.patch, jstack-mac-18567, jstack-win-5488,
> org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup-output.txt.mac,
>
> org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup-output.txt.win,
> test-balancer-with-node-group-timeout.txt
>
>
> When I manually ran TestBalancerWithNodeGroup, it always timed out in my
> machine. Looking at the Jerkins report [build
> #3573|https://builds.apache.org/job/PreCommit-HDFS-Build/3573//testReport/org.apache.hadoop.hdfs.server.balancer/],
> TestBalancerWithNodeGroup somehow was skipped so that the problem was not
> detected.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4261) TestBalancerWithNodeGroup times out
[ https://issues.apache.org/jira/browse/HDFS-4261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551059#comment-13551059 ] Junping Du commented on HDFS-4261: -- Oh, sorry to forget to correct stupid typo here. Yes. resetData() should be in while-loop to work with initNodes(). But initNodes() seems to build everything from 0 (different with trunk) so previous tests didn't complain. Move this resetData() within while-loop but put it to front in case we want to change behaviours in initNodes() as trunk. Update to v2 patch. Thanks Nicholas for carefully review! > TestBalancerWithNodeGroup times out > --- > > Key: HDFS-4261 > URL: https://issues.apache.org/jira/browse/HDFS-4261 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer >Affects Versions: 1.0.4, 1.1.1, 2.0.2-alpha >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Junping Du > Fix For: 3.0.0 > > Attachments: HDFS-4261-branch-1.patch, HDFS-4261-branch-1-v2.patch, > HDFS-4261.patch, HDFS-4261-v2.patch, HDFS-4261-v3.patch, HDFS-4261-v4.patch, > HDFS-4261-v5.patch, HDFS-4261-v6.patch, HDFS-4261-v7.patch, > HDFS-4261-v8.patch, jstack-mac-18567, jstack-win-5488, > org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup-output.txt.mac, > > org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup-output.txt.win, > test-balancer-with-node-group-timeout.txt > > > When I manually ran TestBalancerWithNodeGroup, it always timed out in my > machine. Looking at the Jerkins report [build > #3573|https://builds.apache.org/job/PreCommit-HDFS-Build/3573//testReport/org.apache.hadoop.hdfs.server.balancer/], > TestBalancerWithNodeGroup somehow was skipped so that the problem was not > detected. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4261) TestBalancerWithNodeGroup times out
[ https://issues.apache.org/jira/browse/HDFS-4261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-4261: - Attachment: HDFS-4261-branch-1-v2.patch > TestBalancerWithNodeGroup times out > --- > > Key: HDFS-4261 > URL: https://issues.apache.org/jira/browse/HDFS-4261 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer >Affects Versions: 1.0.4, 1.1.1, 2.0.2-alpha >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Junping Du > Fix For: 3.0.0 > > Attachments: HDFS-4261-branch-1.patch, HDFS-4261-branch-1-v2.patch, > HDFS-4261.patch, HDFS-4261-v2.patch, HDFS-4261-v3.patch, HDFS-4261-v4.patch, > HDFS-4261-v5.patch, HDFS-4261-v6.patch, HDFS-4261-v7.patch, > HDFS-4261-v8.patch, jstack-mac-18567, jstack-win-5488, > org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup-output.txt.mac, > > org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup-output.txt.win, > test-balancer-with-node-group-timeout.txt > > > When I manually ran TestBalancerWithNodeGroup, it always timed out in my > machine. Looking at the Jerkins report [build > #3573|https://builds.apache.org/job/PreCommit-HDFS-Build/3573//testReport/org.apache.hadoop.hdfs.server.balancer/], > TestBalancerWithNodeGroup somehow was skipped so that the problem was not > detected. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4261) TestBalancerWithNodeGroup times out
[ https://issues.apache.org/jira/browse/HDFS-4261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-4261: - Attachment: (was: HDFS-4261-branch-1-v2.patch) > TestBalancerWithNodeGroup times out > --- > > Key: HDFS-4261 > URL: https://issues.apache.org/jira/browse/HDFS-4261 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer >Affects Versions: 1.0.4, 1.1.1, 2.0.2-alpha >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Junping Du > Fix For: 3.0.0 > > Attachments: HDFS-4261-branch-1.patch, HDFS-4261.patch, > HDFS-4261-v2.patch, HDFS-4261-v3.patch, HDFS-4261-v4.patch, > HDFS-4261-v5.patch, HDFS-4261-v6.patch, HDFS-4261-v7.patch, > HDFS-4261-v8.patch, jstack-mac-18567, jstack-win-5488, > org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup-output.txt.mac, > > org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup-output.txt.win, > test-balancer-with-node-group-timeout.txt > > > When I manually ran TestBalancerWithNodeGroup, it always timed out in my > machine. Looking at the Jerkins report [build > #3573|https://builds.apache.org/job/PreCommit-HDFS-Build/3573//testReport/org.apache.hadoop.hdfs.server.balancer/], > TestBalancerWithNodeGroup somehow was skipped so that the problem was not > detected. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4261) TestBalancerWithNodeGroup times out
[ https://issues.apache.org/jira/browse/HDFS-4261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-4261: - Attachment: HDFS-4261-branch-1-v2.patch > TestBalancerWithNodeGroup times out > --- > > Key: HDFS-4261 > URL: https://issues.apache.org/jira/browse/HDFS-4261 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer >Affects Versions: 1.0.4, 1.1.1, 2.0.2-alpha >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Junping Du > Fix For: 3.0.0 > > Attachments: HDFS-4261-branch-1.patch, HDFS-4261-branch-1-v2.patch, > HDFS-4261.patch, HDFS-4261-v2.patch, HDFS-4261-v3.patch, HDFS-4261-v4.patch, > HDFS-4261-v5.patch, HDFS-4261-v6.patch, HDFS-4261-v7.patch, > HDFS-4261-v8.patch, jstack-mac-18567, jstack-win-5488, > org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup-output.txt.mac, > > org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup-output.txt.win, > test-balancer-with-node-group-timeout.txt > > > When I manually ran TestBalancerWithNodeGroup, it always timed out in my > machine. Looking at the Jerkins report [build > #3573|https://builds.apache.org/job/PreCommit-HDFS-Build/3573//testReport/org.apache.hadoop.hdfs.server.balancer/], > TestBalancerWithNodeGroup somehow was skipped so that the problem was not > detected. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4261) TestBalancerWithNodeGroup times out
[ https://issues.apache.org/jira/browse/HDFS-4261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551047#comment-13551047 ] Tsz Wo (Nicholas), SZE commented on HDFS-4261: -- - The branch-1 patch also has the typo MAX_NO_PENDING_BLOCK_INTERATIONS. Let's fix it here so that we don't have to backport HDFS-4382. - Should resetData() be inside the while-loop? > TestBalancerWithNodeGroup times out > --- > > Key: HDFS-4261 > URL: https://issues.apache.org/jira/browse/HDFS-4261 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer >Affects Versions: 1.0.4, 1.1.1, 2.0.2-alpha >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Junping Du > Fix For: 3.0.0 > > Attachments: HDFS-4261-branch-1.patch, HDFS-4261.patch, > HDFS-4261-v2.patch, HDFS-4261-v3.patch, HDFS-4261-v4.patch, > HDFS-4261-v5.patch, HDFS-4261-v6.patch, HDFS-4261-v7.patch, > HDFS-4261-v8.patch, jstack-mac-18567, jstack-win-5488, > org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup-output.txt.mac, > > org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup-output.txt.win, > test-balancer-with-node-group-timeout.txt > > > When I manually ran TestBalancerWithNodeGroup, it always timed out in my > machine. Looking at the Jerkins report [build > #3573|https://builds.apache.org/job/PreCommit-HDFS-Build/3573//testReport/org.apache.hadoop.hdfs.server.balancer/], > TestBalancerWithNodeGroup somehow was skipped so that the problem was not > detected. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4328) TestLargeBlock#testLargeBlockSize is timing out
[ https://issues.apache.org/jira/browse/HDFS-4328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551039#comment-13551039 ] Hudson commented on HDFS-4328: -- Integrated in Hadoop-Yarn-trunk #93 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/93/]) HDFS-4328. TestLargeBlock#testLargeBlockSize is timing out. Contributed by Chris Nauroth. (Revision 1431867) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1431867 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/DataTransferThrottler.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLargeBlock.java > TestLargeBlock#testLargeBlockSize is timing out > --- > > Key: HDFS-4328 > URL: https://issues.apache.org/jira/browse/HDFS-4328 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 3.0.0 >Reporter: Jason Lowe >Assignee: Chris Nauroth > Attachments: HDFS-4328.1.patch > > > For some time now TestLargeBlock#testLargeBlockSize has been timing out on > trunk. It is getting hung up during cluster shutdown, and after 15 minutes > surefire kills it and causes the build to fail since it exited uncleanly. > In addition to fixing the hang, we should consider adding a timeout parameter > to the @Test decorator for this test. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4367) GetDataEncryptionKeyResponseProto does not handle null response
[ https://issues.apache.org/jira/browse/HDFS-4367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551037#comment-13551037 ] Hudson commented on HDFS-4367: -- Integrated in Hadoop-Yarn-trunk #93 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/93/]) HDFS-4367. GetDataEncryptionKeyResponseProto does not handle null response. Contributed by Suresh Srinivas. (Revision 1431459) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1431459 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto > GetDataEncryptionKeyResponseProto does not handle null response > > > Key: HDFS-4367 > URL: https://issues.apache.org/jira/browse/HDFS-4367 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.0.2-alpha >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas >Priority: Blocker > Fix For: 2.0.3-alpha > > Attachments: HDFS-4367.patch > > > GetDataEncryptionKeyResponseProto member dataEncryptionKey should be optional > to handle null response. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4377) Some trivial DN comment cleanup
[ https://issues.apache.org/jira/browse/HDFS-4377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551036#comment-13551036 ] Hudson commented on HDFS-4377: -- Integrated in Hadoop-Yarn-trunk #93 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/93/]) HDFS-4377. Some trivial DN comment cleanup. Contributed by Eli Collins (Revision 1431753) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1431753 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java > Some trivial DN comment cleanup > --- > > Key: HDFS-4377 > URL: https://issues.apache.org/jira/browse/HDFS-4377 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.0-alpha >Reporter: Eli Collins >Assignee: Eli Collins >Priority: Trivial > Fix For: 2.0.3-alpha > > Attachments: hdfs-4377.txt, hdfs-4377.txt > > > DataStorage.java > - The "initilized" member is misspelled > - Comment what the storageID member is > DataNode.java > - Cleanup createNewStorageId comment (should mention the port is included and > is overly verbose) > BlockManager.java > - TreeSet in the comment should be TreeMap -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4382) Fix typo MAX_NOT_CHANGED_INTERATIONS
[
https://issues.apache.org/jira/browse/HDFS-4382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551035#comment-13551035
]
Hudson commented on HDFS-4382:
--
Integrated in Hadoop-Yarn-trunk #93 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/93/])
HDFS-4382. Fix typo MAX_NOT_CHANGED_INTERATIONS. Contributed by Ted Yu.
(Revision 1431726)
Result = SUCCESS
suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1431726
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/NameNodeConnector.java
> Fix typo MAX_NOT_CHANGED_INTERATIONS
>
>
> Key: HDFS-4382
> URL: https://issues.apache.org/jira/browse/HDFS-4382
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 3.0.0
>
> Attachments: hdfs-4382-v1.txt
>
>
> Here is an example:
> {code}
> + if (notChangedIterations >= MAX_NOT_CHANGED_INTERATIONS) {
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4261) TestBalancerWithNodeGroup times out
[ https://issues.apache.org/jira/browse/HDFS-4261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551001#comment-13551001 ] Junping Du commented on HDFS-4261: -- Plan to lunch test-commit (with TestBalancerWithNodeGroup) 30 times to see the results. My local env have Jenkins backed with 4 VMs, so can run 4 times in paralleling, but still need more time. > TestBalancerWithNodeGroup times out > --- > > Key: HDFS-4261 > URL: https://issues.apache.org/jira/browse/HDFS-4261 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer >Affects Versions: 1.0.4, 1.1.1, 2.0.2-alpha >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Junping Du > Fix For: 3.0.0 > > Attachments: HDFS-4261-branch-1.patch, HDFS-4261.patch, > HDFS-4261-v2.patch, HDFS-4261-v3.patch, HDFS-4261-v4.patch, > HDFS-4261-v5.patch, HDFS-4261-v6.patch, HDFS-4261-v7.patch, > HDFS-4261-v8.patch, jstack-mac-18567, jstack-win-5488, > org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup-output.txt.mac, > > org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup-output.txt.win, > test-balancer-with-node-group-timeout.txt > > > When I manually ran TestBalancerWithNodeGroup, it always timed out in my > machine. Looking at the Jerkins report [build > #3573|https://builds.apache.org/job/PreCommit-HDFS-Build/3573//testReport/org.apache.hadoop.hdfs.server.balancer/], > TestBalancerWithNodeGroup somehow was skipped so that the problem was not > detected. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4261) TestBalancerWithNodeGroup times out
[ https://issues.apache.org/jira/browse/HDFS-4261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-4261: - Attachment: HDFS-4261-branch-1.patch backport patch to branch-1. > TestBalancerWithNodeGroup times out > --- > > Key: HDFS-4261 > URL: https://issues.apache.org/jira/browse/HDFS-4261 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer >Affects Versions: 1.0.4, 1.1.1, 2.0.2-alpha >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Junping Du > Fix For: 3.0.0 > > Attachments: HDFS-4261-branch-1.patch, HDFS-4261.patch, > HDFS-4261-v2.patch, HDFS-4261-v3.patch, HDFS-4261-v4.patch, > HDFS-4261-v5.patch, HDFS-4261-v6.patch, HDFS-4261-v7.patch, > HDFS-4261-v8.patch, jstack-mac-18567, jstack-win-5488, > org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup-output.txt.mac, > > org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup-output.txt.win, > test-balancer-with-node-group-timeout.txt > > > When I manually ran TestBalancerWithNodeGroup, it always timed out in my > machine. Looking at the Jerkins report [build > #3573|https://builds.apache.org/job/PreCommit-HDFS-Build/3573//testReport/org.apache.hadoop.hdfs.server.balancer/], > TestBalancerWithNodeGroup somehow was skipped so that the problem was not > detected. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4261) TestBalancerWithNodeGroup times out
[
https://issues.apache.org/jira/browse/HDFS-4261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13550998#comment-13550998
]
Junping Du commented on HDFS-4261:
--
I manually add TestBalancerWithNodeGroup on my test-commit, and following are
result:
{noformat}
+1 overall.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 2.0.1)
warnings.
+1 commit tests. The patch passed commit unit tests.
{noformat}
> TestBalancerWithNodeGroup times out
> ---
>
> Key: HDFS-4261
> URL: https://issues.apache.org/jira/browse/HDFS-4261
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer
>Affects Versions: 1.0.4, 1.1.1, 2.0.2-alpha
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Junping Du
> Fix For: 3.0.0
>
> Attachments: HDFS-4261-branch-1.patch, HDFS-4261.patch,
> HDFS-4261-v2.patch, HDFS-4261-v3.patch, HDFS-4261-v4.patch,
> HDFS-4261-v5.patch, HDFS-4261-v6.patch, HDFS-4261-v7.patch,
> HDFS-4261-v8.patch, jstack-mac-18567, jstack-win-5488,
> org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup-output.txt.mac,
>
> org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup-output.txt.win,
> test-balancer-with-node-group-timeout.txt
>
>
> When I manually ran TestBalancerWithNodeGroup, it always timed out in my
> machine. Looking at the Jerkins report [build
> #3573|https://builds.apache.org/job/PreCommit-HDFS-Build/3573//testReport/org.apache.hadoop.hdfs.server.balancer/],
> TestBalancerWithNodeGroup somehow was skipped so that the problem was not
> detected.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2802) Support for RW/RO snapshots in HDFS
[ https://issues.apache.org/jira/browse/HDFS-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13550983#comment-13550983 ] robert chiu commented on HDFS-2802: --- test > Support for RW/RO snapshots in HDFS > --- > > Key: HDFS-2802 > URL: https://issues.apache.org/jira/browse/HDFS-2802 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Reporter: Hari Mankude >Assignee: Hari Mankude > Attachments: HDFS-2802.20121101.patch, > HDFS-2802-meeting-minutes-121101.txt, HDFSSnapshotsDesign.pdf, snap.patch, > snapshot-design.pdf, snapshot-design.tex, snapshot-one-pager.pdf, > Snapshots20121018.pdf, Snapshots20121030.pdf, Snapshots.pdf > > > Snapshots are point in time images of parts of the filesystem or the entire > filesystem. Snapshots can be a read-only or a read-write point in time copy > of the filesystem. There are several use cases for snapshots in HDFS. I will > post a detailed write-up soon with with more information. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4388) DomainSocket should throw AsynchronousCloseException when appropriate
[
https://issues.apache.org/jira/browse/HDFS-4388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HDFS-4388:
---
Attachment: _05a.patch
> DomainSocket should throw AsynchronousCloseException when appropriate
> -
>
> Key: HDFS-4388
> URL: https://issues.apache.org/jira/browse/HDFS-4388
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
>Priority: Trivial
> Attachments: _05a.patch
>
>
> {{DomainSocket}} should throw {{AsynchronousCloseException}} when appropriate
> (i.e., when an {{accept}} or other blocking operation is interrupted by a
> concurrent close.) This is nicer than throwing a generic {{IOException}} or
> {{SocketException}}.
> Similarly, we should well throw {{ClosedChannelException}} when an operation
> is attempted on a closed {{DomainSocket}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4388) DomainSocket should throw AsynchronousCloseException when appropriate
Colin Patrick McCabe created HDFS-4388:
--
Summary: DomainSocket should throw AsynchronousCloseException when
appropriate
Key: HDFS-4388
URL: https://issues.apache.org/jira/browse/HDFS-4388
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Trivial
{{DomainSocket}} should throw {{AsynchronousCloseException}} when appropriate
(i.e., when an {{accept}} or other blocking operation is interrupted by a
concurrent close.) This is nicer than throwing a generic {{IOException}} or
{{SocketException}}.
Similarly, we should well throw {{ClosedChannelException}} when an operation is
attempted on a closed {{DomainSocket}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
