date:20120904

[
https://issues.apache.org/jira/browse/HDFS-3887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13447512#comment-13447512
]

Hadoop QA commented on HDFS-3887:
-

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12543596/HDFS-3887.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 2 new or modified test
files.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:

org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting

org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/3139//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3139//console

This message is automatically generated.

Remove redundant chooseTarget methods in BlockPlacementPolicy.java
--

Key: HDFS-3887
URL: https://issues.apache.org/jira/browse/HDFS-3887
Project: Hadoop HDFS
Issue Type: Improvement
Affects Versions: 3.0.0
Reporter: Jing Zhao
Assignee: Jing Zhao
Priority: Trivial
Attachments: HDFS-3887.patch

BlockPlacementPolicy.java contains multiple chooseTarget() methods with
different parameter lists. It is difficult to follow and understand the code
since some chooseTarget methods only have minor differences and some of them
are only invoked by testing code.
In this patch, I try to remove some of the chooseTarget methods and only keep
three of them: two abstract methods and the third one using BlockCollection
as its parameter.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2793) Add an admin command to trigger an edit log roll

[
https://issues.apache.org/jira/browse/HDFS-2793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13447518#comment-13447518
]

Hadoop QA commented on HDFS-2793:
-

+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12543618/hdfs-2793.txt
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 3 new or modified test
files.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

+1 core tests. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/3140//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3140//console

This message is automatically generated.

Add an admin command to trigger an edit log roll

Key: HDFS-2793
URL: https://issues.apache.org/jira/browse/HDFS-2793
Project: Hadoop HDFS
Issue Type: New Feature
Components: name-node
Affects Versions: 0.24.0
Reporter: Aaron T. Myers
Assignee: Todd Lipcon
Attachments: hdfs-2793.txt

This seems like it would also be helpful outside of the context of HA, but
especially so given that the standby can currently only read finalized log
segments.

[jira] [Commented] (HDFS-3887) Remove redundant chooseTarget methods in BlockPlacementPolicy.java


[ 
https://issues.apache.org/jira/browse/HDFS-3887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13447521#comment-13447521
 ] 

Jing Zhao commented on HDFS-3887:
-

Have rerun the two failed tests in my local machine and both tests passed.

 Remove redundant chooseTarget methods in BlockPlacementPolicy.java
 --

 Key: HDFS-3887
 URL: https://issues.apache.org/jira/browse/HDFS-3887
 Project: Hadoop HDFS
  Issue Type: Improvement
Affects Versions: 3.0.0
Reporter: Jing Zhao
Assignee: Jing Zhao
Priority: Trivial
 Attachments: HDFS-3887.patch


 BlockPlacementPolicy.java contains multiple chooseTarget() methods with 
 different parameter lists. It is difficult to follow and understand the code 
 since some chooseTarget methods only have minor differences and some of them 
 are only invoked by testing code. 
 In this patch, I try to remove some of the chooseTarget methods and only keep 
 three of them: two abstract methods and the third one using BlockCollection 
 as its parameter.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-1490) TransferFSImage should timeout

2012-09-04 Thread Vinay (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinay updated HDFS-1490:


Attachment: HDFS-1490.patch

Fixed typo.
Added @VisibleForTesting, since 'timeout' is used in test.

We have tested this in cluster, when the active nn's n/w broken. GetImage call 
got timeout.

 TransferFSImage should timeout
 --

 Key: HDFS-1490
 URL: https://issues.apache.org/jira/browse/HDFS-1490
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: name-node
Reporter: Dmytro Molkov
Assignee: Dmytro Molkov
Priority: Minor
 Attachments: HDFS-1490.patch, HDFS-1490.patch, HDFS-1490.patch, 
 HDFS-1490.patch


 Sometimes when primary crashes during image transfer secondary namenode would 
 hang trying to read the image from HTTP connection forever.
 It would be great to set timeouts on the connection so if something like that 
 happens there is no need to restart the secondary itself.
 In our case restarting components is handled by the set of scripts and since 
 the Secondary as the process is running it would just stay hung until we get 
 an alarm saying the checkpointing doesn't happen.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-1490) TransferFSImage should timeout

[
https://issues.apache.org/jira/browse/HDFS-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13447737#comment-13447737
]

Hadoop QA commented on HDFS-1490:
-

+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12543666/HDFS-1490.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 1 new or modified test
files.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

+1 core tests. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/3141//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3141//console

This message is automatically generated.

TransferFSImage should timeout
--

Key: HDFS-1490
URL: https://issues.apache.org/jira/browse/HDFS-1490
Project: Hadoop HDFS
Issue Type: Bug
Components: name-node
Reporter: Dmytro Molkov
Assignee: Dmytro Molkov
Priority: Minor
Attachments: HDFS-1490.patch, HDFS-1490.patch, HDFS-1490.patch,
HDFS-1490.patch

Sometimes when primary crashes during image transfer secondary namenode would
hang trying to read the image from HTTP connection forever.
It would be great to set timeouts on the connection so if something like that
happens there is no need to restart the secondary itself.
In our case restarting components is handled by the set of scripts and since
the Secondary as the process is running it would just stay hung until we get
an alarm saying the checkpointing doesn't happen.

[jira] [Commented] (HDFS-3876) NN should not RPC to self to find trash defaults (causes deadlock)


[ 
https://issues.apache.org/jira/browse/HDFS-3876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13447830#comment-13447830
 ] 

Todd Lipcon commented on HDFS-3876:
---

- When you clobber the trash interval in the configuration, you sohuld do it on 
a copy, rather than modifying the config that the user passed in.

{code}
+  // If we can not determine that trash is enabled server side then
+  // bail rather than potentially deleting a file when trash is enabled.
+  System.err.println(Failed to determine server trash configuration: 
+  + e.getMessage());
+  return false;
{code}

This doesn't seem to be what happens. See the TODO in {{Delete.java}}:
{code}
  // TODO: if the user wants the trash to be used but there is any
  // problem (ie. creating the trash dir, moving the item to be deleted,
  // etc), then the path will just be deleted because moveToTrash returns
  // false and it falls thru to fs.delete.  this doesn't seem right
{code}

if you actually want it to bail, it should probably throw an exception -- or 
return false but remove this comment, and separately address the TODO mentioned 
above.


{code}
+Configuration clientConf = new Configuration(serverConf);
+if (clientTrash) {
+  clientConf.setLong(FS_TRASH_INTERVAL_KEY, 200);
+}
{code}

Since you instantiate the client conf from {{serverConf}}, won't you end up 
with client trash enabled even if {{clientTrash}} is false?

 NN should not RPC to self to find trash defaults (causes deadlock)
 --

 Key: HDFS-3876
 URL: https://issues.apache.org/jira/browse/HDFS-3876
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: name-node
Affects Versions: 2.2.0-alpha
Reporter: Todd Lipcon
Assignee: Eli Collins
Priority: Blocker
 Attachments: hdfs-3876.txt, hdfs-3876.txt, hdfs-3876.txt


 When transitioning a SBN to active, I ran into the following situation:
 - the TrashPolicy first gets loaded by an IPC Server Handler thread. The 
 {{initialize}} function then tries to make an RPC to the same node to find 
 out the defaults.
 - This is happening inside the NN write lock (since it's part of the active 
 initialization). Hence, all of the other handler threads are already blocked 
 waiting to get the NN lock.
 - Since no handler threads are free, the RPC blocks forever and the NN never 
 enters active state.
 We need to have a general policy that the NN should never make RPCs to itself 
 for any reason, due to potential for deadlocks like this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-3884) QJM: Journal format() should reset cached values


[ 
https://issues.apache.org/jira/browse/HDFS-3884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13447853#comment-13447853
 ] 

Aaron T. Myers commented on HDFS-3884:
--

+1

 QJM: Journal format() should reset cached values
 

 Key: HDFS-3884
 URL: https://issues.apache.org/jira/browse/HDFS-3884
 Project: Hadoop HDFS
  Issue Type: Sub-task
Affects Versions: QuorumJournalManager (HDFS-3077)
Reporter: Todd Lipcon
Assignee: Todd Lipcon
 Fix For: QuorumJournalManager (HDFS-3077)

 Attachments: hdfs-3884.txt, hdfs-3884.txt


 Simple bug in the JournalNode: it caches certain values (eg accepted epoch) 
 in memory, and the cached values aren't reset when the journal is formatted. 
 So, after a format, further calls to the same Journal will see the old value 
 for accepted epoch, writer epoch, etc, preventing the journal from being 
 re-used until the JN is restarted.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-3886) Shutdown requests can possibly check for checkpoint issues (corrupted edits) and save a good namespace copy before closing down?

[
https://issues.apache.org/jira/browse/HDFS-3886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13447866#comment-13447866
]

Aaron T. Myers commented on HDFS-3886:
--

bq. I don't think you could easily do much with init.d as that is initiated by
the OS when it's doing a shutdown and it may be unrolling large parts of the
system: fast shutdowns are always appreciated before the monitoring layers
escalate.

I wasn't suggesting we modify the existing behavior of `/etc/init.d/* stop',
but rather that we add an extra, optional command along the lines of
`/etc/init.d/* clean-stop'. This would indeed make an RPC to the NN to enter
safemode, perform a save namespace, and then shut itself down. This wouldn't
affect the behavior of an OS shutdown, since that would still just use the
'stop' command. Does that make sense?

Shutdown requests can possibly check for checkpoint issues (corrupted edits)
and save a good namespace copy before closing down?

Key: HDFS-3886
URL: https://issues.apache.org/jira/browse/HDFS-3886
Project: Hadoop HDFS
Issue Type: Bug
Components: name-node
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Priority: Minor

HDFS-3878 sorta gives me this idea. Aside of having a method to download it
to a different location, we can also lock up the namesystem (or deactivate
the client rpc server) and save the namesystem before we complete up the
shutdown.
The init.d/shutdown scripts would have to work with this somehow though, to
not kill -9 it when in-process. Also, the new image may be stored in a
shutdown.chkpt directory, to not interfere in the regular dirs, but still
allow easier recovery.
Obviously this will still not work if all directories are broken. So maybe we
could have some configs to tackle that as well?
I haven't thought this through, so let me know what part is wrong to do :)

[jira] [Commented] (HDFS-3703) Decrease the datanode failure detection time

2012-09-04 Thread nkeywal (JIRA)

[
https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13447878#comment-13447878
]

nkeywal commented on HDFS-3703:
---

Hi Jing, Suresh,

I'm currently testing the patch with HBase trunk. I rebased your patch on hdfs
branch-2, as HBase does not yet build with hdfs trunk. I will keep you updated.

Decrease the datanode failure detection time

Key: HDFS-3703
URL: https://issues.apache.org/jira/browse/HDFS-3703
Project: Hadoop HDFS
Issue Type: Improvement
Components: data-node, name-node
Affects Versions: 1.0.3, 2.0.0-alpha
Reporter: nkeywal
Assignee: Suresh Srinivas
Attachments: HDFS-3703.patch

By default, if a box dies, the datanode will be marked as dead by the
namenode after 10:30 minutes. In the meantime, this datanode will still be
proposed by the nanenode to write blocks or to read replicas. It happens as
well if the datanode crashes: there is no shutdown hooks to tell the nanemode
we're not there anymore.
It especially an issue with HBase. HBase regionserver timeout for production
is often 30s. So with these configs, when a box dies HBase starts to recover
after 30s and, while 10 minutes, the namenode will consider the blocks on the
same box as available. Beyond the write errors, this will trigger a lot of
missed reads:
- during the recovery, HBase needs to read the blocks used on the dead box
(the ones in the 'HBase Write-Ahead-Log')
- after the recovery, reading these data blocks (the 'HBase region') will
fail 33% of the time with the default number of replica, slowering the data
access, especially when the errors are socket timeout (i.e. around 60s most
of the time).
Globally, it would be ideal if HDFS settings could be under HBase settings.
As a side note, HBase relies on ZooKeeper to detect regionservers issues.

[jira] [Commented] (HDFS-3540) Further improvement on recovery mode and edit log toleration in branch-1


[ 
https://issues.apache.org/jira/browse/HDFS-3540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13447906#comment-13447906
 ] 

Colin Patrick McCabe commented on HDFS-3540:


bq. Correct me if I am wrong: Recovery Mode without HDFS-3479 means the entire 
end-of-log is not checked and, therefore, the silent data loss length is not 
limited. It is even worst.

No, that is incorrect.

Recovery mode has always read up to the end of the log, and it always will.  
The confusion arises because sometimes we are not very good at determining 
where the end of the log is.

I filed and implemented HDFS-3479 because I noticed that in certain scenarios 
we would decide that the edit log ended before it really did because we spotted 
an {{OP_INVALID}}.

The unchecked region which we've been discussing only applied to HDFS-3479 
corruption, not to any other type of corruption.  Frankly, the unchecked region 
was a mistake.

However, none of this has *anything* to do with recovery mode.  HDFS-3004 and 
HDFS-3479 were separate JIRAs, that implemented separate features.

 Further improvement on recovery mode and edit log toleration in branch-1
 

 Key: HDFS-3540
 URL: https://issues.apache.org/jira/browse/HDFS-3540
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: name-node
Affects Versions: 1.2.0
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE

 *Recovery Mode*: HDFS-3479 backported HDFS-3335 to branch-1.  However, the 
 recovery mode feature in branch-1 is dramatically different from the recovery 
 mode in trunk since the edit log implementations in these two branch are 
 different.  For example, there is UNCHECKED_REGION_LENGTH in branch-1 but not 
 in trunk.
 *Edit Log Toleration*: HDFS-3521 added this feature to branch-1 to remedy 
 UNCHECKED_REGION_LENGTH and to tolerate edit log corruption.
 There are overlaps between these two features.  We study potential further 
 improvement in this issue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-3077) Quorum-based protocol for reading and writing edit logs

2012-09-04 Thread Suresh Srinivas (JIRA)

[
https://issues.apache.org/jira/browse/HDFS-3077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13447950#comment-13447950
]

Suresh Srinivas commented on HDFS-3077:
---

Hey Todd, I have not looked at the work in this branch in a while. One thing I
wanted to ask you about is, why are we using journal daemons to decide on an
epoch? Could zookeeper be used for doing the same? What are the advantages of
using journal daemons instead of zk? Adding this information to the document
might also be useful.

Quorum-based protocol for reading and writing edit logs
---

Key: HDFS-3077
URL: https://issues.apache.org/jira/browse/HDFS-3077
Project: Hadoop HDFS
Issue Type: New Feature
Components: ha, name-node
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Fix For: QuorumJournalManager (HDFS-3077)

Attachments: hdfs-3077-partial.txt, hdfs-3077.txt, hdfs-3077.txt,
hdfs-3077.txt, hdfs-3077.txt, hdfs-3077.txt, hdfs-3077.txt, hdfs-3077.txt,
qjournal-design.pdf, qjournal-design.pdf

Currently, one of the weak points of the HA design is that it relies on
shared storage such as an NFS filer for the shared edit log. One alternative
that has been proposed is to depend on BookKeeper, a ZooKeeper subproject
which provides a highly available replicated edit log on commodity hardware.
This JIRA is to implement another alternative, based on a quorum commit
protocol, integrated more tightly in HDFS and with the requirements driven
only by HDFS's needs rather than more generic use cases. More details to
follow.

[jira] [Commented] (HDFS-2793) Add an admin command to trigger an edit log roll


[ 
https://issues.apache.org/jira/browse/HDFS-2793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13448076#comment-13448076
 ] 

Aaron T. Myers commented on HDFS-2793:
--

Patch looks pretty good to me. Just a few little comments:

# I don't understand why this if(...) is necessary:
{code}
if (isFromClient) {
  checkSuperuserPrivilege();
}
{code}
It's certainly the case that arbitrary clients should be super users, but when 
would a non-super user node be calling this method? I think it should be safe 
to always check for super user privileges.
# Should the rollEdits RPC be marked {{@Idempotent}}? I can't think of a case 
when calling it twice would be problematic.

 Add an admin command to trigger an edit log roll
 

 Key: HDFS-2793
 URL: https://issues.apache.org/jira/browse/HDFS-2793
 Project: Hadoop HDFS
  Issue Type: New Feature
  Components: name-node
Affects Versions: 0.24.0
Reporter: Aaron T. Myers
Assignee: Todd Lipcon
 Attachments: hdfs-2793.txt


 This seems like it would also be helpful outside of the context of HA, but 
 especially so given that the standby can currently only read finalized log 
 segments.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-3383) libhdfs does not build on ARM because jni_md.h is not found

2012-09-04 Thread Trevor Robinson (JIRA)


[ 
https://issues.apache.org/jira/browse/HDFS-3383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13448078#comment-13448078
 ] 

Trevor Robinson commented on HDFS-3383:
---

Can this be committed to branch-1, since it has not been changed to use CMake?

 libhdfs does not build on ARM because jni_md.h is not found
 ---

 Key: HDFS-3383
 URL: https://issues.apache.org/jira/browse/HDFS-3383
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: libhdfs
Affects Versions: 0.23.1
 Environment: Linux 3.2.0-1412-omap4 #16-Ubuntu SMP PREEMPT Tue Apr 17 
 19:38:42 UTC 2012 armv7l armv7l armv7l GNU/Linux
 java version 1.7.0_04-ea
 Java(TM) SE Runtime Environment for Embedded (build 1.7.0_04-ea-b20, headless)
 Java HotSpot(TM) Embedded Server VM (build 23.0-b21, mixed mode, experimental)
Reporter: Trevor Robinson
 Attachments: HDFS-3383.patch


 The wrong include directory is used for jni_md.h:
 [INFO] --- make-maven-plugin:1.0-beta-1:make-install (compile) @ hadoop-hdfs 
 ---
 [INFO] /bin/bash ./libtool --tag=CC   --mode=compile gcc 
 -DPACKAGE_NAME=\libhdfs\ -DPACKAGE_TARNAME=\libhdfs\ 
 -DPACKAGE_VERSION=\0.1.0\ -DPACKAGE_STRING=\libhdfs\ 0.1.0\ 
 -DPACKAGE_BUGREPORT=\omal...@apache.org\ -DPACKAGE_URL=\\ 
 -DPACKAGE=\libhdfs\ -DVERSION=\0.1.0\ -DSTDC_HEADERS=1 
 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 
 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 
 -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 -DLT_OBJDIR=\.libs/\ -DHAVE_STRDUP=1 
 -DHAVE_STRERROR=1 -DHAVE_STRTOUL=1 -DHAVE_FCNTL_H=1 -DHAVE__BOOL=1 
 -DHAVE_STDBOOL_H=1 -I. -g -O2 -DOS_LINUX -DDSO_DLFCN -DCPU=\arm\ 
 -I/usr/lib/jvm/ejdk1.7.0_04/include -I/usr/lib/jvm/ejdk1.7.0_04/include/arm 
 -Wall -Wstrict-prototypes -MT hdfs.lo -MD -MP -MF .deps/hdfs.Tpo -c -o 
 hdfs.lo hdfs.c
 [INFO] libtool: compile:  gcc -DPACKAGE_NAME=\libhdfs\ 
 -DPACKAGE_TARNAME=\libhdfs\ -DPACKAGE_VERSION=\0.1.0\ 
 -DPACKAGE_STRING=\libhdfs 0.1.0\ 
 -DPACKAGE_BUGREPORT=\omal...@apache.org\ -DPACKAGE_URL=\\ 
 -DPACKAGE=\libhdfs\ -DVERSION=\0.1.0\ -DSTDC_HEADERS=1 
 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 
 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 
 -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 -DLT_OBJDIR=\.libs/\ -DHAVE_STRDUP=1 
 -DHAVE_STRERROR=1 -DHAVE_STRTOUL=1 -DHAVE_FCNTL_H=1 -DHAVE__BOOL=1 
 -DHAVE_STDBOOL_H=1 -I. -g -O2 -DOS_LINUX -DDSO_DLFCN -DCPU=\arm\ 
 -I/usr/lib/jvm/ejdk1.7.0_04/include -I/usr/lib/jvm/ejdk1.7.0_04/include/arm 
 -Wall -Wstrict-prototypes -MT hdfs.lo -MD -MP -MF .deps/hdfs.Tpo -c hdfs.c  
 -fPIC -DPIC -o .libs/hdfs.o
 [INFO] In file included from hdfs.h:33:0,
 [INFO]  from hdfs.c:19:
 [INFO] /usr/lib/jvm/ejdk1.7.0_04/include/jni.h:45:20: fatal error: jni_md.h: 
 No such file or directory
 [INFO] compilation terminated.
 [INFO] make: *** [hdfs.lo] Error 1
 The problem is caused by 
 hadoop-hdfs-project/hadoop-hdfs/src/main/native/m4/apsupport.m4 overriding 
 supported_os=arm when host_cpu=arm*; supported_os should remain linux, 
 since it determines the jni_md.h include path. OpenJDK 6 and 7 (in Ubuntu 
 12.04, at least) and Oracle EJDK put jni_md.h in include/linux. Not sure 
 if/why this ever worked before.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HDFS-3621) Add a main method to HdfsConfiguration, for debug purposes


 [ 
https://issues.apache.org/jira/browse/HDFS-3621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Plamen Jeliazkov reassigned HDFS-3621:
--

Assignee: Plamen Jeliazkov  (was: Linden Hillenbrand)

 Add a main method to HdfsConfiguration, for debug purposes
 --

 Key: HDFS-3621
 URL: https://issues.apache.org/jira/browse/HDFS-3621
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs client
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Assignee: Plamen Jeliazkov
Priority: Trivial
  Labels: newbie
 Attachments: HDFS-3621.patch


 Just like Configuration has a main() func that dumps XML out for debug 
 purposes, we should have a similar function under the HdfsConfiguration class 
 that does the same. This is useful in testing out app classpath setups at 
 times.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-3621) Add a main method to HdfsConfiguration, for debug purposes


 [ 
https://issues.apache.org/jira/browse/HDFS-3621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Plamen Jeliazkov updated HDFS-3621:
---

Attachment: HDFS-3621.patch

 Add a main method to HdfsConfiguration, for debug purposes
 --

 Key: HDFS-3621
 URL: https://issues.apache.org/jira/browse/HDFS-3621
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs client
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Assignee: Plamen Jeliazkov
Priority: Trivial
  Labels: newbie
 Attachments: HDFS-3621.patch


 Just like Configuration has a main() func that dumps XML out for debug 
 purposes, we should have a similar function under the HdfsConfiguration class 
 that does the same. This is useful in testing out app classpath setups at 
 times.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-3621) Add a main method to HdfsConfiguration, for debug purposes


 [ 
https://issues.apache.org/jira/browse/HDFS-3621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Plamen Jeliazkov updated HDFS-3621:
---

Status: Patch Available  (was: Open)

 Add a main method to HdfsConfiguration, for debug purposes
 --

 Key: HDFS-3621
 URL: https://issues.apache.org/jira/browse/HDFS-3621
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs client
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Assignee: Plamen Jeliazkov
Priority: Trivial
  Labels: newbie
 Attachments: HDFS-3621.patch


 Just like Configuration has a main() func that dumps XML out for debug 
 purposes, we should have a similar function under the HdfsConfiguration class 
 that does the same. This is useful in testing out app classpath setups at 
 times.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2793) Add an admin command to trigger an edit log roll


[ 
https://issues.apache.org/jira/browse/HDFS-2793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13448099#comment-13448099
 ] 

Todd Lipcon commented on HDFS-2793:
---

bq. It's certainly the case that arbitrary clients should be super users, but 
when would a non-super user node be calling this method? I think it should be 
safe to always check for super user privileges.

Is the 2NN's principal always considered a superuser? I think so, but just want 
to confirm.

bq. Should the rollEdits RPC be marked @Idempotent? I can't think of a case 
when calling it twice would be problematic.
Makes sense.

 Add an admin command to trigger an edit log roll
 

 Key: HDFS-2793
 URL: https://issues.apache.org/jira/browse/HDFS-2793
 Project: Hadoop HDFS
  Issue Type: New Feature
  Components: name-node
Affects Versions: 0.24.0
Reporter: Aaron T. Myers
Assignee: Todd Lipcon
 Attachments: hdfs-2793.txt


 This seems like it would also be helpful outside of the context of HA, but 
 especially so given that the standby can currently only read finalized log 
 segments.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2793) Add an admin command to trigger an edit log roll


[ 
https://issues.apache.org/jira/browse/HDFS-2793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13448105#comment-13448105
 ] 

Aaron T. Myers commented on HDFS-2793:
--

bq. Is the 2NN's principal always considered a superuser? I think so, but just 
want to confirm.

Pretty certain that's the case, yes.

 Add an admin command to trigger an edit log roll
 

 Key: HDFS-2793
 URL: https://issues.apache.org/jira/browse/HDFS-2793
 Project: Hadoop HDFS
  Issue Type: New Feature
  Components: name-node
Affects Versions: 0.24.0
Reporter: Aaron T. Myers
Assignee: Todd Lipcon
 Attachments: hdfs-2793.txt


 This seems like it would also be helpful outside of the context of HA, but 
 especially so given that the standby can currently only read finalized log 
 segments.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (HDFS-3866) HttpFS build should download Tomcat via Maven instead of directly


 [ 
https://issues.apache.org/jira/browse/HDFS-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Plamen Jeliazkov reassigned HDFS-3866:
--

Assignee: Plamen Jeliazkov

 HttpFS build should download Tomcat via Maven instead of directly
 -

 Key: HDFS-3866
 URL: https://issues.apache.org/jira/browse/HDFS-3866
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: build
Affects Versions: 2.0.0-alpha
 Environment: CDH4 build on CentOS 6.2
Reporter: Ryan Hennig
Assignee: Plamen Jeliazkov
Priority: Minor
 Attachments: HDFS-3866.patch


 When trying to enable a build of CDH4 in Jenkins, I got a build error due to 
 an attempt to download Tomcat from the internet directly instead of via Maven 
 and thus our internal Maven repository.
 The problem is due to this line in 
 src/hadoop-hdfs-project/hadoop-hdfs-httpfs/target/antrun/build-main.xml:
   get dest=downloads/tomcat.tar.gz skipexisting=true verbose=true 
 src=http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.32/bin/apache-tomcat-6.0.32.tar.gz/
 This build.xml is generated from 
 src/hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml:
 get 
 src=http://archive.apache.org/dist/tomcat/tomcat-6/v${tomcat.version}/bin/apache-tomcat-${tomcat.version}.tar.gz;
   dest=downloads/tomcat.tar.gz verbose=true skipexisting=true/
 Instead of directly downloading from a hardcoded location, the Tomcat 
 dependency should be managed by Maven.  This would enable the use of a local 
 repository for build machines without internet access.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-3866) HttpFS build should download Tomcat via Maven instead of directly


 [ 
https://issues.apache.org/jira/browse/HDFS-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Plamen Jeliazkov updated HDFS-3866:
---

Attachment: HDFS-3866.patch

 HttpFS build should download Tomcat via Maven instead of directly
 -

 Key: HDFS-3866
 URL: https://issues.apache.org/jira/browse/HDFS-3866
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: build
Affects Versions: 2.0.0-alpha
 Environment: CDH4 build on CentOS 6.2
Reporter: Ryan Hennig
Assignee: Plamen Jeliazkov
Priority: Minor
 Attachments: HDFS-3866.patch


 When trying to enable a build of CDH4 in Jenkins, I got a build error due to 
 an attempt to download Tomcat from the internet directly instead of via Maven 
 and thus our internal Maven repository.
 The problem is due to this line in 
 src/hadoop-hdfs-project/hadoop-hdfs-httpfs/target/antrun/build-main.xml:
   get dest=downloads/tomcat.tar.gz skipexisting=true verbose=true 
 src=http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.32/bin/apache-tomcat-6.0.32.tar.gz/
 This build.xml is generated from 
 src/hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml:
 get 
 src=http://archive.apache.org/dist/tomcat/tomcat-6/v${tomcat.version}/bin/apache-tomcat-${tomcat.version}.tar.gz;
   dest=downloads/tomcat.tar.gz verbose=true skipexisting=true/
 Instead of directly downloading from a hardcoded location, the Tomcat 
 dependency should be managed by Maven.  This would enable the use of a local 
 repository for build machines without internet access.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-3866) HttpFS build should download Tomcat via Maven instead of directly


[ 
https://issues.apache.org/jira/browse/HDFS-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13448106#comment-13448106
 ] 

Plamen Jeliazkov commented on HDFS-3866:


Taking Alejandro's advice here by making it a POM property for easy overriding 
with -D or editing.

 HttpFS build should download Tomcat via Maven instead of directly
 -

 Key: HDFS-3866
 URL: https://issues.apache.org/jira/browse/HDFS-3866
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: build
Affects Versions: 2.0.0-alpha
 Environment: CDH4 build on CentOS 6.2
Reporter: Ryan Hennig
Assignee: Plamen Jeliazkov
Priority: Minor
 Attachments: HDFS-3866.patch


 When trying to enable a build of CDH4 in Jenkins, I got a build error due to 
 an attempt to download Tomcat from the internet directly instead of via Maven 
 and thus our internal Maven repository.
 The problem is due to this line in 
 src/hadoop-hdfs-project/hadoop-hdfs-httpfs/target/antrun/build-main.xml:
   get dest=downloads/tomcat.tar.gz skipexisting=true verbose=true 
 src=http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.32/bin/apache-tomcat-6.0.32.tar.gz/
 This build.xml is generated from 
 src/hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml:
 get 
 src=http://archive.apache.org/dist/tomcat/tomcat-6/v${tomcat.version}/bin/apache-tomcat-${tomcat.version}.tar.gz;
   dest=downloads/tomcat.tar.gz verbose=true skipexisting=true/
 Instead of directly downloading from a hardcoded location, the Tomcat 
 dependency should be managed by Maven.  This would enable the use of a local 
 repository for build machines without internet access.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-3866) HttpFS build should download Tomcat via Maven instead of directly


 [ 
https://issues.apache.org/jira/browse/HDFS-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Plamen Jeliazkov updated HDFS-3866:
---

Status: Patch Available  (was: Open)

 HttpFS build should download Tomcat via Maven instead of directly
 -

 Key: HDFS-3866
 URL: https://issues.apache.org/jira/browse/HDFS-3866
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: build
Affects Versions: 2.0.0-alpha
 Environment: CDH4 build on CentOS 6.2
Reporter: Ryan Hennig
Assignee: Plamen Jeliazkov
Priority: Minor
 Attachments: HDFS-3866.patch


 When trying to enable a build of CDH4 in Jenkins, I got a build error due to 
 an attempt to download Tomcat from the internet directly instead of via Maven 
 and thus our internal Maven repository.
 The problem is due to this line in 
 src/hadoop-hdfs-project/hadoop-hdfs-httpfs/target/antrun/build-main.xml:
   get dest=downloads/tomcat.tar.gz skipexisting=true verbose=true 
 src=http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.32/bin/apache-tomcat-6.0.32.tar.gz/
 This build.xml is generated from 
 src/hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml:
 get 
 src=http://archive.apache.org/dist/tomcat/tomcat-6/v${tomcat.version}/bin/apache-tomcat-${tomcat.version}.tar.gz;
   dest=downloads/tomcat.tar.gz verbose=true skipexisting=true/
 Instead of directly downloading from a hardcoded location, the Tomcat 
 dependency should be managed by Maven.  This would enable the use of a local 
 repository for build machines without internet access.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-3888) BlockPlacementPolicyDefault#LOG should be removed

Jing Zhao created HDFS-3888:
---

 Summary: BlockPlacementPolicyDefault#LOG should be removed
 Key: HDFS-3888
 URL: https://issues.apache.org/jira/browse/HDFS-3888
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Jing Zhao
Assignee: Jing Zhao
Priority: Minor


BlockPlacementPolicyDefault#LOG should be removed as it hides LOG from the base 
class BlockPlacementPolicy. Also, in BlockPlacementPolicyDefault#chooseTarget 
method, the logic computing the maxTargetPerLoc can be made a separate method.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-3888) BlockPlacementPolicyDefault#LOG should be removed


 [ 
https://issues.apache.org/jira/browse/HDFS-3888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jing Zhao updated HDFS-3888:


Attachment: HDFS-3888.patch

 BlockPlacementPolicyDefault#LOG should be removed
 -

 Key: HDFS-3888
 URL: https://issues.apache.org/jira/browse/HDFS-3888
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Jing Zhao
Assignee: Jing Zhao
Priority: Minor
 Attachments: HDFS-3888.patch


 BlockPlacementPolicyDefault#LOG should be removed as it hides LOG from the 
 base class BlockPlacementPolicy. Also, in 
 BlockPlacementPolicyDefault#chooseTarget method, the logic computing the 
 maxTargetPerLoc can be made a separate method.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-3888) BlockPlacementPolicyDefault#LOG should be removed


 [ 
https://issues.apache.org/jira/browse/HDFS-3888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jing Zhao updated HDFS-3888:


Status: Patch Available  (was: Open)

 BlockPlacementPolicyDefault#LOG should be removed
 -

 Key: HDFS-3888
 URL: https://issues.apache.org/jira/browse/HDFS-3888
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Jing Zhao
Assignee: Jing Zhao
Priority: Minor
 Attachments: HDFS-3888.patch


 BlockPlacementPolicyDefault#LOG should be removed as it hides LOG from the 
 base class BlockPlacementPolicy. Also, in 
 BlockPlacementPolicyDefault#chooseTarget method, the logic computing the 
 maxTargetPerLoc can be made a separate method.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-3888) BlockPlacementPolicyDefault code cleanup

2012-09-04 Thread Suresh Srinivas (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-3888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suresh Srinivas updated HDFS-3888:
--

Summary: BlockPlacementPolicyDefault code cleanup  (was: 
BlockPlacementPolicyDefault#LOG should be removed)

 BlockPlacementPolicyDefault code cleanup
 

 Key: HDFS-3888
 URL: https://issues.apache.org/jira/browse/HDFS-3888
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Jing Zhao
Assignee: Jing Zhao
Priority: Minor
 Attachments: HDFS-3888.patch


 BlockPlacementPolicyDefault#LOG should be removed as it hides LOG from the 
 base class BlockPlacementPolicy. Also, in 
 BlockPlacementPolicyDefault#chooseTarget method, the logic computing the 
 maxTargetPerLoc can be made a separate method.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-3888) BlockPlacementPolicyDefault code cleanup


 [ 
https://issues.apache.org/jira/browse/HDFS-3888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jing Zhao updated HDFS-3888:


Status: Open  (was: Patch Available)

 BlockPlacementPolicyDefault code cleanup
 

 Key: HDFS-3888
 URL: https://issues.apache.org/jira/browse/HDFS-3888
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Jing Zhao
Assignee: Jing Zhao
Priority: Minor
 Attachments: HDFS-3888.patch


 BlockPlacementPolicyDefault#LOG should be removed as it hides LOG from the 
 base class BlockPlacementPolicy. Also, in 
 BlockPlacementPolicyDefault#chooseTarget method, the logic computing the 
 maxTargetPerLoc can be made a separate method.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-3866) HttpFS build should download Tomcat via Maven instead of directly


[ 
https://issues.apache.org/jira/browse/HDFS-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13448144#comment-13448144
 ] 

Alejandro Abdelnur commented on HDFS-3866:
--

+1

 HttpFS build should download Tomcat via Maven instead of directly
 -

 Key: HDFS-3866
 URL: https://issues.apache.org/jira/browse/HDFS-3866
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: build
Affects Versions: 2.0.0-alpha
 Environment: CDH4 build on CentOS 6.2
Reporter: Ryan Hennig
Assignee: Plamen Jeliazkov
Priority: Minor
 Attachments: HDFS-3866.patch


 When trying to enable a build of CDH4 in Jenkins, I got a build error due to 
 an attempt to download Tomcat from the internet directly instead of via Maven 
 and thus our internal Maven repository.
 The problem is due to this line in 
 src/hadoop-hdfs-project/hadoop-hdfs-httpfs/target/antrun/build-main.xml:
   get dest=downloads/tomcat.tar.gz skipexisting=true verbose=true 
 src=http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.32/bin/apache-tomcat-6.0.32.tar.gz/
 This build.xml is generated from 
 src/hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml:
 get 
 src=http://archive.apache.org/dist/tomcat/tomcat-6/v${tomcat.version}/bin/apache-tomcat-${tomcat.version}.tar.gz;
   dest=downloads/tomcat.tar.gz verbose=true skipexisting=true/
 Instead of directly downloading from a hardcoded location, the Tomcat 
 dependency should be managed by Maven.  This would enable the use of a local 
 repository for build machines without internet access.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-3866) HttpFS POM should have property where to download tomcat from


 [ 
https://issues.apache.org/jira/browse/HDFS-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alejandro Abdelnur updated HDFS-3866:
-

Summary: HttpFS POM should have property where to download tomcat from  
(was: HttpFS build should download Tomcat via Maven instead of directly)

changing summary of JIRA to reflect what the patch does.

 HttpFS POM should have property where to download tomcat from
 -

 Key: HDFS-3866
 URL: https://issues.apache.org/jira/browse/HDFS-3866
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: build
Affects Versions: 2.0.0-alpha
 Environment: CDH4 build on CentOS 6.2
Reporter: Ryan Hennig
Assignee: Plamen Jeliazkov
Priority: Minor
 Attachments: HDFS-3866.patch


 When trying to enable a build of CDH4 in Jenkins, I got a build error due to 
 an attempt to download Tomcat from the internet directly instead of via Maven 
 and thus our internal Maven repository.
 The problem is due to this line in 
 src/hadoop-hdfs-project/hadoop-hdfs-httpfs/target/antrun/build-main.xml:
   get dest=downloads/tomcat.tar.gz skipexisting=true verbose=true 
 src=http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.32/bin/apache-tomcat-6.0.32.tar.gz/
 This build.xml is generated from 
 src/hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml:
 get 
 src=http://archive.apache.org/dist/tomcat/tomcat-6/v${tomcat.version}/bin/apache-tomcat-${tomcat.version}.tar.gz;
   dest=downloads/tomcat.tar.gz verbose=true skipexisting=true/
 Instead of directly downloading from a hardcoded location, the Tomcat 
 dependency should be managed by Maven.  This would enable the use of a local 
 repository for build machines without internet access.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-3866) HttpFS POM should have property where to download tomcat from


 [ 
https://issues.apache.org/jira/browse/HDFS-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alejandro Abdelnur updated HDFS-3866:
-

Issue Type: Improvement  (was: Bug)

 HttpFS POM should have property where to download tomcat from
 -

 Key: HDFS-3866
 URL: https://issues.apache.org/jira/browse/HDFS-3866
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: build
Affects Versions: 2.0.0-alpha
 Environment: CDH4 build on CentOS 6.2
Reporter: Ryan Hennig
Assignee: Plamen Jeliazkov
Priority: Minor
 Attachments: HDFS-3866.patch


 When trying to enable a build of CDH4 in Jenkins, I got a build error due to 
 an attempt to download Tomcat from the internet directly instead of via Maven 
 and thus our internal Maven repository.
 The problem is due to this line in 
 src/hadoop-hdfs-project/hadoop-hdfs-httpfs/target/antrun/build-main.xml:
   get dest=downloads/tomcat.tar.gz skipexisting=true verbose=true 
 src=http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.32/bin/apache-tomcat-6.0.32.tar.gz/
 This build.xml is generated from 
 src/hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml:
 get 
 src=http://archive.apache.org/dist/tomcat/tomcat-6/v${tomcat.version}/bin/apache-tomcat-${tomcat.version}.tar.gz;
   dest=downloads/tomcat.tar.gz verbose=true skipexisting=true/
 Instead of directly downloading from a hardcoded location, the Tomcat 
 dependency should be managed by Maven.  This would enable the use of a local 
 repository for build machines without internet access.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-3866) HttpFS POM should have property where to download tomcat from


 [ 
https://issues.apache.org/jira/browse/HDFS-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alejandro Abdelnur updated HDFS-3866:
-

   Resolution: Fixed
Fix Version/s: 2.2.0-alpha
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Thanks Plamen. Committed to trunk and branch-2.

 HttpFS POM should have property where to download tomcat from
 -

 Key: HDFS-3866
 URL: https://issues.apache.org/jira/browse/HDFS-3866
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: build
Affects Versions: 2.0.0-alpha
 Environment: CDH4 build on CentOS 6.2
Reporter: Ryan Hennig
Assignee: Plamen Jeliazkov
Priority: Minor
 Fix For: 2.2.0-alpha

 Attachments: HDFS-3866.patch


 When trying to enable a build of CDH4 in Jenkins, I got a build error due to 
 an attempt to download Tomcat from the internet directly instead of via Maven 
 and thus our internal Maven repository.
 The problem is due to this line in 
 src/hadoop-hdfs-project/hadoop-hdfs-httpfs/target/antrun/build-main.xml:
   get dest=downloads/tomcat.tar.gz skipexisting=true verbose=true 
 src=http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.32/bin/apache-tomcat-6.0.32.tar.gz/
 This build.xml is generated from 
 src/hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml:
 get 
 src=http://archive.apache.org/dist/tomcat/tomcat-6/v${tomcat.version}/bin/apache-tomcat-${tomcat.version}.tar.gz;
   dest=downloads/tomcat.tar.gz verbose=true skipexisting=true/
 Instead of directly downloading from a hardcoded location, the Tomcat 
 dependency should be managed by Maven.  This would enable the use of a local 
 repository for build machines without internet access.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-3621) Add a main method to HdfsConfiguration, for debug purposes


[ 
https://issues.apache.org/jira/browse/HDFS-3621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13448162#comment-13448162
 ] 

Hadoop QA commented on HDFS-3621:
-

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12543748/HDFS-3621.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

-1 tests included.  The patch doesn't appear to include any new or modified 
tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 eclipse:eclipse.  The patch built with eclipse:eclipse.

+1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) 
warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

+1 core tests.  The patch passed unit tests in 
hadoop-hdfs-project/hadoop-hdfs.

+1 contrib tests.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HDFS-Build/3142//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3142//console

This message is automatically generated.

 Add a main method to HdfsConfiguration, for debug purposes
 --

 Key: HDFS-3621
 URL: https://issues.apache.org/jira/browse/HDFS-3621
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs client
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Assignee: Plamen Jeliazkov
Priority: Trivial
  Labels: newbie
 Attachments: HDFS-3621.patch


 Just like Configuration has a main() func that dumps XML out for debug 
 purposes, we should have a similar function under the HdfsConfiguration class 
 that does the same. This is useful in testing out app classpath setups at 
 times.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-3888) BlockPlacementPolicyDefault code cleanup


 [ 
https://issues.apache.org/jira/browse/HDFS-3888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jing Zhao updated HDFS-3888:


Attachment: HDFS-3888.patch

The code for computing the maxNodePerRack in chooseTarget() method is putting 
back because it may also change the value of numOfReplicas.

 BlockPlacementPolicyDefault code cleanup
 

 Key: HDFS-3888
 URL: https://issues.apache.org/jira/browse/HDFS-3888
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Jing Zhao
Assignee: Jing Zhao
Priority: Minor
 Attachments: HDFS-3888.patch, HDFS-3888.patch


 BlockPlacementPolicyDefault#LOG should be removed as it hides LOG from the 
 base class BlockPlacementPolicy. Also, in 
 BlockPlacementPolicyDefault#chooseTarget method, the logic computing the 
 maxTargetPerLoc can be made a separate method.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-3866) HttpFS POM should have property where to download tomcat from


[ 
https://issues.apache.org/jira/browse/HDFS-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13448171#comment-13448171
 ] 

Hudson commented on HDFS-3866:
--

Integrated in Hadoop-Common-trunk-Commit #2677 (See 
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2677/])
HDFS-3866. HttpFS POM should have property where to download tomcat from 
(zero45 via tucu) (Revision 1380927)

 Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1380927
Files : 
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt


 HttpFS POM should have property where to download tomcat from
 -

 Key: HDFS-3866
 URL: https://issues.apache.org/jira/browse/HDFS-3866
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: build
Affects Versions: 2.0.0-alpha
 Environment: CDH4 build on CentOS 6.2
Reporter: Ryan Hennig
Assignee: Plamen Jeliazkov
Priority: Minor
 Fix For: 2.2.0-alpha

 Attachments: HDFS-3866.patch


 When trying to enable a build of CDH4 in Jenkins, I got a build error due to 
 an attempt to download Tomcat from the internet directly instead of via Maven 
 and thus our internal Maven repository.
 The problem is due to this line in 
 src/hadoop-hdfs-project/hadoop-hdfs-httpfs/target/antrun/build-main.xml:
   get dest=downloads/tomcat.tar.gz skipexisting=true verbose=true 
 src=http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.32/bin/apache-tomcat-6.0.32.tar.gz/
 This build.xml is generated from 
 src/hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml:
 get 
 src=http://archive.apache.org/dist/tomcat/tomcat-6/v${tomcat.version}/bin/apache-tomcat-${tomcat.version}.tar.gz;
   dest=downloads/tomcat.tar.gz verbose=true skipexisting=true/
 Instead of directly downloading from a hardcoded location, the Tomcat 
 dependency should be managed by Maven.  This would enable the use of a local 
 repository for build machines without internet access.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-3866) HttpFS POM should have property where to download tomcat from


[ 
https://issues.apache.org/jira/browse/HDFS-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13448182#comment-13448182
 ] 

Hudson commented on HDFS-3866:
--

Integrated in Hadoop-Hdfs-trunk-Commit #2740 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2740/])
HDFS-3866. HttpFS POM should have property where to download tomcat from 
(zero45 via tucu) (Revision 1380927)

 Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1380927
Files : 
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt


 HttpFS POM should have property where to download tomcat from
 -

 Key: HDFS-3866
 URL: https://issues.apache.org/jira/browse/HDFS-3866
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: build
Affects Versions: 2.0.0-alpha
 Environment: CDH4 build on CentOS 6.2
Reporter: Ryan Hennig
Assignee: Plamen Jeliazkov
Priority: Minor
 Fix For: 2.2.0-alpha

 Attachments: HDFS-3866.patch


 When trying to enable a build of CDH4 in Jenkins, I got a build error due to 
 an attempt to download Tomcat from the internet directly instead of via Maven 
 and thus our internal Maven repository.
 The problem is due to this line in 
 src/hadoop-hdfs-project/hadoop-hdfs-httpfs/target/antrun/build-main.xml:
   get dest=downloads/tomcat.tar.gz skipexisting=true verbose=true 
 src=http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.32/bin/apache-tomcat-6.0.32.tar.gz/
 This build.xml is generated from 
 src/hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml:
 get 
 src=http://archive.apache.org/dist/tomcat/tomcat-6/v${tomcat.version}/bin/apache-tomcat-${tomcat.version}.tar.gz;
   dest=downloads/tomcat.tar.gz verbose=true skipexisting=true/
 Instead of directly downloading from a hardcoded location, the Tomcat 
 dependency should be managed by Maven.  This would enable the use of a local 
 repository for build machines without internet access.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-3540) Further improvement on recovery mode and edit log toleration in branch-1


[ 
https://issues.apache.org/jira/browse/HDFS-3540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13448185#comment-13448185
 ] 

Tsz Wo (Nicholas), SZE commented on HDFS-3540:
--

{quote}
Correct me if I am wrong: Recovery Mode without HDFS-3479 means the entire 
 end-of-log is not checked and, therefore, the silent data loss length is not 
 limited. It is even worst.

No, that is incorrect.

Recovery mode has always read up to the end of the log, and it always will. The 
confusion arises because sometimes we are not very good at determining where 
the end of the log is.
{quote}

I should be more specific: For Recovery Mode without HDFS-3479, if there is 
corruption in the middle of the edit log and the user chooses stop reading in 
recovery mode, then the remaining data of the edit log will not be checked.  Is 
it correct?

 Further improvement on recovery mode and edit log toleration in branch-1
 

 Key: HDFS-3540
 URL: https://issues.apache.org/jira/browse/HDFS-3540
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: name-node
Affects Versions: 1.2.0
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE

 *Recovery Mode*: HDFS-3479 backported HDFS-3335 to branch-1.  However, the 
 recovery mode feature in branch-1 is dramatically different from the recovery 
 mode in trunk since the edit log implementations in these two branch are 
 different.  For example, there is UNCHECKED_REGION_LENGTH in branch-1 but not 
 in trunk.
 *Edit Log Toleration*: HDFS-3521 added this feature to branch-1 to remedy 
 UNCHECKED_REGION_LENGTH and to tolerate edit log corruption.
 There are overlaps between these two features.  We study potential further 
 improvement in this issue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-3888) BlockPlacementPolicyDefault code cleanup


 [ 
https://issues.apache.org/jira/browse/HDFS-3888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsz Wo (Nicholas), SZE updated HDFS-3888:
-

Hadoop Flags: Reviewed

+1 patch looks good.

 BlockPlacementPolicyDefault code cleanup
 

 Key: HDFS-3888
 URL: https://issues.apache.org/jira/browse/HDFS-3888
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Jing Zhao
Assignee: Jing Zhao
Priority: Minor
 Attachments: HDFS-3888.patch, HDFS-3888.patch


 BlockPlacementPolicyDefault#LOG should be removed as it hides LOG from the 
 base class BlockPlacementPolicy. Also, in 
 BlockPlacementPolicyDefault#chooseTarget method, the logic computing the 
 maxTargetPerLoc can be made a separate method.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-3888) BlockPlacementPolicyDefault code cleanup


 [ 
https://issues.apache.org/jira/browse/HDFS-3888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsz Wo (Nicholas), SZE updated HDFS-3888:
-

Status: Patch Available  (was: Open)

 BlockPlacementPolicyDefault code cleanup
 

 Key: HDFS-3888
 URL: https://issues.apache.org/jira/browse/HDFS-3888
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Jing Zhao
Assignee: Jing Zhao
Priority: Minor
 Attachments: HDFS-3888.patch, HDFS-3888.patch


 BlockPlacementPolicyDefault#LOG should be removed as it hides LOG from the 
 base class BlockPlacementPolicy. Also, in 
 BlockPlacementPolicyDefault#chooseTarget method, the logic computing the 
 maxTargetPerLoc can be made a separate method.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-3887) Remove redundant chooseTarget methods in BlockPlacementPolicy.java


 [ 
https://issues.apache.org/jira/browse/HDFS-3887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsz Wo (Nicholas), SZE updated HDFS-3887:
-

 Component/s: name-node
Hadoop Flags: Reviewed

+1 patch looks good.

 Remove redundant chooseTarget methods in BlockPlacementPolicy.java
 --

 Key: HDFS-3887
 URL: https://issues.apache.org/jira/browse/HDFS-3887
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: name-node
Affects Versions: 3.0.0
Reporter: Jing Zhao
Assignee: Jing Zhao
Priority: Trivial
 Attachments: HDFS-3887.patch


 BlockPlacementPolicy.java contains multiple chooseTarget() methods with 
 different parameter lists. It is difficult to follow and understand the code 
 since some chooseTarget methods only have minor differences and some of them 
 are only invoked by testing code. 
 In this patch, I try to remove some of the chooseTarget methods and only keep 
 three of them: two abstract methods and the third one using BlockCollection 
 as its parameter.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-3888) BlockPlacementPolicyDefault code cleanup

[
https://issues.apache.org/jira/browse/HDFS-3888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13448209#comment-13448209
]

Hadoop QA commented on HDFS-3888:
-

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12543755/HDFS-3888.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

+1 core tests. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/3143//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3143//console

This message is automatically generated.

BlockPlacementPolicyDefault code cleanup

Key: HDFS-3888
URL: https://issues.apache.org/jira/browse/HDFS-3888
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Jing Zhao
Assignee: Jing Zhao
Priority: Minor
Attachments: HDFS-3888.patch, HDFS-3888.patch

BlockPlacementPolicyDefault#LOG should be removed as it hides LOG from the
base class BlockPlacementPolicy. Also, in
BlockPlacementPolicyDefault#chooseTarget method, the logic computing the
maxTargetPerLoc can be made a separate method.

[jira] [Updated] (HDFS-3887) Remove redundant chooseTarget methods in BlockPlacementPolicy.java


 [ 
https://issues.apache.org/jira/browse/HDFS-3887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsz Wo (Nicholas), SZE updated HDFS-3887:
-

   Resolution: Fixed
Fix Version/s: 2.2.0-alpha
   Status: Resolved  (was: Patch Available)

I have committed this.  Thanks, Jing!

 Remove redundant chooseTarget methods in BlockPlacementPolicy.java
 --

 Key: HDFS-3887
 URL: https://issues.apache.org/jira/browse/HDFS-3887
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: name-node
Affects Versions: 3.0.0
Reporter: Jing Zhao
Assignee: Jing Zhao
Priority: Trivial
 Fix For: 2.2.0-alpha

 Attachments: HDFS-3887.patch


 BlockPlacementPolicy.java contains multiple chooseTarget() methods with 
 different parameter lists. It is difficult to follow and understand the code 
 since some chooseTarget methods only have minor differences and some of them 
 are only invoked by testing code. 
 In this patch, I try to remove some of the chooseTarget methods and only keep 
 three of them: two abstract methods and the third one using BlockCollection 
 as its parameter.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-3887) Remove redundant chooseTarget methods in BlockPlacementPolicy.java


[ 
https://issues.apache.org/jira/browse/HDFS-3887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13448213#comment-13448213
 ] 

Hudson commented on HDFS-3887:
--

Integrated in Hadoop-Hdfs-trunk-Commit #2741 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2741/])
HDFS-3887. Remove redundant chooseTarget methods in BlockPlacementPolicy.  
Contributed by Jing Zhao (Revision 1380934)

 Result = SUCCESS
szetszwo : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1380934
Files : 
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java


 Remove redundant chooseTarget methods in BlockPlacementPolicy.java
 --

 Key: HDFS-3887
 URL: https://issues.apache.org/jira/browse/HDFS-3887
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: name-node
Affects Versions: 3.0.0
Reporter: Jing Zhao
Assignee: Jing Zhao
Priority: Trivial
 Fix For: 2.2.0-alpha

 Attachments: HDFS-3887.patch


 BlockPlacementPolicy.java contains multiple chooseTarget() methods with 
 different parameter lists. It is difficult to follow and understand the code 
 since some chooseTarget methods only have minor differences and some of them 
 are only invoked by testing code. 
 In this patch, I try to remove some of the chooseTarget methods and only keep 
 three of them: two abstract methods and the third one using BlockCollection 
 as its parameter.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-3887) Remove redundant chooseTarget methods in BlockPlacementPolicy.java


[ 
https://issues.apache.org/jira/browse/HDFS-3887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13448218#comment-13448218
 ] 

Hudson commented on HDFS-3887:
--

Integrated in Hadoop-Common-trunk-Commit #2678 (See 
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2678/])
HDFS-3887. Remove redundant chooseTarget methods in BlockPlacementPolicy.  
Contributed by Jing Zhao (Revision 1380934)

 Result = SUCCESS
szetszwo : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1380934
Files : 
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java


 Remove redundant chooseTarget methods in BlockPlacementPolicy.java
 --

 Key: HDFS-3887
 URL: https://issues.apache.org/jira/browse/HDFS-3887
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: name-node
Affects Versions: 3.0.0
Reporter: Jing Zhao
Assignee: Jing Zhao
Priority: Trivial
 Fix For: 2.2.0-alpha

 Attachments: HDFS-3887.patch


 BlockPlacementPolicy.java contains multiple chooseTarget() methods with 
 different parameter lists. It is difficult to follow and understand the code 
 since some chooseTarget methods only have minor differences and some of them 
 are only invoked by testing code. 
 In this patch, I try to remove some of the chooseTarget methods and only keep 
 three of them: two abstract methods and the third one using BlockCollection 
 as its parameter.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-3888) BlockPlacementPolicyDefault code cleanup


[ 
https://issues.apache.org/jira/browse/HDFS-3888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13448224#comment-13448224
 ] 

Hudson commented on HDFS-3888:
--

Integrated in Hadoop-Hdfs-trunk-Commit #2742 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2742/])
HDFS-3888. Clean up BlockPlacementPolicyDefault.  Contributed by Jing Zhao 
(Revision 1380939)

 Result = SUCCESS
szetszwo : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1380939
Files : 
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java


 BlockPlacementPolicyDefault code cleanup
 

 Key: HDFS-3888
 URL: https://issues.apache.org/jira/browse/HDFS-3888
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Jing Zhao
Assignee: Jing Zhao
Priority: Minor
 Attachments: HDFS-3888.patch, HDFS-3888.patch


 BlockPlacementPolicyDefault#LOG should be removed as it hides LOG from the 
 base class BlockPlacementPolicy. Also, in 
 BlockPlacementPolicyDefault#chooseTarget method, the logic computing the 
 maxTargetPerLoc can be made a separate method.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-3888) BlockPlacementPolicyDefault code cleanup


 [ 
https://issues.apache.org/jira/browse/HDFS-3888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsz Wo (Nicholas), SZE updated HDFS-3888:
-

   Resolution: Fixed
Fix Version/s: 2.2.0-alpha
   Status: Resolved  (was: Patch Available)

I have committed this.  Thanks, Jing!

 BlockPlacementPolicyDefault code cleanup
 

 Key: HDFS-3888
 URL: https://issues.apache.org/jira/browse/HDFS-3888
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Jing Zhao
Assignee: Jing Zhao
Priority: Minor
 Fix For: 2.2.0-alpha

 Attachments: HDFS-3888.patch, HDFS-3888.patch


 BlockPlacementPolicyDefault#LOG should be removed as it hides LOG from the 
 base class BlockPlacementPolicy. Also, in 
 BlockPlacementPolicyDefault#chooseTarget method, the logic computing the 
 maxTargetPerLoc can be made a separate method.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-3888) BlockPlacementPolicyDefault code cleanup


[ 
https://issues.apache.org/jira/browse/HDFS-3888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13448233#comment-13448233
 ] 

Hudson commented on HDFS-3888:
--

Integrated in Hadoop-Common-trunk-Commit #2679 (See 
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2679/])
HDFS-3888. Clean up BlockPlacementPolicyDefault.  Contributed by Jing Zhao 
(Revision 1380939)

 Result = SUCCESS
szetszwo : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1380939
Files : 
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java


 BlockPlacementPolicyDefault code cleanup
 

 Key: HDFS-3888
 URL: https://issues.apache.org/jira/browse/HDFS-3888
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Jing Zhao
Assignee: Jing Zhao
Priority: Minor
 Fix For: 2.2.0-alpha

 Attachments: HDFS-3888.patch, HDFS-3888.patch


 BlockPlacementPolicyDefault#LOG should be removed as it hides LOG from the 
 base class BlockPlacementPolicy. Also, in 
 BlockPlacementPolicyDefault#chooseTarget method, the logic computing the 
 maxTargetPerLoc can be made a separate method.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-3866) HttpFS POM should have property where to download tomcat from


[ 
https://issues.apache.org/jira/browse/HDFS-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13448243#comment-13448243
 ] 

Hudson commented on HDFS-3866:
--

Integrated in Hadoop-Mapreduce-trunk-Commit #2703 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2703/])
HDFS-3866. HttpFS POM should have property where to download tomcat from 
(zero45 via tucu) (Revision 1380927)

 Result = FAILURE
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1380927
Files : 
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt


 HttpFS POM should have property where to download tomcat from
 -

 Key: HDFS-3866
 URL: https://issues.apache.org/jira/browse/HDFS-3866
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: build
Affects Versions: 2.0.0-alpha
 Environment: CDH4 build on CentOS 6.2
Reporter: Ryan Hennig
Assignee: Plamen Jeliazkov
Priority: Minor
 Fix For: 2.2.0-alpha

 Attachments: HDFS-3866.patch


 When trying to enable a build of CDH4 in Jenkins, I got a build error due to 
 an attempt to download Tomcat from the internet directly instead of via Maven 
 and thus our internal Maven repository.
 The problem is due to this line in 
 src/hadoop-hdfs-project/hadoop-hdfs-httpfs/target/antrun/build-main.xml:
   get dest=downloads/tomcat.tar.gz skipexisting=true verbose=true 
 src=http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.32/bin/apache-tomcat-6.0.32.tar.gz/
 This build.xml is generated from 
 src/hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml:
 get 
 src=http://archive.apache.org/dist/tomcat/tomcat-6/v${tomcat.version}/bin/apache-tomcat-${tomcat.version}.tar.gz;
   dest=downloads/tomcat.tar.gz verbose=true skipexisting=true/
 Instead of directly downloading from a hardcoded location, the Tomcat 
 dependency should be managed by Maven.  This would enable the use of a local 
 repository for build machines without internet access.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-3540) Further improvement on recovery mode and edit log toleration in branch-1

[
https://issues.apache.org/jira/browse/HDFS-3540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13448277#comment-13448277
]

Colin Patrick McCabe commented on HDFS-3540:

bq. I should be more specific: For Recovery Mode without HDFS-3479, if there is
corruption in the middle of the edit log and the user chooses stop reading in
recovery mode, then the remaining data of the edit log will not be checked. Is
it correct?

Yes, if the user chooses stop reading in Recovery Mode, the rest of the edit
log will be discarded.

Further improvement on recovery mode and edit log toleration in branch-1

Key: HDFS-3540
URL: https://issues.apache.org/jira/browse/HDFS-3540
Project: Hadoop HDFS
Issue Type: Bug
Components: name-node
Affects Versions: 1.2.0
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE

*Recovery Mode*: HDFS-3479 backported HDFS-3335 to branch-1. However, the
recovery mode feature in branch-1 is dramatically different from the recovery
mode in trunk since the edit log implementations in these two branch are
different. For example, there is UNCHECKED_REGION_LENGTH in branch-1 but not
in trunk.
*Edit Log Toleration*: HDFS-3521 added this feature to branch-1 to remedy
UNCHECKED_REGION_LENGTH and to tolerate edit log corruption.
There are overlaps between these two features. We study potential further
improvement in this issue.

[jira] [Commented] (HDFS-3540) Further improvement on recovery mode and edit log toleration in branch-1


[ 
https://issues.apache.org/jira/browse/HDFS-3540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13448357#comment-13448357
 ] 

Colin Patrick McCabe commented on HDFS-3540:


Here's a table of the abilities of the two features (Recovery mode and edit log 
toleration):

|| ||skip over edits||discard the end of the edit log||requires administrator 
intervention||
|edit log toleration|no|yes|no|
|edit log recovery mode|branch-2 and later|yes|yes|

We have considered enhancing RM in branch-1 so that it can skip over certain 
edits.  When handling corruptions caused by HDFS-3652, it would have been nice 
to have this capability.

 Further improvement on recovery mode and edit log toleration in branch-1
 

 Key: HDFS-3540
 URL: https://issues.apache.org/jira/browse/HDFS-3540
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: name-node
Affects Versions: 1.2.0
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE

 *Recovery Mode*: HDFS-3479 backported HDFS-3335 to branch-1.  However, the 
 recovery mode feature in branch-1 is dramatically different from the recovery 
 mode in trunk since the edit log implementations in these two branch are 
 different.  For example, there is UNCHECKED_REGION_LENGTH in branch-1 but not 
 in trunk.
 *Edit Log Toleration*: HDFS-3521 added this feature to branch-1 to remedy 
 UNCHECKED_REGION_LENGTH and to tolerate edit log corruption.
 There are overlaps between these two features.  We study potential further 
 improvement in this issue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-2793) Add an admin command to trigger an edit log roll


 [ 
https://issues.apache.org/jira/browse/HDFS-2793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Todd Lipcon updated HDFS-2793:
--

Attachment: hdfs-2793.txt

How's this look?

 Add an admin command to trigger an edit log roll
 

 Key: HDFS-2793
 URL: https://issues.apache.org/jira/browse/HDFS-2793
 Project: Hadoop HDFS
  Issue Type: New Feature
  Components: name-node
Affects Versions: 0.24.0
Reporter: Aaron T. Myers
Assignee: Todd Lipcon
 Attachments: hdfs-2793.txt, hdfs-2793.txt


 This seems like it would also be helpful outside of the context of HA, but 
 especially so given that the standby can currently only read finalized log 
 segments.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2793) Add an admin command to trigger an edit log roll


[ 
https://issues.apache.org/jira/browse/HDFS-2793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13448364#comment-13448364
 ] 

Aaron T. Myers commented on HDFS-2793:
--

One nit, I think this should just be ClientProtocol:

{code}
+  @Override // ClientNamenodeProtocol
{code}

 Add an admin command to trigger an edit log roll
 

 Key: HDFS-2793
 URL: https://issues.apache.org/jira/browse/HDFS-2793
 Project: Hadoop HDFS
  Issue Type: New Feature
  Components: name-node
Affects Versions: 0.24.0
Reporter: Aaron T. Myers
Assignee: Todd Lipcon
 Attachments: hdfs-2793.txt, hdfs-2793.txt


 This seems like it would also be helpful outside of the context of HA, but 
 especially so given that the standby can currently only read finalized log 
 segments.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-3889) distcp silently ignores missing checksums

Colin Patrick McCabe created HDFS-3889:
--

 Summary: distcp silently ignores missing checksums
 Key: HDFS-3889
 URL: https://issues.apache.org/jira/browse/HDFS-3889
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: tools
Affects Versions: 2.2.0-alpha
Reporter: Colin Patrick McCabe
Priority: Minor


If distcp can't read the checksum files for the source and destination files-- 
for any reason-- it ignores the checksums and overwrites the destination file.  
It does produce a log message, but I think the correct behavior would be to 
throw an error and stop the distcp.

If the user really wants to ignore checksums, he or she can use 
{{-skipcrccheck}} to do so.

The relevant code is in DistCpUtils#checksumsAreEquals:
{code}
try {
  sourceChecksum = sourceFS.getFileChecksum(source);
  targetChecksum = targetFS.getFileChecksum(target);
} catch (IOException e) {
  LOG.error(Unable to retrieve checksum for  + source +  or  + target, 
e);
}
{code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-3889) distcp overwrites files even when there are missing checksums


 [ 
https://issues.apache.org/jira/browse/HDFS-3889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Patrick McCabe updated HDFS-3889:
---

Summary: distcp overwrites files even when there are missing checksums  
(was: distcp silently ignores missing checksums)

 distcp overwrites files even when there are missing checksums
 -

 Key: HDFS-3889
 URL: https://issues.apache.org/jira/browse/HDFS-3889
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: tools
Affects Versions: 2.2.0-alpha
Reporter: Colin Patrick McCabe
Priority: Minor

 If distcp can't read the checksum files for the source and destination 
 files-- for any reason-- it ignores the checksums and overwrites the 
 destination file.  It does produce a log message, but I think the correct 
 behavior would be to throw an error and stop the distcp.
 If the user really wants to ignore checksums, he or she can use 
 {{-skipcrccheck}} to do so.
 The relevant code is in DistCpUtils#checksumsAreEquals:
 {code}
 try {
   sourceChecksum = sourceFS.getFileChecksum(source);
   targetChecksum = targetFS.getFileChecksum(target);
 } catch (IOException e) {
   LOG.error(Unable to retrieve checksum for  + source +  or  + 
 target, e);
 }
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-3887) Remove redundant chooseTarget methods in BlockPlacementPolicy.java


[ 
https://issues.apache.org/jira/browse/HDFS-3887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13448370#comment-13448370
 ] 

Hudson commented on HDFS-3887:
--

Integrated in Hadoop-Mapreduce-trunk-Commit #2704 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2704/])
HDFS-3887. Remove redundant chooseTarget methods in BlockPlacementPolicy.  
Contributed by Jing Zhao (Revision 1380934)

 Result = FAILURE
szetszwo : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1380934
Files : 
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java


 Remove redundant chooseTarget methods in BlockPlacementPolicy.java
 --

 Key: HDFS-3887
 URL: https://issues.apache.org/jira/browse/HDFS-3887
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: name-node
Affects Versions: 3.0.0
Reporter: Jing Zhao
Assignee: Jing Zhao
Priority: Trivial
 Fix For: 2.2.0-alpha

 Attachments: HDFS-3887.patch


 BlockPlacementPolicy.java contains multiple chooseTarget() methods with 
 different parameter lists. It is difficult to follow and understand the code 
 since some chooseTarget methods only have minor differences and some of them 
 are only invoked by testing code. 
 In this patch, I try to remove some of the chooseTarget methods and only keep 
 three of them: two abstract methods and the third one using BlockCollection 
 as its parameter.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-3888) BlockPlacementPolicyDefault code cleanup


[ 
https://issues.apache.org/jira/browse/HDFS-3888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13448371#comment-13448371
 ] 

Hudson commented on HDFS-3888:
--

Integrated in Hadoop-Mapreduce-trunk-Commit #2704 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2704/])
HDFS-3888. Clean up BlockPlacementPolicyDefault.  Contributed by Jing Zhao 
(Revision 1380939)

 Result = FAILURE
szetszwo : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1380939
Files : 
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java


 BlockPlacementPolicyDefault code cleanup
 

 Key: HDFS-3888
 URL: https://issues.apache.org/jira/browse/HDFS-3888
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Jing Zhao
Assignee: Jing Zhao
Priority: Minor
 Fix For: 2.2.0-alpha

 Attachments: HDFS-3888.patch, HDFS-3888.patch


 BlockPlacementPolicyDefault#LOG should be removed as it hides LOG from the 
 base class BlockPlacementPolicy. Also, in 
 BlockPlacementPolicyDefault#chooseTarget method, the logic computing the 
 maxTargetPerLoc can be made a separate method.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-2793) Add an admin command to trigger an edit log roll


 [ 
https://issues.apache.org/jira/browse/HDFS-2793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Todd Lipcon updated HDFS-2793:
--

Attachment: hdfs-2793.txt

Oops. good catch. New rev.

 Add an admin command to trigger an edit log roll
 

 Key: HDFS-2793
 URL: https://issues.apache.org/jira/browse/HDFS-2793
 Project: Hadoop HDFS
  Issue Type: New Feature
  Components: name-node
Affects Versions: 0.24.0
Reporter: Aaron T. Myers
Assignee: Todd Lipcon
 Attachments: hdfs-2793.txt, hdfs-2793.txt, hdfs-2793.txt


 This seems like it would also be helpful outside of the context of HA, but 
 especially so given that the standby can currently only read finalized log 
 segments.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2793) Add an admin command to trigger an edit log roll


[ 
https://issues.apache.org/jira/browse/HDFS-2793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13448383#comment-13448383
 ] 

Aaron T. Myers commented on HDFS-2793:
--

The latest patch looks good to me. +1 pending Jenkings.

 Add an admin command to trigger an edit log roll
 

 Key: HDFS-2793
 URL: https://issues.apache.org/jira/browse/HDFS-2793
 Project: Hadoop HDFS
  Issue Type: New Feature
  Components: name-node
Affects Versions: 0.24.0
Reporter: Aaron T. Myers
Assignee: Todd Lipcon
 Attachments: hdfs-2793.txt, hdfs-2793.txt, hdfs-2793.txt


 This seems like it would also be helpful outside of the context of HA, but 
 especially so given that the standby can currently only read finalized log 
 segments.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-3054) distcp -skipcrccheck has no effect


[ 
https://issues.apache.org/jira/browse/HDFS-3054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13448386#comment-13448386
 ] 

Colin Patrick McCabe commented on HDFS-3054:


testing done:

I confirmed that copying files from a cluster running branch-1 derived code to 
a cluster running branch-2 derived code did *not* work unless {{-skipcrccheck}} 
was supplied.

The exception was this:

{code}
Error: java.io.IOException: File copy failed: hftp://172.22.1.204:6001/a/xx 
-- hdfs://localhost:6000/b/a/xx
at 
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:267)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:229)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:45)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:726)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:153)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1367)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:148)
Caused by: java.io.IOException: Couldn't run retriable-command: Copying 
hftp://172.22.1.204:6001/a/xx to hdfs://localhost:6000/b/a/xx
at 
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101)
at 
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:263)
... 10 more
Caused by: java.io.IOException: Check-sum mismatch between 
hftp://172.22.1.204:6001/a/xx and 
hdfs://localhost:6000/b/.distcp.tmp.attempt_1346456743556_0010_m_01_0
at 
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.compareCheckSums(RetriableFileCopyCommand.java:145)
at 
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:107)
at 
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:83)
at 
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
... 11 more
{code}

With {{-skipcrccheck}}, this problem did not occur.

 distcp -skipcrccheck has no effect
 --

 Key: HDFS-3054
 URL: https://issues.apache.org/jira/browse/HDFS-3054
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: tools
Affects Versions: 0.23.2, 2.0.0-alpha, 2.0.1-alpha, 2.2.0-alpha
Reporter: patrick white
Assignee: Colin Patrick McCabe
 Attachments: HDFS-3054.002.patch, hdfs-3054.patch


 Using distcp with '-skipcrccheck' still seems to cause CRC checksums to 
 happen. 
 Ran into this while debugging an issue associated with source and destination 
 having different blocksizes, and not using the preserve blocksize parameter 
 (-pb). In both 23.1 and 23.2 builds, trying to bypass the checksum 
 verification by using the '-skipcrcrcheck' parameter had no effect, the 
 distcp still failed on checksum errors.
 Test scenario to reproduce;
 do not use '-pb' and try a distcp from 20.205 (default blksize=128M) to .23 
 (default blksize=256M), the distcp fails on checksum errors, which is 
 expected due to checksum calculation (tiered aggregation of all blks). Trying 
 the same distcp only providing '-skipcrccheck' still fails with the same 
 checksum error, it is expected that checksum would now be bypassed and the 
 distcp would proceed.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-3863) QJM: track last committed txid


[ 
https://issues.apache.org/jira/browse/HDFS-3863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13448387#comment-13448387
 ] 

Eli Collins commented on HDFS-3863:
---

+1 looks great

 QJM: track last committed txid
 

 Key: HDFS-3863
 URL: https://issues.apache.org/jira/browse/HDFS-3863
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: ha
Affects Versions: QuorumJournalManager (HDFS-3077)
Reporter: Todd Lipcon
Assignee: Todd Lipcon
 Attachments: hdfs-3863-prelim.txt, hdfs-3863.txt, hdfs-3863.txt


 Per some discussion with [~stepinto] 
 [here|https://issues.apache.org/jira/browse/HDFS-3077?focusedCommentId=13422579page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13422579],
  we should keep track of the last committed txid on each JournalNode. Then 
 during any recovery operation, we can sanity-check that we aren't asked to 
 truncate a log to an earlier transaction.
 This is also a necessary step if we want to support reading from in-progress 
 segments in the future (since we should only allow reads up to the commit 
 point)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-3383) libhdfs does not build on ARM because jni_md.h is not found


[ 
https://issues.apache.org/jira/browse/HDFS-3383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13448391#comment-13448391
 ] 

Eli Collins commented on HDFS-3383:
---

Happy to review if you post a patch for branch-1.

 libhdfs does not build on ARM because jni_md.h is not found
 ---

 Key: HDFS-3383
 URL: https://issues.apache.org/jira/browse/HDFS-3383
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: libhdfs
Affects Versions: 0.23.1
 Environment: Linux 3.2.0-1412-omap4 #16-Ubuntu SMP PREEMPT Tue Apr 17 
 19:38:42 UTC 2012 armv7l armv7l armv7l GNU/Linux
 java version 1.7.0_04-ea
 Java(TM) SE Runtime Environment for Embedded (build 1.7.0_04-ea-b20, headless)
 Java HotSpot(TM) Embedded Server VM (build 23.0-b21, mixed mode, experimental)
Reporter: Trevor Robinson
 Attachments: HDFS-3383.patch


 The wrong include directory is used for jni_md.h:
 [INFO] --- make-maven-plugin:1.0-beta-1:make-install (compile) @ hadoop-hdfs 
 ---
 [INFO] /bin/bash ./libtool --tag=CC   --mode=compile gcc 
 -DPACKAGE_NAME=\libhdfs\ -DPACKAGE_TARNAME=\libhdfs\ 
 -DPACKAGE_VERSION=\0.1.0\ -DPACKAGE_STRING=\libhdfs\ 0.1.0\ 
 -DPACKAGE_BUGREPORT=\omal...@apache.org\ -DPACKAGE_URL=\\ 
 -DPACKAGE=\libhdfs\ -DVERSION=\0.1.0\ -DSTDC_HEADERS=1 
 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 
 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 
 -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 -DLT_OBJDIR=\.libs/\ -DHAVE_STRDUP=1 
 -DHAVE_STRERROR=1 -DHAVE_STRTOUL=1 -DHAVE_FCNTL_H=1 -DHAVE__BOOL=1 
 -DHAVE_STDBOOL_H=1 -I. -g -O2 -DOS_LINUX -DDSO_DLFCN -DCPU=\arm\ 
 -I/usr/lib/jvm/ejdk1.7.0_04/include -I/usr/lib/jvm/ejdk1.7.0_04/include/arm 
 -Wall -Wstrict-prototypes -MT hdfs.lo -MD -MP -MF .deps/hdfs.Tpo -c -o 
 hdfs.lo hdfs.c
 [INFO] libtool: compile:  gcc -DPACKAGE_NAME=\libhdfs\ 
 -DPACKAGE_TARNAME=\libhdfs\ -DPACKAGE_VERSION=\0.1.0\ 
 -DPACKAGE_STRING=\libhdfs 0.1.0\ 
 -DPACKAGE_BUGREPORT=\omal...@apache.org\ -DPACKAGE_URL=\\ 
 -DPACKAGE=\libhdfs\ -DVERSION=\0.1.0\ -DSTDC_HEADERS=1 
 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 
 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 
 -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 -DLT_OBJDIR=\.libs/\ -DHAVE_STRDUP=1 
 -DHAVE_STRERROR=1 -DHAVE_STRTOUL=1 -DHAVE_FCNTL_H=1 -DHAVE__BOOL=1 
 -DHAVE_STDBOOL_H=1 -I. -g -O2 -DOS_LINUX -DDSO_DLFCN -DCPU=\arm\ 
 -I/usr/lib/jvm/ejdk1.7.0_04/include -I/usr/lib/jvm/ejdk1.7.0_04/include/arm 
 -Wall -Wstrict-prototypes -MT hdfs.lo -MD -MP -MF .deps/hdfs.Tpo -c hdfs.c  
 -fPIC -DPIC -o .libs/hdfs.o
 [INFO] In file included from hdfs.h:33:0,
 [INFO]  from hdfs.c:19:
 [INFO] /usr/lib/jvm/ejdk1.7.0_04/include/jni.h:45:20: fatal error: jni_md.h: 
 No such file or directory
 [INFO] compilation terminated.
 [INFO] make: *** [hdfs.lo] Error 1
 The problem is caused by 
 hadoop-hdfs-project/hadoop-hdfs/src/main/native/m4/apsupport.m4 overriding 
 supported_os=arm when host_cpu=arm*; supported_os should remain linux, 
 since it determines the jni_md.h include path. OpenJDK 6 and 7 (in Ubuntu 
 12.04, at least) and Oracle EJDK put jni_md.h in include/linux. Not sure 
 if/why this ever worked before.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-3869) QJM: expose non-file journal manager details in web UI


[ 
https://issues.apache.org/jira/browse/HDFS-3869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13448392#comment-13448392
 ] 

Eli Collins commented on HDFS-3869:
---

+1 updated patch looks good

 QJM: expose non-file journal manager details in web UI
 --

 Key: HDFS-3869
 URL: https://issues.apache.org/jira/browse/HDFS-3869
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: name-node
Affects Versions: QuorumJournalManager (HDFS-3077)
Reporter: Todd Lipcon
Assignee: Todd Lipcon
 Attachments: dir-failed.png, hdfs-3869.txt, hdfs-3869.txt, 
 hdfs-3869.txt, hdfs-3869.txt, lagging-jn.png, open-for-read.png, 
 open-for-write.png


 Currently, the NN web UI only contains NN storage directories on local disk. 
 It should also include details about any non-file JournalManagers in use.
 This JIRA targets the QJM branch, but will be useful for BKJM as well.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-3889) distcp overwrites files even when there are missing checksums


[ 
https://issues.apache.org/jira/browse/HDFS-3889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13448393#comment-13448393
 ] 

Eli Collins commented on HDFS-3889:
---

Good find.  Or perhaps have an option that checks CRCs but just logs. I imagine 
the motivation for this was to not stop a large distcp job because one call to 
getFileChecksum failed (though it's robust, eg checks multiple DNs, so that 
should probably be rare).

 distcp overwrites files even when there are missing checksums
 -

 Key: HDFS-3889
 URL: https://issues.apache.org/jira/browse/HDFS-3889
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: tools
Affects Versions: 2.2.0-alpha
Reporter: Colin Patrick McCabe
Priority: Minor

 If distcp can't read the checksum files for the source and destination 
 files-- for any reason-- it ignores the checksums and overwrites the 
 destination file.  It does produce a log message, but I think the correct 
 behavior would be to throw an error and stop the distcp.
 If the user really wants to ignore checksums, he or she can use 
 {{-skipcrccheck}} to do so.
 The relevant code is in DistCpUtils#checksumsAreEquals:
 {code}
 try {
   sourceChecksum = sourceFS.getFileChecksum(source);
   targetChecksum = targetFS.getFileChecksum(target);
 } catch (IOException e) {
   LOG.error(Unable to retrieve checksum for  + source +  or  + 
 target, e);
 }
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-3888) BlockPlacementPolicyDefault code cleanup

[
https://issues.apache.org/jira/browse/HDFS-3888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13448399#comment-13448399
]

Hadoop QA commented on HDFS-3888:
-

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12543770/HDFS-3888.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:

org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
org.apache.hadoop.hdfs.TestPersistBlocks

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/3145//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3145//console

This message is automatically generated.

BlockPlacementPolicyDefault code cleanup

Attachments: HDFS-3888.patch, HDFS-3888.patch

[jira] [Commented] (HDFS-2793) Add an admin command to trigger an edit log roll

[
https://issues.apache.org/jira/browse/HDFS-2793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13448401#comment-13448401
]

Hadoop QA commented on HDFS-2793:
-

+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12543780/hdfs-2793.txt
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 1 new or modified test
files.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

+1 core tests. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/3146//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3146//console

This message is automatically generated.

Add an admin command to trigger an edit log roll

This seems like it would also be helpful outside of the context of HA, but
especially so given that the standby can currently only read finalized log
segments.

[jira] [Commented] (HDFS-3889) distcp overwrites files even when there are missing checksums


[ 
https://issues.apache.org/jira/browse/HDFS-3889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13448413#comment-13448413
 ] 

Colin Patrick McCabe commented on HDFS-3889:


I guess I need to look more into why {{getFileChecksum}} would fail.  If the 
block file simply had no checksum to begin with, then skipping it is clearly 
fine.  I think that's the case this was designed to handle (although I could be 
wrong here.)  However, if we expected it to have a checksum and it didn't, then 
that should be a red flag.

 distcp overwrites files even when there are missing checksums
 -

 Key: HDFS-3889
 URL: https://issues.apache.org/jira/browse/HDFS-3889
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: tools
Affects Versions: 2.2.0-alpha
Reporter: Colin Patrick McCabe
Priority: Minor

 If distcp can't read the checksum files for the source and destination 
 files-- for any reason-- it ignores the checksums and overwrites the 
 destination file.  It does produce a log message, but I think the correct 
 behavior would be to throw an error and stop the distcp.
 If the user really wants to ignore checksums, he or she can use 
 {{-skipcrccheck}} to do so.
 The relevant code is in DistCpUtils#checksumsAreEquals:
 {code}
 try {
   sourceChecksum = sourceFS.getFileChecksum(source);
   targetChecksum = targetFS.getFileChecksum(target);
 } catch (IOException e) {
   LOG.error(Unable to retrieve checksum for  + source +  or  + 
 target, e);
 }
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2793) Add an admin command to trigger an edit log roll

[
https://issues.apache.org/jira/browse/HDFS-2793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13448422#comment-13448422
]

Hadoop QA commented on HDFS-2793:
-

+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12543787/hdfs-2793.txt
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 1 new or modified test
files.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

+1 core tests. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/3147//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3147//console

This message is automatically generated.

Add an admin command to trigger an edit log roll

This seems like it would also be helpful outside of the context of HA, but
especially so given that the standby can currently only read finalized log
segments.

[jira] [Resolved] (HDFS-3863) QJM: track last committed txid


 [ 
https://issues.apache.org/jira/browse/HDFS-3863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Todd Lipcon resolved HDFS-3863.
---

   Resolution: Fixed
Fix Version/s: QuorumJournalManager (HDFS-3077)
 Hadoop Flags: Reviewed

Committed to branch, thanks for the reviews, Eli, and for the good ideas, Chao.

 QJM: track last committed txid
 

 Key: HDFS-3863
 URL: https://issues.apache.org/jira/browse/HDFS-3863
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: ha
Affects Versions: QuorumJournalManager (HDFS-3077)
Reporter: Todd Lipcon
Assignee: Todd Lipcon
 Fix For: QuorumJournalManager (HDFS-3077)

 Attachments: hdfs-3863-prelim.txt, hdfs-3863.txt, hdfs-3863.txt


 Per some discussion with [~stepinto] 
 [here|https://issues.apache.org/jira/browse/HDFS-3077?focusedCommentId=13422579page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13422579],
  we should keep track of the last committed txid on each JournalNode. Then 
 during any recovery operation, we can sanity-check that we aren't asked to 
 truncate a log to an earlier transaction.
 This is also a necessary step if we want to support reading from in-progress 
 segments in the future (since we should only allow reads up to the commit 
 point)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HDFS-3869) QJM: expose non-file journal manager details in web UI


 [ 
https://issues.apache.org/jira/browse/HDFS-3869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Todd Lipcon resolved HDFS-3869.
---

   Resolution: Fixed
Fix Version/s: QuorumJournalManager (HDFS-3077)
 Hadoop Flags: Reviewed

Committed to branch, thanks for the review.

 QJM: expose non-file journal manager details in web UI
 --

 Key: HDFS-3869
 URL: https://issues.apache.org/jira/browse/HDFS-3869
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: name-node
Affects Versions: QuorumJournalManager (HDFS-3077)
Reporter: Todd Lipcon
Assignee: Todd Lipcon
 Fix For: QuorumJournalManager (HDFS-3077)

 Attachments: dir-failed.png, hdfs-3869.txt, hdfs-3869.txt, 
 hdfs-3869.txt, hdfs-3869.txt, lagging-jn.png, open-for-read.png, 
 open-for-write.png


 Currently, the NN web UI only contains NN storage directories on local disk. 
 It should also include details about any non-file JournalManagers in use.
 This JIRA targets the QJM branch, but will be useful for BKJM as well.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-3884) QJM: Journal format() should reset cached values


 [ 
https://issues.apache.org/jira/browse/HDFS-3884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Todd Lipcon updated HDFS-3884:
--

Attachment: hdfs-3884.txt

Attaching patch I'm about to commit. Had to slightly modify it due to changes 
in the order the patches were staged vs committed. For it to compile, had to 
add the following getter which was from the patch in HDFS-3870:

{code}
+  synchronized public long getLastWriterEpoch() throws IOException {
+checkFormatted();
+return lastWriterEpoch.get();
+  }
{code}

Since it's a trivial change, I'll commit based on the earlier +1s.

 QJM: Journal format() should reset cached values
 

 Key: HDFS-3884
 URL: https://issues.apache.org/jira/browse/HDFS-3884
 Project: Hadoop HDFS
  Issue Type: Sub-task
Affects Versions: QuorumJournalManager (HDFS-3077)
Reporter: Todd Lipcon
Assignee: Todd Lipcon
 Fix For: QuorumJournalManager (HDFS-3077)

 Attachments: hdfs-3884.txt, hdfs-3884.txt, hdfs-3884.txt


 Simple bug in the JournalNode: it caches certain values (eg accepted epoch) 
 in memory, and the cached values aren't reset when the journal is formatted. 
 So, after a format, further calls to the same Journal will see the old value 
 for accepted epoch, writer epoch, etc, preventing the journal from being 
 re-used until the JN is restarted.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HDFS-3884) QJM: Journal format() should reset cached values


 [ 
https://issues.apache.org/jira/browse/HDFS-3884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Todd Lipcon resolved HDFS-3884.
---

  Resolution: Fixed
Target Version/s: QuorumJournalManager (HDFS-3077)
Hadoop Flags: Reviewed

committed to branch, thx for reviews.

 QJM: Journal format() should reset cached values
 

 Key: HDFS-3884
 URL: https://issues.apache.org/jira/browse/HDFS-3884
 Project: Hadoop HDFS
  Issue Type: Sub-task
Affects Versions: QuorumJournalManager (HDFS-3077)
Reporter: Todd Lipcon
Assignee: Todd Lipcon
 Fix For: QuorumJournalManager (HDFS-3077)

 Attachments: hdfs-3884.txt, hdfs-3884.txt, hdfs-3884.txt


 Simple bug in the JournalNode: it caches certain values (eg accepted epoch) 
 in memory, and the cached values aren't reset when the journal is formatted. 
 So, after a format, further calls to the same Journal will see the old value 
 for accepted epoch, writer epoch, etc, preventing the journal from being 
 re-used until the JN is restarted.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-3870) QJM: add metrics to JournalNode


[ 
https://issues.apache.org/jira/browse/HDFS-3870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13448460#comment-13448460
 ] 

Todd Lipcon commented on HDFS-3870:
---

Just realized I never commented to respond to Chao's suggestion. It was a good 
one, so it's included in the patch. Will commit momentarily.

 QJM: add metrics to JournalNode
 ---

 Key: HDFS-3870
 URL: https://issues.apache.org/jira/browse/HDFS-3870
 Project: Hadoop HDFS
  Issue Type: Sub-task
Affects Versions: QuorumJournalManager (HDFS-3077)
Reporter: Todd Lipcon
Assignee: Todd Lipcon
 Attachments: hdfs-3870.txt


 The JournalNode should expose some basic metrics through the usual interface. 
 In particular:
 - the writer epoch, accepted epoch,
 - the last written transaction ID and last committed txid (which may be newer 
 in case that it's in the process of catching up)
 - latency information for how long the syncs are taking
 Please feel free to suggest others that come to mind.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-3884) QJM: Journal format() should reset cached values


[ 
https://issues.apache.org/jira/browse/HDFS-3884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13448461#comment-13448461
 ] 

Eli Collins commented on HDFS-3884:
---

+1 to the delta, makes sense.

 QJM: Journal format() should reset cached values
 

 Key: HDFS-3884
 URL: https://issues.apache.org/jira/browse/HDFS-3884
 Project: Hadoop HDFS
  Issue Type: Sub-task
Affects Versions: QuorumJournalManager (HDFS-3077)
Reporter: Todd Lipcon
Assignee: Todd Lipcon
 Fix For: QuorumJournalManager (HDFS-3077)

 Attachments: hdfs-3884.txt, hdfs-3884.txt, hdfs-3884.txt


 Simple bug in the JournalNode: it caches certain values (eg accepted epoch) 
 in memory, and the cached values aren't reset when the journal is formatted. 
 So, after a format, further calls to the same Journal will see the old value 
 for accepted epoch, writer epoch, etc, preventing the journal from being 
 re-used until the JN is restarted.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HDFS-3870) QJM: add metrics to JournalNode


 [ 
https://issues.apache.org/jira/browse/HDFS-3870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Todd Lipcon resolved HDFS-3870.
---

   Resolution: Fixed
Fix Version/s: QuorumJournalManager (HDFS-3077)
 Hadoop Flags: Reviewed

Committed to branch, thanks for the review.

 QJM: add metrics to JournalNode
 ---

 Key: HDFS-3870
 URL: https://issues.apache.org/jira/browse/HDFS-3870
 Project: Hadoop HDFS
  Issue Type: Sub-task
Affects Versions: QuorumJournalManager (HDFS-3077)
Reporter: Todd Lipcon
Assignee: Todd Lipcon
 Fix For: QuorumJournalManager (HDFS-3077)

 Attachments: hdfs-3870.txt


 The JournalNode should expose some basic metrics through the usual interface. 
 In particular:
 - the writer epoch, accepted epoch,
 - the last written transaction ID and last committed txid (which may be newer 
 in case that it's in the process of catching up)
 - latency information for how long the syncs are taking
 Please feel free to suggest others that come to mind.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-2793) Add an admin command to trigger an edit log roll


 [ 
https://issues.apache.org/jira/browse/HDFS-2793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Todd Lipcon updated HDFS-2793:
--

   Resolution: Fixed
Fix Version/s: 2.2.0-alpha
   3.0.0
 Release Note: Introduced a new command, hdfs dfsadmin -rollEdits which 
requests that the active NameNode roll its edit log. This can be useful for 
administrators manually backing up log segments.
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Committed to branch-2 and trunk. Thanks for the review, Aaron.

 Add an admin command to trigger an edit log roll
 

 Key: HDFS-2793
 URL: https://issues.apache.org/jira/browse/HDFS-2793
 Project: Hadoop HDFS
  Issue Type: New Feature
  Components: name-node
Affects Versions: 0.24.0
Reporter: Aaron T. Myers
Assignee: Todd Lipcon
 Fix For: 3.0.0, 2.2.0-alpha

 Attachments: hdfs-2793.txt, hdfs-2793.txt, hdfs-2793.txt


 This seems like it would also be helpful outside of the context of HA, but 
 especially so given that the standby can currently only read finalized log 
 segments.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2793) Add an admin command to trigger an edit log roll


[ 
https://issues.apache.org/jira/browse/HDFS-2793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13448470#comment-13448470
 ] 

Hudson commented on HDFS-2793:
--

Integrated in Hadoop-Common-trunk-Commit #2682 (See 
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2682/])
HDFS-2793. Add an admin command to trigger an edit log roll. Contributed by 
Todd Lipcon. (Revision 1380982)

 Result = SUCCESS
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1380982
Files : 
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml


 Add an admin command to trigger an edit log roll
 

 Key: HDFS-2793
 URL: https://issues.apache.org/jira/browse/HDFS-2793
 Project: Hadoop HDFS
  Issue Type: New Feature
  Components: name-node
Affects Versions: 0.24.0
Reporter: Aaron T. Myers
Assignee: Todd Lipcon
 Fix For: 3.0.0, 2.2.0-alpha

 Attachments: hdfs-2793.txt, hdfs-2793.txt, hdfs-2793.txt


 This seems like it would also be helpful outside of the context of HA, but 
 especially so given that the standby can currently only read finalized log 
 segments.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2793) Add an admin command to trigger an edit log roll


[ 
https://issues.apache.org/jira/browse/HDFS-2793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13448473#comment-13448473
 ] 

Hudson commented on HDFS-2793:
--

Integrated in Hadoop-Hdfs-trunk-Commit #2745 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2745/])
HDFS-2793. Add an admin command to trigger an edit log roll. Contributed by 
Todd Lipcon. (Revision 1380982)

 Result = SUCCESS
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1380982
Files : 
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml


 Add an admin command to trigger an edit log roll
 

 Key: HDFS-2793
 URL: https://issues.apache.org/jira/browse/HDFS-2793
 Project: Hadoop HDFS
  Issue Type: New Feature
  Components: name-node
Affects Versions: 0.24.0
Reporter: Aaron T. Myers
Assignee: Todd Lipcon
 Fix For: 3.0.0, 2.2.0-alpha

 Attachments: hdfs-2793.txt, hdfs-2793.txt, hdfs-2793.txt


 This seems like it would also be helpful outside of the context of HA, but 
 especially so given that the standby can currently only read finalized log 
 segments.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-1490) TransferFSImage should timeout


[ 
https://issues.apache.org/jira/browse/HDFS-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13448476#comment-13448476
 ] 

Todd Lipcon commented on HDFS-1490:
---

+1, looks good to me. Will commit momentarily.

 TransferFSImage should timeout
 --

 Key: HDFS-1490
 URL: https://issues.apache.org/jira/browse/HDFS-1490
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: name-node
Reporter: Dmytro Molkov
Assignee: Dmytro Molkov
Priority: Minor
 Attachments: HDFS-1490.patch, HDFS-1490.patch, HDFS-1490.patch, 
 HDFS-1490.patch


 Sometimes when primary crashes during image transfer secondary namenode would 
 hang trying to read the image from HTTP connection forever.
 It would be great to set timeouts on the connection so if something like that 
 happens there is no need to restart the secondary itself.
 In our case restarting components is handled by the set of scripts and since 
 the Secondary as the process is running it would just stay hung until we get 
 an alarm saying the checkpointing doesn't happen.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-1490) TransferFSImage should timeout


 [ 
https://issues.apache.org/jira/browse/HDFS-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Todd Lipcon updated HDFS-1490:
--

Fix Version/s: 2.2.0-alpha
   3.0.0
   Status: In Progress  (was: Patch Available)

Committed to branch-2 and trunk. Thanks Dmytro and Vinay!

 TransferFSImage should timeout
 --

 Key: HDFS-1490
 URL: https://issues.apache.org/jira/browse/HDFS-1490
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: name-node
Reporter: Dmytro Molkov
Assignee: Dmytro Molkov
Priority: Minor
 Fix For: 3.0.0, 2.2.0-alpha

 Attachments: HDFS-1490.patch, HDFS-1490.patch, HDFS-1490.patch, 
 HDFS-1490.patch


 Sometimes when primary crashes during image transfer secondary namenode would 
 hang trying to read the image from HTTP connection forever.
 It would be great to set timeouts on the connection so if something like that 
 happens there is no need to restart the secondary itself.
 In our case restarting components is handled by the set of scripts and since 
 the Secondary as the process is running it would just stay hung until we get 
 an alarm saying the checkpointing doesn't happen.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HDFS-1490) TransferFSImage should timeout


 [ 
https://issues.apache.org/jira/browse/HDFS-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Todd Lipcon resolved HDFS-1490.
---

  Resolution: Fixed
Hadoop Flags: Reviewed

 TransferFSImage should timeout
 --

 Key: HDFS-1490
 URL: https://issues.apache.org/jira/browse/HDFS-1490
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: name-node
Reporter: Dmytro Molkov
Assignee: Dmytro Molkov
Priority: Minor
 Fix For: 3.0.0, 2.2.0-alpha

 Attachments: HDFS-1490.patch, HDFS-1490.patch, HDFS-1490.patch, 
 HDFS-1490.patch


 Sometimes when primary crashes during image transfer secondary namenode would 
 hang trying to read the image from HTTP connection forever.
 It would be great to set timeouts on the connection so if something like that 
 happens there is no need to restart the secondary itself.
 In our case restarting components is handled by the set of scripts and since 
 the Secondary as the process is running it would just stay hung until we get 
 an alarm saying the checkpointing doesn't happen.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-1490) TransferFSImage should timeout


[ 
https://issues.apache.org/jira/browse/HDFS-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13448484#comment-13448484
 ] 

Hudson commented on HDFS-1490:
--

Integrated in Hadoop-Common-trunk-Commit #2683 (See 
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2683/])
HDFS-1490. TransferFSImage should timeout. Contributed by Dmytro Molkov and 
Vinay. (Revision 1380988)

 Result = SUCCESS
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1380988
Files : 
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/TransferFsImage.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestTransferFsImage.java


 TransferFSImage should timeout
 --

 Key: HDFS-1490
 URL: https://issues.apache.org/jira/browse/HDFS-1490
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: name-node
Reporter: Dmytro Molkov
Assignee: Dmytro Molkov
Priority: Minor
 Fix For: 3.0.0, 2.2.0-alpha

 Attachments: HDFS-1490.patch, HDFS-1490.patch, HDFS-1490.patch, 
 HDFS-1490.patch


 Sometimes when primary crashes during image transfer secondary namenode would 
 hang trying to read the image from HTTP connection forever.
 It would be great to set timeouts on the connection so if something like that 
 happens there is no need to restart the secondary itself.
 In our case restarting components is handled by the set of scripts and since 
 the Secondary as the process is running it would just stay hung until we get 
 an alarm saying the checkpointing doesn't happen.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-1490) TransferFSImage should timeout


[ 
https://issues.apache.org/jira/browse/HDFS-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13448485#comment-13448485
 ] 

Hudson commented on HDFS-1490:
--

Integrated in Hadoop-Hdfs-trunk-Commit #2746 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2746/])
HDFS-1490. TransferFSImage should timeout. Contributed by Dmytro Molkov and 
Vinay. (Revision 1380988)

 Result = SUCCESS
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1380988
Files : 
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/TransferFsImage.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestTransferFsImage.java


 TransferFSImage should timeout
 --

 Key: HDFS-1490
 URL: https://issues.apache.org/jira/browse/HDFS-1490
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: name-node
Reporter: Dmytro Molkov
Assignee: Dmytro Molkov
Priority: Minor
 Fix For: 3.0.0, 2.2.0-alpha

 Attachments: HDFS-1490.patch, HDFS-1490.patch, HDFS-1490.patch, 
 HDFS-1490.patch


 Sometimes when primary crashes during image transfer secondary namenode would 
 hang trying to read the image from HTTP connection forever.
 It would be great to set timeouts on the connection so if something like that 
 happens there is no need to restart the secondary itself.
 In our case restarting components is handled by the set of scripts and since 
 the Secondary as the process is running it would just stay hung until we get 
 an alarm saying the checkpointing doesn't happen.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-2793) Add an admin command to trigger an edit log roll