[jira] [Created] (HDFS-3308) hftp/webhdfs can't get tokens if authority has no port
hftp/webhdfs can't get tokens if authority has no port -- Key: HDFS-3308 URL: https://issues.apache.org/jira/browse/HDFS-3308 Project: Hadoop HDFS Issue Type: Bug Components: hdfs client Affects Versions: 0.23.0, 0.24.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Priority: Critical Token acquisition fails if a hftp or webhdfs filesystem is obtained with no port in the authority. Building a token service requires a port, and the renewer needs the port. The default port is not being used when there is no port in the uri. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3307) when a file
when a file --- Key: HDFS-3307 URL: https://issues.apache.org/jira/browse/HDFS-3307 Project: Hadoop HDFS Issue Type: Bug Reporter: yixiaohua -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3306) fuse_dfs: don't lock release operations
fuse_dfs: don't lock release operations --- Key: HDFS-3306 URL: https://issues.apache.org/jira/browse/HDFS-3306 Project: Hadoop HDFS Issue Type: Bug Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor There's no need to lock release operations in FUSE, because release can only be called once on a fuse_file_info structure. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3304) fix fuse_dfs build
fix fuse_dfs build
--
Key: HDFS-3304
URL: https://issues.apache.org/jira/browse/HDFS-3304
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 0.23.0
Reporter: Colin Patrick McCabe
Priority: Minor
The fuse_dfs build is broken in several ways. If you run:
{code}
mvn compile -DskipTests -Pnative
mvn compile -DskipTests -Pfuse
{code}
You get the following error message:
{code}
[exec]
/usr/lib64/gcc/x86_64-suse-linux/4.6/../../../../x86_64-suse-linux/bin/ld:
cannot find -lhdfs
[exec] collect2: ld returned 1 exit status
[exec] make[1]: *** [fuse_dfs] Error 1
[exec] make: *** [all-recursive] Error 1
{code}
libhdfs.so was created, but the -Pfuse build doesn't know where it is and can't
link against it.
Also, should ''mvn install -Pfuse'' be copying fuse_dfs somewhere?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3305) GetImageServlet should considered SBN a valid requestor in a secure HA setup
GetImageServlet should considered SBN a valid requestor in a secure HA setup Key: HDFS-3305 URL: https://issues.apache.org/jira/browse/HDFS-3305 Project: Hadoop HDFS Issue Type: Bug Components: ha, name-node Affects Versions: 2.0.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Right now only the NN and 2NN are considered valid requestors. This won't work if the ANN and SBN use distinct principal names. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3303) RemoteEditLogManifest doesn't need to implements Writable
RemoteEditLogManifest doesn't need to implements Writable - Key: HDFS-3303 URL: https://issues.apache.org/jira/browse/HDFS-3303 Project: Hadoop HDFS Issue Type: Bug Reporter: Brandon Li Assignee: Brandon Li Priority: Minor Since we are using protocol buffers, RemoteEditLogManifest doesn't need to implements Writable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3302) Review and improve HDFS trash documentation
Review and improve HDFS trash documentation --- Key: HDFS-3302 URL: https://issues.apache.org/jira/browse/HDFS-3302 Project: Hadoop HDFS Issue Type: Improvement Components: documentation Affects Versions: 3.0.0 Reporter: Harsh J Assignee: Harsh J Improve Trash documentation for users. (0.23 published release docs are missing original HDFS docs btw...) A set of FAQ-like questions can be found on HDFS-2740 I'll update the ticket shortly with the areas to cover in the docs, as enabling trash by default (HDFS-2740) would be considered as a wide behavior change per its follow ups. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3301) Add public waitOnSafeMode API as HdfsUtils
Add public waitOnSafeMode API as HdfsUtils
---
Key: HDFS-3301
URL: https://issues.apache.org/jira/browse/HDFS-3301
Project: Hadoop HDFS
Issue Type: Sub-task
Components: hdfs client
Affects Versions: 3.0.0
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
Add public waitOnSafeMode API as HdfsUtils.
I have seen this util api in Hbase and using FSCOnstants in it. Currently that
is deprecated and moved the SafeModeActions to HdfsConstants and also marked as
private audience. So, it wil help adding such api in HdfsUtils itself to avoid
the need of accessing HdfsConstants.
from Hbase FSUtils class.
{code}
/**
* If DFS, check safe mode and if so, wait until we clear it.
* @param conf configuration
* @param wait Sleep between retries
* @throws IOException e
*/
public static void waitOnSafeMode(final Configuration conf,
final long wait)
throws IOException {
FileSystem fs = FileSystem.get(conf);
if (!(fs instanceof DistributedFileSystem)) return;
DistributedFileSystem dfs = (DistributedFileSystem)fs;
// Make sure dfs is not in safe mode
while
(dfs.setSafeMode(org.apache.hadoop.hdfs.protocol.FSConstants.SafeModeAction.SAFEMODE_GET))
{
LOG.info("Waiting for dfs to exit safe mode...");
try {
Thread.sleep(wait);
} catch (InterruptedException e) {
//continue
}
}
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3300) stream the edit segments to NameNode when NameNode starts up
stream the edit segments to NameNode when NameNode starts up Key: HDFS-3300 URL: https://issues.apache.org/jira/browse/HDFS-3300 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Brandon Li Edit logs are saved on Journal daemon. When NameNode starts, it loads the latest image file and then streams the edit logs from an active Journal daemon. Currently we are using http to transfer edit files between two journal daemons/nodes or between a journal daemon and a NameNode. To get edit file from Journal daemon, the NameNode has to download it first and then read it from the disk. To avoid the slow start-up time, NameNode should be enhance to read the http data stream and update its in memory name space instead of saving the streamed data on disk first. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3299) handle creation time also in journal service
handle creation time also in journal service Key: HDFS-3299 URL: https://issues.apache.org/jira/browse/HDFS-3299 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Hari Mankude -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3298) Add HdfsDataOutputStream
Add HdfsDataOutputStream Key: HDFS-3298 URL: https://issues.apache.org/jira/browse/HDFS-3298 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3297) Update free space in the DataBlockScanner rather than using du
Update free space in the DataBlockScanner rather than using du -- Key: HDFS-3297 URL: https://issues.apache.org/jira/browse/HDFS-3297 Project: Hadoop HDFS Issue Type: Improvement Components: data-node Affects Versions: 0.23.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor As the DataNode adds new blocks to a BlockPool, it keeps track of how much space that block pool consumes. This information gets sent to the NameNode so we can track statistics and so forth. Periodically, we check what's actually on the disk to make sure that the counts we are keeping are accurate. The DataNode currently kicks off a "du -s" process through the shell every few minutes and takes the result as the new used space number. We should do this in the DataBlockScanner, rather than using a separate du process. The main reason to do this is so that we don't cause a lot of random I/O operations on the disk. Since du has to visit every file in the BlockPool, it is essentially re-doing the work of the block scanner, for no reason. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3296) Running libhdfs tests in mac fails
Running libhdfs tests in mac fails
--
Key: HDFS-3296
URL: https://issues.apache.org/jira/browse/HDFS-3296
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Amareshwari Sriramadasu
Running "ant -Dcompile.c++=true -Dlibhdfs=true test-c++-libhdfs" on Mac fails
with following error:
{noformat}
[exec] dyld: lazy symbol binding failed: Symbol not found:
_JNI_GetCreatedJavaVMs
[exec] Referenced from:
/Users/amareshwari.sr/workspace/hadoop/build/c++/Mac_OS_X-x86_64-64/lib/libhdfs.0.dylib
[exec] Expected in: flat namespace
[exec]
[exec] dyld: Symbol not found: _JNI_GetCreatedJavaVMs
[exec] Referenced from:
/Users/amareshwari.sr/workspace/hadoop/build/c++/Mac_OS_X-x86_64-64/lib/libhdfs.0.dylib
[exec] Expected in: flat namespace
[exec]
[exec]
/Users/amareshwari.sr/workspace/hadoop/src/c++/libhdfs/tests/test-libhdfs.sh:
line 122: 39485 Trace/BPT trap: 5 CLASSPATH=$HADOOP_CONF_DIR:$CLASSPATH
LD_PRELOAD="$LIB_JVM_DIR/libjvm.so:$LIBHDFS_INSTALL_DIR/libhdfs.so:"
$LIBHDFS_BUILD_DIR/$HDFS_TEST
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3295) Mismatch in number of bytes already moved and number of bytes being moved in balancer report
Mismatch in number of bytes already moved and number of bytes being moved in
balancer report
Key: HDFS-3295
URL: https://issues.apache.org/jira/browse/HDFS-3295
Project: Hadoop HDFS
Issue Type: Bug
Components: balancer
Affects Versions: 0.23.0
Reporter: J.Andreina
Fix For: 0.24.0
Scenario:
Replication factor =
1,fs.defaultFS=hdfs://namenodeip:port,dfs.namenode.rpc-address=namenodeip:port.
step 1: started DN1.Pumped 4.67GB of data
step 2: started DN2.
step 3: issued the balancer cmd(./start-balancer.sh -threshold 1)
Totally 848.41 MB has been moved to 2DN and took 4 iterations to move the
blocks to 2DN.
But in balancer output in all the iterations the number of bytes already moved
is alway 0KB and there is mismatch in the Bytes being moved with the bytes
left to move
Balancer output for 2nd Iteration
=
{noformat}
Time Stamp Iteration# Bytes Already Moved Bytes Left To Move
Bytes Being Moved
Apr 18, 2012 12:41:20 PM 0 0 KB 848.41 MB
151.18 MB
Apr 18, 2012 1:06:46 PM 1 0 KB 646.02 MB
151.18 MB
Apr 18, 2012 1:10:28 PM 2 0 KB 417.43 MB
151.18 MB
Apr 18, 2012 1:14:04 PM 3 0 KB 223.84 MB
151.18 MB
Apr 18, 2012 1:18:10 PM 4 0 KB30.34 MB
151.18 MB{noformat}
In the above balancer output
•Bytes already moved is 0KB
• In 3 rd iterarion Bytes left to move is 223.84 MB but Bytes being moved is
151.18 MB . So in next iteration the bytes left to move should be 223.84 MB -
151.18 MB
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3294) Fix indentation in NamenodeWebHdfsMethods and DatanodeWebHdfsMethods
Fix indentation in NamenodeWebHdfsMethods and DatanodeWebHdfsMethods Key: HDFS-3294 URL: https://issues.apache.org/jira/browse/HDFS-3294 Project: Hadoop HDFS Issue Type: Improvement Components: data-node, name-node Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Priority: Trivial These two classes are not well indented. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3293) Implement equals for storageinfo and journainfo class.
Implement equals for storageinfo and journainfo class. --- Key: HDFS-3293 URL: https://issues.apache.org/jira/browse/HDFS-3293 Project: Hadoop HDFS Issue Type: Bug Components: name-node Reporter: Hari Mankude Priority: Minor Implement equals for storageinfo and journalinfo class. Also journalinfo class needs a toString() method. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3292) Remove the deprecated DistributedFileSystem.DiskStatus and the related methods
Remove the deprecated DistributedFileSystem.DiskStatus and the related methods -- Key: HDFS-3292 URL: https://issues.apache.org/jira/browse/HDFS-3292 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3291) add test that covers HttpFS working w/ a non-HDFS Hadoop filesystem
add test that covers HttpFS working w/ a non-HDFS Hadoop filesystem --- Key: HDFS-3291 URL: https://issues.apache.org/jira/browse/HDFS-3291 Project: Hadoop HDFS Issue Type: Test Reporter: Alejandro Abdelnur It would be good to have a test that uses httpfs with local file system to catch the introduction of HDFS-isms. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3290) Use a better local directory layout for the datanode
Use a better local directory layout for the datanode Key: HDFS-3290 URL: https://issues.apache.org/jira/browse/HDFS-3290 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 0.23.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor When the HDFS DataNode stores chunks in a local directory, it currently puts all of the chunk files into one big directory. As the number of files increases, this does not work well at all. Local filesystems are not optimized for the case where there are hundreds of thousands of files in the same directory. It also makes inspecting directories with standard UNIX tools difficult. Similar to the git version control system, HDFS should create a few different top level directories keyed off of a few bits in the chunk ID. Git uses 8 bits. This substantially cuts down on the number of chunk files in the same directory and gives increased performance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3289) Commonize token selectors with fallback behavior
Commonize token selectors with fallback behavior
Key: HDFS-3289
URL: https://issues.apache.org/jira/browse/HDFS-3289
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs client, security
Affects Versions: 0.24.0
Reporter: Daryn Sharp
Priority: Minor
{{WebHdfsDelegationTokenSelector}} and {{HftpDelegationTokenSelector}} are
essentially the same. They could be refactored to a new class in
org.apache.hadoop.hdfs.security.token.delegation (or an inner class in
DelegationTokenSelector) and then pass TOKEN_KIND in the constructor. The
selector should look for the given kind, and if not found, fallback to looking
for the hdfs/rpc token.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3288) Handle block pool ID in journal service
Handle block pool ID in journal service --- Key: HDFS-3288 URL: https://issues.apache.org/jira/browse/HDFS-3288 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Tsz Wo (Nicholas), SZE -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3287) Format command should be able to format the all types of shared dirs.
Format command should be able to format the all types of shared dirs. - Key: HDFS-3287 URL: https://issues.apache.org/jira/browse/HDFS-3287 Project: Hadoop HDFS Issue Type: Bug Reporter: Uma Maheswara Rao G This is the JIRA to support the formatting of shared dirs. for more details see HDFS-3275. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3286) When the threshold value for balancer is 0(zero) ,unexpected output is displayed
When the threshold value for balancer is 0(zero) ,unexpected output is displayed
Key: HDFS-3286
URL: https://issues.apache.org/jira/browse/HDFS-3286
Project: Hadoop HDFS
Issue Type: Bug
Components: balancer
Affects Versions: 0.23.0
Reporter: J.Andreina
Fix For: 0.24.0
Replication factor =1
Step 1: Start NN,DN1.write 4 GB of data
Step 2: Start DN2
Step 3: issue the balancer command(./hdfs balancer -threshold 0)
The threshold parameter is a fraction in the range of (0%, 100%) with a default
value of 10%
When the above scenario is executed the Source DN and Target DN is choosen and
the number of bytes to be moved from source to target DN is also calculated .
Then the balancer is exiting with the following message "No block can be moved.
Exiting..." which is not expected.
{noformat}
HOST-xx-xx-xx-xx:/home/Andreina/APril10/install/hadoop/namenode/bin # ./hdfs
balancer -threshold 0
12/04/16 16:22:07 INFO balancer.Balancer: Using a threshold of 0.0
12/04/16 16:22:07 INFO balancer.Balancer: namenodes =
[hdfs://HOST-xx-xx-xx-xx:9000]
12/04/16 16:22:07 INFO balancer.Balancer: p =
Balancer.Parameters[BalancingPolicy.Node, threshold=0.0]
Time Stamp Iteration# Bytes Already Moved Bytes Left To Move
Bytes Being Moved
12/04/16 16:22:10 INFO net.NetworkTopology: Adding a new node:
/default-rack/yy.yy.yy.yy:50176
12/04/16 16:22:10 INFO net.NetworkTopology: Adding a new node:
/default-rack/xx.xx.xx.xx:50010
12/04/16 16:22:10 INFO balancer.Balancer: 1 over-utilized:
[Source[xx.xx.xx.xx:50010, utilization=7.212458091389678]]
12/04/16 16:22:10 INFO balancer.Balancer: 1 underutilized:
[BalancerDatanode[yy.yy.yy.yy:50176, utilization=4.650670324367203E-5]]
12/04/16 16:22:10 INFO balancer.Balancer: Need to move 1.77 GB to make the
cluster balanced.
No block can be moved. Exiting...
Balancing took 5.142 seconds
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3285) Null pointer execption at ClientNamenodeProtocolTranslatorPB while running fetchdt
Null pointer execption at ClientNamenodeProtocolTranslatorPB while running
fetchdt
---
Key: HDFS-3285
URL: https://issues.apache.org/jira/browse/HDFS-3285
Project: Hadoop HDFS
Issue Type: Bug
Components: name-node
Affects Versions: 2.0.0
Reporter: Brahma Reddy Battula
Priority: Minor
Fix For: 2.0.0, 3.0.0
Scenario:
Run following command
./hdfs fetchdt http://**:50070 then I am getting following nullpointer
execption
{noformat}
Exception in thread "main" java.lang.NullPointerException
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getDelegationToken(ClientNamenodeProtocolTranslatorPB.java:771)
at
org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:650)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:766)
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher$1.run(DelegationTokenFetcher.java:191)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1205)
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.main(DelegationTokenFetcher.java:144)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3284) bootstrapStandby fails in secure cluster
bootstrapStandby fails in secure cluster Key: HDFS-3284 URL: https://issues.apache.org/jira/browse/HDFS-3284 Project: Hadoop HDFS Issue Type: Bug Components: ha, security Affects Versions: 2.0.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Minor HDFS-3247 improved bootstrapStandby to check if the other NN is in active state before trying to bootstrap. But, it forgot to set up the kerberos principals in the config before doing so. So, bootstrapStandby now fails with "Failed to specify server's Kerberos principal name" in a secure cluster. (Credit to Stephen Chu for finding this) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3283) create http server for JournalNode
create http server for JournalNode -- Key: HDFS-3283 URL: https://issues.apache.org/jira/browse/HDFS-3283 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Brandon Li Assignee: Brandon Li This http server can be used to view journal node status and transfer finalized edit log segments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3282) Expose getFileLength API.
Expose getFileLength API. - Key: HDFS-3282 URL: https://issues.apache.org/jira/browse/HDFS-3282 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs client Affects Versions: 3.0.0 Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G This JIRA is to expose the getFileLength API through a new public DistributedFileSystemInfo class. I would appreciate if someone suggest good name for this public class. Nicholas, did you plan any special design for this public client class? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3281) TestParallelRead times out with Clover enabled
TestParallelRead times out with Clover enabled -- Key: HDFS-3281 URL: https://issues.apache.org/jira/browse/HDFS-3281 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 3.0.0 Reporter: Henry Robinson Assignee: Henry Robinson Although it passes in a respectable time in pre-commit builds, when Clover is enabled TestParallelRead timesout (so runs > 15 minutes). I haven't been able to reproduce this on my development machines - the test runs in ~3 minutes and I don't think there's a factor of five difference in speed between this and the Jenkins box. I'm continuing to investigate the cause. If we can't find anything in the short term, we should temporarily disable this test to let the build go back to passing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3280) DFSOutputStream.sync should not be synchronized
DFSOutputStream.sync should not be synchronized
---
Key: HDFS-3280
URL: https://issues.apache.org/jira/browse/HDFS-3280
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs client
Affects Versions: 2.0.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Priority: Critical
HDFS-895 added an optimization to make hflush() much faster by unsynchronizing
it. But, we forgot to un-synchronize the deprecated {{sync()}} wrapper method.
This makes the HBase WAL really slow on 0.23+ since it doesn't take advantage
of HDFS-895 anymore.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3279) One of the FSEditLog constructor should be moved to TestEditLog
One of the FSEditLog constructor should be moved to TestEditLog --- Key: HDFS-3279 URL: https://issues.apache.org/jira/browse/HDFS-3279 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: Tsz Wo (Nicholas), SZE Assignee: Arpit Gupta Priority: Minor The FSEditLog constructor with @VisibleForTesting is used only in TestEditLog. It could be simply declared as a static method in TestEditLog. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3278) Umbrella Jira for HDFS-HA Phase 2
Umbrella Jira for HDFS-HA Phase 2 - Key: HDFS-3278 URL: https://issues.apache.org/jira/browse/HDFS-3278 Project: Hadoop HDFS Issue Type: New Feature Reporter: Sanjay Radia Assignee: Sanjay Radia HDFS-1623 gives a high level architecture and design for hot automatic failover of the NN. Branch HDFS-1623 was merged into trunk for tactical reasons even though the work for HA was not complete, Branch HDFS-1623 contained mechanisms for keeping a standby Hot (ie read from shared journal), dual block reports, fencing of DNs, Zookeeper library for leader election etc. This Umbrella jira covers the remaining work for HA and will link all the jiras for the remaining work. Unlike HDFS-1623 no single branch will be created - work will proceed in parallel branches. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3277) fail over to loading a different FSImage if the first one we try to load is corrupt
fail over to loading a different FSImage if the first one we try to load is corrupt --- Key: HDFS-3277 URL: https://issues.apache.org/jira/browse/HDFS-3277 Project: Hadoop HDFS Issue Type: Bug Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Most users store multiple copies of the FSImage in order to prevent catastrophic data loss if a hard disk fails. However, our image loading code is currently not set up to start reading another FSImage if loading the first one does not succeed. We should add this capability. We should also be sure to remove the FSImage directory that failed from the list of FSImage directories to write to, in the way we normally do when a write (as opopsed to read) fails. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3276) initializeSharedEdits should have a -nonInteractive flag
initializeSharedEdits should have a -nonInteractive flag Key: HDFS-3276 URL: https://issues.apache.org/jira/browse/HDFS-3276 Project: Hadoop HDFS Issue Type: Improvement Components: ha, name-node Affects Versions: 2.0.0 Reporter: Vinithra Varadharajan Priority: Minor Similar to format and bootstrapStandby, would be nice to have -nonInteractive as an option on initializeSharedEdits -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3275) Format command overwrites contents of non-empty shared edits dir if name dirs are empty without any prompting
Format command overwrites contents of non-empty shared edits dir if name dirs are empty without any prompting - Key: HDFS-3275 URL: https://issues.apache.org/jira/browse/HDFS-3275 Project: Hadoop HDFS Issue Type: Bug Components: ha, name-node Affects Versions: 2.0.0 Reporter: Vinithra Varadharajan To reproduce: 1) Configure a NameNode with namedirs and a shared edits dir, all of which are empty. 2) Run hdfs namenode -format. Namedirs and shared edits dir gets populated. 2) Delete the contents of the namedirs. Leave the shared edits dir as is. Check the timestamps of the shared edits dir contents. 3) Run format again. The namedirs as well as the shared edits dir get formatted. The shared edits dir's contents have been replaced without any prompting. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3274) Create a new journal_edits_dir key to support journal nodes
Create a new journal_edits_dir key to support journal nodes Key: HDFS-3274 URL: https://issues.apache.org/jira/browse/HDFS-3274 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Hari Mankude Assignee: Hari Mankude -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3273) Refactor BackupImage and FSEditLog
Refactor BackupImage and FSEditLog -- Key: HDFS-3273 URL: https://issues.apache.org/jira/browse/HDFS-3273 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3272) Make it possible to state MIME type for a webhdfs OPEN operation's result
Make it possible to state MIME type for a webhdfs OPEN operation's result
-
Key: HDFS-3272
URL: https://issues.apache.org/jira/browse/HDFS-3272
Project: Hadoop HDFS
Issue Type: Improvement
Affects Versions: 1.0.1
Reporter: Steve Loughran
Priority: Minor
when you do a GET from the browser with webhdfs, you get the file, but it comes
over as a binary as the browser doesn't know what type it is. Having a mime
mapping table and such like would be one solution, but another is simply to add
a {{mime}} query parameter that would provide a string to be reflected back to
the caller as the Content-Type header in the HTTP response.
e.g.
{code}
http://ranier:50070/webhdfs/v1/results/Debounce/part-r-0.csv?op=open&mime=text/csv
{code}
would generate a 307 redirect to the datanode, with the
{code}
http://dn1:50075/webhdfs/v1/results/Debounce/part-r-0.csv?op=open&mime=text/csv
{code}
which would then generate the result
{code}
200 OK
Content-Type:text/csv
GATE4,eb8bd736445f415e18886ba037f84829,55000,2007-01-14,14:01:54,
GATE4,ec58edcce1049fa665446dc1fa690638,8030803000,2007-01-14,13:52:31,
...
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3271) src/fuse_users.c: use re-entrant versions of getpwuid, getgid, etc
src/fuse_users.c: use re-entrant versions of getpwuid, getgid, etc -- Key: HDFS-3271 URL: https://issues.apache.org/jira/browse/HDFS-3271 Project: Hadoop HDFS Issue Type: Improvement Reporter: Colin Patrick McCabe Priority: Minor Use the re-entrant versions of these functions rather than using locking -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3270) run valgrind on fuse-dfs, fix any memory leaks
run valgrind on fuse-dfs, fix any memory leaks -- Key: HDFS-3270 URL: https://issues.apache.org/jira/browse/HDFS-3270 Project: Hadoop HDFS Issue Type: Improvement Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor run valgrind on fuse-dfs, fix any memory leaks -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3269) End-to-end test for making a non-HA HDFS cluster HA-enabled
End-to-end test for making a non-HA HDFS cluster HA-enabled --- Key: HDFS-3269 URL: https://issues.apache.org/jira/browse/HDFS-3269 Project: Hadoop HDFS Issue Type: Test Components: ha, name-node Affects Versions: 2.0.0 Reporter: Aaron T. Myers Per Eli on HDFS-3259, it would be great if we had a test that did the following: # Starts w/ non HA NN1 # Shutdown, enable HA on NN1, add SBN NN2 # Run initializeSharedEdits # Start and transition to active NN1 # Run bootstrapStandby # Confirm NN1 and NN2 are up and HA -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3268) Hdfs mishandles token service & incompatible with HA
Hdfs mishandles token service & incompatible with HA
Key: HDFS-3268
URL: https://issues.apache.org/jira/browse/HDFS-3268
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, hdfs client
Affects Versions: 0.24.0, 2.0.0
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Priority: Critical
The {{Hdfs AbstractFileSystem}} is overwriting the token service set by the
{{DFSClient}}. The service is not necessarily the correct one since
{{DFSClient}} is responsible for the service. Most importantly, this improper
behavior is overwriting the HA logical service which indirectly renders
{{FileContext}} incompatible with HA.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3267) TestBlocksWithNotEnoughRacks races with DN startup
TestBlocksWithNotEnoughRacks races with DN startup -- Key: HDFS-3267 URL: https://issues.apache.org/jira/browse/HDFS-3267 Project: Hadoop HDFS Issue Type: Bug Components: test Affects Versions: 2.0.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Minor In TestBlocksWithNotEnoughRacks.testCorruptBlockRereplicatedAcrossRacks, it restarts a DN, and then proceeds to call "waitCorruptReplicas". But, because of HDFS-3266, it doesn't actually wait very long while checking for the corrupt block to be reported. Since the DN starts back up asynchronously, the test will fail if it starts too slowly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3266) DFSTestUtil#waitCorruptReplicas doesn't sleep between checks
DFSTestUtil#waitCorruptReplicas doesn't sleep between checks Key: HDFS-3266 URL: https://issues.apache.org/jira/browse/HDFS-3266 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.0.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers DFSTestUtil#waitCorruptReplicas runs a loop waiting for an expected number of corrupt replicas. Unfortunately, it doesn't sleep between iterations of this loop, causing tests to occasionally fail spuriously. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3264) race in removing error edit stream could cause fatalExit()
race in removing error edit stream could cause fatalExit() -- Key: HDFS-3264 URL: https://issues.apache.org/jira/browse/HDFS-3264 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 1.1.0 Reporter: Brandon Li Assignee: Brandon Li An unhealthy storage directory and its edit stream can be removed from editStreams list. There are multiple places to remove unhealthy directory and its stream, such as log sync, checkpointing. Method removeEditsStreamsAndStorageDirs() could unneceissarily invoke fatalExit() if another thread removes one error editStream before current thread. This race could be between logSync requests, or between logSync and any place where a storage directory may be removed, e.g., during checkpointing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3263) HttpFS should read HDFS config from Hadoop site.xml files
HttpFS should read HDFS config from Hadoop site.xml files - Key: HDFS-3263 URL: https://issues.apache.org/jira/browse/HDFS-3263 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 2.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Fix For: 2.0.0 Currently HttpFS reads HDFS client configuration from the httfs-site.xml from any property of the form 'httpfs.hadoop.conf:HADOOP_PROPERTY' This is a bit inconvenient. Instead we should support a single property 'httpfs.hadoop.configuration.dir' that can be pointed to HADOOP conf/ dir and the core-site.xml and hdfs-site.xml files would be read from there. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3262) bootstrapStandby should copy edit logs in addition to images
bootstrapStandby should copy edit logs in addition to images Key: HDFS-3262 URL: https://issues.apache.org/jira/browse/HDFS-3262 Project: Hadoop HDFS Issue Type: Improvement Components: ha, name-node Affects Versions: 2.0.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Currently `hdfs namenode -bootstrapStandby' initializes local name dirs with fsimage files, but not edit log segments, thus requiring that an operator ensure somehow that sufficient edit log segments are in the shared dir or otherwise make it to the standby. It would be nice if this happened automatically. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3261) TestHASafeMode fails on HDFS-3042 branch
TestHASafeMode fails on HDFS-3042 branch Key: HDFS-3261 URL: https://issues.apache.org/jira/browse/HDFS-3261 Project: Hadoop HDFS Issue Type: Bug Components: test Affects Versions: Auto failover (HDFS-3042) Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Trivial TestHASafeMode started failing on the HDFS-3042 branch after the commit of HADOOP-8247. The reason is that testEnterSafeModeInANNShouldNotThrowNPE restarts the active node, and then tries to make an RPC to it right after restarting. The RPC picks up a cached connection to the old (restarted) NN, which causes an EOFException. This was just due to a test change that was made in HADOOP-8247, not due to any change made by the actual patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3260) TestDatanodeRegistration should set minimum DN version in addition to minimum NN version
TestDatanodeRegistration should set minimum DN version in addition to minimum NN version Key: HDFS-3260 URL: https://issues.apache.org/jira/browse/HDFS-3260 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.0.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers This test doesn't actually fail on trunk, but it does fail on branch-2 because of the static version number of branch-2. Regardless, this patch should be committed to both trunk and branch-2 so that if/when trunk changes version numbers things still work as expected. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3259) NameNode#initializeSharedEdits should populate shared edits dir with edit log segments
NameNode#initializeSharedEdits should populate shared edits dir with edit log segments -- Key: HDFS-3259 URL: https://issues.apache.org/jira/browse/HDFS-3259 Project: Hadoop HDFS Issue Type: Improvement Components: ha, name-node Affects Versions: 2.0.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Currently initializeSharedEdits formats the shared dir so that subsequent edit log segments will be written there. However, it would be nice to automatically populate this dir with edit log segments with transactions going back to the last fsimage. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3258) Test for HADOOP-8144 (pseudoSortByDistance in NetworkTopology for first rack local node)
Test for HADOOP-8144 (pseudoSortByDistance in NetworkTopology for first rack local node) Key: HDFS-3258 URL: https://issues.apache.org/jira/browse/HDFS-3258 Project: Hadoop HDFS Issue Type: Test Components: test Reporter: Eli Collins Assignee: Junping Du For updating TestNetworkTopology to cover HADOOP-8144. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3257) Fix synchronization issues with journal service
Fix synchronization issues with journal service --- Key: HDFS-3257 URL: https://issues.apache.org/jira/browse/HDFS-3257 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Hari Mankude Assignee: Hari Mankude -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3256) HDFS considers blocks under-replicated if topology script is configured with only 1 rack
HDFS considers blocks under-replicated if topology script is configured with only 1 rack Key: HDFS-3256 URL: https://issues.apache.org/jira/browse/HDFS-3256 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.0.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers HDFS treats the mere presence of a topology script being configured as of evidence that there are multiple racks. If there is in fact only a single rack, the NN will try to place the blocks on at least two racks, and thus blocks will be considered to be under-replicated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3255) HA DFS returns wrong token service
HA DFS returns wrong token service
--
Key: HDFS-3255
URL: https://issues.apache.org/jira/browse/HDFS-3255
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, hdfs client
Affects Versions: 2.0.0
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Priority: Critical
{{fs.getCanonicalService()}} must be equal to
{{fs.getDelegationToken(renewer).getService()}}. When HA is enabled, the DFS
token's service is a logical uri, but {{dfs.getCanonicalService()}} is only
returning the hostname of the logical uri.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3254) Branch-2 build broken due to wrong version number in fuse-dfs' pom.xml
Branch-2 build broken due to wrong version number in fuse-dfs' pom.xml -- Key: HDFS-3254 URL: https://issues.apache.org/jira/browse/HDFS-3254 Project: Hadoop HDFS Issue Type: Bug Components: contrib/fuse-dfs Reporter: Anupam Seth -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3253) Supporting unzip and untar in hadoop shell
Supporting unzip and untar in hadoop shell -- Key: HDFS-3253 URL: https://issues.apache.org/jira/browse/HDFS-3253 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 0.23.1 Reporter: madhukara phatak Priority: Minor As of now hadoop command shell doesnot support unzipping or untaring files in HDFS. But API wise FileUtil already supports this functionality. Work of this jira will be adding untar and unzip functionality to FSShell. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3252) Include fuse-dfs in the tarball
Include fuse-dfs in the tarball --- Key: HDFS-3252 URL: https://issues.apache.org/jira/browse/HDFS-3252 Project: Hadoop HDFS Issue Type: Improvement Components: build, contrib/fuse-dfs Reporter: Eli Collins The fuse-dfs binary needs to be included in the binary tarball. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3251) Mavenize the fuse-dfs build
Mavenize the fuse-dfs build Key: HDFS-3251 URL: https://issues.apache.org/jira/browse/HDFS-3251 Project: Hadoop HDFS Issue Type: Improvement Components: build, contrib/fuse-dfs Reporter: Eli Collins The fuse-dfs build still uses the old ant-based build, let's integrate it as part of the maven build. Looks like we need to introduce sub-directories under src/main/native as libhdfs is there (w/o it's own subdirectory). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3250) Get the fuse-dfs test running
Get the fuse-dfs test running - Key: HDFS-3250 URL: https://issues.apache.org/jira/browse/HDFS-3250 Project: Hadoop HDFS Issue Type: Improvement Components: contrib/fuse-dfs, test Reporter: Eli Collins Now that fuse-dfs is building again (HDFS-2696) let's get the test running. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3249) Use ToolRunner.confirmPrompt in NameNode
Use ToolRunner.confirmPrompt in NameNode Key: HDFS-3249 URL: https://issues.apache.org/jira/browse/HDFS-3249 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Affects Versions: 2.0.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Trivial One of the recent patches in common copied NameNode.confirmPrompt to be part of ToolRunner, so it's available as utility code. We should now use this utility function from within NameNode.java instead of having two copies. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3248) bootstrapstanby repeated twice in hdfs namenode usage message
bootstrapstanby repeated twice in hdfs namenode usage message
-
Key: HDFS-3248
URL: https://issues.apache.org/jira/browse/HDFS-3248
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
The HDFS usage message repeats "bootstrapStandby" twice.
{code}
Usage: java NameNode [-backup] | [-checkpoint] | [-format[-clusterid cid ]] |
[-upgrade] | [-rollback] | [-finalize] | [-importCheckpoint] |
[-bootstrapStandby] | [-initializeSharedEdits] | [-bootstrapStandby] |
[-recover [ -force ] ]
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3247) Improve bootstrapStandby behavior when original NN is not active
Improve bootstrapStandby behavior when original NN is not active Key: HDFS-3247 URL: https://issues.apache.org/jira/browse/HDFS-3247 Project: Hadoop HDFS Issue Type: Improvement Components: ha Affects Versions: 2.0.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Minor Currently, if you run bootstrapStandby while the first NN is in standby mode, it will spit out an ugly StandbyException with a trace. Instead, it should print an explanation that you should transition the first NN to active before bootstrapping. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3246) pRead equivalent for direct read path
pRead equivalent for direct read path - Key: HDFS-3246 URL: https://issues.apache.org/jira/browse/HDFS-3246 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 3.0.0 Reporter: Henry Robinson Assignee: Henry Robinson There is no pread equivalent in ByteBufferReadable. We should consider adding one. It would be relatively easy to implement for the distributed case (certainly compared to HDFS-2834), since DFSInputStream does most of the heavy lifting. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3245) Add metrics and web UI for cluster version summary
Add metrics and web UI for cluster version summary -- Key: HDFS-3245 URL: https://issues.apache.org/jira/browse/HDFS-3245 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Affects Versions: 2.0.0 Reporter: Todd Lipcon With the introduction of protocol compatibility, once HDFS-2983 is committed, we have the possibility that different nodes in a cluster are running different software versions. To aid operators, we should add the ability to summarize the status of versions in the cluster, so they can easily determine whether a rolling upgrade is in progress or if some nodes "missed" an upgrade (eg maybe they were out of service when the software was updated) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3244) Remove dead writable code from hdfs/protocol
Remove dead writable code from hdfs/protocol Key: HDFS-3244 URL: https://issues.apache.org/jira/browse/HDFS-3244 Project: Hadoop HDFS Issue Type: Improvement Reporter: Eli Collins Assignee: Eli Collins While doing HDFS-3238 I noticed that there's more dead writable code in hdfs/protocol. Let's remove it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3243) TestParallelRead timing out on jenkins
TestParallelRead timing out on jenkins -- Key: HDFS-3243 URL: https://issues.apache.org/jira/browse/HDFS-3243 Project: Hadoop HDFS Issue Type: Bug Components: hdfs client, test Reporter: Todd Lipcon Trunk builds have been failing recently due to a TestParallelRead timeout. It doesn't report in the Jenkins failure list because surefire handles timeouts really poorly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3242) hadoop using scoundary name node
hadoop using scoundary name node Key: HDFS-3242 URL: https://issues.apache.org/jira/browse/HDFS-3242 Project: Hadoop HDFS Issue Type: Bug Reporter: cldoltd I wanto using scoundary name node back up at master and one salve i config in conf/master master slave1 in slave1 i config in hdfs-site: dfs.http.address hmaster90:50070 But when i stop , and start hadop error is: 2012-04-09 19:06:57,971 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: java.lang.NullPointerException at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addChild(FSDirectory.java:1091) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addChild(FSDirectory.java:1103) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addNode(FSDirectory.java:1006) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedAddFile(FSDirectory.java:205) at org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.java:613) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:1013) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:826) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:362) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:97) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:386) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(FSNamesystem.java:360) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:276) at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:496) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1288) i remove file edit.new then runsucess but i data will be lost. How does use scoundaryname node in other computer thank. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3241) BackupNode can't start if its storage directory is not manually hacked
BackupNode can't start if its storage directory is not manually hacked
--
Key: HDFS-3241
URL: https://issues.apache.org/jira/browse/HDFS-3241
Project: Hadoop HDFS
Issue Type: Improvement
Components: name-node
Affects Versions: 3.0.0
Reporter: Brandon Li
Priority: Minor
BackupNode can't start if its storage dir is not formatted (because it extends
NameNode).
After I use "/bin/hadoop namenode -format" to format its storage directory, it
still can't start because the inconsistent namespace with primary namenode.
{noformat}
2012-04-09 18:33:16,464 ERROR namenode.NameNode (NameNode.java:main(958)) -
Exception in namenode join
java.io.IOException: Inconsistent namespace information:
NamespaceInfo has:
LV=-40;NS=1165721067;cTime=0;CID=CID-b8ced26a-4675-476d-b5fb-c9ad337be34a;BPID=BP-1658271424-10.10.10.191-1333658489893.
Storage has:
LV=-40;NS=403924869;cTime=0;CID=CID-57ce694a-172c-4da1-9cd6-d498615e4f1e;BPID=BP-4558626-10.10.10.48-1334021589986.
at
org.apache.hadoop.hdfs.server.protocol.NamespaceInfo.validateStorage(NamespaceInfo.java:109)
at
org.apache.hadoop.hdfs.server.namenode.BackupNode.registerWith(BackupNode.java:332)
at
org.apache.hadoop.hdfs.server.namenode.BackupNode.initialize(BackupNode.java:161)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:546)
at
org.apache.hadoop.hdfs.server.namenode.BackupNode.(BackupNode.java:86)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:893)
{noformat}
The backup node started after I copied the VERSION file from the primary
namenode storage directory to that of the backup node.
This manual hack can be avoided if the backup node can format its local
directory by requesting VERSION info from the primary namenode.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3240) Drop log level of "heartbeat: ..." in BPServiceActor to DEBUG
Drop log level of "heartbeat: ..." in BPServiceActor to DEBUG
-
Key: HDFS-3240
URL: https://issues.apache.org/jira/browse/HDFS-3240
Project: Hadoop HDFS
Issue Type: Improvement
Components: data-node
Affects Versions: 2.0.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Priority: Trivial
Attachments: hdfs-3240.txt
The following log message is at INFO level, but should be at DEBUG level:
{code}
LOG.info("heartbeat: " + this);
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3239) Hadoop-1.0.2 is taking the wrong class path during setup
Hadoop-1.0.2 is taking the wrong class path during setup - Key: HDFS-3239 URL: https://issues.apache.org/jira/browse/HDFS-3239 Project: Hadoop HDFS Issue Type: Bug Environment: I am using 32 bit ubuntu 11.10 Reporter: Srikanth Kommineni /usr/libexec/../bin/hadoop: line 321: /usr/lib/jvm/java-6-sun/bin/java: No such file or directory /usr/libexec/../bin/hadoop: line 387: /usr/lib/jvm/java-6-sun/bin/java: No such file or directory -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3238) ServerCommand and friends don't need to be writables
ServerCommand and friends don't need to be writables Key: HDFS-3238 URL: https://issues.apache.org/jira/browse/HDFS-3238 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 2.0.0 Reporter: Eli Collins Assignee: Eli Collins Attachments: hdfs-3238.txt We can remove writable infrastructure from the ServerCommand classes as they're not uses across clients and we're PB within the server side. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3237) DatanodeInfo should have a DatanodeID rather than extend it
DatanodeInfo should have a DatanodeID rather than extend it --- Key: HDFS-3237 URL: https://issues.apache.org/jira/browse/HDFS-3237 Project: Hadoop HDFS Issue Type: Improvement Reporter: Eli Collins Assignee: Eli Collins Priority: Minor DatanodeInfo currently extends DatanodeID, the code would be more clear if it had a DatanodeID member instead, as DatanodeInfo is private within the server side and DatanodeID is passed to clients. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3236) NameNode does not initialize generic conf keys when started with -initializeSharedEditsDir
NameNode does not initialize generic conf keys when started with -initializeSharedEditsDir -- Key: HDFS-3236 URL: https://issues.apache.org/jira/browse/HDFS-3236 Project: Hadoop HDFS Issue Type: Bug Components: ha, name-node Affects Versions: 2.0.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Priority: Minor Attachments: HDFS-3236.patch This means that configurations that scope the location of the name/edits/shared edits dirs by nameserice or namenode won't work with `hdfs namenode -initializeSharedEdits'. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3235) MiniDFSClusterManager doesn't correctly support -format option
MiniDFSClusterManager doesn't correctly support -format option -- Key: HDFS-3235 URL: https://issues.apache.org/jira/browse/HDFS-3235 Project: Hadoop HDFS Issue Type: Bug Reporter: Henry Robinson Assignee: Henry Robinson Priority: Minor Fix For: 3.0.0 MiniDFSClusterManager.java correctly honours -format for setting StartupOption.FORMAT, but does not set .format(true) on the MiniDFSClusterBuilder. This means the datanodes' data dirs will be formatted every time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3234) Accidentally left log message in GetConf after HDFS-3226
Accidentally left log message in GetConf after HDFS-3226 Key: HDFS-3234 URL: https://issues.apache.org/jira/browse/HDFS-3234 Project: Hadoop HDFS Issue Type: Bug Components: tools Affects Versions: 2.0.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Trivial I accidentally left a debug printout in. It doesn't cause a functionality regression, but it does cause noisy output on the command line: $ ./bin/hdfs getconf -confKey fs.defaultFS key: fs.defaultFS hdfs://nameserviceId1 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3233) Move IP to FQDN conversion from DatanodeJSPHelper to DatanodeID
Move IP to FQDN conversion from DatanodeJSPHelper to DatanodeID --- Key: HDFS-3233 URL: https://issues.apache.org/jira/browse/HDFS-3233 Project: Hadoop HDFS Issue Type: Improvement Reporter: Eli Collins Assignee: Eli Collins Priority: Minor In a handful of places DatanodeJSPHelper looks up the IP for a DN and then determines a FQDN for the IP. We should move this code to a single place, a new DatanodeID to return the FQDN for a DatanodeID. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3232) Cleanup DatanodeInfo vs DatanodeID handling in DN servlets
Cleanup DatanodeInfo vs DatanodeID handling in DN servlets
--
Key: HDFS-3232
URL: https://issues.apache.org/jira/browse/HDFS-3232
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Eli Collins
Assignee: Eli Collins
Priority: Minor
The DN servlets currently have code like the following:
{code}
final String hostname = host instanceof DatanodeInfo
? ((DatanodeInfo)host).getHostName() : host.getIpAddr();
{code}
I believe this outdated, that we now always get one or the other (at least when
not running the tests). Need to verify that. We should clean this code up as
well, eg always use the IP (which we'll lookup the FQDN for) since the hostname
isn't necessarily valid to put in a URL (the DN hostname isn't necesarily a
FQDN).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3231) NN Host2NodesMap should use hostnames
NN Host2NodesMap should use hostnames - Key: HDFS-3231 URL: https://issues.apache.org/jira/browse/HDFS-3231 Project: Hadoop HDFS Issue Type: Improvement Reporter: Eli Collins Assignee: Eli Collins The NN's Host2NodesMap maps "host names" to datanode descriptors. It actually uses IP addresses and should use hostnames instead, as hostnames are a better key (eg a Datanode has one hostname but may have multiple IPs). Per HDFS-3216 there's actually a bug in that it's sometimes accessed with IP:port instead of IP, so that jira should be fixed before this one. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3230) Cleanup DatanodeID creation in the tests
Cleanup DatanodeID creation in the tests Key: HDFS-3230 URL: https://issues.apache.org/jira/browse/HDFS-3230 Project: Hadoop HDFS Issue Type: Improvement Components: test Reporter: Eli Collins Assignee: Eli Collins Priority: Minor A lot of tests create dummy DatanodeIDs for testing, often use bogus values when creating the objects (eg hostname in the IP field), which they can get away with because the IDs aren't actually used. Let's add a test utility method for creating a DatanodeID for testing and use it throughout. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3229) add JournalProtocol RPCs to list finalized edit segments, and read edit segment file from JournalNode.
add JournalProtocol RPCs to list finalized edit segments, and read edit segment file from JournalNode. --- Key: HDFS-3229 URL: https://issues.apache.org/jira/browse/HDFS-3229 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, name-node Reporter: Brandon Li Assignee: Brandon Li -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3228) Use fadvise in local read path
Use fadvise in local read path -- Key: HDFS-3228 URL: https://issues.apache.org/jira/browse/HDFS-3228 Project: Hadoop HDFS Issue Type: Improvement Reporter: Henry Robinson Assignee: Henry Robinson Fix For: 3.0.0 The read path through BlockReaderLocal does not take advantage of readahead or drop-behind in the way that BlockSender does. We could arguably stand to gain even more from hinting about read patterns to the kernel here, so we should add the same mechanisms to BlockReaderLocal. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3227) Mavenise libhdfs tests
Mavenise libhdfs tests -- Key: HDFS-3227 URL: https://issues.apache.org/jira/browse/HDFS-3227 Project: Hadoop HDFS Issue Type: Improvement Components: libhdfs Reporter: Henry Robinson Fix For: 3.0.0 There's a TODO in hadoop-hdfs/pom.xml that refers to getting the native tests wired up. hdfs_test is built as part of the compilation step. The trick is to get HADOOP_HOME set up correctly for test-libhdfs.sh to run, and to fix test-libhdfs.sh so it can run from a tree that hasn't yet had mvn package run to establish an easy CLASSPATH. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3226) Allow GetConf tool to print arbitrary keys
Allow GetConf tool to print arbitrary keys -- Key: HDFS-3226 URL: https://issues.apache.org/jira/browse/HDFS-3226 Project: Hadoop HDFS Issue Type: Improvement Components: tools Affects Versions: 2.0.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Currently the "hdfs getconf" tool can only print out certain keys, like the list of NNs, etc. It would be handy to allow it to fetch an arbitrary configuration. For example, users may wish to write shell scripts that interact with their hadoop cluster, and it is useful to be able to fetch configs like the name of the superuser, or the state of whether HA is enabled. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3225) Revist upgrade snapshots, roll back, finalize to enable rolling upgrades
Revist upgrade snapshots, roll back, finalize to enable rolling upgrades Key: HDFS-3225 URL: https://issues.apache.org/jira/browse/HDFS-3225 Project: Hadoop HDFS Issue Type: New Feature Reporter: Sanjay Radia -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3224) Bug in check for DN re-registration with different storage ID
Bug in check for DN re-registration with different storage ID - Key: HDFS-3224 URL: https://issues.apache.org/jira/browse/HDFS-3224 Project: Hadoop HDFS Issue Type: Bug Reporter: Eli Collins Priority: Minor DatanodeManager#registerDatanode checks the host to node map using an IP:port key, however the map is keyed on IP, so this check will always fail. It's performing the check to determine if a DN with the same IP and storage ID has already registered, and if so to remove this DN from the map and indicate that eg it's no longer hosting these blocks. This bug has been here forever. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3223) Auto HA: add zkfc to hadoop-daemon.sh script
Auto HA: add zkfc to hadoop-daemon.sh script Key: HDFS-3223 URL: https://issues.apache.org/jira/browse/HDFS-3223 Project: Hadoop HDFS Issue Type: Improvement Components: auto-failover, scripts Affects Versions: Auto failover (HDFS-3042) Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Trivial In order to start the ZKFC, we need to add it to the list of daemons in this script. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3222) DFSInputStream#openInfo should not silently get the length as 0 when locations length is zero for last partial block.
DFSInputStream#openInfo should not silently get the length as 0 when locations
length is zero for last partial block.
-
Key: HDFS-3222
URL: https://issues.apache.org/jira/browse/HDFS-3222
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs client
Affects Versions: 1.0.3, 2.0.0, 3.0.0
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
I have seen one situation with Hbase cluster.
Scenario is as follows:
1)1.5 blocks has been written and synced.
2)Suddenly cluster has been restarted.
Reader opened the file and trying to get the length., By this time partial
block contained DNs are not reported to NN. So, locations for this partial
block would be 0. In this case, DFSInputStream assumes that, 1 block size as
final size.
But reader also assuming that, 1 block size is the final length and setting his
end marker. Finally reader ending up reading only partial data. Due to this,
HMaster could not replay the complete edits.
Actually this happend with 20 version. Looking at the code, same should present
in trunk as well.
{code}
int replicaNotFoundCount = locatedblock.getLocations().length;
for(DatanodeInfo datanode : locatedblock.getLocations()) {
..
..
// Namenode told us about these locations, but none know about the replica
// means that we hit the race between pipeline creation start and end.
// we require all 3 because some other exception could have happened
// on a DN that has it. we want to report that error
if (replicaNotFoundCount == 0) {
return 0;
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3221) Update docs for HDFS-3140 (multiple interfaces)
Update docs for HDFS-3140 (multiple interfaces) --- Key: HDFS-3221 URL: https://issues.apache.org/jira/browse/HDFS-3221 Project: Hadoop HDFS Issue Type: Sub-task Components: documentation Reporter: Eli Collins Assignee: Eli Collins Need to update the docs to cover: - How to configure mulithoming (binding to the wildcard, the default) - The new client and server configuration options -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3220) Improve some block recovery log messages
Improve some block recovery log messages Key: HDFS-3220 URL: https://issues.apache.org/jira/browse/HDFS-3220 Project: Hadoop HDFS Issue Type: Improvement Reporter: Eli Collins FsDatasetImpl has three cases that throw exceptions with the message "THIS IS NOT SUPPOSED TO HAPPEN". These could happen in real life (eg with a corrupt block file). Let's improve these messages to indicate what case we've actually hit instead of this message, which isn't very useful. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3219) Disambiguate "visible length" in the code and docs
Disambiguate "visible length" in the code and docs
--
Key: HDFS-3219
URL: https://issues.apache.org/jira/browse/HDFS-3219
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Eli Collins
Priority: Minor
HDFS-2288 there are two definition of visible length, or rather we're using the
same name for two things:
# The HDFS-265 design doc which defines it as property of the replica:
{quote}
visible length is the "number of bytes that have been acknowledged by the
downstream DataNodes". It is replica (not block) specific, meaning it can be
different for different replicas at a given time. In the document it is called
BA (bytes acknowledged), compared to BR (bytes received).
{quote}
# The definition in HDFS-814 and DFSClient#getVisibleLength which defines it as
a property of a file:
{quote}
The visible length is the length that *all* datanodes in the pipeline contain
at least such amount of data. Therefore, these data are visible to the readers.
According to this definition the visible length of a file is the floor of all
visible lengths of all the replicas of the last block. It's a static property
set on open, eg is not updated when a writer calls hflush. Also
DFSInputStream#readBlockLength returns the 1st visible length of a replica it
finds, so it seems possible (though unlikely) in a failure scenario it could
return a length that was longer than what all replicas had.
{quote}
This has caused confusion in a number of other jiras. We should update the
design doc, java doc, perhaps rename DFSClient#getVisibleLength etc to
disambiguate this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3218) The client should be able to use multiple remote DN interfaces for block transfer
The client should be able to use multiple remote DN interfaces for block transfer - Key: HDFS-3218 URL: https://issues.apache.org/jira/browse/HDFS-3218 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs client Reporter: Eli Collins Assignee: Eli Collins HDFS-3146 and HDFS-3216 expose multiple DN interfaces to the client. In order for clients, in aggregate, to use multiple DN interfaces clients should pick different interfaces when transferring blocks. Given that we cache client <-> DN connections the policy of picking a remote interface at random for each new connection seems best (vs round robin for example). In the future we could make the client congestion aware. We could also establish multiple connections between the client and DN and therefore use multiple interfaces for a single block transfer. Both of those are out of scope for this jira. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3217) ZKFC should restart NN when healthmonitor gets a SERVICE_NOT_RESPONDING exception
ZKFC should restart NN when healthmonitor gets a SERVICE_NOT_RESPONDING exception - Key: HDFS-3217 URL: https://issues.apache.org/jira/browse/HDFS-3217 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Hari Mankude -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3216) DatanodeID should support multiple IP addresses
DatanodeID should support multiple IP addresses --- Key: HDFS-3216 URL: https://issues.apache.org/jira/browse/HDFS-3216 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Eli Collins Assignee: Eli Collins The DatanodeID has a single field for the IP address, for HDFS-3146 we need to extend it to support multiple addresses. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3215) Block size is logging as zero Even blockrecevied command received by DN
Block size is logging as zero Even blockrecevied command received by DN
Key: HDFS-3215
URL: https://issues.apache.org/jira/browse/HDFS-3215
Project: Hadoop HDFS
Issue Type: Bug
Components: name-node
Affects Versions: 2.0.0
Reporter: Brahma Reddy Battula
Priority: Minor
Fix For: 2.0.0, 3.0.0
Scenario
==
Start NN and DN.
write file.
Block size is logging as zero Even blockrecevied command received by DN
*NN log*
2012-03-14 20:23:40,541 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
NameSystem.allocateBlock: /hadoop-create-user.sh._COPYING_.
BP-1166515020-10.18.40.24-1331736264353
blk_1264419582929433995_1002{blockUCState=UNDER_CONSTRUCTION,
primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[XXX:50010|RBW]]}
2012-03-14 20:24:26,357 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
addStoredBlock: blockMap updated: XXX:50010 is added to
blk_1264419582929433995_1002{blockUCState=UNDER_CONSTRUCTION,
primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[XXX:50010|RBW]]}
size 0
*DN log*
2012-03-14 20:24:17,519 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Receiving block
BP-1166515020-XXX-1331736264353:blk_1264419582929433995_1002 src:
/XXX:53141 dest: /XXX:50010
2012-03-14 20:24:26,517 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
/XXX:53141, dest: /XXX:50010, bytes: 512, op: HDFS_WRITE, cliID:
DFSClient_NONMAPREDUCE_1612873957_1, offset: 0, srvID:
DS-1639667928-XXX-50010-1331736284942, blockid:
BP-1166515020-XXX-1331736264353:blk_1264419582929433995_1002, duration:
1286482503
2012-03-14 20:24:26,517 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
PacketResponder:
BP-1166515020-XXX-1331736264353:blk_1264419582929433995_1002,
type=LAST_IN_PIPELINE, downstreams=0:[] terminating
2012-03-14 20:24:31,533 INFO
org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification
succeeded for BP-1166515020-XXX-1331736264353:blk_1264419582929433995_1002
[ Show » ] Brahma Reddy Battula added a comment - 16-Mar-12 10:13 AM Scenario
2: == Start NN and DN. write file. Block size is logging as zero Even
blockrecevied command received by DN NN log 2012-03-14 20:23:40,541 INFO
org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.allocateBlock:
/hadoop-create-user.sh._COPYING_. BP-1166515020-XXX-1331736264353
blk_1264419582929433995_1002{blockUCState=UNDER_CONSTRUCTION,
primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[XXX:50010|RBW]]}
2012-03-14 20:24:26,357 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
addStoredBlock: blockMap updated: XXX:50010 is added to
blk_1264419582929433995_1002{blockUCState=UNDER_CONSTRUCTION,
primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[XXX:50010|RBW]]}
size 0
DN log 2012-03-14 20:24:17,519 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
BP-1166515020-XXX-1331736264353:blk_1264419582929433995_1002 src:
/XXX:53141 dest: /XXX:50010
2012-03-14 20:24:26,517 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
/XXX:53141, dest: /XXX:50010, bytes: 512, op: HDFS_WRITE, cliID:
DFSClient_NONMAPREDUCE_1612873957_1, offset: 0, srvID:
DS-1639667928-XXX-50010-1331736284942, blockid:
BP-1166515020-XXX-1331736264353:blk_1264419582929433995_1002, duration:
1286482503
2012-03-14 20:24:26,517 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
PacketResponder:
BP-1166515020-XXX-1331736264353:blk_1264419582929433995_1002,
type=LAST_IN_PIPELINE, downstreams=0:[] terminating
2012-03-14 20:24:31,533 INFO
org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification
succeeded for BP-1166515020-XXX-1331736264353:blk_1264419582929433995_1002
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3214) InterDatanodeProtocolServerSideTranslatorPB doesn't handle null response from initReplicaRecovery
InterDatanodeProtocolServerSideTranslatorPB doesn't handle null response from initReplicaRecovery - Key: HDFS-3214 URL: https://issues.apache.org/jira/browse/HDFS-3214 Project: Hadoop HDFS Issue Type: Bug Components: data-node Affects Versions: 2.0.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Blocker The initReplicaRecovery function may return null to indicate that the block doesn't exist on the local node. However, the translator doesn't handle this case, which results in NPEs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3213) JournalDaemon (server) should persist the cluster id and nsid in the storage directory
JournalDaemon (server) should persist the cluster id and nsid in the storage directory -- Key: HDFS-3213 URL: https://issues.apache.org/jira/browse/HDFS-3213 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Hari Mankude -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3212) Persist the epoch received by the JournalService
Persist the epoch received by the JournalService Key: HDFS-3212 URL: https://issues.apache.org/jira/browse/HDFS-3212 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, name-node Affects Versions: Shared journals (HDFS-3092) Reporter: Suresh Srinivas epoch received over JournalProtocol should be persisted by JournalService. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3211) JournalProtocol changes required for introducing epoch and fencing
JournalProtocol changes required for introducing epoch and fencing -- Key: HDFS-3211 URL: https://issues.apache.org/jira/browse/HDFS-3211 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, name-node Reporter: Suresh Srinivas JournalProtocol changes to introduce epoch in every request. Adding new method fence for fencing a JournalService. On BackupNode fence is a no-op. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3210) JsonUtil#toJsonMap for for a DatanodeInfo should use "ipAddr" instead of "name"
JsonUtil#toJsonMap for for a DatanodeInfo should use "ipAddr" instead of "name" --- Key: HDFS-3210 URL: https://issues.apache.org/jira/browse/HDFS-3210 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.0.0 Reporter: Eli Collins Assignee: Eli Collins Attachments: hdfs-3210.txt In HDFS-3144 I missed a spot when renaming the "name" field. Let's fix that and add a test. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3209) dfs.namenode.hosts* configuration options are unused
dfs.namenode.hosts* configuration options are unused Key: HDFS-3209 URL: https://issues.apache.org/jira/browse/HDFS-3209 Project: Hadoop HDFS Issue Type: Improvement Reporter: Eli Collins Priority: Minor HDFS-631 introduced dfs.namenode.hosts and dfs.namenode.hosts.exclude but never actually used them, so they're dead code (dfs.hosts and dfs.hosts.excludes are used instead). IMO the current names are better (even though they're inconsistent) so I'd actually prefer we just remove the dead defines. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3208) Bogus entries in hosts files are incorrectly displayed in the report
Bogus entries in hosts files are incorrectly displayed in the report - Key: HDFS-3208 URL: https://issues.apache.org/jira/browse/HDFS-3208 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 2.0.0 Reporter: Eli Collins Assignee: Eli Collins DM#getDatanodeListForReport incorrectly creates the DatanodeID for the "dead" report for bogus entries in the host files (eg that an invalid hostname). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
