[jira] Commented: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12829152#action_12829152 ] Sanjay Radia commented on HDFS-245: --- Feedback on patch 31: TestFcHdfsSymlink * dfsClient is unused. * FileContext fc = FileContext.getFileContext(fs.getUri()); should be FileContext fc = FileContext.getFileContext(cluster.getUri()); The FileSystem interface should not be used all all in any of the FileContext tests. * Teardown move the deletes for base dir to teardown - easier to add new tests * testDiffFileContextRelTarget() why set wd? * testAccessLinkFromFileSystem()- this should really be FromAbsFileSystem (or FromAFS). * testSetPermission Also add a few additional lines to test set-permission on link to a dir * Add test for quota - setQuota should not follow symlink. Create symbolic links in HDFS - Key: HDFS-245 URL: https://issues.apache.org/jira/browse/HDFS-245 Project: Hadoop HDFS Issue Type: New Feature Reporter: dhruba borthakur Assignee: Eli Collins Attachments: 4044_20081030spi.java, designdocv1.txt, designdocv2.txt, designdocv3.txt, HADOOP-4044-strawman.patch, symlink-0.20.0.patch, symlink-25-hdfs.patch, symlink-26-hdfs.patch, symlink-26-hdfs.patch, symLink1.patch, symLink1.patch, symLink11.patch, symLink12.patch, symLink13.patch, symLink14.patch, symLink15.txt, symLink15.txt, symlink16-common.patch, symlink16-hdfs.patch, symlink16-mr.patch, symlink17-common.txt, symlink17-hdfs.txt, symlink18-common.txt, symlink19-common-delta.patch, symlink19-common.txt, symlink19-common.txt, symlink19-hdfs-delta.patch, symlink19-hdfs.txt, symlink20-common.patch, symlink20-hdfs.patch, symlink21-common.patch, symlink21-hdfs.patch, symlink22-common.patch, symlink22-hdfs.patch, symlink23-common.patch, symlink23-hdfs.patch, symlink24-hdfs.patch, symlink27-hdfs.patch, symlink28-hdfs.patch, symlink29-hdfs.patch, symlink29-hdfs.patch, symlink30-hdfs.patch, symlink31-hdfs.patch, symlink33-hdfs.patch, symlink35-hdfs.patch, symlink36-hdfs.patch, symLink4.patch, symLink5.patch, symLink6.patch, symLink8.patch, symLink9.patch HDFS should support symbolic links. A symbolic link is a special type of file that contains a reference to another file or directory in the form of an absolute or relative path and that affects pathname resolution. Programs which read or write to files named by a symbolic link will behave as if operating directly on the target file. However, archiving utilities can handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HDFS-945) Make NameNode resilient to DoS attacks (malicious or otherwise)
Make NameNode resilient to DoS attacks (malicious or otherwise) --- Key: HDFS-945 URL: https://issues.apache.org/jira/browse/HDFS-945 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: Arun C Murthy We've seen defective applications cause havoc on the NameNode, for e.g. by doing 100k+ 'listStatus' on very large directories (60k files) etc. I'd like to start a discussion around how we prevent such, and possibly malicious applications in the future, taking down the NameNode. Thoughts? -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-945) Make NameNode resilient to DoS attacks (malicious or otherwise)
[ https://issues.apache.org/jira/browse/HDFS-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12829185#action_12829185 ] Philip Zeyliger commented on HDFS-945: -- A solution is rate limiting (by client IP or user id or something else). It ain't fool-proof, but it would probably get the job done. Make NameNode resilient to DoS attacks (malicious or otherwise) --- Key: HDFS-945 URL: https://issues.apache.org/jira/browse/HDFS-945 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: Arun C Murthy We've seen defective applications cause havoc on the NameNode, for e.g. by doing 100k+ 'listStatus' on very large directories (60k files) etc. I'd like to start a discussion around how we prevent such, and possibly malicious applications in the future, taking down the NameNode. Thoughts? -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-945) Make NameNode resilient to DoS attacks (malicious or otherwise)
[ https://issues.apache.org/jira/browse/HDFS-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12829189#action_12829189 ] Todd Lipcon commented on HDFS-945: -- What's the scope of this? It seems there are a number of DoS scenarios to worry about: - RPC flooding (as you noted above) - Malformed packets (it's probably not too hard to find a spot where you can make the NN allocate way too much memory and crash some important thread) - Open socket limit exhaustion - what if a client just opened thousands of connections to the NN's RPC ports without actually sending commands? At some point you'll hit the ulimit -n - lots of others I imagine some of these are high priority and others less so. Focusing on non-malicious clients first is probably easiest. Although bugs can make non-malicious clients act like malicious ones for sure, I think your point is good that we should focus on well-meaning but stupid applications first :) Make NameNode resilient to DoS attacks (malicious or otherwise) --- Key: HDFS-945 URL: https://issues.apache.org/jira/browse/HDFS-945 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: Arun C Murthy We've seen defective applications cause havoc on the NameNode, for e.g. by doing 100k+ 'listStatus' on very large directories (60k files) etc. I'd like to start a discussion around how we prevent such, and possibly malicious applications in the future, taking down the NameNode. Thoughts? -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-945) Make NameNode resilient to DoS attacks (malicious or otherwise)
[ https://issues.apache.org/jira/browse/HDFS-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12829190#action_12829190 ] Zlatin Balevsky commented on HDFS-945: -- Any type of rate-limiting should be either optional or configurable on per-application basis. Make NameNode resilient to DoS attacks (malicious or otherwise) --- Key: HDFS-945 URL: https://issues.apache.org/jira/browse/HDFS-945 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: Arun C Murthy We've seen defective applications cause havoc on the NameNode, for e.g. by doing 100k+ 'listStatus' on very large directories (60k files) etc. I'd like to start a discussion around how we prevent such, and possibly malicious applications in the future, taking down the NameNode. Thoughts? -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-717) Proposal for exceptions thrown by FileContext and Abstract File System
[ https://issues.apache.org/jira/browse/HDFS-717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-717: - Attachment: hdfs-717.1.patch Proposal for exceptions thrown by FileContext and Abstract File System -- Key: HDFS-717 URL: https://issues.apache.org/jira/browse/HDFS-717 Project: Hadoop HDFS Issue Type: Improvement Reporter: Jitendra Nath Pandey Assignee: Suresh Srinivas Fix For: 0.22.0 Attachments: hdfs-717.1.patch, hdfs-717.patch, hdfs-717.patch Currently the APIs in FileContext throw only IOException. Going forward these APIs will throw more specific exceptions. This jira proposes following hierarchy of exceptions to be thrown by FileContext and AFS (Abstract File System) classes. InterruptedException (java.lang.InterruptedException) IOException /* Following exceptions extend IOException */ FileNotFoundException FileAlreadyExistsException DirectoryNotEmptyException NotDirectoryException AccessDeniedException IsDirectoryException InvalidPathNameException FileSystemException /* Following exceptions extend FileSystemException */ FileSystemNotReadyException ReadOnlyFileSystemException QuotaExceededException OutOfSpaceException RemoteException (java.rmi.RemoteException) Most of the IOExceptions above are caused by invalid user input, while FileSystemException is thrown when FS is in such a state that the requested operation cannot proceed. Please note that the proposed RemoteException is from standard java rmi package, which also extends IOException. HDFS throws many exceptions which are not in the above list. The DFSClient will unwrap the exceptions thrown by HDFS, and any exception not in the above list will be thrown as IOException or FileSystemException. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-945) Make NameNode resilient to DoS attacks (malicious or otherwise)
[ https://issues.apache.org/jira/browse/HDFS-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12829209#action_12829209 ] Kay Kay commented on HDFS-945: -- Echoing Zlatin's comment - this should be an optional feature , for those apps within trusted network that does not need to be paranoid about this. Make NameNode resilient to DoS attacks (malicious or otherwise) --- Key: HDFS-945 URL: https://issues.apache.org/jira/browse/HDFS-945 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: Arun C Murthy We've seen defective applications cause havoc on the NameNode, for e.g. by doing 100k+ 'listStatus' on very large directories (60k files) etc. I'd like to start a discussion around how we prevent such, and possibly malicious applications in the future, taking down the NameNode. Thoughts? -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-945) Make NameNode resilient to DoS attacks (malicious or otherwise)
[ https://issues.apache.org/jira/browse/HDFS-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12829211#action_12829211 ] Todd Lipcon commented on HDFS-945: -- bq. apps within trusted network that does not need to be paranoid about this I disagree. It's very easy to write a well-meaning app that does awful things. Certainly the limits should be configurable, but they should default on and be set at a high enough threshold that they only trigger in a case where it would make the NN fall over otherwise. Make NameNode resilient to DoS attacks (malicious or otherwise) --- Key: HDFS-945 URL: https://issues.apache.org/jira/browse/HDFS-945 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: Arun C Murthy We've seen defective applications cause havoc on the NameNode, for e.g. by doing 100k+ 'listStatus' on very large directories (60k files) etc. I'd like to start a discussion around how we prevent such, and possibly malicious applications in the future, taking down the NameNode. Thoughts? -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-945) Make NameNode resilient to DoS attacks (malicious or otherwise)
[ https://issues.apache.org/jira/browse/HDFS-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12829219#action_12829219 ] Eli Collins commented on HDFS-945: -- Let's keep this jira to mechanism, can discuss policy (eg defaults) later. If we really want to prevent DoS then HDFS needs some notion of QOS reservations (and even then you can just use more clients with different IPs etc). I suspect for now it's best to try to limit the impact of any one operation (eg lsr /), and then rate limit operations by client. Agree with Todd about focusing on the non-malicious use case first. Make NameNode resilient to DoS attacks (malicious or otherwise) --- Key: HDFS-945 URL: https://issues.apache.org/jira/browse/HDFS-945 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: Arun C Murthy We've seen defective applications cause havoc on the NameNode, for e.g. by doing 100k+ 'listStatus' on very large directories (60k files) etc. I'd like to start a discussion around how we prevent such, and possibly malicious applications in the future, taking down the NameNode. Thoughts? -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-877) Client-driven block verification not functioning
[ https://issues.apache.org/jira/browse/HDFS-877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12829233#action_12829233 ] stack commented on HDFS-877: I'm +1 on commit though non-critical. It has comprehensive test and it seems silly not picking up fixes (yeah, there is a freeze but its going on too long). Client-driven block verification not functioning Key: HDFS-877 URL: https://issues.apache.org/jira/browse/HDFS-877 Project: Hadoop HDFS Issue Type: Bug Components: hdfs client, test Affects Versions: 0.20.1, 0.21.0, 0.22.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: 0.22.0 Attachments: hdfs-877-branch20.txt, hdfs-877.txt, hdfs-877.txt, hdfs-877.txt, hdfs-877.txt, hdfs-877.txt, hdfs-877.txt This is actually the reason for HDFS-734 (TestDatanodeBlockScanner timing out). The issue is that DFSInputStream relies on readChunk being called one last time at the end of the file in order to receive the lastPacketInBlock=true packet from the DN. However, DFSInputStream.read checks pos getFileLength() before issuing the read. Thus gotEOS never shifts to true and checksumOk() is never called. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HDFS-946) NameNode should not return full path name when lisitng a diretory or getting the status of a file
NameNode should not return full path name when lisitng a diretory or getting
the status of a file
-
Key: HDFS-946
URL: https://issues.apache.org/jira/browse/HDFS-946
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Hairong Kuang
Fix For: 0.22.0
FSDirectory#getListring(String src) has the following code:
int i = 0;
for (INode cur : contents) {
listing[i] = createFileStatus(srcs+cur.getLocalName(), cur);
i++;
}
So listing a directory will return an array of FileStatus. Each FileStatus
element has the full path name. This increases the return message size and adds
non-negligible CPU time to the operation.
FSDirectory#getFileInfo(String) does not need to return the file name either.
Another optimization is that in the version of FileStatus that's used in the
wire protocol, the field path does not need to be Path; It could be a String or
a byte array ideally. This could avoid unnecessary creation of the Path objects
at NameNode, thus help reduce the GC problem observed when a large number of
getFileInfo or getListing operations hit NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-946) NameNode should not return full path name when lisitng a diretory or getting the status of a file
[
https://issues.apache.org/jira/browse/HDFS-946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12829250#action_12829250
]
Doug Cutting commented on HDFS-946:
---
This sounds reasonable. But the client would still return fully-qualified
paths, no?
NameNode should not return full path name when lisitng a diretory or getting
the status of a file
-
Key: HDFS-946
URL: https://issues.apache.org/jira/browse/HDFS-946
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Hairong Kuang
Fix For: 0.22.0
FSDirectory#getListring(String src) has the following code:
int i = 0;
for (INode cur : contents) {
listing[i] = createFileStatus(srcs+cur.getLocalName(), cur);
i++;
}
So listing a directory will return an array of FileStatus. Each FileStatus
element has the full path name. This increases the return message size and
adds non-negligible CPU time to the operation.
FSDirectory#getFileInfo(String) does not need to return the file name either.
Another optimization is that in the version of FileStatus that's used in the
wire protocol, the field path does not need to be Path; It could be a String
or a byte array ideally. This could avoid unnecessary creation of the Path
objects at NameNode, thus help reduce the GC problem observed when a large
number of getFileInfo or getListing operations hit NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-909) Race condition between rollEditLog or rollFSImage ant FSEditsLog.write operations corrupts edits log
[
https://issues.apache.org/jira/browse/HDFS-909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12829262#action_12829262
]
Todd Lipcon commented on HDFS-909:
--
Hi Konstantin. Thanks for the in depth exploration of that race.
Let me ask a question - sorry if I'm being dense here :)
In FSImage.saveFSImage(boolean) (FSImage.java:1243) we create a new current
edit log file before rolling the FSImage.
This seems to be related to the race, and I don't quite understand the design
here. It seems to me that the correct operation
of saveFSImage should be:
1. Dump the image in IMAGE_NEW
2. Create a new empty edit log in EDITS_NEW (while leaving EDITS alone)
3. Call purgeEditLogs which rolls the new image/edits pair in.
Analyzing the failure cases:
- During step 1, if the NN restarts, it can recover from the preexisting
IMAGE/EDITS as if the saveFSImage never started
- Between step 1 and 2, it can recover from either the preexisting IMAGE/EDITS
or the IMAGE_NEW.
IMAGE_NEW is consistent and up to date because saveFSImage is synchronized, and
thus it represents the exact state
of the namesystem.
- The same goes for step 2 - the empty edit log is correct against the new log.
If we decide to recover from current instead
of NEW at this point it would also be correct.
- We must assume that step 3 performs rolling in a way that allows recovery -
this is what allows rolling to work in general
safely.
In the implementation as I understand it, step 2 is implemented differently
such that it calls createEditLogFile on EDITS, not
just EDITS_NEW. This truncates the edits, which means we can only recover from
the NEW image at this point. This makes
for tricky recovery logic, since we have an old image with truncated edit log,
plus a new image (which we don't have any great way
of knowing is complete). Additionally, I don't think there's a guarantee that
steps 1 and 2 happen in that order - the order of
saveFSImage is determined by the order of the storage directories in the
directory iterator. It seems like edits _usually_
come last, but in the case of failed/recovered storage directories, I don't
think it's always true. If the order of step 1 and 2 are
inverted, it could truncate the current edits log before saving the new image,
and thus end up losing all those edits if the
NN failed in between.
Am I misunderstanding something? The simple fix to the issue you suggested
above is to add a waitForSyncToFinish in
createEditLogFile, but I'm worried there may be more subtle issues as described
above.
Thanks!
Race condition between rollEditLog or rollFSImage ant FSEditsLog.write
operations corrupts edits log
-
Key: HDFS-909
URL: https://issues.apache.org/jira/browse/HDFS-909
Project: Hadoop HDFS
Issue Type: Bug
Components: name-node
Affects Versions: 0.20.1, 0.20.2, 0.21.0, 0.22.0
Environment: CentOS
Reporter: Cosmin Lehene
Assignee: Todd Lipcon
Priority: Blocker
Fix For: 0.21.0, 0.22.0
Attachments: hdfs-909-unittest.txt, hdfs-909.txt, hdfs-909.txt,
hdfs-909.txt
closing the edits log file can race with write to edits log file operation
resulting in OP_INVALID end-of-file marker being initially overwritten by the
concurrent (in setReadyToFlush) threads and then removed twice from the
buffer, losing a good byte from edits log.
Example:
{code}
FSNameSystem.rollEditLog() - FSEditLog.divertFileStreams() -
FSEditLog.closeStream() - EditLogOutputStream.setReadyToFlush()
FSNameSystem.rollEditLog() - FSEditLog.divertFileStreams() -
FSEditLog.closeStream() - EditLogOutputStream.flush() -
EditLogFileOutputStream.flushAndSync()
OR
FSNameSystem.rollFSImage() - FSIMage.rollFSImage() -
FSEditLog.purgeEditLog() - FSEditLog.revertFileStreams() -
FSEditLog.closeStream() -EditLogOutputStream.setReadyToFlush()
FSNameSystem.rollFSImage() - FSIMage.rollFSImage() -
FSEditLog.purgeEditLog() - FSEditLog.revertFileStreams() -
FSEditLog.closeStream() -EditLogOutputStream.flush() -
EditLogFileOutputStream.flushAndSync()
VERSUS
FSNameSystem.completeFile - FSEditLog.logSync() -
EditLogOutputStream.setReadyToFlush()
FSNameSystem.completeFile - FSEditLog.logSync() -
EditLogOutputStream.flush() - EditLogFileOutputStream.flushAndSync()
OR
Any FSEditLog.write
{code}
Access on the edits flush operations is synchronized only in the
FSEdits.logSync() method level. However at a lower level access to
EditsLogOutputStream setReadyToFlush(), flush() or flushAndSync() is NOT
synchronized. These can be called from concurrent threads like in the example
above
So if a rollEditLog or rollFSIMage is happening at the same time with a write
operation it can race for
[jira] Commented: (HDFS-946) NameNode should not return full path name when lisitng a diretory or getting the status of a file
[
https://issues.apache.org/jira/browse/HDFS-946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12829269#action_12829269
]
Hairong Kuang commented on HDFS-946:
For this jira, the client will still return fully-qualified paths.
But I am thinking that even at the FileContext level it is not necessary to
return fully-qualified paths. However this is a user-facing incompatible
change. I would prefer to discuss it in a different jira.
NameNode should not return full path name when lisitng a diretory or getting
the status of a file
-
Key: HDFS-946
URL: https://issues.apache.org/jira/browse/HDFS-946
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Hairong Kuang
Fix For: 0.22.0
FSDirectory#getListring(String src) has the following code:
int i = 0;
for (INode cur : contents) {
listing[i] = createFileStatus(srcs+cur.getLocalName(), cur);
i++;
}
So listing a directory will return an array of FileStatus. Each FileStatus
element has the full path name. This increases the return message size and
adds non-negligible CPU time to the operation.
FSDirectory#getFileInfo(String) does not need to return the file name either.
Another optimization is that in the version of FileStatus that's used in the
wire protocol, the field path does not need to be Path; It could be a String
or a byte array ideally. This could avoid unnecessary creation of the Path
objects at NameNode, thus help reduce the GC problem observed when a large
number of getFileInfo or getListing operations hit NameNode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-938) Replace calls to UGI.getUserName() with UGI.getShortUserName()
[ https://issues.apache.org/jira/browse/HDFS-938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated HDFS-938: Attachment: HDFS-938-BP20-1.patch HDFS-938-BP20-1.patch: for previous version, not to commit Replace calls to UGI.getUserName() with UGI.getShortUserName() -- Key: HDFS-938 URL: https://issues.apache.org/jira/browse/HDFS-938 Project: Hadoop HDFS Issue Type: Bug Components: hdfs client, name-node Reporter: Jakob Homan Assignee: Jakob Homan Fix For: 0.22.0 Attachments: HDFS-938-BP20-1.patch, HDFS-938.patch HADOOP-6526 details why UGI.getUserName() will not work to identify users. Until the proposed UGI.getLocalName() is implemented, calls to getUserName() should be replaced with the short name. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads
[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12829293#action_12829293 ] ryan rawson commented on HDFS-918: -- I have done some thinking about HBase performance in relation to HDFS, and right now we are currently bottlenecking reads to a single file. By moving to a re-entrant API (pread) we are looking to unleash the parallelism. This is important I think, because we want to push as many parallel reads from our clients down into Datanode then down into the kernel to benefit from the IO scheduling in the kernel hardware. This could mean we might expect literally dozens of parallel reads per node on a busy cluster. Perhaps even hundreds! Per node. To ensure scalbility we'd probably want to get away from the xciever model, for more than 1 reason... If I remember correctly, xcivers not only consume threads (hundreds of threads is OK but non ideal) but it also consumes epolls, and there is just so many epolls available. So I heartily approve of the direction of this JIRA! Use single Selector and small thread pool to replace many instances of BlockSender for reads Key: HDFS-918 URL: https://issues.apache.org/jira/browse/HDFS-918 Project: Hadoop HDFS Issue Type: Improvement Components: data-node Reporter: Jay Booth Fix For: 0.22.0 Attachments: hdfs-918-20100201.patch, hdfs-918-20100203.patch, hdfs-multiplex.patch Currently, on read requests, the DataXCeiver server allocates a new thread per request, which must allocate its own buffers and leads to higher-than-optimal CPU and memory usage by the sending threads. If we had a single selector and a small threadpool to multiplex request packets, we could theoretically achieve higher performance while taking up fewer resources and leaving more CPU on datanodes available for mapred, hbase or whatever. This can be done without changing any wire protocols. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HDFS-947) The namenode should redirect a hftp request to read a file to the datanode that has the maximum number of local replicas
The namenode should redirect a hftp request to read a file to the datanode that has the maximum number of local replicas Key: HDFS-947 URL: https://issues.apache.org/jira/browse/HDFS-947 Project: Hadoop HDFS Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur A client that uses the Hftp protocol to read a file is redirected by the namenode to a random datanode. It would be nice if the client gets redirected to a datanode that has the maximum number of local replicas of the blocks of the file. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-947) The namenode should redirect a hftp request to read a file to the datanode that has the maximum number of local replicas
[ https://issues.apache.org/jira/browse/HDFS-947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated HDFS-947: -- Attachment: hftpRedirection.patch This is a patch fro hadoop 0.20. The namenode redirects a hftp read request to the datanode that has the max number of local replicas of the file. The namenode should redirect a hftp request to read a file to the datanode that has the maximum number of local replicas Key: HDFS-947 URL: https://issues.apache.org/jira/browse/HDFS-947 Project: Hadoop HDFS Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur Attachments: hftpRedirection.patch A client that uses the Hftp protocol to read a file is redirected by the namenode to a random datanode. It would be nice if the client gets redirected to a datanode that has the maximum number of local replicas of the blocks of the file. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HDFS-948) Allow setting permissions on links
Allow setting permissions on links
--
Key: HDFS-948
URL: https://issues.apache.org/jira/browse/HDFS-948
Project: Hadoop HDFS
Issue Type: New Feature
Components: hdfs client, name-node
Reporter: Eli Collins
Priority: Minor
In HDFS-245 symlink permissions are ignored, access to a symlink is determined
by the permissions of the symlink target. FileContext#setPermission and
setOwner work like {{chmod}} and {{chown}}, symlinks are fully resolved. This
jira is for adding FileContext#setLinkOwner and friends, ie a {{lchown}} (IEEE
1003.1) equivalent. The motivation is to allow an admin to create a link in a
directory that users can write but disallow them from deleting the link.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Created: (HDFS-949) Move Delegation token into Common so that we can use it for MapReduce also
Move Delegation token into Common so that we can use it for MapReduce also -- Key: HDFS-949 URL: https://issues.apache.org/jira/browse/HDFS-949 Project: Hadoop HDFS Issue Type: Improvement Reporter: Owen O'Malley Assignee: Owen O'Malley We need to support a MapReduce job that launches another MapReduce job inside its Mapper. Since the task doesn't have any Kerberos tickets, we need a delegation token. Moving the HDFS Delegation token to Common will allow both projects to use it. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-528) Add ability for safemode to wait for a minimum number of live datanodes
[ https://issues.apache.org/jira/browse/HDFS-528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-528: - Status: Patch Available (was: Open) somehow hudson missed this in the patch queue... retriggering. Add ability for safemode to wait for a minimum number of live datanodes --- Key: HDFS-528 URL: https://issues.apache.org/jira/browse/HDFS-528 Project: Hadoop HDFS Issue Type: New Feature Components: scripts Affects Versions: 0.22.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Attachments: hdfs-528-v2.txt, hdfs-528-v3.txt, hdfs-528.txt, hdfs-528.txt When starting up a fresh cluster programatically, users often want to wait until DFS is writable before continuing in a script. dfsadmin -safemode wait doesn't quite work for this on a completely fresh cluster, since when there are 0 blocks on the system, 100% of them are accounted for before any DNs have reported. This JIRA is to add a command which waits until a certain number of DNs have reported as alive to the NN. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-528) Add ability for safemode to wait for a minimum number of live datanodes
[ https://issues.apache.org/jira/browse/HDFS-528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-528: - Status: Open (was: Patch Available) Add ability for safemode to wait for a minimum number of live datanodes --- Key: HDFS-528 URL: https://issues.apache.org/jira/browse/HDFS-528 Project: Hadoop HDFS Issue Type: New Feature Components: scripts Affects Versions: 0.22.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Attachments: hdfs-528-v2.txt, hdfs-528-v3.txt, hdfs-528.txt, hdfs-528.txt When starting up a fresh cluster programatically, users often want to wait until DFS is writable before continuing in a script. dfsadmin -safemode wait doesn't quite work for this on a completely fresh cluster, since when there are 0 blocks on the system, 100% of them are accounted for before any DNs have reported. This JIRA is to add a command which waits until a certain number of DNs have reported as alive to the NN. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: symlink37-hdfs.patch Hey Sanjay, Thanks for posting, these are all addressed in patches 34-36, modulo using FileContextTestHelper which I'll defer to when HADOOP-5916 is resolved. Attached a patch that's a minor update to the last one to fix a bug in ImageLoaderCurrent. Thanks, Eli Create symbolic links in HDFS - Key: HDFS-245 URL: https://issues.apache.org/jira/browse/HDFS-245 Project: Hadoop HDFS Issue Type: New Feature Reporter: dhruba borthakur Assignee: Eli Collins Attachments: 4044_20081030spi.java, designdocv1.txt, designdocv2.txt, designdocv3.txt, HADOOP-4044-strawman.patch, symlink-0.20.0.patch, symlink-25-hdfs.patch, symlink-26-hdfs.patch, symlink-26-hdfs.patch, symLink1.patch, symLink1.patch, symLink11.patch, symLink12.patch, symLink13.patch, symLink14.patch, symLink15.txt, symLink15.txt, symlink16-common.patch, symlink16-hdfs.patch, symlink16-mr.patch, symlink17-common.txt, symlink17-hdfs.txt, symlink18-common.txt, symlink19-common-delta.patch, symlink19-common.txt, symlink19-common.txt, symlink19-hdfs-delta.patch, symlink19-hdfs.txt, symlink20-common.patch, symlink20-hdfs.patch, symlink21-common.patch, symlink21-hdfs.patch, symlink22-common.patch, symlink22-hdfs.patch, symlink23-common.patch, symlink23-hdfs.patch, symlink24-hdfs.patch, symlink27-hdfs.patch, symlink28-hdfs.patch, symlink29-hdfs.patch, symlink29-hdfs.patch, symlink30-hdfs.patch, symlink31-hdfs.patch, symlink33-hdfs.patch, symlink35-hdfs.patch, symlink36-hdfs.patch, symlink37-hdfs.patch, symLink4.patch, symLink5.patch, symLink6.patch, symLink8.patch, symLink9.patch HDFS should support symbolic links. A symbolic link is a special type of file that contains a reference to another file or directory in the form of an absolute or relative path and that affects pathname resolution. Programs which read or write to files named by a symbolic link will behave as if operating directly on the target file. However, archiving utilities can handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: design-doc-v4.txt Attached the latest design doc that captures the final path resolution and API changes, as well as incorporates feedback from Hairong. Create symbolic links in HDFS - Key: HDFS-245 URL: https://issues.apache.org/jira/browse/HDFS-245 Project: Hadoop HDFS Issue Type: New Feature Reporter: dhruba borthakur Assignee: Eli Collins Attachments: 4044_20081030spi.java, design-doc-v4.txt, designdocv1.txt, designdocv2.txt, designdocv3.txt, HADOOP-4044-strawman.patch, symlink-0.20.0.patch, symlink-25-hdfs.patch, symlink-26-hdfs.patch, symlink-26-hdfs.patch, symLink1.patch, symLink1.patch, symLink11.patch, symLink12.patch, symLink13.patch, symLink14.patch, symLink15.txt, symLink15.txt, symlink16-common.patch, symlink16-hdfs.patch, symlink16-mr.patch, symlink17-common.txt, symlink17-hdfs.txt, symlink18-common.txt, symlink19-common-delta.patch, symlink19-common.txt, symlink19-common.txt, symlink19-hdfs-delta.patch, symlink19-hdfs.txt, symlink20-common.patch, symlink20-hdfs.patch, symlink21-common.patch, symlink21-hdfs.patch, symlink22-common.patch, symlink22-hdfs.patch, symlink23-common.patch, symlink23-hdfs.patch, symlink24-hdfs.patch, symlink27-hdfs.patch, symlink28-hdfs.patch, symlink29-hdfs.patch, symlink29-hdfs.patch, symlink30-hdfs.patch, symlink31-hdfs.patch, symlink33-hdfs.patch, symlink35-hdfs.patch, symlink36-hdfs.patch, symlink37-hdfs.patch, symLink4.patch, symLink5.patch, symLink6.patch, symLink8.patch, symLink9.patch HDFS should support symbolic links. A symbolic link is a special type of file that contains a reference to another file or directory in the form of an absolute or relative path and that affects pathname resolution. Programs which read or write to files named by a symbolic link will behave as if operating directly on the target file. However, archiving utilities can handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads
[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12829364#action_12829364 ] dhruba borthakur commented on HDFS-918: --- because we want to push as many parallel reads from our clients down into Datanode Which part of HBase are you referring to here? I can understand that HBase can use pread to read the contents of a file but am not able to understand why the same file be read by multiple region servers at the same time. Use single Selector and small thread pool to replace many instances of BlockSender for reads Key: HDFS-918 URL: https://issues.apache.org/jira/browse/HDFS-918 Project: Hadoop HDFS Issue Type: Improvement Components: data-node Reporter: Jay Booth Fix For: 0.22.0 Attachments: hdfs-918-20100201.patch, hdfs-918-20100203.patch, hdfs-multiplex.patch Currently, on read requests, the DataXCeiver server allocates a new thread per request, which must allocate its own buffers and leads to higher-than-optimal CPU and memory usage by the sending threads. If we had a single selector and a small threadpool to multiplex request packets, we could theoretically achieve higher performance while taking up fewer resources and leaving more CPU on datanodes available for mapred, hbase or whatever. This can be done without changing any wire protocols. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-918) Use single Selector and small thread pool to replace many instances of BlockSender for reads
[ https://issues.apache.org/jira/browse/HDFS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12829368#action_12829368 ] ryan rawson commented on HDFS-918: -- the problem was we were using a stateful interface previously because it was faster in scan tests, so we serialized reads within 1 RS to any given HFile. With multiple client handler threads asking for different parts of a large file, we get a serialized behaviour which hurts random get performance. So we are moving back to pread, which means we will get more parallelism - depending your table read pattern of course. But I want to get even more parallelism, by preading multiple hfiles during a scan/get for example. This will just up the thread pressure on the datanode. Use single Selector and small thread pool to replace many instances of BlockSender for reads Key: HDFS-918 URL: https://issues.apache.org/jira/browse/HDFS-918 Project: Hadoop HDFS Issue Type: Improvement Components: data-node Reporter: Jay Booth Fix For: 0.22.0 Attachments: hdfs-918-20100201.patch, hdfs-918-20100203.patch, hdfs-multiplex.patch Currently, on read requests, the DataXCeiver server allocates a new thread per request, which must allocate its own buffers and leads to higher-than-optimal CPU and memory usage by the sending threads. If we had a single selector and a small threadpool to multiplex request packets, we could theoretically achieve higher performance while taking up fewer resources and leaving more CPU on datanodes available for mapred, hbase or whatever. This can be done without changing any wire protocols. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HDFS-950) Add concat to FsShell
Add concat to FsShell - Key: HDFS-950 URL: https://issues.apache.org/jira/browse/HDFS-950 Project: Hadoop HDFS Issue Type: New Feature Components: tools Reporter: Eli Collins Would be nice if concat (HDFS-222) was exposed up to FsShell so users don't have to use hadoop jar. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-945) Make NameNode resilient to DoS attacks (malicious or otherwise)
[ https://issues.apache.org/jira/browse/HDFS-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12829443#action_12829443 ] Sharad Agarwal commented on HDFS-945: - Some of the things like rate limiting is applicable to mapreduce as well, so I assume to have it in the RPC layer, though there may be use cases to have different limits for different operations. Make NameNode resilient to DoS attacks (malicious or otherwise) --- Key: HDFS-945 URL: https://issues.apache.org/jira/browse/HDFS-945 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: Arun C Murthy We've seen defective applications cause havoc on the NameNode, for e.g. by doing 100k+ 'listStatus' on very large directories (60k files) etc. I'd like to start a discussion around how we prevent such, and possibly malicious applications in the future, taking down the NameNode. Thoughts? -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-914) Refactor DFSOutputStream and DFSInputStream out of DFSClient
[ https://issues.apache.org/jira/browse/HDFS-914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-914: - Status: Open (was: Patch Available) Patch got out of date by HDFS-927 being committed. Refactor DFSOutputStream and DFSInputStream out of DFSClient Key: HDFS-914 URL: https://issues.apache.org/jira/browse/HDFS-914 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs client Affects Versions: 0.22.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Attachments: hdfs-914.txt, hdfs-914.txt, hdfs-914.txt I'd like to propose refactoring DFSClient to extract DFSOutputStream and DFSInputStream into a new org.apache.hadoop.hdfs.client package. DFSClient has become unmanageably large, containing 8 inner classes.and approaching 4kloc. Factoring out the non-static inner classes will also make them easier to test in isolation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-914) Refactor DFSOutputStream and DFSInputStream out of DFSClient
[ https://issues.apache.org/jira/browse/HDFS-914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-914: - Status: Patch Available (was: Open) Refactor DFSOutputStream and DFSInputStream out of DFSClient Key: HDFS-914 URL: https://issues.apache.org/jira/browse/HDFS-914 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs client Affects Versions: 0.22.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Attachments: hdfs-914.txt, hdfs-914.txt, hdfs-914.txt I'd like to propose refactoring DFSClient to extract DFSOutputStream and DFSInputStream into a new org.apache.hadoop.hdfs.client package. DFSClient has become unmanageably large, containing 8 inner classes.and approaching 4kloc. Factoring out the non-static inner classes will also make them easier to test in isolation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
