[jira] [Updated] (HDFS-1850) DN should transmit absolute failed volume count rather than increments to the NN
[ https://issues.apache.org/jira/browse/HDFS-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-1850: -- Status: Patch Available (was: Open) > DN should transmit absolute failed volume count rather than increments to the > NN > > > Key: HDFS-1850 > URL: https://issues.apache.org/jira/browse/HDFS-1850 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node, name-node >Reporter: Eli Collins >Assignee: Eli Collins > Fix For: 0.22.0 > > Attachments: hdfs-1850-1.patch, hdfs-1850-2.patch, hdfs-1850-3.patch, > hdfs-1850-4.patch, hdfs-1850-5.patch > > > The API added in HDFS-811 for the DN to report volume failures to the NN is > "inc(DN)". However the given sequence of events will result in the NN > forgetting about reported failed volumes: > # DN loses a volume and reports it > # NN restarts > # DN re-registers to the new NN > A more robust interface would be to have the DN report the total number of > volume failures to the NN each heart beat (the same way other volume state is > transmitted). > This will likely be an incompatible change since it requires changing the > Datanode protocol. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1850) DN should transmit absolute failed volume count rather than increments to the NN
[ https://issues.apache.org/jira/browse/HDFS-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-1850: -- Status: Open (was: Patch Available) > DN should transmit absolute failed volume count rather than increments to the > NN > > > Key: HDFS-1850 > URL: https://issues.apache.org/jira/browse/HDFS-1850 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node, name-node >Reporter: Eli Collins >Assignee: Eli Collins > Fix For: 0.22.0 > > Attachments: hdfs-1850-1.patch, hdfs-1850-2.patch, hdfs-1850-3.patch, > hdfs-1850-4.patch, hdfs-1850-5.patch > > > The API added in HDFS-811 for the DN to report volume failures to the NN is > "inc(DN)". However the given sequence of events will result in the NN > forgetting about reported failed volumes: > # DN loses a volume and reports it > # NN restarts > # DN re-registers to the new NN > A more robust interface would be to have the DN report the total number of > volume failures to the NN each heart beat (the same way other volume state is > transmitted). > This will likely be an incompatible change since it requires changing the > Datanode protocol. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1850) DN should transmit absolute failed volume count rather than increments to the NN
[ https://issues.apache.org/jira/browse/HDFS-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-1850: -- Attachment: hdfs-1850-5.patch Patch attached, rebased on trunk. > DN should transmit absolute failed volume count rather than increments to the > NN > > > Key: HDFS-1850 > URL: https://issues.apache.org/jira/browse/HDFS-1850 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node, name-node >Reporter: Eli Collins >Assignee: Eli Collins > Fix For: 0.22.0 > > Attachments: hdfs-1850-1.patch, hdfs-1850-2.patch, hdfs-1850-3.patch, > hdfs-1850-4.patch, hdfs-1850-5.patch > > > The API added in HDFS-811 for the DN to report volume failures to the NN is > "inc(DN)". However the given sequence of events will result in the NN > forgetting about reported failed volumes: > # DN loses a volume and reports it > # NN restarts > # DN re-registers to the new NN > A more robust interface would be to have the DN report the total number of > volume failures to the NN each heart beat (the same way other volume state is > transmitted). > This will likely be an incompatible change since it requires changing the > Datanode protocol. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1053) A client side mount table to give per-application/per-job file system view
[ https://issues.apache.org/jira/browse/HDFS-1053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13027276#comment-13027276 ] Konstantin Boudnik commented on HDFS-1053: -- Quickly skimming over the patch I have noticed that assertions in the tests do not have any messages which will result in Assert(null) error messages if tests fail. This makes failure analyze harder. Please have them added. > A client side mount table to give per-application/per-job file system view > -- > > Key: HDFS-1053 > URL: https://issues.apache.org/jira/browse/HDFS-1053 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.22.0 >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: ViewFs - javdoc2.pdf, ViewFsJavaDoc.pdf, viewFs2.patch, > viewfs1.patch > > > This jira proposes a client side mount table to allow application-centric (or > job-centric) filesystem views. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-1873) Federation Cluster Management Web Console
Federation Cluster Management Web Console - Key: HDFS-1873 URL: https://issues.apache.org/jira/browse/HDFS-1873 Project: Hadoop HDFS Issue Type: New Feature Affects Versions: 0.23.0 Reporter: Tanping Wang Assignee: Tanping Wang Fix For: 0.23.0 The Federation cluster management console provides # Cluster summary information that shows overall cluster utilization. A list of the name nodes that reports the used space, files and directories, blocks, live and dead datanodes of each name space. # decommissioning status of all the data nodes who are decommissioning in process or decommissioned. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1053) A client side mount table to give per-application/per-job file system view
[ https://issues.apache.org/jira/browse/HDFS-1053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sanjay Radia updated HDFS-1053: --- Attachment: (was: ViewFs - javdoc2.pdf) > A client side mount table to give per-application/per-job file system view > -- > > Key: HDFS-1053 > URL: https://issues.apache.org/jira/browse/HDFS-1053 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.22.0 >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: ViewFs - javdoc2.pdf, ViewFsJavaDoc.pdf, viewFs2.patch, > viewfs1.patch > > > This jira proposes a client side mount table to allow application-centric (or > job-centric) filesystem views. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1053) A client side mount table to give per-application/per-job file system view
[ https://issues.apache.org/jira/browse/HDFS-1053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sanjay Radia updated HDFS-1053: --- Attachment: ViewFs - javdoc2.pdf > A client side mount table to give per-application/per-job file system view > -- > > Key: HDFS-1053 > URL: https://issues.apache.org/jira/browse/HDFS-1053 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.22.0 >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: ViewFs - javdoc2.pdf, ViewFsJavaDoc.pdf, viewFs2.patch, > viewfs1.patch > > > This jira proposes a client side mount table to allow application-centric (or > job-centric) filesystem views. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1799) Refactor log rolling and filename management out of FSEditLog
[ https://issues.apache.org/jira/browse/HDFS-1799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13027251#comment-13027251 ] Jitendra Nath Pandey commented on HDFS-1799: Which of the patches contains changes related to LogSegment? I looked at the latest one uploaded one 04/29, but didn't see LogSegment. What am I missing? > Refactor log rolling and filename management out of FSEditLog > - > > Key: HDFS-1799 > URL: https://issues.apache.org/jira/browse/HDFS-1799 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: Edit log branch (HDFS-1073) >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: Edit log branch (HDFS-1073) > > Attachments: 0001-Added-state-management-to-FSEditLog.patch, > 0002-Standardised-error-pattern.patch, > 0003-Add-JournalFactory-and-move-divert-revert-out-of-FSE.patch, > HDFS-1799-all.diff, hdfs-1799.txt, hdfs-1799.txt, hdfs-1799.txt, hdfs-1799.txt > > > This is somewhat similar to HDFS-1580, but less ambitious. While that JIRA > focuses on pluggability, this task is simply the minimum needed for HDFS-1073: > - Refactor the filename-specific code for rolling, diverting, and reverting > log streams out of FSEditLog into a new class > - Clean up the related code in FSEditLog a bit > Notably, this JIRA is going to temporarily break the BackupNode. I plan to > circle back on the BackupNode later on this branch. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1053) A client side mount table to give per-application/per-job file system view
[ https://issues.apache.org/jira/browse/HDFS-1053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13027248#comment-13027248 ] Sanjay Radia commented on HDFS-1053: The patch provides an implementation for viewfs ie. client-side mount tables. They are implemented as Hadoop file systems and hence can be transparently used where ever a Hadoop file systems is used. (e.g. all the commandline tools work with it). The patch has an implementation for both FileSystem and AbstractFileSystem called ViewFileSystem and ViewFs I have attach the pdf of the javadoc for viewfs which explains how to use this feature. I will later post a user guide document. The patch contains a large number of additional tests and improves some of the existing file system tests. It also fixes a bug in AbstractFileSystem in how it deals with URIs. > A client side mount table to give per-application/per-job file system view > -- > > Key: HDFS-1053 > URL: https://issues.apache.org/jira/browse/HDFS-1053 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.22.0 >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: ViewFs - javdoc2.pdf, ViewFsJavaDoc.pdf, viewFs2.patch, > viewfs1.patch > > > This jira proposes a client side mount table to allow application-centric (or > job-centric) filesystem views. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1053) A client side mount table to give per-application/per-job file system view
[ https://issues.apache.org/jira/browse/HDFS-1053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sanjay Radia updated HDFS-1053: --- Attachment: ViewFs - javdoc2.pdf > A client side mount table to give per-application/per-job file system view > -- > > Key: HDFS-1053 > URL: https://issues.apache.org/jira/browse/HDFS-1053 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.22.0 >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: ViewFs - javdoc2.pdf, ViewFsJavaDoc.pdf, viewFs2.patch, > viewfs1.patch > > > This jira proposes a client side mount table to allow application-centric (or > job-centric) filesystem views. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1053) A client side mount table to give per-application/per-job file system view
[ https://issues.apache.org/jira/browse/HDFS-1053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sanjay Radia updated HDFS-1053: --- Attachment: viewFs2.patch > A client side mount table to give per-application/per-job file system view > -- > > Key: HDFS-1053 > URL: https://issues.apache.org/jira/browse/HDFS-1053 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.22.0 >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: ViewFsJavaDoc.pdf, viewFs2.patch, viewfs1.patch > > > This jira proposes a client side mount table to allow application-centric (or > job-centric) filesystem views. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1053) A client side mount table to give per-application/per-job file system view
[ https://issues.apache.org/jira/browse/HDFS-1053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sanjay Radia updated HDFS-1053: --- Attachment: viewFs1.patch > A client side mount table to give per-application/per-job file system view > -- > > Key: HDFS-1053 > URL: https://issues.apache.org/jira/browse/HDFS-1053 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.22.0 >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: ViewFsJavaDoc.pdf, viewfs1.patch > > > This jira proposes a client side mount table to allow application-centric (or > job-centric) filesystem views. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1053) A client side mount table to give per-application/per-job file system view
[ https://issues.apache.org/jira/browse/HDFS-1053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sanjay Radia updated HDFS-1053: --- Attachment: (was: viewFs1.patch) > A client side mount table to give per-application/per-job file system view > -- > > Key: HDFS-1053 > URL: https://issues.apache.org/jira/browse/HDFS-1053 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.22.0 >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: ViewFsJavaDoc.pdf, viewfs1.patch > > > This jira proposes a client side mount table to allow application-centric (or > job-centric) filesystem views. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-1872) BPOfferService.cleanUp(..) throws NullPointerException
BPOfferService.cleanUp(..) throws NullPointerException
--
Key: HDFS-1872
URL: https://issues.apache.org/jira/browse/HDFS-1872
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Tsz Wo (Nicholas), SZE
{noformat}
NullPointerException
at
org.apache.hadoop.hdfs.server.datanode.DataNode$BPOfferService.cleanUp(DataNode.java:1005)
at
org.apache.hadoop.hdfs.server.datanode.DataNode$BPOfferService.run(DataNode.java:1220)
at java.lang.Thread.run(Thread.java:662)
{noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-1871) Tests using MiniDFSCluster fail to compile due to HDFS-1052 changes
[ https://issues.apache.org/jira/browse/HDFS-1871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HDFS-1871. --- Resolution: Fixed Hadoop Flags: [Reviewed] I committed the patch. > Tests using MiniDFSCluster fail to compile due to HDFS-1052 changes > --- > > Key: HDFS-1871 > URL: https://issues.apache.org/jira/browse/HDFS-1871 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 0.23.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.23.0 > > Attachments: HDFS-1871.patch > > > MiniDFSCluster public method signature changes from HDFS-1052 breaks build of > mapreduce tests. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1871) Tests using MiniDFSCluster fail to compile due to HDFS-1052 changes
[ https://issues.apache.org/jira/browse/HDFS-1871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13027213#comment-13027213 ] Jitendra Nath Pandey commented on HDFS-1871: +1 for the patch. > Tests using MiniDFSCluster fail to compile due to HDFS-1052 changes > --- > > Key: HDFS-1871 > URL: https://issues.apache.org/jira/browse/HDFS-1871 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 0.23.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.23.0 > > Attachments: HDFS-1871.patch > > > MiniDFSCluster public method signature changes from HDFS-1052 breaks build of > mapreduce tests. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1871) Tests using MiniDFSCluster fail to compile due to HDFS-1052 changes
[ https://issues.apache.org/jira/browse/HDFS-1871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-1871: -- Attachment: HDFS-1871.patch Attached patch fixes the build error. > Tests using MiniDFSCluster fail to compile due to HDFS-1052 changes > --- > > Key: HDFS-1871 > URL: https://issues.apache.org/jira/browse/HDFS-1871 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 0.23.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.23.0 > > Attachments: HDFS-1871.patch > > > MiniDFSCluster public method signature changes from HDFS-1052 breaks build of > mapreduce tests. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-1871) Tests using MiniDFSCluster fail to compile due to HDFS-1052 changes
Tests using MiniDFSCluster fail to compile due to HDFS-1052 changes --- Key: HDFS-1871 URL: https://issues.apache.org/jira/browse/HDFS-1871 Project: Hadoop HDFS Issue Type: Bug Components: test Affects Versions: 0.23.0 Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: 0.23.0 MiniDFSCluster public method signature changes from HDFS-1052 breaks build of mapreduce tests. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1870) Refactor DFSClient.LeaseChecker
[
https://issues.apache.org/jira/browse/HDFS-1870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE updated HDFS-1870:
-
Status: Patch Available (was: Open)
> Refactor DFSClient.LeaseChecker
> ---
>
> Key: HDFS-1870
> URL: https://issues.apache.org/jira/browse/HDFS-1870
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs client
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
>Priority: Minor
> Attachments: h1870_20110129.patch
>
>
> Two simply changes:
> - move the inner class {{LeaseChecker}} from {{DFSClient}} to a separated
> class;
> - rename {{LeaseChecker}} to {{LeaseRenewer}} since it is actually renewing
> lease instead of checking lease.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1869) mkdirs should use the supplied permission for all of the created directories
[ https://issues.apache.org/jira/browse/HDFS-1869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated HDFS-1869: -- Status: Patch Available (was: Open) > mkdirs should use the supplied permission for all of the created directories > > > Key: HDFS-1869 > URL: https://issues.apache.org/jira/browse/HDFS-1869 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.23.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: HDFS-1869.patch > > > Mkdirs only uses the supplied FsPermission for the last directory of the > path. Paths 0..N-1 will all inherit the parent dir's permissions -even if- > inheritPermission is false. This is a regression from somewhere around > 0.20.9 and does not follow posix semantics. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1870) Refactor DFSClient.LeaseChecker
[
https://issues.apache.org/jira/browse/HDFS-1870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE updated HDFS-1870:
-
Attachment: h1870_20110129.patch
h1870_20110129.patch: see description.
> Refactor DFSClient.LeaseChecker
> ---
>
> Key: HDFS-1870
> URL: https://issues.apache.org/jira/browse/HDFS-1870
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs client
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
>Priority: Minor
> Attachments: h1870_20110129.patch
>
>
> Two simply changes:
> - move the inner class {{LeaseChecker}} from {{DFSClient}} to a separated
> class;
> - rename {{LeaseChecker}} to {{LeaseRenewer}} since it is actually renewing
> lease instead of checking lease.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1869) mkdirs should use the supplied permission for all of the created directories
[ https://issues.apache.org/jira/browse/HDFS-1869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated HDFS-1869: -- Attachment: HDFS-1869.patch Mkdirs will now honor the inheritPermission flag correctly. If false, the user's requested perms are used. Note that files that have auto-generated directories still implicitly inherit permissions. I find the entire concept of inheriting permissions to be dubious, so I'd like guidance on whether I should fix the file issue on this jira or a new jira? > mkdirs should use the supplied permission for all of the created directories > > > Key: HDFS-1869 > URL: https://issues.apache.org/jira/browse/HDFS-1869 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.23.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: HDFS-1869.patch > > > Mkdirs only uses the supplied FsPermission for the last directory of the > path. Paths 0..N-1 will all inherit the parent dir's permissions -even if- > inheritPermission is false. This is a regression from somewhere around > 0.20.9 and does not follow posix semantics. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-1870) Refactor DFSClient.LeaseChecker
Refactor DFSClient.LeaseChecker
---
Key: HDFS-1870
URL: https://issues.apache.org/jira/browse/HDFS-1870
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs client
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE
Priority: Minor
Two simply changes:
- move the inner class {{LeaseChecker}} from {{DFSClient}} to a separated class;
- rename {{LeaseChecker}} to {{LeaseRenewer}} since it is actually renewing
lease instead of checking lease.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-1869) mkdirs should use the supplied permission for all of the created directories
mkdirs should use the supplied permission for all of the created directories Key: HDFS-1869 URL: https://issues.apache.org/jira/browse/HDFS-1869 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.23.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Mkdirs only uses the supplied FsPermission for the last directory of the path. Paths 0..N-1 will all inherit the parent dir's permissions -even if- inheritPermission is false. This is a regression from somewhere around 0.20.9 and does not follow posix semantics. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1799) Refactor log rolling and filename management out of FSEditLog
[
https://issues.apache.org/jira/browse/HDFS-1799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13027159#comment-13027159
]
Todd Lipcon commented on HDFS-1799:
---
bq. JournalManager currently deals only with output
bq. Lastly, this patch doesn't hide the fact that files are in use here
These two are explicitly out of scope for this JIRA (note in the description
that I call out that this is supposed to be less ambitious than 1580). This is
meant to be a start but not an end to the refactoring of these interfaces.
bq. Regarding LogSegment, I'm still on the fence about this. I see some of the
merits in it, and I can see why JournalMananger is possibly needed for it. If
you retain the list of EditLogOutputStreams as it is now, you'll have to
maintain a separate list of some other objects for the creation of new log
segments. This could be problematic as these lists need to be kept in sync for
error handling etc
Exactly what I was thinking - thanks for expressing this better than I could.
The issue is that, if an EditLogOutputStream gets an IOException, I don't want
to keep using that object. JournalManager allows us to abort that stream and
get a new one next time we start a new segment.
How would you feel about the following API as a goal:
{noformat}
JournalManager:
void setBufferCapacity(int size);
EditLogOutputStream startLogSegment(long txId);
EditLogOutputStream:
void close(); // closes and "finalizes"
void abort(); // closes but marks as possibly truncated (eg after an IOE)
... current calls related to write/flush/sync ...
{noformat}
Then, in FSEditLog, whenever we start a new segment, we'd call startLogSegment
on all JournalManagers and insert the ELOS instances into a list. Journal
Managers never get marked as "failed" -- we only remove the failed ELOS after
aborting it. Then on the next segment we always give each JM a new change to
start a segment.
> Refactor log rolling and filename management out of FSEditLog
> -
>
> Key: HDFS-1799
> URL: https://issues.apache.org/jira/browse/HDFS-1799
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: Edit log branch (HDFS-1073)
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
> Fix For: Edit log branch (HDFS-1073)
>
> Attachments: 0001-Added-state-management-to-FSEditLog.patch,
> 0002-Standardised-error-pattern.patch,
> 0003-Add-JournalFactory-and-move-divert-revert-out-of-FSE.patch,
> HDFS-1799-all.diff, hdfs-1799.txt, hdfs-1799.txt, hdfs-1799.txt, hdfs-1799.txt
>
>
> This is somewhat similar to HDFS-1580, but less ambitious. While that JIRA
> focuses on pluggability, this task is simply the minimum needed for HDFS-1073:
> - Refactor the filename-specific code for rolling, diverting, and reverting
> log streams out of FSEditLog into a new class
> - Clean up the related code in FSEditLog a bit
> Notably, this JIRA is going to temporarily break the BackupNode. I plan to
> circle back on the BackupNode later on this branch.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1052) HDFS scalability with multiple namenodes
[ https://issues.apache.org/jira/browse/HDFS-1052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13027130#comment-13027130 ] Konstantin Boudnik commented on HDFS-1052: -- It'd be nice to have the test plan attached to the JIRA if any. Thanks. > HDFS scalability with multiple namenodes > > > Key: HDFS-1052 > URL: https://issues.apache.org/jira/browse/HDFS-1052 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Affects Versions: 0.22.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.23.0 > > Attachments: Block pool proposal.pdf, HDFS-1052.3.patch, > HDFS-1052.4.patch, HDFS-1052.5.patch, HDFS-1052.6.patch, HDFS-1052.patch, > Mulitple Namespaces5.pdf, high-level-design.pdf > > > HDFS currently uses a single namenode that limits scalability of the cluster. > This jira proposes an architecture to scale the nameservice horizontally > using multiple namenodes. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1052) HDFS scalability with multiple namenodes
[ https://issues.apache.org/jira/browse/HDFS-1052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-1052: -- Resolution: Fixed Fix Version/s: 0.23.0 Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I merged the changes from federation branch HDFS-1052 into trunk. Thanks every one for participating in the discussions and help move this issue forward. > HDFS scalability with multiple namenodes > > > Key: HDFS-1052 > URL: https://issues.apache.org/jira/browse/HDFS-1052 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Affects Versions: 0.22.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.23.0 > > Attachments: Block pool proposal.pdf, HDFS-1052.3.patch, > HDFS-1052.4.patch, HDFS-1052.5.patch, HDFS-1052.6.patch, HDFS-1052.patch, > Mulitple Namespaces5.pdf, high-level-design.pdf > > > HDFS currently uses a single namenode that limits scalability of the cluster. > This jira proposes an architecture to scale the nameservice horizontally > using multiple namenodes. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1860) when renewing/canceling DelegationToken over http we need to pass exception information back to the caller.
[ https://issues.apache.org/jira/browse/HDFS-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated HDFS-1860: - Status: Open (was: Patch Available) > when renewing/canceling DelegationToken over http we need to pass exception > information back to the caller. > --- > > Key: HDFS-1860 > URL: https://issues.apache.org/jira/browse/HDFS-1860 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Boris Shkolnik >Assignee: Boris Shkolnik > Attachments: HDFS-1860-2.patch, HDFS-1860-4.patch, HDFS-1860-5.patch, > HDFS-1860.20-1.patch, HDFS-1860.22-1.patch, HDFS-1860.patch, MR2420.20-1.patch > > > Current implementation is not using XML for that, so we will pass it as a > part of response message. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1860) when renewing/canceling DelegationToken over http we need to pass exception information back to the caller.
[ https://issues.apache.org/jira/browse/HDFS-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated HDFS-1860: - Attachment: HDFS-1860.20-1.patch renamed patch for previous version (.20) > when renewing/canceling DelegationToken over http we need to pass exception > information back to the caller. > --- > > Key: HDFS-1860 > URL: https://issues.apache.org/jira/browse/HDFS-1860 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Boris Shkolnik >Assignee: Boris Shkolnik > Attachments: HDFS-1860-2.patch, HDFS-1860-4.patch, HDFS-1860-5.patch, > HDFS-1860.20-1.patch, HDFS-1860.22-1.patch, HDFS-1860.patch, MR2420.20-1.patch > > > Current implementation is not using XML for that, so we will pass it as a > part of response message. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1860) when renewing/canceling DelegationToken over http we need to pass exception information back to the caller.
[ https://issues.apache.org/jira/browse/HDFS-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated HDFS-1860: - Attachment: HDFS-1860.22-1.patch patch for earlier release (.22) > when renewing/canceling DelegationToken over http we need to pass exception > information back to the caller. > --- > > Key: HDFS-1860 > URL: https://issues.apache.org/jira/browse/HDFS-1860 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Boris Shkolnik >Assignee: Boris Shkolnik > Attachments: HDFS-1860-2.patch, HDFS-1860-4.patch, HDFS-1860-5.patch, > HDFS-1860.20-1.patch, HDFS-1860.22-1.patch, HDFS-1860.patch, MR2420.20-1.patch > > > Current implementation is not using XML for that, so we will pass it as a > part of response message. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1860) when renewing/canceling DelegationToken over http we need to pass exception information back to the caller.
[ https://issues.apache.org/jira/browse/HDFS-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13027108#comment-13027108 ] Hadoop QA commented on HDFS-1860: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12477821/MR2420.20-1.patch against trunk revision 1097648. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/433//console This message is automatically generated. > when renewing/canceling DelegationToken over http we need to pass exception > information back to the caller. > --- > > Key: HDFS-1860 > URL: https://issues.apache.org/jira/browse/HDFS-1860 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Boris Shkolnik >Assignee: Boris Shkolnik > Attachments: HDFS-1860-2.patch, HDFS-1860-4.patch, HDFS-1860-5.patch, > HDFS-1860.patch, MR2420.20-1.patch > > > Current implementation is not using XML for that, so we will pass it as a > part of response message. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-101) DFS write pipeline : DFSClient sometimes does not detect second datanode failure
[ https://issues.apache.org/jira/browse/HDFS-101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13027104#comment-13027104 ] Hairong Kuang commented on HDFS-101: By simply looking at your client-side log, it seems to me that the datanode sent an ack with two fields but the client expects an ack with 3 fields. Could you please create a jira and post logs there? Thanks! > DFS write pipeline : DFSClient sometimes does not detect second datanode > failure > - > > Key: HDFS-101 > URL: https://issues.apache.org/jira/browse/HDFS-101 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node >Affects Versions: 0.20.1, 0.20-append >Reporter: Raghu Angadi >Assignee: Hairong Kuang >Priority: Blocker > Fix For: 0.20.2, 0.20-append, 0.21.0 > > Attachments: HDFS-101_20-append.patch, detectDownDN-0.20.patch, > detectDownDN1-0.20.patch, detectDownDN2.patch, > detectDownDN3-0.20-yahoo.patch, detectDownDN3-0.20.patch, > detectDownDN3.patch, hdfs-101-branch-0.20-append-cdh3.txt, hdfs-101.tar.gz, > pipelineHeartbeat.patch, pipelineHeartbeat_yahoo.patch > > > When the first datanode's write to second datanode fails or times out > DFSClient ends up marking first datanode as the bad one and removes it from > the pipeline. Similar problem exists on DataNode as well and it is fixed in > HADOOP-3339. From HADOOP-3339 : > "The main issue is that BlockReceiver thread (and DataStreamer in the case of > DFSClient) interrupt() the 'responder' thread. But interrupting is a pretty > coarse control. We don't know what state the responder is in and interrupting > has different effects depending on responder state. To fix this properly we > need to redesign how we handle these interactions." > When the first datanode closes its socket from DFSClient, DFSClient should > properly read all the data left in the socket.. Also, DataNode's closing of > the socket should not result in a TCP reset, otherwise I think DFSClient will > not be able to read from the socket. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1860) when renewing/canceling DelegationToken over http we need to pass exception information back to the caller.
[ https://issues.apache.org/jira/browse/HDFS-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated HDFS-1860: - Attachment: MR2420.20-1.patch patch for previous release (.20) > when renewing/canceling DelegationToken over http we need to pass exception > information back to the caller. > --- > > Key: HDFS-1860 > URL: https://issues.apache.org/jira/browse/HDFS-1860 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Boris Shkolnik >Assignee: Boris Shkolnik > Attachments: HDFS-1860-2.patch, HDFS-1860-4.patch, HDFS-1860-5.patch, > HDFS-1860.patch, MR2420.20-1.patch > > > Current implementation is not using XML for that, so we will pass it as a > part of response message. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-101) DFS write pipeline : DFSClient sometimes does not detect second datanode failure
[ https://issues.apache.org/jira/browse/HDFS-101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13027003#comment-13027003 ] Hari commented on HDFS-101: --- TestFileAppend4 is failing randomnly even with this patch in 20-append branch .. We start 3 dns in the MiniDfsCluster . When the second datanode is killed , the first datanode correctly sends the ack as SUCCESS(for itself) FAILED(2nd dn) to the client .. But client is getting an EOFException while reading this ack from the first dn and hence incorrectly assumes the first dn as bad .. As a result , first dn (which is fine) is removed from the pipeline and eventually 2nd dn ll also be removed during recovery . now only 1 replica (3rd dn) is remaining for the block and the assertion fails .. While the 1st dn is fine and is sending ack correctly to the client , it is not clear why he is getting EOFException . Is this normal ?? The relevant part of the log is : ".. 11-04-27 14:48:16,796 DEBUG hdfs.DFSClient (DFSClient.java:run(2445)) - DataStreamer block blk_-1353652352417823279_1001 wrote packet seqno:-1 size:25 offsetInBlock:0 lastPacketInBlock:false 2011-04-27 14:48:16,796 DEBUG datanode.DataNode (BlockReceiver.java:run(891)) - PacketResponder 2 for block blk_-1353652352417823279_1001 responded an ack: Replies for seqno -1 are SUCCESS FAILED 2011-04-27 14:48:16,796 WARN hdfs.DFSClient (DFSClient.java:run(2596)) - DFSOutputStream ResponseProcessor exception for block blk_-1353652352417823279_1001java.io.EOFException at java.io.DataInputStream.readShort(Unknown Source) at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:125) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2548) 2011-04-27 14:48:16,796 DEBUG datanode.DataNode (BlockReceiver.java:run(789)) - PacketResponder 2 seqno = -2 for block blk_-1353652352417823279_1001 waiting for local datanode to finish write. 2011-04-27 14:48:16,796 WARN hdfs.DFSClient (DFSClient.java:processDatanodeError(2632)) - Error Recovery for block blk_-1353652352417823279_1001 bad datanode[0] 127.0.0.1:4956 " > DFS write pipeline : DFSClient sometimes does not detect second datanode > failure > - > > Key: HDFS-101 > URL: https://issues.apache.org/jira/browse/HDFS-101 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node >Affects Versions: 0.20.1, 0.20-append >Reporter: Raghu Angadi >Assignee: Hairong Kuang >Priority: Blocker > Fix For: 0.20.2, 0.20-append, 0.21.0 > > Attachments: HDFS-101_20-append.patch, detectDownDN-0.20.patch, > detectDownDN1-0.20.patch, detectDownDN2.patch, > detectDownDN3-0.20-yahoo.patch, detectDownDN3-0.20.patch, > detectDownDN3.patch, hdfs-101-branch-0.20-append-cdh3.txt, hdfs-101.tar.gz, > pipelineHeartbeat.patch, pipelineHeartbeat_yahoo.patch > > > When the first datanode's write to second datanode fails or times out > DFSClient ends up marking first datanode as the bad one and removes it from > the pipeline. Similar problem exists on DataNode as well and it is fixed in > HADOOP-3339. From HADOOP-3339 : > "The main issue is that BlockReceiver thread (and DataStreamer in the case of > DFSClient) interrupt() the 'responder' thread. But interrupting is a pretty > coarse control. We don't know what state the responder is in and interrupting > has different effects depending on responder state. To fix this properly we > need to redesign how we handle these interactions." > When the first datanode closes its socket from DFSClient, DFSClient should > properly read all the data left in the socket.. Also, DataNode's closing of > the socket should not result in a TCP reset, otherwise I think DFSClient will > not be able to read from the socket. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1860) when renewing/canceling DelegationToken over http we need to pass exception information back to the caller.
[ https://issues.apache.org/jira/browse/HDFS-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13026952#comment-13026952 ] Hudson commented on HDFS-1860: -- Integrated in Hadoop-Hdfs-trunk #651 (See [https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/651/]) HDFS-1860. when renewing/canceling DelegationToken over http we need to pass exception information back to the caller. > when renewing/canceling DelegationToken over http we need to pass exception > information back to the caller. > --- > > Key: HDFS-1860 > URL: https://issues.apache.org/jira/browse/HDFS-1860 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Boris Shkolnik >Assignee: Boris Shkolnik > Attachments: HDFS-1860-2.patch, HDFS-1860-4.patch, HDFS-1860-5.patch, > HDFS-1860.patch > > > Current implementation is not using XML for that, so we will pass it as a > part of response message. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1799) Refactor log rolling and filename management out of FSEditLog
[ https://issues.apache.org/jira/browse/HDFS-1799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13026940#comment-13026940 ] Ivan Kelly commented on HDFS-1799: -- My primary criticism of this patch is that it introduces the JournalManager interface which is unnecessary in my view. It creates a intermediatory layer between FSEditLog and OutputStream. In one sense this is a straight refactor. You've just moved revert and divert into a different class. However, this straight refactor has required that you change almost every method in FSEditLog to use a list of JournalManagers instead of EditLogOutputStreams. JournalManager currently deals only with output. Its not clear how input of the streams will be handled. open(), close(), abort(), restore(), getCurrentStream(), setBufferCapacity() are all output only. How do you plan to add input methods to this? If you don't, then you've added a new abstraction for output streams, which should be rolled directly into output stream. Lastly, this patch doesn't hide the fact that files are in use here. FSEditLog still knows that it's using edits and edits.new. Are different filenames ever used? The only other place I know of is the Backup spool, which is disabled on this branch. In summary, this patch... 1. Changes too much code. 2. Unclear how input functions will be added in the future. 3. Doesn't hide the fact that files are still in use. Sorry for being a pain in the ass about this. I expect you're quite frustrated by it. I do think there needs to be unified view of how this will look in the future though, as getting the right abstractions now will prevent a lot of pain later. Regarding LogSegment, I'm still on the fence about this. I see some of the merits in it, and I can see why JournalMananger is possibly needed for it. If you retain the list of EditLogOutputStreams as it is now, you'll have to maintain a separate list of some other objects for the creation of new log segments. This could be problematic as these lists need to be kept in sync for error handling etc. > Refactor log rolling and filename management out of FSEditLog > - > > Key: HDFS-1799 > URL: https://issues.apache.org/jira/browse/HDFS-1799 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: Edit log branch (HDFS-1073) >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: Edit log branch (HDFS-1073) > > Attachments: 0001-Added-state-management-to-FSEditLog.patch, > 0002-Standardised-error-pattern.patch, > 0003-Add-JournalFactory-and-move-divert-revert-out-of-FSE.patch, > HDFS-1799-all.diff, hdfs-1799.txt, hdfs-1799.txt, hdfs-1799.txt, hdfs-1799.txt > > > This is somewhat similar to HDFS-1580, but less ambitious. While that JIRA > focuses on pluggability, this task is simply the minimum needed for HDFS-1073: > - Refactor the filename-specific code for rolling, diverting, and reverting > log streams out of FSEditLog into a new class > - Clean up the related code in FSEditLog a bit > Notably, this JIRA is going to temporarily break the BackupNode. I plan to > circle back on the BackupNode later on this branch. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
