[jira] Created: (HDFS-1249) with fuse-dfs, chown which only has owner (or only group) argument fails with Input/output error.
with fuse-dfs, chown which only has owner (or only group) argument fails with
Input/output error.
-
Key: HDFS-1249
URL: https://issues.apache.org/jira/browse/HDFS-1249
Project: Hadoop HDFS
Issue Type: Bug
Components: contrib/fuse-dfs
Affects Versions: 0.20.2, 0.20.1
Environment: x86 linux (ubuntu 10.04)
Reporter: matsusaka kentaro
Priority: Minor
with fuse-dfs, chown which only has owner (or only group) argument fails with
Input/output error.
--
/mnt/hdfs/tmp# chown root file1
chown: changing ownership of `file1': Input/output error
/mnt/hdfs/tmp# chown root:root file1
/mnt/hdfs/tmp# chown :root file1
chown: changing group of `file1': Input/output error
--
I think it should be treated as unchanged for missing part(owner or group)
instead of returning an error.
I took fuse_dfs log and it is saying
--
unique: 25, opcode: SETATTR (4), nodeid: 14, insize: 128
chown /tmp/file1 0 4294967295
could not lookup group -1
unique: 25, error: -5 (Input/output error), outsize: 16
unique: 26, opcode: SETATTR (4), nodeid: 14, insize: 128
chown /tmp/file1 0 0
getattr /tmp/file1
unique: 26, success, outsize: 120
unique: 27, opcode: SETATTR (4), nodeid: 14, insize: 128
chown /tmp/file1 4294967295 0
could not lookup userid -1
unique: 27, error: -5 (Input/output error), outsize: 16
--
therefore this should happen because dfs_chown() in
src/contrib/fuse-dfs/src/fuse_impls_chown.c has following
--
...
user = getUsername(uid);
if (NULL == user) {
syslog(LOG_ERR,"Could not lookup the user id string %d\n",(int)uid);
fprintf(stderr, "could not lookup userid %d\n", (int)uid);
ret = -EIO;
}
if (0 == ret) {
group = getGroup(gid);
if (group == NULL) {
syslog(LOG_ERR,"Could not lookup the group id string %d\n",(int)gid);
fprintf(stderr, "could not lookup group %d\n", (int)gid);
ret = -EIO;
}
}
...
--
but actually, hdfsChown() in src/c++/libhdfs/hdfs.c has this
--
...
if (owner == NULL && group == NULL) {
fprintf(stderr, "Both owner and group cannot be null in chown");
errno = EINVAL;
return -1;
}
...
--
and also, setOwner seems allowing NULL
--
username - If it is null, the original username remains unchanged.
groupname - If it is null, the original groupname remains unchanged.
--
according to the api document.
therefore, I think fuse_impls_chown.c should not treat only user(or only group)
lookup fail as an error.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1249) with fuse-dfs, chown which only has owner (or only group) argument fails with Input/output error.
[
https://issues.apache.org/jira/browse/HDFS-1249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
matsusaka kentaro updated HDFS-1249:
Attachment: HDFS1249.1
I think it should behave like this attached diff.
> with fuse-dfs, chown which only has owner (or only group) argument fails with
> Input/output error.
> -
>
> Key: HDFS-1249
> URL: https://issues.apache.org/jira/browse/HDFS-1249
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: contrib/fuse-dfs
>Affects Versions: 0.20.1, 0.20.2
> Environment: x86 linux (ubuntu 10.04)
>Reporter: matsusaka kentaro
>Priority: Minor
> Attachments: HDFS1249.1
>
>
> with fuse-dfs, chown which only has owner (or only group) argument fails with
> Input/output error.
> --
> /mnt/hdfs/tmp# chown root file1
> chown: changing ownership of `file1': Input/output error
> /mnt/hdfs/tmp# chown root:root file1
> /mnt/hdfs/tmp# chown :root file1
> chown: changing group of `file1': Input/output error
> --
> I think it should be treated as unchanged for missing part(owner or group)
> instead of returning an error.
> I took fuse_dfs log and it is saying
> --
> unique: 25, opcode: SETATTR (4), nodeid: 14, insize: 128
> chown /tmp/file1 0 4294967295
> could not lookup group -1
>unique: 25, error: -5 (Input/output error), outsize: 16
> unique: 26, opcode: SETATTR (4), nodeid: 14, insize: 128
> chown /tmp/file1 0 0
> getattr /tmp/file1
>unique: 26, success, outsize: 120
> unique: 27, opcode: SETATTR (4), nodeid: 14, insize: 128
> chown /tmp/file1 4294967295 0
> could not lookup userid -1
>unique: 27, error: -5 (Input/output error), outsize: 16
> --
> therefore this should happen because dfs_chown() in
> src/contrib/fuse-dfs/src/fuse_impls_chown.c has following
> --
> ...
> user = getUsername(uid);
> if (NULL == user) {
> syslog(LOG_ERR,"Could not lookup the user id string %d\n",(int)uid);
> fprintf(stderr, "could not lookup userid %d\n", (int)uid);
> ret = -EIO;
> }
> if (0 == ret) {
> group = getGroup(gid);
> if (group == NULL) {
> syslog(LOG_ERR,"Could not lookup the group id string %d\n",(int)gid);
> fprintf(stderr, "could not lookup group %d\n", (int)gid);
> ret = -EIO;
> }
> }
> ...
> --
> but actually, hdfsChown() in src/c++/libhdfs/hdfs.c has this
> --
> ...
> if (owner == NULL && group == NULL) {
> fprintf(stderr, "Both owner and group cannot be null in chown");
> errno = EINVAL;
> return -1;
> }
> ...
> --
> and also, setOwner seems allowing NULL
> --
> username - If it is null, the original username remains unchanged.
> groupname - If it is null, the original groupname remains unchanged.
> --
> according to the api document.
> therefore, I think fuse_impls_chown.c should not treat only user(or only
> group) lookup fail as an error.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-752) Add interface classification stable & scope to HDFS
[ https://issues.apache.org/jira/browse/HDFS-752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-752: - Attachment: HDFS-752.patch This patch adds interface classification to HDFS classes. Contrib and tests are excluded from this. Approach: - All protocol related classes, Implementations of FileSystem and AbstractFileSystem are tagged private and evolving - All exceptions exposed in protocol classes are marked private and evolving. These classes should perhaps move to protocol package. - All the other public classes are marked private Tagged DatanodeDescriptor.BlockTargetPair as private and evolving, since it is used in BlockCommand. This should be moved to protocol directory, to ensure all the classes used in protocols is in protocol packages. Will open a bug to track this. Tom, can you please review this. > Add interface classification stable & scope to HDFS > --- > > Key: HDFS-752 > URL: https://issues.apache.org/jira/browse/HDFS-752 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.21.0, 0.22.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.21.0, 0.22.0 > > Attachments: HDFS-752.patch, hdfs.interface.txt > > > This jira addresses adding interface classification for the classes in hadoop > hdfs, based on the mechanism described in Hadoop-5073. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1057) Concurrent readers hit ChecksumExceptions if following a writer to very end of file
[
https://issues.apache.org/jira/browse/HDFS-1057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12880278#action_12880278
]
sam rash commented on HDFS-1057:
So removing setBytesOnDisk() means:
{code}
if (replicaInfo instanceof ReplicaBeingWritten) {
((ReplicaBeingWritten) replicaInfo)
.setLastChecksumAndDataLen(offsetInBlock, lastChunkChecksum);
}
replicaInfo.setBytesOnDisk(offsetInBlock);
{code}
will not have the latter. So all other implementations of Replica will have a
valid value for getByteOnDIsk()?
Does this also mean that the impl of getBytesOnDisk for ReplicaInPipeline will
move to ReplicaBeingWritten ?
> Concurrent readers hit ChecksumExceptions if following a writer to very end
> of file
> ---
>
> Key: HDFS-1057
> URL: https://issues.apache.org/jira/browse/HDFS-1057
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
>Affects Versions: 0.20-append, 0.21.0, 0.22.0
>Reporter: Todd Lipcon
>Assignee: sam rash
>Priority: Blocker
> Attachments: conurrent-reader-patch-1.txt,
> conurrent-reader-patch-2.txt, conurrent-reader-patch-3.txt,
> hdfs-1057-trunk-1.txt, hdfs-1057-trunk-2.txt, hdfs-1057-trunk-3.txt
>
>
> In BlockReceiver.receivePacket, it calls replicaInfo.setBytesOnDisk before
> calling flush(). Therefore, if there is a concurrent reader, it's possible to
> race here - the reader will see the new length while those bytes are still in
> the buffers of BlockReceiver. Thus the client will potentially see checksum
> errors or EOFs. Additionally, the last checksum chunk of the file is made
> accessible to readers even though it is not stable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1071) savenamespace should write the fsimage to all configured fs.name.dir in parallel
[ https://issues.apache.org/jira/browse/HDFS-1071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmytro Molkov updated HDFS-1071: Status: Patch Available (was: Open) > savenamespace should write the fsimage to all configured fs.name.dir in > parallel > > > Key: HDFS-1071 > URL: https://issues.apache.org/jira/browse/HDFS-1071 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Reporter: dhruba borthakur >Assignee: Dmytro Molkov > Attachments: HDFS-1071.2.patch, HDFS-1071.3.patch, HDFS-1071.4.patch, > HDFS-1071.5.patch, HDFS-1071.patch > > > If you have a large number of files in HDFS, the fsimage file is very big. > When the namenode restarts, it writes a copy of the fsimage to all > directories configured in fs.name.dir. This takes a long time, especially if > there are many directories in fs.name.dir. Make the NN write the fsimage to > all these directories in parallel. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1071) savenamespace should write the fsimage to all configured fs.name.dir in parallel
[ https://issues.apache.org/jira/browse/HDFS-1071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmytro Molkov updated HDFS-1071: Attachment: HDFS-1071.5.patch Attaching new patch that applies to trunk since there are some conflicts with the older version. > savenamespace should write the fsimage to all configured fs.name.dir in > parallel > > > Key: HDFS-1071 > URL: https://issues.apache.org/jira/browse/HDFS-1071 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Reporter: dhruba borthakur >Assignee: Dmytro Molkov > Attachments: HDFS-1071.2.patch, HDFS-1071.3.patch, HDFS-1071.4.patch, > HDFS-1071.5.patch, HDFS-1071.patch > > > If you have a large number of files in HDFS, the fsimage file is very big. > When the namenode restarts, it writes a copy of the fsimage to all > directories configured in fs.name.dir. This takes a long time, especially if > there are many directories in fs.name.dir. Make the NN write the fsimage to > all these directories in parallel. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1057) Concurrent readers hit ChecksumExceptions if following a writer to very end of file
[ https://issues.apache.org/jira/browse/HDFS-1057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12880291#action_12880291 ] sam rash commented on HDFS-1057: another way to ask this: only ReplicaBeingWritten needs to have the bytes on disk set in BlockRecevier? > Concurrent readers hit ChecksumExceptions if following a writer to very end > of file > --- > > Key: HDFS-1057 > URL: https://issues.apache.org/jira/browse/HDFS-1057 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: data-node >Affects Versions: 0.20-append, 0.21.0, 0.22.0 >Reporter: Todd Lipcon >Assignee: sam rash >Priority: Blocker > Attachments: conurrent-reader-patch-1.txt, > conurrent-reader-patch-2.txt, conurrent-reader-patch-3.txt, > hdfs-1057-trunk-1.txt, hdfs-1057-trunk-2.txt, hdfs-1057-trunk-3.txt > > > In BlockReceiver.receivePacket, it calls replicaInfo.setBytesOnDisk before > calling flush(). Therefore, if there is a concurrent reader, it's possible to > race here - the reader will see the new length while those bytes are still in > the buffers of BlockReceiver. Thus the client will potentially see checksum > errors or EOFs. Additionally, the last checksum chunk of the file is made > accessible to readers even though it is not stable. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1000) libhdfs needs to be updated to use the new UGI
[ https://issues.apache.org/jira/browse/HDFS-1000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HDFS-1000: -- Status: Resolved (was: Patch Available) Fix Version/s: (was: 0.22.0) Resolution: Fixed I just committed this. > libhdfs needs to be updated to use the new UGI > -- > > Key: HDFS-1000 > URL: https://issues.apache.org/jira/browse/HDFS-1000 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.21.0, 0.22.0 >Reporter: Devaraj Das >Assignee: Devaraj Das >Priority: Blocker > Fix For: 0.21.0 > > Attachments: fs-javadoc.patch, hdfs-1000-bp20.3.patch, > hdfs-1000-bp20.4.patch, hdfs-1000-bp20.patch, hdfs-1000-trunk.1.patch > > > libhdfs needs to be updated w.r.t the APIs in the new UGI. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-752) Add interface classification stable & scope to HDFS
[ https://issues.apache.org/jira/browse/HDFS-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12880319#action_12880319 ] Tom White commented on HDFS-752: +1 This looks fine to me. A few public classes seem to have been missed, including HdfsConstants, Upgradeable, FSDatasetInterface, Replica, FSDatasetMBean, FSClusterStats, FSInodeInfo, FSNamesystemMBean, DatanodeProtocol, InterDatanodeProtocol, NamenodeProtocol, NamenodeProtocols, NodeRegistration, GSet, LightWeightGSet. Do you want to add annotations to these? Thanks. > Add interface classification stable & scope to HDFS > --- > > Key: HDFS-752 > URL: https://issues.apache.org/jira/browse/HDFS-752 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.21.0, 0.22.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.21.0, 0.22.0 > > Attachments: HDFS-752.patch, hdfs.interface.txt > > > This jira addresses adding interface classification for the classes in hadoop > hdfs, based on the mechanism described in Hadoop-5073. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1140) Speedup INode.getPathComponents
[ https://issues.apache.org/jira/browse/HDFS-1140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmytro Molkov updated HDFS-1140: Issue Type: Improvement (was: Bug) Priority: Minor (was: Major) > Speedup INode.getPathComponents > --- > > Key: HDFS-1140 > URL: https://issues.apache.org/jira/browse/HDFS-1140 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Dmytro Molkov >Assignee: Dmytro Molkov >Priority: Minor > Attachments: HDFS-1140.2.patch, HDFS-1140.patch > > > When the namenode is loading the image there is a significant amount of time > being spent in the DFSUtil.string2Bytes. We have a very specific workload > here. The path that namenode does getPathComponents for shares N - 1 > component with the previous path this method was called for (assuming current > path has N components). > Hence we can improve the image load time by caching the result of previous > conversion. > We thought of using some simple LRU cache for components, but the reality is, > String.getBytes gets optimized during runtime and LRU cache doesn't perform > as well, however using just the latest path components and their translation > to bytes in two arrays gives quite a performance boost. > I could get another 20% off of the time to load the image on our cluster (30 > seconds vs 24) and I wrote a simple benchmark that tests performance with and > without caching. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HDFS-1245) Plugable block id generation
[ https://issues.apache.org/jira/browse/HDFS-1245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmytro Molkov reassigned HDFS-1245: --- Assignee: Dmytro Molkov > Plugable block id generation > - > > Key: HDFS-1245 > URL: https://issues.apache.org/jira/browse/HDFS-1245 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Reporter: Dmytro Molkov >Assignee: Dmytro Molkov > > The idea is to have a way to easily create block id generation engines that > may fit a certain purpose. One of them could be HDFS-898 started by > Konstantin, but potentially others. > We chatted with Dhruba about this for a while and came up with the following > approach: > There should be a BlockIDGenerator interface that has following methods: > void blockAdded(Block) > void blockRemoved(Block) > Block nextBlock() > First two methods are needed for block generation engines that hold a certain > state. During the restart, when namenode reads the fsimage it will notify > generator about all the blocks it reads from the image and during runtime > namenode will notify the generator about block removals on file deletion. > The instance of the generator will also have a reference to the block > registry, the interface that BlockManager implements. The only method there > is __blockExists(Block)__, so that the current random block id generation can > be implemented, since it needs to check with the block manager if the id is > already present. > What does the community think about this proposal? -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1057) Concurrent readers hit ChecksumExceptions if following a writer to very end of file
[ https://issues.apache.org/jira/browse/HDFS-1057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12880321#action_12880321 ] Hairong Kuang commented on HDFS-1057: - I meant removing setBytesOnDisk and moving setLastChecksumAndDataLen to ReplicaPipelineInterface. > Concurrent readers hit ChecksumExceptions if following a writer to very end > of file > --- > > Key: HDFS-1057 > URL: https://issues.apache.org/jira/browse/HDFS-1057 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: data-node >Affects Versions: 0.20-append, 0.21.0, 0.22.0 >Reporter: Todd Lipcon >Assignee: sam rash >Priority: Blocker > Attachments: conurrent-reader-patch-1.txt, > conurrent-reader-patch-2.txt, conurrent-reader-patch-3.txt, > hdfs-1057-trunk-1.txt, hdfs-1057-trunk-2.txt, hdfs-1057-trunk-3.txt > > > In BlockReceiver.receivePacket, it calls replicaInfo.setBytesOnDisk before > calling flush(). Therefore, if there is a concurrent reader, it's possible to > race here - the reader will see the new length while those bytes are still in > the buffers of BlockReceiver. Thus the client will potentially see checksum > errors or EOFs. Additionally, the last checksum chunk of the file is made > accessible to readers even though it is not stable. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-752) Add interface classification stable & scope to HDFS
[ https://issues.apache.org/jira/browse/HDFS-752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-752: - Attachment: HDFS-752.1.patch Good catch... I had missed tagging public interfaces. Attached patch fixes all the interfaces. One addition to your list StorageDirType. > Add interface classification stable & scope to HDFS > --- > > Key: HDFS-752 > URL: https://issues.apache.org/jira/browse/HDFS-752 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.21.0, 0.22.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.21.0, 0.22.0 > > Attachments: HDFS-752.1.patch, HDFS-752.patch, hdfs.interface.txt > > > This jira addresses adding interface classification for the classes in hadoop > hdfs, based on the mechanism described in Hadoop-5073. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-752) Add interface classification stable & scope to HDFS
[ https://issues.apache.org/jira/browse/HDFS-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12880336#action_12880336 ] Tom White commented on HDFS-752: +1 > Add interface classification stable & scope to HDFS > --- > > Key: HDFS-752 > URL: https://issues.apache.org/jira/browse/HDFS-752 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.21.0, 0.22.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.21.0, 0.22.0 > > Attachments: HDFS-752.1.patch, HDFS-752.patch, hdfs.interface.txt > > > This jira addresses adding interface classification for the classes in hadoop > hdfs, based on the mechanism described in Hadoop-5073. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-752) Add interface classification stable & scope to HDFS
[ https://issues.apache.org/jira/browse/HDFS-752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-752: - Status: Patch Available (was: Open) > Add interface classification stable & scope to HDFS > --- > > Key: HDFS-752 > URL: https://issues.apache.org/jira/browse/HDFS-752 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.21.0, 0.22.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.21.0, 0.22.0 > > Attachments: HDFS-752.1.patch, HDFS-752.patch, hdfs.interface.txt > > > This jira addresses adding interface classification for the classes in hadoop > hdfs, based on the mechanism described in Hadoop-5073. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HDFS-1250) Namenode accepts block report from dead datanodes
Namenode accepts block report from dead datanodes - Key: HDFS-1250 URL: https://issues.apache.org/jira/browse/HDFS-1250 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.20.2, 0.22.0 Reporter: Suresh Srinivas Assignee: Suresh Srinivas When a datanode heartbeat times out namenode marks it dead. The subsequent heartbeat from the datanode is rejected with a command to datanode to re-register. However namenode accepts block report from the datanode although it is marked dead. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1250) Namenode accepts block report from dead datanodes
[ https://issues.apache.org/jira/browse/HDFS-1250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12880342#action_12880342 ] Suresh Srinivas commented on HDFS-1250: --- This creates several problems: # Blocks from dead datanode is added to block map. Opening a file returns blocks from the dead datanode. The client experience failure to read it repeatedly. # Decommissioning the datanode does not resolve this problem as the blocks continue to remain in the block map. Only way to solve this problem is by restarting namenode. This problem was observed on a yahoo cluster, where a datanode was marked dead, however a block report from it made it to namenode. > Namenode accepts block report from dead datanodes > - > > Key: HDFS-1250 > URL: https://issues.apache.org/jira/browse/HDFS-1250 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.20.2, 0.22.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > > When a datanode heartbeat times out namenode marks it dead. The subsequent > heartbeat from the datanode is rejected with a command to datanode to > re-register. However namenode accepts block report from the datanode although > it is marked dead. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1114) Reducing NameNode memory usage by an alternate hash table
[ https://issues.apache.org/jira/browse/HDFS-1114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-1114: - Attachment: benchmark20100618.patch benchmark20100618.patch: additional codes for benchmark and memory footprint measurement. > Reducing NameNode memory usage by an alternate hash table > - > > Key: HDFS-1114 > URL: https://issues.apache.org/jira/browse/HDFS-1114 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Fix For: 0.22.0 > > Attachments: benchmark20100618.patch, GSet20100525.pdf, > gset20100608.pdf, h1114_20100607.patch, h1114_20100614b.patch, > h1114_20100615.patch, h1114_20100616b.patch, h1114_20100617.patch, > h1114_20100617b.patch > > > NameNode uses a java.util.HashMap to store BlockInfo objects. When there are > many blocks in HDFS, this map uses a lot of memory in the NameNode. We may > optimize the memory usage by a light weight hash table implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1057) Concurrent readers hit ChecksumExceptions if following a writer to very end of file
[ https://issues.apache.org/jira/browse/HDFS-1057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12880346#action_12880346 ] sam rash commented on HDFS-1057: got it. i will make the changes and get a patch soon > Concurrent readers hit ChecksumExceptions if following a writer to very end > of file > --- > > Key: HDFS-1057 > URL: https://issues.apache.org/jira/browse/HDFS-1057 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: data-node >Affects Versions: 0.20-append, 0.21.0, 0.22.0 >Reporter: Todd Lipcon >Assignee: sam rash >Priority: Blocker > Attachments: conurrent-reader-patch-1.txt, > conurrent-reader-patch-2.txt, conurrent-reader-patch-3.txt, > hdfs-1057-trunk-1.txt, hdfs-1057-trunk-2.txt, hdfs-1057-trunk-3.txt > > > In BlockReceiver.receivePacket, it calls replicaInfo.setBytesOnDisk before > calling flush(). Therefore, if there is a concurrent reader, it's possible to > race here - the reader will see the new length while those bytes are still in > the buffers of BlockReceiver. Thus the client will potentially see checksum > errors or EOFs. Additionally, the last checksum chunk of the file is made > accessible to readers even though it is not stable. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1192) refreshSuperUserGroupsConfiguration should use server side configuration for the refresh (for HADOOP-6815)
[ https://issues.apache.org/jira/browse/HDFS-1192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated HDFS-1192: - Attachment: HDFS-1192-9.patch > refreshSuperUserGroupsConfiguration should use server side configuration for > the refresh (for HADOOP-6815) > -- > > Key: HDFS-1192 > URL: https://issues.apache.org/jira/browse/HDFS-1192 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Boris Shkolnik >Assignee: Boris Shkolnik > Attachments: HDFS-1192-1.patch, HDFS-1192-9.patch > > > Currently refreshSuperUserGroupsConfiguration is using client side > Configuration. > One of the issues with this is that if the cluster is restarted it will loose > the "refreshed' values (unless they are copied to the NameNode/JobTracker > machine). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-599) Improve Namenode robustness by prioritizing datanode heartbeats over client requests
[ https://issues.apache.org/jira/browse/HDFS-599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hairong Kuang updated HDFS-599: --- Status: Resolved (was: Patch Available) Resolution: Fixed I just committed this. Thanks Dmytro! > Improve Namenode robustness by prioritizing datanode heartbeats over client > requests > > > Key: HDFS-599 > URL: https://issues.apache.org/jira/browse/HDFS-599 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Reporter: dhruba borthakur >Assignee: Dmytro Molkov > Fix For: 0.22.0 > > Attachments: HDFS-599.3.patch, HDFS-599.patch > > > The namenode processes RPC requests from clients that are reading/writing to > files as well as heartbeats/block reports from datanodes. > Sometime, because of various reasons (Java GC runs, inconsistent performance > of NFS filer that stores HDFS transacttion logs, etc), the namenode > encounters transient slowness. For example, if the device that stores the > HDFS transaction logs becomes sluggish, the Namenode's ability to process > RPCs slows down to a certain extent. During this time, the RPCs from clients > as well as the RPCs from datanodes suffer in similar fashion. If the > underlying problem becomes worse, the NN's ability to process a heartbeat > from a DN is severly impacted, thus causing the NN to declare that the DN is > dead. Then the NN starts replicating blocks that used to reside on the > now-declared-dead datanode. This adds extra load to the NN. Then the > now-declared-datanode finally re-establishes contact with the NN, and sends a > block report. The block report processing on the NN is another heavyweight > activity, thus casing more load to the already overloaded namenode. > My proposal is tha the NN should try its best to continue processing RPCs > from datanodes and give lesser priority to serving client requests. The > Datanode RPCs are integral to the consistency and performance of the Hadoop > file system, and it is better to protect it at all costs. This will ensure > that NN recovers from the hiccup much faster than what it does now. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1110) Namenode heap optimization - reuse objects for commonly used file names
[
https://issues.apache.org/jira/browse/HDFS-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12880355#action_12880355
]
Tsz Wo (Nicholas), SZE commented on HDFS-1110:
--
Hi Suresh, new commit should be appended at end of the list in CHANGE.txt but
not inserted in the beginning.
{noformat}
--- hadoop/hdfs/trunk/CHANGES.txt 2010/06/11 18:09:48 953802
+++ hadoop/hdfs/trunk/CHANGES.txt 2010/06/11 18:15:17 953803
@@ -9,6 +9,9 @@
IMPROVEMENTS
+HDFS-1110. Reuses objects for commonly used file names in namenode to
+reduce the heap usage. (suresh)
+
HDFS-1096. fix for prev. commit. (boryas)
{noformat}
> Namenode heap optimization - reuse objects for commonly used file names
> ---
>
> Key: HDFS-1110
> URL: https://issues.apache.org/jira/browse/HDFS-1110
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Suresh Srinivas
>Assignee: Suresh Srinivas
> Fix For: 0.22.0
>
> Attachments: hdfs-1110.2.patch, hdfs-1110.3.patch, hdfs-1110.4.patch,
> hdfs-1110.5.patch, hdfs-1110.6.patch, hdfs-1110.patch, hdfs-1110.y20.patch
>
>
> There are a lot of common file names used in HDFS, mainly created by
> mapreduce, such as file names starting with "part". Reusing byte[]
> corresponding to these recurring file names will save significant heap space
> used for storing the file names in millions of INodeFile objects.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1096) allow dfsadmin/mradmin refresh of superuser proxy group mappings
[
https://issues.apache.org/jira/browse/HDFS-1096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12880356#action_12880356
]
Tsz Wo (Nicholas), SZE commented on HDFS-1096:
--
Hi Boris,
- new commit should be appended at end of the list in CHANGE.txt but not
inserted in the beginning.
- you don't now have to add new item in CHANGE.txt for fixing previous commit.
{noformat}
HDFS-1096. fix for prev. commit. (boryas)
HDFS-1096. allow dfsadmin/mradmin refresh of superuser proxy group
mappings (boryas)
{noformat}
> allow dfsadmin/mradmin refresh of superuser proxy group mappings
>
>
> Key: HDFS-1096
> URL: https://issues.apache.org/jira/browse/HDFS-1096
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Boris Shkolnik
>Assignee: Boris Shkolnik
> Attachments: HDFS-1096-3.patch, HDFS-1096-BP20-4.patch,
> HDFS-1096-BP20-6-fix.patch, HDFS-1096-BP20-6.patch, HDFS-1096-BP20-7.patch,
> HDFS-1096-fix.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1110) Namenode heap optimization - reuse objects for commonly used file names
[ https://issues.apache.org/jira/browse/HDFS-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12880358#action_12880358 ] Suresh Srinivas commented on HDFS-1110: --- Sure. Next time I commit some thing I will fix it. > Namenode heap optimization - reuse objects for commonly used file names > --- > > Key: HDFS-1110 > URL: https://issues.apache.org/jira/browse/HDFS-1110 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.22.0 > > Attachments: hdfs-1110.2.patch, hdfs-1110.3.patch, hdfs-1110.4.patch, > hdfs-1110.5.patch, hdfs-1110.6.patch, hdfs-1110.patch, hdfs-1110.y20.patch > > > There are a lot of common file names used in HDFS, mainly created by > mapreduce, such as file names starting with "part". Reusing byte[] > corresponding to these recurring file names will save significant heap space > used for storing the file names in millions of INodeFile objects. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1119) Refactor BlocksMap with GettableSet
[ https://issues.apache.org/jira/browse/HDFS-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-1119: - Attachment: h1119_20100525_y0.20.1xx.patch h1119_20100525_y0.20.1xx.patch: for y0.20.1xx > Refactor BlocksMap with GettableSet > --- > > Key: HDFS-1119 > URL: https://issues.apache.org/jira/browse/HDFS-1119 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Fix For: 0.22.0 > > Attachments: h1119_20100429.patch, h1119_20100506.patch, > h1119_20100521.patch, h1119_20100525.patch, h1119_20100525_y0.20.1xx.patch > > > The data structure required in BlocksMap is a GettableSet. See also [this > comment|https://issues.apache.org/jira/browse/HDFS-1114?focusedCommentId=12862118&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12862118]. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1250) Namenode accepts block report from dead datanodes
[ https://issues.apache.org/jira/browse/HDFS-1250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12880366#action_12880366 ] Suresh Srinivas commented on HDFS-1250: --- Hudson is stuck. I ran testpatch; here is the result: [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] -1 tests included. The patch doesn't appear to include any new or modified tests. [exec] Please justify why no new tests are needed for this patch. [exec] Also please list what manual steps were performed to verify this patch. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. This change just tags the code with interface classification. Hence tests are not included. I also ran unit tests and it ran without failures. > Namenode accepts block report from dead datanodes > - > > Key: HDFS-1250 > URL: https://issues.apache.org/jira/browse/HDFS-1250 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.20.2, 0.22.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > > When a datanode heartbeat times out namenode marks it dead. The subsequent > heartbeat from the datanode is rejected with a command to datanode to > re-register. However namenode accepts block report from the datanode although > it is marked dead. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-752) Add interface classification stable & scope to HDFS
[ https://issues.apache.org/jira/browse/HDFS-752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-752: - Status: Resolved (was: Patch Available) Hadoop Flags: [Reviewed] Fix Version/s: (was: 0.21.0) Resolution: Fixed Committed the patch to trunk. > Add interface classification stable & scope to HDFS > --- > > Key: HDFS-752 > URL: https://issues.apache.org/jira/browse/HDFS-752 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.21.0, 0.22.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.22.0 > > Attachments: HDFS-752.1.patch, HDFS-752.patch, hdfs.interface.txt > > > This jira addresses adding interface classification for the classes in hadoop > hdfs, based on the mechanism described in Hadoop-5073. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1140) Speedup INode.getPathComponents
[ https://issues.apache.org/jira/browse/HDFS-1140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmytro Molkov updated HDFS-1140: Attachment: HDFS-1140.3.patch I removed one array copy from the routine. This should speed it up even more. I will submit this for tests right away. > Speedup INode.getPathComponents > --- > > Key: HDFS-1140 > URL: https://issues.apache.org/jira/browse/HDFS-1140 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Dmytro Molkov >Assignee: Dmytro Molkov >Priority: Minor > Attachments: HDFS-1140.2.patch, HDFS-1140.3.patch, HDFS-1140.patch > > > When the namenode is loading the image there is a significant amount of time > being spent in the DFSUtil.string2Bytes. We have a very specific workload > here. The path that namenode does getPathComponents for shares N - 1 > component with the previous path this method was called for (assuming current > path has N components). > Hence we can improve the image load time by caching the result of previous > conversion. > We thought of using some simple LRU cache for components, but the reality is, > String.getBytes gets optimized during runtime and LRU cache doesn't perform > as well, however using just the latest path components and their translation > to bytes in two arrays gives quite a performance boost. > I could get another 20% off of the time to load the image on our cluster (30 > seconds vs 24) and I wrote a simple benchmark that tests performance with and > without caching. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1140) Speedup INode.getPathComponents
[ https://issues.apache.org/jira/browse/HDFS-1140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmytro Molkov updated HDFS-1140: Status: Patch Available (was: Open) > Speedup INode.getPathComponents > --- > > Key: HDFS-1140 > URL: https://issues.apache.org/jira/browse/HDFS-1140 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Dmytro Molkov >Assignee: Dmytro Molkov >Priority: Minor > Attachments: HDFS-1140.2.patch, HDFS-1140.3.patch, HDFS-1140.patch > > > When the namenode is loading the image there is a significant amount of time > being spent in the DFSUtil.string2Bytes. We have a very specific workload > here. The path that namenode does getPathComponents for shares N - 1 > component with the previous path this method was called for (assuming current > path has N components). > Hence we can improve the image load time by caching the result of previous > conversion. > We thought of using some simple LRU cache for components, but the reality is, > String.getBytes gets optimized during runtime and LRU cache doesn't perform > as well, however using just the latest path components and their translation > to bytes in two arrays gives quite a performance boost. > I could get another 20% off of the time to load the image on our cluster (30 > seconds vs 24) and I wrote a simple benchmark that tests performance with and > without caching. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1245) Plugable block id generation
[ https://issues.apache.org/jira/browse/HDFS-1245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12880379#action_12880379 ] dhruba borthakur commented on HDFS-1245: Maybe you can write up just the proposed Interface class (with empty methods) along with an imbedded javadoc docs? That might help in understanding this proposal. > Plugable block id generation > - > > Key: HDFS-1245 > URL: https://issues.apache.org/jira/browse/HDFS-1245 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Reporter: Dmytro Molkov >Assignee: Dmytro Molkov > > The idea is to have a way to easily create block id generation engines that > may fit a certain purpose. One of them could be HDFS-898 started by > Konstantin, but potentially others. > We chatted with Dhruba about this for a while and came up with the following > approach: > There should be a BlockIDGenerator interface that has following methods: > void blockAdded(Block) > void blockRemoved(Block) > Block nextBlock() > First two methods are needed for block generation engines that hold a certain > state. During the restart, when namenode reads the fsimage it will notify > generator about all the blocks it reads from the image and during runtime > namenode will notify the generator about block removals on file deletion. > The instance of the generator will also have a reference to the block > registry, the interface that BlockManager implements. The only method there > is __blockExists(Block)__, so that the current random block id generation can > be implemented, since it needs to check with the block manager if the id is > already present. > What does the community think about this proposal? -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-752) Add interface classification stable & scope to HDFS
[ https://issues.apache.org/jira/browse/HDFS-752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-752: - Attachment: HDFS-752.rel21.patch Patch for branch 0.21 > Add interface classification stable & scope to HDFS > --- > > Key: HDFS-752 > URL: https://issues.apache.org/jira/browse/HDFS-752 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.21.0, 0.22.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.22.0 > > Attachments: HDFS-752.1.patch, HDFS-752.patch, HDFS-752.rel21.patch, > hdfs.interface.txt > > > This jira addresses adding interface classification for the classes in hadoop > hdfs, based on the mechanism described in Hadoop-5073. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-752) Add interface classification stable & scope to HDFS
[ https://issues.apache.org/jira/browse/HDFS-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12880385#action_12880385 ] Suresh Srinivas commented on HDFS-752: -- Tom, should this be committed 0.21 branch? > Add interface classification stable & scope to HDFS > --- > > Key: HDFS-752 > URL: https://issues.apache.org/jira/browse/HDFS-752 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.21.0, 0.22.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.22.0 > > Attachments: HDFS-752.1.patch, HDFS-752.patch, HDFS-752.rel21.patch, > hdfs.interface.txt > > > This jira addresses adding interface classification for the classes in hadoop > hdfs, based on the mechanism described in Hadoop-5073. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-752) Add interface classification stable & scope to HDFS
[ https://issues.apache.org/jira/browse/HDFS-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12880412#action_12880412 ] Tom White commented on HDFS-752: Yes, please. > Add interface classification stable & scope to HDFS > --- > > Key: HDFS-752 > URL: https://issues.apache.org/jira/browse/HDFS-752 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.21.0, 0.22.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.22.0 > > Attachments: HDFS-752.1.patch, HDFS-752.patch, HDFS-752.rel21.patch, > hdfs.interface.txt > > > This jira addresses adding interface classification for the classes in hadoop > hdfs, based on the mechanism described in Hadoop-5073. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1114) Reducing NameNode memory usage by an alternate hash table
[ https://issues.apache.org/jira/browse/HDFS-1114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12880423#action_12880423 ] Scott Carey commented on HDFS-1114: --- bq. This is a good point. Is there a way to determine if UseCompressedOops is set in runtime? Well, there is ManagementFactory.getRuntimeMXBean().getInputArguments(), but later versions of Java are going to be making +UseCompressedOops the default. There is also a way to check if the VM is 64 bit or 32 bit, either its out of ManagementFactory or one of the system properties. Digging around I don't see it, but I have used it before. I think it is vendor specific though. > Reducing NameNode memory usage by an alternate hash table > - > > Key: HDFS-1114 > URL: https://issues.apache.org/jira/browse/HDFS-1114 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Fix For: 0.22.0 > > Attachments: benchmark20100618.patch, GSet20100525.pdf, > gset20100608.pdf, h1114_20100607.patch, h1114_20100614b.patch, > h1114_20100615.patch, h1114_20100616b.patch, h1114_20100617.patch, > h1114_20100617b.patch > > > NameNode uses a java.util.HashMap to store BlockInfo objects. When there are > many blocks in HDFS, this map uses a lot of memory in the NameNode. We may > optimize the memory usage by a light weight hash table implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HDFS-1251) Unnecessary log messages are printed in info level
Unnecessary log messages are printed in info level
--
Key: HDFS-1251
URL: https://issues.apache.org/jira/browse/HDFS-1251
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs client, security
Reporter: Tsz Wo (Nicholas), SZE
Priority: Minor
When I run a job on a secure cluster, the following messages are printed on the
console.
{noformat}
10/06/19 05:05:18 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token
125033 for tsz
10/06/19 05:05:18 INFO security.TokenCache: Got dt for
hdfs://namenode/user/tsz/.staging/job_201006091641_23763;uri=xx.xx.xx.xx:8020;t.service=xx.xx.xx.xx:8020
{noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-752) Add interface classification stable & scope to HDFS
[ https://issues.apache.org/jira/browse/HDFS-752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12880430#action_12880430 ] Suresh Srinivas commented on HDFS-752: -- Committed the patch to 0.21 branch. > Add interface classification stable & scope to HDFS > --- > > Key: HDFS-752 > URL: https://issues.apache.org/jira/browse/HDFS-752 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.21.0, 0.22.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.22.0 > > Attachments: HDFS-752.1.patch, HDFS-752.patch, HDFS-752.rel21.patch, > hdfs.interface.txt > > > This jira addresses adding interface classification for the classes in hadoop > hdfs, based on the mechanism described in Hadoop-5073. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-947) The namenode should redirect a hftp request to read a file to the datanode that has the maximum number of local replicas
[ https://issues.apache.org/jira/browse/HDFS-947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated HDFS-947: -- Status: Resolved (was: Patch Available) Hadoop Flags: [Reviewed] Fix Version/s: 0.22.0 Resolution: Fixed I just committed this. Thanks Dmytro! > The namenode should redirect a hftp request to read a file to the datanode > that has the maximum number of local replicas > > > Key: HDFS-947 > URL: https://issues.apache.org/jira/browse/HDFS-947 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: dhruba borthakur >Assignee: Dmytro Molkov > Fix For: 0.22.0 > > Attachments: HDFS-947.2.patch, HDFS-947.patch, hftpRedirection.patch > > > A client that uses the Hftp protocol to read a file is redirected by the > namenode to a random datanode. It would be nice if the client gets redirected > to a datanode that has the maximum number of local replicas of the blocks of > the file. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
