Fwd: HDFS replication factor
Hi: It's my understanding that HDFS write operation is not considered completd until all of the replicas have been successfully written.If so, does the replication factor affect the write latency? the mapreduce\spark task will be affected? is there the way to set HDFS write the first replica synchronously and return ,the others in an asynchronous. Thanks in advance.
Re: Apache Hadoop 3.0.1 Release plan
Hi Aaron/Lei, Do you plan to roll an RC with an uncommitted fix? That isn't the right approach. This issue has good visibility and enough discussion. If there is a binding veto in effect then the change must be abandoned. Else you should be able to proceed with committing. However, 3.0.0 must be called out as an abandoned release if we commit it. Regards, Arpit On 2/1/18, 3:01 PM, "Lei Xu" wrote: Sounds good to me, ATM. On Thu, Feb 1, 2018 at 2:34 PM, Aaron T. Myers wrote: > Hey Anu, > > My feeling on HDFS-12990 is that we've discussed it quite a bit already and > it doesn't seem at this point like either side is going to budge. I'm > certainly happy to have a phone call about it, but I don't expect that we'd > make much progress. > > My suggestion is that we simply include the patch posted to HDFS-12990 in > the 3.0.1 RC and call this issue out clearly in the subsequent VOTE thread > for the 3.0.1 release. Eddy, are you up for that? > > Best, > Aaron > > On Thu, Feb 1, 2018 at 1:13 PM, Lei Xu wrote: >> >> +Xiao >> >> My understanding is that we will have this for 3.0.1. Xiao, could >> you give your inputs here? >> >> On Thu, Feb 1, 2018 at 11:55 AM, Anu Engineer >> wrote: >> > Hi Eddy, >> > >> > Thanks for driving this release. Just a quick question, do we have time >> > to close this issue? >> > https://issues.apache.org/jira/browse/HDFS-12990 >> > >> > or are we abandoning it? I believe that this is the last window for us >> > to fix this issue. >> > >> > Should we have a call and get this resolved one way or another? >> > >> > Thanks >> > Anu >> > >> > On 2/1/18, 10:51 AM, "Lei Xu" wrote: >> > >> > Hi, All >> > >> > I just cut branch-3.0.1 from branch-3.0. Please make sure all >> > patches >> > targeted to 3.0.1 being checked in both branch-3.0 and branch-3.0.1. >> > >> > Thanks! >> > Eddy >> > >> > On Tue, Jan 9, 2018 at 11:17 AM, Lei Xu wrote: >> > > Hi, All >> > > >> > > We have released Apache Hadoop 3.0.0 in December [1]. To further >> > > improve the quality of release, we plan to cut branch-3.0.1 branch >> > > tomorrow for the preparation of Apache Hadoop 3.0.1 release. The >> > focus >> > > of 3.0.1 will be fixing blockers (3), critical bugs (1) and bug >> > fixes >> > > [2]. No new features and improvement should be included. >> > > >> > > We plan to cut branch-3.0.1 tomorrow (Jan 10th) and vote for RC on >> > Feb >> > > 1st, targeting for Feb 9th release. >> > > >> > > Please feel free to share your insights. >> > > >> > > [1] >> > https://www.mail-archive.com/general@hadoop.apache.org/msg07757.html >> > > [2] https://issues.apache.org/jira/issues/?filter=12342842 >> > > >> > > Best, >> > > -- >> > > Lei (Eddy) Xu >> > > Software Engineer, Cloudera >> > >> > >> > >> > -- >> > Lei (Eddy) Xu >> > Software Engineer, Cloudera >> > >> > >> > - >> > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org >> > For additional commands, e-mail: common-dev-h...@hadoop.apache.org >> > >> > >> > >> >> >> >> -- >> Lei (Eddy) Xu >> Software Engineer, Cloudera >> >> - >> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org >> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org >> > -- Lei (Eddy) Xu Software Engineer, Cloudera - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13102) Implement SnapshotSkipList class to store Multi level DirectoryDiffs
Shashikant Banerjee created HDFS-13102: -- Summary: Implement SnapshotSkipList class to store Multi level DirectoryDiffs Key: HDFS-13102 URL: https://issues.apache.org/jira/browse/HDFS-13102 Project: Hadoop HDFS Issue Type: Improvement Reporter: Shashikant Banerjee Assignee: Shashikant Banerjee HDFS-11225 explains an issue where deletion of older snapshots can take a very long time in case the no of snapshot diffs is quite large for directories. For any directory under a snapshot, to construct the children list , it needs to combine all the diffs from that particular snapshot to the last snapshotDiff record and reverseApply to the current children list of the directory on live fs. This can take a significant time if the no of snapshot diffs are quite large and changes per diff is significant. This Jira proposes to store the Directory diffs in a SnapshotSkip list, where we store multi level DirectoryDiffs. At each level, the Directory Diff will be cumulative diff of k snapshot diffs, where k is the level of a node in the list. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: Apache Hadoop 3.0.1 Release plan
On Fri, Feb 2, 2018 at 10:22 AM, Arpit Agarwal wrote: > Do you plan to roll an RC with an uncommitted fix? That isn't the right > approach. The fix will be committed to the release branch. We'll vote on the release, and if it receives a majority of +1 votes then it becomes 3.0.1. That's how the PMC decides how to move forward. In this case, that will also resolve whether or not it can be committed to trunk. If this logic is unpersuasive, then we can require a 2/3 majority to replace the codebase. Either way, the PMC will vote to define the consensus view when it is not emergent. > This issue has good visibility and enough discussion. Yes, it has. We always prefer consensus to voting, but when discussion reveals that complete consensus is impossible, we still need a way forward. This is rare, and usually reserved for significant changes (like merging YARN). Frankly, it's embarrassing to resort to it here, but here we are. > If there is a binding veto in effect then the change must be abandoned. Else > you should be able to proceed with committing. However, 3.0.0 must be called > out as an abandoned release if we commit it. This is not accurate. A binding veto from any committer halts progress, but the PMC sets the direction of the project. That includes making decisions that are not universally accepted. -C > On 2/1/18, 3:01 PM, "Lei Xu" wrote: > > Sounds good to me, ATM. > > On Thu, Feb 1, 2018 at 2:34 PM, Aaron T. Myers wrote: > > Hey Anu, > > > > My feeling on HDFS-12990 is that we've discussed it quite a bit already > and > > it doesn't seem at this point like either side is going to budge. I'm > > certainly happy to have a phone call about it, but I don't expect that > we'd > > make much progress. > > > > My suggestion is that we simply include the patch posted to HDFS-12990 > in > > the 3.0.1 RC and call this issue out clearly in the subsequent VOTE > thread > > for the 3.0.1 release. Eddy, are you up for that? > > > > Best, > > Aaron > > > > On Thu, Feb 1, 2018 at 1:13 PM, Lei Xu wrote: > >> > >> +Xiao > >> > >> My understanding is that we will have this for 3.0.1. Xiao, could > >> you give your inputs here? > >> > >> On Thu, Feb 1, 2018 at 11:55 AM, Anu Engineer > > >> wrote: > >> > Hi Eddy, > >> > > >> > Thanks for driving this release. Just a quick question, do we have > time > >> > to close this issue? > >> > https://issues.apache.org/jira/browse/HDFS-12990 > >> > > >> > or are we abandoning it? I believe that this is the last window for > us > >> > to fix this issue. > >> > > >> > Should we have a call and get this resolved one way or another? > >> > > >> > Thanks > >> > Anu > >> > > >> > On 2/1/18, 10:51 AM, "Lei Xu" wrote: > >> > > >> > Hi, All > >> > > >> > I just cut branch-3.0.1 from branch-3.0. Please make sure all > >> > patches > >> > targeted to 3.0.1 being checked in both branch-3.0 and > branch-3.0.1. > >> > > >> > Thanks! > >> > Eddy > >> > > >> > On Tue, Jan 9, 2018 at 11:17 AM, Lei Xu > wrote: > >> > > Hi, All > >> > > > >> > > We have released Apache Hadoop 3.0.0 in December [1]. To > further > >> > > improve the quality of release, we plan to cut branch-3.0.1 > branch > >> > > tomorrow for the preparation of Apache Hadoop 3.0.1 release. > The > >> > focus > >> > > of 3.0.1 will be fixing blockers (3), critical bugs (1) and bug > >> > fixes > >> > > [2]. No new features and improvement should be included. > >> > > > >> > > We plan to cut branch-3.0.1 tomorrow (Jan 10th) and vote for > RC on > >> > Feb > >> > > 1st, targeting for Feb 9th release. > >> > > > >> > > Please feel free to share your insights. > >> > > > >> > > [1] > >> > https://www.mail-archive.com/general@hadoop.apache.org/msg07757.html > >> > > [2] https://issues.apache.org/jira/issues/?filter=12342842 > >> > > > >> > > Best, > >> > > -- > >> > > Lei (Eddy) Xu > >> > > Software Engineer, Cloudera > >> > > >> > > >> > > >> > -- > >> > Lei (Eddy) Xu > >> > Software Engineer, Cloudera > >> > > >> > > >> > - > >> > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org > >> > For additional commands, e-mail: > common-dev-h...@hadoop.apache.org > >> > > >> > > >> > > >> > >> > >> > >> -- > >> Lei (Eddy) Xu > >> Software Engineer, Cloudera > >> > >> - > >> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > >> For additio
[jira] [Created] (HDFS-13103) HDFS Client write acknowledgement timeout should not depend on read timeout
Wei-Chiu Chuang created HDFS-13103:
--
Summary: HDFS Client write acknowledgement timeout should not
depend on read timeout
Key: HDFS-13103
URL: https://issues.apache.org/jira/browse/HDFS-13103
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode, hdfs-client
Affects Versions: 3.0.0-alpha1, 2.8.0
Environment: CDH5.7.0 and above. HBase Region Server.
Reporter: Wei-Chiu Chuang
HDFS-8311 added a timeout for client write acknowledgement for both
# transferring blocks
# writing to a DataNode.
The timeout shares the same configuration as client read timeout
(dfs.client.socket-timeout).
While I agree having a timeout is good, it does not make sense for the write
acknowledgement timeout to depend on read timeout. We saw a case where cluster
admin wants to reduce HBase RegionServer read timeout so as to detect DataNode
crash quickly, but did not realize it affects write acknowledgement timeout.
In the end, the effective DataNode write timeout is shorter than the effective
client write acknowledgement timeout. If the last two DataNodes in the write
pipeline crashes, client would think the first DataNode is faulty (the DN
appears unresponsive because it is still waiting for the ack from downstream
DNs), dropping it and then HBase RS would crash because it is unable to write
to any good DataNode. This scenario is possible during a rack failure.
This problem is even worse for Cloudera Manager-managed cluster. By default,
CM-managed HBase RegionServer sets
{{dfs.client.block.write.replace-datanode-on-failure.enable = true}}. Even one
unresponsive DataNode could crash HBase RegionServer.
I am raising this Jira to discuss two possible solutions
# add a new config for write acknowledgement timeout. Do not depend on read
timeout
# or, update the description of dfs.client.socket-timeout in core-default.xml
so that admin is aware write acknowledgement timeout depends on this
configuration.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/677/ [Feb 2, 2018 2:03:01 AM] (aengineer) HDFS-12942. Synchronization issue in FSDataSetImpl#moveBlock. [Feb 2, 2018 3:25:41 AM] (yqlin) HDFS-13068. RBF: Add router admin option to manage safe mode. [Feb 2, 2018 5:34:07 AM] (aajisaka) HDFS-13048. LowRedundancyReplicatedBlocks metric can be negative [Feb 2, 2018 5:33:26 PM] (jlowe) HADOOP-15170. Add symlink support to FileUtil#unTarUsingJava. [Feb 2, 2018 6:28:22 PM] (arun suresh) YARN-7839. Modify PlacementAlgorithm to Check node capacity before [Feb 2, 2018 7:10:47 PM] (jianhe) YARN-7868. Provide improved error message when YARN service is disabled. [Feb 2, 2018 7:37:51 PM] (arp) HADOOP-15198. Correct the spelling in CopyFilter.java. Contributed by [Feb 2, 2018 8:51:27 PM] (hanishakoneru) HADOOP-15168. Add kdiag tool to hadoop command. Contributed by Bharat [Feb 2, 2018 10:38:33 PM] (jianhe) YARN-7831. YARN Service CLI should use hadoop.http.authentication.type [Feb 2, 2018 10:46:20 PM] (kkaranasos) YARN-7778. Merging of placement constraints defined at different levels. [Feb 3, 2018 12:28:03 AM] (hanishakoneru) HDFS-13073. Cleanup code in InterQJournalProtocol.proto. Contributed by [Feb 3, 2018 12:48:57 AM] (szegedim) YARN-7879. NM user is unable to access the application filecache due to [Feb 3, 2018 1:18:42 AM] (weichiu) HDFS-11187. Optimize disk access for last partial chunk checksum of - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13104) Make hadoop proxy user changes reconfigurable in Namenode
Mukul Kumar Singh created HDFS-13104: Summary: Make hadoop proxy user changes reconfigurable in Namenode Key: HDFS-13104 URL: https://issues.apache.org/jira/browse/HDFS-13104 Project: Hadoop HDFS Issue Type: Bug Components: namenode Reporter: Mukul Kumar Singh Assignee: Mukul Kumar Singh Currently any changes to add/delete a new proxy user requires NN restart requiring a downtime. This jira proposes to make the changes in proxy/user configuration reconfiguration via that ReconfigurationProtocol so that the changes can take effect without a NN restart. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
