[jira] [Resolved] (MAPREDUCE-7278) Speculative execution behavior is observed even when mapreduce.map.speculative and mapreduce.reduce.speculative are false
[ https://issues.apache.org/jira/browse/MAPREDUCE-7278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wilfred Spiegelenburg resolved MAPREDUCE-7278. -- Resolution: Fixed Pulled this back to branch-3.1 for 3.1.5 closing, thank you [~tarunparimi] for your contribution > Speculative execution behavior is observed even when > mapreduce.map.speculative and mapreduce.reduce.speculative are false > - > > Key: MAPREDUCE-7278 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7278 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: task >Affects Versions: 2.8.0, 3.4.0 >Reporter: Tarun Parimi >Assignee: Tarun Parimi >Priority: Major > Fix For: 3.3.0, 3.2.2, 3.4.0, 3.1.5 > > Attachments: MAPREDUCE-7278.001.patch, MAPREDUCE-7278.002.patch, > MAPREDUCE-7278.003.patch, MAPREDUCE-7278.004.patch, Screen Shot 2020-04-30 at > 8.04.27 PM.png > > > When a failed task attempt container is stuck in FAIL_FINISHING_CONTAINER > state for some time, we observe two task attempts are launched simultaneously > even when speculative execution is disabled. > This results in the below message shown in the killed attempts, indicating > speculation has occurred. This is an issue for jobs which require speculative > execution to be strictly disabled. > !Screen Shot 2020-04-30 at 8.04.27 PM.png! > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
Re: [VOTE] EOL Hadoop branch-2.8
+1 Wilfred > On 3 Mar 2020, at 05:48, Wei-Chiu Chuang wrote: > > I am sorry I forgot to start a VOTE thread. > > This is the "official" vote thread to mark branch-2.8 End of Life. This is > based on the following thread and the tracking jira (HADOOP-16880 > <https://issues.apache.org/jira/browse/HADOOP-16880>). > > This vote will run for 7 days and conclude on March 9th (Mon) 11am PST. > > Please feel free to share your thoughts. > > Thanks, > Weichiu > > On Mon, Feb 24, 2020 at 10:28 AM Wei-Chiu Chuang > wrote: > >> Looking at the EOL policy wiki: >> https://cwiki.apache.org/confluence/display/HADOOP/EOL+%28End-of-life%29+Release+Branches >> >> The Hadoop community can still elect to make security update for EOL'ed >> releases. >> >> I think the EOL is to give more clarity to downstream applications (such >> as HBase) the guidance of which Hadoop release lines are still active. >> Additionally, I don't think it is sustainable to maintain 6 concurrent >> release lines in this big project, which is why I wanted to start this >> discussion. >> >> Thoughts? >> >> On Mon, Feb 24, 2020 at 10:22 AM Sunil Govindan wrote: >> >>> Hi Wei-Chiu >>> >>> Extremely sorry for the late reply here. >>> Cud u pls help to add more clarity on defining what will happen for >>> branch-2.8 when we call EOL. >>> Does this mean that, no more release coming out from this branch, or some >>> more additional guidelines? >>> >>> - Sunil >>> >>> >>> On Mon, Feb 24, 2020 at 11:47 PM Wei-Chiu Chuang >>> wrote: >>> >>>> This thread has been running for 7 days and no -1. >>>> >>>> Don't think we've established a formal EOL process, but to publicize the >>>> EOL, I am going to file a jira, update the wiki and post the >>> announcement >>>> to general@ and user@ >>>> >>>> On Wed, Feb 19, 2020 at 1:40 PM Dinesh Chitlangia < >>> dineshc@gmail.com> >>>> wrote: >>>> >>>>> Thanks Wei-Chiu for initiating this. >>>>> >>>>> +1 for 2.8 EOL. >>>>> >>>>> On Tue, Feb 18, 2020 at 10:48 PM Akira Ajisaka >>>>> wrote: >>>>> >>>>>> Thanks Wei-Chiu for starting the discussion, >>>>>> >>>>>> +1 for the EoL. >>>>>> >>>>>> -Akira >>>>>> >>>>>> On Tue, Feb 18, 2020 at 4:59 PM Ayush Saxena >>>> wrote: >>>>>> >>>>>>> Thanx Wei-Chiu for initiating this >>>>>>> +1 for marking 2.8 EOL >>>>>>> >>>>>>> -Ayush >>>>>>> >>>>>>>> On 17-Feb-2020, at 11:14 PM, Wei-Chiu Chuang < >>> weic...@apache.org> >>>>>> wrote: >>>>>>>> >>>>>>>> The last Hadoop 2.8.x release, 2.8.5, was GA on September 15th >>>> 2018. >>>>>>>> >>>>>>>> It's been 17 months since the release and the community by and >>>> large >>>>>> have >>>>>>>> moved up to 2.9/2.10/3.x. >>>>>>>> >>>>>>>> With Hadoop 3.3.0 over the horizon, is it time to start the EOL >>>>>>> discussion >>>>>>>> and reduce the number of active branches? >>>>>>> >>>>>>> >>> - >>>>>>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org >>>>>>> For additional commands, e-mail: >>> common-dev-h...@hadoop.apache.org >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> Wilfred Spiegelenburg Staff Software Engineer <https://www.cloudera.com/>
[jira] [Created] (MAPREDUCE-7249) Invalid event TA_TOO_MANY_FETCH_FAILURE at SUCCESS_CONTAINER_CLEANUP cause job
Wilfred Spiegelenburg created MAPREDUCE-7249:
Summary: Invalid event TA_TOO_MANY_FETCH_FAILURE at
SUCCESS_CONTAINER_CLEANUP cause job
Key: MAPREDUCE-7249
URL: https://issues.apache.org/jira/browse/MAPREDUCE-7249
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, mrv2
Affects Versions: 3.1.0
Reporter: Wilfred Spiegelenburg
Assignee: Wilfred Spiegelenburg
Same issue as in MAPREDUCE-7240 but this one has a different state in which the
Exception {{TA_TOO_MANY_FETCH_FAILURE}} event is received:
{code}
2019-11-18 23:03:40,270 ERROR [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Can't handle this
event at current state for attempt_1568654141590_630203_m_003108_1
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
TA_TOO_MANY_FETCH_FAILURE at SUCCESS_CONTAINER_CLEANUP
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:1183)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:148)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:1388)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:1380)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:182)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:109)
{code}
The stack trace is from a CDH release which is highly patched 2.6 release.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.1.2 - RC1
+1 (non binding) - build from source on MacOSX 10.14.2, 1.8.0u181 - successful native build on Ubuntu 16.04.3 - deployed a single node cluster (openjdk 1.8u191 / centos 7.5) - uploaded the MR framework - configured YARN with the FS - ran multiple MR jobs > On 29 Jan 2019, at 17:19, Sunil G wrote: > > Hi Folks, > > On behalf of Wangda, we have an RC1 for Apache Hadoop 3.1.2. > > The artifacts are available here: > http://home.apache.org/~sunilg/hadoop-3.1.2-RC1/ > > The RC tag in git is release-3.1.2-RC1: > https://github.com/apache/hadoop/commits/release-3.1.2-RC1 > > The maven artifacts are available via repository.apache.org at > https://repository.apache.org/content/repositories/orgapachehadoop-1215 > > This vote will run 5 days from now. > > 3.1.2 contains 325 [1] fixed JIRA issues since 3.1.1. > > We have done testing with a pseudo cluster and distributed shell job. > > My +1 to start. > > Best, > Wangda Tan and Sunil Govindan > > [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.2) > ORDER BY priority DESC Wilfred Spiegelenburg | Software Engineer cloudera.com <https://www.cloudera.com/>
Re: [VOTE] Release Apache Hadoop 3.2.0 - RC1
+1 (non binding) - build from source on MacOSX 10.14.2, 1.8.0u181 - successful native build on Ubuntu 16.04.3 - confirmed the checksum and signature - deployed a single node cluster (openjdk 1.8u191 / centos 7.5) - uploaded the MR framework - configured YARN with the FS - ran multiple MR jobs > On 8 Jan 2019, at 22:42, Sunil G wrote: > > Hi folks, > > > Thanks to all of you who helped in this release [1] and for helping to vote > for RC0. I have created second release candidate (RC1) for Apache Hadoop > 3.2.0. > > > Artifacts for this RC are available here: > > http://home.apache.org/~sunilg/hadoop-3.2.0-RC1/ > > > RC tag in git is release-3.2.0-RC1. > > > > The maven artifacts are available via repository.apache.org at > https://repository.apache.org/content/repositories/orgapachehadoop-1178/ > > > This vote will run 7 days (5 weekdays), ending on 14th Jan at 11:59 pm PST. > > > > 3.2.0 contains 1092 [2] fixed JIRA issues since 3.1.0. Below feature > additions > > are the highlights of this release. > > 1. Node Attributes Support in YARN > > 2. Hadoop Submarine project for running Deep Learning workloads on YARN > > 3. Support service upgrade via YARN Service API and CLI > > 4. HDFS Storage Policy Satisfier > > 5. Support Windows Azure Storage - Blob file system in Hadoop > > 6. Phase 3 improvements for S3Guard and Phase 5 improvements S3a > > 7. Improvements in Router-based HDFS federation > > > > Thanks to Wangda, Vinod, Marton for helping me in preparing the release. > > I have done few testing with my pseudo cluster. My +1 to start. > > > > Regards, > > Sunil > > > > [1] > > https://lists.apache.org/thread.html/68c1745dcb65602aecce6f7e6b7f0af3d974b1bf0048e7823e58b06f@%3Cyarn-dev.hadoop.apache.org%3E > > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.2.0) > AND fixVersion not in (3.1.0, 3.0.0, 3.0.0-beta1) AND status = Resolved > ORDER BY fixVersion ASC Wilfred Spiegelenburg | Software Engineer cloudera.com <https://www.cloudera.com/>
Re: [VOTE] Release Apache Hadoop 3.2.0 - RC0
+1 (non binding) - build from source on MacOSX 10.14.1, 1.8.0u181 - successful native build on Ubuntu 16.04.3 - confirmed the checksum and signature - deployed a single node cluster (jdk 1.8u191 / centos 7.5) Wilfred > On 23 Nov 2018, at 23:06, Sunil G wrote: > > Hi folks, > > > > Thanks to all contributors who helped in this release [1]. I have created > > first release candidate (RC0) for Apache Hadoop 3.2.0. > > > Artifacts for this RC are available here: > > http://home.apache.org/~sunilg/hadoop-3.2.0-RC0/ > > > > RC tag in git is release-3.2.0-RC0. > > > > The maven artifacts are available via repository.apache.org at > > https://repository.apache.org/content/repositories/orgapachehadoop-1174/ > > > This vote will run 7 days (5 weekdays), ending on Nov 30 at 11:59 pm PST. > > > > 3.2.0 contains 1079 [2] fixed JIRA issues since 3.1.0. Below feature > additions > > are the highlights of this release. > > 1. Node Attributes Support in YARN > > 2. Hadoop Submarine project for running Deep Learning workloads on YARN > > 3. Support service upgrade via YARN Service API and CLI > > 4. HDFS Storage Policy Satisfier > > 5. Support Windows Azure Storage - Blob file system in Hadoop > > 6. Phase 3 improvements for S3Guard and Phase 5 improvements S3a > > 7. Improvements in Router-based HDFS federation > > > > Thanks to Wangda, Vinod, Marton for helping me in preparing the release. > > I have done few testing with my pseudo cluster. My +1 to start. > > > > Regards, > > Sunil > > > > [1] > > https://lists.apache.org/thread.html/68c1745dcb65602aecce6f7e6b7f0af3d974b1bf0048e7823e58b06f@%3Cyarn-dev.hadoop.apache.org%3E > > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.2.0) > AND fixVersion not in (3.1.0, 3.0.0, 3.0.0-beta1) AND status = Resolved > ORDER BY fixVersion ASC Wilfred Spiegelenburg | Software Engineer cloudera.com <https://www.cloudera.com/>
[jira] [Created] (MAPREDUCE-7072) mapred job -history prints duplicate counter in human output
Wilfred Spiegelenburg created MAPREDUCE-7072:
Summary: mapred job -history prints duplicate counter in human
output
Key: MAPREDUCE-7072
URL: https://issues.apache.org/jira/browse/MAPREDUCE-7072
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Affects Versions: 3.0.0
Reporter: Wilfred Spiegelenburg
Assignee: Wilfred Spiegelenburg
'mapred job -history' command prints duplicate entries for counters only for
the human output format. It does not do this for the JSON format.
mapred job -history /user/history/somefile.jhist -format human
{code}
|Job Counters |Total megabyte-seconds taken by all map tasks|0 |0 |268,288,000
...
|Job Counters |Total megabyte-seconds taken by all map tasks|0 |0 |268,288,000
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-6739) allow specifying range on the port that MR AM web server binds to
[ https://issues.apache.org/jira/browse/MAPREDUCE-6739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wilfred Spiegelenburg resolved MAPREDUCE-6739. -- Resolution: Duplicate Closing this as a duplicate of MAPREDUCE-6404. There has been progress on that jira and none on this one > allow specifying range on the port that MR AM web server binds to > - > > Key: MAPREDUCE-6739 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6739 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mr-am >Affects Versions: 2.7.2 >Reporter: Haibo Chen >Assignee: Haibo Chen > Labels: supportability > > MR AM web server binds itself to an arbitrary port. This means if the RM web > proxy lives outside of a cluster, the whole port range needs to be wide open. > It'd be nice to reuse yarn.app.mapreduce.am.job.client.port-range to place a > port range restriction on MR AM web server, so that connection from outside > the cluster can be restricted within a range of ports. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6558) multibyte delimiters with compressed input files generate duplicate records
Wilfred Spiegelenburg created MAPREDUCE-6558: Summary: multibyte delimiters with compressed input files generate duplicate records Key: MAPREDUCE-6558 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6558 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1, mrv2 Affects Versions: 2.7.2 Reporter: Wilfred Spiegelenburg Assignee: Wilfred Spiegelenburg This is the follow up for MAPREDUCE-6549. Compressed files cause record duplications as shown in different junit tests. The number of duplicated records changes with the splitsize: Unexpected number of records in split (splitsize = 10) Expected: 41051 Actual: 45062 Unexpected number of records in split (splitsize = 10) Expected: 41051 Actual: 41052 Test passes with splitsize = 147445 which is the compressed file length.The file is a bzip2 file with 100k blocks and a total of 11 blocks -- This message was sent by Atlassian JIRA (v6.3.4#6332)
