Re: Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

[Chris Douglas]

Yeah, consistently failing nightly builds are just burning resources.
Until someone starts working to fix it, we shouldn't keep submitting
the job. Too bad; I thought build times and resource usage were
becoming more manageable on branch-2.

If anyone has cycles to work on this, the job is here [1]. -C

[1]: https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/

On Tue, May 15, 2018 at 10:27 AM, Allen Wittenauer
<a...@effectivemachines.com> wrote:
>
>> On May 15, 2018, at 10:16 AM, Chris Douglas <cdoug...@apache.org> wrote:
>>
>> They've been failing for a long time. It can't install bats, and
>> that's fatal? -C
>
>
> The bats error is new and causes the build to fail enough that it 
> produces the email output.  For the past few months, it hasn’t been producing 
> email output at all because the builds have been timing out.  (The last 
> ‘good’ report was Feb 26.)  Since no one [*] is paying attention to them 
> enough to notice, I figured it was better to free up the cycles for the rest 
> of the ASF.
>
> * - I noticed a while back, but for various reasons I’ve mostly moved to only 
> working on Hadoop things where I’m getting paid.

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

[Chris Douglas]

They've been failing for a long time. It can't install bats, and
that's fatal? -C

On Tue, May 15, 2018 at 9:43 AM, Allen Wittenauer
 wrote:
>
>
> FYI:
>
> I’m going to disable the branch-2 nightly jobs.
> -
> To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
>

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: Apache Hadoop qbt Report: trunk+JDK8 on Windows/x64

[Chris Douglas]

Thanks, Allen.

On Thu, Mar 15, 2018 at 10:20 AM, Iñigo Goiri  wrote:
>> * It ALWAYS applies HADOOP-14667.05.patch prior to running.  As a result,
>> this is only set up for trunk with no parameterization to run other
>> branches.

I tried to get this running in my environment a few weeks ago, but I
suspect my problems are with my environment, not the patch.

Let's at least get this committed to trunk. We can look at other
branches later. -C

>> * The URI handling for file paths in hadoop-common and elsewhere is pretty
>> broken on Windows, so many many many unit tests are failing and I wouldn't
>> be surprised if Windows hadoop installs are horked as a result.
>>
>> * Runtime is about 12-13 hours with many tests taking significantly longer
>> than their UNIX counterparts.  My guess is that this caused by winutils.
>> Changing from winutils to Java 7 API calls would get this more in line and
>> be a significant performance boost for Windows clients/servers as well.
>>
>> Have fun.
>>
>> =
>>
>> For more details, see https://builds.apache.org/job/hadoop-trunk-win/406/
>> 
>>
>> [Mar 14, 2018 6:26:58 PM] (xyao) HDFS-13251. Avoid using hard coded
>> datanode data dirs in unit tests.
>> [Mar 14, 2018 8:05:24 PM] (jlowe) MAPREDUCE-7064. Flaky test
>> [Mar 14, 2018 8:14:36 PM] (inigoiri) HDFS-13198. RBF:
>> RouterHeartbeatService throws out CachedStateStore
>> [Mar 14, 2018 8:36:53 PM] (wangda) Revert "HADOOP-13707. If kerberos is
>> enabled while HTTP SPNEGO is not
>> [Mar 14, 2018 10:47:56 PM] (fabbri) HADOOP-15278 log s3a at info.
>> Contributed by Steve Loughran.
>>
>>
>>
>>
>> -1 overall
>>
>>
>> The following subsystems voted -1:
>>unit
>>
>>
>> The following subsystems are considered long running:
>> (runtime bigger than 1h 00m 00s)
>>unit
>>
>>
>> Specific tests:
>>
>>Failed CTEST tests :
>>
>>   test_test_libhdfs_threaded_hdfs_static
>>
>>Failed junit tests :
>>
>>   hadoop.crypto.TestCryptoStreamsWithOpensslAesCtrCryptoCodec
>>   hadoop.fs.contract.rawlocal.TestRawlocalContractAppend
>>   hadoop.fs.TestFsShellCopy
>>   hadoop.fs.TestFsShellList
>>   hadoop.fs.TestLocalFileSystem
>>   hadoop.http.TestHttpServer
>>   hadoop.http.TestHttpServerLogs
>>   hadoop.io.compress.TestCodec
>>   hadoop.io.nativeio.TestNativeIO
>>   hadoop.ipc.TestSocketFactory
>>   hadoop.metrics2.impl.TestStatsDMetrics
>>   hadoop.metrics2.sink.TestRollingFileSystemSinkWithLocal
>>   hadoop.security.TestSecurityUtil
>>   hadoop.security.TestShellBasedUnixGroupsMapping
>>   hadoop.security.token.TestDtUtilShell
>>   hadoop.util.TestNativeCodeLoader
>>   hadoop.fs.TestWebHdfsFileContextMainOperations
>>   hadoop.hdfs.client.impl.TestBlockReaderLocalLegacy
>>   hadoop.hdfs.crypto.TestHdfsCryptoStreams
>>   hadoop.hdfs.qjournal.client.TestQuorumJournalManager
>>   hadoop.hdfs.qjournal.server.TestJournalNode
>>   hadoop.hdfs.qjournal.server.TestJournalNodeSync
>>   hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
>>   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
>>   hadoop.hdfs.server.blockmanagement.TestNameNodePrunesMissingStorages
>>   hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks
>>   hadoop.hdfs.server.blockmanagement.TestReplicationPolicy
>>   hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
>>   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles
>>   hadoop.hdfs.server.datanode.fsdataset.impl.
>> TestLazyPersistLockedMemory
>>   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistPolicy
>>   hadoop.hdfs.server.datanode.fsdataset.impl.
>> TestLazyPersistReplicaPlacement
>>   hadoop.hdfs.server.datanode.fsdataset.impl.
>> TestLazyPersistReplicaRecovery
>>   hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter
>>   hadoop.hdfs.server.datanode.fsdataset.impl.TestProvidedImpl
>>   hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation
>>   hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica
>>   hadoop.hdfs.server.datanode.TestBlockPoolSliceStorage
>>   hadoop.hdfs.server.datanode.TestBlockRecovery
>>   hadoop.hdfs.server.datanode.TestBlockScanner
>>   hadoop.hdfs.server.datanode.TestDataNodeFaultInjector
>>   hadoop.hdfs.server.datanode.TestDataNodeMetrics
>>   hadoop.hdfs.server.datanode.TestDataNodeUUID
>>   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
>>   hadoop.hdfs.server.datanode.TestDirectoryScanner
>>   hadoop.hdfs.server.datanode.TestHSync
>>   hadoop.hdfs.server.datanode.web.TestDatanodeHttpXFrame
>>   hadoop.hdfs.server.diskbalancer.command.TestDiskBalancerCommand
>>   hadoop.hdfs.server.diskbalancer.TestDiskBalancerRPC
>>   hadoop.hdfs.server.federation.router.TestRouterAdminCLI
>>   

Re: [EVENT] HDFS Bug Bash: March 12

[Chris Douglas]

Thanks, Vinod. -C

On Mon, Mar 12, 2018 at 2:38 PM, Vinod Kumar Vavilapalli
<vino...@apache.org> wrote:
> Was out of country and away from email for weeks.
>
> We have been using this in the past:
> https://www.meetup.com/Hadoop-Contributors/. I believe this doesn't have the
> per-meetup-charge issue, but will double check. Let me know if there are
> more meetups planned in the near future and we can use this.
>
> Thanks
> +Vinod
>
> On Mar 6, 2018, at 7:48 PM, Chris Douglas <cdoug...@apache.org> wrote:
>
> Found a meetup alternative (thanks Subru):
> https://meetingstar.io/event/fk13172f1d75KN
>
> So we can get a rough headcount, please add (local) if you plan to
> attend in-person. -C
>
>
> On Mon, Mar 5, 2018 at 4:03 PM, Chris Douglas <cdoug...@apache.org> wrote:
>
> [Cross-posting, as this affects the rest of the project]
>
> Hey folks-
>
> As discussed last month [1], the HDFS build hasn't been healthy
> recently. We're dedicating a bug bash to stabilize the build and
> address some longstanding issues with our unit tests. We rely on our
> CI infrastructure to keep the project releasable, and in its current
> state, it's not protecting us from regressions. While we probably
> won't achieve all our goals in this session, we can develop the
> conditions for reestablishing a firm foundation.
>
> If you're new to the project, please consider attending and
> contributing. Committers often prioritize large or complicated
> patches, and the issues that make the project livable don't get enough
> attention. A bug bash is a great opportunity to pull reviewers'
> collars, and fix the annoyances that slow us all down.
>
> If you're a committer, please join us! While some of the proposed
> repairs are rote, many unit tests rely on implementation details and
> non-obvious invariants. We need domain experts to help untangle
> complex dependencies and to prevent breakage of deliberate, but
> counter-intuitive code.
>
> We're collecting tasks in wiki [2] and will include a dial-in option
> for folks who aren't local.
>
> Meetup has started charging for creating new events, so we'll have to
> find another way to get an approximate headcount and publish the
> address. Please ping me if you have a preferred alternative. -C
>
> [1]: https://s.apache.org/nEoQ
> [2]:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75965105
>
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [EVENT] HDFS Bug Bash: March 12

[Chris Douglas]

We're using a shared doc to track work in progress, PA/review ready,
and committed [1]. -C

[1]: https://s.apache.org/RLlx

On Mon, Mar 12, 2018 at 9:17 AM, Chris Douglas <cdoug...@apache.org> wrote:
> On Fri, Mar 9, 2018 at 7:52 PM, 俊平堵 <junping...@apache.org> wrote:
>> Thanks for organizing this, Chris! Please let me know if anything else I
>> can help here.
>
> The plan is to work through items on the wiki [1], including build
> reliability and other "quality of life" issues in the project. -C
>
> [1]: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75965105
>
>> 2018-03-05 16:03 GMT-08:00 Chris Douglas <cdoug...@apache.org>:
>>
>>> [Cross-posting, as this affects the rest of the project]
>>>
>>> Hey folks-
>>>
>>> As discussed last month [1], the HDFS build hasn't been healthy
>>> recently. We're dedicating a bug bash to stabilize the build and
>>> address some longstanding issues with our unit tests. We rely on our
>>> CI infrastructure to keep the project releasable, and in its current
>>> state, it's not protecting us from regressions. While we probably
>>> won't achieve all our goals in this session, we can develop the
>>> conditions for reestablishing a firm foundation.
>>>
>>> If you're new to the project, please consider attending and
>>> contributing. Committers often prioritize large or complicated
>>> patches, and the issues that make the project livable don't get enough
>>> attention. A bug bash is a great opportunity to pull reviewers'
>>> collars, and fix the annoyances that slow us all down.
>>>
>>> If you're a committer, please join us! While some of the proposed
>>> repairs are rote, many unit tests rely on implementation details and
>>> non-obvious invariants. We need domain experts to help untangle
>>> complex dependencies and to prevent breakage of deliberate, but
>>> counter-intuitive code.
>>>
>>> We're collecting tasks in wiki [2] and will include a dial-in option
>>> for folks who aren't local.
>>>
>>> Meetup has started charging for creating new events, so we'll have to
>>> find another way to get an approximate headcount and publish the
>>> address. Please ping me if you have a preferred alternative. -C
>>>
>>> [1]: https://s.apache.org/nEoQ
>>> [2]: https://cwiki.apache.org/confluence/pages/viewpage.
>>> action?pageId=75965105
>>>
>>> -
>>> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
>>> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>>>
>>>

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [EVENT] HDFS Bug Bash: March 12

[Chris Douglas]

On Fri, Mar 9, 2018 at 7:52 PM, 俊平堵 <junping...@apache.org> wrote:
> Thanks for organizing this, Chris! Please let me know if anything else I
> can help here.

The plan is to work through items on the wiki [1], including build
reliability and other "quality of life" issues in the project. -C

[1]: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75965105

> 2018-03-05 16:03 GMT-08:00 Chris Douglas <cdoug...@apache.org>:
>
>> [Cross-posting, as this affects the rest of the project]
>>
>> Hey folks-
>>
>> As discussed last month [1], the HDFS build hasn't been healthy
>> recently. We're dedicating a bug bash to stabilize the build and
>> address some longstanding issues with our unit tests. We rely on our
>> CI infrastructure to keep the project releasable, and in its current
>> state, it's not protecting us from regressions. While we probably
>> won't achieve all our goals in this session, we can develop the
>> conditions for reestablishing a firm foundation.
>>
>> If you're new to the project, please consider attending and
>> contributing. Committers often prioritize large or complicated
>> patches, and the issues that make the project livable don't get enough
>> attention. A bug bash is a great opportunity to pull reviewers'
>> collars, and fix the annoyances that slow us all down.
>>
>> If you're a committer, please join us! While some of the proposed
>> repairs are rote, many unit tests rely on implementation details and
>> non-obvious invariants. We need domain experts to help untangle
>> complex dependencies and to prevent breakage of deliberate, but
>> counter-intuitive code.
>>
>> We're collecting tasks in wiki [2] and will include a dial-in option
>> for folks who aren't local.
>>
>> Meetup has started charging for creating new events, so we'll have to
>> find another way to get an approximate headcount and publish the
>> address. Please ping me if you have a preferred alternative. -C
>>
>> [1]: https://s.apache.org/nEoQ
>> [2]: https://cwiki.apache.org/confluence/pages/viewpage.
>> action?pageId=75965105
>>
>> -
>> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>>
>>

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [EVENT] HDFS Bug Bash: March 12

[Chris Douglas]

If you want to join remotely, we have a bridge set up:

Skype: https://meet.lync.com/microsoft/cdoug/DM56LHTJ
Web app: https://meet.lync.com/microsoft/cdoug/DM56LHTJ?sl=1
Phone: +1 (425) 616-0754 Conference ID: 65908231
Find a local number:
https://dialin.lync.com/8551f4c1-bea3-441a-8738-69aa517a91c5?id=65908231

Please let me know if this doesn't work for you. -C


On Fri, Mar 9, 2018 at 7:52 PM, 俊平堵 <junping...@apache.org> wrote:
> I cannot join this in person as I am currently oversea. but +1 on this
> event.
> Thanks for organizing this, Chris! Please let me know if anything else I
> can help here.
>
> Thanks,
>
> Junping
>
> 2018-03-05 16:03 GMT-08:00 Chris Douglas <cdoug...@apache.org>:
>
>> [Cross-posting, as this affects the rest of the project]
>>
>> Hey folks-
>>
>> As discussed last month [1], the HDFS build hasn't been healthy
>> recently. We're dedicating a bug bash to stabilize the build and
>> address some longstanding issues with our unit tests. We rely on our
>> CI infrastructure to keep the project releasable, and in its current
>> state, it's not protecting us from regressions. While we probably
>> won't achieve all our goals in this session, we can develop the
>> conditions for reestablishing a firm foundation.
>>
>> If you're new to the project, please consider attending and
>> contributing. Committers often prioritize large or complicated
>> patches, and the issues that make the project livable don't get enough
>> attention. A bug bash is a great opportunity to pull reviewers'
>> collars, and fix the annoyances that slow us all down.
>>
>> If you're a committer, please join us! While some of the proposed
>> repairs are rote, many unit tests rely on implementation details and
>> non-obvious invariants. We need domain experts to help untangle
>> complex dependencies and to prevent breakage of deliberate, but
>> counter-intuitive code.
>>
>> We're collecting tasks in wiki [2] and will include a dial-in option
>> for folks who aren't local.
>>
>> Meetup has started charging for creating new events, so we'll have to
>> find another way to get an approximate headcount and publish the
>> address. Please ping me if you have a preferred alternative. -C
>>
>> [1]: https://s.apache.org/nEoQ
>> [2]: https://cwiki.apache.org/confluence/pages/viewpage.
>> action?pageId=75965105
>>
>> -
>> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>>
>>

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [EVENT] HDFS Bug Bash: March 12

[Chris Douglas]

Hey folks-

Remember we're getting together on Monday to fix some "quality of
life" issues in the build. Please RSVP [1] so we can preprint badges,
that sort of thing. The meetup site only allows for 6 hour events, but
we have the room through 6:00PM PST and can keep the bridge/channel
open for other timezones that want to keep going.

We'll post remote access information when we get it. Please update the
wiki [2] as you think of issues we should address. -C

[1] https://meetingstar.io/event/fk13172f1d75KN
[2]: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75965105

On Tue, Mar 6, 2018 at 7:48 PM, Chris Douglas <cdoug...@apache.org> wrote:
> Found a meetup alternative (thanks Subru):
> https://meetingstar.io/event/fk13172f1d75KN
>
> So we can get a rough headcount, please add (local) if you plan to
> attend in-person. -C
>
>
> On Mon, Mar 5, 2018 at 4:03 PM, Chris Douglas <cdoug...@apache.org> wrote:
>> [Cross-posting, as this affects the rest of the project]
>>
>> Hey folks-
>>
>> As discussed last month [1], the HDFS build hasn't been healthy
>> recently. We're dedicating a bug bash to stabilize the build and
>> address some longstanding issues with our unit tests. We rely on our
>> CI infrastructure to keep the project releasable, and in its current
>> state, it's not protecting us from regressions. While we probably
>> won't achieve all our goals in this session, we can develop the
>> conditions for reestablishing a firm foundation.
>>
>> If you're new to the project, please consider attending and
>> contributing. Committers often prioritize large or complicated
>> patches, and the issues that make the project livable don't get enough
>> attention. A bug bash is a great opportunity to pull reviewers'
>> collars, and fix the annoyances that slow us all down.
>>
>> If you're a committer, please join us! While some of the proposed
>> repairs are rote, many unit tests rely on implementation details and
>> non-obvious invariants. We need domain experts to help untangle
>> complex dependencies and to prevent breakage of deliberate, but
>> counter-intuitive code.
>>
>> We're collecting tasks in wiki [2] and will include a dial-in option
>> for folks who aren't local.
>>
>> Meetup has started charging for creating new events, so we'll have to
>> find another way to get an approximate headcount and publish the
>> address. Please ping me if you have a preferred alternative. -C
>>
>> [1]: https://s.apache.org/nEoQ
>> [2]: 
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75965105

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [EVENT] HDFS Bug Bash: March 12

[Chris Douglas]

Found a meetup alternative (thanks Subru):
https://meetingstar.io/event/fk13172f1d75KN

So we can get a rough headcount, please add (local) if you plan to
attend in-person. -C


On Mon, Mar 5, 2018 at 4:03 PM, Chris Douglas <cdoug...@apache.org> wrote:
> [Cross-posting, as this affects the rest of the project]
>
> Hey folks-
>
> As discussed last month [1], the HDFS build hasn't been healthy
> recently. We're dedicating a bug bash to stabilize the build and
> address some longstanding issues with our unit tests. We rely on our
> CI infrastructure to keep the project releasable, and in its current
> state, it's not protecting us from regressions. While we probably
> won't achieve all our goals in this session, we can develop the
> conditions for reestablishing a firm foundation.
>
> If you're new to the project, please consider attending and
> contributing. Committers often prioritize large or complicated
> patches, and the issues that make the project livable don't get enough
> attention. A bug bash is a great opportunity to pull reviewers'
> collars, and fix the annoyances that slow us all down.
>
> If you're a committer, please join us! While some of the proposed
> repairs are rote, many unit tests rely on implementation details and
> non-obvious invariants. We need domain experts to help untangle
> complex dependencies and to prevent breakage of deliberate, but
> counter-intuitive code.
>
> We're collecting tasks in wiki [2] and will include a dial-in option
> for folks who aren't local.
>
> Meetup has started charging for creating new events, so we'll have to
> find another way to get an approximate headcount and publish the
> address. Please ping me if you have a preferred alternative. -C
>
> [1]: https://s.apache.org/nEoQ
> [2]: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75965105

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[EVENT] HDFS Bug Bash: March 12

[Chris Douglas]

[Cross-posting, as this affects the rest of the project]

Hey folks-

As discussed last month [1], the HDFS build hasn't been healthy
recently. We're dedicating a bug bash to stabilize the build and
address some longstanding issues with our unit tests. We rely on our
CI infrastructure to keep the project releasable, and in its current
state, it's not protecting us from regressions. While we probably
won't achieve all our goals in this session, we can develop the
conditions for reestablishing a firm foundation.

If you're new to the project, please consider attending and
contributing. Committers often prioritize large or complicated
patches, and the issues that make the project livable don't get enough
attention. A bug bash is a great opportunity to pull reviewers'
collars, and fix the annoyances that slow us all down.

If you're a committer, please join us! While some of the proposed
repairs are rote, many unit tests rely on implementation details and
non-obvious invariants. We need domain experts to help untangle
complex dependencies and to prevent breakage of deliberate, but
counter-intuitive code.

We're collecting tasks in wiki [2] and will include a dial-in option
for folks who aren't local.

Meetup has started charging for creating new events, so we'll have to
find another way to get an approximate headcount and publish the
address. Please ping me if you have a preferred alternative. -C

[1]: https://s.apache.org/nEoQ
[2]: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75965105

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-7016) Avoid making separate RPC calls for FileStatus and block locations in FileInputFormat

[Chris Douglas (JIRA)]

Chris Douglas created MAPREDUCE-7016:


 Summary: Avoid making separate RPC calls for FileStatus and block 
locations in FileInputFormat
 Key: MAPREDUCE-7016
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7016
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Reporter: Chris Douglas


{{FileInputFormat::getSplits}} uses {{FileSystem::globStatus}} to determine its 
inputs. When the glob returns directories, each is traversed and 
{{LocatedFileStatus}} instances are returned with the block locations. However, 
when the glob returns files, each requires a second RPC to obtain its locations.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-7013) Tests of internal logic should not use the local FS as scratch space

[Chris Douglas (JIRA)]

Chris Douglas created MAPREDUCE-7013:


 Summary: Tests of internal logic should not use the local FS as 
scratch space
 Key: MAPREDUCE-7013
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7013
 Project: Hadoop Map/Reduce
  Issue Type: Test
Reporter: Chris Douglas


MapReduce often manipulates files/permissions to ensure splits, dependencies, 
and other user data are consistently managed. Unit tests of these internal 
methods sometimes set up temporary hierarchies in a scratch directory on the 
local FS to exercise these modules. However, dev environment quirks (e.g., 
umask) can cause these tests to fail spuriously. Instead, this logic should be 
validated by mocking the filesystem.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC3)

[Chris Douglas]

+1 (binding)

Verified source tarball. Checksum and signature match, built from
source, ran some unit tests. Skimmed NOTICE/LICENSE. -C

On Mon, Nov 13, 2017 at 4:10 PM, Arun Suresh  wrote:
> Hi Folks,
>
> Apache Hadoop 2.9.0 is the first release of Hadoop 2.9 line and will be the
> starting release for Apache Hadoop 2.9.x line - it includes 30 New Features
> with 500+ subtasks, 407 Improvements, 790 Bug fixes new fixed issues since
> 2.8.2.
>
> More information about the 2.9.0 release plan can be found here:
> *https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-Version2.9
> *
>
> New RC is available at: *https://home.apache.org/~asuresh/hadoop-2.9.0-RC3/
> *
>
> The RC tag in git is: release-2.9.0-RC3, and the latest commit id is:
> 756ebc8394e473ac25feac05fa493f6d612e6c50.
>
> The maven artifacts are available via repository.apache.org at:
> *https://repository.apache.org/content/repositories/orgapachehadoop-1068/
> *
>
> We are carrying over the votes from the previous RC given that the delta is
> the license fix.
>
> Given the above - we are also going to stick with the original deadline for
> the vote : ending on Friday 17th November 2017 2pm PT time.
>
> Thanks,
> -Arun/Subru

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC0)

[Chris Douglas]

The labor required for these release formalisms is exceeding their
value. Our minor releases have more bugs than our patch releases (we
hope), but every consumer should understand how software versioning
works. Every device I own has bugs on major OS updates. That doesn't
imply that every minor release is strictly less stable than a patch
release, and users need to be warned off it.

In contrast, we should warn users about features that compromise
invariants like security or durability, either by design or due to
their early stage of development. We can't reasonably expect them to
understand those tradeoffs, since they depend on internal details of
Hadoop.

On Wed, Nov 8, 2017 at 5:34 PM, Vinod Kumar Vavilapalli
 wrote:
> When we tried option (b), we used to make .0 as a GA release, but downstream 
> projects like Tez, Hive, Spark would come back and find an incompatible 
> change - and now we were forced into a conundrum - is fixing this 
> incompatible change itself an incompatibility?

Every project takes these case-by-case. Most of the time we'll
accommodate the old semantics- and we try to be explicit where we
promise compatibility- but this isn't a logic problem, it's a
practical one. If it's an easy fix to an obscure API, we probably
won't even hear about it.

> Long story short, I'd just add to your voting thread and release notes that 
> 2.9.0 still needs to be tested downstream and so users may want to wait for 
> subsequent point releases.

It's uncomfortable to have four active release branches, with 3.1
coming in early 2018. We all benefit from the shared deployment
experiences that harden these releases, and fragmentation creates
incentives to compete for that attention. Rather than tacitly
scuffling over waning interest in the 2.x series, I'd endorse your
other thread encouraging consolidation around 3.x.

To that end, there is no policy or precedent that requires that new
minor releases be labeled as "alpha". If there is cause to believe
that 2.9.0 is not ready to release in the stable line, then we
shouldn't release it. -C

>> On Nov 8, 2017, at 12:43 AM, Subru Krishnan  wrote:
>>
>> We are canceling the RC due to the issue that Rohith/Sunil identified. The
>> issue was difficult to track down as it only happens when you use IP for ZK
>> (works fine with host names) and moreover if ZK and RM are co-located on
>> same machine. We are hopeful to get the fix in tomorrow and roll out RC1.
>>
>> Thanks to everyone for the extensive testing/validation. Hopefully cost to
>> replicate with RC1 is much lower.
>>
>> -Subru/Arun.
>>
>> On Tue, Nov 7, 2017 at 5:27 PM, Konstantinos Karanasos >> wrote:
>>
>>> +1 from me too.
>>>
>>> Did the following:
>>> 1) set up a 9-node cluster;
>>> 2) ran some Gridmix jobs;
>>> 3) ran (2) after enabling opportunistic containers (used a mix of
>>> guaranteed and opportunistic containers for each job);
>>> 4) ran (3) but this time enabling distributed scheduling of opportunistic
>>> containers.
>>>
>>> All the above worked with no issues.
>>>
>>> Thanks for all the effort guys!
>>>
>>> Konstantinos
>>>
>>>
>>>
>>> Konstantinos
>>>
>>> On Tue, Nov 7, 2017 at 2:56 PM, Eric Badger 
>>> wrote:
>>>
 +1 (non-binding) pending the issue that Sunil/Rohith pointed out

 - Verified all hashes and checksums
 - Built from source on macOS 10.12.6, Java 1.8.0u65
 - Deployed a pseudo cluster
 - Ran some example jobs

 Thanks,

 Eric

 On Tue, Nov 7, 2017 at 4:03 PM, Wangda Tan  wrote:

> Sunil / Rohith,
>
> Could you check if your configs are same as Jonathan posted configs?
> https://issues.apache.org/jira/browse/YARN-7453?
 focusedCommentId=16242693&
> page=com.atlassian.jira.plugin.system.issuetabpanels:
> comment-tabpanel#comment-16242693
>
> And could you try if using Jonathan's configs can still reproduce the
> issue?
>
> Thanks,
> Wangda
>
>
> On Tue, Nov 7, 2017 at 1:52 PM, Arun Suresh 
>>> wrote:
>
>> Thanks for testing Rohith and Sunil
>>
>> Can you please confirm if it is not a config issue at your end ?
>> We (both Jonathan and myself) just tried testing this on a fresh
 cluster
>> (both automatic and manual) and we are not able to reproduce this.
>>> I've
>> updated the YARN-7453 >> jira/browse/YARN-7453
>
>> JIRA
>> with details of testing.
>>
>> Cheers
>> -Arun/Subru
>>
>> On Tue, Nov 7, 2017 at 3:17 AM, Rohith Sharma K S <
>> rohithsharm...@apache.org
>>> wrote:
>>
>>> Thanks Sunil for confirmation. Btw, I have raised YARN-7453
>>>  JIRA to track
>>> this
>>> issue.
>>>
>>> - Rohith Sharma K S
>>>
>>> On 7 November 2017 at 16:44, Sunil G 

Re: Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

[Chris Douglas]

Sean/Junping-

Ignoring the epistemology, it's a problem. Let's figure out what's
causing memory to balloon and then we can work out the appropriate
remedy.

Is this reproducible outside the CI environment? To Junping's point,
would YETUS-561 provide more detailed information to aid debugging? -C

On Tue, Oct 24, 2017 at 2:50 PM, Junping Du  wrote:
> In general, the "solid evidence" of memory leak comes from analysis of 
> heapdump, jastack, gc log, etc. In many cases, we can locate/conclude which 
> piece of code are leaking memory from the analysis.
>
> Unfortunately, I cannot find any conclusion from previous comments and it 
> even cannot tell which daemons/components of HDFS consumes unexpected high 
> memory. Don't sounds like a solid bug report to me.
>
>
>
> Thanks,?
>
>
> Junping
>
>
> 
> From: Sean Busbey 
> Sent: Tuesday, October 24, 2017 2:20 PM
> To: Junping Du
> Cc: Allen Wittenauer; Hadoop Common; Hdfs-dev; 
> mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
> Subject: Re: Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86
>
> Just curious, Junping what would "solid evidence" look like? Is the 
> supposition here that the memory leak is within HDFS test code rather than 
> library runtime code? How would such a distinction be shown?
>
> On Tue, Oct 24, 2017 at 4:06 PM, Junping Du 
> > wrote:
> Allen,
>  Do we have any solid evidence to show the HDFS unit tests going through 
> the roof are due to serious memory leak by HDFS? Normally, I don't expect 
> memory leak are identified in our UTs - mostly, it (test jvm gone) is just 
> because of test or deployment issues.
>  Unless there is concrete evidence, my concern on seriously memory leak 
> for HDFS on 2.8 is relatively low given some companies (Yahoo, Alibaba, etc.) 
> have deployed 2.8 on large production environment for months. Non-serious 
> memory leak (like forgetting to close stream in non-critical path, etc.) and 
> other non-critical bugs always happens here and there that we have to live 
> with.
>
> Thanks,
>
> Junping
>
> 
> From: Allen Wittenauer 
> >
> Sent: Tuesday, October 24, 2017 8:27 AM
> To: Hadoop Common
> Cc: Hdfs-dev; 
> mapreduce-dev@hadoop.apache.org; 
> yarn-...@hadoop.apache.org
> Subject: Re: Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86
>
>> On Oct 23, 2017, at 12:50 PM, Allen Wittenauer 
>> > wrote:
>>
>>
>>
>> With no other information or access to go on, my current hunch is that one 
>> of the HDFS unit tests is ballooning in memory size.  The easiest way to 
>> kill a Linux machine is to eat all of the RAM, thanks to overcommit and 
>> that's what this "feels" like.
>>
>> Someone should verify if 2.8.2 has the same issues before a release goes out 
>> ...
>
>
> FWIW, I ran 2.8.2 last night and it has the same problems.
>
> Also: the node didn't die!  Looking through the workspace (so the 
> next run will destroy them), two sets of logs stand out:
>
> https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/ws/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
>
> and
>
> https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/ws/sourcedir/hadoop-hdfs-project/hadoop-hdfs/
>
> It looks like my hunch is correct:  RAM in the HDFS unit tests are 
> going through the roof.  It's also interesting how MANY log files there are.  
> Is surefire not picking up that jobs are dying?  Maybe not if memory is 
> getting tight.
>
> Anyway, at the point, branch-2.8 and higher are probably fubar'd. 
> Additionally, I've filed YETUS-561 so that Yetus-controlled Docker containers 
> can have their RAM limits set in order to prevent more nodes going catatonic.
>
>
>
> -
> To unsubscribe, e-mail: 
> yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: 
> yarn-dev-h...@hadoop.apache.org
>
>
>
> -
> To unsubscribe, e-mail: 
> common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: 
> common-dev-h...@hadoop.apache.org
>
>
>
>
> --
> busbey

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.8.2 (RC1)

[Chris Douglas]

+1 (binding)

Looked through the src distribution. Checksum, signatures match, ran
some of the unit tests. Also checked the site docs; thanks for
updating the Docker container security docs.

Thanks, Junping. -C

On Thu, Oct 19, 2017 at 5:42 PM, Junping Du  wrote:
> Hi folks,
>  I've created our new release candidate (RC1) for Apache Hadoop 2.8.2.
>
>  Apache Hadoop 2.8.2 is the first stable release of Hadoop 2.8 line and 
> will be the latest stable/production release for Apache Hadoop - it includes 
> 315 new fixed issues since 2.8.1 and 69 fixes are marked as blocker/critical 
> issues.
>
>   More information about the 2.8.2 release plan can be found here: 
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
>
>   New RC is available at: 
> http://home.apache.org/~junping_du/hadoop-2.8.2-RC1
>
>   The RC tag in git is: release-2.8.2-RC1, and the latest commit id is: 
> 66c47f2a01ad9637879e95f80c41f798373828fb
>
>   The maven artifacts are available via 
> repository.apache.org at: 
> https://repository.apache.org/content/repositories/orgapachehadoop-1064
>
>   Please try the release and vote; the vote will run for the usual 5 
> days, ending on 10/24/2017 6pm PST time.
>
> Thanks,
>
> Junping
>

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)

[Chris Douglas]

On Mon, Sep 11, 2017 at 2:52 PM, Junping Du <j...@hortonworks.com> wrote:
> I don't think this -1 is reasonable, because:
> - If you look at YARN-6622 closely, it targets to fix a problematic 
> documentation work on YARN-5258 which get checked into 2.9 and 3.0 branch 
> only. It means it targets to fix a problem that 2.8.2 never exists.

...we're not going to document security implications- which include
escalations to root- because we don't have _any_ documentation? Why
don't we backport the documentation?

> - New docker container support (replace of old DockerContainerExectutor) is 
> still an alpha feature now which doesn't highlight in 2.8 major 
> features/improvement (http://hadoop.apache.org/docs/r2.8.0/index.html). So 
> adding documentation here is also not a blocker.

YARN-6622 is *documenting* the fact that this is an alpha feature and
that it shouldn't be enabled in secure environments. How are users
supposed to make this determination without it?

> Vote still continue until a real blocker comes.

Soright. I remain -1. -C

> ________
> From: Chris Douglas <cdoug...@apache.org>
> Sent: Monday, September 11, 2017 12:00 PM
> To: Junping Du
> Cc: Miklos Szegedi; Mingliang Liu; Hadoop Common; Hdfs-dev; 
> mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; junping_du
> Subject: Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)
>
> -1 (binding)
>
> I don't think we should release this without YARN-6622.
>
> Since this doesn't happen often: a -1 in this case is NOT a veto.
> Releases are approved by majority vote of the PMC. -C
>
> On Mon, Sep 11, 2017 at 11:45 AM, Junping Du <j...@hortonworks.com> wrote:
>> Thanks Mikols for notifying on this. I think docker support is general known 
>> as alpha feature so document it as experimental is nice to have but not a 
>> blocker for 2.8.2. I also noticed that our 2.7.x document 
>> (https://hadoop.apache.org/docs/r2.7.4/hadoop-yarn/hadoop-yarn-site/DockerContainerExecutor.html)
>>  without mentioning docker support is experimental. We may need to fix that 
>> as well in following releases.
>>
>> I can also add it (mentioning docker container support feature is 
>> experimental) to release message in public website just like previous 
>> release we call 2.7.0/2.8.0 as non-production release.
>>
>> I think vote should continue until we could find a real blocker.
>>
>>
>> Thanks,
>>
>>
>> Junping
>>
>>
>> 
>> From: Miklos Szegedi <miklos.szeg...@cloudera.com>
>> Sent: Monday, September 11, 2017 10:07 AM
>> To: Mingliang Liu
>> Cc: Hadoop Common; Hdfs-dev; mapreduce-dev@hadoop.apache.org; 
>> yarn-...@hadoop.apache.org; junping_du; Junping Du
>> Subject: Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)
>>
>> Hello Junping,
>>
>> Thank you for working on this. Should not YARN-6622 be addressed first? 
>> "Summary: Document Docker work as experimental".
>>
>> Thank you,
>> Miklos
>>
>>
>> On Sun, Sep 10, 2017 at 6:39 PM, Mingliang Liu 
>> <lium...@gmail.com<mailto:lium...@gmail.com>> wrote:
>> Thanks Junping for doing this!
>>
>> +1 (non-binding)
>>
>> - Download the hadoop-2.8.2-src.tar.gz file and checked the md5 value
>> - Build package using maven (skipping tests) with Java 8
>> - Spin up a test cluster in Docker containers having 1 master node (NN/RM) 
>> and 3 slave nodes (DN/NM)
>> - Operate the basic HDFS/YARN operations from command line, both client and 
>> admin
>> - Check NN/RM Web UI
>> - Run distcp to copy files from/to local and HDFS
>> - Run hadoop mapreduce examples: grep and wordcount
>> - Check the HDFS service logs
>>
>> All looked good to me.
>>
>> Mingliang
>>
>>> On Sep 10, 2017, at 5:00 PM, Junping Du 
>>> <j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote:
>>>
>>> Hi folks,
>>> With fix of HADOOP-14842 get in, I've created our first release 
>>> candidate (RC0) for Apache Hadoop 2.8.2.
>>>
>>> Apache Hadoop 2.8.2 is the first stable release of Hadoop 2.8 line and 
>>> will be the latest stable/production release for Apache Hadoop - it 
>>> includes 305 new fixed issues since 2.8.1 and 63 fixes are marked as 
>>> blocker/critical issues.
>>>
>>>  More information about the 2.8.2 release plan can be found here: 
>>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
>>>
>>>  New RC is available at: 
>>> ht

Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)

[Chris Douglas]

-1 (binding)

I don't think we should release this without YARN-6622.

Since this doesn't happen often: a -1 in this case is NOT a veto.
Releases are approved by majority vote of the PMC. -C

On Mon, Sep 11, 2017 at 11:45 AM, Junping Du  wrote:
> Thanks Mikols for notifying on this. I think docker support is general known 
> as alpha feature so document it as experimental is nice to have but not a 
> blocker for 2.8.2. I also noticed that our 2.7.x document 
> (https://hadoop.apache.org/docs/r2.7.4/hadoop-yarn/hadoop-yarn-site/DockerContainerExecutor.html)
>  without mentioning docker support is experimental. We may need to fix that 
> as well in following releases.
>
> I can also add it (mentioning docker container support feature is 
> experimental) to release message in public website just like previous release 
> we call 2.7.0/2.8.0 as non-production release.
>
> I think vote should continue until we could find a real blocker.
>
>
> Thanks,
>
>
> Junping
>
>
> 
> From: Miklos Szegedi 
> Sent: Monday, September 11, 2017 10:07 AM
> To: Mingliang Liu
> Cc: Hadoop Common; Hdfs-dev; mapreduce-dev@hadoop.apache.org; 
> yarn-...@hadoop.apache.org; junping_du; Junping Du
> Subject: Re: [VOTE] Release Apache Hadoop 2.8.2 (RC0)
>
> Hello Junping,
>
> Thank you for working on this. Should not YARN-6622 be addressed first? 
> "Summary: Document Docker work as experimental".
>
> Thank you,
> Miklos
>
>
> On Sun, Sep 10, 2017 at 6:39 PM, Mingliang Liu 
> > wrote:
> Thanks Junping for doing this!
>
> +1 (non-binding)
>
> - Download the hadoop-2.8.2-src.tar.gz file and checked the md5 value
> - Build package using maven (skipping tests) with Java 8
> - Spin up a test cluster in Docker containers having 1 master node (NN/RM) 
> and 3 slave nodes (DN/NM)
> - Operate the basic HDFS/YARN operations from command line, both client and 
> admin
> - Check NN/RM Web UI
> - Run distcp to copy files from/to local and HDFS
> - Run hadoop mapreduce examples: grep and wordcount
> - Check the HDFS service logs
>
> All looked good to me.
>
> Mingliang
>
>> On Sep 10, 2017, at 5:00 PM, Junping Du 
>> > wrote:
>>
>> Hi folks,
>> With fix of HADOOP-14842 get in, I've created our first release 
>> candidate (RC0) for Apache Hadoop 2.8.2.
>>
>> Apache Hadoop 2.8.2 is the first stable release of Hadoop 2.8 line and 
>> will be the latest stable/production release for Apache Hadoop - it includes 
>> 305 new fixed issues since 2.8.1 and 63 fixes are marked as blocker/critical 
>> issues.
>>
>>  More information about the 2.8.2 release plan can be found here: 
>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
>>
>>  New RC is available at: 
>> http://home.apache.org/~junping_du/hadoop-2.8.2-RC0
>>
>>  The RC tag in git is: release-2.8.2-RC0, and the latest commit id is: 
>> e6597fe3000b06847d2bf55f2bab81770f4b2505
>>
>>  The maven artifacts are available via 
>> repository.apache.org at: 
>> https://repository.apache.org/content/repositories/orgapachehadoop-1062
>>
>>  Please try the release and vote; the vote will run for the usual 5 
>> days, ending on 09/15/2017 5pm PST time.
>>
>> Thanks,
>>
>> Junping
>>
>
>
> -
> To unsubscribe, e-mail: 
> mapreduce-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: 
> mapreduce-dev-h...@hadoop.apache.org
>
>

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.7.4 (RC0)

[Chris Douglas]

+1 (binding)

Looked through the src tarball. Checksum and signature match, skimmed
NOTICE/LICENSE, ran some unit tests. -C

On Sat, Jul 29, 2017 at 4:29 PM, Konstantin Shvachko
 wrote:
> Hi everybody,
>
> Here is the next release of Apache Hadoop 2.7 line. The previous stable
> release 2.7.3 was available since 25 August, 2016.
> Release 2.7.4 includes 264 issues fixed after release 2.7.3, which are
> critical bug fixes and major optimizations. See more details in Release
> Note:
> http://home.apache.org/~shv/hadoop-2.7.4-RC0/releasenotes.html
>
> The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.4-RC0/
>
> Please give it a try and vote on this thread. The vote will run for 5 days
> ending 08/04/2017.
>
> Please note that my up to date public key are available from:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> Please don't forget to refresh the page if you've been there recently.
> There are other place on Apache sites, which may contain my outdated key.
>
> Thanks,
> --Konstantin

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.7.4 (RC0)

[Chris Douglas]

On Wed, Aug 2, 2017 at 12:02 AM, Zhe Zhang <z...@apache.org> wrote:
> Quick question for Chris: was your +1 for the RC, or to support
> Konstantin's statement regarding packaging?

For Konstantin's statement. I haven't had a chance to look through the
RC yet, but it's on my list. -C

> On Mon, Jul 31, 2017 at 3:56 PM Chris Douglas <cdoug...@apache.org> wrote:
>
>> On Mon, Jul 31, 2017 at 3:02 PM, Konstantin Shvachko
>> <shv.had...@gmail.com> wrote:
>> > For the packaging, here is the exact phrasing from the sited
>> release-policy
>> > document relevant to binaries:
>> > "As a convenience to users that might not have the appropriate tools to
>> > build a compiled version of the source, binary/bytecode packages MAY be
>> > distributed alongside official Apache releases. In all such cases, the
>> > binary/bytecode package MUST have the same version number as the source
>> > release and MUST only add binary/bytecode files that are the result of
>> > compiling that version of the source code release and its dependencies."
>> > I don't think my binary package violates any of these.
>>
>> +1 The PMC VOTE applies to source code, only. If someone wants to
>> rebuild the binary tarball with native libs and replace this one,
>> that's fine.
>>
>> My reading of the above is that source code must be distributed with
>> binaries, not that we omit the source code from binary releases... -C
>>
>> > But I'll upload an additional tar.gz with native bits and no src, as you
>> > guys requested.
>> > Will keep it as RC0 as there is no source code change and it comes from
>> the
>> > same build.
>> > Hope this is satisfactory.
>> >
>> > Thanks,
>> > --Konstantin
>> >
>> > On Mon, Jul 31, 2017 at 1:53 PM, Andrew Wang <andrew.w...@cloudera.com>
>> > wrote:
>> >
>> >> I agree with Brahma on the two issues flagged (having src in the binary
>> >> tarball, missing native libs). These are regressions from prior
>> releases.
>> >>
>> >> As an aside, "we release binaries as a convenience" doesn't relax the
>> >> quality bar. The binaries are linked on our website and distributed
>> through
>> >> official Apache channels. They have to adhere to Apache release
>> >> requirements. And, most users consume our work via Maven dependencies,
>> >> which are binary artifacts.
>> >>
>> >> http://www.apache.org/legal/release-policy.html goes into this in more
>> >> detail. A release must minimally include source packages, and can also
>> >> include binary artifacts.
>> >>
>> >> Best,
>> >> Andrew
>> >>
>> >> On Mon, Jul 31, 2017 at 12:30 PM, Konstantin Shvachko <
>> >> shv.had...@gmail.com> wrote:
>> >>
>> >>> To avoid any confusion in this regard. I built RC0 manually in
>> compliance
>> >>> with Apache release policy
>> >>> http://www.apache.org/legal/release-policy.html
>> >>> I edited the HowToReleasePreDSBCR page to make sure people don't use
>> >>> Jenkins option for building.
>> >>>
>> >>> A side note. This particular build is broken anyways, so no worries
>> there.
>> >>> I think though it would be useful to have it working for testing and
>> as a
>> >>> packaging standard.
>> >>>
>> >>> Thanks,
>> >>> --Konstantin
>> >>>
>> >>> On Mon, Jul 31, 2017 at 11:40 AM, Allen Wittenauer <
>> >>> a...@effectivemachines.com
>> >>> > wrote:
>> >>>
>> >>> >
>> >>> > > On Jul 31, 2017, at 11:20 AM, Konstantin Shvachko <
>> >>> shv.had...@gmail.com>
>> >>> > wrote:
>> >>> > >
>> >>> > > https://wiki.apache.org/hadoop/HowToReleasePreDSBCR
>> >>> >
>> >>> > FYI:
>> >>> >
>> >>> > If you are using ASF Jenkins to create an ASF release
>> >>> > artifact, it's pretty much an automatic vote failure as any such
>> >>> release is
>> >>> > in violation of ASF policy.
>> >>> >
>> >>> >
>> >>>
>> >>
>> >>
>>
>> -
>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>>
>> --
> Zhe Zhang
> Apache Hadoop Committer
> http://zhe-thoughts.github.io/about/ | @oldcap

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.7.4 (RC0)

[Chris Douglas]

On Mon, Jul 31, 2017 at 3:02 PM, Konstantin Shvachko
 wrote:
> For the packaging, here is the exact phrasing from the sited release-policy
> document relevant to binaries:
> "As a convenience to users that might not have the appropriate tools to
> build a compiled version of the source, binary/bytecode packages MAY be
> distributed alongside official Apache releases. In all such cases, the
> binary/bytecode package MUST have the same version number as the source
> release and MUST only add binary/bytecode files that are the result of
> compiling that version of the source code release and its dependencies."
> I don't think my binary package violates any of these.

+1 The PMC VOTE applies to source code, only. If someone wants to
rebuild the binary tarball with native libs and replace this one,
that's fine.

My reading of the above is that source code must be distributed with
binaries, not that we omit the source code from binary releases... -C

> But I'll upload an additional tar.gz with native bits and no src, as you
> guys requested.
> Will keep it as RC0 as there is no source code change and it comes from the
> same build.
> Hope this is satisfactory.
>
> Thanks,
> --Konstantin
>
> On Mon, Jul 31, 2017 at 1:53 PM, Andrew Wang 
> wrote:
>
>> I agree with Brahma on the two issues flagged (having src in the binary
>> tarball, missing native libs). These are regressions from prior releases.
>>
>> As an aside, "we release binaries as a convenience" doesn't relax the
>> quality bar. The binaries are linked on our website and distributed through
>> official Apache channels. They have to adhere to Apache release
>> requirements. And, most users consume our work via Maven dependencies,
>> which are binary artifacts.
>>
>> http://www.apache.org/legal/release-policy.html goes into this in more
>> detail. A release must minimally include source packages, and can also
>> include binary artifacts.
>>
>> Best,
>> Andrew
>>
>> On Mon, Jul 31, 2017 at 12:30 PM, Konstantin Shvachko <
>> shv.had...@gmail.com> wrote:
>>
>>> To avoid any confusion in this regard. I built RC0 manually in compliance
>>> with Apache release policy
>>> http://www.apache.org/legal/release-policy.html
>>> I edited the HowToReleasePreDSBCR page to make sure people don't use
>>> Jenkins option for building.
>>>
>>> A side note. This particular build is broken anyways, so no worries there.
>>> I think though it would be useful to have it working for testing and as a
>>> packaging standard.
>>>
>>> Thanks,
>>> --Konstantin
>>>
>>> On Mon, Jul 31, 2017 at 11:40 AM, Allen Wittenauer <
>>> a...@effectivemachines.com
>>> > wrote:
>>>
>>> >
>>> > > On Jul 31, 2017, at 11:20 AM, Konstantin Shvachko <
>>> shv.had...@gmail.com>
>>> > wrote:
>>> > >
>>> > > https://wiki.apache.org/hadoop/HowToReleasePreDSBCR
>>> >
>>> > FYI:
>>> >
>>> > If you are using ASF Jenkins to create an ASF release
>>> > artifact, it's pretty much an automatic vote failure as any such
>>> release is
>>> > in violation of ASF policy.
>>> >
>>> >
>>>
>>
>>

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Reopened] (MAPREDUCE-6433) launchTime may be negative

[Chris Douglas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Douglas reopened MAPREDUCE-6433:
--

> launchTime may be negative
> --
>
> Key: MAPREDUCE-6433
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6433
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: jobhistoryserver, mrv2
>Affects Versions: 2.4.1
>Reporter: Allen Wittenauer
>Assignee: zhihai xu
> Fix For: 2.8.0, 3.0.0-alpha1
>
> Attachments: MAPREDUCE-6433.000.patch, MAPREDUCE-6433.001.patch, 
> MAPREDUCE-6433-branch-2.7.001.patch, REPRODUCE.patch
>
>
> Under extremely rare conditions (.0017% in our sample size), launchTime in 
> the jhist files may be set to -1.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: Can we update protobuf's version on trunk?

[Chris Douglas]

On Wed, Mar 29, 2017 at 4:59 PM, Stack  wrote:
>> The former; an intermediate handler decoding, [modifying,] and
>> encoding the record without losing unknown fields.
>>
>
> I did not try this. Did you? Otherwise I can.

Yeah, I did. Same format. -C

>> This looks fine. -C
>>
>> > Thanks,
>> > St.Ack
>> >
>> >
>> > # Using the protoc v3.0.2 tool
>> > $ protoc --version
>> > libprotoc 3.0.2
>> >
>> > # I have a simple proto definition with two fields in it
>> > $ more pb.proto
>> > message Test {
>> >   optional string one = 1;
>> >   optional string two = 2;
>> > }
>> >
>> > # This is a text-encoded instance of a 'Test' proto message:
>> > $ more pb.txt
>> > one: "one"
>> > two: "two"
>> >
>> > # Now I encode the above as a pb binary
>> > $ protoc --encode=Test pb.proto < pb.txt > pb.bin
>> > [libprotobuf WARNING google/protobuf/compiler/parser.cc:546] No syntax
>> > specified for the proto file: pb.proto. Please use 'syntax = "proto2";'
>> > or
>> > 'syntax = "proto3";' to specify a syntax version. (Defaulted to proto2
>> > syntax.)
>> >
>> > # Here is a dump of the binary
>> > $ od -xc pb.bin
>> > 000  030a6e6f126574036f77
>> >   \n 003   o   n   e 022 003   t   w   o
>> > 012
>> >
>> > # Here is a proto definition file that has a Test Message minus the
>> > 'two'
>> > field.
>> > $ more pb_drops_two.proto
>> > message Test {
>> >   optional string one = 1;
>> > }
>> >
>> > # Use it to decode the bin file:
>> > $ protoc --decode=Test pb_drops_two.proto < pb.bin
>> > [libprotobuf WARNING google/protobuf/compiler/parser.cc:546] No syntax
>> > specified for the proto file: pb_drops_two.proto. Please use 'syntax =
>> > "proto2";' or 'syntax = "proto3";' to specify a syntax version.
>> > (Defaulted
>> > to proto2 syntax.)
>> > one: "one"
>> > 2: "two"
>> >
>> > Note how the second field is preserved (absent a field name). It is not
>> > dropped.
>> >
>> > If I change the syntax of pb_drops_two.proto to be proto3, the field IS
>> > dropped.
>> >
>> > # Here proto file with proto3 syntax specified (had to drop the
>> > 'optional'
>> > qualifier -- not allowed in proto3):
>> > $ more pb_drops_two.proto
>> > syntax = "proto3";
>> > message Test {
>> >   string one = 1;
>> > }
>> >
>> > $ protoc --decode=Test pb_drops_two.proto < pb.bin  > pb_drops_two.txt
>> > $ more pb_drops_two.txt
>> > one: "one"
>> >
>> >
>> > I cannot reencode the text output using pb_drops_two.proto. It
>> > complains:
>> >
>> > $ protoc --encode=Test pb_drops_two.proto < pb_drops_two.txt >
>> > pb_drops_two.bin
>> > [libprotobuf WARNING google/protobuf/compiler/parser.cc:546] No syntax
>> > specified for the proto file: pb_drops_two.proto. Please use 'syntax =
>> > "proto2";' or 'syntax = "proto3";' to specify a syntax version.
>> > (Defaulted
>> > to proto2 syntax.)
>> > input:2:1: Expected identifier, got: 2
>> >
>> > Proto 2.5 does same:
>> >
>> > $ ~/bin/protobuf-2.5.0/src/protoc --encode=Test pb_drops_two.proto <
>> > pb_drops_two.txt > pb_drops_two.bin
>> > input:2:1: Expected identifier.
>> > Failed to parse input.
>> >
>> > St.Ack
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Wed, Mar 29, 2017 at 10:14 AM, Stack  wrote:
>> >>
>> >> On Tue, Mar 28, 2017 at 4:18 PM, Andrew Wang 
>> >> wrote:
>> >>>
>> >>> >
>> >>> > > If unknown fields are dropped, then applications proxying tokens
>> >>> > > and
>> >>> > other
>> >>> > >> data between servers will effectively corrupt those messages,
>> >>> > >> unless
>> >>> > >> we
>> >>> > >> make everything opaque bytes, which- absent the convenient,
>> >>> > >> prenominate
>> >>> > >> semantics managing the conversion- obviate the compatibility
>> >>> > >> machinery
>> >>> > that
>> >>> > >> is the whole point of PB. Google is removing the features that
>> >>> > >> justified
>> >>> > >> choosing PB over its alternatives. Since we can't require that
>> >>> > >> our
>> >>> > >> applications compile (or link) against our updated schema, this
>> >>> > >> creates
>> >>> > a
>> >>> > >> problem that PB was supposed to solve.
>> >>> > >
>> >>> > >
>> >>> > > This is scary, and it potentially affects services outside of the
>> >>> > > Hadoop
>> >>> > > codebase. This makes it difficult to assess the impact.
>> >>> >
>> >>> > Stack mentioned a compatibility mode that uses the proto2 semantics.
>> >>> > If that carries unknown fields through intermediate handlers, then
>> >>> > this objection goes away. -C
>> >>>
>> >>>
>> >>> Did some more googling, found this:
>> >>>
>> >>> https://groups.google.com/d/msg/protobuf/Z6pNo81FiEQ/fHkdcNtdAwAJ
>> >>>
>> >>> Feng Xiao appears to be a Google engineer, and suggests workarounds
>> >>> like
>> >>> packing the fields into a byte type. No mention of a PB2 compatibility
>> >>> mode. Also here:
>> >>>
>> >>> https://groups.google.com/d/msg/protobuf/bO2L6-_t91Q/-zIaJAR9AAAJ
>> >>>
>> >>> Participants say that unknown fields were dropped for automatic JSON
>> >>> encoding, since 

Re: Can we update protobuf's version on trunk?

[Chris Douglas]

On Wed, Mar 29, 2017 at 1:13 PM, Stack  wrote:
> Is the below evidence enough that pb3 in proto2 syntax mode does not drop
> 'unknown' fields? (Maybe you want evidence that java tooling behaves the
> same?)

I reproduced your example with the Java tooling, including changing
some of the fields in the intermediate representation. As long as the
syntax is "proto2", it seems to have compatible semantics.

> To be clear, when we say proxy above, are we expecting that a pb message
> deserialized by a process down-the-line that happens to have a crimped proto
> definition that is absent a couple of fields somehow can re-serialize and at
> the end of the line, all fields are present? Or are we talking pass-through
> of the message without rewrite?

The former; an intermediate handler decoding, [modifying,] and
encoding the record without losing unknown fields.

This looks fine. -C

> Thanks,
> St.Ack
>
>
> # Using the protoc v3.0.2 tool
> $ protoc --version
> libprotoc 3.0.2
>
> # I have a simple proto definition with two fields in it
> $ more pb.proto
> message Test {
>   optional string one = 1;
>   optional string two = 2;
> }
>
> # This is a text-encoded instance of a 'Test' proto message:
> $ more pb.txt
> one: "one"
> two: "two"
>
> # Now I encode the above as a pb binary
> $ protoc --encode=Test pb.proto < pb.txt > pb.bin
> [libprotobuf WARNING google/protobuf/compiler/parser.cc:546] No syntax
> specified for the proto file: pb.proto. Please use 'syntax = "proto2";' or
> 'syntax = "proto3";' to specify a syntax version. (Defaulted to proto2
> syntax.)
>
> # Here is a dump of the binary
> $ od -xc pb.bin
> 000  030a6e6f126574036f77
>   \n 003   o   n   e 022 003   t   w   o
> 012
>
> # Here is a proto definition file that has a Test Message minus the 'two'
> field.
> $ more pb_drops_two.proto
> message Test {
>   optional string one = 1;
> }
>
> # Use it to decode the bin file:
> $ protoc --decode=Test pb_drops_two.proto < pb.bin
> [libprotobuf WARNING google/protobuf/compiler/parser.cc:546] No syntax
> specified for the proto file: pb_drops_two.proto. Please use 'syntax =
> "proto2";' or 'syntax = "proto3";' to specify a syntax version. (Defaulted
> to proto2 syntax.)
> one: "one"
> 2: "two"
>
> Note how the second field is preserved (absent a field name). It is not
> dropped.
>
> If I change the syntax of pb_drops_two.proto to be proto3, the field IS
> dropped.
>
> # Here proto file with proto3 syntax specified (had to drop the 'optional'
> qualifier -- not allowed in proto3):
> $ more pb_drops_two.proto
> syntax = "proto3";
> message Test {
>   string one = 1;
> }
>
> $ protoc --decode=Test pb_drops_two.proto < pb.bin  > pb_drops_two.txt
> $ more pb_drops_two.txt
> one: "one"
>
>
> I cannot reencode the text output using pb_drops_two.proto. It complains:
>
> $ protoc --encode=Test pb_drops_two.proto < pb_drops_two.txt >
> pb_drops_two.bin
> [libprotobuf WARNING google/protobuf/compiler/parser.cc:546] No syntax
> specified for the proto file: pb_drops_two.proto. Please use 'syntax =
> "proto2";' or 'syntax = "proto3";' to specify a syntax version. (Defaulted
> to proto2 syntax.)
> input:2:1: Expected identifier, got: 2
>
> Proto 2.5 does same:
>
> $ ~/bin/protobuf-2.5.0/src/protoc --encode=Test pb_drops_two.proto <
> pb_drops_two.txt > pb_drops_two.bin
> input:2:1: Expected identifier.
> Failed to parse input.
>
> St.Ack
>
>
>
>
>
>
> On Wed, Mar 29, 2017 at 10:14 AM, Stack  wrote:
>>
>> On Tue, Mar 28, 2017 at 4:18 PM, Andrew Wang 
>> wrote:
>>>
>>> >
>>> > > If unknown fields are dropped, then applications proxying tokens and
>>> > other
>>> > >> data between servers will effectively corrupt those messages, unless
>>> > >> we
>>> > >> make everything opaque bytes, which- absent the convenient,
>>> > >> prenominate
>>> > >> semantics managing the conversion- obviate the compatibility
>>> > >> machinery
>>> > that
>>> > >> is the whole point of PB. Google is removing the features that
>>> > >> justified
>>> > >> choosing PB over its alternatives. Since we can't require that our
>>> > >> applications compile (or link) against our updated schema, this
>>> > >> creates
>>> > a
>>> > >> problem that PB was supposed to solve.
>>> > >
>>> > >
>>> > > This is scary, and it potentially affects services outside of the
>>> > > Hadoop
>>> > > codebase. This makes it difficult to assess the impact.
>>> >
>>> > Stack mentioned a compatibility mode that uses the proto2 semantics.
>>> > If that carries unknown fields through intermediate handlers, then
>>> > this objection goes away. -C
>>>
>>>
>>> Did some more googling, found this:
>>>
>>> https://groups.google.com/d/msg/protobuf/Z6pNo81FiEQ/fHkdcNtdAwAJ
>>>
>>> Feng Xiao appears to be a Google engineer, and suggests workarounds like
>>> packing the fields into a byte type. No mention of a PB2 compatibility
>>> mode. Also here:
>>>
>>> 

Re: Can we update protobuf's version on trunk?

[Chris Douglas]

On Tue, Mar 28, 2017 at 4:18 PM, Andrew Wang  wrote:
> Unfortunately, it sounds like these are intrinsic differences with PB3.

That's too bad... but possibly not fatal: most of the data we proxy
through client code is, if not opaque, it's at least immutable
(particularly tokens). If PB3 does support reading valid PB fields as
bytes, then we could proxy the payload through application code as an
opaque blob. That opacity has a drawback: if clients could use that
information (e.g., StorageType), we'd need to include it in a
redundant field.

Ewan Higgs used a technique in HDFS-11026 [1] to handle a transition
from Writable to protobuf. This probably could be used for most of our
token types. It's not a general solution, but it would be sufficient
for existing applications to continue working, with some accommodation
for proxy versioning and rolling upgrades.

I haven't seen data identifying PB as a bottleneck, but the
non-x86/non-Linux and dev setup arguments may make this worthwhile. -C

[1] https://issues.apache.org/jira/browse/HDFS-11026

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: Can we update protobuf's version on trunk?

[Chris Douglas]

On Tue, Mar 28, 2017 at 3:04 PM, Andrew Wang  wrote:
> There's no mention of the convenient "Embedded messages are compatible with
>> bytes if the bytes contain an encoded version of the message" semantics in
>> proto3.
>
>
> I checked the proto3 guide, and I think this is supported:
> https://developers.google.com/protocol-buffers/docs/proto3#updating

You're right, it looks like this is supported.

> If unknown fields are dropped, then applications proxying tokens and other
>> data between servers will effectively corrupt those messages, unless we
>> make everything opaque bytes, which- absent the convenient, prenominate
>> semantics managing the conversion- obviate the compatibility machinery that
>> is the whole point of PB. Google is removing the features that justified
>> choosing PB over its alternatives. Since we can't require that our
>> applications compile (or link) against our updated schema, this creates a
>> problem that PB was supposed to solve.
>
>
> This is scary, and it potentially affects services outside of the Hadoop
> codebase. This makes it difficult to assess the impact.

Stack mentioned a compatibility mode that uses the proto2 semantics.
If that carries unknown fields through intermediate handlers, then
this objection goes away. -C

> Paraphrasing, the issues with PB2.5 are:
>
>1. poor support for non-x86, non-Linux platforms
>2. not as available, so harder to setup a dev environment
>3. missing zero-copy support, which helped performance in HBase
>
> #1 and #2 can be addressed if we rehosted PB (with cross-OS compilation
> patches) elsewhere.
> #3 I don't think we benefit from, since we don't pass around big PB byte
> arrays (at least in HDFS).
>
> So the way I see it, upgrading to PB3 has risk from the behavior change wrt
> unknown fields, while there are other ways of addressing the stated issues
> with PB2.5.
>
> Best,
> Andrew

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: Understanding task commit/abort protocol

[Chris Douglas]

IIRC the call to commit is correct, if the OutputCommitter supports
partial commit. That was the idea: checkpoint the state of the reducer
and promote its output, so it picks up from that point when
rescheduled. It worked well in our experiments.

This also included changes that allow intermediate output to be
written to non-local FileSystems, so the reduce could stream output
instead of first copying/merging locally. It significantly improved
runtimes for small/medium jobs (around 30%), though naively enabling
it could DDoS a NN. It made checkpoints very inexpensive.

Anyway- I'm unsure what to do with this. Augusto Souza picked up the
code a couple years ago, but we couldn't sync up to merge the code
with the encrypted shuffle and polish a couple rough points [1].
There's little risk in merging it, since at worst it's never enabled,
but we wanted to make sure it was hardened enough for production.

Is there interest in this? AFAIK MapReduce jobs are still very common
in production clusters, preemption is (now) often enabled, and it'd be
an interesting example for other YARN applications. However, we don't
have cycles to redo this work on trunk. We can offer review/guidance,
if someone wanted to dig into the MR pipeline and complete it. -C


[1] e.g., writing the task attempt ID into the IFile header. This
anticipated aggregation trees, but in practice is used only for
preemption during the shuffle phase. This isn't wrong, just verbose.

On Wed, Feb 8, 2017 at 6:52 AM, Steve Loughran <ste...@hortonworks.com> wrote:
>
>> On 3 Feb 2017, at 20:02, Chris Douglas <chris.doug...@gmail.com> wrote:
>>
>> It's been a long time, but IIRC this isn't going to be invoked. The AM
>> will never set the preempt flag in the umbilical, so the task will
>> never transition to this state.
>>
>> MapReduce checkpoint/restart of reduce tasks was going to be part of
>> MAPREDUCE-5269, which signals a ReduceTask to promote its partial
>> output if both the Reducer and OutputCommitter are tagged as
>> @Checkpointable. If either is not, then the flag is never set. The
>> code that would have implemented this was not committed, so it's
>> really-really not going to be set. -C
>
> I didn't think it was being used, but thanks for clarifying this.
>
> Should that code snippet be culled? Or at least the abort operation to 
> actually call abortTask?
>
>>
>> On Fri, Feb 3, 2017 at 6:41 AM, Steve Loughran <ste...@hortonworks.com> 
>> wrote:
>>>
>>> In HADOOP-13786 I'm adding a new committer, one which writes to S3 without 
>>> doing renames. It does this by submitting all the data to S3 targeted at 
>>> the final destination, but doesn't send the POST needed to materialize it 
>>> until the tasks commits. Abort the task and it cancels these pending 
>>> commits.
>>>
>>> this algorithm should be robust provided that only one attempt for a task 
>>> is committed, which comes down to
>>>
>>> 1.  Only those tasks which have succeeded are committed
>>> 2   those tasks which have not succeeded have their pending writes aborted
>>>
>>>
>>> Which is where I now have a question. In the class 
>>> org.apache.hadoop.mapred.Task, OutputCommitter.commitTask() is called when 
>>> a task is pre-empted:
>>>
>>>
>>>  public void done(TaskUmbilicalProtocol umbilical,
>>>   TaskReporter reporter
>>>   ) throws IOException, InterruptedException {
>>>updateCounters();
>>>if (taskStatus.getRunState() == TaskStatus.State.PREEMPTED ) {
>>>  // If we are preempted, do no output promotion; signal done and exit
>>>  committer.commitTask(taskContext); / * HERE */
>>>  umbilical.preempted(taskId, taskStatus);
>>>  taskDone.set(true);
>>>  reporter.stopCommunicationThread();
>>>  return;
>>>}
>>>
>>> That's despite the line above saying "do no output promotion", and, judging 
>>> by its place in the code, looking like it's the handler for task preempted 
>>> state.
>>>
>>> Shouldn't it be doing a task abort here?
>>>
>>> I suspect the sole reason this hasn't shown up as a problem before is that 
>>> this is the sole use of TaskStatus.State.PREEMPTED in the hadoop code: this 
>>> particular codepath is never executed. In which case, culling it may be 
>>> correct option.
>>>
>>> Thoughts?
>>>
>>> -Steve
>>>
>>> -
>>> To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
>>> For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
>>>
>>
>

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0-alpha2 RC0

[Chris Douglas]

On Wed, Jan 25, 2017 at 11:42 AM, Andrew Wang  wrote:
> Chris and Karthik, could you clarify the contingency of your votes? Is
> fixing just the release notes sufficient?

My +1 was not contingent on any changes.

The release is fine as-is. Fixing any subset of the release notes,
minicluster jar, and the organization of LICENSE files within jars
should not reset the clock on the VOTE. -C

> On Wed, Jan 25, 2017 at 11:14 AM, Karthik Kambatla 
> wrote:
>
>> Thanks for driving the alphas, Andrew. I don't see the need to restart the
>> vote and I feel it is okay to fix the minor issues before releasing.
>>
>> +1 (binding). Downloaded source, stood up a pseudo-distributed cluster
>> with FairScheduler, ran example jobs, and played around with the UI.
>>
>> Thanks
>> Karthik
>>
>>
>> On Fri, Jan 20, 2017 at 2:36 PM, Andrew Wang 
>> wrote:
>>
>>> Hi all,
>>>
>>> With heartfelt thanks to many contributors, the RC0 for 3.0.0-alpha2 is
>>> ready.
>>>
>>> 3.0.0-alpha2 is the second alpha in the planned 3.0.0 release line leading
>>> up to a 3.0.0 GA. It comprises 857 fixes, improvements, and new features
>>> since alpha1 was released on September 3rd, 2016.
>>>
>>> More information about the 3.0.0 release plan can be found here:
>>>
>>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0.0+release
>>>
>>> The artifacts can be found here:
>>>
>>> http://home.apache.org/~wang/3.0.0-alpha2-RC0/
>>>
>>> This vote will run 5 days, ending on 01/25/2017 at 2PM pacific.
>>>
>>> I ran basic validation with a local pseudo cluster and a Pi job. RAT
>>> output
>>> was clean.
>>>
>>> My +1 to start.
>>>
>>> Thanks,
>>> Andrew
>>>
>>
>>

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0-alpha2 RC0

[Chris Douglas]

On Mon, Jan 23, 2017 at 9:32 PM, Allen Wittenauer
 wrote:
> The problem here is that there is a 'license' directory and a file called 
> 'LICENSE'.  If this gets extracted by jar via jar xf, it will fail.  unzip 
> can be made to extract it via an option like -o.  To make matters worse, none 
> of these license files match the one in the generated tarball. :(

Ah, got it. I didn't strip the trailing slash on directories. With
that, it looks like the "license" directory and "LICENSE" file are the
only conflict?

I've not followed the development of packaging LICENSE/NOTICE in the
jar files. AFAIK, it's sufficient that we have the top-level
LICENSE/NOTICE in the tarball. Unless there's a LEGAL thread to the
contrary, it's OK as-is.

Again, I don't think we need to restart the clock on the RC vote if
the release notes and LICENSE/NOTICE were fixed, but it's Andrew's
time and I don't think any of these are blockers for the release. -C

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0-alpha2 RC0

[Chris Douglas]

Thanks for all your work on this, Andrew. It's great to see the 3.x
series moving forward.

If you were willing to modify the release notes and add the LICENSE to
the jar, we don't need to reset the clock on the VOTE, IMO.

What's the issue with the minicluster jar [1]? I tried to reproduce,
but had no issues with 1.8.0_92-b14.

+1 Verified checksums, signature, built src tarball. -C

[1] 
hadoop-3.0.0-alpha2/share/hadoop/client/hadoop-client-minicluster-3.0.0-alpha2.jar


On Mon, Jan 23, 2017 at 10:51 AM, Andrew Wang  wrote:
> There are 5 JIRAs by my count that are missing release notes, which isn't
> that bad but could of course be improved. Four of those I had already
> checked earlier and forgot to follow up, and they were very minorly
> incompatible (affecting private APIs) or mistakenly marked incompatible.
>
> I'm not too concerned about the shaded minicluster since it's a new
> feature, this is an alpha, and we have an IT test against the shaded
> minicluster. Multiple files with the same name are actually also allowed by
> the zip standard, so it's not clear if there is a functionality bug.
>
> Could I get some additional PMC input on this vote? The most critical issue
> in my mind is the missing LICENSE on that one new artifact. If we end up
> spinning a new RC, I'll also handle the missing release notes that Allen
> mentioned.
>
> Thanks,
> Andrew
>
> On Sun, Jan 22, 2017 at 10:45 PM, Allen Wittenauer > wrote:
>
>>
>> > On Jan 22, 2017, at 9:05 PM, Allen Wittenauer 
>> wrote:
>> >
>> >
>> >
>> >
>> >
>> >> On Jan 20, 2017, at 2:36 PM, Andrew Wang 
>> wrote:
>> >>
>> >> http://home.apache.org/~wang/3.0.0-alpha2-RC0/
>> >
>> >   There are quite a few JIRA issues that need release notes.
>> >
>>
>>
>> One other thing, before I forget... I'm not sure the
>> hadoop-client-minicluster jar is getting built properly.  If you look
>> inside, you'll find a real mishmash of things, including files and
>> directories with the same names but different cases.  This means it won't
>> extract properly on OS X.  (jar xf on that jar file literally stack traces
>> on my El Capitan machine. Neat!)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release cadence and EOL

[Chris Douglas]

On Fri, Jan 20, 2017 at 2:50 PM, Sangjin Lee  wrote:
> The security patch for the 2.6.x line is a case in point. Without any
> guideline, we would start with "What should we do for 2.6.x? Should we
> continue to patch it?" With this guideline, the baseline is already "it's
> been 2 years since 2.6.0 is released and we should consider stopping
> releasing from 2.6.x and encourage users to upgrade to 2.7.x."

Unless it falls under the "good reason" clause. To invent an example,
if 3.6.x were within the two year/minor release window, but 3.5.x was
more widely deployed/stable, then we'd use this escape hatch to patch
3.5.x and likely just violate our policy on 3.6.x (when the
implementation cost is high). How do we "force" a fix to 3.6.x?

We can't actually compel work from people. Even when we can point to a
prior consensus, someone needs to be motivated to actually complete
that task. That's the rub: this proposal doesn't only allow us to stop
working on old code, it also promises that we'll work on code we want
to abandon.

Pointing to a bylaw, and telling a contributor they "must" support a
branch/release isn't going to result in shorter discussions, either.
In the preceding hypothetical, if someone wants a fix in the 3.6 line,
they either need to convince others that it's important or they need
to do the work themselves.

> Actually the decision on security issues is a pretty strong indicator of our
> desire for EOL. If we say we do not want to patch that line for security
> vulnerability, then there would be even *less* rationale for fixing any
> other issue on that line. So the decision to stop backporting security
> patches is a sufficient indication of EOL in my mind.

Agreed. If we're not backporting security patches to a branch, then we
need to release a CVE, file a JIRA, and move on. If someone wants to
fix it and roll an RC for that release line, it lives. Otherwise it
dies as people move to later versions (unpatched security flaws are
motivating). A branch is EOL when we stop releasing from it. Two years
or two minor releases is a good heuristic based on recent history, but
overfitting policy to experience doesn't seem to buy us anything.

I'm all for spending less time discussing release criteria, but if
it's sufficient to observe which release lines are getting updates and
label them accordingly, that's cheaper to implement than a curated set
of constraints. -C

>> We can still embargo security flaws if someone asks (to give them time
>> time to implement a fix and call a vote). If there's nothing else in
>> the release, then we're effectively announcing it. In those cases, we
>> call a vote on private@ (cc: security@). -C
>>
>>
>> On Thu, Jan 19, 2017 at 1:30 PM, Andrew Wang 
>> wrote:
>> > I don't think the motivation here is vendor play or taking away power
>> > from
>> > committers. Having a regular release cadence helps our users understand
>> > when a feature will ship so they can plan their upgrades. Having an EOL
>> > policy and a minimum support period helps users choose a release line,
>> > and
>> > understand when they will need to upgrade.
>> >
>> > In the earlier thread, we discussed how these are not rules, but
>> > guidelines. There's a lot of flexibility if someone wants to keep
>> > maintaining a release line (particularly if they are willing to do the
>> > backporting work). More power to them; more releases are a good thing
>> > for
>> > the project.
>> >
>> > My main concern (which I raised on the earlier thread) is that without
>> > significant improvements to the release process and upstream integration
>> > testing, it's unlikely we'll actually ship more releases. Too often,
>> > branches are simply not in a releaseable state, or they have latent
>> > blocker
>> > bugs due to a lack of testing. This is what we've been struggling with
>> > on
>> > both the 2.8.x and 3.0.0-x release lines.
>> >
>> > So, in the abstract, I'm +1 on having a published policy on release
>> > cadence
>> > and EOL. This information helps users.
>> >
>> > However, I don't think we're ready to actually execute on this policy
>> > for
>> > the above reasons. This leaves me ambivalent overall, perhaps -0 since
>> > publishing a policy we don't follow is more confusing to users.
>> >
>> > My 2c,
>> > Andrew
>> >
>> >
>> >
>> > On Thu, Jan 19, 2017 at 12:28 PM, Arpit Agarwal
>> > 
>> > wrote:
>> >
>> >> The ASF release policy says releases may not be vetoed [1] so the EOL
>> >> policy sounds unenforceable. Not sure a release cadence is enforceable
>> >> either since Release Managers are volunteers.
>> >>
>> >> 1. https://www.apache.org/dev/release.html#approving-a-release
>> >>
>> >>
>> >>
>> >> On 1/18/17, 7:06 PM, "Junping Du"  wrote:
>> >>
>> >> +1 on Sangjin's proposal -
>> >> "A minor release line is end-of-lifed 2 years after it is released
>> >> or
>> >> there
>> >> are 2 newer minor 

Re: [VOTE] Release cadence and EOL

[Chris Douglas]

Sorry, I'd missed the end of the EOL discussion thread.

As several people have pointed out, this is unenforceable. The release
dates on the front page are a decent signal for liveness... do we need
something more formal? All these hypothetical situations would be
decided with more context. The "good reason" clause allows revive a
release line if two "live" branches straddle a dud, so this proposal
only commits us to maintain our mistakes. For two years? As Andrew
points out, while this heuristic usually holds, we're not set up to
enforce it.

We may want to have an informal policy for security issues, since
there are known holes in 2.6.x and earlier that aren't going to be
patched. We need to issue CVEs for those. A policy would simplify
tracking (e.g., announce vulnerabilities no more than a month after a
fix is available in a later release), so we don't wait indefinitely to
announce. Additionally, creating a JIRA and flagging the release on
the download page would be ample warning.

We can still embargo security flaws if someone asks (to give them time
time to implement a fix and call a vote). If there's nothing else in
the release, then we're effectively announcing it. In those cases, we
call a vote on private@ (cc: security@). -C


On Thu, Jan 19, 2017 at 1:30 PM, Andrew Wang  wrote:
> I don't think the motivation here is vendor play or taking away power from
> committers. Having a regular release cadence helps our users understand
> when a feature will ship so they can plan their upgrades. Having an EOL
> policy and a minimum support period helps users choose a release line, and
> understand when they will need to upgrade.
>
> In the earlier thread, we discussed how these are not rules, but
> guidelines. There's a lot of flexibility if someone wants to keep
> maintaining a release line (particularly if they are willing to do the
> backporting work). More power to them; more releases are a good thing for
> the project.
>
> My main concern (which I raised on the earlier thread) is that without
> significant improvements to the release process and upstream integration
> testing, it's unlikely we'll actually ship more releases. Too often,
> branches are simply not in a releaseable state, or they have latent blocker
> bugs due to a lack of testing. This is what we've been struggling with on
> both the 2.8.x and 3.0.0-x release lines.
>
> So, in the abstract, I'm +1 on having a published policy on release cadence
> and EOL. This information helps users.
>
> However, I don't think we're ready to actually execute on this policy for
> the above reasons. This leaves me ambivalent overall, perhaps -0 since
> publishing a policy we don't follow is more confusing to users.
>
> My 2c,
> Andrew
>
>
>
> On Thu, Jan 19, 2017 at 12:28 PM, Arpit Agarwal 
> wrote:
>
>> The ASF release policy says releases may not be vetoed [1] so the EOL
>> policy sounds unenforceable. Not sure a release cadence is enforceable
>> either since Release Managers are volunteers.
>>
>> 1. https://www.apache.org/dev/release.html#approving-a-release
>>
>>
>>
>> On 1/18/17, 7:06 PM, "Junping Du"  wrote:
>>
>> +1 on Sangjin's proposal -
>> "A minor release line is end-of-lifed 2 years after it is released or
>> there
>> are 2 newer minor releases, whichever is sooner. The community
>> reserves the
>> right to extend or shorten the life of a release line if there is a
>> good
>> reason to do so."
>>
>> I also noticed Karthik bring up some new proposals - some of them
>> looks interesting to me and I have some ideas as well. Karthik, can you
>> bring it out in a separated discussion threads so that we can discuss from
>> there?
>>
>> About Chris Trezzo's question about definition of EOL of hadoop
>> release, I think potentially changes could be:
>> 1. For users of Apache hadoop, they would expect to upgrade to a new
>> minor/major releases after EOL of their current release because there is no
>> guarantee of new maintenance release.
>>
>> 2. For release effort, apache law claim that committer can volunteer
>> RM for any release. With this release EOL proposal passes and written into
>> hadoop bylaw, anyone want to call for a release which is EOL then she/he
>> have to provide a good reason to community and get voted before to start
>> release effort. We don't want to waste community time/resource to
>> verify/vote a narrow interested release.
>>
>> 3. About committer's responsibility, I think the bottom line is
>> committer should commit patch contributor's target release and her/his own
>> interest release which I conservatively agree with Allen's point that this
>> vote doesn't change anything. But if a committer want to take care more
>> interest from the whole community like most committers are doing today,
>> he/she should understand which branches can benefit more people and could
>> skip some EOL release branches for backport 

Re: [VOTE] Release Apache Hadoop 2.6.5 (RC0)

[Chris Douglas]

+1 Verified checksum and signature. Unpacked the jar, started
single-node HDFS cluster, did some cursory checks.

Read through the commit log from 2.6.4; particularly happy to see
HADOOP-12893. -C

On Tue, Sep 27, 2016 at 1:28 PM, Sangjin Lee  wrote:
> Hi folks,
>
> I have created a release candidate RC0 for the Apache Hadoop 2.6.5 release
> (the next maintenance release in the 2.6.x release line). Below are the
> details of this release candidate:
>
> The RC is available for validation at:
> http://home.apache.org/~sjlee/hadoop-2.6.5-RC0/.
>
> The RC tag in git is release-2.6.5-RC0 and its git commit is
> 6939fc935fba5651fdb33386d88aeb8e875cf27a.
>
> The maven artifacts are staged via repository.apache.org at:
> https://repository.apache.org/content/repositories/orgapachehadoop-1048/.
>
> You can find my public key at
> http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS.
>
> Please try the release and vote. The vote will run for the usual 5 days.
> Huge thanks to Chris Trezzo for spearheading the release management and
> doing all the work!
>
> Thanks,
> Sangjin

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Merge HADOOP-13341

[Chris Douglas]

+1 (also on JIRA) -C

On Wed, Sep 7, 2016 at 6:44 AM, Allen Wittenauer
 wrote:
>
> I’d like to call for a vote to run for 5 days (ending  Mon 12, 2016 
> at 7AM PT) to merge the HADOOP-13341 feature branch into trunk. This branch 
> was developed exclusively by me.  As usual with large shell script changes, 
> it's been broken up into several smaller commits to make it easier to read.  
> The core of the functionality is almost entirely in hadoop-functions.sh with 
> the majority of the rest of the new additions either being documentation or 
> test code. In addition, large swaths of code is removed from the hadoop, 
> hdfs, mapred, and yarn executables.
>
> Here's a quick summary:
>
> * makes the rules around _OPTS consistent across all the projects
> * makes it possible to provide custom _OPTS for every hadoop, hdfs, mapred, 
> and yarn subcommand
> * with the exception of deprecations, removes all of the custom daemon _OPTS 
> handling sprinkled around the hadoop, hdfs, mapred, and yarn subcommands
> * removes the custom handling handling of HADOOP_CLIENT_OPTS and makes it 
> consistent for non-daemon subcommands
> * makes the _USER blocker consistent with _OPTS as well as providing better 
> documentation around this feature's existence.  Note that this is an 
> incompatible change against -alpha1.
> * by consolidating all of this code, makes it possible to finally fix a good 
> chunk of the "directory name containing spaces blows up the bash code" 
> problems that's been around since the beginning of the project
>
> Thanks!
>
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [DISCUSS] Release numbering semantics with concurrent (>2) releases [Was Setting JIRA fix versions for 3.0.0 releases]

[Chris Douglas]

> I'm certainly open to alternate proposals for versioning and fix versions,
> but to reiterate, I like this versioning since it imitates other enterprise
> software. RHEL has versions like 6.2 Beta 2 and 7.0 Beta, so versions like
> 3.0.0-alpha1 will be immediately familiar to end users. Conversely, I've
> met people who were confused by the 2.0/2.1/2.2 progression. As far as I
> know, we were unique in using this style of versioning.

Yes, but even after a stable release, we haven't blocked new (or
incompatible) features in minor releases. Some minor releases- 0.16,
0.21, 2.6.0- included complicated features that took awhile to
stabilize. Most of our x.y.0 releases are not stable, because
downstream reports come back only after we cut a release. Signaling
stability is useful, but this identifier is not reliable. At least,
it's less reliable than a section on the website recommending the
latest stable release beside alpha/beta versions.

Anyway- if we do use this, then we can use Maven's schema:
..--

Though I hope we won't need it, starting with 3.0.0-alpha-01 will
avoid -alpha10 as preceding -alpha2. We've discussed it enough; I'll
let it go.

> I also think what we're doing now *is* considered releasing from trunk.
> Maybe I'm missing something, but we can't literally release from trunk
> without a release branch (e.g. branch-3.0.0-alpha1) unless we hold commits
> until the release or change fix versions afterwards.

We're not constrained on where we cut releases. If subsequent releases
will be off of trunk- not the -alpha branch- and we aren't
removing/holding any change until stabilizing at -beta, then we don't
need to maintain a separate release branch concurrent with development
on trunk. Bumping the release version, etc. might be useful, but a
living branch just mixes up the lineage and adds steps to commit
(trunk > 3.0.0-alpha > branch-2 > branch-2.8 > ...). If it's easier
for you to create the RC then OK. -C

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: Why there are so many revert operations on trunk?

[Chris Douglas]

Reading through HDFS-9924, a request for a design doc- and a -1 on
committing to trunk- was raised in mid-May, but commits to trunk
continued. Why is that? Shouldn't this have paused while the details
were discussed? Branching is neutral to the pace of feature
development, but consensus on the result is required. Working through
possibilities in a branch- or in multiple branches- seems like a
reasonable way to determine which approach has support and code to
back it.

Reverting code is not "illegal"; the feature will be in/out of any
release by appealing to bylaws. Our rules exist to facilitate
consensus, not declare it a fiat accompli.

An RM only exists by creating an RC. Someone can declare themselves
Grand Marshall of trunk and stomp around in a fancy hat, but it
doesn't affect anything. -C


On Mon, Jun 6, 2016 at 9:36 AM, Junping Du  wrote:
> Thanks Aaron for pointing it out. I didn't see any consensus on HDFS-9924 so 
> I think we should bring it here with broader audiences for more discussions.
>
> I saw several very bad practices here:
>
> 1. committer (no need to say who) revert all commits from trunk without 
> making consensus with all related contributors/committers.
>
> 2. Someone's comments on feature branch are very misleading... If I didn't 
> remember wrong, feature development doesn't have to go through feature branch 
> which is just an optional process. This creative process of feature branch 
> and branch committer - I believe the intention is trying to accelerate 
> features development but not to slow them down.
>
> 3. Someone (again, no need to say who) seems to claim himself as RM for 
> trunk. I don't think we need any RM for trunk. Even for RM of 3.0.0-alpha, I 
> think we need someone else who demonstrates he/she is more responsible, work 
> hardly and carefully and open communication with all community. Only through 
> this, the success of Hadoop in age of 3.0 are guranteed.
>
>
> Thanks,
>
>
> Junping
>
>
> 
> From: Aaron T. Myers 
> Sent: Monday, June 06, 2016 4:46 PM
> To: Junping Du
> Cc: Andrew Wang; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
> mapreduce-dev@hadoop.apache.org; yarn-...@hadoop.apache.org
> Subject: Re: Why there are so many revert operations on trunk?
>
> Junping,
>
> All of this is being discussed on HDFS-9924. Suggest you follow the 
> conversation there.
>
> --
> Aaron T. Myers
> Software Engineer, Cloudera
>
> On Mon, Jun 6, 2016 at 7:20 AM, Junping Du 
> > wrote:
> Hi Andrew,
>
>  I just noticed you revert 8 commits on trunk last Friday:
>
> HADOOP-13226
>
> HDFS-10430
>
> HDFS-10431
>
> HDFS-10390
>
> HADOOP-13168
>
> HDFS-10390
>
> HADOOP-13168
>
> HDFS-10346
>
> HADOOP-12957
>
> HDFS-10224
>
>And I didn't see you have any comments on JIRA or email discussion before 
> you did this. I don't think we are legally allowed to do this even as 
> committer/PMC member. Can you explain what's your intention to do this?
>
>BTW, thanks Nicolas to revert all these "illegal" revert operations.
>
>
>
> Thanks,
>
>
> Junping
>

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: Looking to a Hadoop 3 release

[Chris Douglas]

If we're not starting branch-3/trunk, what would distinguish it from
trunk/trunk-incompat? Is it the same mechanism with different labels?

That may be a reasonable strategy when we create branch-3, as a
release branch for beta. Releasing 3.x from trunk will help us figure
out which incompatibilities can be called out in an upgrade guide
(e.g., "new feature X is incompatible with uncommon configuration Y")
and which require code changes (e.g., "data loss upgrading a cluster
with feature X"). Given how long trunk has been unreleased, we need
more data from deployments to triage. How to manage transitions
between major versions will always be case-by-case; consensus on how
we'll address generic incompatible changes is not saving any work.

Once created, removing functionality from branch-3 (leaving it in
trunk) _because_ nobody volunteers cycles to address urgent
compatibility issues is fair. It's also more workable than asking that
features be committed to a branch that we have no plan to release,
even as alpha. -C

On Fri, Apr 22, 2016 at 6:50 PM, Vinod Kumar Vavilapalli
 wrote:
> Tx for your replies, Andrew.
>
>>> For exit criteria, how about we time box it? My plan was to do monthly
>> alphas through the summer, leading up to beta in late August / early Sep.
>> At that point we freeze and stabilize for GA in Nov/Dec.
>
>
> Time-boxing is a reasonable exit-criterion.
>
>
>> In this case, does trunk-incompat essentially become the new trunk? Or are
>> we treating trunk-incompat as a feature branch, which periodically merges
>> changes from trunk?
>
>
> It’s the later. Essentially
>  - trunk-incompat = trunk + only incompatible changes, periodically kept 
> up-to-date to trunk
>  - trunk is always ready to ship
>  - and no compatible code gets left behind
>
> The reason for my proposal like this is to address the tension between “there 
> is lot of compatible code in trunk that we are not shipping” and “don’t ship 
> trunk, it has incompatibilities”. With this, we will not have (compatible) 
> code not getting shipped to users.
>
> Obviously, we can forget about all of my proposal completely if everyone puts 
> in all compatible code into branch-2 / branch-3 or whatever the main 
> releasable branch is. This didn’t work in practice, have seen this not 
> happening prominently during 0.21, and now 3.x.
>
> There is another related issue - "my feature is nearly ready, so I’ll just 
> merge it into trunk as we don’t release that anyways, but not the current 
> releasable branch - I’m lazy to fix the last few stability related issues”. 
> With this, we will (should) get more disciplined, take feature stability on a 
> branch seriously and merge a feature branch only when it is truly ready!
>
>> For 3.x, my strawman was to release off trunk for the alphas, then branch a
>> branch-3 for the beta and onwards.
>
>
> Repeating above, I’m proposing continuing to make GA 3.x releases also off of 
> trunk! This way only incompatible changes don’t get shipped to users - by 
> design! Eventually, trunk-incompat will be latest 3.x GA + enough 
> incompatible code to warrant a 4.x, 5.x etc.
>
> +Vinod

Re: [DISCUSS] Looking to a 2.8.0 release

[Chris Douglas]

With two active sustaining branches (2.6, 2.7), what would you think
of releasing trunk as 3.x instead of pushing 2.8? There are many new
features (EC, Y1197, etc.), and trunk could be the source of several
alpha/beta releases before we fork the 3.x line. -C

On Sat, Sep 26, 2015 at 12:49 PM, Vinod Vavilapalli
 wrote:
> As you may have noted, 2.8.0 got completely derailed what with 2.7.x and the 
> unusually long 2.6.1 release.
>
> With 2.6.1 out of the way, and two parallel threads in progress for 2.6.2 and 
> 2.7.2, it’s time for us to look back at where we are with Hadoop 2.8.
>
> I’ll do a quick survey of where the individual features are and the amount of 
> content already present in 2.8 and kick-start 2.8.0 process again.
>
> +Vinod
>
>
>> On Apr 21, 2015, at 2:39 PM, vino...@apache.org wrote:
>>
>> With 2.7.0 out of the way, and with more maintenance releases to stabilize 
>> it, I propose we start thinking about 2.8.0.
>>
>> Here's my first cut of the proposal, will update the Roadmap wiki.
>>  - Support *both* JDK7 and JDK8 runtimes: HADOOP-11090
>>  - Compatibility tools to catch backwards, forwards compatibility issues at 
>> patch submission, release times. Some of it is captured at YARN-3292. This 
>> also involves resurrecting jdiff (HADOOP-11776/YARN-3426/MAPREDUCE-6310) 
>> and/or investing in new tools.
>>  - HADOOP-11656 Classpath isolation for downstream clients
>>  - Support for Erasure Codes in HDFS HDFS-7285
>>  - Early work for disk and network isolation in YARN: YARN-2139, YARN-2140
>>  - YARN Timeline Service Next generation: YARN-2928. At least branch-merge + 
>> early peek.
>>  - Supporting non-exclusive node-labels: YARN-3214
>>
>> I'm experimenting with more agile 2.7.x releases and would like to continue 
>> the same by volunteering as the RM for 2.8.x too.
>>
>> Given the long time we took with 2.7.0, the timeline I am looking at is 8-12 
>> weeks. We can pick as many features as they finish along and make a more 
>> predictable releases instead of holding up releases for ever.
>>
>> Thoughts?
>>
>> Thanks
>> +Vinod
>

Re: Looking to a Hadoop 3 release

[Chris Douglas]

On Fri, Mar 6, 2015 at 4:32 PM, Vinod Kumar Vavilapalli
vino...@hortonworks.com wrote:
 I'd encourage everyone to post their wish list on the Roadmap wiki that 
 *warrants* making incompatible changes forcing us to go 3.x.

This is a useful exercise, but not a prerequisite to releasing 3.0.0
as an alpha off of trunk, right? Andrew summarized the operating
assumptions for anyone working on it: rolling upgrades still work,
wire compat is preserved, breaking changes may get rolled back when
branch-3 is in beta (so be very conservative, notify others loudly).
This applies to branches merged to trunk, also.

 +1 to Jason's comments on general. We can keep rolling alphas that downstream 
 can pick up, but I'd also like us to clarify the exit criterion for a GA 
 release of 3.0 and its relation to the life of 2.x if we are going this 
 route. This brings us back to the roadmap discussion, and a collective 
 agreement about a logical step at a future point in time where we say we have 
 enough incompatible features in 3.x that we can stop putting more of them and 
 start stabilizing it.

We'll have this discussion again. We don't need to reach consensus on
the roadmap, just that each artifact reflects the output of the
project.

 Irrespective of that, here is my proposal in the interim:
  - Run JDK7 + JDK8 first in a compatible manner like I mentioned before for 
 atleast two releases in branch-2: say 2.8 and 2.9 before we consider taking 
 up the gauntlet on 3.0.
  - Continue working on the classpath isolation effort and try making it as 
 compatible as is possible for users to opt in and migrate easily.

+1 for 2.x, but again I don't understand the sequencing. -C

 On Mar 5, 2015, at 1:44 PM, Jason Lowe jl...@yahoo-inc.com.INVALID wrote:

 I'm OK with a 3.0.0 release as long as we are minimizing the pain of 
 maintaining yet another release line and conscious of the incompatibilities 
 going into that release line.
 For the former, I would really rather not see a branch-3 cut so soon.  It's 
 yet another line onto which to cherry-pick, and I don't see why we need to 
 add this overhead at such an early phase.  We should only create branch-3 
 when there's an incompatible change that the community wants and it should 
 _not_ go into the next major release (i.e.: it's for Hadoop 4.0).  We can 
 develop 3.0 alphas and betas on trunk and release from trunk in the interim. 
  IMHO we need to stop treating trunk as a place to exile patches.

 For the latter, I think as a community we need to evaluate the benefits of 
 breaking compatibility against the costs of migrating.  Each time we break 
 compatibility we create a hurdle for people to jump when they move to the 
 new release, and we should make those hurdles worth their time.  For 
 example, wire-compatibility has been mentioned as part of this.  Any feature 
 that breaks wire compatibility better be absolutely amazing, as it creates a 
 huge hurdle for people to jump.
 To summarize:+1 for a community-discussed roadmap of what we're breaking in 
 Hadoop 3 and why it's worth it for users
 -1 for creating branch-3 now, we can release from trunk until the next 
 incompatibility for Hadoop 4 arrives
 +1 for baking classpath isolation as opt-in on 2.x and eventually default on 
 in 3.0
 Jason
  From: Andrew Wang andrew.w...@cloudera.com
 To: hdfs-...@hadoop.apache.org hdfs-...@hadoop.apache.org
 Cc: common-...@hadoop.apache.org common-...@hadoop.apache.org; 
 mapreduce-dev@hadoop.apache.org mapreduce-dev@hadoop.apache.org; 
 yarn-...@hadoop.apache.org yarn-...@hadoop.apache.org
 Sent: Wednesday, March 4, 2015 12:15 PM
 Subject: Re: Looking to a Hadoop 3 release

 Let's not dismiss this quite so handily.

 Sean, Jason, and Stack replied on HADOOP-11656 pointing out that while we
 could make classpath isolation opt-in via configuration, what we really
 want longer term is to have it on by default (or just always on). Stack in
 particular points out the practical difficulties in using an opt-in method
 in 2.x from a downstream project perspective. It's not pretty.

 The plan that both Sean and Jason propose (which I support) is to have an
 opt-in solution in 2.x, bake it there, then turn it on by default
 (incompatible) in a new major release. I think this lines up well with my
 proposal of some alphas and betas leading up to a GA 3.x. I'm also willing
 to help with 2.x release management if that would help with testing this
 feature.

 Even setting aside classpath isolation, a new major release is still
 justified by JDK8. Somehow this is being ignored in the discussion. Allen,
 historically the voice of the user in our community, just highlighted it as
 a major compatibility issue, and myself and Tucu have also expressed our
 very strong concerns about bumping this in a minor release. 2.7's bump is a
 unique exception, but this is not something to be cited as precedent or
 policy.

 Where does this resistance to a new major release stem from? As I've
 described 

Re: Looking to a Hadoop 3 release

[Chris Douglas]

On Mon, Mar 2, 2015 at 11:04 PM, Konstantin Shvachko
shv.had...@gmail.com wrote:
 2. If Hadoop 3 and 2.x are meant to exist together, we run a risk to
 manifest split-brain behavior again, as we had with hadoop-1, hadoop-2 and
 other versions. If that somehow beneficial for commercial vendors, which I
 don't see how, for the community it was proven to be very disruptive. Would
 be really good to avoid it this time.

Agreed; let's try to minimize backporting headaches. Pulling trunk 
branch-2  branch-2.x is already tedious. Adding a branch-3,
branch-3.x would be obnoxious.

 3. Could we release Hadoop 3 directly from trunk? With a proper feature
 freeze in advance. Current trunk is in the best working condition I've seen
 in years - much better, than when hadoop-2 was coming to life. It could
 make a good alpha.

+1 This sounds like a good approach. Marked as alpha, we can break
compatibility in minor versions. Stabilizing a beta can correspond
with cutting branch-3, since that will be winding down branch-2. This
shouldn't disrupt existing plans for branch-2.

However, this requires that committers not accumulate too much
compatibility debt in trunk. Undoing all that in branch-3 imposes a
burdensome tax. Scanning through Allen's diff: that doesn't appear to
be the case so far, but it recommends against developing features in
place on trunk. Just be considerate of users and developers who will
need to move from (and maintain) branch-2.

 I believe we can start planning 3.0 from trunk right after 2.7 is out.

If we're publishing a snapshot, we don't need too much planning. -C

 On Mon, Mar 2, 2015 at 3:19 PM, Andrew Wang andrew.w...@cloudera.com
 wrote:

 Hi devs,

 It's been a year and a half since 2.x went GA, and I think we're about due
 for a 3.x release.
 Notably, there are two incompatible changes I'd like to call out, that will
 have a tremendous positive impact for our users.

 First, classpath isolation being done at HADOOP-11656, which has been a
 long-standing request from many downstreams and Hadoop users.

 Second, bumping the source and target JDK version to JDK8 (related to
 HADOOP-11090), which is important since JDK7 is EOL in April 2015 (two
 months from now). In the past, we've had issues with our dependencies
 discontinuing support for old JDKs, so this will future-proof us.

 Between the two, we'll also have quite an opportunity to clean up and
 upgrade our dependencies, another common user and developer request.

 I'd like to propose that we start rolling a series of monthly-ish series of
 3.0 alpha releases ASAP, with myself volunteering to take on the RM and
 other cat herding responsibilities. There are already quite a few changes
 slated for 3.0 besides the above (for instance the shell script rewrite) so
 there's already value in a 3.0 alpha, and the more time we give downstreams
 to integrate, the better.

 This opens up discussion about inclusion of other changes, but I'm hoping
 to freeze incompatible changes after maybe two alphas, do a beta (with no
 further incompat changes allowed), and then finally a 3.x GA. For those
 keeping track, that means a 3.x GA in about four months.

 I would also like to stress though that this is not intended to be a big
 bang release. For instance, it would be great if we could maintain wire
 compatibility between 2.x and 3.x, so rolling upgrades work. Keeping
 branch-2 and branch-3 similar also makes backports easier, since we're
 likely maintaining 2.x for a while yet.

 Please let me know any comments / concerns related to the above. If people
 are friendly to the idea, I'd like to cut a branch-3 and start working on
 the first alpha.

 Best,
 Andrew

Re: [VOTE] Merge branch MAPREDUCE-2841 to trunk

[Chris Douglas]

+1

The change to the existing code is very limited and the perf is impressive. -C

On Fri, Sep 5, 2014 at 4:58 PM, Todd Lipcon t...@apache.org wrote:
 Hi all,

 As I've reported recently [1], work on the MAPREDUCE-2841 branch has
 progressed well and the development team working on it feels that it is
 ready to be merged into trunk.

 For those not familiar with the JIRA (it's a bit lengthy to read from start
 to finish!) the goal of this work is to build a native implementation of
 the map-side sort code. The native implementation's primary advantage is
 its speed: for example, terasort is 30% faster on a wall-clock basis and
 60% faster on a resource consumption basis. For clusters which make heavy
 use of MapReduce, this is a substantial improvement to their efficiency.
 Users may enable the feature by switching a single configuration flag, and
 it will fall back to the original implementation in cases where the native
 code doesn't support the configured features/types.

 The new work is entirely pluggable and off-by-default to mitigate risk. The
 merge patch itself does not modify even a single line of existing code: all
 necessary plug-points have already been committed to trunk for some time.

 Though we do not yet have a full +1 precommit Jenkins run on the JIRA,
 there are only a few small nits to fix before merge, so I figured that we
 could start the vote in parallel. Of course we will not merge until it has
 a positive precommit run.

 Though this branch is a new contribution to the Apache repository, it
 represents work done over several years by a large community of developers
 including the following:

 Binglin Chang
 Yang Dong
 Sean Zhong
 Manu Zhang
 Zhongliang Zhu
 Vincent Wang
 Yan Dong
 Cheng Lian
 Xusen Yin
 Fangqin Dai
 Jiang Weihua
 Gansha Wu
 Avik Dey

 The vote will run for 7 days, ending Friday 9/12 EOD PST.

 I'll start the voting with my own +1.

 -Todd

 [1]
 http://search-hadoop.com/m/09oay13EwlV/native+task+progresssubj=Native+task+branch+progress

Re: [VOTE] Migration from subversion to git for version control

[Chris Douglas]

+1 -C

On Fri, Aug 8, 2014 at 7:57 PM, Karthik Kambatla ka...@cloudera.com wrote:
 I have put together this proposal based on recent discussion on this topic.

 Please vote on the proposal. The vote runs for 7 days.

1. Migrate from subversion to git for version control.
2. Force-push to be disabled on trunk and branch-* branches. Applying
changes from any of trunk/branch-* to any of branch-* should be through
git cherry-pick -x.
3. Force-push on feature-branches is allowed. Before pulling in a
feature, the feature-branch should be rebased on latest trunk and the
changes applied to trunk through git rebase --onto or git cherry-pick
commit-range.
4. Every time a feature branch is rebased on trunk, a tag that
identifies the state before the rebase needs to be created (e.g.
tag_feature_JIRA-2454_2014-08-07_rebase). These tags can be deleted once
the feature is pulled into trunk and the tags are no longer useful.
5. The relevance/use of tags stay the same after the migration.

 Thanks
 Karthik

 PS: Per Andrew Wang, this should be a Adoption of New Codebase kind of
 vote and will be Lazy 2/3 majority of PMC members.

Re: [VOTE] Change by-laws on release votes: 5 days instead of 7

[Chris Douglas]

+1 -C

On Tue, Jun 24, 2014 at 1:53 AM, Arun C Murthy a...@hortonworks.com wrote:
 Folks,

  As discussed, I'd like to call a vote on changing our by-laws to change 
 release votes from 7 days to 5.

  I've attached the change to by-laws I'm proposing.

  Please vote, the vote will the usual period of 7 days.

 thanks,
 Arun

 

 [main]$ svn diff
 Index: author/src/documentation/content/xdocs/bylaws.xml
 ===
 --- author/src/documentation/content/xdocs/bylaws.xml   (revision 1605015)
 +++ author/src/documentation/content/xdocs/bylaws.xml   (working copy)
 @@ -344,7 +344,16 @@
  pVotes are open for a period of 7 days to allow all active
  voters time to consider the vote. Votes relating to code
  changes are not subject to a strict timetable but should be
 -made as timely as possible./p/li
 +made as timely as possible./p
 +
 + ul
 + li strongProduct Release - Vote Timeframe/strong
 +   pRelease votes, alone, run for a period of 5 days. All other
 + votes are subject to the above timeframe of 7 days./p
 + /li
 +   /ul
 +   /li
 +
 /ul
 /section
  /body
 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the reader
 of this message is not the intended recipient, you are hereby notified that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender immediately
 and delete it from your system. Thank You.

Re: [DISCUSS] sort improvement

[Chris Douglas]

The sort implementation is pluggable (see MAPREDUCE-2454,
MAPREDUCE-4049), so please feel free to fork and improve it. Selecting
a sort implementation based on job configuration (e.g.,
BinaryComparable keys) would allow for much more efficient and
specialized implementations. -C

On Thu, Mar 20, 2014 at 8:40 PM, 蒋达晟 dsjiang...@gmail.com wrote:
 I'd like to discuss about some improvements of the sort stage in current
 hadoop. There are no formal documents available right now, so I will just
 hope someone would be interested in it. If so I can give more detail
 information.



 To start with, I have conducted a series of experiments on version 2.2 and
 I will take the example job Terasort as a demonstration of my thoughts.



 Machine: 19 slave machines, 12-core E5-2420(1.9GHZ), 7 disks

 Configuration: jvm=-Xmx512m, io.sort.mb=300, make sure no extra spill on
 both map/reduce side

 Datasize: 1TB (1e12), 4000 map/reduce



 Baseline: version 2.2

 Result: around 109000 CPU time

 (The terasort job is optimized because I just want to know the costage of
 the framework, anyone redo the test using the Terasort job in the example
 would see a surprisingly higher CPU time because of the counter system)

 Sort takes about 45% percentage of the total CPU time.



 Improvement 0+1

 Result: around 78000 CPU time



 Improvement 0+1+2

 Result: around 63000 CPU time



 The improvement 0 is use SSE to accelerate IFile checksum. Currently java
 implementation costs too much CPU, but I believe it worth another discuss,
 so I don't further extend it. Let me know if it worth doing.



 Improvement 1 uses a reduce side sort, which the intermediate result
 contain unsorted data, and reduce receive the data and sort it before the
 reduce function.

 The reason it saves some CPU is based on a fact that quick sort is
 efficient than heap-based merge sort. It saves comparison count, which in
 some circumstance is a direct reflection of the CPU cost.

 For large job with thousands of reduces, each map will send little data to
 reduce side. So the majority work of sort stage is at reduce side. In our
 case, each map sort approximately 62500 bytes use quick sort, but each
 reduce has to merge 4000 segments.

 There are several drawbacks of using a reduce side sort.

 First of all, it requires a rewrite of the whole data path. Although it is
 not much work to write the code, a few strategies in current implementation
 will be re-considered in new implementation.

 Secondary, combiner would be tricky to implement. One choice is to use a
 in-memory hash table, which needs a precise track of the memory usage of
 Java object. Another choice is to use current data path if combiner is
 required. But maintaining two sepearate data path seems to cause much extra
 work in future.

 I have tried to use in-memory hash table as a choice, when selectivity is
 high (the intermediate key number is much smaller than the input key
 number), it is feasible and has certain performance gain. But when
 selectivity is low, I don't see any performance advantage. Anyway, I
 believe there is a way to implement a combiner with lower cost, I just
 don't have much thought on it right now.

 Last drawback is about memory usage. We still need a large buffer as
 io.sort.mb indicates at map side to avoid a small spill file. And we also
 need a buffer at reduce side to sort data into a large segment. I don't
 think it is a big problem, just to complete my thought.



 Improvement 2 is about the sorting algorithm. I know some issues have been
 promoted to improve it, but there is still a lot of space for further
 improvement.

 First of all, current MapOutputBuffer implementation use a single buffer to
 store data and index. It is an adaptive strategy for both long record and
 short record. But it's not a good choice for performance. Implementation
 use direct int[] for the 16-bytes index would cut the CPU cost by half in
 my experiment.

 But even the implementation of QuickSort is optimized, it will still be
 much slower than RadixSort. On a 250MB terasort data, RadixSort can be
 6.5x~10x faster in my test.

 One thing need to clarify is how to adapt RadixSort into current hadoop.
 The plain answer is that we can't replace QuickSort to RadixSort as a
 genral framework, but for the majority usage we can.

 The assumption here is that most of the key type we use in hadoop is
 IntWritable, Text or some other sub-classes of BinaryComparable. As long as
 we can get four bytes from the key share the same order of the original
 key, we can sort it with RadixSort and then use QuickSort on those equal
 four bytes. 
 MAPREDUCE-1639https://issues.apache.org/jira/browse/MAPREDUCE-1639
 MAPREDUCE-3235 https://issues.apache.org/jira/browse/MAPREDUCE-3235 has
 related discussion.

 Another important thing to point out is that if we don't combine
 improvement 2 with improvement 1, it will be less effective as the majority
 of the sort happens as reduce side using the low 

Re: [VOTE] Release Apache Hadoop 2.3.0

[Chris Douglas]

+1 (binding)

Verified checksum, signature. Built from src, poked at single-node
cluster, ran some unit tests. -C

On Tue, Feb 11, 2014 at 6:49 AM, Arun C Murthy a...@hortonworks.com wrote:
 Folks,

 I've created a release candidate (rc0) for hadoop-2.3.0 that I would like to 
 get released.

 The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.3.0-rc0
 The RC tag in svn is here: 
 https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.3.0-rc0

 The maven artifacts are available via repository.apache.org.

 Please try the release and vote; the vote will run for the usual 7 days.

 thanks,
 Arun

 PS: Thanks to Andrew, Vinod  Alejandro for all their help in various release 
 activities.
 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the reader
 of this message is not the intended recipient, you are hereby notified that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender immediately
 and delete it from your system. Thank You.

Re: [VOTE] Release Apache Hadoop 2.2.0

[Chris Douglas]

+1

Verified checksum and signature, built tarball, ran some unit tests. -C

On Mon, Oct 7, 2013 at 12:00 AM, Arun C Murthy a...@hortonworks.com wrote:
 Folks,

 I've created a release candidate (rc0) for hadoop-2.2.0 that I would like to 
 get released - this release fixes a small number of bugs and some 
 protocol/api issues which should ensure they are now stable and will not 
 change in hadoop-2.x.

 The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.2.0-rc0
 The RC tag in svn is here: 
 http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.2.0-rc0

 The maven artifacts are available via repository.apache.org.

 Please try the release and vote; the vote will run for the usual 7 days.

 thanks,
 Arun

 P.S.: Thanks to Colin, Andrew, Daryn, Chris and others for helping nail down 
 the symlinks-related issues. I'll release note the fact that we have disabled 
 it in 2.2. Also, thanks to Vinod for some heavy-lifting on the YARN side in 
 the last couple of weeks.





 --
 Arun C. Murthy
 Hortonworks Inc.
 http://hortonworks.com/



 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the reader
 of this message is not the intended recipient, you are hereby notified that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender immediately
 and delete it from your system. Thank You.

Re: [VOTE] Release Apache Hadoop 2.0.5-alpha (rc2)

[Chris Douglas]

+1

Checksum and signature match, ran some unit tests, checked diff
against 2.0.4-alpha.

Thanks for seeing this through, Cos. -C

On Mon, Jun 3, 2013 at 1:13 PM, Alejandro Abdelnur t...@cloudera.com wrote:
 +1 RC2. Verified MD5  signature, checked CHANGES.txt files, built,
 configured pseudo cluster, run a couple of sample jobs, tested HTTPFS.


 On Mon, Jun 3, 2013 at 12:51 PM, Konstantin Boudnik c...@apache.org wrote:

 I have rolled out release candidate (rc2) for hadoop-2.0.5-alpha.

 The difference between rc1 and rc2 is the optimistic release date is set
 for
 06/06/2013 in the CHANGES.txt files.

 The binary artifact is the same - there's no need to rebuild it. The maven
 artifacts are the same.

 The difference between the two RCs:

 svn diff \

 https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.5-alpha-rc1/\

 https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.5-alpha-rc2/

 New RC builds are uploaded to the web.
 The RC is available at:
 http://people.apache.org/~cos/hadoop-2.0.5-alpha-rc2/
 The RC tag in svn is here:
 http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.5-alpha-rc2

 I would like to extend the vote for another three days for it is such a
 minor
 change that doesn't affect anything but the recorded release date. Please
 cast your vote before 06/06/2013 5pm PDT.

 Thanks for your patience!
   Cos

 On Fri, May 31, 2013 at 09:27PM, Konstantin Boudnik wrote:
  All,
 
  I have created a release candidate (rc1) for hadoop-2.0.5-alpha that I
 would
  like to release.
 
  This is a stabilization release that includes fixed for a couple a of
 issues
  discovered in the testing with BigTop 0.6.0 release candidate.
 
  The RC is available at:
 http://people.apache.org/~cos/hadoop-2.0.5-alpha-rc1/
  The RC tag in svn is here:
 http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.5-alpha-rc1
 
  The maven artifacts will be available via repository.apache.org on Sat,
 June
  1st, 2013 at 2 pm PDT as outlined here
  http://s.apache.org/WKD
 
  Please try the release bits and vote; the vote will run for the 3 days,
  because this is just a version name change. The bits are identical to
 the ones
  voted on before in
  http://s.apache.org/2041move
 
  Thanks for your voting
Cos
 




 --
 Alejandro

Re: [VOTE] Release Apache Hadoop 0.23.8

[Chris Douglas]

+1

Checksum and signature match, ran some tests. -C

On Mon, Jun 3, 2013 at 1:19 PM, Sandy Ryza sandy.r...@cloudera.com wrote:
 +1 (non-binding).  Did a full build from source and ran a few sample jobs
 on a pseudo-distributed cluster.

 -Sandy


 On Mon, Jun 3, 2013 at 11:29 AM, Kihwal Lee kih...@yahoo-inc.com wrote:

 +1 I've downloaded  built the RC and ran several tests on a single node
 cluster.

 Kihwal

 On 5/28/13 11:00 AM, Thomas Graves tgra...@yahoo-inc.com wrote:

 
 I've created a release candidate (RC0) for hadoop-0.23.8 that I would like
 to release.
 
 This release is a sustaining release with several important bug fixes in
 it.  The most critical one is MAPREDUCE-5211.
 
 The RC is available at:
 http://people.apache.org/~tgraves/hadoop-0.23.8-candidate-0/
 The RC tag in svn is here:
 http://svn.apache.org/viewvc/hadoop/common/tags/release-0.23.8-rc0/
 
 The maven artifacts are available via repository.apache.org.
 
 Please try the release and vote; the vote will run for the usual 7 days.
 
 I am +1 (binding).
 
 thanks,
 Tom Graves
 

Re: [VOTE] Release Apache Hadoop 2.0.4.1-alpha

[Chris Douglas]

On Thu, May 30, 2013 at 10:57 AM, Arun C Murthy a...@hortonworks.com wrote:
 Why not include MAPREDUCE-4211 as well rather than create one release per 
 patch?

From Cos's description, it sounded like these were backports of fixes
to help Sqoop2 and fix some build issues. If it's not just to fixup
leftover bugs in 2.0.4 *once* so downstream projects can integrate
against 2.0.4.1, and this a release series, then I've completely
misunderstood the purpose.

Cos, are you planning 2.0.4.2?

 Also, this is the first time we are seeing a four-numbered scheme in Hadoop. 
 Why not call this 2.0.5-alpha?

Good point. Since it contains only backports from branch-2, it would
make sense for it to be an intermediate release.

I shouldn't have to say this, but I'm changing my vote to -1 while we
work this out. -C

 On May 24, 2013, at 8:48 PM, Konstantin Boudnik wrote:

 All,

 I have created a release candidate (rc0) for hadoop-2.0.4.1-alpha that I 
 would
 like to release.

 This is a stabilization release that includes fixed for a couple a of issues
 discovered in the testing with BigTop 0.6.0 release candidate.

 The RC is available at: 
 http://people.apache.org/~cos/hadoop-2.0.4.1-alpha-rc0/
 The RC tag in svn is here: 
 http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.4.1-alpha-rc0

 The maven artifacts are available via repository.apache.org.

 Please try the release bits and vote; the vote will run for the usual 7 days.

 Thanks for your voting
  Cos

Re: [VOTE] Release Apache Hadoop 2.0.4.1-alpha

[Chris Douglas]

On Thu, May 30, 2013 at 2:39 PM, Konstantin Boudnik c...@apache.org wrote:
 There's no misunderstanding Chris - this release is to unblock downstream.

 As for your question: I don't have a crystal ball; I wish though. I think the
 answer depends on will be there more blocking bugs found in the later releases
 of Bigtop or other downstream components.
 This is bugfix release and, I guess, if there are more bugs found in the
 future - more releases would have to be cut. Isn't this is why the software is
 being released?

Sure, but they're all backports from the release currently marked for
2.0.5. Either (a) these are really blocker bugs and we should roll a
patch release or (b) some bleeding-edge work needs to work around this
while branch-2 is released in the next few weeks. If it's not severe
enough to justify disrupting the versioning of snapshot maven
artifacts in branch-2, then we're clearly not in case (a).

I thought this was the result of extensive testing, and 2.0.4.1 was a
release to enable additional integration before 2.0.5. If we plan to
roll more releases as a subset of the bug fixes committed to branch-2
then just call it 2.0.5. Please make sure it- and any future,
intermediate release- is worth the disruption.

 Now, the -1: I am not clear about the justification. What exactly we expect to
 work out?

It's become fashionable to close threads and count votes in the middle
of the discussion. I changed my vote instead of trusting you. -C

 On Thu, May 30, 2013 at 01:48PM, Chris Douglas wrote:
 On Thu, May 30, 2013 at 10:57 AM, Arun C Murthy a...@hortonworks.com wrote:
  Why not include MAPREDUCE-4211 as well rather than create one release per 
  patch?

 From Cos's description, it sounded like these were backports of fixes
 to help Sqoop2 and fix some build issues. If it's not just to fixup
 leftover bugs in 2.0.4 *once* so downstream projects can integrate
 against 2.0.4.1, and this a release series, then I've completely
 misunderstood the purpose.

 Cos, are you planning 2.0.4.2?

  Also, this is the first time we are seeing a four-numbered scheme in 
  Hadoop. Why not call this 2.0.5-alpha?

 Good point. Since it contains only backports from branch-2, it would
 make sense for it to be an intermediate release.

 I shouldn't have to say this, but I'm changing my vote to -1 while we
 work this out. -C

  On May 24, 2013, at 8:48 PM, Konstantin Boudnik wrote:
 
  All,
 
  I have created a release candidate (rc0) for hadoop-2.0.4.1-alpha that I 
  would
  like to release.
 
  This is a stabilization release that includes fixed for a couple a of 
  issues
  discovered in the testing with BigTop 0.6.0 release candidate.
 
  The RC is available at: 
  http://people.apache.org/~cos/hadoop-2.0.4.1-alpha-rc0/
  The RC tag in svn is here: 
  http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.4.1-alpha-rc0
 
  The maven artifacts are available via repository.apache.org.
 
  Please try the release bits and vote; the vote will run for the usual 7 
  days.
 
  Thanks for your voting
   Cos
 
 
 

Re: [VOTE] Release Apache Hadoop 2.0.4.1-alpha

[Chris Douglas]

On Thu, May 30, 2013 at 3:25 PM, Konstantin Boudnik c...@apache.org wrote:
 There's no plans to release anything else at this point - this is a bug-fix
 release, as I pointed out on a numerous occasions. There's no new features -
 just 2 fixes.

If you're worried about extending the voting by a week, I don't think
anyone can reasonably object to a truncated schedule if the only
change is the version number. Downstream fixes discovered in Bigtop
are a sufficient reason for a patch release and I think we'd all like
them to become routine. Not just in 2.0.x, but in later release
branches.

 2.0.5 matter became and still is too controversial at some point. The vote
 started by Arun to override the results of the Konstantin's vote never been
 closed.

For the nth time, neither release plan vote was binding. We recently
amended the bylaws to eliminate this confusion. There is no ambiguity
between votes when neither is in scope.

 The downstream projects are handing in the middle of the air because
 of that confusion.

Can we please ground our discussion when discussing compatibility and
bugs? Breathless alarm is not proportional to the severity, here.

 Have I missed something or you just called me a cheater and a lair right to 
 my face?

I called you neither. The prenominate votes were closed, counted, and
declared to be binding over objections. I'm annoyed that I have to
toggle my vote to prevent that tactic, but based on recent experience
I don't expect you to forgo it. I'd be happy to learn my caution is
unnecessary. -C

  On Thu, May 30, 2013 at 01:48PM, Chris Douglas wrote:
  On Thu, May 30, 2013 at 10:57 AM, Arun C Murthy a...@hortonworks.com 
  wrote:
   Why not include MAPREDUCE-4211 as well rather than create one release 
   per patch?
 
  From Cos's description, it sounded like these were backports of fixes
  to help Sqoop2 and fix some build issues. If it's not just to fixup
  leftover bugs in 2.0.4 *once* so downstream projects can integrate
  against 2.0.4.1, and this a release series, then I've completely
  misunderstood the purpose.
 
  Cos, are you planning 2.0.4.2?
 
   Also, this is the first time we are seeing a four-numbered scheme in 
   Hadoop. Why not call this 2.0.5-alpha?
 
  Good point. Since it contains only backports from branch-2, it would
  make sense for it to be an intermediate release.
 
  I shouldn't have to say this, but I'm changing my vote to -1 while we
  work this out. -C
 
   On May 24, 2013, at 8:48 PM, Konstantin Boudnik wrote:
  
   All,
  
   I have created a release candidate (rc0) for hadoop-2.0.4.1-alpha that 
   I would
   like to release.
  
   This is a stabilization release that includes fixed for a couple a of 
   issues
   discovered in the testing with BigTop 0.6.0 release candidate.
  
   The RC is available at: 
   http://people.apache.org/~cos/hadoop-2.0.4.1-alpha-rc0/
   The RC tag in svn is here: 
   http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.4.1-alpha-rc0
  
   The maven artifacts are available via repository.apache.org.
  
   Please try the release bits and vote; the vote will run for the usual 
   7 days.
  
   Thanks for your voting
Cos
  
  
  

Re: [VOTE] Release Apache Hadoop 2.0.4.1-alpha

[Chris Douglas]

On Thu, May 30, 2013 at 5:51 PM, Konstantin Boudnik c...@apache.org wrote:
 I have no issues of changing the version to 2.0.5-alpha and restarting to vote
 for the release content, e.g. 2 bug fixes. Shall I call 3 days re-vote because
 of the number change?

+1 Sounds great.

 Does the result of bylaw vote nullifies the unfinished vote started by Arun?
 Sorry, I am dense, apparently.

Yes, nobody should feel bound by either vote. The bylaw change
clarifies that release plans are for RMs to solicit feedback and gauge
PMC support for an artifact, not pre-approvals for doing work.

 Can we limit the vote thread to the merits of the release then?

Happily.

 That sound like adding an insult to injury, if my forth-language skills do not
 mislead me.

They do mislead you, or I've expressed the point imprecisely. We can
take this offline. -C

   On Thu, May 30, 2013 at 01:48PM, Chris Douglas wrote:
   On Thu, May 30, 2013 at 10:57 AM, Arun C Murthy a...@hortonworks.com 
   wrote:
Why not include MAPREDUCE-4211 as well rather than create one 
release per patch?
  
   From Cos's description, it sounded like these were backports of fixes
   to help Sqoop2 and fix some build issues. If it's not just to fixup
   leftover bugs in 2.0.4 *once* so downstream projects can integrate
   against 2.0.4.1, and this a release series, then I've completely
   misunderstood the purpose.
  
   Cos, are you planning 2.0.4.2?
  
Also, this is the first time we are seeing a four-numbered scheme in 
Hadoop. Why not call this 2.0.5-alpha?
  
   Good point. Since it contains only backports from branch-2, it would
   make sense for it to be an intermediate release.
  
   I shouldn't have to say this, but I'm changing my vote to -1 while we
   work this out. -C
  
On May 24, 2013, at 8:48 PM, Konstantin Boudnik wrote:
   
All,
   
I have created a release candidate (rc0) for hadoop-2.0.4.1-alpha 
that I would
like to release.
   
This is a stabilization release that includes fixed for a couple a 
of issues
discovered in the testing with BigTop 0.6.0 release candidate.
   
The RC is available at: 
http://people.apache.org/~cos/hadoop-2.0.4.1-alpha-rc0/
The RC tag in svn is here: 
http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.4.1-alpha-rc0
   
The maven artifacts are available via repository.apache.org.
   
Please try the release bits and vote; the vote will run for the 
usual 7 days.
   
Thanks for your voting
 Cos
   
   
   

Re: [VOTE] Release Apache Hadoop 2.0.4.1-alpha

[Chris Douglas]

+1

Checksum and signature match, ran some unit tests, verified w/ a diff
of release-2.0.4-alpha that the release contains MAPREDUCE-5240 and
HADOOP-9407, plus some fixups to the release notes. -C

On Fri, May 24, 2013 at 8:48 PM, Konstantin Boudnik c...@apache.org wrote:
 All,

 I have created a release candidate (rc0) for hadoop-2.0.4.1-alpha that I would
 like to release.

 This is a stabilization release that includes fixed for a couple a of issues
 discovered in the testing with BigTop 0.6.0 release candidate.

 The RC is available at: 
 http://people.apache.org/~cos/hadoop-2.0.4.1-alpha-rc0/
 The RC tag in svn is here: 
 http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.4.1-alpha-rc0

 The maven artifacts are available via repository.apache.org.

 Please try the release bits and vote; the vote will run for the usual 7 days.

 Thanks for your voting
   Cos

Re: Heads up - 2.0.5-beta

[Chris Douglas]

On Wed, May 1, 2013 at 6:24 PM, Arun C Murthy a...@hortonworks.com wrote:
 Having a strict policy leads to all sorts of further dialogues and issues we 
 could do well without.

+1

Can anyone remember why we vote on release plans? -C

Re: Heads up - 2.0.5-beta

[Chris Douglas]

On Thu, May 2, 2013 at 2:11 AM, Konstantin Shvachko
shv.had...@gmail.com wrote:
 On Thu, May 2, 2013 at 12:07 AM, Chris Douglas cdoug...@apache.org wrote:
 Can anyone remember why we vote on release plans? -C

 To vote on features to include in the release.

Since most features are developed in branches (requiring a merge
vote), each change is RTC, and the release itself requires a vote... a
vote on the executive summary for a release is a poor time to engage
development. It doesn't seem to accomplish anything when it's not a
formality, so maybe we're better without it. Thoughts?

 I am arguing against invasive and destructive features proposed for the
 release.

Heh; do we need new tags in JIRA?

Setting aside the choice of words, we don't assign work by voting.
Stability is a shared goal, but conflating it with inertia after our
experiences with the 0.20 forks, 0.21, and 0.22 takes exactly the
wrong lessons from those episodes.

If you want to create a 2.x branch, pull out the features you view as
high-risk, and invite others to join your effort: you don't need
anyone's permission. If the bylaws contradict this, then that's a bug.
But one can't vote a set of priorities into preeminence, he can only
convince others to share them *and work on them.* It's cheap to
reassign versions to accommodate whatever shape the community takes,
but voting first and expecting everyone to follow the result has never
worked. Cos's chart gives the lie to the attempt: every time we've
tried, we end up reassigning the versions according to reality,
anyway. -C

[jira] [Created] (MAPREDUCE-5194) Heed interrupts during Fetcher shutdown

[Chris Douglas (JIRA)]

Chris Douglas created MAPREDUCE-5194:


 Summary: Heed interrupts during Fetcher shutdown
 Key: MAPREDUCE-5194
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5194
 Project: Hadoop Map/Reduce
  Issue Type: Task
  Components: task
Reporter: Chris Douglas
Priority: Minor


In the current implementation, {{Fetcher}} instances usually exit gracefully 
when the shuffle succeeds. When it fails, threads are interrupted, but may 
continue running harmlessly until the JVM shuts down.

However, to generate consistent checkpoints, these threads should exit cleanly 
to quiesce the state of the shuffle.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-5192) Separate TCE resolution from fetch

[Chris Douglas (JIRA)]

Chris Douglas created MAPREDUCE-5192:


 Summary: Separate TCE resolution from fetch
 Key: MAPREDUCE-5192
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5192
 Project: Hadoop Map/Reduce
  Issue Type: Task
  Components: task
Reporter: Chris Douglas
Priority: Minor


The {{EventFetcher}} thread grounds task completion events as URIs before 
passing them to the {{ShuffleScheduler}}. If the former deferred this to the 
scheduler, one could interpret the TCE metadata differently

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: [VOTE] Release Apache Hadoop 0.23.7

[Chris Douglas]

+1

Verified checksums and signatures, ran some tests, built the tarball. -C

On Thu, Apr 11, 2013 at 12:55 PM, Thomas Graves tgra...@yahoo-inc.com wrote:
 I've created a release candidate (RC0) for hadoop-0.23.7 that I would like
 to release.

 This release is a sustaining release with several important bug fixes in
 it.

 The RC is available at:
 http://people.apache.org/~tgraves/hadoop-0.23.7-candidate-0/
 The RC tag in svn is here:
 http://svn.apache.org/viewvc/hadoop/common/tags/release-0.23.7-rc0/

 The maven artifacts are available via repository.apache.org.

 Please try the release and vote; the vote will run for the usual 7 days.

 thanks,
 Tom Graves

Re: [VOTE] Release Apache Hadoop 2.0.4-alpha

[Chris Douglas]

+1

Verified checksum, signatures. Ran some tests, built the package. -C

On Fri, Apr 12, 2013 at 2:56 PM, Arun C Murthy a...@hortonworks.com wrote:
 Folks,

 I've created a release candidate (RC2) for hadoop-2.0.4-alpha that I would 
 like to release.

 The RC is available at: 
 http://people.apache.org/~acmurthy/hadoop-2.0.4-alpha-rc2/
 The RC tag in svn is here: 
 http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.4-alpha-rc2

 The maven artifacts are available via repository.apache.org.

 Please try the release and vote; the vote will run for the usual 7 days.

 thanks,
 Arun


 --
 Arun C. Murthy
 Hortonworks Inc.
 http://hortonworks.com/

Re: [Vote] Merge branch-trunk-win to trunk

[Chris Douglas]

Konstantin-

There's no debate on the necessity of CI and related infrastructure to
support the platform well. Suresh outlined the support to effect this
here: http://s.apache.org/s1

Is the commitment to establish this infrastructure after the merge
sufficient? -C

On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
shv.had...@gmail.com wrote:
 -1
 We should have a CI infrastructure in place before we can commit to
 supporting Windows platform.

 Eric is right Win/Cygwin was supported since day one.
 I had a Windows box under my desk running nightly builds back in 2006-07.
 People were irritated but I was filing windows bugs until 0.22 release.
 Times changing and I am glad to see wider support for Win platform.

 But in order to make it work you guys need to put the CI process in place

 1. windows jenkins build: could be nightly or PreCommit.
 - Nightly would mean that changes can be committed to trunk based on
 linux PreCommit build. And people will file bugs if the change broke
 Windows nightly build.
 - PreCommit-win build will mean automatic reporting failed tests to
 respective jira blocking commits the same way as it is now with linux
 PreCommit builds.
 We should discuss which way is more efficient for developers.

 2. On-demand-windows Jenkins build.
 I see it as a build to which I can attach my patch and the build will
 run my changes on a dedicated windows box.
 That way people can test their changes without having personal windows nodes.

 I think this is the minimal set of requirement for us to be able to
 commit to the new platform.
 Right now I see only one windows related build
 https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
 Which was failing since Sept 8, 2012 and did not run in the last month.

 Thanks,
 --Konst

 On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
 eri...@hortonworks.com wrote:
 +1 (non-binding)

 A few of observations:

 - Windows has actually been a supported platform for Hadoop since 0.1 .  
 Doug championed supporting windows then and we've continued to do it with 
 varying vigor over time.  To my knowledge we've never made a decision to 
 drop windows support.  The change here is improving our support and dropping 
 the requirement of cigwin.  We had Nutch windows users on the list in 2006 
 and we've been supporting windows FS requirements since inception.

 - A little pragmatism will go a long way.  As a community we've got to stay 
 committed to keeping hadoop simple (so it does work on many platforms) and 
 extending it to take advantage of key emerging OS/hardware features, such as 
 containers, new FSs, virtualization, flash ...  We should all plan to let 
 new features  optimizations emerge that don't work everywhere, if they are 
 compelling and central to hadoop's mission of being THE best fabric for 
 storing and processing big data.

 - A UI project like KDE has to deal with the MANY differences between 
 windows and linux UI APIs.  Hadoop faces no such complex challenge and hence 
 can be maintained from a single codeline IMO.  It is mostly abstracted from 
 the OS APIs via Java and our design choices.  Where it is not we can 
 continue to add plugable abstractions.

Re: [Vote] Merge branch-trunk-win to trunk

[Chris Douglas]

On Fri, Mar 1, 2013 at 1:57 PM, Konstantin Shvachko
shv.had...@gmail.com wrote:
 Commitment is a good thing.
 I think the two builds that I proposed are a prerequisite for Win support.
 If we commit windows patch people will start breaking it the next day.
 Which we wont know without the nightly build and wont be able to fix
 without the on-demand one.

As several people have pointed out already, the surface of possible
conflicts is relatively limited, and- as you did in 2007- the devs on
Windows will report and fix bugs in that platform as they find them.
CI is important for detecting and preventing bugs, but this isn't
software we're launching into orbit.

 Making two builds is less than 2 days work, imho, given that there is
 a Windows node available and that mvn targets are in place. Correct me
 if I missed any complications in the process.

On Fri, Mar 1, 2013 at 3:47 PM, Konstantin Boudnik c...@apache.org wrote:
 It seems that with the HW in place, the matter of setting at least nightly
 build is trivial for anyone with up to date Windows knowledge. I wish I could
 help. Going without a validation is a recipe for a disaster IMO.

Fair enough, though that also implies that the window for regressions
is small, and it leaves little room to doubt that this will receive
priority. Until it's merged, spurious notifications that the current
trunk breaks Windows are an awkward introduction to devs' workflow.
The order of merge/CI is a choice between mild annoyances, really.

But it might be moot. Giri: you're doing the work on this. When do you
think it can be complete? -C

[jira] [Created] (MAPREDUCE-5032) MapTask.MapOutputBuffer contains arithmetic overflows

[Chris Douglas (JIRA)]

Chris Douglas created MAPREDUCE-5032:


 Summary: MapTask.MapOutputBuffer contains arithmetic overflows
 Key: MAPREDUCE-5032
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5032
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: task
Affects Versions: 0.23.5, 2.0.3-alpha, 1.1.1
Reporter: Chris Douglas
Assignee: Chris Douglas


There are several places where offsets into the collection buffer can overflow 
when applied to large buffers. These should be accommodated.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Yarn NodeManager services

[Chris Douglas]

On Tue, Oct 11, 2011 at 2:05 PM,  milind.bhandar...@emc.com wrote:
 As part of MPI implementation in Yarn (aka Hamster), I was looking at
 refactoring some of the functionality into node manager services, so that
 it can be reused by other frameworks too. (Based on the discussion with
 some folks led me to believe that suffle etc is also being planned as NM
 service, separating it from deep integration with node manager.)

A general-purpose transfer protocol for intermediate data was designed
and scrapped for time. So was a local service allocated and shared by
resident containers. Instead, the NM supports loading modules (like
ShuffleHandler) at startup that implement auxiliary (or its
misspelling; my bad), ops-approved services, particularly shared
protocols independent of container and application allocation.

 I could not find the documentation for building NM services, in the MR-279
 architecture PDF. Does the service need to reside inside the NM jvm ?

Yes.

  How
 does NM talk to the service ?

The NM loads and starts a list of services from the config by
classname. Each of these services follows the same lifecycle as the
other NM components. Each service is given a name in the config, which
is also its address for containers launched on that machine. Each
container can pass an opaque blob to the service to initialize it with
container-specific data. For the MR shuffle, these are the job
credentials.

Don't bother listing everything that's wrong with this model. Its
principal virtue was speed of implementation.

 How do containers discover the service ?

They don't. They need to know the name of the service a priori. -C

 (I
 know I can look at the code, but if there is already some docs (or even
 philosophical musings :-), it will make my life easier.

 Thanks,

 - milind

 ---
 Milind Bhandarkar
 Greenplum Labs, EMC
 (Disclaimer: Opinions expressed in this email are those of the author, and
 do not necessarily represent the views of any organization, past or
 present, the author might be affiliated with.)

Re: [VOTE] Merge MR-279 to trunk.

[Chris Douglas]

+1 -C

On Tue, Aug 16, 2011 at 2:14 PM, Mahadev Konar maha...@hortonworks.com wrote:
 Hi all,

  We are excited to let you know that we have MR-279 ready to be merged to 
 trunk. I have uploaded necessary details on 
 https://issues.apache.org/jira/browse/MAPREDUCE-279.

  Please take a look and vote.

  Clearly I am +1 on it.

 thanks
 mahadev

[jira] [Resolved] (MAPREDUCE-2535) JobClient creates a RunningJob with null status and profile

[Chris Douglas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-2535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Douglas resolved MAPREDUCE-2535.
--

Resolution: Fixed

Committed the follow-up. Thanks for the quick fix.

 JobClient creates a RunningJob with null status and profile
 ---

 Key: MAPREDUCE-2535
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2535
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client
Affects Versions: 0.20.204.0
Reporter: Robert Joseph Evans
Assignee: Robert Joseph Evans
 Attachments: MR-2535-0.20.20X-V1.patch, MR-2535-failures-v1.patch


 Exception occurred because the job was retired and is removed from 
 RetireJobCcahe and CompletedJobStatusStore. But, the
 JobClient creates a RunningJob with null status and profile, if getJob(JobID) 
 is called again.
 So, Even-though not null check is there in the following user code, it did 
 not help.
 466 runningJob = jobClient.getJob(mapRedJobID);
 467 if(runningJob != null) {
 JobClient.getJob() should return null if status is null.
 In trunk this is fixed by validating that the job status is not null every 
 time it is updated, and also verifying that that the profile data is not null 
 when created.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Reopened] (MAPREDUCE-2470) Receiving NPE occasionally on RunningJob.getCounters() call

[Chris Douglas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Douglas reopened MAPREDUCE-2470:
--


Reverted while FI breakage is investigated

 Receiving NPE occasionally on RunningJob.getCounters() call
 ---

 Key: MAPREDUCE-2470
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2470
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client
Affects Versions: 0.21.0
 Environment: FreeBSD, Java6, Hadoop r0.21.0
Reporter: Aaron Baff
Assignee: Robert Joseph Evans
 Fix For: 0.23.0

 Attachments: MAPREDUCE-2470-v1.patch, MAPREDUCE-2470-v2.patch, 
 counters_null_data.pcap


 This is running in a Java daemon that is used as an interface (Thrift) to get 
 information and data from MR Jobs. Using JobClient.getJob(JobID) I 
 successfully get a RunningJob object (I'm checking for NULL), and then rarely 
 I get an NPE when I do RunningJob.getCounters(). This seems to occur after 
 the daemon has been up and running for a while, and in the event of an 
 Exception, I close the JobClient, set it to NULL, and a new one should then 
 be created on the next request for data. Yet, I still seem to be unable to 
 fetch the Counters. Below is the stack trace.
 java.lang.NullPointerException
 at org.apache.hadoop.mapred.Counters.downgrade(Counters.java:77)
 at 
 org.apache.hadoop.mapred.JobClient$NetworkedJob.getCounters(JobClient.java:381)
 at 
 com.telescope.HadoopThrift.service.ServiceImpl.getReportResults(ServiceImpl.java:350)
 at 
 com.telescope.HadoopThrift.gen.HadoopThrift$Processor$getReportResults.process(HadoopThrift.java:545)
 at 
 com.telescope.HadoopThrift.gen.HadoopThrift$Processor.process(HadoopThrift.java:421)
 at 
 org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:697)
 at 
 org.apache.thrift.server.THsHaServer$Invocation.run(THsHaServer.java:317)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:619)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (MAPREDUCE-2520) InputSampler.RandomSampler only accepts Text keys

[Chris Douglas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-2520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Douglas resolved MAPREDUCE-2520.
--

Resolution: Invalid

I see. This doesn't appear to be a bug in Hadoop, so I'm going to close this. 
You should contact the vendor for a fix to that code or download the latest 
stable release from Apache: http://hadoop.apache.org/common/releases.html

 InputSampler.RandomSampler only accepts Text keys
 -

 Key: MAPREDUCE-2520
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2520
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: William McNeill
Priority: Minor

 I want to do a total sort on some data whose key type is Writable but not 
 Text.  I wrote an InputSampler.RandomSampler object following the example in 
 the Total Sort section of Hadoop: The Definitive Guide.  When I call 
 InputSampler.writePartitionFile() I get a runtime class cast exception 
 because my key type cannot be cast to Text.  Specifically the issue seems to 
 be the following section of InputSampler.getSample():
 K key = reader.getCurrentKey();
 
 Text keyCopy = WritableUtils.Textclone((Text)key, 
 job.getConfiguration());
 You can only use a RandomSampler on data with Text keys despite the fact that 
 InputSampler takes Key, Value generic parameters.
 InputSampler.getSample() should be changed to cast the key to type K instead 
 of type Text.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (MAPREDUCE-2468) MR-279: Metrics for shuffle

[Chris Douglas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Douglas resolved MAPREDUCE-2468.
--

   Resolution: Fixed
Fix Version/s: (was: 0.23.0)
 Hadoop Flags: [Reviewed]

+1

I committed this to the MR-279 branch. Thanks Luke!

 MR-279: Metrics for shuffle
 ---

 Key: MAPREDUCE-2468
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2468
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
  Components: mrv2
Reporter: Luke Lu
Assignee: Luke Lu
 Attachments: mr-2468-shuffle-metrics-v1.patch


 Metrics for MR shuffle service.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (MAPREDUCE-2411) When you submit a job to a queue with no ACLs you get an inscrutible NPE

[Chris Douglas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-2411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Douglas resolved MAPREDUCE-2411.
--

  Resolution: Fixed
Assignee: Dick King
Hadoop Flags: [Reviewed]

I committed this to branch-0.20-security.

Thanks, Dick!

 When you submit a job to a queue with no ACLs you get an inscrutible NPE
 

 Key: MAPREDUCE-2411
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2411
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 0.20.3
Reporter: Dick King
Assignee: Dick King
Priority: Minor
 Fix For: 0.20.3

 Attachments: M2411-1.patch, mapreduce-2411--2011-03-30.patch


 With this patch we'll check for that, and print a message in the logs.  Then 
 at submission time you find out about it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Resolved: (MAPREDUCE-1401) [Gridmix] Add a load generating factory

[Chris Douglas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-1401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Douglas resolved MAPREDUCE-1401.
--

Resolution: Duplicate

Resolved in MAPREDUCE-1840

 [Gridmix] Add a load generating factory
 ---

 Key: MAPREDUCE-1401
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1401
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: contrib/gridmix
Reporter: Chris Douglas

 To replace previous Gridmix benchmarks (HADOOP-2369 , HADOOP-3770), it must 
 be possible to put a sustained, saturating load on a cluster. While tools for 
 manipulating traces (MAPREDUCE-1295) allow one to produce lighter or heavier 
 load than observed, a client monitoring and responding to observed load would 
 let one write easier-to-interpret, end-to-end benchmarks.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Resolved: (MAPREDUCE-1376) Support for varied user submission in Gridmix

[Chris Douglas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-1376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Douglas resolved MAPREDUCE-1376.
--

 Hadoop Flags: [Reviewed]
Fix Version/s: 0.22.0
   Resolution: Fixed

Fixed in MAPREDUCE-1840

 Support for varied user submission in Gridmix
 -

 Key: MAPREDUCE-1376
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1376
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: contrib/gridmix
Reporter: Chris Douglas
Assignee: Chris Douglas
 Fix For: 0.22.0

 Attachments: 1376-2-yhadoop-security.patch, 
 1376-3-yhadoop20.100.patch, 1376-4-yhadoop20.100.patch, 
 1376-5-yhadoop20-100.patch, 1376-yhadoop-security.patch, M1376-0.patch, 
 M1376-1.patch, M1376-2.patch, M1376-3.patch, M1376-4.patch


 Gridmix currently submits all synthetic jobs as the client user. It should be 
 possible to map users in the trace to a set of users appropriate for the 
 target cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Resolved: (MAPREDUCE-1711) Gridmix should provide an option to submit jobs to the same queues as specified in the trace.

[Chris Douglas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Douglas resolved MAPREDUCE-1711.
--

 Hadoop Flags: [Reviewed]
Fix Version/s: 0.22.0
   Resolution: Fixed

Fixed in MAPREDUCE-1840

 Gridmix should provide an option to submit jobs to the same queues as 
 specified in the trace.
 -

 Key: MAPREDUCE-1711
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1711
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: contrib/gridmix
Reporter: Hong Tang
Assignee: rahul k singh
 Fix For: 0.22.0

 Attachments: diff-gridmix.patch, diff-rumen.patch, 
 MR-1711-yhadoop-20-1xx-2.patch, MR-1711-yhadoop-20-1xx-3.patch, 
 MR-1711-yhadoop-20-1xx-4.patch, MR-1711-yhadoop-20-1xx-5.patch, 
 MR-1711-yhadoop-20-1xx-6.patch, MR-1711-yhadoop-20-1xx-7.patch, 
 MR-1711-yhadoop-20-1xx.patch, MR-1711-Yhadoop-20-crossPort-1.patch, 
 MR-1711-Yhadoop-20-crossPort-2.patch, MR-1711-Yhadoop-20-crossPort.patch, 
 mr-1711-yhadoop-20.1xx-20100416.patch


 Gridmix should provide an option to submit jobs to the same queues as 
 specified in the trace.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Resolved: (MAPREDUCE-1594) Support for Sleep Jobs in gridmix

[Chris Douglas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-1594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Douglas resolved MAPREDUCE-1594.
--

 Hadoop Flags: [Reviewed]
 Assignee: rahul k singh
Fix Version/s: 0.22.0
   Resolution: Fixed

Fixed in MAPREDUCE-1840

 Support for Sleep Jobs in gridmix
 -

 Key: MAPREDUCE-1594
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1594
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
  Components: contrib/gridmix
Reporter: rahul k singh
Assignee: rahul k singh
 Fix For: 0.22.0

 Attachments: 1376-5-yhadoop20-100-3.patch, 1594-diff-4-5.patch, 
 1594-yhadoop-20-1xx-1-2.patch, 1594-yhadoop-20-1xx-1-3.patch, 
 1594-yhadoop-20-1xx-1-4.patch, 1594-yhadoop-20-1xx-1-5.patch, 
 1594-yhadoop-20-1xx-1.patch, 1594-yhadoop-20-1xx.patch


 Support for Sleep jobs in gridmix

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Resolved: (MAPREDUCE-1844) Tests failing with java.lang.NoClassDefFoundError

[Chris Douglas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-1844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Douglas resolved MAPREDUCE-1844.
--

Resolution: Fixed

Fixed by HDFS-1193

 Tests failing with java.lang.NoClassDefFoundError
 -

 Key: MAPREDUCE-1844
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1844
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: test
Reporter: Amar Kamat
Priority: Blocker

 Tests are failing with java.lang.NoClassDefFoundError (see 
 http://pastebin.com/Y3E8iDw0). Steps to reproduce on trunk
 1) Delete ~/.ivy2
 2) checkout trunk
 3) ant -Dtestcase=TestMRCLI run-test-mapred

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

New Common/MapReduce committer: Amareshwari Sriramadasu

[Chris Douglas]

The Hadoop PMC has voted to make Amareshwari Sriramadasu a committer
on the Common and MapReduce subprojects.

Congratulations Amareshwari! Thanks for all your past and continued
work on Hadoop. -C

Contributor Meeting Minutes 05/28/2010

[Chris Douglas]

This month, the MapReduce + HDFS contributor meeting was held at
Cloudera Headquarters.

Announcements for contributor meetings are here:
http://www.meetup.com/Hadoop-Contributors/

Minutes follow. No decisions were made at this meeting, but the
following issues were discussed and may presage future discussion and
decisions on these lists.

Eli, I think you have all the slides. Would you mind sending them out? -C

== 0.21 release update ==
* Continuing to close blockers, ping people for updates and suggestions
* About 20 open blockers. Many are MapReduce documentation that may be
pushed. Speak up if 0.21 is missing anything substantive.
* Common/HDFS visibility and annotations are close to consensus;
MapReduce annotations are committed to trunk and the 0.21 branch

== HEP proposal ==
(what follows is the sketch presented at the meeting. A full proposal
with concrete details will be circulated on the list)

* Based on- and very similar to- the PEP (Python Enhancement Proposal) Process
* Audience is HDFS and MapReduce; not necessarily adopted by other subprojects
  - Addresses the perception that there is friction between
innovation/experimentation and stability
* Not for small enhancements, features, and bug fixes. This should not
slow down typical development or impede casual contribution to Hadoop
* Primary mechanism for new features, collecting input, documenting
design decisions
* JIRA is good for details, but not for deciding on wide shifts in direction
* Purpose is for author to build consensus and gather dissenting opinions.
  - All may comment, but Editors will review incoming HEP material
  - Editors determine only whether the HEP is complete, not whether
they believe it is a sound idea
  - Editors are appointed by the PMC
  - Mechanism for appointing Editors and term of service TBD
- Apache Board appoints Shepherds for projects somewhat randomly,
to projects. A similar mechanism could work for incoming HEPs
  - Proposal *may* come with code, but not necessarily.
Drafting/baking of the HEP occurs in public on a list dedicated to
that particular proposal. Once Editors certify the HEP as complete, it
is sent to general@ for wider discussion.
- The discussion phase begins on gene...@. The mailing list exists
to ensure the HEP is complete enough to present to the community.
  - Some discussion on the difference between posting to general@ and
posting to the HEP list. Completeness is, of course, subjective. If
the Editor and Author disagree whether the proposal affects an aspect
of the framework enough to merit special consideration, it is not
entirely clear how to resolve the disagreement.
- In general, the role of the Editor in the community-driven
process of Hadoop is not entirely clear. It may be possible to
optimize it out.
  - Once discussion ends, the HEP is passed (or fails to pass) by a
vote of the PMC (mechanics undefined). In Python, the result is
committed to the repository. A similar practice would make sense in
Hadoop.
* Which issues require HEPs?
  - Discussion ranged. Append, backup namenode, edit log rewrite, et
al. were examples of features substantial enough to merit a HEP. Pure
Java CRC is an example of an enhancement that would not. Whether an
explicit process must be in place to determine whether an issue
requires a HEP is not clear.
  - Viewing HEPs as a way of soliciting consensus for an approach
might be more accurate. Going through the HEP process should always
improve the chances of a successful proposal

* Evaluation
  - The proposal may be rejected if it is redundant with existing
functionality, technically unsound, insufficiently motivated, no
backwards compatibility story, etc.
  - Implementation is not necessary, and is lightly discouraged.
Feedback is less welcome once code is in hand.
  - Purpose is to be clear about the acceptance criteria for that
issue, e.g. concerns that the proposal may not scale or may harm
performance
  - Dissenting opinions must be recorded accurately. Quoting would be
a safe practice for the Author to encourage HEP reviewers not to block
the product of the proposal.

* The testing burden and completion strategy may be ambiguous
  - Whether the proposal affects scalability may not be testable by
the implementer. Completing the proposal to address all use cases may
require considerably more work than the Author is willing or motivated
to invest.
  - The HEP discussion on general@ should explore whether such
objections are merited and reasonable. For example, a particularly
obscure/esoteric use case could be included as a condition for
acceptance if the dissenter is willing to invest the resources to
test/validate it. The process is flexible in this regard.
- But it is not infinitely flexible. Backwards compatibility,
performance regression, availability, and other considerations need
not be called out in every HEP.
- Traditional concerns need to be documented. Acceptance criteria
should ideally be automated and reproducible 

[jira] Created: (MAPREDUCE-1683) Remove JNI calls from ClusterStatus cstr

[Chris Douglas (JIRA)]

Remove JNI calls from ClusterStatus cstr


 Key: MAPREDUCE-1683
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1683
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobtracker
Affects Versions: 0.20.2
Reporter: Chris Douglas


The {{ClusterStatus}} constructor makes two JNI calls to the {{Runtime}} to 
fetch memory information. {{ClusterStatus}} instances are often created inside 
the {{JobTracker}} to obtain other, unrelated metrics (sometimes from 
schedulers' inner loops). Given that this information is related to the 
{{JobTracker}} process and not the cluster, the metrics are also available via 
{{JvmMetrics}}, and the jsps can gather this information for themselves: these 
fields can be removed from {{ClusterStatus}}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (MAPREDUCE-1622) Include slf4j dependencies in binary tarball

[Chris Douglas (JIRA)]

Include slf4j dependencies in binary tarball


 Key: MAPREDUCE-1622
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1622
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build
Reporter: Chris Douglas
Priority: Minor
 Fix For: 0.22.0


After MAPREDUCE-1556 and HADOOP-6486, starting \*Trackers from a binary tarball 
produces the following warning:
{noformat}
SLF4J: Failed to load class org.slf4j.impl.StaticLoggerBinder.
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further 
details.
2010-03-18 01:11:32.988::INFO:  Logging to STDERR via org.mortbay.log.StdErrLog
2010-03-18 01:11:33.056::INFO:  jetty-6.1.14
{noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Resolved: (MAPREDUCE-1571) OutOfMemoryError during shuffle

[Chris Douglas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-1571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Douglas resolved MAPREDUCE-1571.
--

Resolution: Duplicate

This is a duplicate of MAPREDUCE-1182.

{{MAX_SINGLE_SHUFFLE_SEGMENT_FRACTION}} governs the maximum size of a map 
output segment that will be stored in memory. The aggregate limit is enforced 
by a separate mechanism.

 OutOfMemoryError during shuffle
 ---

 Key: MAPREDUCE-1571
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1571
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 0.20.1, 0.20.2
Reporter: Jacob Rideout

 A OutOfMemoryError can occur when determining if the shuffle can be 
 accomplished in memory
 2010-03-06 07:54:49,621 INFO org.apache.hadoop.mapred.ReduceTask:
 Shuffling 4191933 bytes (435311 raw bytes) into RAM from
 attempt_201003060739_0002_m_61_0
 2010-03-06 07:54:50,222 INFO org.apache.hadoop.mapred.ReduceTask: Task
 attempt_201003060739_0002_r_00_0: Failed fetch #1 from
 attempt_201003060739_0002_m_000202_0
 2010-03-06 07:54:50,223 WARN org.apache.hadoop.mapred.ReduceTask:
 attempt_201003060739_0002_r_00_0 adding host
 hd37.dfs.returnpath.net to penalty box, next contact in 4 seconds
 2010-03-06 07:54:50,223 INFO org.apache.hadoop.mapred.ReduceTask:
 attempt_201003060739_0002_r_00_0: Got 1 map-outputs from previous
 failures
 2010-03-06 07:54:50,223 FATAL org.apache.hadoop.mapred.TaskRunner:
 attempt_201003060739_0002_r_00_0 : Map output copy failure :
 java.lang.OutOfMemoryError: Java heap space
at 
 org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleInMemory(ReduceTask.java:1508)
at 
 org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1408)
at 
 org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1261)
at 
 org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1195)
 Ted Yu identified the following potential solution:
 I think there is mismatch (in ReduceTask.java) between:
  this.numCopiers = conf.getInt(mapred.reduce.parallel.copies, 5);
 and:
maxSingleShuffleLimit = (long)(maxSize *
 MAX_SINGLE_SHUFFLE_SEGMENT_FRACTION);
 where MAX_SINGLE_SHUFFLE_SEGMENT_FRACTION is 0.25f
 because
  copiers = new ArrayListMapOutputCopier(numCopiers);
 so the total memory allocated for in-mem shuffle is 1.25 * maxSize
 A JIRA should be filed to correlate the constant 5 above and
 MAX_SINGLE_SHUFFLE_SEGMENT_FRACTION.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (MAPREDUCE-1551) TestMiniMRLocalFS fails

[Chris Douglas (JIRA)]

TestMiniMRLocalFS fails
---

 Key: MAPREDUCE-1551
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1551
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: test
Affects Versions: 0.22.0
Reporter: Chris Douglas


{{TestMiniMRLocalFS}} fails consistently on trunk:
{noformat}
Testcase: testWithLocal took 38.957 sec
Caused an ERROR
File QuasiMonteCarlo_TMP_3_141592654/out/reduce-out does not exist.
java.io.FileNotFoundException: File 
QuasiMonteCarlo_TMP_3_141592654/out/reduce-out does not exist.
at 
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:420)
at 
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:246)
at 
org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1466)
at 
org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1447)
at 
org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:313)
at 
org.apache.hadoop.mapred.TestMiniMRWithDFS.runPI(TestMiniMRWithDFS.java:235)
at 
org.apache.hadoop.mapred.TestMiniMRLocalFS.testWithLocal(TestMiniMRLocalFS.java:68)
{noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Resolved: (MAPREDUCE-623) Resolve javac warnings in mapred

[Chris Douglas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Douglas resolved MAPREDUCE-623.
-

   Resolution: Fixed
Fix Version/s: (was: 0.21.0)
   0.20.2

I committed this to the 0.20 branch

 Resolve javac warnings in mapred
 

 Key: MAPREDUCE-623
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-623
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: build
Reporter: Jothi Padmanabhan
Assignee: Jothi Padmanabhan
 Fix For: 0.20.2

 Attachments: M623-0v20.patch, mapreduce-623.patch


 Towards a solution for HADOOP-5628, we need to resolve all javac warnings. 
 This jira will try to resolve javac warnings where ever possible and suppress 
 them where resolution is not possible.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Reopened: (MAPREDUCE-623) Resolve javac warnings in mapred

[Chris Douglas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Douglas reopened MAPREDUCE-623:
-


These changes should also be applied to 0.20

 Resolve javac warnings in mapred
 

 Key: MAPREDUCE-623
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-623
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: build
Reporter: Jothi Padmanabhan
Assignee: Jothi Padmanabhan
 Fix For: 0.21.0

 Attachments: M623-0v20.patch, mapreduce-623.patch


 Towards a solution for HADOOP-5628, we need to resolve all javac warnings. 
 This jira will try to resolve javac warnings where ever possible and suppress 
 them where resolution is not possible.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Reopened: (MAPREDUCE-433) TestReduceFetch failed.

[Chris Douglas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Douglas reopened MAPREDUCE-433:
-


 TestReduceFetch failed.
 ---

 Key: MAPREDUCE-433
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-433
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Chris Douglas
 Fix For: 0.20.2

 Attachments: 6029-0.patch, 6029-1.patch, 6029-2.patch, 
 FAILING-PARTIALMEM-TEST-org.apache.hadoop.mapred.TestReduceFetch.txt, 
 M433-0v20.patch, M433-v20.another.patch, 
 TEST-org.apache.hadoop.mapred.TestReduceFetch.txt, 
 TEST-org.apache.hadoop.mapred.TestReduceFetch.txt, testReduceFromMem.txt


 {noformat}
 Testcase: testReduceFromMem took 23.625 sec
   FAILED
 Non-zero read from local: 83
 junit.framework.AssertionFailedError: Non-zero read from local: 83
   at 
 org.apache.hadoop.mapred.TestReduceFetch.testReduceFromMem(TestReduceFetch.java:289)
   at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
   at junit.extensions.TestSetup$1.protect(TestSetup.java:23)
   at junit.extensions.TestSetup.run(TestSetup.java:27)
 {noformat}
 Ran TestReduceFetch a few times on a clean trunk.  It failed consistently.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Resolved: (MAPREDUCE-433) TestReduceFetch failed.

[Chris Douglas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Douglas resolved MAPREDUCE-433.
-

   Resolution: Fixed
Fix Version/s: (was: 0.21.0)
   0.20.2

Committed to 0.20

 TestReduceFetch failed.
 ---

 Key: MAPREDUCE-433
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-433
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Chris Douglas
 Fix For: 0.20.2

 Attachments: 6029-0.patch, 6029-1.patch, 6029-2.patch, 
 FAILING-PARTIALMEM-TEST-org.apache.hadoop.mapred.TestReduceFetch.txt, 
 M433-0v20.patch, M433-v20.another.patch, 
 TEST-org.apache.hadoop.mapred.TestReduceFetch.txt, 
 TEST-org.apache.hadoop.mapred.TestReduceFetch.txt, testReduceFromMem.txt


 {noformat}
 Testcase: testReduceFromMem took 23.625 sec
   FAILED
 Non-zero read from local: 83
 junit.framework.AssertionFailedError: Non-zero read from local: 83
   at 
 org.apache.hadoop.mapred.TestReduceFetch.testReduceFromMem(TestReduceFetch.java:289)
   at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
   at junit.extensions.TestSetup$1.protect(TestSetup.java:23)
   at junit.extensions.TestSetup.run(TestSetup.java:27)
 {noformat}
 Ran TestReduceFetch a few times on a clean trunk.  It failed consistently.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (MAPREDUCE-1405) unchecked cast warnings in trunk

[Chris Douglas (JIRA)]

unchecked cast warnings in trunk


 Key: MAPREDUCE-1405
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1405
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Chris Douglas
Priority: Trivial


{noformat}
compile-mapred-classes:
[jsp-compile] 10/01/24 15:16:55 WARN compiler.TldLocationsCache: Internal 
Error: File /WEB-INF/web.xml not found
[javac] Compiling 494 source files to /.../mapred/build/classes
[javac] /.../mapred/src/java/org/apache/hadoop/mapred/Child.java:159: 
warning: [unchecked] unchecked cast
[javac] found   : org.apache.hadoop.security.token.Tokencapture#981 of ? 
extends org.apache.hadoop.security.token.TokenIdentifier
[javac] required: 
org.apache.hadoop.security.token.Tokenorg.apache.hadoop.mapreduce.security.token.JobTokenIdentifier
[javac] TokenJobTokenIdentifier jt = 
(TokenJobTokenIdentifier)ts.getJobToken();
[javac] 
^
[javac] /.../mapred/src/java/org/apache/hadoop/mapred/TaskTracker.java:922: 
warning: [unchecked] unchecked cast
[javac] found   : org.apache.hadoop.security.token.Tokencapture#810 of ? 
extends org.apache.hadoop.security.token.TokenIdentifier
[javac] required: 
org.apache.hadoop.security.token.Tokenorg.apache.hadoop.mapreduce.security.token.JobTokenIdentifier
[javac] TokenJobTokenIdentifier jt = 
(TokenJobTokenIdentifier)ts.getJobToken();
{noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (MAPREDUCE-1406) JobContext.MAP_COMBINE_MIN_SPILLS is misspelled

[Chris Douglas (JIRA)]

JobContext.MAP_COMBINE_MIN_SPILLS is misspelled
---

 Key: MAPREDUCE-1406
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1406
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 0.21.0
Reporter: Chris Douglas
Priority: Trivial
 Attachments: M1406-0.patch

{{JobContext.MAP_COMBINE_MIN_SPILLS}} is misspelled as 
{{JobContext.MAP_COMBINE_MIN_SPISS}}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (MAPREDUCE-1401) [Gridmix] Add a load generating factory

[Chris Douglas (JIRA)]

[Gridmix] Add a load generating factory
---

 Key: MAPREDUCE-1401
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1401
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: contrib/gridmix
Reporter: Chris Douglas


To replace previous Gridmix benchmarks (HADOOP-2369 , HADOOP-3770), it must be 
possible to put a sustained, saturating load on a cluster. While tools for 
manipulating traces (MAPREDUCE-1295) allow one to produce lighter or heavier 
load than observed, a client monitoring and responding to observed load would 
let one write easier-to-interpret, end-to-end benchmarks.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (MAPREDUCE-1402) Remove src/benchmarks

[Chris Douglas (JIRA)]

Remove src/benchmarks
-

 Key: MAPREDUCE-1402
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1402
 Project: Hadoop Map/Reduce
  Issue Type: Task
  Components: benchmarks
Reporter: Chris Douglas
Assignee: Chris Douglas
 Fix For: 0.22.0


src/benchmarks has not attracted many contributions. The versions of gridmix 
currently there are deprecated in in favor of the version in contrib 
(MAPREDUCE-776) and will be redundant after MAPREDUCE-1401.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Reopened: (MAPREDUCE-433) TestReduceFetch failed.

[Chris Douglas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Douglas reopened MAPREDUCE-433:
-


Per MAPREDUCE-1172, this needs to be ported to 0.20

 TestReduceFetch failed.
 ---

 Key: MAPREDUCE-433
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-433
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Chris Douglas
 Fix For: 0.21.0

 Attachments: 6029-0.patch, 6029-1.patch, 6029-2.patch, 
 FAILING-PARTIALMEM-TEST-org.apache.hadoop.mapred.TestReduceFetch.txt, 
 M433-0v20.patch, TEST-org.apache.hadoop.mapred.TestReduceFetch.txt, 
 TEST-org.apache.hadoop.mapred.TestReduceFetch.txt, testReduceFromMem.txt


 {noformat}
 Testcase: testReduceFromMem took 23.625 sec
   FAILED
 Non-zero read from local: 83
 junit.framework.AssertionFailedError: Non-zero read from local: 83
   at 
 org.apache.hadoop.mapred.TestReduceFetch.testReduceFromMem(TestReduceFetch.java:289)
   at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
   at junit.extensions.TestSetup$1.protect(TestSetup.java:23)
   at junit.extensions.TestSetup.run(TestSetup.java:27)
 {noformat}
 Ran TestReduceFetch a few times on a clean trunk.  It failed consistently.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (MAPREDUCE-1384) Rumen should return UGI information for users rather than String

[Chris Douglas (JIRA)]

Rumen should return UGI information for users rather than String


 Key: MAPREDUCE-1384
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1384
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Reporter: Chris Douglas


It would be cleaner for downstream tools if Rumen were to return the 
{{UserGroupInformation}} about users in the trace- including groups- instead of 
the username only

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (MAPREDUCE-1376) Support for varied user submission in Gridmix

[Chris Douglas (JIRA)]

Support for varied user submission in Gridmix
-

 Key: MAPREDUCE-1376
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1376
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: contrib/gridmix
Reporter: Chris Douglas


Gridmix currently submits all synthetic jobs as the client user. It should be 
possible to map users in the trace to a set of users appropriate for the target 
cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (MAPREDUCE-1324) Make MapTask.MapOutputBuffer a standalone class

[Chris Douglas (JIRA)]

Make MapTask.MapOutputBuffer a standalone class
---

 Key: MAPREDUCE-1324
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1324
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: task
Reporter: Chris Douglas
Assignee: Chris Douglas
Priority: Minor


{{MapTask.MapOutputBuffer}} has few dependencies on its outer class, but is 
more than half its total length. It should be factored out into a separate 
class.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Resolved: (MAPREDUCE-1190) Add package.html to pi and pi.math packages.

[Chris Douglas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-1190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Douglas resolved MAPREDUCE-1190.
--

   Resolution: Fixed
Fix Version/s: 0.22.0
 Hadoop Flags: [Reviewed]

I committed this. Thanks, Nicholas!

 Add package.html to pi and pi.math packages.
 

 Key: MAPREDUCE-1190
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1190
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
  Components: documentation
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE
Priority: Minor
 Fix For: 0.22.0

 Attachments: m1190_20091105.patch


 package.html is missing in the pi and pi.math packages.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Reopened: (MAPREDUCE-972) distcp can timeout during rename operation to s3

[Chris Douglas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Douglas reopened MAPREDUCE-972:
-


 distcp can timeout during rename operation to s3
 

 Key: MAPREDUCE-972
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-972
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: distcp, documentation
Affects Versions: 0.20.1
Reporter: Aaron Kimball
Assignee: Aaron Kimball
 Fix For: 0.21.0

 Attachments: MAPREDUCE-972.2.patch, MAPREDUCE-972.3.patch, 
 MAPREDUCE-972.4.patch, MAPREDUCE-972.5.patch, MAPREDUCE-972.6.patch, 
 MAPREDUCE-972.patch


 rename() in S3 is implemented as copy + delete. The S3 copy operation can 
 perform very slowly, which may cause task timeout.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Resolved: (MAPREDUCE-972) distcp can timeout during rename operation to s3

[Chris Douglas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Douglas resolved MAPREDUCE-972.
-

Resolution: Fixed

 distcp can timeout during rename operation to s3
 

 Key: MAPREDUCE-972
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-972
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: distcp, documentation
Affects Versions: 0.20.1
Reporter: Aaron Kimball
Assignee: Aaron Kimball
 Fix For: 0.21.0

 Attachments: MAPREDUCE-972.2.patch, MAPREDUCE-972.3.patch, 
 MAPREDUCE-972.4.patch, MAPREDUCE-972.5.patch, MAPREDUCE-972.6.patch, 
 MAPREDUCE-972.patch


 rename() in S3 is implemented as copy + delete. The S3 copy operation can 
 perform very slowly, which may cause task timeout.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (MAPREDUCE-1147) Map output records counter missing for map-only jobs in new API

[Chris Douglas (JIRA)]

Map output records counter missing for map-only jobs in new API
---

 Key: MAPREDUCE-1147
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1147
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 0.20.1, 0.21.0
Reporter: Chris Douglas
Priority: Blocker
 Fix For: 0.20.2


In the new API, the counter for map output records is not incremented for 
map-only jobs

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Reopened: (MAPREDUCE-1109) ConcurrentModificationException in jobtracker.jsp

[Chris Douglas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-1109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Douglas reopened MAPREDUCE-1109:
--


This still exists in 0.20, right? Sorry, I only mentioned MAPREDUCE-679 because 
this was fixed as a detour in that issue and the fix should either be 
backported or changed to match whatever approach is agreed on here.

 ConcurrentModificationException in jobtracker.jsp
 -

 Key: MAPREDUCE-1109
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1109
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobtracker
Affects Versions: 0.20.1
Reporter: dhruba borthakur
Assignee: dhruba borthakur

 The jobtracker.jsp invoked methods tracker.runningJobs(), 
 tracker.completedJobs() and tracker.failedJobs() but these methods are not 
 synchronized and can cause ConcurrentModificationException if the JT is 
 concurrently changing the contents of these datastructures.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Resolved: (MAPREDUCE-819) DistCP Guide - updates

[Chris Douglas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Douglas resolved MAPREDUCE-819.
-

Resolution: Fixed

This was committed. Thanks, Corinne!

 DistCP Guide - updates
 --

 Key: MAPREDUCE-819
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-819
 Project: Hadoop Map/Reduce
  Issue Type: Task
  Components: documentation
Affects Versions: 0.21.0
Reporter: Corinne Chandel
Assignee: Corinne Chandel
 Fix For: 0.21.0

 Attachments: distcp.pdf, MAPREDUCE-819.patch


 DistCp Guide - updates to examples. Changes suggested and approved by 
 engineer.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (MAPREDUCE-1063) Document Gridmix benchmark

[Chris Douglas (JIRA)]

Document Gridmix benchmark
--

 Key: MAPREDUCE-1063
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1063
 Project: Hadoop Map/Reduce
  Issue Type: Task
  Components: benchmarks
Reporter: Chris Douglas
Priority: Minor
 Fix For: 0.21.0
 Attachments: M1063-0.patch

The Gridmix benchmark should have forrest documentation and a README pointing 
to it.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (MAPREDUCE-1061) Gridmix unit test should validate input/output bytes

[Chris Douglas (JIRA)]

Gridmix unit test should validate input/output bytes


 Key: MAPREDUCE-1061
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1061
 Project: Hadoop Map/Reduce
  Issue Type: Test
Affects Versions: 0.21.0
Reporter: Chris Douglas
 Fix For: 0.21.0


TestGridmixSubmission currently verifies only that the correct number of jobs 
have been run. The test should validate the I/O parameters it claims to satisfy.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.