Re: Myriad Vagrant Setup Issue

[Darin Johnson]

Matt, if you can't access the UI, on the slave you should still be able to
access stderr and stdout going to:

/tmp/mesos/slaves//frameworks//executors/myriad_executor/runs/latest/stderr

/tmp/mesos/slaves//frameworks//executors/myriad_executor/runs/latest/stdout
Replace /tmp/mesos/ with your workdir (likely /var/run/mesos/ or
/tmp/mesos).  The error messages here are usually informative.

On Fri, Jan 15, 2016 at 11:13 AM, Matthew J. Loppatto <
mloppa...@keywcorp.com> wrote:

> Hey Darin,
>
> For some reason my Mesos UI hangs when loading the logs but I posted the
> contents of my mesos slave logs in /var/log/mesos to this public Gist:
> https://gist.github.com/FearTheParrot/b00aa7eee9ae169498d3
>
> Matt
>
> -----Original Message-
> From: Darin Johnson [mailto:dbjohnson1...@gmail.com]
> Sent: Friday, January 15, 2016 10:55 AM
> To: Dev
> Subject: Re: Myriad Vagrant Setup Issue
>
> Hey Matt, if you look at the mesos ui is there any information in the
> stderr or stdout of the Slave Host it's staging on?
>
> Darin
>
> On Fri, Jan 15, 2016 at 10:36 AM, Matthew J. Loppatto <
> mloppa...@keywcorp.com> wrote:
>
> > I've gotten a little farther on this issue by increasing the mesos
> > slave memory to 4 GB from 2GB.  The node manager task get launched and
> > sits in the STAGING state for a minute and then the mesos-slave.INFO log
> shows:
> >
> > I0115 15:19:12.114537 30903 slave.cpp:3841] Terminating executor
> > myriad_executor20160115-145750-344821002-5050-30838-20160115-14575
> > 0-344821002-5050-30838-O18020160115-145750-344821002-5050-30838-S0
> > of framework 20160115-145750-344821002-5050-30838- because it did
> > not register within 1mins
> >
> > I then increased the mesos slave's executor_registration_timeout
> > setting from 1mins to 5mins to see if that would make a difference but
> > still get the following in the log:
> >
> > I0115 15:19:12.114537 30903 slave.cpp:3841] Terminating executor
> > myriad_executor20160115-145750-344821002-5050-30838-20160115-14575
> > 0-344821002-5050-30838-O18020160115-145750-344821002-5050-30838-S0
> > of framework 20160115-145750-344821002-5050-30838- because it did
> > not register within 5mins
> >
> > Is there any guidance on why the Myriad executor fails to register
> > with the Mesos slave?
> >
> > Thanks,
> > Matt
> >
> > -Original Message-
> > From: Matthew J. Loppatto
> > Sent: Thursday, January 14, 2016 2:25 PM
> > To: 'dev@myriad.incubator.apache.org'
> > Subject: RE: Myriad Vagrant Setup Issue
> >
> > Sarjeet,
> >
> > Thanks for the reply.  I modified the medium profile in my
> > myriad-config-default.yml file to use 1 cpu and 1024 MB mem and am
> > seeing a similar issue in the YARN resource manager log:
> >
> > Offer not sufficient for task with, cpu: 1.4, memory: 2432.0, ports:
> > 1001
> >
> > If I try lowering the medium profile memory below 1024 I get the
> > following message in the log:
> >
> > NodeManager from vagrant-ubuntu-trusty-64 doesn’t satisfy minimum
> > allocations, Sending SHUTDOWN signal to NodeManager.
> >
> > Increasing the memory of the VM to 6 GB also didn't solve the issue.
> > Are there any other measures I can take to resolve the insufficient
> > resource messages?
> >
> > Thanks,
> > Matt
> >
> > -Original Message-
> > From: sarjeet singh [mailto:sarje...@usc.edu]
> > Sent: Thursday, January 14, 2016 12:41 PM
> > To: dev@myriad.incubator.apache.org
> > Subject: Re: Myriad Vagrant Setup Issue
> >
> > Matthew,
> >
> > You can modify profile configurations for Nodemanagers in
> > myriad-config-default.yml and reduce medium (default) NM configuration
> > to match with your VM capacity so a default NM (medium profile) could
> > launch without any issue.
> >
> > - Sarjeet Singh
> >
> > On Thu, Jan 14, 2016 at 10:56 PM, Matthew J. Loppatto <
> > mloppa...@keywcorp.com> wrote:
> >
> > > Hi,
> > >
> > > I'm trying to setup Myriad for an R project at my company but I'm
> > > having some trouble even getting the Vagrant VM working properly.  I
> > > followed the instructions here:
> > >
> > > https://github.com/apache/incubator-myriad/blob/master/docs/vagrant.
> > > md
> > >
> > > with some minor corrections but the Node Manager fails to start.  It
> > > looks like a resource issue based on the log output.  The Mesos UI
> > > shows a slave process with 2 cpu and 2 GB mem, but the log states
> > > the task requires 4 cpu and 5.5 GB mem.
> > >
> > > I've detailed my configuration and log output in this public Gist:
> > >
> > > https://gist.github.com/FearTheParrot/626259c23a854645fcbf
> > >
> > > Would it be possible to provision the Mesos slave with more
> > > resources while also reducing the profile size of the Node Manager?
> > > The Vagrant VM only has 4 GB ram and 2 cpu.
> > >
> > > Any help would be appreciated.
> > >
> > > Thanks!
> > > Matt
> > >
> >
>

Re: Next dev sync hangout will be on 1/6/2016

[Darin Johnson]

I'll be available.

On Tue, Jan 12, 2016 at 3:05 PM, Adam Bordelon <a...@mesosphere.io> wrote:

> I'll be ready for a meeting tomorrow.
>
> On Tue, Jan 12, 2016 at 11:03 AM, Sarjeet Singh <sarjeetsi...@maprtech.com
> >
> wrote:
>
> > +1.
> >
> > Can we confirm this for tomorrow if this is happening?
> >
> > -Sarjeet
> >
> > On Fri, Jan 8, 2016 at 6:53 PM, Darin Johnson <dbjohnson1...@gmail.com>
> > wrote:
> >
> > > Sounds good to me.  I think we might have another possible east coast
> > > contributor join.
> > >
> > > Darin
> > >
> > > On Thu, Jan 7, 2016 at 9:09 PM, Adam Bordelon <a...@mesosphere.io>
> > wrote:
> > >
> > > > Sorry, this slipped off my calendar, so I didn't even try to attend.
> I
> > > > guess we'll pick up again next week?
> > > >
> > > > On Wed, Jan 6, 2016 at 11:22 AM, Paul Curtis <pcur...@maprtech.com>
> > > wrote:
> > > >
> > > > > I attempted to join as well  it seems that no one was in the
> > > > > hangout. I gave up after about 15 minutes.
> > > > >
> > > > > paul
> > > > >
> > > > > On Wed, Jan 6, 2016 at 12:19 PM, Darin Johnson <
> > > dbjohnson1...@gmail.com>
> > > > > wrote:
> > > > > > Can't seem to join...
> > > > > > On Jan 6, 2016 12:16 PM, "Darin Johnson" <
> dbjohnson1...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > >> Trying to join
> > > > > >> On Jan 6, 2016 12:06 PM, "yuliya Feldman"
> > > <yufeld...@yahoo.com.invalid
> > > > >
> > > > > >> wrote:
> > > > > >>
> > > > > >>> Do we have a sync today?
> > > > > >>>
> > > > > >>>
> > > > > >>>   From: Santosh Marella <smare...@maprtech.com>
> > > > > >>>  To: dev@myriad.incubator.apache.org
> > > > > >>>  Sent: Wednesday, December 16, 2015 9:47 AM
> > > > > >>>  Subject: Next dev sync hangout will be on 1/6/2016
> > > > > >>>
> > > > > >>> We have decided to hold the next dev sync on 1/6/2016 (instead
> of
> > > > > >>> 12/30/2015).
> > > > > >>>
> > > > > >>> Meeting notes from today's hangout:
> > > > > >>>
> > > > > >>>
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1JGmJrgeg98bHw_0_sSRmyX6WiAe13OdErcFlaz6Aa04/edit#
> > > > > >>>
> > > > > >>> Thanks,
> > > > > >>> Santosh
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > > >>
> > > > > >>
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Paul Curtis - Senior Product Technologist
> > > > > O: +1 203-660-0015 - M: +1 203-539-9705
> > > > >
> > > > > Now Available - Free Hadoop On-Demand Training
> > > > >
> > > >
> > >
> >
>

Re: Next dev sync hangout will be on 1/6/2016

[Darin Johnson]

Sounds good to me.  I think we might have another possible east coast
contributor join.

Darin

On Thu, Jan 7, 2016 at 9:09 PM, Adam Bordelon <a...@mesosphere.io> wrote:

> Sorry, this slipped off my calendar, so I didn't even try to attend. I
> guess we'll pick up again next week?
>
> On Wed, Jan 6, 2016 at 11:22 AM, Paul Curtis <pcur...@maprtech.com> wrote:
>
> > I attempted to join as well  it seems that no one was in the
> > hangout. I gave up after about 15 minutes.
> >
> > paul
> >
> > On Wed, Jan 6, 2016 at 12:19 PM, Darin Johnson <dbjohnson1...@gmail.com>
> > wrote:
> > > Can't seem to join...
> > > On Jan 6, 2016 12:16 PM, "Darin Johnson" <dbjohnson1...@gmail.com>
> > wrote:
> > >
> > >> Trying to join
> > >> On Jan 6, 2016 12:06 PM, "yuliya Feldman" <yufeld...@yahoo.com.invalid
> >
> > >> wrote:
> > >>
> > >>> Do we have a sync today?
> > >>>
> > >>>
> > >>>   From: Santosh Marella <smare...@maprtech.com>
> > >>>  To: dev@myriad.incubator.apache.org
> > >>>  Sent: Wednesday, December 16, 2015 9:47 AM
> > >>>  Subject: Next dev sync hangout will be on 1/6/2016
> > >>>
> > >>> We have decided to hold the next dev sync on 1/6/2016 (instead of
> > >>> 12/30/2015).
> > >>>
> > >>> Meeting notes from today's hangout:
> > >>>
> > >>>
> >
> https://docs.google.com/document/d/1JGmJrgeg98bHw_0_sSRmyX6WiAe13OdErcFlaz6Aa04/edit#
> > >>>
> > >>> Thanks,
> > >>> Santosh
> > >>>
> > >>>
> > >>>
> > >>
> > >>
> >
> >
> >
> > --
> > Paul Curtis - Senior Product Technologist
> > O: +1 203-660-0015 - M: +1 203-539-9705
> >
> > Now Available - Free Hadoop On-Demand Training
> >
>

Re: Next dev sync hangout will be on 1/6/2016

[Darin Johnson]

Trying to join
On Jan 6, 2016 12:06 PM, "yuliya Feldman" 
wrote:

> Do we have a sync today?
>
>
>   From: Santosh Marella 
>  To: dev@myriad.incubator.apache.org
>  Sent: Wednesday, December 16, 2015 9:47 AM
>  Subject: Next dev sync hangout will be on 1/6/2016
>
> We have decided to hold the next dev sync on 1/6/2016 (instead of
> 12/30/2015).
>
> Meeting notes from today's hangout:
>
> https://docs.google.com/document/d/1JGmJrgeg98bHw_0_sSRmyX6WiAe13OdErcFlaz6Aa04/edit#
>
> Thanks,
> Santosh
>
>
>

Re: Myriad is 0.1.0

[Darin Johnson]

Sweet, thanks for all the work on the release guys!

On Thu, Dec 10, 2015 at 1:31 PM, mohit soni  wrote:

> WooHoo! Perfect holiday gift. Thanks everyone.
>
> On Thu, Dec 10, 2015 at 10:19 AM, Aashreya Shankar 
> wrote:
>
> > This is great !
> > Good job everyone.
> >
> > On Thu, Dec 10, 2015 at 8:02 AM, Yuliya 
> > wrote:
> >
> > > Great news!!!
> > >
> > > We are finally there
> > >
> > >
> > > > On Dec 10, 2015, at 1:13 AM, Santosh Marella 
> > > wrote:
> > > >
> > > > Hi All,
> > > >
> > > >  Congratulations on a the first Apache Myriad release..! Kudos to
> > > everyone
> > > > involved for making this happen.
> > > >
> > > >  As we now have IPMC's approval, there are a few things that I did to
> > > wrap
> > > > up the release:
> > > >  - Make 0.1.0 artifacts available from the release SVN repo [1].
> > > >  - Git tag the voted RC as the "0.1.0" release [2].
> > > >  - Delete the previously marked git RC tags.
> > > >  - Closed the remaining JIRAs marked for 0.1.0 version and marked the
> > > > 0.1.0 version as "released" [3].
> > > >  - Submitted a PR [4] with scripts to prepare a RC and release a RC
> > > > (automates the above git and svn steps)
> > > >  - Updated the release guide [5] with voting links to help future
> > release
> > > > managers.
> > > >
> > > >  Here are a couple of things still remaining:
> > > >  - Update the downloads page on Myriad's website with links to 0.1.0
> > > > artifacts on svn, git tag, release notes.
> > > >  - Do an announcement blog post. Here is a draft [6]. Please suggest
> > any
> > > > changes.
> > > >
> > > >  If I may be missing something for 0.1.0, appreciate if you could
> bring
> > > to
> > > > my notice.
> > > >
> > > >   1.
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/release/incubator/myriad/myriad-0.1.0-incubating/
> > > >   2.
> > > >
> > >
> >
> https://github.com/apache/incubator-myriad/releases/tag/myriad-0.1.0-incubating
> > > >   3.
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/MYRIAD/?selectedTab=com.atlassian.jira.jira-projects-plugin:versions-panel
> > > >   4. https://github.com/apache/incubator-myriad/pull/53
> > > >   5.
> https://cwiki.apache.org/confluence/display/MYRIAD/Release+Guide
> > > >   6.
> > > >
> > >
> >
> https://docs.google.com/document/d/1zCXnDlqzNhj0BL_CqRz5-poCap9QFah7R3tKkHdspYg/edit
> > > >
> > > > Thanks,
> > > > Santosh
> > >
> >
>

Re: [PROPOSAL(s)] Use Release Branches, and Delete Obsolete Branches

[Darin Johnson]

My experience with alt 1 is it takes a lot of discipline or it devolves
into develop just being master.  I'd be curious how others have found it.
On Dec 3, 2015 10:07 PM, "Darin Johnson" <dbjohnson1...@gmail.com> wrote:

> +1 A, +1 B.
> On Dec 3, 2015 7:12 PM, "Sarjeet Singh" <sarjeetsi...@maprtech.com> wrote:
>
>> +1 for Proposal A -> Alt 1, and +1 for Proposal B.
>>
>> Should we also maintain 'develop' & 'master' branch as described on
>> nvie.com,
>> it was easy to read through the branching model, and understand the
>> branching flow without any complexity involved?
>>
>> Btw, Good pro/con list with references. thanks Adam!!
>>
>> -Sarjeet
>>
>> On Thu, Dec 3, 2015 at 2:49 PM, Santosh Marella <smare...@maprtech.com>
>> wrote:
>>
>> > Yup.
>> >
>> > +1 for Proposal A -> Alternative 1.
>> > +1 for Proposal B
>> >
>> > Santosh
>> >
>> > On Thu, Dec 3, 2015 at 1:03 PM, yuliya Feldman
>> <yufeld...@yahoo.com.invalid
>> > >
>> > wrote:
>> >
>> > > I fully second Todd.
>> > > Thanks,Yuliya
>> > >   From: Todd Richmond <trichm...@maprtech.com>
>> > >  To: dev@myriad.incubator.apache.org
>> > >  Sent: Thursday, December 3, 2015 8:59 AM
>> > >  Subject: Re: [PROPOSAL(s)] Use Release Branches, and Delete Obsolete
>> > > Branches
>> > >
>> > > excellent pro/con list
>> > >
>> > > +1 for either A or + .5 for Alt 1. A branch is easy to follow and
>> allows
>> > > for long term patch support. Each new 0.x.y patch release becomes the
>> > only
>> > > “supported” 0.x version but more than one “x” can be supported
>> > > simultaneously
>> > >
>> > > Alt 2 tags work best when you release very often (such as monthly) to
>> > > users who are willing to upgrade and so can quickly deprecate previous
>> > > releases. Cherry-picking is more manual effort and possibly error
>> prone
>> > as
>> > > the committer count grows
>> > >
>> > > +1 for proposal B. feature branches can usually be done on private
>> forks
>> > > instead and should definitely be removed once the feature is merged to
>> > > develop
>> > >
>> > >   Todd
>> > >
>> > >
>> > >
>> > >
>> > > > On Dec 3, 2015, at 12:36 AM, Adam Bordelon <a...@mesosphere.io>
>> wrote:
>> > > >
>> > > > Proposal A: Use Release Branches
>> > > > I propose that we create a '0.1.x' branch that will contain all of
>> the
>> > > > 0.1.0-rc tags, the final 0.1.0 release tag, and any tags related to
>> > > hotfix
>> > > > releases on top (0.1.1, 0.1.2). A hotfix release like 0.1.1 can
>> > > cherry-pick
>> > > > fixes from master directly on top of the 0.1.0 tag in the 0.1.x
>> branch,
>> > > and
>> > > > we'll iterate on its rc's in the 0.1.x branch. Development for
>> > > > features/fixes for 0.2.0 go into the master branch, and whenever
>> 0.2.0
>> > is
>> > > > feature-complete/ready, we'll cut the new '0.2.x' branch from master
>> > and
>> > > > tag a 0.2.0-rc1, then cherry pick any necessary fixes from master
>> into
>> > > > 0.2.x, for future release candidates and hotfix releases.
>> > > > + Easy to create/review github PRs to merge fixes into release
>> > candidates
>> > > > and hotfix releases.
>> > > > + Release candidates and hotfixes are handled in the appropriate
>> > release
>> > > > branch, while normal development can continue in master.
>> > > > + Minimal additional branches/commands required; no need for
>> ephemeral
>> > > > branches for each release (candidate).
>> > > >
>> > > > Alternative 1: Follow
>> > > >
>> >
>> http://nvie.com/posts/a-successful-git-branching-model/#release-branches
>> > > > My proposal is similar to the model described by nvie except that we
>> > use
>> > > > the myriad 'master' branch for what he calls the 'develop' branch,
>> and
>> > we
>> > > > use our 0.1.x,0.2.x release branches as longer-lived branches for
>> > hotfix
>> > > > releases. (Note: Feature branches are a separ

Re: [PROPOSAL(s)] Use Release Branches, and Delete Obsolete Branches

[Darin Johnson]

+1 A, +1 B.
On Dec 3, 2015 7:12 PM, "Sarjeet Singh"  wrote:

> +1 for Proposal A -> Alt 1, and +1 for Proposal B.
>
> Should we also maintain 'develop' & 'master' branch as described on
> nvie.com,
> it was easy to read through the branching model, and understand the
> branching flow without any complexity involved?
>
> Btw, Good pro/con list with references. thanks Adam!!
>
> -Sarjeet
>
> On Thu, Dec 3, 2015 at 2:49 PM, Santosh Marella 
> wrote:
>
> > Yup.
> >
> > +1 for Proposal A -> Alternative 1.
> > +1 for Proposal B
> >
> > Santosh
> >
> > On Thu, Dec 3, 2015 at 1:03 PM, yuliya Feldman
>  > >
> > wrote:
> >
> > > I fully second Todd.
> > > Thanks,Yuliya
> > >   From: Todd Richmond 
> > >  To: dev@myriad.incubator.apache.org
> > >  Sent: Thursday, December 3, 2015 8:59 AM
> > >  Subject: Re: [PROPOSAL(s)] Use Release Branches, and Delete Obsolete
> > > Branches
> > >
> > > excellent pro/con list
> > >
> > > +1 for either A or + .5 for Alt 1. A branch is easy to follow and
> allows
> > > for long term patch support. Each new 0.x.y patch release becomes the
> > only
> > > “supported” 0.x version but more than one “x” can be supported
> > > simultaneously
> > >
> > > Alt 2 tags work best when you release very often (such as monthly) to
> > > users who are willing to upgrade and so can quickly deprecate previous
> > > releases. Cherry-picking is more manual effort and possibly error prone
> > as
> > > the committer count grows
> > >
> > > +1 for proposal B. feature branches can usually be done on private
> forks
> > > instead and should definitely be removed once the feature is merged to
> > > develop
> > >
> > >   Todd
> > >
> > >
> > >
> > >
> > > > On Dec 3, 2015, at 12:36 AM, Adam Bordelon 
> wrote:
> > > >
> > > > Proposal A: Use Release Branches
> > > > I propose that we create a '0.1.x' branch that will contain all of
> the
> > > > 0.1.0-rc tags, the final 0.1.0 release tag, and any tags related to
> > > hotfix
> > > > releases on top (0.1.1, 0.1.2). A hotfix release like 0.1.1 can
> > > cherry-pick
> > > > fixes from master directly on top of the 0.1.0 tag in the 0.1.x
> branch,
> > > and
> > > > we'll iterate on its rc's in the 0.1.x branch. Development for
> > > > features/fixes for 0.2.0 go into the master branch, and whenever
> 0.2.0
> > is
> > > > feature-complete/ready, we'll cut the new '0.2.x' branch from master
> > and
> > > > tag a 0.2.0-rc1, then cherry pick any necessary fixes from master
> into
> > > > 0.2.x, for future release candidates and hotfix releases.
> > > > + Easy to create/review github PRs to merge fixes into release
> > candidates
> > > > and hotfix releases.
> > > > + Release candidates and hotfixes are handled in the appropriate
> > release
> > > > branch, while normal development can continue in master.
> > > > + Minimal additional branches/commands required; no need for
> ephemeral
> > > > branches for each release (candidate).
> > > >
> > > > Alternative 1: Follow
> > > >
> > http://nvie.com/posts/a-successful-git-branching-model/#release-branches
> > > > My proposal is similar to the model described by nvie except that we
> > use
> > > > the myriad 'master' branch for what he calls the 'develop' branch,
> and
> > we
> > > > use our 0.1.x,0.2.x release branches as longer-lived branches for
> > hotfix
> > > > releases. (Note: Feature branches are a separate discussion,
> unrelated
> > to
> > > > release management.)
> > > > + Easy to follow guide.
> > > > + Good separation of concerns/responsibility.
> > > > - Doesn't explain how release candidates are handled.
> > > > - So many branches.
> > > >
> > > > Alternative 2: Use tags for releases, no branches (like Mesos does)
> > > > See the discussion at:
> > > >
> > >
> >
> http://stackoverflow.com/questions/9810050/git-tag-vs-release-beta-branches
> > > > + No mess of branches in the repo; no merging between branches.
> > > > + Since release candidates and releases are cherry-picked and tagged,
> > > > normal development can continue on master without
> > > interruption/corruption.
> > > > - Github PRs cannot use a tag (Dealbreaker?).
> > > > http://stackoverflow.com/a/12279290/4056606
> > > >
> > > > Please let me know your thoughts on release branches. I went ahead
> and
> > > > created the '0.1.x' branch from the 0.1.0-rc3 tag so you can check it
> > out
> > > > and play around, and so you can push 0.2.0 features to master without
> > > > worrying about messing up the 0.1.0 release. We can cherry-pick any
> > > > rc4/0.1.1 patches out of master, and we can always
> delete/rename/reorg
> > > the
> > > > release branch later if desired.
> > > >
> > >
> >
> https://git-wip-us.apache.org/repos/asf?p=incubator-myriad.git;a=shortlog;h=refs/heads/0.1.x
> > > >
> > > >
> > > >
> > > > Proposal B: Delete all these obsolete branches from the Apache git
> > repo:
> > > > 9/23/15 phase1 (72 behind 

Re: Merges to Myriad master branch

[Darin Johnson]

I like Adam's idea for RCs, but I also really like the idea of a 0.1.0
branch for bug fixes.  That way we can cut a 0.1.1 maintenance release much
easier than trying to cut off master.  Seems like a lot of apache projects
handle it that way.

Darin

Otherwise there will likely be some really ugly merges.
On Mon, Nov 30, 2015 at 1:49 PM, Adam Bordelon  wrote:

> Sounds like a code freeze/thaw. What are the conditions for a *major* PR?
> Even a minor PR could introduce major bugs.
> I will point out that you have the option of cherry-picking specific new
> patches on top of the 0.1.0-rc3 tag to create a new rc. This ensures that
> 0.1.0 only includes changes that were tested in the previous rc's plus
> specific critical fixes. This is how Mesos handles patch releases (e.g.
> 0.23.0 -> 0.23.1) or release candidates after the first. Cut the rc0/1
from
> HEAD, then cherry-pick on top for all future rcs.
>
>
>
>
Do you mean cherry-pick into branches (e.g. 0.1.x branch) instead of tag
which is supposed to be immutable ? Having a branch also enables future
releases based on  0.1.0 release.



--
Luciano Resende
http://people.apache.org/~lresende
http://twitter.com/lresende1975
http://lresende.blogspot.com/

Re: [VOTE] Release apache-myriad-0.1.0-incubating (release candidate 3)

[Darin Johnson]

+1
D/L'd built and ran and 4 node vanilla hadoop cluster (remote distro).
verified md5 and sha hashes
Ran M/R Job on CGS/FGS
Flexup and down nodes.
killed RM with HA enabled verified nodes up

noticed if JHS is up in HA mode and the RM is restarted a new JHS is
launched resulting in two JHS running (Minor should fix in next release).
noticed if HA is enabled and the the framework time expires you must
manually delete the statestore (Minor, should add documentation).

On Mon, Nov 23, 2015 at 10:34 PM, Sarjeet Singh 
wrote:

> +1 (Non-Binding)
>
> Verified md5 & sha512 checksums.
> D/L myriad-0.1.0-incubating-rc3.tar.gz, install gradle, Compiled code &
> deployed it on a 4 node MapR cluster.
> Tried FGS/CGS NM flex up/down, and ran hadoop M/R jobs.
> Tried myriad HA with RM restart/kill.
> Tried framework shutdown, and start myriad again.
> Tried JHS configuration flex up/down and its functionality.
>
> -Sarjeet
>
> On Thu, Nov 19, 2015 at 10:37 PM, Santosh Marella 
> wrote:
>
> > Hi All,
> >
> > Firstly, thanks everyone for the valuable contributions to the project
> and
> > for holding on tight as we move along the release process. We're almost
> > home!
> >
> > I have created a source tar ball for Apache Myriad 0.1.0-incubating,
> > release candidate 3. This includes the feedback from the recent IPMC
> > voting.
> > Here’s the release notes:
> > https://cwiki.apache.org/confluence/display/MYRIAD/Release+Notes
> >
> > The commit to be voted upon is tagged with "myriad-0.1.0-incubating-rc3"
> > and is available here:
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=incubator-myriad.git;a=shortlog;h=refs/tags/myriad-0.1.0-incubating-rc3
> >
> > The artifacts to be voted upon are located below. Please note that this
> is
> > a source release:
> >
> >
> https://dist.apache.org/repos/dist/dev/incubator/myriad/myriad-0.1.0-incubating-rc3/
> >
> > Release artifacts are signed with the following key:
> > https://people.apache.org/keys/committer/smarella.asc
> >
> > **Please note that the release tar ball does not include the gradlew
> script
> > to build. You need to generate one in order to build.**
> >
> > Please try out the release candidate and vote. The vote is open for a
> > minimum of 72 hours or until the necessary number of votes (3 binding
> +1s)
> > is reached.
> >
> > If/when this vote succeeds, I will call for a vote with IPMC seeking
> > permission to release RC3 as Apache Myriad 0.1.0 (incubating).
> >
> > [ ] +1 Release this package as Apache Myriad 0.1.0-incubating
> > [ ]  0 I don't feel strongly about it, but I'm okay with the release
> > [ ] -1 Do not release this package because...
> >
> > Thanks,
> > Santosh
> >
>

Re: [VOTE] Release apache-myriad-0.1.0-incubating (release candidate 3)

[Darin Johnson]

Any guidance on why we can't include the gradlew script?  I took a look at
Apache Kafka and Apache Tapestry, their artifacts both include gradlew,
just not the jar.

Meanwhile I'll check the release ...

Darin

On Mon, Nov 23, 2015 at 4:55 PM, yuliya Feldman  wrote:

> When I download release gradlew file is not there, so I can not build
> Am I missing anything?
> Thanks,Yuliya
>   From: Santosh Marella 
>  To: dev@myriad.incubator.apache.org
>  Sent: Monday, November 23, 2015 12:36 PM
>  Subject: Re: [VOTE] Release apache-myriad-0.1.0-incubating (release
> candidate 3)
>
> +1 (binding)
>
> - Downloaded the RC
> - Verified signatures
> - Built binaries by following instructions from the README page [1]
> - Deployed binaries on a 2.7.0 MapR cluster as described in the
> documentation [2]
> - Verified the following:
>   - flexup/flexdown NMs of zero/low/medium profiles from Myriad's Web UI.
>   - Successfully ran a 10G terasort M/R job with 60 mappers and 10 reducers
>   - Myriad/RM HA: Killed RM and restarted it while a M/R job is running.
>   - Verified the job completed successfully.
>   - Verified that the Myriad UI showed the active NM tasks correctly.
>   - Verified "Shutdown Framework"
>
> [1] https://github.com/apache/incubator-myriad#build-myriad
> [2]
>
> https://github.com/apache/incubator-myriad/blob/master/docs/myriad-dev.md#step-2-deploy-the-myriad-binaries
>
> Santosh
>
>
>
> On Thu, Nov 19, 2015 at 10:37 PM, Santosh Marella 
> wrote:
>
> > Hi All,
> >
> > Firstly, thanks everyone for the valuable contributions to the project
> and
> > for holding on tight as we move along the release process. We're almost
> > home!
> >
> > I have created a source tar ball for Apache Myriad 0.1.0-incubating,
> > release candidate 3. This includes the feedback from the recent IPMC
> > voting.
> > Here’s the release notes:
> > https://cwiki.apache.org/confluence/display/MYRIAD/Release+Notes
> >
> > The commit to be voted upon is tagged with "myriad-0.1.0-incubating-rc3"
> > and is available here:
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=incubator-myriad.git;a=shortlog;h=refs/tags/myriad-0.1.0-incubating-rc3
> >
> > The artifacts to be voted upon are located below. Please note that this
> > is a source release:
> >
> >
> https://dist.apache.org/repos/dist/dev/incubator/myriad/myriad-0.1.0-incubating-rc3/
> >
> > Release artifacts are signed with the following key:
> > https://people.apache.org/keys/committer/smarella.asc
> >
> > **Please note that the release tar ball does not include the gradlew
> > script to build. You need to generate one in order to build.**
> >
> > Please try out the release candidate and vote. The vote is open for a
> > minimum of 72 hours or until the necessary number of votes (3 binding
> > +1s) is reached.
> >
> > If/when this vote succeeds, I will call for a vote with IPMC seeking
> > permission to release RC3 as Apache Myriad 0.1.0 (incubating).
> >
> > [ ] +1 Release this package as Apache Myriad 0.1.0-incubating
> > [ ]  0 I don't feel strongly about it, but I'm okay with the release
> > [ ] -1 Do not release this package because...
> >
> > Thanks,
> > Santosh
> >
>
>
>

Re: Struggling with Permissions

[Darin Johnson]

Yuliya: Are you referencing yarn.nodemanager.hostname or a mapr specific
option?

I'm working right now on passing a
-Dyarn.nodemanager.hostname=offer.getHostName().  Useful if you've got
extra ip's for a san or management network.

John: Yeah the permissions on the tarball are a pain to get right.  I'm
working on Docker Support and a build script for the tarball, which should
make things easier.  Also, to the point of using world writable directories
it's a little scary from the security side of things to allow executables
to run there, especially things running as privileged users.  Many distro's
of linux will mount /tmp noexec.

Darin

On Tue, Nov 17, 2015 at 2:53 PM, yuliya Feldman  wrote:

> Please change workdir directory for mesos slave to one that is not /tmp
> and make sure that dir is owned by root.
> There is one more caveat with binary distro and MapR - in Myriad code for
> binary distro configuration is copied from RM to NMs - it doe snot work for
> MapR since we need hostname (yes for the sake of local volumes) to be
> unique.
> MapR will have Myriad release to handle this situation.
>   From: John Omernik 
>  To: dev@myriad.incubator.apache.org
>  Sent: Tuesday, November 17, 2015 11:37 AM
>  Subject: Re: Struggling with Permissions
>
> Oh hey, I found a post by me back on Sept 9.  I looked at the Jiras and
> followed the instructions with the same errors. At this point do I still
> need to have a place where the entire path is owned by root? That seems
> like a an odd requirement (a changed of each node to facilitate a
> framework)
>
>
>
>
>
> On Tue, Nov 17, 2015 at 1:25 PM, John Omernik  wrote:
>
> > Hey all, I am struggling with permissions on myriad, trying to get the
> > right permissions in the tgz as well as who to run as.  I am running in
> > MapR, which means I need to run as mapr or root (otherwise my volume
> > creation scripts will fail on MapR, MapR folks, we should talk more about
> > those scripts)
> >
> > But back to the code, I've had lots issues. When I run the Frameworkuser
> > and Superuser as mapr, it unpacks everything as MapR and I get a
> > "/bin/container-executor" must be owned by root but is owned by 700 (my
> > mapr UID).
> >
> > So now I am running as root, and I am getting the error below as it
> > relates to /tmp. I am not sure which /tmp this refers to. the /tmp that
> my
> > slave is executing in? (i.e. my local mesos agent /tmp directory) or my
> > MaprFS /tmp directory (both of which are world writable, as /tmp
> typically
> > is... or am I mistaken here?)
> >
> > Any thoughts on how to get this to resolve? This is when nodemanager is
> > trying to start running as root and root for both of my Myriad users.
> >
> > Thanks!
> >
> >
> > Caused by: ExitCodeException exitCode=24: File /tmp must not be world or
> group writable, but is 1777
> >
> >
> >
> >
>
>
>
>

Re: Struggling with Permissions

[Darin Johnson]

John,

I'm not super familiar with MapR, but I think I might have some thought and
the the MapR people can chime it :).

I think the mapr.host thing is due to the fact in the remote distribution,
Myriad pulls it's config from the resource manager.  As I mentioned in my
note to Yuyila, I'm working on adding the ability to add
yarn.nodemanager.hostname as a -D option, I think the right thing may by to
expose an environment variable $HOSTNAME and then in yarnEnvironment: you
could set a YARN_OPTS=-Dmapr.hostname=$HOSTNAME
-Dyarn.nodemanager.hostname=$HOSTNAME ... option.

One could imagine a similar option for ports as this is kind of what
Marathon does.

Maybe best to JIRA this, as I don't think we necessarily expose a lot of
things we should just yet.


On Tue, Nov 17, 2015 at 4:41 PM, John Omernik <j...@omernik.com> wrote:

> What's even stranger is I can't for life of me find where "mapr.host" gets
> set or used.  I did a grep -P -R "mapr\.host" ./*  in /opt/mapr (which
> included me pulling down the myriad code into
> /opt/mapr/myriad/incubator-myriad) and found only one reference in
> /opt/mapr/server/mapr_yarn_install.sh
>
> 
>
>   yarn.nodemanager.hostname
>
>   \${mapr.host}
>
> " | sudo tee -a ${YARN_CONF_FILE}
>
>
> But I don't think that is being called at all by the resource manager...
>
>
> (Note when I create my tarball from /opt/mapr/hadoop/hadoop-2.7.0 directory
> I am using tar -zcfhp  to both preserver permissions and include the files
> that symlinked... not sure if that affects things here.. )
>
>
>
>
>
> On Tue, Nov 17, 2015 at 3:15 PM, John Omernik <j...@omernik.com> wrote:
>
> > Well sure /tmp is world writeable but /tmp/mesos is not world writable
> > thus there is a sandbox to play in there... or am I missing something.
> Not
> > to mention my tmp is rwt which is world writable but only the creator or
> > root can modify (based on the googles).
> > Yuliya:
> >
> > I am seeing a weird behavior with MapR as it relates to (I believe) the
> > mapr_direct_shuffle.
> >
> > In the Node Manager logs, I see things starting and it saying "Checking
> > for local volume, if local volume is not present command will create and
> > mount it"
> >
> > Command invoked is : /opt/mapr/server/createTTVolume.sh
> > hadoopmapr7.brewingintel.com /var/mapr/local/
> > hadoopmapr2.brewingintel.com/mapred /var/mapr/local/
> > hadoopmapr2.brewingintel.com/mapred/nodeManager yarn
> >
> >
> > What is interesting here is hadoopmapr7 is the nodemanager it's trying to
> > start on, however the mount point it's trying to create is hadoopmapr2
> > which is the node the resource manager happened to fall on...  I was very
> > confused by that because in no place should hadoopmapr2 be "known" to the
> > nodemanager, because it thinks the resource manager hostname is
> > myriad.marathon.mesos.
> >
> > So why was it hard coding to the node the resource manager is running on?
> >
> > Well if I look at the conf file in the sandbox (the file that gets copied
> > to be yarn-site.xml for node managers.  There ARE four references the
> > hadoopmapr2. Three of the four say "source programatically" and one is
> just
> > set... that's mapr.host.  Could there be some down stream hinkyness going
> > on with how MapR is setting hostnames?  All of these variables seem
> "wrong"
> > in that mapr.host (on the node manager) should be hadoopmapr7 in this
> case,
> > and the resource managers should all be myriad.marathon.mesos.   I'd be
> > interested in your thoughts here, because I am stumped at how these are
> > getting set.
> >
> >
> >
> >
> >
> yarn.resourcemanager.addresshadoopmapr2:8032programatically
> > mapr.hosthadoopmapr2.brewingintel.com
> > 
> >
> >
> yarn.resourcemanager.resource-tracker.addresshadoopmapr2:8031programatically
> >
> >
> yarn.resourcemanager.admin.addresshadoopmapr2:8033programatically
> >
> >
> >
> >
> >
> > On Tue, Nov 17, 2015 at 2:51 PM, Darin Johnson <dbjohnson1...@gmail.com>
> > wrote:
> >
> >> Yuliya: Are you referencing yarn.nodemanager.hostname or a mapr specific
> >> option?
> >>
> >> I'm working right now on passing a
> >> -Dyarn.nodemanager.hostname=offer.getHostName().  Useful if you've got
> >> extra ip's for a san or management network.
> >>
> >> John: Yeah the permissions on the tarball are a pain to get right.  I'm
> >> working on Docker Support and a build script for the tarball, which
> shoul

Re: [VOTE] Release apache-myriad-0.1.0-incubating (release candidate 2)

[Darin Johnson]

I agree with Adam, it's really hard for to find blocks of time on the
weekends with so little notice.

Darin

On Sun, Nov 15, 2015 at 8:01 PM, Adam Bordelon <a...@mesosphere.io> wrote:

> In the future, let's try to allow 3 business days for votes, rather than
> just 72 hours. This gives people more of a chance to test and vote.
>
> That said, I think we can call this a successful vote, since there were 3
> binding +1s and no -1s. Also, we got additional binding +1s on the previous
> release candidate, which was not substantially different. Next we'll have
> to get the IPMC to vote on it.
>
> Great work, team!
>
> On Sun, Nov 15, 2015 at 4:13 PM, Darin Johnson <dbjohnson1...@gmail.com>
> wrote:
>
> > +1
> > On Nov 15, 2015 6:33 PM, "yuliya Feldman" <yufeld...@yahoo.com.invalid>
> > wrote:
> >
> > > Thank you Jim
> > > You are not late - it is still 3:30 PM PDT :)
> > > Committers - we need at least one more vote from you for RC2 - you have
> > > 1.5 hours left.
> > > Thanks,Yuliya
> > >   From: Jim Klucar <klu...@gmail.com>
> > >  To: "dev@myriad.incubator.apache.org" <
> dev@myriad.incubator.apache.org>
> > >  Sent: Sunday, November 15, 2015 3:12 PM
> > >  Subject: Re: [VOTE] Release apache-myriad-0.1.0-incubating (release
> > > candidate 2)
> > >
> > > Even though I'm late, I had no problems.
> > >
> > > +1 NB
> > >
> > > On Saturday, November 14, 2015, Santosh Marella <smare...@maprtech.com
> >
> > > wrote:
> > >
> > > > A friendly reminder that the vote ends at 4:55 PM (PDT) on Sunday
> > > > (tomorrow), 11/15.
> > > >
> > > > Thanks,
> > > > Santosh
> > > >
> > > > On Fri, Nov 13, 2015 at 5:17 PM, Sarjeet Singh <
> > > sarjeetsi...@maprtech.com
> > > > <javascript:;>>
> > > > wrote:
> > > >
> > > > > +1 (Non-Binding)
> > > > >
> > > > > Verified checksums.
> > > > > D/L myriad-0.1.0-incubating-rc2.tar.gz, Compiled & deployed it on
> a 4
> > > > node
> > > > > MapR cluster.
> > > > > Tried FGS/CGS flex up/down, and ran hadoop M/R jobs.
> > > > > Tried myriad HA with RM restart and kill -9.
> > > > > Tried framework shutdown, and restart myriad again.
> > > > > Tried JHS configuration flex up/down and its functionality.
> > > > >
> > > > > -Sarjeet
> > > > >
> > > > > On Fri, Nov 13, 2015 at 4:17 PM, Aashreya Shankar <
> > > ashan...@maprtech.com
> > > > <javascript:;>>
> > > > > wrote:
> > > > >
> > > > > > +1 (non binding)
> > > > > >
> > > > > > Successfully built binaries from rc2 tar.gz
> > > > > > Tried it 5 node MapR cluster
> > > > > > Flex up/down works accordingly
> > > > > > Hadoop jobs running fine.
> > > > > > Build was successful through Vagrant
> > > > > >
> > > > > > Thank you
> > > > > > Aashreya
> > > > > >
> > > > > > On Fri, Nov 13, 2015 at 3:13 PM, Swapnil Daingade <
> > > > > > swapnil.daing...@gmail.com <javascript:;>> wrote:
> > > > > >
> > > > > > > Downloaded rc2 tar.gz
> > > > > > > * Verified md5 and sha512 hashes successfully
> > > > > > > * Built binaries successfully
> > > > > > > * Deployed on 3 node MapR cluster
> > > > > > > * Tested NM flexup/flexdown with HA enabled and disabled.
> > > > > > > * Tried HA
> > > > > > > * Tried Framework Shutdown.
> > > > > > > All operations worked as expected.
> > > > > > >
> > > > > > > +1
> > > > > > >
> > > > > > > Regards
> > > > > > > Swapnil
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Nov 12, 2015 at 4:55 PM, Santosh Marella <
> > > > > smare...@maprtech.com <javascript:;>>
> > >
> > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > &g

Re: [VOTE] Release apache-myriad-0.1.0-incubating (release candidate 2)

[Darin Johnson]

+1
On Nov 15, 2015 6:33 PM, "yuliya Feldman" 
wrote:

> Thank you Jim
> You are not late - it is still 3:30 PM PDT :)
> Committers - we need at least one more vote from you for RC2 - you have
> 1.5 hours left.
> Thanks,Yuliya
>   From: Jim Klucar 
>  To: "dev@myriad.incubator.apache.org" 
>  Sent: Sunday, November 15, 2015 3:12 PM
>  Subject: Re: [VOTE] Release apache-myriad-0.1.0-incubating (release
> candidate 2)
>
> Even though I'm late, I had no problems.
>
> +1 NB
>
> On Saturday, November 14, 2015, Santosh Marella 
> wrote:
>
> > A friendly reminder that the vote ends at 4:55 PM (PDT) on Sunday
> > (tomorrow), 11/15.
> >
> > Thanks,
> > Santosh
> >
> > On Fri, Nov 13, 2015 at 5:17 PM, Sarjeet Singh <
> sarjeetsi...@maprtech.com
> > >
> > wrote:
> >
> > > +1 (Non-Binding)
> > >
> > > Verified checksums.
> > > D/L myriad-0.1.0-incubating-rc2.tar.gz, Compiled & deployed it on a 4
> > node
> > > MapR cluster.
> > > Tried FGS/CGS flex up/down, and ran hadoop M/R jobs.
> > > Tried myriad HA with RM restart and kill -9.
> > > Tried framework shutdown, and restart myriad again.
> > > Tried JHS configuration flex up/down and its functionality.
> > >
> > > -Sarjeet
> > >
> > > On Fri, Nov 13, 2015 at 4:17 PM, Aashreya Shankar <
> ashan...@maprtech.com
> > >
> > > wrote:
> > >
> > > > +1 (non binding)
> > > >
> > > > Successfully built binaries from rc2 tar.gz
> > > > Tried it 5 node MapR cluster
> > > > Flex up/down works accordingly
> > > > Hadoop jobs running fine.
> > > > Build was successful through Vagrant
> > > >
> > > > Thank you
> > > > Aashreya
> > > >
> > > > On Fri, Nov 13, 2015 at 3:13 PM, Swapnil Daingade <
> > > > swapnil.daing...@gmail.com > wrote:
> > > >
> > > > > Downloaded rc2 tar.gz
> > > > > * Verified md5 and sha512 hashes successfully
> > > > > * Built binaries successfully
> > > > > * Deployed on 3 node MapR cluster
> > > > > * Tested NM flexup/flexdown with HA enabled and disabled.
> > > > > * Tried HA
> > > > > * Tried Framework Shutdown.
> > > > > All operations worked as expected.
> > > > >
> > > > > +1
> > > > >
> > > > > Regards
> > > > > Swapnil
> > > > >
> > > > >
> > > > > On Thu, Nov 12, 2015 at 4:55 PM, Santosh Marella <
> > > smare...@maprtech.com >
>
>
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I have created a build for Apache Myriad 0.1.0-incubating,
> release
> > > > > > candidate 2.
> > > > > >
> > > > > > Thanks to everyone who has contributed to this release.
> > > > > >
> > > > > > Here’s the release notes:
> > > > > > https://cwiki.apache.org/confluence/display/MYRIAD/Release+Notes
> > > > > >
> > > > > > The commit to be voted upon is tagged with
> > > > "myriad-0.1.0-incubating-rc2"
> > > > > > and is available here:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://git1-us-west.apache.org/repos/asf/incubator-myriad/repo?p=incubator-myriad.git;a=commit;h=fb93291e9377cccf625bed93a9ad1ae1c4b76529
> > > > > >
> > > > > > The artifacts to be voted upon are located here:
> > > > > > *
> > > > > >
> > > > >
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/incubator/myriad/myriad-0.1.0-incubating-rc2/
> > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/incubator/myriad/myriad-0.1.0-incubating-rc2/
> > > > > > >*
> > > > > >
> > > > > > Release artifacts are signed with the following key:
> > > > > > https://people.apache.org/keys/committer/smarella.asc
> > > > > >
> > > > > > Please vote on releasing this package as Apache Myriad
> > > > 0.1.0-incubating.
> > > > > >
> > > > > > The vote is open for the next 72 hours and passes if a majority
> of
> > > > > > at least three +1 PPMC votes are cast.
> > > > > >
> > > > > > [ ] +1 Release this package as Apache Myriad 0.1.0-incubating
> > > > > > [ ]  0 I don't feel strongly about it, but I'm okay with the
> > release
> > > > > > [ ] -1 Do not release this package because...
> > > > > >
> > > > > > Here is my vote:
> > > > > > +1 (binding)
> > > > > >
> > > > > > Thanks,
> > > > > > Santosh
> > > > > >
> > > > >
> > > >
> > >
> >
>
>

Re: [VOTE] Release apache-myriad-0.1.0-incubating (release candidate 1)

[Darin Johnson]

That's a good idea.

On Wed, Nov 11, 2015 at 10:03 AM, Jim Klucar <klu...@gmail.com> wrote:

> 0 (non-binding)
>
> Vagrant environment is broken.
> I did a `vagrant up` and ran the setup-yarn-1.sh and setup-yarn-2.sh
> scripts. The first had a slight problem, the second failed.
> I then tried `./gradlew build` from inside vagrant and the build failed in
> the web-ui. I believe it is due to how vagrant maps things to /vagrant but
> didn't really dig into it. It builds fine on my local machine.
>
> I recommend removing the Vagrantfile and the setup-yarn-* scripts and
> releasing. We can then decide to revamp or permanently remove the Vagrant
> setup for a separate release.
>
>
>
> On Tue, Nov 10, 2015 at 10:42 PM, Darin Johnson <dbjohnson1...@gmail.com>
> wrote:
>
> > +1
> > D/L'd tar ball verified checksums
> > Flexed up/down nodes and JHS
> > Ran MR job with FGS
> >
> >
> >
> > On Tue, Nov 10, 2015 at 9:12 PM, Sarjeet Singh <
> sarjeetsi...@maprtech.com>
> > wrote:
> >
> > > +1 (Non-Binding)
> > >
> > > Verified checksums.
> > > Downloaded myriad-0.1.0-incubating-rc1.tar.gz, Compiled the code and
> > > deployed it on a 4 node MapR cluster.
> > > Tried basic functionality tests for FGS/CGS flex up/down and it worked
> > > fine.
> > > Tried running M/R job and it completed successfully.
> > > Tried framework shutdown, shutdown went smooth.
> > > Tried JHS configuration and service flex-up, and it worked fine.
> > >
> > >
> > > Thanks,
> > > Sarjeet Singh
> > >
> > > On Tue, Nov 10, 2015 at 3:05 PM, Santosh Marella <
> smare...@maprtech.com>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I have created a build for Apache Myriad 0.1.0-incubating, release
> > > > candidate 1.
> > > >
> > > > Thanks to everyone who has contributed to this release.
> > > >
> > > > Here’s the release notes:
> > > > https://cwiki.apache.org/confluence/display/MYRIAD/Release+Notes
> > > >
> > > > The commit to be voted upon is tagged with
> > "myriad-0.1.0-incubating-rc1"
> > > > and is available here:
> > > >
> > > >
> > >
> >
> https://git1-us-west.apache.org/repos/asf/incubator-myriad/repo?p=incubator-myriad.git;a=commit;h=9f0fa15bfaa4fdc309ada27126567a2aa5bf296b
> > > >
> > > > The artifacts to be voted upon are located here:
> > > > *
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/incubator/myriad/myriad-0.1.0-incubating-rc1/
> > > > <
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/incubator/myriad/myriad-0.1.0-incubating-rc1/
> > > > >*
> > > >
> > > > Release artifacts are signed with the following key:
> > > > https://people.apache.org/keys/committer/smarella.asc
> > > >
> > > > Please vote on releasing this package as Apache Myriad
> > 0.1.0-incubating.
> > > >
> > > > The vote is open for the next 72 hours and passes if a majority of
> > > > at least three +1 PPMC votes are cast.
> > > >
> > > > [ ] +1 Release this package as Apache Myriad 0.1.0-incubating
> > > > [ ]  0 I don't feel strongly about it, but I'm okay with the release
> > > > [ ] -1 Do not release this package because...
> > > >
> > > > Here is my vote:
> > > > +1 (binding)
> > > >
> > > > Thanks,
> > > > Santosh
> > > >
> > >
> >
>

Re: New Committer: Swapnil Daingade

[Darin Johnson]

Congrats Swapnil!

On Thu, Nov 5, 2015 at 11:16 AM, Sarjeet Singh 
wrote:

> Congrats Swapnil, Very nice work on HA :)
>
> On Thu, Nov 5, 2015 at 8:13 AM, Santosh Marella 
> wrote:
>
> > Congratulations Swapnil.
> >
> > --
> > Sent from mobile
> > On Nov 5, 2015 7:31 AM, "yuliya Feldman" 
> > wrote:
> >
> > > Congratulations Swapnil!!!
> > > Well done
> > > Yuliya
> > >   From: Adam Bordelon 
> > >  To: dev@myriad.incubator.apache.org
> > >  Sent: Thursday, November 5, 2015 4:38 AM
> > >  Subject: New Committer: Swapnil Daingade
> > >
> > > The Podling Project Management Committee (PPMC) for Apache Myriad has
> > asked
> > > Swapnil Daingade to become a committer and PPMC member and we are
> pleased
> > > to announce that he has accepted.
> > >
> > > Please join me in welcoming Swapnil as a Myriad committer, and let's
> > thank
> > > him for all his contributions so far. Looking forward to more!
> > >
> > > Cheers,
> > > -Adam-
> > >
> > >
> > >
> >
>

Re: JIRA work for 0.1.0

[Darin Johnson]

yuliya, jim's right this belongs on JIRA, I only commented here as you
mentioned it.  Let's continue there.

On Fri, Oct 23, 2015 at 1:38 PM, yuliya Feldman <yufeld...@yahoo.com.invalid
> wrote:

> Do you have other suggestions?
> I understand that there might be clashes even with randomization, but it
> is much lower chance then consistently getting the same port w/o
> randomization.
> Thanks,Yuliya  From: Darin Johnson <dbjohnson1...@gmail.com>
>  To: Dev <dev@myriad.incubator.apache.org>; yuliya Feldman <
> yufeld...@yahoo.com>
>  Sent: Friday, October 23, 2015 9:57 AM
>  Subject: Re: JIRA work for 0.1.0
>
> On MYRIAD-160, it's a really bad idea to let Myriad pick a random port
> outside of Mesos as other frameworks might select that port the probability
> of that happening is 1-(number of ports requested by other
> frameworks)/(number of ports in use).  This could get near 50% and cause
> frameworks that are being good citizens to crash.
>
> I'm also not convinced randomizing the port is in fact the correct fix for
> this in the long term, as there is still a non-zero chance you'll get that
> port again.
>
> Darin
>
>
>
>
>
> On Fri, Oct 23, 2015 at 12:56 AM, yuliya Feldman <
> yufeld...@yahoo.com.invalid> wrote:
>
> > Great list.
> > I would include MYRIAD-160 to the list - I am working on that one. Of
> > course as a workaround we could not use Mesos ports and let NM ports
> > randomization kick in.I also almost done with MYRIAD-148 - it was really
> > tricky. Should submit PR tonight.
> > Thanks,Yuliya
> >  From: Darin Johnson <dbjohnson1...@gmail.com>
> >  To: Dev <dev@myriad.incubator.apache.org>
> >  Sent: Thursday, October 22, 2015 8:29 PM
> >  Subject: Re: JIRA work for 0.1.0
> >
> > I think this sounds good about right.  A few Jim marked were new
> features,
> > gotta leave something for the 0.2.0 release :).
> >
> >
> >
> >
> > On Thu, Oct 22, 2015 at 8:08 AM, Santosh Marella <smare...@maprtech.com>
> > wrote:
> >
> > > I looked at the JIRAs currently marked with fix version "myriad-0.1.0".
> > > There were 19 of them. I moved a few out. We are currently at 14.
> > >
> > > However, IMO the show stoppers are really the following:
> > >
> > > MYRIAD-43 Replace com.ebay namespace with org.apache
> > > MYRIAD-44 Prepare for 0.1.0 release
> > > MYRIAD-98 Move from 4 spaces to 2 spaces for indentation
> > > MYRIAD-114 Automatic dashboard building
> > > MYRIAD-145 Document Myriad Release Process
> > > MYRIAD-150 Update NOTICE file
> > > MYRIAD-159 Change default mesos version to 0.24
> > >
> > > Unless anyone thinks there are other JIRAs that are show stoppers, I
> > think
> > > we should stick to the above list
> > > and cut a RC as soon as we address the above.
> > >
> > > **I'm positive the above can be fixed by early next week (10/27) and we
> > can
> > > have a RC out for voting mid next week (10/28).**
> > >
> > > If we can't fix the above JIRAs in time or if new ones come up as "show
> > > stoppers", we will have a revised date.
> > > And, of course, more fixes are welcome, as long as they can be merged
> > > before 10/27.
> > >
> > > Just to let everyone know about the Apache release process (@Adam, feel
> > > free to chime in):
> > >  - Apache requires that a RC be put out for voting on
> > dev@myriad.incubator
> > > for 72 hrs or until 3 binding +1s and no binding -1s,
> > >  - followed by a similar voting round on general@incubator.
> > >
> > > Now to run the last mile..!
> > >
> > > Cheers,
> > > Santosh
> > >
> > > On Wed, Oct 21, 2015 at 2:48 PM, Adam Bordelon <a...@mesosphere.io>
> > wrote:
> > >
> > > > Keep in mind that Santosh (as Release Manager) has final authority
> over
> > > > pushing things out of 0.1.0.
> > > >
> > > > I created a (hopefully public) filter for Unresolved 0.1.0 Myriad
> > JIRAs:
> > > > https://issues.apache.org/jira/browse/MYRIAD-44?filter=12333786
> > > >
> > > > Maybe we should create a dashboard too, like
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12327111
> > > >
> > > > On Wed, Oct 21, 2015 at 1:38 PM, Jim Klucar <klu...@gmail.com>
> wrote:
> > > >
> > > > > I went through JIRA and assigned a bunch of the tickets to the
> Myriad
> > > > 0.1.0
> > > > > release. Some of them are really low hanging fruit, so I think we
> can
> > > > knock
> > > > > most of these out. As Adam said in the meeting, we'd rather have
> too
> > > many
> > > > > flagged for the release and pair down rather than miss some. To
> that
> > > end,
> > > > > please go through the tickets that are still unmarked and set the
> Fix
> > > > > Version/s: to Myriad 0.1.0 if you think they should be fixed or by
> > the
> > > > > release.
> > > > >
> > > > > Jim
> > > > >
> > > >
> > >
> >
> >
> >
> >
>
>
>
>

Re: JIRA work for 0.1.0

[Darin Johnson]

On MYRIAD-160, it's a really bad idea to let Myriad pick a random port
outside of Mesos as other frameworks might select that port the probability
of that happening is 1-(number of ports requested by other
frameworks)/(number of ports in use).  This could get near 50% and cause
frameworks that are being good citizens to crash.

I'm also not convinced randomizing the port is in fact the correct fix for
this in the long term, as there is still a non-zero chance you'll get that
port again.

Darin



On Fri, Oct 23, 2015 at 12:56 AM, yuliya Feldman <
yufeld...@yahoo.com.invalid> wrote:

> Great list.
> I would include MYRIAD-160 to the list - I am working on that one. Of
> course as a workaround we could not use Mesos ports and let NM ports
> randomization kick in.I also almost done with MYRIAD-148 - it was really
> tricky. Should submit PR tonight.
> Thanks,Yuliya
>   From: Darin Johnson <dbjohnson1...@gmail.com>
>  To: Dev <dev@myriad.incubator.apache.org>
>  Sent: Thursday, October 22, 2015 8:29 PM
>  Subject: Re: JIRA work for 0.1.0
>
> I think this sounds good about right.  A few Jim marked were new features,
> gotta leave something for the 0.2.0 release :).
>
>
>
>
> On Thu, Oct 22, 2015 at 8:08 AM, Santosh Marella <smare...@maprtech.com>
> wrote:
>
> > I looked at the JIRAs currently marked with fix version "myriad-0.1.0".
> > There were 19 of them. I moved a few out. We are currently at 14.
> >
> > However, IMO the show stoppers are really the following:
> >
> > MYRIAD-43 Replace com.ebay namespace with org.apache
> > MYRIAD-44 Prepare for 0.1.0 release
> > MYRIAD-98 Move from 4 spaces to 2 spaces for indentation
> > MYRIAD-114 Automatic dashboard building
> > MYRIAD-145 Document Myriad Release Process
> > MYRIAD-150 Update NOTICE file
> > MYRIAD-159 Change default mesos version to 0.24
> >
> > Unless anyone thinks there are other JIRAs that are show stoppers, I
> think
> > we should stick to the above list
> > and cut a RC as soon as we address the above.
> >
> > **I'm positive the above can be fixed by early next week (10/27) and we
> can
> > have a RC out for voting mid next week (10/28).**
> >
> > If we can't fix the above JIRAs in time or if new ones come up as "show
> > stoppers", we will have a revised date.
> > And, of course, more fixes are welcome, as long as they can be merged
> > before 10/27.
> >
> > Just to let everyone know about the Apache release process (@Adam, feel
> > free to chime in):
> >  - Apache requires that a RC be put out for voting on
> dev@myriad.incubator
> > for 72 hrs or until 3 binding +1s and no binding -1s,
> >  - followed by a similar voting round on general@incubator.
> >
> > Now to run the last mile..!
> >
> > Cheers,
> > Santosh
> >
> > On Wed, Oct 21, 2015 at 2:48 PM, Adam Bordelon <a...@mesosphere.io>
> wrote:
> >
> > > Keep in mind that Santosh (as Release Manager) has final authority over
> > > pushing things out of 0.1.0.
> > >
> > > I created a (hopefully public) filter for Unresolved 0.1.0 Myriad
> JIRAs:
> > > https://issues.apache.org/jira/browse/MYRIAD-44?filter=12333786
> > >
> > > Maybe we should create a dashboard too, like
> > >
> >
> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12327111
> > >
> > > On Wed, Oct 21, 2015 at 1:38 PM, Jim Klucar <klu...@gmail.com> wrote:
> > >
> > > > I went through JIRA and assigned a bunch of the tickets to the Myriad
> > > 0.1.0
> > > > release. Some of them are really low hanging fruit, so I think we can
> > > knock
> > > > most of these out. As Adam said in the meeting, we'd rather have too
> > many
> > > > flagged for the release and pair down rather than miss some. To that
> > end,
> > > > please go through the tickets that are still unmarked and set the Fix
> > > > Version/s: to Myriad 0.1.0 if you think they should be fixed or by
> the
> > > > release.
> > > >
> > > > Jim
> > > >
> > >
> >
>
>
>
>

Re: JIRA work for 0.1.0

[Darin Johnson]

I think this sounds good about right.  A few Jim marked were new features,
gotta leave something for the 0.2.0 release :).


On Thu, Oct 22, 2015 at 8:08 AM, Santosh Marella 
wrote:

> I looked at the JIRAs currently marked with fix version "myriad-0.1.0".
> There were 19 of them. I moved a few out. We are currently at 14.
>
> However, IMO the show stoppers are really the following:
>
> MYRIAD-43 Replace com.ebay namespace with org.apache
> MYRIAD-44 Prepare for 0.1.0 release
> MYRIAD-98 Move from 4 spaces to 2 spaces for indentation
> MYRIAD-114 Automatic dashboard building
> MYRIAD-145 Document Myriad Release Process
> MYRIAD-150 Update NOTICE file
> MYRIAD-159 Change default mesos version to 0.24
>
> Unless anyone thinks there are other JIRAs that are show stoppers, I think
> we should stick to the above list
> and cut a RC as soon as we address the above.
>
> **I'm positive the above can be fixed by early next week (10/27) and we can
> have a RC out for voting mid next week (10/28).**
>
> If we can't fix the above JIRAs in time or if new ones come up as "show
> stoppers", we will have a revised date.
> And, of course, more fixes are welcome, as long as they can be merged
> before 10/27.
>
> Just to let everyone know about the Apache release process (@Adam, feel
> free to chime in):
>  - Apache requires that a RC be put out for voting on dev@myriad.incubator
> for 72 hrs or until 3 binding +1s and no binding -1s,
>  - followed by a similar voting round on general@incubator.
>
> Now to run the last mile..!
>
> Cheers,
> Santosh
>
> On Wed, Oct 21, 2015 at 2:48 PM, Adam Bordelon  wrote:
>
> > Keep in mind that Santosh (as Release Manager) has final authority over
> > pushing things out of 0.1.0.
> >
> > I created a (hopefully public) filter for Unresolved 0.1.0 Myriad JIRAs:
> > https://issues.apache.org/jira/browse/MYRIAD-44?filter=12333786
> >
> > Maybe we should create a dashboard too, like
> >
> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12327111
> >
> > On Wed, Oct 21, 2015 at 1:38 PM, Jim Klucar  wrote:
> >
> > > I went through JIRA and assigned a bunch of the tickets to the Myriad
> > 0.1.0
> > > release. Some of them are really low hanging fruit, so I think we can
> > knock
> > > most of these out. As Adam said in the meeting, we'd rather have too
> many
> > > flagged for the release and pair down rather than miss some. To that
> end,
> > > please go through the tickets that are still unmarked and set the Fix
> > > Version/s: to Myriad 0.1.0 if you think they should be fixed or by the
> > > release.
> > >
> > > Jim
> > >
> >
>

Re: Yarn-Site

[Darin Johnson]

sudo rm $YARN_HOME/etc/hadoop/yarn-site.xml <- This is slightly off, I'll
go correct.

On Mon, Oct 5, 2015 at 4:38 PM, Darin Johnson <dbjohnson1...@gmail.com>
wrote:

> John, I'm running off: https://github.com/apache/incubator-myriad, it
> seems to run OK there's a couple of NPE issues though they're rare events
> (I had to invent a way to make one occur), I've got a PR for one if you
> want to test/review it for me :).
>
> I'll try to update the wiki with instructions on running the resource
> manager via marathon.
>
> Darin
>
>
> On Mon, Oct 5, 2015 at 4:27 PM, John Omernik <j...@omernik.com> wrote:
>
>> I see. That makes sense.  Thanks for the tip.
>>
>> Is it safe to pull down a recent version at this point? Are we using the
>> official "master" or phase1?  (the lazy man in me is asking for a link to
>> the current repo so I don't have to read back over emails to see where I
>> should go :)
>>
>>
>>
>> On Mon, Oct 5, 2015 at 3:25 PM, Darin Johnson <dbjohnson1...@gmail.com>
>> wrote:
>>
>> > Hey John,
>> >
>> > Are you trying to run the resource manager from the tar ball via
>> marathon?
>> > It's doable, my suggested approach would be to use a json like this:
>> >
>> > {
>> >   "id": "resource-manager",
>> >   "uris": ["hdfs://namenode:port/dist/hadoop-2.7.0.tgz",
>> >  "hdfs://namenode:port/dist/conf/hadoop/yarn-site.xml",
>> >  "hdfs://namenode:port/dist/conf/hadoop/hdfs-site.xml",
>> >  "hdfs:///dist/conf/hadoop/core-site.xml",
>> >  "hdfs://namenode:port/dist/conf/hadoop/mapred-site.xml"],
>> >   "cmd": "cp *.xml hadoop-2.7.0/etc/hadoop && cd hadoop-2.7.0 &&
>> bin/yarn
>> > resourcemanager",
>> >   "mem": 16,
>> >   "cpu": 1
>> >   "instances" : 1,
>> >   "user": "yarn"
>> > }
>> >
>> > Basically it keeps you from redoing the tar ball every time you edit a
>> > config, instead you just upload the new yarn-site.xml.  The Node Manager
>> > gets it's config from the Resource Manager (I'm assuming this is all for
>> > remote distribution, otherwise creating the tar ball is optional).
>> >
>> > Darin
>> >
>> > On Mon, Oct 5, 2015 at 2:36 PM, John Omernik <j...@omernik.com> wrote:
>> >
>> > > Hey all, I've been waiting until the chaos of the code move has died
>> > down.
>> > > I am looking to get this working on my MapR cluster now, and would
>> like
>> > > some clarification on instructions here:
>> > >
>> > >
>> > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/MYRIAD/Installing+for+Administrators
>> > >
>> > > Basically, in the instructions below, it has the "remove the
>> > > yarn-site.xml.  Yet to run the resource manager with myriad, you need
>> the
>> > > yarn-site to be packaged with things (unless I am reading that
>> > incorrectly)
>> > > Is the only option right now to created a tarball for nodemanagers,
>> and
>> > > have this be different from the tarball for the resource manager?
>> > >
>> > > Step 5: Create the Tarball
>> > >
>> > > The tarball has all of the files needed for the Node Managers and
>> > Resource
>> > > Managers. The following shows how to create the tarball and place it
>> in
>> > > HDFS:
>> > > cd ~
>> > > sudo cp -rp $YARN_HOME .
>> > > sudo rm $YARN_HOME/etc/hadoop/yarn-site.xml
>> > > sudo tar -zcpf ~/hadoop-2.7.1.tar.gz hadoop-2.7.1
>> > > hadoop fs -put ~/hadoop-2.7.1.tar.gz /dist
>> > >
>> >
>>
>
>

Re: Myriad Open Source Doc Review

[Darin Johnson]

Ruth,

Looked briefly at this there are significant changes that need to be made
to this document since Swapnil's last PR.  I'll see what I can do about
updating these, I'm rebuilding my dev environment at the moment so this is
a convenient time.  I am traveling early in the week though so it maybe
Wednesday before you see changes.

Darin

On Wed, Sep 16, 2015 at 4:35 PM, Ruth Harris  wrote:

> Hi Team,
>
> Could you all take some to review the documentation on the Apache Wiki?
> https://cwiki.apache.org/confluence/display/MYRIAD/Myriad+Home
>
> *Darin*, Could you take a good look at the Installing for Administrators
> section?
>
> https://cwiki.apache.org/confluence/display/MYRIAD/Installing+for+Administrators
> I'm not sure if there was a resolution from your discussion with John that
> impacts the documentation.
>
> If you could respond with which section(s) that you plan on reviewing, then
> I can make sure that everything gets covered.
>
> Thanks, Ruth
>
> --
> Ruth Harris
> Sr. Technical Writer, MapR
>

Re: Question about the Wiki Instructions on yarn-site.xml

[Darin Johnson]

Hey John I'm going to try to recreate issue using vanilla hadoop later
today.  Any other settings I should know about?
Darin
On Sep 9, 2015 9:42 AM, "John Omernik"  wrote:

> This was another "slipped in" question in my other thread, I am breaking
> out for specific instructions.  Basically, I was struggling with with some
> things in the wiki on this page:
>
> https://cwiki.apache.org/confluence/display/MYRIAD/Installing+for+Administrators
>
> In step 5:
> Step 5: Configure YARN to use Myriad
>
> Modify the */opt/hadoop-2.7.0/etc/hadoop/yarn-site.xml* file as instructed
> in Sample: myriad-config-default.yml
> <
> https://cwiki.apache.org/confluence/display/MYRIAD/Sample%3A+myriad-config-default.yml
> >
> .
>
>
> Issue 1: It should link to the yarn-site.xml page, not hte
> myriad-config.default.yml page
>
> Issue 2:
> It has us put that information in the yarn-site.xml This makes sense.  The
> resource manager needs to be aware of the myriad stuff.
>
> Then I go to create a tarball, (which I SHOULD be able to use for both
> resource manager and nodemanager... right?) However, the instructions state
> to remove the *.xml files.
>
> Step 6: Create the Tarball
>
> The tarball has all of the files needed for the Node Managers and  Resource
> Managers. The following shows how to create the tarball and place it in
> HDFS:
> cd ~
> sudo cp -rp /opt/hadoop-2.7.0 .
> sudo rm hadoop-2.7.0/etc/hadoop/*.xml
> sudo tar -zcpf ~/hadoop-2.7.0.tar.gz hadoop-2.7.0
> hadoop fs -put ~/hadoop-2.7.0.tar.gz /dist
>
>
> What I ended up doing... since I am running the resourcemanager (myriad) in
> marathon, is I created two tarballs. One is my hadoop-2.7.0-RM.tar.gz which
> has the all the xml files still in the tar ball for shipping to marathon.
> Then other is hadoop-2.7.0-NM.tar.gz which per the instructions removes the
> *.xml files from the /etc/hadoop/ directory.
>
>
> I guess... my logic is that myriad creates the conf directory for the
> nodemanagers... but then I thought, and I overthinking something? Am I
> missing something? Could that be factoring into what I am doing here?
>
>
> Obviously my first steps are to add the extra yarn-site.xml entries, but in
> this current setup, they are only going into the resource manager yarn-site
> as the the node-managers don't have a yarn-site in their directories.  Am I
> looking at this correctly?  Perhaps we could rethink the removal process of
> the XML files in the tarball to allow this to work correctly with a single
> tarball?
>
> If I am missing something here, please advise!
>
>
> John
>

Re: Complete Myriad HA implementation

[Darin Johnson]

That's fine.
On Sep 5, 2015 4:54 AM, "Swapnil Daingade" <swapnil.daing...@gmail.com>
wrote:

> Hi Darin,
>
> I was wondering if it would be possible to address them in a follow up PR.
>
> Addressed new review comments since Wednesday and updated PR
> https://github.com/mesos/myriad/pull/123
>
> Regards
> Swapnil
>
>
> On Fri, Sep 4, 2015 at 9:55 AM, Darin Johnson <dbjohnson1...@gmail.com>
> wrote:
>
> > I've got some more comments but can really respond adequately until
> > tomorrow due to travel.  I'll defer if everyone else is OK with the
> merge.
> > Hi All,
> >
> > I have address all the review comments I received since the last PR
> update
> > on Monday.
> > The latest updated PR is here https://github.com/mesos/myriad/pull/123
> >
> > I am wondering, if there are no further review comments by Friday, can we
> > considering merging this pull request ?
> >
> > Regards
> > Swapnil
> >
> >
> > On Mon, Aug 24, 2015 at 7:58 PM, Darin Johnson <dbjohnson1...@gmail.com>
> > wrote:
> >
> > > Swapnil,
> > >
> > > Generally no, they don't which makes it confusing and so I asked.
> > Started
> > > playing with it, going to be travelling tomorrow but will try out soon.
> > >
> > > Darin
> > >
> > >
> > > On Mon, Aug 24, 2015 at 10:50 PM, Swapnil Daingade <
> > > swapnil.daing...@gmail.com> wrote:
> > >
> > > > Hi Darin,
> > > >
> > > > Its a dfs path. I tried it on MapRFS.
> > > > Does hdfs require it to be prefixed by something like hdfs:// ?
> > > > If yes, I'll make the change.
> > > >
> > > > Regards
> > > > Swapnil
> > > >
> > > >
> > > > On Mon, Aug 24, 2015 at 7:28 PM, Darin Johnson <
> > dbjohnson1...@gmail.com>
> > > > wrote:
> > > >
> > > > > Question on the yarn.resourcemanager.fs.state-store.uri that's a
> > local
> > > fs
> > > > > in /var/ if that's we're you're keeping the state how is it
> regained
> > if
> > > > the
> > > > > RM is restarted on a different node in marathon?  Haven't read
> > through
> > > > all
> > > > > the code yet but I'm trying to get oriented, sorry.
> > > > >
> > > > > On Mon, Aug 24, 2015 at 9:35 PM, Swapnil Daingade <
> > > > > swapnil.daing...@gmail.com> wrote:
> > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > Here is a document on how to configure and try out Myriad HA.
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> https://docs.google.com/document/d/1PPqQmiWgCsxrMEq56fNra2Z6JZI8vlF5HK9fHKmA9_Q/edit
> > > > > >
> > > > > > Please let me know your thoughts.
> > > > > > Once I incorporate community feedback and Myriad HA makes it to
> > > phase1,
> > > > > > I'll move it to a more
> > > > > > permanent place like perhaps the wiki.
> > > > > >
> > > > > > Regards
> > > > > > Swapnil
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, Aug 20, 2015 at 11:39 AM, Darin Johnson <
> > > > dbjohnson1...@gmail.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Sweet, look forward to checking it out.
> > > > > > > Hi All,
> > > > > > >
> > > > > > > I have updated my pull request with the complete Myriad HA
> > > > > implementation
> > > > > > > rebased
> > > > > > > on top of the FGS changes here
> > > > > > >
> > > > > > > https://github.com/mesos/myriad/pull/123
> > > > > > >
> > > > > > > I am planning to send out another email with details on how to
> > > > > configure
> > > > > > > it.
> > > > > > >
> > > > > > > Regards
> > > > > > > Swapnil
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Complete Myriad HA implementation

[Darin Johnson]

I've got some more comments but can really respond adequately until
tomorrow due to travel.  I'll defer if everyone else is OK with the merge.
Hi All,

I have address all the review comments I received since the last PR update
on Monday.
The latest updated PR is here https://github.com/mesos/myriad/pull/123

I am wondering, if there are no further review comments by Friday, can we
considering merging this pull request ?

Regards
Swapnil


On Mon, Aug 24, 2015 at 7:58 PM, Darin Johnson <dbjohnson1...@gmail.com>
wrote:

> Swapnil,
>
> Generally no, they don't which makes it confusing and so I asked.  Started
> playing with it, going to be travelling tomorrow but will try out soon.
>
> Darin
>
>
> On Mon, Aug 24, 2015 at 10:50 PM, Swapnil Daingade <
> swapnil.daing...@gmail.com> wrote:
>
> > Hi Darin,
> >
> > Its a dfs path. I tried it on MapRFS.
> > Does hdfs require it to be prefixed by something like hdfs:// ?
> > If yes, I'll make the change.
> >
> > Regards
> > Swapnil
> >
> >
> > On Mon, Aug 24, 2015 at 7:28 PM, Darin Johnson <dbjohnson1...@gmail.com>
> > wrote:
> >
> > > Question on the yarn.resourcemanager.fs.state-store.uri that's a local
> fs
> > > in /var/ if that's we're you're keeping the state how is it regained
if
> > the
> > > RM is restarted on a different node in marathon?  Haven't read through
> > all
> > > the code yet but I'm trying to get oriented, sorry.
> > >
> > > On Mon, Aug 24, 2015 at 9:35 PM, Swapnil Daingade <
> > > swapnil.daing...@gmail.com> wrote:
> > >
> > > > Hi All,
> > > >
> > > > Here is a document on how to configure and try out Myriad HA.
> > > >
> > > >
> > >
> >
>
https://docs.google.com/document/d/1PPqQmiWgCsxrMEq56fNra2Z6JZI8vlF5HK9fHKmA9_Q/edit
> > > >
> > > > Please let me know your thoughts.
> > > > Once I incorporate community feedback and Myriad HA makes it to
> phase1,
> > > > I'll move it to a more
> > > > permanent place like perhaps the wiki.
> > > >
> > > > Regards
> > > > Swapnil
> > > >
> > > >
> > > >
> > > > On Thu, Aug 20, 2015 at 11:39 AM, Darin Johnson <
> > dbjohnson1...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Sweet, look forward to checking it out.
> > > > > Hi All,
> > > > >
> > > > > I have updated my pull request with the complete Myriad HA
> > > implementation
> > > > > rebased
> > > > > on top of the FGS changes here
> > > > >
> > > > > https://github.com/mesos/myriad/pull/123
> > > > >
> > > > > I am planning to send out another email with details on how to
> > > configure
> > > > > it.
> > > > >
> > > > > Regards
> > > > > Swapnil
> > > > >
> > > >
> > >
> >
>

Re: Flex API

[Darin Johnson]

One additional thing I'd like is the ability to flexup a nmnode on all
agents with a certain attribute as opposed to a fixed number.

Also, when I get back from vacation I plan on scoping mesos-1739 (dynamic
attributes) which would allow for tighter integration with the hdfs
framework. An alternative would be to get hostnames from the namenode,
though not as seemless.
(Reviving this thread)

We've discussed several great points in the thread (PUT vs POST, need for
GET, JSON payload vs parameters in URL, declarative interface etc).
Just to get us going, I think we should focus on a couple of things that
will be useful for Myriad users, while leaving them flexible enough to be
evolved in the future.

What I heard from several folks (some of it brought up again at MesosCon)
about the flex up/down APIs is this:
- flexup doesn't support launching NMs on specific set of hosts. This is
especially needed to launch NMs on same set of nodes that have HDFS
DataNode running.
- flexdown lacks an option to shut down NMs with a specific profile. Today,
we bring down ANY arbitrary NM.
- flexdown lacks an option to shutdown NMs running on specific hosts.

I captured my thoughts in a document here:
https://docs.google.com/document/d/1PA_POY_abP6J4youM2Q0VJ48T4OCSe258-OAz_-EO6k/edit#heading=h.1atlx0ag9s8t

@Jim: Happy to collaborate at one single place (Swagger/Google Doc) to
finalize the APIs. Just let me know.

Thanks,
Santosh

On Sun, Jun 14, 2015 at 5:29 PM, Jim Klucar klu...@gmail.com wrote:

 Seems like POST is a winner with people.

 Another thing to consider is how we want the REST interface to be vs what
 we want the UI to do. The UI could support flexup/flexdown like it is
while
 the REST interface is just a declarative state like Adam suggested. The UI
 would just be responsible for translating the request into the new state.

 Tomorrow I'll try to put together another swagger doc with some of the
 suggested options.


 On Sun, Jun 14, 2015 at 6:37 PM, yuliya Feldman
 yufeld...@yahoo.com.invalid
  wrote:

  I think we are at the point to list all the options we want flex API
to
  support.
  1. Do we continue supporting flexup/down or just flex with additional
  preposition like up/down:https://hostname:port/flex/up(down)
  2. I think we should switch to POST and may be maintain PUT for legacy
 (if
  even needed to keep it). We are not DB after all and not storing any
  retrievable info here :)
  3. We need to add status (GET) to see the status - though I think we
have
  one
  4. Define JSON payload to support different casesa. providing
  different profiles together: [{profile:big,
  instances:2},{profile:medium,instances:6}]b. provide what state we
  want Myriad to be in: I want 10 medium instances and then Myriad will
 do
  whatever isnecessary to transition to that state,
 adding/removing/resizing
  NMsc. flex/down particular instance IDsd. flex up/down
preferred
  hosts, delays, others
  5. How all this fits into FineGrain Scaling? With it we would do
 automatic
  flex up/down. And the less knobs admin will have to turn the easier it
is
  for admin and the end users.
 
From: Adam Bordelon a...@mesosphere.io
   To: dev@myriad.incubator.apache.org
   Sent: Sunday, June 14, 2015 2:54 PM
   Subject: Re: Flex API
 
  (In addition,) I'd also like to see a more declarative interface.
Instead
  of add two more instances, the user(s) could just specify the desired
  state of I want 10 medium instances and then Myriad will do whatever
is
  necessary to transition to that state, adding/removing/resizing NMs as
  necessary.
 
 
 
  On Fri, Jun 12, 2015 at 5:23 PM, Will Ochandarena 
  wochandar...@maprtech.com
   wrote:
 
   On Fri, Jun 12, 2015 at 5:11 PM, Jim Klucar klu...@gmail.com wrote:
  
What verb to use when outside of database land can be argued. I
would
   vote
for POST over PUT just because I tend to default to POST. PUT was
 there
when I showed up, so I left it.
  
  
   Last time I agonized about PUT vs POST the most logical distinction I
  found
   was that PUT should be used for idempotent operations, while POST for
   non-idempotent (like we have here with flex-up, since instance-ids are
   generated).
  
   Since the api doesn't wait until the
instances are created to return, we can't really return the instance
  IDs
   we
created.
   
  
   That seems OK to me.
  
  
The GET would just return some status?
   
  
   Yeah, I was thinking that this would be needed for a future GUI where
 we
   list all instances with parameters and status for each (profile,
 current
   cpu/ram/disk, node, uptime).  I'm picturing checkboxes next  to each
so
   users can multi-select and hit 'delete' to wipe them away (like
 flex-down
   does now).
  
   The PATCH is interesting
   
  
   Yeah, I started to write PUT but to REST geeks PUT implies you always
  have
   to rewrite the complete object when making changes.  PATCH allows more
   flexible modifications.
  
   The 

Re: Google Hangout Link

[Darin Johnson]

Ken,

Was going through the dcos guide, are there any service discovery options
in dcos for finding webservice ports haproxy, nginx?

Thanks,
Darin
PS sorry bought missing the sync I'm on vacation in the Midwest.
On Aug 26, 2015 12:05 PM, Ken Sipe k...@mesosphere.io wrote:

 I sent an invite… and BTW… the sync notes have the hangout link at the
 top:
 https://docs.google.com/document/d/1JGmJrgeg98bHw_0_sSRmyX6WiAe13OdErcFlaz6Aa04/edit#heading=h.rnolkdpzfc8u
 
 https://docs.google.com/document/d/1JGmJrgeg98bHw_0_sSRmyX6WiAe13OdErcFlaz6Aa04/edit#heading=h.rnolkdpzfc8u
 

 ken
  On Aug 26, 2015, at 11:04 AM, Brandon Gulla gulla.bran...@gmail.com
 wrote:
 
  Can someone send out the active google hangout link please?
 
  thanks
 
  --
  Brandon

Re: Complete Myriad HA implementation

[Darin Johnson]

Question on the yarn.resourcemanager.fs.state-store.uri that's a local fs
in /var/ if that's we're you're keeping the state how is it regained if the
RM is restarted on a different node in marathon?  Haven't read through all
the code yet but I'm trying to get oriented, sorry.

On Mon, Aug 24, 2015 at 9:35 PM, Swapnil Daingade 
swapnil.daing...@gmail.com wrote:

 Hi All,

 Here is a document on how to configure and try out Myriad HA.

 https://docs.google.com/document/d/1PPqQmiWgCsxrMEq56fNra2Z6JZI8vlF5HK9fHKmA9_Q/edit

 Please let me know your thoughts.
 Once I incorporate community feedback and Myriad HA makes it to phase1,
 I'll move it to a more
 permanent place like perhaps the wiki.

 Regards
 Swapnil



 On Thu, Aug 20, 2015 at 11:39 AM, Darin Johnson dbjohnson1...@gmail.com
 wrote:

  Sweet, look forward to checking it out.
  Hi All,
 
  I have updated my pull request with the complete Myriad HA implementation
  rebased
  on top of the FGS changes here
 
  https://github.com/mesos/myriad/pull/123
 
  I am planning to send out another email with details on how to configure
  it.
 
  Regards
  Swapnil
 

Re: Complete Myriad HA implementation

[Darin Johnson]

Sweet, look forward to checking it out.
Hi All,

I have updated my pull request with the complete Myriad HA implementation
rebased
on top of the FGS changes here

https://github.com/mesos/myriad/pull/123

I am planning to send out another email with details on how to configure it.

Regards
Swapnil

Re: Odd Errors

[Darin Johnson]

Could you paste the stderr/stdout of the executor as well?

I think this may be a bug so you won't get booted:).

Darin
On Aug 18, 2015 3:07 PM, John Omernik j...@omernik.com wrote:

 I am working to stumble through this as Santosh helped me get a pre
 incubator version of Myriad running, and now I upgraded a bunch of stuff
 and wanted to try some of the more recent features. I setup the remote
 distribution, created what I think would be a good a json for marathon and
 then I am getting the dreaded Null Pointer Exception without much help...

 Based on the logs, it appears to be pulling my URI down with the proper
 pathing and trying to execute the resource manager from the tar ball,
 perhaps this will get me kicked off the dev list but my dev foo is weak,
 thus I am not sure how to troubleshoot this. :) Any help would be
 appreciated.


 15/08/18 17:00:28 INFO mortbay.log: Started
 SelectChannelConnector@0.0.0.0:8192
 15/08/18 17:00:28 INFO myriad.Main: Initializing HealthChecks
 15/08/18 17:00:28 INFO myriad.Main: Initializing Profiles
 15/08/18 17:00:28 INFO scheduler.NMProfileManager: Adding profile tiny
 with CPU: 1 and Memory: 4096
 15/08/18 17:00:28 INFO scheduler.NMProfileManager: Adding profile
 small with CPU: 2 and Memory: 8192
 15/08/18 17:00:28 INFO scheduler.NMProfileManager: Adding profile
 medium with CPU: 4 and Memory: 16384
 15/08/18 17:00:28 INFO scheduler.NMProfileManager: Adding profile
 large with CPU: 8 and Memory: 32768
 15/08/18 17:00:28 INFO scheduler.NMProfileManager: Adding profile huge
 with CPU: 12 and Memory: 49152
 15/08/18 17:00:28 INFO myriad.Main: Validating nmInstances..
 15/08/18 17:00:28 INFO service.AbstractService: Service
 org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler
 failed in state INITED; cause: java.lang.RuntimeException: Failed to
 initialize myriad
 java.lang.RuntimeException: Failed to initialize myriad
 at
 com.ebay.myriad.scheduler.yarn.interceptor.MyriadInitializationInterceptor.init(MyriadInitializationInterceptor.java:35)
 at
 com.ebay.myriad.scheduler.yarn.interceptor.CompositeInterceptor.init(CompositeInterceptor.java:76)
 at
 com.ebay.myriad.scheduler.yarn.MyriadFairScheduler.serviceInit(MyriadFairScheduler.java:50)
 at
 org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
 at
 org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
 at
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:570)
 at
 org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
 at
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:997)
 at
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:262)
 at
 org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
 at
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1220)
 Caused by: java.lang.NullPointerException
 at com.ebay.myriad.Main.validateNMInstances(Main.java:166)
 at com.ebay.myriad.Main.run(Main.java:98)
 at com.ebay.myriad.Main.initialize(Main.java:80)
 at
 com.ebay.myriad.scheduler.yarn.interceptor.MyriadInitializationInterceptor.init(MyriadInitializationInterceptor.java:32)
 ... 10 more
 15/08/18 17:00:28 INFO service.AbstractService: Service
 RMActiveServices failed in state INITED; cause:
 java.lang.RuntimeException: Failed to initialize myriad
 java.lang.RuntimeException: Failed to initialize myriad
 at
 com.ebay.myriad.scheduler.yarn.interceptor.MyriadInitializationInterceptor.init(MyriadInitializationInterceptor.java:35)
 at
 com.ebay.myriad.scheduler.yarn.interceptor.CompositeInterceptor.init(CompositeInterceptor.java:76)
 at
 com.ebay.myriad.scheduler.yarn.MyriadFairScheduler.serviceInit(MyriadFairScheduler.java:50)
 at
 org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
 at
 org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
 at
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:570)
 at
 org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
 at
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:997)
 at
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:262)
 at
 org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
 at
 org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1220)
 Caused by: java.lang.NullPointerException
 at com.ebay.myriad.Main.validateNMInstances(Main.java:166)
 at 

Re: Myriad 0.1 release scope

[Darin Johnson]

John,
The remote distribution doesn't require the nm to be run from marathon
though it's possible.  Essentially, it's the same configuration for the rm
you'd do for the non remote version + adding a uri for the tarball.
I've got jsons for running the rm in marathon, I'll try to get them and
some documentation up soon.  Currently at a conference though which means
probably next week.

Darin
Darin
On Aug 18, 2015 2:49 PM, John Omernik j...@omernik.com wrote:

 Ok, so I tried the remote distribution of the Myriad per the docs, I
 guess,it could probably use some information related to how to run
 resource manager if it's in the tar.gz.  Perhaps an example marathon json.
 I am playing with it now to figure it out.

 On Tue, Aug 18, 2015 at 3:48 PM, yuliya Feldman
 yufeld...@yahoo.com.invalid
  wrote:

  mesos/myriad is the right one so far
From: John Omernik j...@omernik.com
   To: dev@myriad.incubator.apache.org; yuliya Feldman 
 yufeld...@yahoo.com
   Sent: Tuesday, August 18, 2015 1:44 PM
   Subject: Re: Myriad 0.1 release scope
 
  (So if I clone that repo, am I cloning the right one?)
 
 
 
  On Tue, Aug 18, 2015 at 3:43 PM, John Omernik j...@omernik.com wrote:
 
   Ok, I was going off
  
 https://github.com/mesos/myriad/blob/phase1/docs/myriad-configuration.md
  
   I will try it.
  
   John
  
   On Tue, Aug 18, 2015 at 3:40 PM, yuliya Feldman 
   yufeld...@yahoo.com.invalid wrote:
  
   You actually do not need to rebuild even today - just keep this file
 in
   hadoop config directory that is on the classpath: like .../etc/hadoop
From: John Omernik j...@omernik.com
To: dev@myriad.incubator.apache.org
Sent: Tuesday, August 18, 2015 1:35 PM
Subject: Re: Myriad 0.1 release scope
  
   On the release scope, will having the myriad configuration file exist
   outside the jar (i.e. you can change configuration without rebuilding)
  be
   part of the .1 release scope?
  
  
  
   On Mon, Aug 10, 2015 at 10:01 PM, Santosh Marella 
  smare...@maprtech.com
   wrote:
  
Hello All,
   
 I've merged the FGS changes into phase1. Built and tested both
 coarse
grained scaling and fine grained scaling, UI on a 4 node cluster.
   
 If anyone finds things are not working as expected, please let me
  know.
   
Thanks,
Santosh
   
On Fri, Aug 7, 2015 at 10:46 AM, Santosh Marella 
  smare...@maprtech.com
   
wrote:
   
 Hello guys,

 I propose merging FGS into phase1. As I said before, I think it's
  at a
 point where the functionality works reasonably well.
 Any future improvements/fixes/UI changes can be done via separate
   JIRAs.

 Unless there are any major concerns, I'd like to merge FGS into
  phase1
 *EOD Monday* (PDT).

 Thanks,
 Santosh

 On Wed, Aug 5, 2015 at 8:16 PM, Santosh Marella 
   smare...@maprtech.com
 wrote:

 I feel FGS is very close to making it into 0.1. PR 116 addresses
   moving
 to hadoop 2.7 and making FGS and CGS coexist. This PR was
 recently
reviewed
 by Yulia and Darin. Darin had also tried out FGS on hadoop 2.6.x
  and
2.7.x
 clusters and it seemed to have worked as expected. Unless there
 are
   more
 reviews/feedback, it can be merged into issue_14. Once PR 116 is
   merged
 into issue_14, issue_14 can be merged into phase1.

 Thanks,
 Santosh

 On Tue, Aug 4, 2015 at 4:54 PM, Adam Bordelon 
 a...@mesosphere.io
wrote:

 We do have a JIRA 0.1.0 fix version field, but none of our
  issues
   use
 it
 yet.
 I think the goal was just to take what we have and make it work
   under
 Apache infrastructure, then vote on that for 0.1.0.
 Although other features like HA or FGS would be great, let's try
  to
   get
 our
 first Apache release out ASAP.
 We can create 0.1.1 or 0.2.0 fix versions for subsequent
 releases
   with
 other issues/features. Roadmap would be great.
 (I'm just summarizing what we discussed a month or two ago. Feel
   free
to
 correct me or disagree with this approach.)

 On Tue, Aug 4, 2015 at 4:44 PM, Swapnil Daingade 
 swapnil.daing...@gmail.com
  wrote:

  Hi all,
 
  Was wondering what would be the scope for the Myriad 0.1
  release.
  It would be nice to have a roadmap page somewhere and target
  features to releases (JIRA 'fix version' field perhaps)
 
  Regards
  Swapnil
 




   
  
  
  
  
  
  
 
 
 
 

Re: Myriad 0.1 release scope

[Darin Johnson]

So I compiled the 2.5 fgs against 2.6 when I was testing.  If we abstract
this right it may just be an if statement or two.
On Aug 7, 2015 6:47 PM, Santosh Marella smare...@maprtech.com wrote:

  Myriad code base compiled against hadoop 2.7 should work on hadoop 2.5
  cluster as long as FGS (i.e. zero profile NM) is not used.

 Verified the above. As long as FGS (zero profile NM) is not used,
 Myriad compiled against hadoop 2.7 will work on hadoop 2.5.

 Thanks,
 Santosh

 On Fri, Aug 7, 2015 at 2:20 PM, Santosh Marella smare...@maprtech.com
 wrote:

   It will make working on HA easier
  Oh Yes!
 
   how do we facilitate that? Profiles?
  Profiles might be one way. Currently, FGS is supported for zero profile
  only.
  And we have seen there was an API incompatibility from 2.5 to 2.6+ in FGS
  code.
  So, ideally (since I haven't tried it myself), when FGS is merged into
  phase1,
  the Myriad code base compiled against hadoop 2.7 should work on hadoop
 2.5
  cluster as long as FGS (i.e. zero profile NM) is not used. (I'll try this
  out and
  post back what I find)
 
  However, in the long term we need a mechanism to abstract out the APIs
  that are incompatible across versions.
 
  Thanks,
  Santosh
 
  On Fri, Aug 7, 2015 at 12:12 PM, Darin Johnson dbjohnson1...@gmail.com
  wrote:
 
  It will make working on HA easier.  However, one concern that's been
  addressed previously is that FGS works for Hadoop 2.6.0+. Do we plan to
  support 2.5.X (anything lower?) also as Santosh has a way to do that, if
  so
  how do we facilitate that? Profiles?
 
  Darin
 
  On Fri, Aug 7, 2015 at 1:46 PM, Santosh Marella smare...@maprtech.com
  wrote:
 
   Hello guys,
  
   I propose merging FGS into phase1. As I said before, I think it's at a
   point where the functionality works reasonably well.
   Any future improvements/fixes/UI changes can be done via separate
 JIRAs.
  
   Unless there are any major concerns, I'd like to merge FGS into phase1
  *EOD
   Monday* (PDT).
  
   Thanks,
   Santosh
  
   On Wed, Aug 5, 2015 at 8:16 PM, Santosh Marella 
 smare...@maprtech.com
   wrote:
  
I feel FGS is very close to making it into 0.1. PR 116 addresses
  moving
   to
hadoop 2.7 and making FGS and CGS coexist. This PR was recently
  reviewed
   by
Yulia and Darin. Darin had also tried out FGS on hadoop 2.6.x and
  2.7.x
clusters and it seemed to have worked as expected. Unless there are
  more
reviews/feedback, it can be merged into issue_14. Once PR 116 is
  merged
into issue_14, issue_14 can be merged into phase1.
   
Thanks,
Santosh
   
On Tue, Aug 4, 2015 at 4:54 PM, Adam Bordelon a...@mesosphere.io
   wrote:
   
We do have a JIRA 0.1.0 fix version field, but none of our issues
  use
   it
yet.
I think the goal was just to take what we have and make it work
 under
Apache infrastructure, then vote on that for 0.1.0.
Although other features like HA or FGS would be great, let's try to
  get
our
first Apache release out ASAP.
We can create 0.1.1 or 0.2.0 fix versions for subsequent releases
  with
other issues/features. Roadmap would be great.
(I'm just summarizing what we discussed a month or two ago. Feel
  free to
correct me or disagree with this approach.)
   
On Tue, Aug 4, 2015 at 4:44 PM, Swapnil Daingade 
swapnil.daing...@gmail.com
 wrote:
   
 Hi all,

 Was wondering what would be the scope for the Myriad 0.1 release.
 It would be nice to have a roadmap page somewhere and target
 features to releases (JIRA 'fix version' field perhaps)

 Regards
 Swapnil

   
   
   
  
 
 
 

Re: Myriad 0.1 release scope

[Darin Johnson]

It will make working on HA easier.  However, one concern that's been
addressed previously is that FGS works for Hadoop 2.6.0+. Do we plan to
support 2.5.X (anything lower?) also as Santosh has a way to do that, if so
how do we facilitate that? Profiles?

Darin

On Fri, Aug 7, 2015 at 1:46 PM, Santosh Marella smare...@maprtech.com
wrote:

 Hello guys,

 I propose merging FGS into phase1. As I said before, I think it's at a
 point where the functionality works reasonably well.
 Any future improvements/fixes/UI changes can be done via separate JIRAs.

 Unless there are any major concerns, I'd like to merge FGS into phase1 *EOD
 Monday* (PDT).

 Thanks,
 Santosh

 On Wed, Aug 5, 2015 at 8:16 PM, Santosh Marella smare...@maprtech.com
 wrote:

  I feel FGS is very close to making it into 0.1. PR 116 addresses moving
 to
  hadoop 2.7 and making FGS and CGS coexist. This PR was recently reviewed
 by
  Yulia and Darin. Darin had also tried out FGS on hadoop 2.6.x and 2.7.x
  clusters and it seemed to have worked as expected. Unless there are more
  reviews/feedback, it can be merged into issue_14. Once PR 116 is merged
  into issue_14, issue_14 can be merged into phase1.
 
  Thanks,
  Santosh
 
  On Tue, Aug 4, 2015 at 4:54 PM, Adam Bordelon a...@mesosphere.io
 wrote:
 
  We do have a JIRA 0.1.0 fix version field, but none of our issues use
 it
  yet.
  I think the goal was just to take what we have and make it work under
  Apache infrastructure, then vote on that for 0.1.0.
  Although other features like HA or FGS would be great, let's try to get
  our
  first Apache release out ASAP.
  We can create 0.1.1 or 0.2.0 fix versions for subsequent releases with
  other issues/features. Roadmap would be great.
  (I'm just summarizing what we discussed a month or two ago. Feel free to
  correct me or disagree with this approach.)
 
  On Tue, Aug 4, 2015 at 4:44 PM, Swapnil Daingade 
  swapnil.daing...@gmail.com
   wrote:
 
   Hi all,
  
   Was wondering what would be the scope for the Myriad 0.1 release.
   It would be nice to have a roadmap page somewhere and target
   features to releases (JIRA 'fix version' field perhaps)
  
   Regards
   Swapnil
  
 
 
 

Re: Issues 16 and Issue 12

[Darin Johnson]

Swapnil,

Looked over both Docs, HA and NM restart.  It's pretty high level so I'll
look forward to the details.  Initial thoughts:

1. Getting framework reconciliation going would likely eliminate certain
issues, such as sendFrameworkMessage being unreliable.  So should be
implemented sooner than later.

2. How stable is the RMStateStore API? If there's changes between versions
of Hadoop, might be best to use Mesos's State API.

3. There was no mention of running two RM's in traditional Hadoop RM HA
(maybe in marathon even), but this should be considered a possibility. That
may have been implicit.

Saw the PR will look at it.

Darin
Hi Darin,

The Myriad HA work will involve work related to issue 16.
I already have the Myriad HA design doc for review.
Your feedback on it would be really helpful.
I also plan to send out for review parts of the Myriad HA implementation
(although it does not address task reconciliation yet). I was planning to
work on it next.

Regards
Swapnil


On Mon, Aug 3, 2015 at 12:08 PM, Darin Johnson dbjohnson1...@gmail.com
wrote:

 Is anyone actively working these?  I'm interested in both of these and
 should have some cycles to work on them.

 One question I have on issue 12 is how the generalize Scheduling Policies
 if we have autoscaling, fine grain scheduling, and fixed resources (with a
 flexup/flexdown option).  Currently it seems as though FGS is embedded
 pretty deeply.  Ideally though we could Have a SchedulerPolicy interface,
 and users could specify the SchedulerPolicy via the Myriad config.

 If I don't get a response, I'll probably start issue 16 as it's straight
 forward and write something up on 12.

 Darin

Re: Some thoughts after a crash course installation

[Darin Johnson]

Take a look at https://github.com/mesos/myriad/pull/83.  I think this
addresses your question, if so its being worked on.
On Apr 21, 2015 7:35 AM, Zhongyue Luo zhongyue@gmail.com wrote:

 Hi team,

 I finally got Myriad installed in our small Mesos cluster but I have some
 questions regarding Myriads implementation.

 I see that each slave needs a JRE and Hadoop binaries extracted in
 HADOOP_HOME.

 I understand why JRE is required on each slave but why the Hadoop binary
 files?

 Shouldn't the files needed for executing a Node manager be placed in the
 Myriad executor?

 Or is this a feature not yet implemented?

 I haven't yet seen Myriad's source code yet but it would be cool if someone
 in the list could help me out here. Thanks.

 --
 *Intel SSG/STO/BDT*
 880 Zixing Road, Zizhu Science Park, Minhang District, 200241, Shanghai,
 China
 +862161166500

Re: skip chown in mesos patched

[Darin Johnson]

That will eventually simplify the getCommandInfo in TaskFactory.java code I
submitted.  That's essentially what I had to do to keep the permissions
correct, definitely belongs in mesos proper.

On Wed, Apr 8, 2015 at 5:32 PM, Adam Bordelon a...@mesosphere.io wrote:

 Saw that. Thanks!
 Added the RB link to the JIRA and asked Vinod if he would Shepherd the
 patch.
 In case others want to review: https://reviews.apache.org/r/32975/

 On Wed, Apr 8, 2015 at 2:19 PM, Jim Klucar klu...@gmail.com wrote:

  In case anyone hasn't seen it, I supplied a patch to Mesos that allows us
  to skip the chown step when distributing myriad. Its in the Mesos review
  system now, and I've been in touch with Vinod.
 
  https://issues.apache.org/jira/browse/MESOS-1790