Re: 0.2.0 release
Swanil, I concur and want to keep both options for Mesos and Docker networking available, and putting the configuration for both in should be a priority. However, one has to be careful with this as the NM's register with the RM via heartbeats with their container port (Not the host port), this isn't an issue if NM and RM are in the same Docker Network, via Weave or Kubernetes but is with simple bridged networking. We also have to be careful as Myriad currently doesn't run HDFS itself so we'd lose data locality. My idea was the start with Host Networking so we could make Myriad easier to deploy but leave room to add additional networking options: basically exposing all the protobuf options for Docker Parameters (used to configure docker networking) and NetworkInfo (used to configure Mesos networking). Darin On Tue, Mar 22, 2016 at 2:48 PM, Swapnil Daingade wrote: > Hi Darin, > > I feel docker networking is something we should spent time to think > through. > A user should be able to use multiple options provided by Mesos, Docker, > 3rd party etc > > It would be great if we can abstract the specific implementation to provide > container ip addresses behind interfaces. User should be able to switch > implementations by making simple changes in configuration files. > > Regards > Swapnil > > > On Tue, Mar 22, 2016 at 8:20 AM, Darin Johnson > wrote: > > > Swapnil, > > > > Any help would be appreciated. I'll try to write up what I'm working on > > tomorrow. But essentially the ideas are: > > 1. Ability to launch the resource manager and node managers in docker > > containers > > 2. Use host networking for now (Ports configured to be pulled from mesos > - > > ability to use ports reserved by role), but leave hooks to easily add IP > > per container. > > 3. Ability to get configuration files for a URI > > 4. Ability to mount local volumes for local directories in the shuffle > > phase etc (though will require more config). > > > > Darin > > >
Re: 0.2.0 release
Hi Darin, I feel docker networking is something we should spent time to think through. A user should be able to use multiple options provided by Mesos, Docker, 3rd party etc It would be great if we can abstract the specific implementation to provide container ip addresses behind interfaces. User should be able to switch implementations by making simple changes in configuration files. Regards Swapnil On Tue, Mar 22, 2016 at 8:20 AM, Darin Johnson wrote: > Swapnil, > > Any help would be appreciated. I'll try to write up what I'm working on > tomorrow. But essentially the ideas are: > 1. Ability to launch the resource manager and node managers in docker > containers > 2. Use host networking for now (Ports configured to be pulled from mesos - > ability to use ports reserved by role), but leave hooks to easily add IP > per container. > 3. Ability to get configuration files for a URI > 4. Ability to mount local volumes for local directories in the shuffle > phase etc (though will require more config). > > Darin >
Re: 0.2.0 release
Swapnil, Any help would be appreciated. I'll try to write up what I'm working on tomorrow. But essentially the ideas are: 1. Ability to launch the resource manager and node managers in docker containers 2. Use host networking for now (Ports configured to be pulled from mesos - ability to use ports reserved by role), but leave hooks to easily add IP per container. 3. Ability to get configuration files for a URI 4. Ability to mount local volumes for local directories in the shuffle phase etc (though will require more config). Darin
Re: 0.2.0 release
+1 on Darin as release manager I'd like to see 0.2 have: - Usable FGS - Dockerized NM (for multitenancy) On Tue, Mar 15, 2016 at 9:46 AM, Darin Johnson wrote: > We've talked about a 0.2.0 release slated for mid April at the dev sync. > I'd like to nail down any features people would like and have time to work > on. > > I've been spend some time fixing major bugs to the FGS feature and plan to > address MYRIAD-136 and MYRIAD-189. > > I'd also be willing to be the release manager on this release if necessary. > > Darin >
Re: 0.2.0 release
With a Docker container, I think it still follows under the same cgroup hierarchy. Dockerized NM is not a _requirement_ for multitenancy, but can help because it forces us to support dynamic ports/paths/etc so that multiple NMs don't share any resources/binaries and can run on the same node. On Thu, Mar 17, 2016 at 1:31 PM, Swapnil Daingade wrote: > +1 Darin for RM. > > Would like be involved in the multi-tenency with Dockerized NM work. > Perhaps we can start filing JIRA's targeted for 0.2 and start adding design > proposals > for people to review and comment. > > Regards > Swapnil > > > On Thu, Mar 17, 2016 at 8:13 AM, Darin Johnson > wrote: > > > Happy to report as of the last two PRS, FGS is usable no memory leaks or > > crashes, could likely be improved with fancier schedulers but that's for > > the future. I'm currently looking at running some terasort benchmarks > with > > FGS and a reserved resources vs statically sized NMs to figure out the > > performance hit. Might be worth a blog post in the near future. > > > > Adam, I've been looking through the cgroups code for myriad recently, > > apparently we need the mod the path YARN uses for it's Hierarchy. Does > > that change at all within a Docker container or is it the same? > > > > Darin > > > > On Mar 16, 2016 8:48 PM, "Adam Bordelon" wrote: > > > > > +1 on Darin as release manager > > > > > > I'd like to see 0.2 have: > > > - Usable FGS > > > - Dockerized NM (for multitenancy) > > > > > > On Tue, Mar 15, 2016 at 9:46 AM, Darin Johnson < > dbjohnson1...@gmail.com> > > > wrote: > > > > > > > We've talked about a 0.2.0 release slated for mid April at the dev > > sync. > > > > I'd like to nail down any features people would like and have time to > > > work > > > > on. > > > > > > > > I've been spend some time fixing major bugs to the FGS feature and > plan > > > to > > > > address MYRIAD-136 and MYRIAD-189. > > > > > > > > I'd also be willing to be the release manager on this release if > > > necessary. > > > > > > > > Darin > > > > > > > > > >
Re: 0.2.0 release
+1 Darin for RM. Would like be involved in the multi-tenency with Dockerized NM work. Perhaps we can start filing JIRA's targeted for 0.2 and start adding design proposals for people to review and comment. Regards Swapnil On Thu, Mar 17, 2016 at 8:13 AM, Darin Johnson wrote: > Happy to report as of the last two PRS, FGS is usable no memory leaks or > crashes, could likely be improved with fancier schedulers but that's for > the future. I'm currently looking at running some terasort benchmarks with > FGS and a reserved resources vs statically sized NMs to figure out the > performance hit. Might be worth a blog post in the near future. > > Adam, I've been looking through the cgroups code for myriad recently, > apparently we need the mod the path YARN uses for it's Hierarchy. Does > that change at all within a Docker container or is it the same? > > Darin > > On Mar 16, 2016 8:48 PM, "Adam Bordelon" wrote: > > > +1 on Darin as release manager > > > > I'd like to see 0.2 have: > > - Usable FGS > > - Dockerized NM (for multitenancy) > > > > On Tue, Mar 15, 2016 at 9:46 AM, Darin Johnson > > wrote: > > > > > We've talked about a 0.2.0 release slated for mid April at the dev > sync. > > > I'd like to nail down any features people would like and have time to > > work > > > on. > > > > > > I've been spend some time fixing major bugs to the FGS feature and plan > > to > > > address MYRIAD-136 and MYRIAD-189. > > > > > > I'd also be willing to be the release manager on this release if > > necessary. > > > > > > Darin > > > > > >
Re: 0.2.0 release
Thanks Darin for willing to be the release manager for 0.2. Happy to help with anything related to release management (and beyond, if possible). Santosh On Tue, Mar 15, 2016 at 9:46 AM, Darin Johnson wrote: > We've talked about a 0.2.0 release slated for mid April at the dev sync. > I'd like to nail down any features people would like and have time to work > on. > > I've been spend some time fixing major bugs to the FGS feature and plan to > address MYRIAD-136 and MYRIAD-189. > > I'd also be willing to be the release manager on this release if necessary. > > Darin >
Re: 0.2.0 release
Adam: I agree with your suggestion for 0.2. I am, however, curious to understand why a dockerized NM would be a requirement for multi-tenancy. I don't disagree, I'm just trying to understand. paul On Wed, Mar 16, 2016 at 8:48 PM, Adam Bordelon wrote: > +1 on Darin as release manager > > I'd like to see 0.2 have: > - Usable FGS > - Dockerized NM (for multitenancy) > > On Tue, Mar 15, 2016 at 9:46 AM, Darin Johnson > wrote: > > > We've talked about a 0.2.0 release slated for mid April at the dev sync. > > I'd like to nail down any features people would like and have time to > work > > on. > > > > I've been spend some time fixing major bugs to the FGS feature and plan > to > > address MYRIAD-136 and MYRIAD-189. > > > > I'd also be willing to be the release manager on this release if > necessary. > > > > Darin > > > -- *Paul Curtis *- Senior Product Technologist *O: *+1 203-660-0015 - *M:* +1 203-539-9705 Now Available - Free Hadoop On-Demand Training <http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available>
Re: 0.2.0 release
Happy to report as of the last two PRS, FGS is usable no memory leaks or crashes, could likely be improved with fancier schedulers but that's for the future. I'm currently looking at running some terasort benchmarks with FGS and a reserved resources vs statically sized NMs to figure out the performance hit. Might be worth a blog post in the near future. Adam, I've been looking through the cgroups code for myriad recently, apparently we need the mod the path YARN uses for it's Hierarchy. Does that change at all within a Docker container or is it the same? Darin On Mar 16, 2016 8:48 PM, "Adam Bordelon" wrote: > +1 on Darin as release manager > > I'd like to see 0.2 have: > - Usable FGS > - Dockerized NM (for multitenancy) > > On Tue, Mar 15, 2016 at 9:46 AM, Darin Johnson > wrote: > > > We've talked about a 0.2.0 release slated for mid April at the dev sync. > > I'd like to nail down any features people would like and have time to > work > > on. > > > > I've been spend some time fixing major bugs to the FGS feature and plan > to > > address MYRIAD-136 and MYRIAD-189. > > > > I'd also be willing to be the release manager on this release if > necessary. > > > > Darin > > >
0.2.0 release
We've talked about a 0.2.0 release slated for mid April at the dev sync. I'd like to nail down any features people would like and have time to work on. I've been spend some time fixing major bugs to the FGS feature and plan to address MYRIAD-136 and MYRIAD-189. I'd also be willing to be the release manager on this release if necessary. Darin
