Updated Invitation: Myriad Dev Sync @ Every 2 weeks from 9am to 9:50am on Wednesday (a...@mesosphere.io)
BEGIN:VCALENDAR PRODID:-//Google Inc//Google Calendar 70.9054//EN VERSION:2.0 CALSCALE:GREGORIAN METHOD:REQUEST BEGIN:VTIMEZONE TZID:America/Los_Angeles X-LIC-LOCATION:America/Los_Angeles BEGIN:DAYLIGHT TZOFFSETFROM:-0800 TZOFFSETTO:-0700 TZNAME:PDT DTSTART:19700308T02 RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=2SU END:DAYLIGHT BEGIN:STANDARD TZOFFSETFROM:-0700 TZOFFSETTO:-0800 TZNAME:PST DTSTART:19701101T02 RRULE:FREQ=YEARLY;BYMONTH=11;BYDAY=1SU END:STANDARD END:VTIMEZONE BEGIN:VEVENT DTSTART;TZID=America/Los_Angeles:20160113T09 DTEND;TZID=America/Los_Angeles:20160113T095000 RRULE:FREQ=WEEKLY;INTERVAL=2;BYDAY=WE DTSTAMP:20160113T165837Z ORGANIZER;CN=Adam Bordelon:mailto:a...@mesosphere.io UID:f941u4avg2vf4319366dajm73o_r20160113t170...@google.com ATTENDEE;CUTYPE=INDIVIDUAL;ROLE=REQ-PARTICIPANT;PARTSTAT=ACCEPTED;RSVP=TRUE ;CN=Adam Bordelon;X-NUM-GUESTS=0:mailto:a...@mesosphere.io ATTENDEE;CUTYPE=INDIVIDUAL;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP= TRUE;CN=dev@myriad.incubator.apache.org;X-NUM-GUESTS=0:mailto:d...@myriad.in cubator.apache.org ATTENDEE;CUTYPE=INDIVIDUAL;ROLE=REQ-PARTICIPANT;PARTSTAT=ACCEPTED;RSVP=TRUE ;CN=mo...@mesosphere.io;X-NUM-GUESTS=0:mailto:mo...@mesosphere.io ATTENDEE;CUTYPE=INDIVIDUAL;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP= TRUE;CN=Ken Sipe;X-NUM-GUESTS=0:mailto:k...@mesosphere.io CREATED:20150408T091607Z DESCRIPTION:Biweekly meeting for the Myriad developer community to discuss: \n- Announcements and Current Events (conferences\, meetups\, etc.)\n- Incu bator Updates\n- Pull Requests in Review\n- Features/issues in progress\n\n Agenda/Notes: https://docs.google.com/a/mesosphere.io/document/d/1JGmJrgeg9 8bHw_0_sSRmyX6WiAe13OdErcFlaz6Aa04/edit#heading=h.aaws1n2fsgsi\n\nThis even t has a Google Hangouts video call.\nJoin: https://plus.google.com/hangouts /_/mesosphere.io/myriad?hceid=YWRhbUBtZXNvc3BoZXJlLmlv.f941u4avg2vf4319366d ajm73o&hs=121\n\nView your event at https://www.google.com/calendar/event?a ction=VIEW&eid=Zjk0MXU0YXZnMnZmNDMxOTM2NmRham03M29fUjIwMTYwMTEzVDE3MDAwMCBk ZXZAbXlyaWFkLmluY3ViYXRvci5hcGFjaGUub3Jn&tok=MTgjYWRhbUBtZXNvc3BoZXJlLmlvMz IzZjAwNDFkNzQ4ODcwMjRmN2YwMzhlYTMzNjBhMGNmMTgxZjljMw&ctz=America/Los_Angele s&hl=en. LAST-MODIFIED:20160113T165836Z LOCATION: SEQUENCE:1 STATUS:CONFIRMED SUMMARY:Myriad Dev Sync TRANSP:OPAQUE ATTACH;FILENAME=Myriad Sync Notes;FMTTYPE=application/vnd.google-apps.docum ent:https://docs.google.com/a/mesosphere.io/document/d/1JGmJrgeg98bHw_0_sSR myX6WiAe13OdErcFlaz6Aa04/edit?usp=drive_web END:VEVENT END:VCALENDAR invite.ics Description: application/ics
[jira] [Updated] (MYRIAD-153) Placeholder tasks yarn_container_* is not cleaned after yarn job is complete.
[ https://issues.apache.org/jira/browse/MYRIAD-153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Santosh Marella updated MYRIAD-153: --- Fix Version/s: Myriad 0.2.0 > Placeholder tasks yarn_container_* is not cleaned after yarn job is complete. > - > > Key: MYRIAD-153 > URL: https://issues.apache.org/jira/browse/MYRIAD-153 > Project: Myriad > Issue Type: Bug >Reporter: Sarjeet Singh >Assignee: DarinJ > Fix For: Myriad 0.2.0 > > Attachments: Mesos_UI_screeshot_placeholder_tasks_running.png > > > Observed the placeholder tasks for containers launched on FGS are still in > RUNNING state on mesos. These container tasks are not cleaned up properly > after job is finished completely. > see screenshot attached for mesos UI with placeholder tasks still running. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MYRIAD-182) Ability to ignore certificate warnings on config download from SSL secured RM
[ https://issues.apache.org/jira/browse/MYRIAD-182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15096652#comment-15096652 ] DarinJ commented on MYRIAD-182: --- The config is pulled using the Mesos Fetcher. We will need to determine if it's possible to not use certs with Mesos and if so how. > Ability to ignore certificate warnings on config download from SSL secured RM > - > > Key: MYRIAD-182 > URL: https://issues.apache.org/jira/browse/MYRIAD-182 > Project: Myriad > Issue Type: Bug > Components: Executor >Affects Versions: Myriad 0.1.0 >Reporter: John Omernik > > When SSL is enabled for the Resource Manager, and the executor tries download > the config from /conf, if the CA is not valid a warning is thrown the > download fails. There are many cases where SSL certificate may not be valid > (especially in test, but maybe in production) thus we need the ability to > specify that certificate warnings should be ignored. in that case. The > warning received is: > Failed to fetch 'https://myriadprod.marathonprod.mesos:8090/conf': Error > downloading resource: Peer certificate cannot be authenticated with given CA > certificates -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MYRIAD-185) Incubator Myriad 0.1.0 Build Takes Long
Miguel Bernadin created MYRIAD-185: -- Summary: Incubator Myriad 0.1.0 Build Takes Long Key: MYRIAD-185 URL: https://issues.apache.org/jira/browse/MYRIAD-185 Project: Myriad Issue Type: Bug Affects Versions: Myriad 0.1.0 Environment: OS: Ubuntu 14.04 Kernel: 3.13.0-66-generic Reporter: Miguel Bernadin Priority: Minor Hello guys, I've noticed that the build for incubator-0.1.0 took 28 mins to complete. I performed, git clone https://github.com/apache/incubator-myriad and ./gradlew build and here are the logs below. As you can see below it hangs at > Building 65% for 18 mins or so. It throws an error then runs for about 3 mins more minutes and says it finishes successfully. Any thoughts? --- LOGS Starting a new Gradle Daemon for this build (subsequent builds will be faster). Parallel execution is an incubating feature. :myriad-commons:compileJava :myriad-scheduler:nodeSetup :myriad-executor:processResources UP-TO-DATE :myriad-executor:processTestResources UP-TO-DATE warning: [options] bootstrap class path not set in conjunction with -source 1.7 1 warning :myriad-commons:compileGroovy UP-TO-DATE :myriad-commons:processResources UP-TO-DATE :myriad-commons:classes :myriad-commons:jar :myriad-commons:startScripts :myriad-executor:compileJava :myriad-scheduler:compileJava warning: [options] bootstrap class path not set in conjunction with -source 1.7 1 warning :myriad-commons:distTar :myriad-executor:compileGroovy UP-TO-DATE :myriad-executor:classes :myriad-executor:jar :myriad-executor:startScripts warning: [options] bootstrap class path not set in conjunction with -source 1.7 :myriad-executor:distTar :myriad-commons:distZip :myriad-commons:assemble :myriad-executor:distZip :myriad-commons:checkstyleMain :myriad-commons:compileTestJava UP-TO-DATE :myriad-commons:compileTestGroovy UP-TO-DATE :myriad-commons:processTestResources UP-TO-DATE :myriad-commons:testClasses UP-TO-DATE :myriad-commons:checkstyleTest UP-TO-DATE :myriad-commons:findbugsMain 1 warning :myriad-scheduler:compileGroovy UP-TO-DATE :myriad-scheduler:installGulp :myriad-executor:assemble :myriad-executor:capsule :myriad-executor:checkstyleMain :myriad-executor:compileTestJava UP-TO-DATE :myriad-executor:compileTestGroovy UP-TO-DATE :myriad-executor:testClasses UP-TO-DATE :myriad-executor:checkstyleTest UP-TO-DATE :myriad-executor:findbugsMain :myriad-commons:findbugsTest UP-TO-DATE :myriad-commons:pmdMain Scanning archives (68 / 68) 2 analysis passes to perform :myriad-commons:pmdTest UP-TO-DATE :myriad-commons:test UP-TO-DATE :myriad-commons:check :myriad-commons:build Pass 1: Analyzing classes (39 / 67) - 58% complete Pass 1: Analyzing classes (67 / 67) - 100% complete Pass 2: Analyzing classes (3 / 3) - 100% complete Done with analysis :myriad-executor:findbugsTest UP-TO-DATE :myriad-executor:pmdMain :myriad-executor:pmdTest UP-TO-DATE :myriad-executor:test UP-TO-DATE :myriad-executor:check :myriad-executor:build > Building 65% :myriad-scheduler:npmSetup SKIPPED :myriad-scheduler:npmInstall npm WARN peerDependencies The peer dependency stream-browserify@* included from bl will no npm WARN peerDependencies longer be automatically installed to fulfill the peerDependency npm WARN peerDependencies in npm 3+. Your application will need to depend on it explicitly. npm WARN deprecated lodash@2.4.2: lodash@<3.0.0 is no longer maintained. Upgrade to lodash@^3.0.0. npm WARN optional dep failed, continuing fsevents@0.3.8 npm WARN deprecated lodash@1.0.2: lodash@<3.0.0 is no longer maintained. Upgrade to lodash@^3.0.0. > jpegtran-bin@3.0.6 postinstall > /root/.dsh/mesos-configurator/incubator-myriad-0.1.0/myriad-scheduler/src/main/resources/webapp/node_modules/gulp-imagemin/node_modules/imagemin/node_modules/imagemin-jpegtran/node_modules/jpegtran-bin > node lib/install.js module.js:340 throw err; > optipng-bin@3.0.3 postinstall > /root/.dsh/mesos-configurator/incubator-myriad-0.1.0/myriad-scheduler/src/main/resources/webapp/node_modules/gulp-imagemin/node_modules/imagemin/node_modules/imagemin-optipng/node_modules/optipng-bin ^ > node lib/install.js Error: Cannot find module '/root/.dsh/mesos-configurator/incubator-myriad-0.1.0/myriad-scheduler/src/main/resources/webapp/node_modules/gulp-imagemin/node_modules/imagemin/node_modules/imagemin-jpegtran/node_modules/jpegtran-bin/lib/install.js' at Function.Module._resolveFilename (module.js:338:15) at Function.Module._load (module.js:280:25) at Function.Module.runMain (module.js:497:10) at startup (node.js:119:16) at node.js:902:3 npm WARN cann
[jira] [Commented] (MYRIAD-179) Support Revocable resources in Mesos
[ https://issues.apache.org/jira/browse/MYRIAD-179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15096851#comment-15096851 ] DarinJ commented on MYRIAD-179: --- This would be a really cool feature, I have some concerns on the difficulty and maintainability though. In particular, if we want to kill a placeholder task we'll need to go deeper into the NodeManager than MyriadExecutorAuxServices to kill the actual container. This might mean we will have to extend the NodeManager class (and others), and possibly using reflection to access some private variables and methods. While possible it may be very difficult to maintain, hopefully we could contribute changes to the YARN project to make this easier. There's been some recent commits/issues on the 2.8.0 branch that may help in particular YARN-291. > Support Revocable resources in Mesos > > > Key: MYRIAD-179 > URL: https://issues.apache.org/jira/browse/MYRIAD-179 > Project: Myriad > Issue Type: Improvement > Components: Scheduler >Affects Versions: Myriad 0.1.0 >Reporter: John Omernik > > Mesos has introduced revocable resources. Based on my reading of things, > Myriad would be an awesome use case for over subscription, especially when > you combine it with the Fine Grain Scaling (FGS). > Based on what I've read on oversubscription, if Myriad was aware of > oversubscription, we could have Myriad be smart about various Yarn > containers. Have some jobs that may be production jobs, be tagged in such a > way that they could run on non-revocable resources, but we could have other > yarn jobs with certain users/flags, especially in FGS mode, be submitted > using the revocable resources. This would be exceptionally powerful for big > map reduce jobs etc. > These are the jobs that would be adhoc in nature, and in addition to not > using resources when no jobs are running, the node managers, when they did > run certain jobs would run on the revocable resources so they could be killed > if needed. > I am speaking now not from a Dev perspective, so this may be a lot harder > than it seems, I am just trying to outline use cases. > Another use case (I think both are very valid and worth pursuing) would be > once we have the the multi-tenancy built in, have a whole myriad framework > dedicated to adhoc type jobs, and have another myriad framework dedicated to > production jobs. These adhoc jobs could be setup in such a way that all > submissions would be run with revocable resources. Thus being appropriate for > dev work, or other non production type jobs. Obviously this hinges on being > able to run two Myriad clusters on the same Mesos cluster. > The other thing, is a whole frame was set to be revocable resources, we'd > have to ensure the resource manager was running on non-revocable resources... > while containers for Yarn jobs can be killed, we don't want the whole > framework to die. > I see use cases for both, this just seems to add another layer of awesome > flexibility as it pertains to jobs on the cluster. > I'd be interested in flushing this idea out more with the dev team. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
