Re: scheduler braindump: architecture, gotchas, etc.
Thx Xiao! On Mon, Feb 4, 2019 at 9:04 AM Xiao Li wrote: > Thank you, Imran! > > Also, I attached the slides of "Deep Dive: Scheduler of Apache Spark". > > Cheers, > > Xiao > > > > John Zhuge 于2019年2月4日周一 上午8:59写道: > >> Thanks Imran! >> >> On Mon, Feb 4, 2019 at 8:42 AM Imran Rashid >> wrote: >> >>> The scheduler has been pretty error-prone and hard to work on, and I >>> feel like there may be a dwindling core of active experts. I'm sure its >>> very discouraging to folks trying to make what seem like simple changes, >>> and then find they are in a rats nest of complex issues they weren't >>> expecting. But for those who are still trying, THANK YOU! more >>> involvement and more folks becoming experts is definitely needed. >>> >>> I put together a doc going over the architecture of the scheduler, and >>> things I've seen us get bitten by in the past. Its sort of a brain dump, >>> but I'm hopeful it'll help orient new folks to the scheduler. I also hope >>> more experts will chime in -- there are places in the doc I know I've >>> missed things, and called that out, but there are probably even more that >>> should be discussed, & mistakes I've made. All input welcome. >>> >>> >>> https://docs.google.com/document/d/1oiE21t-8gXLXk5evo-t-BXpO5Hdcob5D-Ps40hogsp8/edit?usp=sharing >>> >> >> >> -- >> John Zhuge >> > -- John Zhuge
Re: scheduler braindump: architecture, gotchas, etc.
Thanks Li and Imran for providing us an overview about one of the complex module in spark 👍 Excellent sharing. Regards Sujith. On Mon, 4 Feb 2019 at 10:54 PM, Xiao Li wrote: > Thank you, Imran! > > Also, I attached the slides of "Deep Dive: Scheduler of Apache Spark". > > Cheers, > > Xiao > > > > John Zhuge 于2019年2月4日周一 上午8:59写道: > >> Thanks Imran! >> >> On Mon, Feb 4, 2019 at 8:42 AM Imran Rashid >> wrote: >> >>> The scheduler has been pretty error-prone and hard to work on, and I >>> feel like there may be a dwindling core of active experts. I'm sure its >>> very discouraging to folks trying to make what seem like simple changes, >>> and then find they are in a rats nest of complex issues they weren't >>> expecting. But for those who are still trying, THANK YOU! more >>> involvement and more folks becoming experts is definitely needed. >>> >>> I put together a doc going over the architecture of the scheduler, and >>> things I've seen us get bitten by in the past. Its sort of a brain dump, >>> but I'm hopeful it'll help orient new folks to the scheduler. I also hope >>> more experts will chime in -- there are places in the doc I know I've >>> missed things, and called that out, but there are probably even more that >>> should be discussed, & mistakes I've made. All input welcome. >>> >>> >>> https://docs.google.com/document/d/1oiE21t-8gXLXk5evo-t-BXpO5Hdcob5D-Ps40hogsp8/edit?usp=sharing >>> >> >> >> -- >> John Zhuge >> > > - > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
Re: scheduler braindump: architecture, gotchas, etc.
Thank you Imran, this is quite helpful. Regards, Parth Kamlesh Gandhi On Mon, Feb 4, 2019 at 11:01 AM Rubén Berenguel wrote: > Thanks Imran, will definitely give it a look (even if just out of sheer > interest on how the sausage is done) > > R > > > On 4 February 2019 at 17:59:33, John Zhuge (jzh...@apache.org) wrote: > > Thanks Imran! > > On Mon, Feb 4, 2019 at 8:42 AM Imran Rashid > wrote: > >> The scheduler has been pretty error-prone and hard to work on, and I feel >> like there may be a dwindling core of active experts. I'm sure its very >> discouraging to folks trying to make what seem like simple changes, and >> then find they are in a rats nest of complex issues they weren't >> expecting. But for those who are still trying, THANK YOU! more >> involvement and more folks becoming experts is definitely needed. >> >> I put together a doc going over the architecture of the scheduler, and >> things I've seen us get bitten by in the past. Its sort of a brain dump, >> but I'm hopeful it'll help orient new folks to the scheduler. I also hope >> more experts will chime in -- there are places in the doc I know I've >> missed things, and called that out, but there are probably even more that >> should be discussed, & mistakes I've made. All input welcome. >> >> >> https://docs.google.com/document/d/1oiE21t-8gXLXk5evo-t-BXpO5Hdcob5D-Ps40hogsp8/edit?usp=sharing >> > > > -- > John Zhuge > >
Re: scheduler braindump: architecture, gotchas, etc.
Thanks Imran, will definitely give it a look (even if just out of sheer interest on how the sausage is done) R On 4 February 2019 at 17:59:33, John Zhuge (jzh...@apache.org) wrote: Thanks Imran! On Mon, Feb 4, 2019 at 8:42 AM Imran Rashid wrote: > The scheduler has been pretty error-prone and hard to work on, and I feel > like there may be a dwindling core of active experts. I'm sure its very > discouraging to folks trying to make what seem like simple changes, and > then find they are in a rats nest of complex issues they weren't > expecting. But for those who are still trying, THANK YOU! more > involvement and more folks becoming experts is definitely needed. > > I put together a doc going over the architecture of the scheduler, and > things I've seen us get bitten by in the past. Its sort of a brain dump, > but I'm hopeful it'll help orient new folks to the scheduler. I also hope > more experts will chime in -- there are places in the doc I know I've > missed things, and called that out, but there are probably even more that > should be discussed, & mistakes I've made. All input welcome. > > > https://docs.google.com/document/d/1oiE21t-8gXLXk5evo-t-BXpO5Hdcob5D-Ps40hogsp8/edit?usp=sharing > -- John Zhuge
Re: scheduler braindump: architecture, gotchas, etc.
Thanks Imran! On Mon, Feb 4, 2019 at 8:42 AM Imran Rashid wrote: > The scheduler has been pretty error-prone and hard to work on, and I feel > like there may be a dwindling core of active experts. I'm sure its very > discouraging to folks trying to make what seem like simple changes, and > then find they are in a rats nest of complex issues they weren't > expecting. But for those who are still trying, THANK YOU! more > involvement and more folks becoming experts is definitely needed. > > I put together a doc going over the architecture of the scheduler, and > things I've seen us get bitten by in the past. Its sort of a brain dump, > but I'm hopeful it'll help orient new folks to the scheduler. I also hope > more experts will chime in -- there are places in the doc I know I've > missed things, and called that out, but there are probably even more that > should be discussed, & mistakes I've made. All input welcome. > > > https://docs.google.com/document/d/1oiE21t-8gXLXk5evo-t-BXpO5Hdcob5D-Ps40hogsp8/edit?usp=sharing > -- John Zhuge
scheduler braindump: architecture, gotchas, etc.
The scheduler has been pretty error-prone and hard to work on, and I feel like there may be a dwindling core of active experts. I'm sure its very discouraging to folks trying to make what seem like simple changes, and then find they are in a rats nest of complex issues they weren't expecting. But for those who are still trying, THANK YOU! more involvement and more folks becoming experts is definitely needed. I put together a doc going over the architecture of the scheduler, and things I've seen us get bitten by in the past. Its sort of a brain dump, but I'm hopeful it'll help orient new folks to the scheduler. I also hope more experts will chime in -- there are places in the doc I know I've missed things, and called that out, but there are probably even more that should be discussed, & mistakes I've made. All input welcome. https://docs.google.com/document/d/1oiE21t-8gXLXk5evo-t-BXpO5Hdcob5D-Ps40hogsp8/edit?usp=sharing