That's correct, it needs define dependency between paragraphs, e.g. %spark(deps=p1), so that we can build DAG for the whole pipeline.
Rick Moritz <rah...@gmail.com>于2017年4月6日周四 下午3:28写道： > This actually calls for a dependency definition of notes within a > notebook, so the scheduler can decide which tasks to run simultaneously. > I suggest a simple counter of dependency levels, which by default > increases with every new note and can be decremented to allow notes to run > simultaneously. Run-all then submits each level into the target > interpreters for this level, awaits termination of all results, and then > starts the next level's note. > > > On Thu, Apr 6, 2017 at 12:57 AM, moon soo Lee <m...@apache.org> wrote: > > Hi, > > That's expected behavior at the moment. The reason is > > Each interpreter has it's own scheduler (either FIFO, Parallel), and > run-all just submit all paragraphs into target interpreter's scheduler. > > I think we can add feature such as run-all-sequentially. > Do you mind file a JIRA issue? > > Thanks, > moon > > On Thu, Apr 6, 2017 at 5:35 AM <murexconsult...@googlemail.com> wrote: > > I often have notebooks that have a %sh as the 1st paragraph. This scps > some file from another server, and then a number of spark or sparksql > paragraphs are after that. > > If I click on the run-all paragraphs at the top of the notebook the 1st > %sh paragraph kicks off as expected, but the 2nd %spark notebook starts too > at the same time. The others go into pending state and then start once the > spark one has completed. > > Is this a bug? Or am I doing something wrong? > > Thanks > > >
