Hi Mykola, I'd have thought it would be worth giving it some time to review, given that it is such a significant change. I'm personally of the opinion that significant changes like this shouldn't be merged without review. I was hoping to do some review on it this week myself, though I don't really get time to work on Brooklyn these days, so I was also hoping that others would review it who are still close to the code. Did you get a chance to look through it yourself?
What do you all think? I guess if it is in place and doesn't break anything then it's OK? Geoff On Mon, 12 Sept 2022 at 09:04, Mykola Mandra <mykola.man...@cloudsoftcorp.com> wrote: > > Hell All, > > Was I too hasty to merge the PR - > https://github.com/apache/brooklyn-server/pull/1358? > <https://github.com/apache/brooklyn-server/pull/1358?> > We agreed to split the some of the work with Alex, and I did not see why it > could not be merged to continue with what I’m working on. > > Regards, > Mykola > > > On 9 Sep 2022, at 20:17, Geoff Macartney <geom...@apache.org> wrote: > > > > Hi Alex, > > > > Thanks for the link, will have a look at the PR. > > > > As for the new shorthand proposal, +1 from me. I think it will be a > > majority of cases when things *are* simple - setting sensors, etc. > > Your example above goes from eight lines of markup for the steps to > > two, I think that's a compelling argument in itself in favour of the > > proposal. > > > > Cheers > > Geoff > > > > On Thu, 8 Sept 2022 at 09:35, Alex Heneveld <a...@cloudsoft.io> wrote: > >> > >> Hi team, > >> > >> An initial PR with a few types and the ability to define an effector is > >> available [1]. > >> > >> This is enough for the next steps to be parallelized, e.g. new steps > >> added. The proposal has been updated with a work plan / list of tasks > >> [2]. Any volunteers to help with some of the upcoming tasks let me know. > >> > >> Finally I've been thinking about the "shorthand syntax" and how to bring us > >> closer to Peter's proposal of a DSL. The original proposal allowed instead > >> of a map e.g. > >> > >> step_sleep: > >> type: sleep > >> duration: 5s > >> > >> or > >> > >> step_update_service_up: > >> type: set-sensor > >> sensor: > >> name: service.isUp > >> type: boolean > >> value: true > >> > >> being able to use a shorthand _map_ with a single key being the type, and > >> value interpreted by that type, so in the OLD SHORTHAND PROPOSAL the above > >> could be written: > >> > >> step_sleep: > >> sleep: 5s > >> > >> step_update_service_up: > >> set-sensor: service.isUp = true > >> > >> Having played with syntaxes a bit I wonder if we should instead say the > >> shorthand DSL kicks in when the step _body_ is a string (instead of a > >> single-key map), and the first word of the string being the type, and the > >> remainder interpreted by the type, and we allow it to be a bit more > >> ambitious. > >> > >> Concretely this NEW SHORTHAND PROPOSAL would look something like: > >> > >> step_sleep: sleep 5s > >> step_update_service_up: set-sensor service.isUp = true > >> # also supporting a type, ie `set-sensor [TYPE] NAME = VALUE`, eg > >> step_update_service_up: set-sensor boolean service.isUp = true > >> > >> You would still need the full map syntax whenever defining flow logic -- eg > >> condition, next, retry, or timeout -- or any property not supported by the > >> shorthand syntax. But for the (majority?) simple cases the expression > >> would be very concise. In most cases I think it would feel like a DSL but > >> has the virtue of a very clear translation to the actual workflow model and > >> the underlying (YAML) model needed for resumption and UI. > >> > >> As a final example, the example used at the start of the proposal > >> (simplified a little -- removing on-error retry and env map as those > >> wouldn't be supported by shorthand): > >> > >> brooklyn.initializers: > >> - type: workflow-effector > >> name: run-spark-on-gcp > >> steps: > >> 1: > >> type: container > >> image: my/google-cloud > >> command: gcloud dataproc jobs submit spark > >> --BUCKET=gs://$brooklyn:config("bucket") > >> 2: > >> type: set-sensor > >> sensor: spark-output > >> value: ${1.stdout} > >> > >> Could be written in this shorthand as follows: > >> > >> steps: > >> 1: container my/google-cloud command "gcloud dataproc jobs submit spark > >> --BUCKET=gs://${entity.config.bucket}" > >> 2: set-sensor spark-output ${1.stdout} > >> > >> Thoughts? > >> > >> Best > >> Alex > >> > >> > >> [1] https://github.com/apache/brooklyn-server/pull/1358 > >> [2] > >> https://docs.google.com/document/d/1u02Bi6sS8Fkf1s7UzRRMnvLhA477bqcyxGa0nJesqkI/edit#heading=h.gbadaqa2yql6 > >> > >> > >> On Wed, 7 Sept 2022 at 09:58, Alex Heneveld <a...@cloudsoft.io> wrote: > >> > >>> Hi Peter, > >>> > >>> Yes - thanks for the extra details. I did take your suggestion to be a > >>> procedural DSL not YAML, per the illustration at [1] (second code block). > >>> Probably where I was confusing was in saying that unlike DSLs which just > >>> run (and where the execution can be delegated to eg java/groovy/ruby), > >>> here > >>> we need to understand and display, store and resume the workflow progress. > >>> So I think it needs to be compiled to some representation that is well > >>> described and that new Apache Brooklyn code can reason about, both in the > >>> UI (JS) and backend (Java). Parsing a DSL is much harder than using YAML > >>> for this "reasonable" representation (as in we can reason _about_ it :) ), > >>> because we already have good backend processing, persistence, > >>> serialization; and frontend processing and visualization support for > >>> YAML-based models. So I think we almost definitely want a well-described > >>> declarative YAML model of the workflow. > >>> > >>> We might *also* want a Workflow DSL because I agree with you a DSL would > >>> be nicer for a user to write (if writing by hand; although if composing > >>> visually a drag-and-drop to YAML is probably easier). However it should > >>> probably get "compiled" into a Workflow YAML. So I'm suggesting we do the > >>> workflow YAML at this stage, and a DSL that compiles into that YAML can be > >>> designed later. (Designing a good DSL and parser and reason-about-able > >>> representation is a big task, so being able to separate it feels good > >>> too!) > >>> > >>> Best > >>> Alex > >>> > >>> [1] > >>> https://docs.google.com/document/d/1u02Bi6sS8Fkf1s7UzRRMnvLhA477bqcyxGa0nJesqkI/edit#heading=h.75wm48pjvx0h > >>> > >>> > >>> On Fri, 2 Sept 2022 at 20:17, Geoff Macartney <geoff.macart...@gmail.com> > >>> wrote: > >>> > >>>> Hi Peter, > >>>> > >>>> Thanks for such a detailed writeup of how you see this working. I fear > >>>> I've too little experience with this sort of thing to be able to say > >>>> anything very useful about it. My thought on the matter would be, > >>>> let's get started with the yaml based approach and see how it goes. I > >>>> think that experience would then give us a much better feel for what a > >>>> really nice and usable DSL for workflows would look like (probably to > >>>> address all the pain points of the yaml approach! :-) The outline > >>>> above will then be a good starting point, I'm sure. > >>>> > >>>> Cheers > >>>> Geoff > >>>> > >>>> On Thu, 1 Sept 2022 at 21:26, Peter Abramowitsch > >>>> <pabramowit...@gmail.com> wrote: > >>>>> > >>>>> Hi All > >>>>> I just wanted to clarify something in my comment the other day about > >>>> DSLs > >>>>> since I see that the acronym was also used in Alex's original document. > >>>>> Unless I misunderstood, Alex was proposing to create a DSL for Brooklyn > >>>>> using yaml as syntax and writing a code layer to translate between that > >>>>> syntax and underlying APIs which are presumably all in Java. > >>>>> > >>>>> What I was suggesting was a DSL written directly in Java (I guess) > >>>> whose > >>>>> syntax would be that language, but whose grammar would be keywords that > >>>>> were also Java functions. Some of these functions would be pre-defined > >>>> in > >>>>> the DSL, while others could be defined by the user and could use other > >>>>> functions of the DSL. The result would be turned into a JAR file (or > >>>>> equivalent in another platform) But during the compile phase, it > >>>> would be > >>>>> checked for errors, and it could be debugged line by line either > >>>> invoking > >>>>> live functionality or using a library of mock versions of the Brooklyn > >>>> API. > >>>>> > >>>>> In this 'native' DSL one could provide different types of workflow > >>>>> constructs as functions (In the BaseClass), taking function names as > >>>> method > >>>>> pointers, or using Lambdas. It would be a lot easier in Ruby or Python > >>>>> > >>>>> // linear > >>>>> brooklynRun(NamedTaskMethod, NamedTaskMethod) > >>>>> > >>>>> // chained > >>>>> TaskMethodA()TaskMethodB(). > >>>>> > >>>>> // asynchronous > >>>>> brooklynJoin(NamedTaskMethod, NamedTaskMethod,...) > >>>>> > >>>>> // conditional > >>>>> brooklynRunIf(NamedTaskMethod, NamedConditionMethod,...) > >>>>> > >>>>> // iterative > >>>>> brooklynRunWhile(NamedTaskMethod, NamedConditionMethod,...) > >>>>> brooklynRunUntil(NamedTaskMethod, NamedConditionMethod,...) > >>>>> > >>>>> // there could even be a utility to implement legacy syntax (this of > >>>> course > >>>>> would require the extra code layer I was trying to avoid) > >>>>> runYaml(Path) > >>>>> > >>>>> A basic class structure might be > >>>>> > >>>>> // where BrooklynRecipeBase implements the utility functions including, > >>>>> among others Join, Run, If, While, Until mentioned above > >>>>> // and the BrooklynWorkflowInterface would dictate the functional > >>>>> requirements for the mandatory aspects of the Recipe. > >>>>> class MyRecipe extends BrooklynRecipeBase implements, > >>>>> BrooklynWorkflowInterface > >>>>> { > >>>>> Initialize() > >>>>> createContext() - spin up resources > >>>>> workflow() - the main launch sequence using aspects of the DSL > >>>>> monitoring() - an asynchronous workflow used to manage sensor output or > >>>> for > >>>>> whatever needs to be done while the "orchestra" is plating > >>>>> shutdownHook() - called whenever shutdown is happening > >>>>> } > >>>>> > >>>>> For those who don't like the smell of Java, the source file could just > >>>> be > >>>>> the contents, which would then be injected into the class framing code > >>>>> before compilation. > >>>>> > >>>>> These are just ideas. I'm not familiar enough with Brooklyn in its > >>>> current > >>>>> implementation to be able to create realistic pseudocode. > >>>>> > >>>>> Peter > >>>>> > >>>>> On Thu, Sep 1, 2022 at 9:24 AM Geoff Macartney < > >>>> geoff.macart...@gmail.com> > >>>>> wrote: > >>>>> > >>>>>> Hi Alex, > >>>>>> > >>>>>> That's great, I'll be excited to hear all about it. 7th September > >>>>>> suits me fine; I would probably prefer 4.00 p.m. over 11.00. > >>>>>> > >>>>>> Cheers > >>>>>> Geoff > >>>>>> > >>>>>> On Thu, 1 Sept 2022 at 12:41, Alex Heneveld <a...@cloudsoft.io> > >>>> wrote: > >>>>>>> > >>>>>>> Thanks for the excellent feedback Geoff and yes there are some very > >>>> cool > >>>>>> and exciting things added recently -- containers, conditions, and > >>>> terraform > >>>>>> and kubernetes support, all of which make writing complex blueprints > >>>> much > >>>>>> easier. > >>>>>>> > >>>>>>> I'd love to host a session to showcase these. > >>>>>>> > >>>>>>> How does Wed 7 Sept sound? I could do 11am UK or 4pm UK -- > >>>> depending > >>>>>> what time suits for people who are interested. Please RSVP and > >>>> indicate > >>>>>> your time preference! > >>>>>>> > >>>>>>> Best > >>>>>>> Alex > >>>>>>> > >>>>>>> > >>>>>>> On Wed, 31 Aug 2022 at 22:17, Geoff Macartney < > >>>> geoff.macart...@gmail.com> > >>>>>> wrote: > >>>>>>>> > >>>>>>>> Hi Alex, > >>>>>>>> > >>>>>>>> Another thought occurred to me when reading that workflow > >>>> proposal. You > >>>>>> wrote > >>>>>>>> > >>>>>>>> "and with the recent support for container-based tasks and > >>>> declarative > >>>>>>>> conditions, we have taken big steps towards enabling YAML > >>>> authorship" > >>>>>>>> > >>>>>>>> Unfortunately over the past while I haven't been able to keep up as > >>>>>>>> closely as I would like with developments in Brooklyn. I'm just > >>>>>>>> wondering if it might be possible to get together some time, on > >>>> Google > >>>>>>>> Meet or Zoom or whatnot, if you or a colleague could spare half an > >>>>>>>> hour to demo some of these recent developments? But don't worry > >>>> about > >>>>>>>> it if you're too busy at present. > >>>>>>>> > >>>>>>>> Adding dev@ to this in CC for the sake of Openness. Others might > >>>> also > >>>>>>>> be interested! > >>>>>>>> > >>>>>>>> Cheers > >>>>>>>> Geoff > >>>>>> > >>>> > >>> >
