Hi Alex,
Thanks for the link, will have a look at the PR.
As for the new shorthand proposal, +1 from me. I think it will be a
majority of cases when things *are* simple - setting sensors, etc.
Your example above goes from eight lines of markup for the steps to
two, I think that's a compelling argument in itself in favour of the
proposal.
Cheers
Geoff
On Thu, 8 Sept 2022 at 09:35, Alex Heneveld <a...@cloudsoft.io> wrote:
>
> Hi team,
>
> An initial PR with a few types and the ability to define an effector is
> available [1].
>
> This is enough for the next steps to be parallelized, e.g. new steps
> added. The proposal has been updated with a work plan / list of tasks
> [2]. Any volunteers to help with some of the upcoming tasks let me know.
>
> Finally I've been thinking about the "shorthand syntax" and how to bring us
> closer to Peter's proposal of a DSL. The original proposal allowed instead
> of a map e.g.
>
> step_sleep:
> type: sleep
> duration: 5s
>
> or
>
> step_update_service_up:
> type: set-sensor
> sensor:
> name: service.isUp
> type: boolean
> value: true
>
> being able to use a shorthand _map_ with a single key being the type, and
> value interpreted by that type, so in the OLD SHORTHAND PROPOSAL the above
> could be written:
>
> step_sleep:
> sleep: 5s
>
> step_update_service_up:
> set-sensor: service.isUp = true
>
> Having played with syntaxes a bit I wonder if we should instead say the
> shorthand DSL kicks in when the step _body_ is a string (instead of a
> single-key map), and the first word of the string being the type, and the
> remainder interpreted by the type, and we allow it to be a bit more
> ambitious.
>
> Concretely this NEW SHORTHAND PROPOSAL would look something like:
>
> step_sleep: sleep 5s
> step_update_service_up: set-sensor service.isUp = true
> # also supporting a type, ie `set-sensor [TYPE] NAME = VALUE`, eg
> step_update_service_up: set-sensor boolean service.isUp = true
>
> You would still need the full map syntax whenever defining flow logic -- eg
> condition, next, retry, or timeout -- or any property not supported by the
> shorthand syntax. But for the (majority?) simple cases the expression
> would be very concise. In most cases I think it would feel like a DSL but
> has the virtue of a very clear translation to the actual workflow model and
> the underlying (YAML) model needed for resumption and UI.
>
> As a final example, the example used at the start of the proposal
> (simplified a little -- removing on-error retry and env map as those
> wouldn't be supported by shorthand):
>
> brooklyn.initializers:
> - type: workflow-effector
> name: run-spark-on-gcp
> steps:
> 1:
> type: container
> image: my/google-cloud
> command: gcloud dataproc jobs submit spark
> --BUCKET=gs://$brooklyn:config("bucket")
> 2:
> type: set-sensor
> sensor: spark-output
> value: ${1.stdout}
>
> Could be written in this shorthand as follows:
>
> steps:
> 1: container my/google-cloud command "gcloud dataproc jobs submit spark
> --BUCKET=gs://${entity.config.bucket}"
> 2: set-sensor spark-output ${1.stdout}
>
> Thoughts?
>
> Best
> Alex
>
>
> [1] https://github.com/apache/brooklyn-server/pull/1358
> [2]
> https://docs.google.com/document/d/1u02Bi6sS8Fkf1s7UzRRMnvLhA477bqcyxGa0nJesqkI/edit#heading=h.gbadaqa2yql6
>
>
> On Wed, 7 Sept 2022 at 09:58, Alex Heneveld <a...@cloudsoft.io> wrote:
>
> > Hi Peter,
> >
> > Yes - thanks for the extra details. I did take your suggestion to be a
> > procedural DSL not YAML, per the illustration at [1] (second code block).
> > Probably where I was confusing was in saying that unlike DSLs which just
> > run (and where the execution can be delegated to eg java/groovy/ruby), here
> > we need to understand and display, store and resume the workflow progress.
> > So I think it needs to be compiled to some representation that is well
> > described and that new Apache Brooklyn code can reason about, both in the
> > UI (JS) and backend (Java). Parsing a DSL is much harder than using YAML
> > for this "reasonable" representation (as in we can reason _about_ it :) ),
> > because we already have good backend processing, persistence,
> > serialization; and frontend processing and visualization support for
> > YAML-based models. So I think we almost definitely want a well-described
> > declarative YAML model of the workflow.
> >
> > We might *also* want a Workflow DSL because I agree with you a DSL would
> > be nicer for a user to write (if writing by hand; although if composing
> > visually a drag-and-drop to YAML is probably easier). However it should
> > probably get "compiled" into a Workflow YAML. So I'm suggesting we do the
> > workflow YAML at this stage, and a DSL that compiles into that YAML can be
> > designed later. (Designing a good DSL and parser and reason-about-able
> > representation is a big task, so being able to separate it feels good too!)
> >
> > Best
> > Alex
> >
> > [1]
> > https://docs.google.com/document/d/1u02Bi6sS8Fkf1s7UzRRMnvLhA477bqcyxGa0nJesqkI/edit#heading=h.75wm48pjvx0h
> >
> >
> > On Fri, 2 Sept 2022 at 20:17, Geoff Macartney <geoff.macart...@gmail.com>
> > wrote:
> >
> >> Hi Peter,
> >>
> >> Thanks for such a detailed writeup of how you see this working. I fear
> >> I've too little experience with this sort of thing to be able to say
> >> anything very useful about it. My thought on the matter would be,
> >> let's get started with the yaml based approach and see how it goes. I
> >> think that experience would then give us a much better feel for what a
> >> really nice and usable DSL for workflows would look like (probably to
> >> address all the pain points of the yaml approach! :-) The outline
> >> above will then be a good starting point, I'm sure.
> >>
> >> Cheers
> >> Geoff
> >>
> >> On Thu, 1 Sept 2022 at 21:26, Peter Abramowitsch
> >> <pabramowit...@gmail.com> wrote:
> >> >
> >> > Hi All
> >> > I just wanted to clarify something in my comment the other day about
> >> DSLs
> >> > since I see that the acronym was also used in Alex's original document.
> >> > Unless I misunderstood, Alex was proposing to create a DSL for Brooklyn
> >> > using yaml as syntax and writing a code layer to translate between that
> >> > syntax and underlying APIs which are presumably all in Java.
> >> >
> >> > What I was suggesting was a DSL written directly in Java (I guess)
> >> whose
> >> > syntax would be that language, but whose grammar would be keywords that
> >> > were also Java functions. Some of these functions would be pre-defined
> >> in
> >> > the DSL, while others could be defined by the user and could use other
> >> > functions of the DSL. The result would be turned into a JAR file (or
> >> > equivalent in another platform) But during the compile phase, it
> >> would be
> >> > checked for errors, and it could be debugged line by line either
> >> invoking
> >> > live functionality or using a library of mock versions of the Brooklyn
> >> API.
> >> >
> >> > In this 'native' DSL one could provide different types of workflow
> >> > constructs as functions (In the BaseClass), taking function names as
> >> method
> >> > pointers, or using Lambdas. It would be a lot easier in Ruby or Python
> >> >
> >> > // linear
> >> > brooklynRun(NamedTaskMethod, NamedTaskMethod)
> >> >
> >> > // chained
> >> > TaskMethodA()TaskMethodB().
> >> >
> >> > // asynchronous
> >> > brooklynJoin(NamedTaskMethod, NamedTaskMethod,...)
> >> >
> >> > // conditional
> >> > brooklynRunIf(NamedTaskMethod, NamedConditionMethod,...)
> >> >
> >> > // iterative
> >> > brooklynRunWhile(NamedTaskMethod, NamedConditionMethod,...)
> >> > brooklynRunUntil(NamedTaskMethod, NamedConditionMethod,...)
> >> >
> >> > // there could even be a utility to implement legacy syntax (this of
> >> course
> >> > would require the extra code layer I was trying to avoid)
> >> > runYaml(Path)
> >> >
> >> > A basic class structure might be
> >> >
> >> > // where BrooklynRecipeBase implements the utility functions including,
> >> > among others Join, Run, If, While, Until mentioned above
> >> > // and the BrooklynWorkflowInterface would dictate the functional
> >> > requirements for the mandatory aspects of the Recipe.
> >> > class MyRecipe extends BrooklynRecipeBase implements,
> >> > BrooklynWorkflowInterface
> >> > {
> >> > Initialize()
> >> > createContext() - spin up resources
> >> > workflow() - the main launch sequence using aspects of the DSL
> >> > monitoring() - an asynchronous workflow used to manage sensor output or
> >> for
> >> > whatever needs to be done while the "orchestra" is plating
> >> > shutdownHook() - called whenever shutdown is happening
> >> > }
> >> >
> >> > For those who don't like the smell of Java, the source file could just
> >> be
> >> > the contents, which would then be injected into the class framing code
> >> > before compilation.
> >> >
> >> > These are just ideas. I'm not familiar enough with Brooklyn in its
> >> current
> >> > implementation to be able to create realistic pseudocode.
> >> >
> >> > Peter
> >> >
> >> > On Thu, Sep 1, 2022 at 9:24 AM Geoff Macartney <
> >> geoff.macart...@gmail.com>
> >> > wrote:
> >> >
> >> > > Hi Alex,
> >> > >
> >> > > That's great, I'll be excited to hear all about it. 7th September
> >> > > suits me fine; I would probably prefer 4.00 p.m. over 11.00.
> >> > >
> >> > > Cheers
> >> > > Geoff
> >> > >
> >> > > On Thu, 1 Sept 2022 at 12:41, Alex Heneveld <a...@cloudsoft.io>
> >> wrote:
> >> > > >
> >> > > > Thanks for the excellent feedback Geoff and yes there are some very
> >> cool
> >> > > and exciting things added recently -- containers, conditions, and
> >> terraform
> >> > > and kubernetes support, all of which make writing complex blueprints
> >> much
> >> > > easier.
> >> > > >
> >> > > > I'd love to host a session to showcase these.
> >> > > >
> >> > > > How does Wed 7 Sept sound? I could do 11am UK or 4pm UK --
> >> depending
> >> > > what time suits for people who are interested. Please RSVP and
> >> indicate
> >> > > your time preference!
> >> > > >
> >> > > > Best
> >> > > > Alex
> >> > > >
> >> > > >
> >> > > > On Wed, 31 Aug 2022 at 22:17, Geoff Macartney <
> >> geoff.macart...@gmail.com>
> >> > > wrote:
> >> > > >>
> >> > > >> Hi Alex,
> >> > > >>
> >> > > >> Another thought occurred to me when reading that workflow
> >> proposal. You
> >> > > wrote
> >> > > >>
> >> > > >> "and with the recent support for container-based tasks and
> >> declarative
> >> > > >> conditions, we have taken big steps towards enabling YAML
> >> authorship"
> >> > > >>
> >> > > >> Unfortunately over the past while I haven't been able to keep up as
> >> > > >> closely as I would like with developments in Brooklyn. I'm just
> >> > > >> wondering if it might be possible to get together some time, on
> >> Google
> >> > > >> Meet or Zoom or whatnot, if you or a colleague could spare half an
> >> > > >> hour to demo some of these recent developments? But don't worry
> >> about
> >> > > >> it if you're too busy at present.
> >> > > >>
> >> > > >> Adding dev@ to this in CC for the sake of Openness. Others might
> >> also
> >> > > >> be interested!
> >> > > >>
> >> > > >> Cheers
> >> > > >> Geoff
> >> > >
> >>
> >
