Hi Colin, no worries, responses below-
On Thu, Mar 5, 2015 at 2:31 PM, Colin Yates <colin.ya...@gmail.com> wrote:
> Hi Jonah,
>
> Coffee consumed, post read - thoughts emerging:
>
> - so the 'context' is essentially a service registry as well as
> run-time state?
>
More the latter- about transient, intermediate state rather than global
state like a registry.
A service registry could properly be a Component that depended on other
components, and then the registry could be delivered in the context map. I
usually include Components in the context map, though not the whole System
map. Flows are like functions, they should be "called" by Components, not
the other way around.
> - if 'renew-session' and 'return-renewed-token' both needed to
> execute in a single transaction that was separate to
> 'some-thing-else', how would that be co-ordinated - would each tx
> boundary mandate a distinct flow for example
>
Right- generally since transaction boundaries are an edge semantic,
transaction scope should ideally be confined to a single top level edge
function, and data needed for the transaction should be prepared
beforehand- perhaps including generating primary key values separately. If
the transaction is complex and requires in-scope processing, perhaps it
should live inside the database. In theory an isolated "flow" could be used
*inside* transaction scope, but any complex logic occurring inside
transaction scope is a smell that perhaps transaction scope could be
rethought.
> - the whole thing feels very 'state-machiney' - is that the point?
>
>
I would say "data-driven".
> Thanks again for sharing. I have to say that the 'flattening' of the
> control, or rather conditional logic seems quite expensive. My mind
> thinks in graphs not flattened lists, but maybe my resistance is just
> the pain of a new paradigm...
>
> At the moment I am slightly nervous as it seems to be conflating a
> whole bunch for the win of a consistent way of moving state around.
>
>
In a way- it's just another view of an execution chain.
We get the stack-based model "for free", where functions call into
functions, into other functions, etc, then are popped back out when they're
done, and conditional logic is typically "embedded" inside functions. When
those functions need data, it's supposed to be available in the "class", or
less often, arrive on the stack, or they have to reach out onto the heap-
into global space shared by all threads. Conditional decisions aren't first
class entities- an "if" doesn't give you a new stack frame. etc. There are
good things about that model for certain kinds of solutions. Clojure begins
to decouple it by, e.g. by not hiding data inside classes.
A data flow model has different priorities and different costs and
benefits, and though it is less "native" to the JVM one can imagine a data
flow "compiler" that makes it a little friendlier.
> Thanks for the time to respond and these questions are those of
> clarity not criticism :).
>
>
Of course!
>
> On 5 March 2015 at 16:45, Jonah Benton <jo...@jonah.com> wrote:
> > Hi Colin- a simple example that uses a flow operator would be
> >
> > (-> context
> > do-something
> > do-some-io
> > do-a-last-thing )
> >
> > with
> >
> > (defn do-something [context]
> > (let [some-data (get-in context [:path :to :data :element])
> > result (pure-edge-function some-data)]
> > (assoc-in context [:path :to :revised :data] result)))
> >
> > and
> >
> > (defn do-some-io [context]
> > (let [some-other-data (get-in context [:path :to :data :to :save])
> > database (get-in context [:path :to :database :resource])]
> > (save! database some-other-data)
> > context)))
> >
> > Then save! can be a plain function, or
> >
> > (defprotocol Saver
> > (save! [this data]))
> >
> > (defrecord DatabaseSaver [conn ... ]
> > Saver
> > (save! ))
> >
> > (defrecord AtomSaver [a ... ]
> > Saver
> > (save! ))
> >
> > And context is a map of maps that contains both application state and
> edge
> > resources that should provide or receive data.
> >
> > Encoding conditional steps in a data flow presents a challenge. Several
> > variants for doing this are in flow operators in Clojure and there are
> also
> > a number of third party approaches. The below is a concrete example from
> an
> > internal library that is not open source but doesn't differ significantly
> > from approaches that are.
> >
> > A "flow" looks like this:
> >
> > (defflow renew
> > { :start valid-request?
> > valid-request? { true valid-token? false invalid-request }
> > valid-token? { true ingest-token false not-authorized }
> > ingest-token renew-session
> > renew-session renewed?
> > renewed? { true return-renewed-token false
> not-authorized
> > }
> > return-renewed-token :end
> > not-authorized :end
> > invalid-request :end })
> >
> > It's a regular map with at least a :start key and an :end value. Symbols
> in
> > a flow map are plain functions that take a context map.
> >
> > Semantically, those functions represent either actions or decisions. A
> > function whose value in the flow map is itself a map with true and false
> > keys is a decision. A function whose value in the map is another
> function is
> > an action.
> >
> > This particular flow performs a renewal of a web-like "session" token. It
> > runs in a Pedestal interceptor, so it receives a context map (with
> :request,
> > :response and other keys) as a parameter and at the end it populates the
> > :response key of the map.
> >
> > The way to read this is-
> >
> > 0. a context map is prepared.
> > 1. the data flow runner looks for the :start key. the value of :start is
> > valid-request?
> > 2. The value of valid-request? is a map, rather than a function, so the
> flow
> > determines valid-request? is a decision function- an ordinary function
> that
> > takes a context and is expected to return true or false.
> > 3. the flow runner executes valid-request? If it returns true, the flow
> > runner proceeds to valid-token? using the same context map. If false,
> > proceed to invalid-request
> > 4. valid-token? is similarly a decision function, it's executed similarly
> > and is also expected to return true or false. invalid-request is an
> action.
> > It's expected to return a new context map- in this case it populates the
> > :response with a 4xx :status and some other response attributes.
> > 5. the flow engine keeps going between keys and values until it comes to
> a
> > value of :end. Then the resulting context is the output of the flow.
> >
> > The action function in the above called "renew-session" can do IO to
> renew a
> > user session in the database. It's copied below.
> >
> > (defn renew-session [context]
> > (let [database (get-in context [:components :database])
> > session-key (get-in context [:work :session-key])
> > renewed? (db/renew-session! database session-key)]
> > (if (not (nil? renewed?))
> > (assoc-in context [:work :renewed-key] session-key)
> > context)))
> >
> > It simply pulls the database connection from the context, calls into a db
> > specific function, db/renew-session!, which takes a connection and a key,
> > and could be a protocol function that accepts a Record that encapsulates
> a
> > specific storage resource, or could be a multimethod.
> >
> > This model of a data flow with actions and decisions follows Amazon's
> Simple
> > Workflow Service, and the clever encoding of that flow as a map is due to
> > Cognitect.
> >
> > A few projects exist now that encode flows in data form with an engine to
> > process them. There are some trade-offs- it takes practice to read them,
> you
> > can trade off a little in performance. In the above there's a fair
> amount of
> > boilerplate that could be abstracted away along the lines of of
> Prismatic's
> > Graph.
> >
> > You get a lot of other benefits- a consistent system shape, separation of
> > concerns between core logic and edge IO, consistent error handling,
> > logging/auditing, etc. Flows can be reused and can themselves compose,
> > because everything takes a context map. You could also generate a visual
> > decision tree of the model, or a real time heat map of decisions or of
> > action performance.
> >
> > Anyway- sorry, this isn't intended to pitch an internal library, rather
> to
> > share another thought process around the problems you raised that points
> in
> > the direction of modeling and then executing the data flow, rather than
> > relying solely on the call stack.
> >
> >
> >
> > On Thu, Mar 5, 2015 at 9:39 AM, Colin Yates <colin.ya...@gmail.com>
> wrote:
> >>
> >> Sounds interesting - are there are instances that I can look at?
> >>
> >> On 5 March 2015 at 14:15, Jonah Benton <jo...@jonah.com> wrote:
> >> > Yes, exactly.
> >> >
> >> > That's fine, a node in the "middle" of the data flow can be
> responsible
> >> > for
> >> > reaching out to the edge to synchronously get some data to feed into a
> >> > decision performed by the data flow. It would just do that through the
> >> > edge
> >> > functions.
> >> >
> >> > A request for data could also be asynchronously initiated once enough
> >> > context had been determined by the data flow, and a promise added to
> the
> >> > context map for later derefing (likely with a timeout).
> >> >
> >> > If the request for the data is context-free, it could be pulled in at
> >> > the
> >> > start of the data flow.
> >> >
> >> > By edge, it's is not that IO can't be done in the middle of a
> dataflow,
> >> > it's
> >> > just that functions that deal with dataflow follow one pattern
> (taking a
> >> > context map) so they can be easily composed and rearranged, while
> >> > functions
> >> > that deal with edge resources take those resources along with other
> >> > parameters more explicitly, so the data flow can run in an insulated
> >> > fashion.
> >> >
> >> >
> >> >
> >> > On Thu, Mar 5, 2015 at 8:55 AM, Colin Yates <colin.ya...@gmail.com>
> >> > wrote:
> >> >>
> >> >> Hi Jonah,
> >> >>
> >> >> This sounds very much like the model layer in DDD, which ironically
> is
> >> >> exactly what I am building.
> >> >>
> >> >> However, in the middle of a data flow a function needs to reach out
> to
> >> >> a repository to make a decision - how does that fit in with the data
> >> >> flow approach?
> >> >>
> >> >>
> >> >> On 5 March 2015 at 13:39, Jonah Benton <jo...@jonah.com> wrote:
> >> >> > Hi Colin,
> >> >> >
> >> >> > Another option, other than HOFs or dynamic binding, is what might
> be
> >> >> > called
> >> >> > a data flow approach.
> >> >> >
> >> >> > At the edge of the application are the functions that explicitly
> take
> >> >> > parameterized resources to perform edge state IO. These might be
> bare
> >> >> > functions, or they might be protocol implementations that await the
> >> >> > delivery
> >> >> > of a type to perform their operation, or multimethods.
> >> >> >
> >> >> > At the center are functions that take a context map, which contains
> >> >> > all
> >> >> > relevant application state- IO resources and transient state data.
> >> >> > Those
> >> >> > functions are arranged in a data flow and capture the logic or
> state
> >> >> > transitions of the application independent of any specific IO
> >> >> > commitments.
> >> >> > They can use schema or type annotation or pre/post conditions to
> >> >> > enforce
> >> >> > invariants.
> >> >> >
> >> >> > When data flow processing arrives at a place where some edge IO
> >> >> > should
> >> >> > occur, these data flow functions act as adapters to get-in the
> >> >> > appropriate
> >> >> > resource or Record from the context map and call the edge functions
> >> >> > to
> >> >> > perform IO.
> >> >> >
> >> >> > The result is a wider, flatter system, propagating state explicitly
> >> >> > rather
> >> >> > than implicitly through the call stack.
> >> >> >
> >> >> > Jonah
> >> >> >
> >> >> >
> >> >> >
> >> >> > On Wed, Mar 4, 2015 at 12:58 PM, Colin Yates <
> colin.ya...@gmail.com>
> >> >> > wrote:
> >> >> >>
> >> >> >> Hi,
> >> >> >>
> >> >> >> I am looking for the Clojure equivalent of:
> >> >> >>
> >> >> >> class Whatever {
> >> >> >> @Transactional
> >> >> >> void doSomething(IDoSomething one, IDoSomethingElse two) {
> >> >> >> one.doSomething()
> >> >> >> two.doSomething()
> >> >> >> }
> >> >> >> }
> >> >> >>
> >> >> >> where both one and two are dependency injected with a proxy which
> >> >> >> resolves
> >> >> >> to a thread local database connection. In addition, one might
> itself
> >> >> >> have a
> >> >> >> collaborator which itself has a collaborator which needs a
> >> >> >> datasource.
> >> >> >>
> >> >> >> So far I have two protocols:
> >> >> >>
> >> >> >> (defprotocol IDoSomething
> >> >> >> (do-something [this ...])
> >> >> >>
> >> >> >> (defprotocol IDoSomethingElse
> >> >> >> (do-something [this ...])
> >> >> >>
> >> >> >> Each protocol may have a number of implementations, one of which
> is
> >> >> >> a
> >> >> >> JDBC
> >> >> >> implementation:
> >> >> >>
> >> >> >> (defrecord JdbcIDoSomething [db]
> >> >> >> (do-something [this ...] ...))
> >> >> >>
> >> >> >> The problem is that the calling code only gets provided an
> >> >> >> IDoSomething
> >> >> >> and an IDoSomethingElse and it wants to do something like:
> >> >> >>
> >> >> >> (let [one (->JdbcDoSomething db) two (->JdbcDoSomethingElse db)]
> >> >> >> (with-transaction [tx db]
> >> >> >> (do-something one)
> >> >> >> (do-something-else two)))
> >> >> >>
> >> >> >> The problem here is that the implementations of do-something and
> >> >> >> do-something-else won't have access to the local bound 'tx', they
> >> >> >> will
> >> >> >> have
> >> >> >> their own 'db'.
> >> >> >>
> >> >> >> I realise the general argument is to be explicit and pass a db as
> >> >> >> the
> >> >> >> first argument to the protocol but this isn't appropriate in this
> >> >> >> case
> >> >> >> as
> >> >> >> there are validly multiple implementations. I could abstract a
> >> >> >> 'unit-of-work' and pass that as the first argument to the
> protocols
> >> >> >> but
> >> >> >> that
> >> >> >> seems a bit painful.
> >> >> >>
> >> >> >> Also, these protocols may be used quite far away from where the
> >> >> >> database
> >> >> >> code lives and passing a parameter all the way through the call
> >> >> >> stack
> >> >> >> is
> >> >> >> painful.
> >> >> >>
> >> >> >> I am using Stuart Sierra's components if that makes any
> difference.
> >> >> >>
> >> >> >> I can't be the first person to run into this but google is
> >> >> >> surprisingly
> >> >> >> unhelpful which makes me think I have missed something
> fundamental,
> >> >> >> and
> >> >> >> that
> >> >> >> I have something upside down.
> >> >> >>
> >> >> >> What do you all do?
> >> >> >>
> >> >> >>
> >> >> >> --
> >> >> >> You received this message because you are subscribed to the Google
> >> >> >> Groups "Clojure" group.
> >> >> >> To post to this group, send email to clojure@googlegroups.com
> >> >> >> Note that posts from new members are moderated - please be patient
> >> >> >> with
> >> >> >> your first post.
> >> >> >> To unsubscribe from this group, send email to
> >> >> >> clojure+unsubscr...@googlegroups.com
> >> >> >> For more options, visit this group at
> >> >> >> http://groups.google.com/group/clojure?hl=en
> >> >> >> ---
> >> >> >> You received this message because you are subscribed to the Google
> >> >> >> Groups
> >> >> >> "Clojure" group.
> >> >> >> To unsubscribe from this group and stop receiving emails from it,
> >> >> >> send
> >> >> >> an
> >> >> >> email to clojure+unsubscr...@googlegroups.com.
> >> >> >> For more options, visit https://groups.google.com/d/optout.
> >> >> >
> >> >> >
> >> >> > --
> >> >> > You received this message because you are subscribed to the Google
> >> >> > Groups "Clojure" group.
> >> >> > To post to this group, send email to clojure@googlegroups.com
> >> >> > Note that posts from new members are moderated - please be patient
> >> >> > with
> >> >> > your
> >> >> > first post.
> >> >> > To unsubscribe from this group, send email to
> >> >> > clojure+unsubscr...@googlegroups.com
> >> >> > For more options, visit this group at
> >> >> > http://groups.google.com/group/clojure?hl=en
> >> >> > ---
> >> >> > You received this message because you are subscribed to the Google
> >> >> > Groups
> >> >> > "Clojure" group.
> >> >> > To unsubscribe from this group and stop receiving emails from it,
> >> >> > send
> >> >> > an
> >> >> > email to clojure+unsubscr...@googlegroups.com.
> >> >> > For more options, visit https://groups.google.com/d/optout.
> >> >>
> >> >> --
> >> >> You received this message because you are subscribed to the Google
> >> >> Groups "Clojure" group.
> >> >> To post to this group, send email to clojure@googlegroups.com
> >> >> Note that posts from new members are moderated - please be patient
> with
> >> >> your first post.
> >> >> To unsubscribe from this group, send email to
> >> >> clojure+unsubscr...@googlegroups.com
> >> >> For more options, visit this group at
> >> >> http://groups.google.com/group/clojure?hl=en
> >> >> ---
> >> >> You received this message because you are subscribed to the Google
> >> >> Groups
> >> >> "Clojure" group.
> >> >> To unsubscribe from this group and stop receiving emails from it,
> send
> >> >> an
> >> >> email to clojure+unsubscr...@googlegroups.com.
> >> >> For more options, visit https://groups.google.com/d/optout.
> >> >
> >> >
> >> > --
> >> > You received this message because you are subscribed to the Google
> >> > Groups "Clojure" group.
> >> > To post to this group, send email to clojure@googlegroups.com
> >> > Note that posts from new members are moderated - please be patient
> with
> >> > your
> >> > first post.
> >> > To unsubscribe from this group, send email to
> >> > clojure+unsubscr...@googlegroups.com
> >> > For more options, visit this group at
> >> > http://groups.google.com/group/clojure?hl=en
> >> > ---
> >> > You received this message because you are subscribed to the Google
> >> > Groups
> >> > "Clojure" group.
> >> > To unsubscribe from this group and stop receiving emails from it, send
> >> > an
> >> > email to clojure+unsubscr...@googlegroups.com.
> >> > For more options, visit https://groups.google.com/d/optout.
> >>
> >> --
> >> You received this message because you are subscribed to the Google
> >> Groups "Clojure" group.
> >> To post to this group, send email to clojure@googlegroups.com
> >> Note that posts from new members are moderated - please be patient with
> >> your first post.
> >> To unsubscribe from this group, send email to
> >> clojure+unsubscr...@googlegroups.com
> >> For more options, visit this group at
> >> http://groups.google.com/group/clojure?hl=en
> >> ---
> >> You received this message because you are subscribed to the Google
> Groups
> >> "Clojure" group.
> >> To unsubscribe from this group and stop receiving emails from it, send
> an
> >> email to clojure+unsubscr...@googlegroups.com.
> >> For more options, visit https://groups.google.com/d/optout.
> >
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "Clojure" group.
> > To post to this group, send email to clojure@googlegroups.com
> > Note that posts from new members are moderated - please be patient with
> your
> > first post.
> > To unsubscribe from this group, send email to
> > clojure+unsubscr...@googlegroups.com
> > For more options, visit this group at
> > http://groups.google.com/group/clojure?hl=en
> > ---
> > You received this message because you are subscribed to the Google Groups
> > "Clojure" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to clojure+unsubscr...@googlegroups.com.
> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
