Hi Jonah,
Coffee consumed, post read - thoughts emerging:
- so the 'context' is essentially a service registry as well as
run-time state?
- if 'renew-session' and 'return-renewed-token' both needed to
execute in a single transaction that was separate to
'some-thing-else', how would that be co-ordinated - would each tx
boundary mandate a distinct flow for example
- the whole thing feels very 'state-machiney' - is that the point?
Thanks again for sharing. I have to say that the 'flattening' of the
control, or rather conditional logic seems quite expensive. My mind
thinks in graphs not flattened lists, but maybe my resistance is just
the pain of a new paradigm...
At the moment I am slightly nervous as it seems to be conflating a
whole bunch for the win of a consistent way of moving state around.
Thanks for the time to respond and these questions are those of
clarity not criticism :).
On 5 March 2015 at 16:45, Jonah Benton <jo...@jonah.com> wrote:
> Hi Colin- a simple example that uses a flow operator would be
>
> (-> context
> do-something
> do-some-io
> do-a-last-thing )
>
> with
>
> (defn do-something [context]
> (let [some-data (get-in context [:path :to :data :element])
> result (pure-edge-function some-data)]
> (assoc-in context [:path :to :revised :data] result)))
>
> and
>
> (defn do-some-io [context]
> (let [some-other-data (get-in context [:path :to :data :to :save])
> database (get-in context [:path :to :database :resource])]
> (save! database some-other-data)
> context)))
>
> Then save! can be a plain function, or
>
> (defprotocol Saver
> (save! [this data]))
>
> (defrecord DatabaseSaver [conn ... ]
> Saver
> (save! ))
>
> (defrecord AtomSaver [a ... ]
> Saver
> (save! ))
>
> And context is a map of maps that contains both application state and edge
> resources that should provide or receive data.
>
> Encoding conditional steps in a data flow presents a challenge. Several
> variants for doing this are in flow operators in Clojure and there are also
> a number of third party approaches. The below is a concrete example from an
> internal library that is not open source but doesn't differ significantly
> from approaches that are.
>
> A "flow" looks like this:
>
> (defflow renew
> { :start valid-request?
> valid-request? { true valid-token? false invalid-request }
> valid-token? { true ingest-token false not-authorized }
> ingest-token renew-session
> renew-session renewed?
> renewed? { true return-renewed-token false not-authorized
> }
> return-renewed-token :end
> not-authorized :end
> invalid-request :end })
>
> It's a regular map with at least a :start key and an :end value. Symbols in
> a flow map are plain functions that take a context map.
>
> Semantically, those functions represent either actions or decisions. A
> function whose value in the flow map is itself a map with true and false
> keys is a decision. A function whose value in the map is another function is
> an action.
>
> This particular flow performs a renewal of a web-like "session" token. It
> runs in a Pedestal interceptor, so it receives a context map (with :request,
> :response and other keys) as a parameter and at the end it populates the
> :response key of the map.
>
> The way to read this is-
>
> 0. a context map is prepared.
> 1. the data flow runner looks for the :start key. the value of :start is
> valid-request?
> 2. The value of valid-request? is a map, rather than a function, so the flow
> determines valid-request? is a decision function- an ordinary function that
> takes a context and is expected to return true or false.
> 3. the flow runner executes valid-request? If it returns true, the flow
> runner proceeds to valid-token? using the same context map. If false,
> proceed to invalid-request
> 4. valid-token? is similarly a decision function, it's executed similarly
> and is also expected to return true or false. invalid-request is an action.
> It's expected to return a new context map- in this case it populates the
> :response with a 4xx :status and some other response attributes.
> 5. the flow engine keeps going between keys and values until it comes to a
> value of :end. Then the resulting context is the output of the flow.
>
> The action function in the above called "renew-session" can do IO to renew a
> user session in the database. It's copied below.
>
> (defn renew-session [context]
> (let [database (get-in context [:components :database])
> session-key (get-in context [:work :session-key])
> renewed? (db/renew-session! database session-key)]
> (if (not (nil? renewed?))
> (assoc-in context [:work :renewed-key] session-key)
> context)))
>
> It simply pulls the database connection from the context, calls into a db
> specific function, db/renew-session!, which takes a connection and a key,
> and could be a protocol function that accepts a Record that encapsulates a
> specific storage resource, or could be a multimethod.
>
> This model of a data flow with actions and decisions follows Amazon's Simple
> Workflow Service, and the clever encoding of that flow as a map is due to
> Cognitect.
>
> A few projects exist now that encode flows in data form with an engine to
> process them. There are some trade-offs- it takes practice to read them, you
> can trade off a little in performance. In the above there's a fair amount of
> boilerplate that could be abstracted away along the lines of of Prismatic's
> Graph.
>
> You get a lot of other benefits- a consistent system shape, separation of
> concerns between core logic and edge IO, consistent error handling,
> logging/auditing, etc. Flows can be reused and can themselves compose,
> because everything takes a context map. You could also generate a visual
> decision tree of the model, or a real time heat map of decisions or of
> action performance.
>
> Anyway- sorry, this isn't intended to pitch an internal library, rather to
> share another thought process around the problems you raised that points in
> the direction of modeling and then executing the data flow, rather than
> relying solely on the call stack.
>
>
>
> On Thu, Mar 5, 2015 at 9:39 AM, Colin Yates <colin.ya...@gmail.com> wrote:
>>
>> Sounds interesting - are there are instances that I can look at?
>>
>> On 5 March 2015 at 14:15, Jonah Benton <jo...@jonah.com> wrote:
>> > Yes, exactly.
>> >
>> > That's fine, a node in the "middle" of the data flow can be responsible
>> > for
>> > reaching out to the edge to synchronously get some data to feed into a
>> > decision performed by the data flow. It would just do that through the
>> > edge
>> > functions.
>> >
>> > A request for data could also be asynchronously initiated once enough
>> > context had been determined by the data flow, and a promise added to the
>> > context map for later derefing (likely with a timeout).
>> >
>> > If the request for the data is context-free, it could be pulled in at
>> > the
>> > start of the data flow.
>> >
>> > By edge, it's is not that IO can't be done in the middle of a dataflow,
>> > it's
>> > just that functions that deal with dataflow follow one pattern (taking a
>> > context map) so they can be easily composed and rearranged, while
>> > functions
>> > that deal with edge resources take those resources along with other
>> > parameters more explicitly, so the data flow can run in an insulated
>> > fashion.
>> >
>> >
>> >
>> > On Thu, Mar 5, 2015 at 8:55 AM, Colin Yates <colin.ya...@gmail.com>
>> > wrote:
>> >>
>> >> Hi Jonah,
>> >>
>> >> This sounds very much like the model layer in DDD, which ironically is
>> >> exactly what I am building.
>> >>
>> >> However, in the middle of a data flow a function needs to reach out to
>> >> a repository to make a decision - how does that fit in with the data
>> >> flow approach?
>> >>
>> >>
>> >> On 5 March 2015 at 13:39, Jonah Benton <jo...@jonah.com> wrote:
>> >> > Hi Colin,
>> >> >
>> >> > Another option, other than HOFs or dynamic binding, is what might be
>> >> > called
>> >> > a data flow approach.
>> >> >
>> >> > At the edge of the application are the functions that explicitly take
>> >> > parameterized resources to perform edge state IO. These might be bare
>> >> > functions, or they might be protocol implementations that await the
>> >> > delivery
>> >> > of a type to perform their operation, or multimethods.
>> >> >
>> >> > At the center are functions that take a context map, which contains
>> >> > all
>> >> > relevant application state- IO resources and transient state data.
>> >> > Those
>> >> > functions are arranged in a data flow and capture the logic or state
>> >> > transitions of the application independent of any specific IO
>> >> > commitments.
>> >> > They can use schema or type annotation or pre/post conditions to
>> >> > enforce
>> >> > invariants.
>> >> >
>> >> > When data flow processing arrives at a place where some edge IO
>> >> > should
>> >> > occur, these data flow functions act as adapters to get-in the
>> >> > appropriate
>> >> > resource or Record from the context map and call the edge functions
>> >> > to
>> >> > perform IO.
>> >> >
>> >> > The result is a wider, flatter system, propagating state explicitly
>> >> > rather
>> >> > than implicitly through the call stack.
>> >> >
>> >> > Jonah
>> >> >
>> >> >
>> >> >
>> >> > On Wed, Mar 4, 2015 at 12:58 PM, Colin Yates <colin.ya...@gmail.com>
>> >> > wrote:
>> >> >>
>> >> >> Hi,
>> >> >>
>> >> >> I am looking for the Clojure equivalent of:
>> >> >>
>> >> >> class Whatever {
>> >> >> @Transactional
>> >> >> void doSomething(IDoSomething one, IDoSomethingElse two) {
>> >> >> one.doSomething()
>> >> >> two.doSomething()
>> >> >> }
>> >> >> }
>> >> >>
>> >> >> where both one and two are dependency injected with a proxy which
>> >> >> resolves
>> >> >> to a thread local database connection. In addition, one might itself
>> >> >> have a
>> >> >> collaborator which itself has a collaborator which needs a
>> >> >> datasource.
>> >> >>
>> >> >> So far I have two protocols:
>> >> >>
>> >> >> (defprotocol IDoSomething
>> >> >> (do-something [this ...])
>> >> >>
>> >> >> (defprotocol IDoSomethingElse
>> >> >> (do-something [this ...])
>> >> >>
>> >> >> Each protocol may have a number of implementations, one of which is
>> >> >> a
>> >> >> JDBC
>> >> >> implementation:
>> >> >>
>> >> >> (defrecord JdbcIDoSomething [db]
>> >> >> (do-something [this ...] ...))
>> >> >>
>> >> >> The problem is that the calling code only gets provided an
>> >> >> IDoSomething
>> >> >> and an IDoSomethingElse and it wants to do something like:
>> >> >>
>> >> >> (let [one (->JdbcDoSomething db) two (->JdbcDoSomethingElse db)]
>> >> >> (with-transaction [tx db]
>> >> >> (do-something one)
>> >> >> (do-something-else two)))
>> >> >>
>> >> >> The problem here is that the implementations of do-something and
>> >> >> do-something-else won't have access to the local bound 'tx', they
>> >> >> will
>> >> >> have
>> >> >> their own 'db'.
>> >> >>
>> >> >> I realise the general argument is to be explicit and pass a db as
>> >> >> the
>> >> >> first argument to the protocol but this isn't appropriate in this
>> >> >> case
>> >> >> as
>> >> >> there are validly multiple implementations. I could abstract a
>> >> >> 'unit-of-work' and pass that as the first argument to the protocols
>> >> >> but
>> >> >> that
>> >> >> seems a bit painful.
>> >> >>
>> >> >> Also, these protocols may be used quite far away from where the
>> >> >> database
>> >> >> code lives and passing a parameter all the way through the call
>> >> >> stack
>> >> >> is
>> >> >> painful.
>> >> >>
>> >> >> I am using Stuart Sierra's components if that makes any difference.
>> >> >>
>> >> >> I can't be the first person to run into this but google is
>> >> >> surprisingly
>> >> >> unhelpful which makes me think I have missed something fundamental,
>> >> >> and
>> >> >> that
>> >> >> I have something upside down.
>> >> >>
>> >> >> What do you all do?
>> >> >>
>> >> >>
>> >> >> --
>> >> >> You received this message because you are subscribed to the Google
>> >> >> Groups "Clojure" group.
>> >> >> To post to this group, send email to clojure@googlegroups.com
>> >> >> Note that posts from new members are moderated - please be patient
>> >> >> with
>> >> >> your first post.
>> >> >> To unsubscribe from this group, send email to
>> >> >> clojure+unsubscr...@googlegroups.com
>> >> >> For more options, visit this group at
>> >> >> http://groups.google.com/group/clojure?hl=en
>> >> >> ---
>> >> >> You received this message because you are subscribed to the Google
>> >> >> Groups
>> >> >> "Clojure" group.
>> >> >> To unsubscribe from this group and stop receiving emails from it,
>> >> >> send
>> >> >> an
>> >> >> email to clojure+unsubscr...@googlegroups.com.
>> >> >> For more options, visit https://groups.google.com/d/optout.
>> >> >
>> >> >
>> >> > --
>> >> > You received this message because you are subscribed to the Google
>> >> > Groups "Clojure" group.
>> >> > To post to this group, send email to clojure@googlegroups.com
>> >> > Note that posts from new members are moderated - please be patient
>> >> > with
>> >> > your
>> >> > first post.
>> >> > To unsubscribe from this group, send email to
>> >> > clojure+unsubscr...@googlegroups.com
>> >> > For more options, visit this group at
>> >> > http://groups.google.com/group/clojure?hl=en
>> >> > ---
>> >> > You received this message because you are subscribed to the Google
>> >> > Groups
>> >> > "Clojure" group.
>> >> > To unsubscribe from this group and stop receiving emails from it,
>> >> > send
>> >> > an
>> >> > email to clojure+unsubscr...@googlegroups.com.
>> >> > For more options, visit https://groups.google.com/d/optout.
>> >>
>> >> --
>> >> You received this message because you are subscribed to the Google
>> >> Groups "Clojure" group.
>> >> To post to this group, send email to clojure@googlegroups.com
>> >> Note that posts from new members are moderated - please be patient with
>> >> your first post.
>> >> To unsubscribe from this group, send email to
>> >> clojure+unsubscr...@googlegroups.com
>> >> For more options, visit this group at
>> >> http://groups.google.com/group/clojure?hl=en
>> >> ---
>> >> You received this message because you are subscribed to the Google
>> >> Groups
>> >> "Clojure" group.
>> >> To unsubscribe from this group and stop receiving emails from it, send
>> >> an
>> >> email to clojure+unsubscr...@googlegroups.com.
>> >> For more options, visit https://groups.google.com/d/optout.
>> >
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups "Clojure" group.
>> > To post to this group, send email to clojure@googlegroups.com
>> > Note that posts from new members are moderated - please be patient with
>> > your
>> > first post.
>> > To unsubscribe from this group, send email to
>> > clojure+unsubscr...@googlegroups.com
>> > For more options, visit this group at
>> > http://groups.google.com/group/clojure?hl=en
>> > ---
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "Clojure" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an
>> > email to clojure+unsubscr...@googlegroups.com.
>> > For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clojure@googlegroups.com
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+unsubscr...@googlegroups.com
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to clojure+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with your
> first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
