Hi everyone, The crowdmatch mechanism code is still a work in progress, and I want to talk about what I've done so far to get some feedback. This email has become rather long! Feel free to skim it to find parts that sound interesting. This was written for my benefit as much as yours, and in the hope that I get some feedback. Anything at all, like "Well hurry up and do it already", is very welcome.
It is also written for posterity and for your amusement. :)
The new code is on a branch called split-mechanism. A giant comparison
can be viewed, but it's longer than this email(!)[1] and I'm going to
call out specific items instead. This will be a technical discussion,
but not too nutty. I will talk about how I structured the crowdmatch
mechanism code, how I've designed it to be used, how I've designed it
to be testable, and how some quirks popped up during my headlong crash
through coding it. Please reply with feedback ABOUT ANY PART OF THIS! :D
### Table of Contents:
1. A separate module for the crowdmatch mechanism
2. API for the library
2.a. Client code requirements
2.b. Usage example
2.c. Exported actions
3. Database management
3.a. Migrations
3.b. The test database
3.c. run-persist library
4. Implementation to allow property testing
5. Implementation to allow more-than-one Stripe
6. Difference between MechAction and StripeI
7. What's left to do
Appendix A: Questions
### 1. A separate module for the crowdmatch mechanism
Although there is only one foreseeable client to this code, namely the
Snowdrift website, I went ahead and structured it as a standalone
library. It is in a directory called 'crowdmatch'. I did this so it can
be tested independently. It will be a lot more stable than the website
eventually, too, which is a good decider for whether something should
be independent or not.
### 2. API for the library
The crowdmatch library puts certain constraints on client code because
it requires a database. Then, it does what any API does: provides
actions and data types.
### 2.a. Client code requirements
A client of this library must be using Postgres via Persistent.
The only way to be agnostic about databases would be to completely
reimplement a database. That's nuts. The library will just use Postgres,
and thus requires client code to use it as well.
The library also needs to do IO to talk to Stripe. Rather than rely
on monad transformers or mtl-style classes to manage these different
layers, I use plain old functions. Each API method needs to be passed
a handler for running database actions. Will this be a pain in the
ass? Probably not. It's also the easiest style to change FROM, should
that become necessary.
The library maintains its own notion of what a "Patron" is. To
interface with the website, I created ToMechPatron and FromMechPatron
type classes. These are easy to define for the User type. Thus, it's
easy to create a relationship between "website Users" and "crowdmatch
Patrons".
### 2.b. Usage example
Here's a usage example that covers these parts so far.
Given:
fetchPatron
:: (ToMechPatron usr, MonadIO io, MonadIO env)
=> SqlRunner io env
-> usr
-> env Patron
You can run:
someHandler = do
Entity uid user <- requireAuth
patron <- Crowdmatch.fetchPatron runDB uid
...
See how 'runDB' is passed in as the database runner.
### 2.c. Exported actions
There are about ten operations that will be exported, as well as a
handful of data types. I am starting with fetching, storing, and
deleting two items: payment-method tokens and pledges. The payment
tokens are what Stripe gives us in lieu of credit card info. Pledges are
what you think they are: the record of a donor pledged to a project.
Next in the API are two super important methods (prototyped but not
implemented yet): 'runCrowdmatch' and 'processPayments'.
The first looks at all pledged patrons and calculates an outstanding
donation balance. After a crowdmatch event, each pledged patron will
then "owe" a certain amount to the project. That all happens in the
database in a single transaction.
Later, at our leisure, we can run 'processPayments'. That method
inspects outstanding balances and sends payment commands via
Stripe. This is where fee limits take effect.
The website won't trigger the crowdmatch event or payment processing.
Instead I'll have two simple utilities that do that stuff. I'll just
run them manually at first.
This section was longer than I intended, but that's probably good.
It's one of the more important. Feedback on the API is highly welcome.
API to date:
https://git.snowdrift.coop/sd/snowdrift/blob/split-mechanism/crowdmatch/src/Crowdmatch.hs#L24
### 3. Database management
Since this library requires a database, it requires database
management. Like I said, it uses Persistent, and has its own 'Model'
module that works like the website's Model, where the database schema
is defined.
### 3.a. Migrations
The library exports a Persistent-generated migration action that clients
need to ensure gets run. This handles "safe" migrations. I will also be
adding manual migrations for the cases that Persistent can't handle. I
am leaning towards using the 'drift' library, but this is actually an
open question.
Does anyone have experience with manual migration libraries?
So far I've looked at drift, postgresql-simple-migrations, and
dbmigrations. I haven't decided which to use yet.
### 3.b. The test database
I had to set up a test database, since there's no point in running tests
without one. I wrote code that creates a tablespace in memory, so the
tests are relatively quick. (400 iterations of 0-100 actions in sixteen
seconds. Yes, "relatively quick" is definitely *relative*.)
### 3.c. run-persist library
Along the way, I wrote a whole 'nother library called run-persist. I
got tired of playing Monad Jenga just to run a SqlPersistT action in
IO. Right now this library is included in the project, but I plan on
splitting it out once it is good enough.
The run-persist library is pretty short:
https://git.snowdrift.coop/sd/snowdrift/blob/split-mechanism/run-persist/RunPersist.hs
### 4. Implementation to allow property testing
If you look at the API for the crowdmatch library linked above, you'll
see a thin wrapper over 'runMech' applied to values of type 'MechAction
ret'. Why this design? I wanted to be able to create a long, random list
of actions to blast at the library, so I could inspect invariants. This
should catch any weird combination of actions that could lead to edge
case behavior. I have two tests that use it: 'prop_pledgeHist' and
'prop_pledgeCapability'.
https://git.snowdrift.coop/sd/snowdrift/blob/split-mechanism/crowdmatch/test/main.hs#L157
Something I struggled with here is the type variable in 'MechAction
ret'. The 'ret' represents the return value of the action. MechAction
is a GADT, and each constructor can specify its unique return type
concretely. It took a while to figure out how to run a bunch of actions
with these heterogeneous return types. That's why right now all the
constructors all have type 'MechAction ()'. :) I know how to change that
now, which I will have to do to handle Stripe errors. But fixing this
in a satisfying way is an open question. I'll be "fixing" it using a
hack that is possible because the properties being tested are already
monadic.
In summary, the GADT is nice but creates some complications. This is
my first time using this design, so feedback is welcome.
### 5. Implementation to allow more-than-one Stripe
Going one level deeper, I parameterized over the method of calling out
to Stripe. This is primarily for testability. I don't want to actually
hit Stripe hundreds of times when running the property tests discussed
above. Once again I am using plain old functions, side stepping any
long-winded discussion about "dependency injection".
The Haskell library for Stripe actually already does this! But I wrapped
over it anyway, to ignore the dozens of Stripe commands that I don't
use, and to allow some flexibility in munging return types.
The wrapper is another GADT named StripeI. The name will be explained in
the next section.
There is a dummy Stripe runner for tests:
https://git.snowdrift.coop/sd/snowdrift/blob/split-mechanism/crowdmatch/test/main.hs#L48
Its implementation will be explained in the next section, as well.
### 6. Differences between MechAction and StripeI
Both of these GADTs are used to define an API independent of how they
are implemented. They only look different because I started out with
different goals, and didn't realize I was solving the same problem.
For a good intro to GADTs, read here:
http://www.haskellforall.com/2012/06/gadts.html
I wrote MechAction because I knew I would be creating an Arbitrary list
of actions for tests. I used a GADT just so I could have different, but
concrete, return types. Amusingly, that didn't work so well, since I
still needed to monomorphize them some how when running tests. I think I
will stick with it for another benefit, though: The GADT is like an OO
interface, describing what the module can do without actually doing it.
I wrote StripeI because I knew I wanted more than one way of running the
actions. I used the 'operational' package, which is designed for exactly
that. StripeI corresponds to the way instruction-types are built (and
named!) in operational's documentation. This also explains dummyStripe's
implementation, which follows the operational pattern.
After I wrote both of these, however, I realized they were two versions
of the same thing. Both describe the available commands without
actually running them. My abstraction over the API's internals, and my
abstraction over Stripe, could take on the exact same shape. At the very
least, they should be named similarly (FoobarAction versus FoobarI?!).
This is my second time using GADTs in anger. Super convenient, except
when they are not.
### 7. What's left to do
I am going to continue working on the mechanism until there is enough
in place to let people make an actual pledge. Roughly, that entails
the following:
- Write more tests
- Add Stripe return codes to MechAction results
- Write a StripeI runner that actually calls out to Stripe
- Choose a migrations library, and use it to add a constraint that
pledges can only be made if a user has their payment info on file
(Persistent can't handle this)
- Update the website to use the library instead of the prototypal code
- Pass -Wall -Werror everywhere
Then I will switch to coordinating the work required for launching at
SeaGL. There is work to do on the website and in operations. On the
production website, we need the banner with instructions for verifying
email addresses. Jason (JazzyEagle) has begun working on that, and could
use more help.
In operations, I need to put everything in place to use our Stripe
keys, and I need to prepare for the user database migration.
At the same time, the mechanism needs to implement the crowdmatch and
payment-processing functions! I will get to that after the project is
ready for launch, unless someone beats me to it. HINT HINT. :)
### Appendix A: Questions
This is just a list of questions extracted/inferred from the body of the
email, repeated (with a bit of context) for your convenience.
1. Any experience with manual migration libraries? drift,
postgresql-simple-migrations, dbmigrations, ...
2. Thoughts on requiring client code to pass in database- and
Stripe-runners to each API methods? I think it has a rewarding
simplicity. Currying the API methods should make them really
convenient.
3. Any good ideas on creating an Arbitrary list of heterogeneous
MechActions to blast at the library for testing? I'm going to be using a
low-sophistication method, so SOME solution exists, but I hope something
better exists. Something something HList? But HList makes my spidey
sense tingle in a bad way, and there are a lot of ways to mimic such
a thing. I was directed to consider extensible-effects or possibly
"reappropriating vinyl machinery".
4. Anybody want to help write mechanism methods? Both runCrowdmatch
and processPayments are interesting and available.
WHEW! If you're still here, wow. Thanks. It has been good to write
this description and rationalization for the library's design.
Hopefully it has been enlightening and inspiring. Good night.
-Bryan
aka chreekat
---------------
[1]: For reference, the entire list of changes (including diffs) is at:
https://git.snowdrift.coop/sd/snowdrift/compare/master...split-mechanism
signature.asc
Description: Digital signature
_______________________________________________ Dev mailing list Dev@lists.snowdrift.coop https://lists.snowdrift.coop/mailman/listinfo/dev
