Hi Terence,
... first: thanks for your encouraging and insightful comments.
The current status of this proposal:
- I've added the capability to save incoming write requests (aka
commands) to the database (brings us a bit closer to the initial idea of
CQRS replayable commands to restore the system at any given point in time);
there is some error during the build process that I still have to track down
- Manoj made some comments around maker-checker functionality in the
current production version of Fineract. I didn't reply extensively to that,
but my view is that command processing and maker-checker are two different
problem domains that should not be mixed together. I'd like to address the
maker-checker concern in a different proposal that would (maybe) help us on
multiple fronts:
- maker-checker is more related to human tasks/interactions in a
workflow... maybe (!) we can address this in a way that we can later
optionally (!) drop in a proper workflow engine...
- having maker-checker as a separate component should have the same
benefits as command processing (type safety, clear readable
execution path,
extensible/replaceable, better documentation)
- maker-checker is tightly related to our security model; an
improved implementation should help also with a long standing effort to
modularize the security of Fineract; we have a classic RBAC model baked
into Fineract that can't really be replaced without major
refactoring; even
adding roles/permissions to adjust to the deployment environment is not
really easy... let alone using some more advanced concepts like
Open Policy
Agent or OpenFGA
- again, this would be subject to a separate proposal... don't want
to overload
The first services that will be migrated to the new command processing are
free of maker-checker requirements and are NOT critical ones (like loan,
general ledger etc.). I think that way we won't create a monster pull
request and validate the two improved concepts (command processing,
maker-checker) step by step.
Cheers,
Aleks
On Mon, Mar 10, 2025 at 11:14 AM Terence Monteiro <[email protected]>
wrote:
> Hi Aleks and fellow Fineract Devs,
>
> Just wanted to bump this thread and share some thoughts..
> 1. I'm certain of the benefit such accurate typing and class usage for
> data interchange will bring. From my experience with both strongly typed
> (C++, Java, Python) and weakly typed languages (C, Perl5, PHP) the
> flexibility of the latter and not using classes becomes a challenge when
> the project scales up to around 10000+ lines of code. I think Fineract is
> at the stage of its evolution where this kind of work will bring a sea
> change in ease of ongoing Development and auto-generated Documentation.
> 2. I like the bullet point summary of changes expressed in the
> fineract-command module README from the PR. From an architecture
> standpoint, I see a far reaching impact such a change can have. I feel the
> questions raised after the initial mail by Aleks have been satisfactorily
> answered so I'd like to know whether we have progressed beyond the PR
> shared in the initial mail.
> 3. Practically having worked with developers of varying degrees of
> experience and IDE (Eclipse, Intellij-Idea, VS Code) adoption and
> utilization, this will yield, if done properly, much more accurate
> auto-generated Class names for function signatures and accurate
> implementation stubs. Circling back to the earlier point about the stage we
> are at with Fineract having all these lines of code, I feel this FSIP can
> lay the groundwork for a lot of contributions to
> a). Clean, refactor current code
> b). introduce new features
> c). Inject more quality API consuming projects like Web app and mobile
> app from Mifos, as well as the dozens of open-source and
> proprietary consumer apps
>
> Given this promise and opening of possibilities, I strongly support this
> proposal. I'd be glad to assist in any way possible working you Aleks on
> fleshing out required parts of making this happen as long as I can get some
> more native Java eyes to review (I think in algorithms preferably through
> an OOP lens, but don't always think in Java unless deep into a Java
> project).
>
> Regards,
> Terence Monteiro.
>
>
> On Fri, Jan 31, 2025 at 12:21 AM Aleksandar Vidakovic <
> [email protected]> wrote:
>
>> ... thanks Manoj for reviewing the proposal. The additional features you
>> mentioned can be of course added... as soon as I have a moment I'll have a
>> stab at this and will provide an example in a unit test that demonstrates
>> it.
>>
>> Cheers
>>
>> On Thu, Jan 30, 2025 at 7:36 PM Manoj Mohanan <[email protected]> wrote:
>>
>>>
>>> Hi Aleks,
>>>
>>> Thank you for your initiative in proposing this system enhancement. I
>>> strongly support this change, as it addresses a critical need in our
>>> current architecture.
>>>
>>> The existing synchronous command handler facilitates the maker-checker
>>> workflow. It allowed direct storage of serialized commands (including
>>> request payload JSON) in the database, enabling easy retrieval and replay
>>> during checker approval. As we transition to the new approach, Are there
>>> proposed modifications to the maker-checker workflow itself?
>>>
>>> Additionally, the existing implementation includes a permission check
>>> (authorization) prior to invoking the handler, as well as a Hook Event
>>> Processor integrated into the control flow to invoke external APIs
>>> asynchronously. To ensure continuity of these functionalities, can we
>>> include these in the proposed command executors?
>>>
>>> This will help maintain critical auditability standards while adopting
>>> the updated architecture.
>>>
>>> Regards,
>>> Manoj
>>>
>>> On 2025/01/26 09:38:46 Aleksandar Vidakovic wrote:
>>> > On Sun, Jan 26, 2025 at 6:59 AM VICTOR MANUEL ROMERO RODRIGUEZ <
>>> > [email protected]> wrote:
>>> >
>>> > > Hello,
>>> > >
>>> > > I have been reading the discussion, the FISP and the GitHub PR...
>>> this
>>> > > must receive more feedback from the community.
>>> > >
>>> >
>>> > That's why we are here... thanks again for participating.
>>> >
>>> >
>>> > >
>>> > > It causes me some noise that the FSIP is focused on the API Rest
>>> layer and
>>> > > the PR is introducing a new header that could also be used for
>>> > > Idempotency... We can understand the reasons for having it in the
>>> Rest
>>> > > layer. But what about the idempotency that is being used in the
>>> batch/event
>>> > > processing?
>>> > >
>>> >
>>> > Sorry for the noise...
>>> > I'm not really seeing the "focus" on the REST API layer... the
>>> proposal is
>>> > kept explicitly free of any assumption which web stack you are using...
>>> > this stuff could even work with Webflux.
>>> > Introducing a new header? There is not even an implementation for
>>> > idempotency in this proposal... maybe you are referring to the example
>>> in
>>> > the unit test? That stuff (everything in the "sample" package) is
>>> purely
>>> > for demonstration.
>>> > Could you maybe elaborate what problem you see concerning the batch
>>> > processing?
>>> >
>>> >
>>> > >
>>> > > I have found the implementation for sync and async commands... What
>>> about
>>> > > streaming? Also in the same package there are pipeline, executor,
>>> router ..
>>> > > What about notifications? My questions are for getting feedback from
>>> you if
>>> > > they are expected/discarded on this new proposed infrastructure.
>>> > >
>>> >
>>> > ... I would add to your list "what about non-blocking"? Well, the
>>> reason
>>> > why it's not in my proposal is, because right now we have a classic
>>> > synchronous blocking execution path in place upstream... and the first
>>> > step would be to get the whole thing gradually to a more performant
>>> > solution with clearly structured internal API. Another important
>>> thought
>>> > here is to avoid/reduce any additional learning curve; the community is
>>> > used to the classic "I call a function and get a result" paradigm of
>>> > programming; introducing more complex paradigms would need more time
>>> for
>>> > adoption. That being said: if the community decided tomorrow to go all
>>> in
>>> > for non-blocking, reactive, GRPC, [place your preference here] and
>>> > bulldozer things then I would happily participate... but my feeling is
>>> that
>>> > this is very unlikely. And before anything like this happens we would
>>> need
>>> > all these hard-wired references to JSON data structures from the
>>> business
>>> > logic services anyway.
>>> >
>>> > As for notifications: noted, good point.
>>> > And concerning feedback: I have no claim here that what I wrote in the
>>> PR
>>> > is set in stone... that's why we discuss it here (I hope more people
>>> join).
>>> > Things that I didn't have on the radar and that you and others find
>>> > indispensable can be of course added.
>>> >
>>> >
>>> > >
>>> > > I think that Apache Camel is a good tool... I don't see it as a
>>> vendor
>>> > > lock-in... well If do that (to see it as a vendor lock-in)... then
>>> what
>>> > > about spring boot itself? i.e. nowadays we have Quarkus and
>>> Micronaut...
>>> > > and I think it is more complex to move some Apache Fineract code to
>>> these
>>> > > frameworks or add plugins developed in these frameworks to Apache
>>> Fineract
>>> > > runtime.
>>> > >
>>> >
>>> > Again, the idea here is to keep this proposal as self contained as
>>> possible
>>> > to avoid having to decide on too many fundamental changes... hence,
>>> you see
>>> > no Apache Camel. That being said: I can write you an adapter in 10min
>>> that
>>> > runs the whole thing over Camel... but doesn't leak any details about
>>> Camel
>>> > being used to the rest of the code base. Don't see what the flaw would
>>> be
>>> > here to properly abstract implementation details?
>>> >
>>> > Is there a ready to use solution in Spring Boot for command processing
>>> > and/or CQRS? I think not... but let me know if I am missing something
>>> here.
>>> > What is out there is Axon which is a complete framework implemented ON
>>> TOP
>>> > OF Spring/Boot... but that thing would require us to jump on their
>>> internal
>>> > APIs and how they think things should be processed... personally I
>>> think
>>> > these guys thought about this subject a bit longer and have a way more
>>> > complete solution... but again, given the need for backwards
>>> compatibility
>>> > and the requirement to be not (too) disruptive to any other upstream
>>> > development: very unlikely that such a fundamental change would happen
>>> and
>>> > cause a major refactoring fest.
>>> >
>>> > I'm not so sure why Quarkus and Micronaut are mentioned here... yes,
>>> very
>>> > nice and capable modern frameworks... but they also have no specific
>>> notion
>>> > of command processing, but are generic frameworks like Spring/Boot...
>>> my
>>> > intention is not to change any fundamental underlying frameworks (give
>>> and
>>> > take they have similar features)... again, I'd like to keep this as
>>> small
>>> > and self contained as possible.
>>> >
>>> >
>>> > >
>>> > > Happy to read your feedback.
>>> > >
>>> > > Regards
>>> > >
>>> > > Victor
>>> > >
>>> > >
>>> > >
>>> > >
>>> > > El jue, 23 ene 2025 a las 23:40, James Dailey (<[email protected]>)
>>> > > escribió:
>>> > >
>>> > >> On Thu, Jan 23, 2025 at 9:20 AM Aleksandar Vidakovic
>>> > >> <[email protected]> wrote:
>>> > >> >
>>> > >> > ... thanks James for the input... I'll try to answer your last
>>> couple
>>> > >> of questions from my perspective (read: opinionated... take with a
>>> pinch of
>>> > >> salt):
>>> > >>
>>> > >> JD: Aleks - thank you. I always learn something from this back
>>> and
>>> > >> forth with you.
>>> > >> >
>>> > >> > too clever: the current implementation I suggest that anyone
>>> tries to
>>> > >> draw a sequence diagram that explains the flow of execution and
>>> make it fit
>>> > >> on one page vs the new proposal will most likely contain less than a
>>> > >> handful of lines. You can apply the same if you take lines of code
>>> as a
>>> > >> metric... overall the new proposal has less than 50 lines of code
>>> that are
>>> > >> relevant (I don't know the number for upstream, but I think it's
>>> safe to
>>> > >> say it's more). If we assume that we can achieve the same results
>>> with less
>>> > >> code then I think the answer is easy here
>>> > >> > maintainability: well, see above... the current solution is not
>>> > >> documented at all and I am pretty sure I am not alone when I saw "I
>>> really
>>> > >> can't explain all the steps" (doesn't mean they are not necessary);
>>> what I
>>> > >> want to say is that the existing solution would really need a lot
>>> more
>>> > >> explanation than just "CQRS", I think that would be a fair
>>> requirement.
>>> > >> Admittedly, the new proposal also has no documentation (other than
>>> the wiki
>>> > >> page and what I wrote in this message). But: I think if I did write
>>> it it
>>> > >> can fit on one page (with diagrams), this module (it's a real one)
>>> has
>>> > >> (almost) no external dependencies (other than the frameworks that
>>> we use
>>> > >> anyway), it makes no assumption about any of the business logic
>>> that might
>>> > >> or might be passing through (existing implementation fails already
>>> there...
>>> > >> see CommandWrapper and the various entity IDs that are buried
>>> there... this
>>> > >> wrapper class should not be aware of anything it transports)...
>>> which
>>> > >> brings me back to the point of less code which is I think from a
>>> > >> maintenance point of view preferrable
>>> > >>
>>> > >> JD: When I say "overly clever" that is in contrast to simplicity
>>> > >> through elegant design. A favorite quote "There are two ways of
>>> > >> constructing a software design: One way is to make it so simple that
>>> > >> there are obviously no deficiencies, and the other way is to make it
>>> > >> so complicated that there are no obvious deficiencies. The first
>>> > >> method is far more difficult." I think if you are aiming for
>>> > >> something simple enough to have obviously no (or much fewer)
>>> > >> deficiencies, that is, an improvement. But, could you write some
>>> > >> documentation about the concept? It should be simple to describe "on
>>> > >> paper", yes?
>>> > >>
>>> > >> >
>>> > >> > Apache Camel: ... disclaimer, I really like that framework and
>>> used it
>>> > >> on a ton of occasions. That being said: choosing a framework is a
>>> > >> commitment pretty much like a vendor lock-in. Depending on how you
>>> > >> integrate a framework like Camel (this will be more than a JAR file
>>> and you
>>> > >> can either hide the fact you use Camel from the rest of your app or
>>> you
>>> > >> fully expose it...) upstream means if for some reason it turns out
>>> that
>>> > >> Camel is not a good choice or the community doesn't want yet another
>>> > >> dependency then we might find ourselves in a refactoring fest to
>>> revert
>>> > >> things. If you look closely in the proposed sources you will see
>>> that first
>>> > >> of all there are Java interfaces that propose a contract on how to
>>> wire
>>> > >> things together... and there not many... which leaves a lot of room
>>> for
>>> > >> actual implementations (Camel or something else). In fact, 3 or 4
>>> years ago
>>> > >> I actually created a drop-in replacement for the upstream
>>> > >> SynchronousCommandProcessing service and ran Camel behind the
>>> scenes and
>>> > >> was actually very happey with the outcome. When I did this there
>>> were
>>> > >> basically 2 relevant functions that needed to be taken care of.
>>> Today there
>>> > >> is a lot more going on there and I am not so sure if you could just
>>> drop-in
>>> > >> Camel effortlessly with the current incarnation of the command
>>> processing
>>> > >> service.
>>> > >>
>>> > >> JD: Ok. I can buy not wanting another dependency, but only if our
>>> > >> level of effort is relatively small ongoing. Otherwise we are taking
>>> > >> on code maintenance for our "own thing" when a perfectly suited
>>> > >> solution is in the same software foundation.
>>> > >>
>>> > >> > Asking Apache Camel's community for opinion: well, can't hurt...
>>> they
>>> > >> do stuff like this literally every day, so I am pretty sure
>>> whatever we'll
>>> > >> exchange with them will be very informative. But that doesn't
>>> relieve us
>>> > >> from deciding if you want to go all in on Apache Camel it would be
>>> anyway a
>>> > >> good practice to abstract these implementation details away (aka
>>> hide to
>>> > >> the rest of Fineract that you are using Camel). If that is the case
>>> then we
>>> > >> need a contract (aka Java interface). The one that is there won't
>>> do it
>>> > >> anymore... without major rework... and that is the point. The
>>> proposal
>>> > >> intends to ensure a gradual non disruptive migration (not open heart
>>> > >> surgery)
>>> > >>
>>> > >> JD: Sure, that makes sense, you need to new Java interface... but
>>> > >> wouldn't it be better to spend a bit of time in design and
>>> validation
>>> > >> at this early stage. I think we're talking about a pretty
>>> significant
>>> > >> optimization from its location in the stack. Who should reach out?
>>> > >>
>>> > >> > whitepapers, alternatives: I think the first thing that Google or
>>> > >> ChatGPT searches will tell you is "use an existing CQRS
>>> framework"... and
>>> > >> this will most likely show you AxonIQ (a CQRS framework implemented
>>> with
>>> > >> Spring/Boot)... but that is then even more of a vendor lock in than
>>> using a
>>> > >> more generic solution like Camel... Axon will force us to use their
>>> > >> contracts (internal APIs, Java interfaces etc.), in short: refactor
>>> fest,
>>> > >> disruptive. There are other low level "solutions" (like LMAX
>>> Disruptor)
>>> > >> that are somewhat in the vicinity of this type of application, but
>>> require
>>> > >> work, to my knowledge there is nothing out there we could just
>>> magically
>>> > >> drop and use without any refactoring. Disclaimer: in one of the 3
>>> drop-in
>>> > >> implementations of the proposed command processing I am actually
>>> using LMAX
>>> > >> Disruptor... its implementation details just don't leak into the
>>> rest of
>>> > >> the system
>>> > >> > Spring Boot 3 compliance: yes (buzzword drop:
>>> "auto-configuration")
>>> > >>
>>> > >> JD: Excellent
>>> > >>
>>> > >> > cutting edge: not sure how to read this here... is this meant as a
>>> > >> requirement or as an argument against the adoption of the proposal
>>> as in
>>> > >> "too experimental"... as I've written the code I am obviously
>>> biased so I
>>> > >> leave that to the community to decide and come up with improvements
>>> and/or
>>> > >> alternatives/arguments if someone doesn't agree
>>> > >>
>>> > >> JD: Yep. The ambiguity is on purpose - cutting edge can be great in
>>> > >> getting results, or it can make you bleed.
>>> > >>
>>> > >> >
>>> > >> > Let me know if I skipped something, made an error or was not clear
>>> > >> enough.
>>> > >>
>>> > >> JD: Very clear. Now, before this code is committed, I would also
>>> > >> like to be sure we have a sensible way of documenting the progress
>>> so
>>> > >> that if we are doing a release, we make note of how much of the code
>>> > >> base is using the new methods. I also think we should discuss this
>>> in
>>> > >> context of the next release. (coming up soon).
>>> > >>
>>> > >> >
>>> > >> > On Thu, Jan 23, 2025 at 5:00 PM <[email protected]> wrote:
>>> > >> >>
>>> > >> >> Thanks for bringing this to the list. It looks to be a very low
>>> level
>>> > >> (in the stack) and therefore, highly impactful. I was there when the
>>> > >> decision was made to adopt this pattern and
>>> > >> SynchronousCommandProcessingService as a flexible improvement to the
>>> > >> existing CQRS. I remember asking some questions, but this was and
>>> is,
>>> > >> beyond my direct experience.
>>> > >> >>
>>> > >> >>
>>> > >>
>>> https://cwiki.apache.org/confluence/display/FINERACT/FSIP-5:+New+command+processing+infrastructure
>>> > >> >>
>>> > >> >> What I do know is that we should be deliberate with this
>>> process, and
>>> > >> I appreciate your write up on wiki. Definitely other architects
>>> here
>>> > >> should take a look.
>>> > >> >>
>>> > >> >> At times over the past decisions - it feels to me that we try to
>>> be
>>> > >> "too clever", and this creates a problem with maintainability. I'd
>>> like to
>>> > >> make sure we understand the alternatives as we dig into this. You
>>> raised
>>> > >> Apache Camel as an option - would it be worth it to ask someone
>>> over in
>>> > >> that project to comment on this? Is there some whitepaper or
>>> comparison
>>> > >> out there between the alternatives available? Is this consistent
>>> with
>>> > >> Spring Boot 3 ? Is this on the cutting edge?
>>> > >>
>>> > >
>>> >
>>>
>>
