Hi Michael,
Thank you so much for spending your time reading and sifting through
everything, I really appreciate and value your feedback.
I need to highlight that we deployed mixed and somewhat crazy solutions
including multiple mobile apps, live video / chat streaming systems and
other projects that are booming in Kuwait and exploding both in user
bases and technical requirements which is why we cannot operate some of
these projects the way we used to in the past.
Anyway I think I'm not going to elaborate too much as we wait for more
opinions from the community to see if this is worth pushing. However, I
want to reply to a few points you raised:
- There is a huge difference between _few_ modifications, and ZERO
modifications to the framework or core components. This way you don't
even need to freeze a version of OFBiz to develop against at all!
- Breaking to multiple git repositories means having something like
./gradlew gitPullAll to update and you're done. Nothing breaks, no
surprises. You have to try it to believe it :) I have client projects
that have 15 git repos and I can upgrade with my eyes closed. Also there
is always entanglement that you don't feel or care for until you
actually cut the repo into pieces and then realize we have too much
entanglement. Having many small git repositories is in fact the most
enjoyable part of how we're writing software now and I can assign
different modules to different teams and they work independent of each
other. It's just a totally different ballgame.
- All or nothing in terms of code means you have to onboard a lot of
code that you don't need. I'm not saying you cannot it turn it off, I'm
saying it should be opt-in, not opt-out. All this code is potential
maintenance, bugs, security, and so on that you shouldn't carry on if
you don't want it. This makes things the messiest in terms of seed /
demo data but also in other entanglements.
- Yes entities and services are re-usable but you're limited to only
that. Powerful re-usability is applied to everything (configurations,
routers, widgets, scripts, etc ...). What is mostly re-usable in OFBiz
is entities and services, the others are mostly "usable" not "reusable".
It would take too long to explain and provide examples so I'll delay
this part.
- It's so awesome to use groovy exactly where you need it, without
having to jump into yet another file to only write 2 lines of code. I
have a service that needs 5-6 lines of code, why do I have to point to
an external script. Put the implementation RIGHT THERE next to the
definition of your service for example, the same is true everywhere you
need XML actions. So scripting in lacking in XML actions, and also they
are inconsistent (works differently in different places).
- Mixing XML with groovy is just part of the flexibility and having your
code where you need it. Even debugging XML code improves by putting
groovy snippets above / below to investigate the context. Nobody would
want to do that by opening a new file just to print something from the
context for example.
- Services are pure business logic, controller is routing logic. Routing
logic does not mean NO logic, it means logic that is focused on
coordination and decisions on where to go and what to do and putting
that logic in services is actually mixing layers and making the code
harder to understand / debug. I cannot show you how severely limited
routing logic is in OFBiz is without showing you 5 / 6 examples of
powerful things we've done that we simply can not do in OFBiz.
- We should IMHO by default have ONLY one web-app with the ability to
add more in an "opt-in" fashion. However, this means we need to be able
to break the screen definitions and routes, routing needs a major change
to achieve this objective. The controller.xml would simply not cut it.
Again I need examples to really highlight what I mean and I might do it
later.
- It is not enough to integrate REST to be able to have dynamic
libraries integrated with OFBiz if you want that to work with your
widgets and other constructs. You also need multiple rendering modes
that allow you to select from a variety of different technologies /
options with different macros and different theming constructs. Again I
cannot say more without examples to illustrate clearly.
- Finally, many of the changes that I'm proposing require substantial
changes to the framework and redesigning it. Well we cannot do that
without first refactoring it, which means we have to get rid of / update
lots of code. which means we need to start using libraries to control
the large size of that thing. It's a chain, one thing pulling another
thing, which makes it inevitable that a major shakeup of the core
framework is necessary. I'm not proposing to change it just to change
it. I'm proposing changing it because without that many of the things
I'm proposing simply won't happen.
I can understand why you might not agree with some of the points. In a
way it's kind of hard to imagine a different way of doing things when
we're used to working a certain way, but once we get exposed we realize
that OH .. I've been torturing myself :). However, I do so your
very-good point on being conservative because there is substantial risk
in doing everything that I suggest.
Anyway opening this to the rest of the community and thank you in
advance for sharing your thoughts.
Cheers,
Taher Alkhateeb
On 12/27/21 17:20, Michael Brohl wrote:
Hi Taher,
thanks for bringing this up! I will try to briefly explain my views on
the raised topics.
My perspective is from nearly 20 years of intensive work with OFBiz,
both for internal and external projects. Those projects have/had a
volume reaching from about 50 to over 1000 project days and we have
active customer projects which are running for almost 20 years now,
still being developed further with the business. OFBiz often became a
crucial part of the business ecosystem of those users. It is also the
central toolbox for our services around development of web based
software solutions and system integration in our own company since 7
years now. I consider myself both a user of OFBiz and a service
provider/integrator.
After so many years of working with OFBiz if have learned that - from
a business users perspective - the functionality and integration
aspects along with the powerful datamodel, services etc. as well as
flexibility, performance and scalability is way more important than
detailed technical aspects (in the sense of “modern” or “newest
technology”).
As a service provider for OFBiz, we are feeling responsible to provide
a reliable, constant and upgradeable software ecosystem which is why
we are on a more "conservative" path regarding big or long running
changes in the project. I have seen several new, ambititious
contributors who wanted to change everything, using the newest
frameworks and tools around etc., but often did not have an extensive
background of real world projects or who are not able to explain why
they want to use other tools and mechanisms and which user problems
they really want to solve.
Most of these people are not active anymore or unfinished solutions
are put into the codebase which is another reason why I am really
careful when it comes to massive changes. I think the project needs
good planning, a good migration path and constant support when those
changes are decided to take place.
All in all, I do not feel such a great pain as you seem to feel but
there is definetely much room for improvement, for sure.
I will try to give my views on the different topics inline. They are
brief and need/deserve more in-depth preparation from my side which I
cannot spare at the moment.
For further discussions I recommend to split those topics in different
threads because they would become unreadable soon with more
participants. Also writing up the outcome of the discussions and the
further roadmap in the wiki might help.
Let's see how it works.
Michael Brohl
ecomify GmbH - www.ecomify.de
Am 22.12.21 um 18:23 schrieb Taher Alkhateeb:
Problems
...
_1. Lack of Code Isolation_:It is unlikely to do anything substantial
in OFBiz without touching the core repositories. Compare this with
moqui or some other web frameworks like say Ruby-On-Rails or django
and you will see that you can work independently of the development
path in these frameworks. This makes upgrades quite painfulwith
patches and manual interventions. Some in the community even
introduced a whole patching framework in the past.
We do not have this experience, at least not as much as you seem to
have. Customer projects are developed completely in plugins, using the
framework and avoiding changes there. There ARE some changes tagged as
mods but not too much to be a real pain during a vendor update. Some
mods are also contributed back to OFBiz so these mods disappear during
the next vendor import.
We are not directly using the original framework but rather use the
original as an upstream branch for our own projects. This makes it
easy to track changes and merge them into our codebase.
It would be helpful to have some specific examples on where you are
having those painful changes.
_2. All or nothing:_In OFBiz, if you want to use the system, either
take the whole thing or take nothing. What if I don't want any user
interface for example, and just want to utilize the entities and
services?
We are running projects which do not use the original user interface
or are being headless / just REST interfaces. You are not forced to
use the UI and you can switch off the web apps, do we really have a
problem there?
_3. Limited Re-usability:_Because of the way current components are
designed, it's difficult to take advantage of components in other
components. Sharing artifacts is pretty much limited to entities and
services. Even then they're not very shareable because they're
designed in a non-usable way for a certain context and certain
settings and database parameters.*
Datamodel and services, the core business artifacts, are shareable and
in my view that's what's needed mostly. Can you be more specific
what's missing or designed in a way that it is not (re)usable ?
*
_4. Fragmented Actions:_**Instead of having unified XML actions, you
have fragmentation across different contexts (screens, services,
SECAs, etc). Furthermore, you cannot mix XML actions with groovy or
other artifacts except by calling into external scripts and things
like that. In comparison to moqui which is more flexible in XML
actions I find it restrictive to work with OFBiz actions and it's
hard to review, debug, or modify. I'm listing something below as an
example for comparison [1]*
I will have to investigate more how Moqui does those things. I'm not
sure what the example shows. Is it part of the controller logic?
Generally, I think that it is a strength to have these tasks separated
in an MVC pattern and not being able to mix controller logic with
business logic.
When it comes to debugging: how do you debug the shown XML code?
*
_5. Limited Routing capability:_**Routing is very minimal
(controller.xml). It cannot contain business logic and is limited
either to chaining requests or rendering screens. Sophisticated
routing logic is important for making rich systems that react to
differing circumstances.
**
See above: the controller part should NOT contain too much business
logic. If the business logic would be mixed with the controller logic,
how would you then make the business logic more shareable (see point 3
above)?
Maybe I'm missing something and some examples would help me understand
the point.
_6. Core framework difficult to change:_**OFBiz core was written
a/long/time ago, and back then you didn't have many libraries for
caching, transactions, DB pooling, Concurrency stuff, and others, so
a lot was hand crafted, and we have a lots of interconnected API that
is difficult to change, and most of these APIs depend on static util
files that just kept piling and piling over the years. Where is the
service engine implementation exactly? What about the entity engine?
What about widgets? They are all over the place! Pieces for XML
parsing, others for DOM models, and so on. The code is scattered and
fixing it means going through a maze of many classes that are
interconnected and all referencing each other. It's also hard to
identify the signature or figure out the "big picture" since you have
essentially minimal interfaces and no clear public API that you can
refactor against. Pretty much the whole thing is public API**[2].
**
That's a big field and I think we should break this into subtopics to
discuss. Can you be more specific, for me this is too broad to
discuss. What is the real problem, what would be a better solution, ...?
In general, a framework like OFBiz IS complex, it will never be easy
to read/understand. And of course, there is code which should be
refactored and better maintained (deprecation etc.). It would maybe be
a good start to identify those and do refactorings there first to
reduce complexity and pave the path for bigger refactorings?
From my experience, e.g. the caching capabilities of OFBiz are a big
strength and to change those parts, we need some really good proof of
concepts to check if other solutions are really better. I've had
several bad experiences with, for example, object relational mapping
frameworks (performance, complexity) and I'm still a fan of the entity
engine.
_7. Web-apps design problem__:_ A big problem (in my opinion) in the
design is around having multiple web-apps, one for each component (or
more). This interrupts the flow of the app and makes it difficult to
come up with smooth solutions that work across the various screens
and instead we need to pass a key between the components (and face
other problems around session management). This restricts you in what
you can offer as a user experience.*
*
Do you have a solution in mind to solve this and also keep those
applications separated (see above)? Single Sign On?
_8. Problematic UI design__:_ Another/big/problem (in my opinion) is
having the UI show everything (reference implementation). I never
understood the value of showing/everything/! If I want to understand
the data model, I will just look at the data model. The purpose of
the UI is not to show everything, but to implement specific useful
use-cases. The reference implementation we have is not doing that,
and not offering something very useful, instead it is showing
everything! To improve this, we need to switch from data-model based
interface (party, order, etc ...) to use-case based interface
(project management, e-commerce, ERP, CRM, etc ...)*
*
That's indeed a big issue and part of the discussions with every new
user who wants to utilize the existing web apps. On one hand it is
great to be able to show the potential of the ecosystem, on the other
hand it's too complex, especially for new users.
IMO this is not so much a problem of the framework in general, which
does not prevent us from developing better UI and workflows.
Personally, I see OFBiz as an ERP framework which has to be customized
to match the target users's needs and not so much as an ready-to-use ERP.
But yes, this can and should be improved with additional web apps or
even ONE web app which tightly integrates all core applications into a
one-stop ERP solution which is more user-friendly than the base
applications. On the other hand, this will make separation more
difficult...
_9. Limited client rendering:_Because it is difficult to refactor the
framework, it's hard to pin-point where to go exactly to refactor all
the code needed to support more client-rendering technology (react,
vue, angular, etc ...) and make it interact smoothly with the
back-end (drop-downs, complex forms, etc ...). A lot of work needs to
happen both to the /themes and to /framework to allow and provide
more flexibility (also moving things to DB). If this step is not done
you will be stuck with your UI capabilities and won't catch up
(unless you develop your own UI which makes the UI of the community
redundant and you let go of the whole widget system anyway).*
*
I guess it would help a lot if we have REST integrated into the core
framework and expose a proper REST API. For the HTML based web UI, I
have not much to change as the widget/freemarker templating gives us
much flexibility.
What do you miss exactly / in which direction do you want to do changes?
_10. Security is difficult to implement:_One of my favorite features
in moqui is having security baked in the artifacts themselves to
avoid sprinkling all screens / services with security logic. I never
even worry about writing security related stuff until way close to
the end of delivering the project. So there needs to be a whole
security layer that sits below all the artifacts such that we don't
even need to write security code in the various screens and services
and around certain entities.*
*
Can you elaborate more on how you want to achieve fine grained
permissions for screens and services if not tied to the screen/service
configurations (having in mind that they should be reusable)?
_11. Limited Integration Solutions__:_ I know we have now a REST
component, but this is not the same as being a core feature of the
framework (essential for any modern web framework). Why should it be
tightly integrated? Many reasons: security, localization, mapping to
service engine, mapping to entities, etc ...*
*
I'm currently working on the rest-api plugin and I agree that it would
be great to have it in the framework instead of being a plugin.
_12. No Specialization:_Because of the big repository, and having
everything point to everything, it's hard to specialize. Now compare
this to the approach we have in our work which is easier to
specialize in. For example, I have a component for payment gateway
integration, another one for twilio (SMS integration), and I have
application components for each solution in its own repository (HR,
insurance stuff, many others) and each is living in its own git repo
yet they work together just fine and without touching one line of
code in the framework or even core components (despite depending on
them and using pieces of them). I can jump into new versions of the
framework everyday without breaking a sweat. When I work on component
X, I'm only thinking of that and nothing else. I don't have to worry
about changing something that would crash another component or bring
the system down (happened to me many times in the past).
For our projects, we were going the other way: we have framework +
plugins in the same repository instead of handling different
repositories. I think it would be a real pain to split the project in
dozens of different repositories only to bring them together to
integrate during development or deployment again.
Can you describe how you are handling those different repositories in
an efficient way to get a better picture?
As long as your work in a component/application/plugin and don't touch
the framework code I also do not see why you should break something
for other components? Of course, there ARE applications which are more
fundamental than other (e.g. party, security) and if those are changed
they might break other functionality. I see those applications
belonging to the framework more than being independent applications.
I agree that we should try to identify false or unnecessary
dependencies though.
Solutions
Whatever solutions we come up with I think they're going to be
difficult and would require community buy-in and support. I'm
breaking this down into things to keep, update, add and remove as
follows:
Keep it
- XSD signatures (so entity definitions, service definitions, XML
actions though should expand)
- Basic constructs of what is a component (with basic directory
structure)
- Data model
- Existing services (core data-model services)
- Java / groovy code that doesn't use core framework constructs or
goes too deep into the internals.
- Most used and useful APIs (Making queries, calling services, etc
...) although it should all be converted to interfaces with separate
implementations where it's lacking.
Agree to all above.
Improve It
- Break the project to smaller git repositories.
- framework
- data model
- services
- common UI
- use-case applications (ERP, CRM, e-commerce, etc ...)
- plugins
- entire framework directory needs refurbishment and big chunks
probably require a delete / rewrite
- the web architecture (get rid of web-apps)
- the tests (lots of them need to be deleted anyway)
- Routing needs to be programmatic with ability to inject code.
Add It
- Configuration override mechanism to remove the need to touch the
core. Overrides include everything including routing, component
registration, etc ...
- REST integration into the core
- Artifact based security that maps users / groups to artifacts
without the need to inject security code into every screen
- A super-set of XML actions that can be applied in all contexts and
can embed groovy code
- New, simple use-case based user interface. It needs to be designed
such that it is easy to extend to avoid the all-or-nothing UI that
currently exists.
Remove It / Deprecate It / Overwrite It
- Most of the existing UI (needs to be replaced with simpler use-case
based interface)
- Lots of the core framework code (needs major rewrites, needs
interfaces, import libraries for things like transactions, security,
concurrency, etc ...)
- Probably some of the existing plugins might need to be deprecated.
See my questions above for some of the topics. I will add more details
and my point-of-view once we've split up the topics.
The overall change would be huge and we should paint a clear picture
of what we want to achieve and which way to go in detail.
We'll have to prioritize with a good cost/effort consideration and do
implementations step-by-step with a proper migration plan. Maybe
having feature branches for a longer time to be able to put out
releases if the development gets stuck or lasts longer.
Quality Control
Investing in such a huge effort requires that we change the way we
view the project after / while work is being completed. Code should
be approached with greater care and a lot of emphasis should be put
on high quality, readability, avoiding anti-patterns, enforcing good
patterns and best-practices. Some of the daily workflows need to
improve / change, more thorough auditing and peer-reviews, and so on.
A big +1 for quality control. This will be a real challenge for the
project as it requires dedication and enough ressources for review.
Tradeoffs
It goes without saying that the above is a _lot_ of work and
commitment and requires a lot of effort. That's why I hesitated in
starting this thread, but I thought maybe showing the tradeoffs would
make it easier to weigh things out and decide on a direction to push.
Pros
- Pretty much most of the above mentioned problems will be resolved.
- The framework would be resilient and future proof.
- We will have more options and capabilities in the future, it would
be easier to incorporate new technologies as they emerge
- Development would go faster
- Technical Debt will be reduced
Cons
- Huge (maybe too huge) upfront costs in time and effort to make the
switch
- Extra effort needed to ensure a smooth upgrade path for existing
users / adopters / system integrators. A whole layer of code needs to
be written to ensure the ability to upgrade without massive rewrites.
- Risk of failure, exhaustion, inability to push through to the end
- Inconvenience to those with existing systems / solutions
Summary
Change is the only constant. Inability to adapt to the future means
not surviving. I'm not confident what is the best change possible or
at what pace, but I do know that lack of change is a death sentence
to any project. Maybe it's time to reflect and think of where we'd
like to go and this thread is sort of a brain-dump of possible ideas.
I look forward to hearing your thoughts.
Thanks, Taher, for the thoughtful write-up of your view on the project
and the fields of change you see to be tackled. I do not agree with
all of them and I have a sensible view on cost/effort, thorough
planning and the need to always have existing users/projects in mind.
But I hope that we/the project will find a way to constantly work on
improvements.
Again, I would propose to split up the most important topics into
threads with accompanying documentation in the wiki to help avoid
confusion.
I am curious what others have to say and if we can activate enough
interest to improve.
Cheers,
Taher Alkhateeb
[1] Example mixed XML actions and groovy from moqui
<if condition="settlementTerm?.orderPmtServiceRegisterId">
<then>
<service-call name="update#mantle.order.OrderPart"
in-map="[orderId:orderId,
orderPartSeqId:orderPartSeqId, settlementTermId:settlementTermId]"/>
<entity-find-one entity-name="moqui.service.ServiceRegister"
value-field="serviceRegister">
<field-map field-name="serviceRegisterId"
from="settlementTerm.orderPmtServiceRegisterId"/>
</entity-find-one>
<script><![CDATA[
try {
ec.service.sync().name(serviceRegister.serviceName).parameters(context).softValidate(true).disableAuthz().call()
if (ec.message.hasError()) return
} catch (Throwable t) {
ec.logger.log(300, "Error calling auto order payments
service for SettlementTerm ${settlementTermId}", t)
}
]]></script>
</then>
<else>
<service-call
name="mantle.account.PaymentServices.create#Payment" out-map="context"
in-map="context + [amount:amount,
amountUomId:amountUomId, fromPartyId:fromPartyId,
toPartyId:toPartyId]"/>
</else>
</if>
[2] https://ci.apache.org/projects/ofbiz/site/trunk/javadocs/
