Hi Taher,
thanks for bringing this up! I will try to briefly explain my views on
the raised topics.
My perspective is from nearly 20 years of intensive work with OFBiz,
both for internal and external projects. Those projects have/had a
volume reaching from about 50 to over 1000 project days and we have
active customer projects which are running for almost 20 years now,
still being developed further with the business. OFBiz often became a
crucial part of the business ecosystem of those users. It is also the
central toolbox for our services around development of web based
software solutions and system integration in our own company since 7
years now. I consider myself both a user of OFBiz and a service
provider/integrator.
After so many years of working with OFBiz if have learned that - from a
business users perspective - the functionality and integration aspects
along with the powerful datamodel, services etc. as well as flexibility,
performance and scalability is way more important than detailed
technical aspects (in the sense of “modern” or “newest technology”).
As a service provider for OFBiz, we are feeling responsible to provide a
reliable, constant and upgradeable software ecosystem which is why we
are on a more "conservative" path regarding big or long running changes
in the project. I have seen several new, ambititious contributors who
wanted to change everything, using the newest frameworks and tools
around etc., but often did not have an extensive background of real
world projects or who are not able to explain why they want to use other
tools and mechanisms and which user problems they really want to solve.
Most of these people are not active anymore or unfinished solutions are
put into the codebase which is another reason why I am really careful
when it comes to massive changes. I think the project needs good
planning, a good migration path and constant support when those changes
are decided to take place.
All in all, I do not feel such a great pain as you seem to feel but
there is definetely much room for improvement, for sure.
I will try to give my views on the different topics inline. They are
brief and need/deserve more in-depth preparation from my side which I
cannot spare at the moment.
For further discussions I recommend to split those topics in different
threads because they would become unreadable soon with more
participants. Also writing up the outcome of the discussions and the
further roadmap in the wiki might help.
Let's see how it works.
Michael Brohl
ecomify GmbH - www.ecomify.de
Am 22.12.21 um 18:23 schrieb Taher Alkhateeb:
Problems
...
_1. Lack of Code Isolation_:It is unlikely to do anything substantial
in OFBiz without touching the core repositories. Compare this with
moqui or some other web frameworks like say Ruby-On-Rails or django
and you will see that you can work independently of the development
path in these frameworks. This makes upgrades quite painfulwith
patches and manual interventions. Some in the community even
introduced a whole patching framework in the past.
We do not have this experience, at least not as much as you seem to
have. Customer projects are developed completely in plugins, using the
framework and avoiding changes there. There ARE some changes tagged as
mods but not too much to be a real pain during a vendor update. Some
mods are also contributed back to OFBiz so these mods disappear during
the next vendor import.
We are not directly using the original framework but rather use the
original as an upstream branch for our own projects. This makes it easy
to track changes and merge them into our codebase.
It would be helpful to have some specific examples on where you are
having those painful changes.
_2. All or nothing:_In OFBiz, if you want to use the system, either
take the whole thing or take nothing. What if I don't want any user
interface for example, and just want to utilize the entities and
services?
We are running projects which do not use the original user interface or
are being headless / just REST interfaces. You are not forced to use the
UI and you can switch off the web apps, do we really have a problem there?
_3. Limited Re-usability:_Because of the way current components are
designed, it's difficult to take advantage of components in other
components. Sharing artifacts is pretty much limited to entities and
services. Even then they're not very shareable because they're
designed in a non-usable way for a certain context and certain
settings and database parameters.*
Datamodel and services, the core business artifacts, are shareable and
in my view that's what's needed mostly. Can you be more specific what's
missing or designed in a way that it is not (re)usable ?
*
_4. Fragmented Actions:_**Instead of having unified XML actions, you
have fragmentation across different contexts (screens, services,
SECAs, etc). Furthermore, you cannot mix XML actions with groovy or
other artifacts except by calling into external scripts and things
like that. In comparison to moqui which is more flexible in XML
actions I find it restrictive to work with OFBiz actions and it's hard
to review, debug, or modify. I'm listing something below as an example
for comparison [1]*
I will have to investigate more how Moqui does those things. I'm not
sure what the example shows. Is it part of the controller logic?
Generally, I think that it is a strength to have these tasks separated
in an MVC pattern and not being able to mix controller logic with
business logic.
When it comes to debugging: how do you debug the shown XML code?
*
_5. Limited Routing capability:_**Routing is very minimal
(controller.xml). It cannot contain business logic and is limited
either to chaining requests or rendering screens. Sophisticated
routing logic is important for making rich systems that react to
differing circumstances.
**
See above: the controller part should NOT contain too much business
logic. If the business logic would be mixed with the controller logic,
how would you then make the business logic more shareable (see point 3
above)?
Maybe I'm missing something and some examples would help me understand
the point.
_6. Core framework difficult to change:_**OFBiz core was written
a/long/time ago, and back then you didn't have many libraries for
caching, transactions, DB pooling, Concurrency stuff, and others, so a
lot was hand crafted, and we have a lots of interconnected API that is
difficult to change, and most of these APIs depend on static util
files that just kept piling and piling over the years. Where is the
service engine implementation exactly? What about the entity engine?
What about widgets? They are all over the place! Pieces for XML
parsing, others for DOM models, and so on. The code is scattered and
fixing it means going through a maze of many classes that are
interconnected and all referencing each other. It's also hard to
identify the signature or figure out the "big picture" since you have
essentially minimal interfaces and no clear public API that you can
refactor against. Pretty much the whole thing is public API**[2].
**
That's a big field and I think we should break this into subtopics to
discuss. Can you be more specific, for me this is too broad to discuss.
What is the real problem, what would be a better solution, ...?
In general, a framework like OFBiz IS complex, it will never be easy to
read/understand. And of course, there is code which should be refactored
and better maintained (deprecation etc.). It would maybe be a good start
to identify those and do refactorings there first to reduce complexity
and pave the path for bigger refactorings?
From my experience, e.g. the caching capabilities of OFBiz are a big
strength and to change those parts, we need some really good proof of
concepts to check if other solutions are really better. I've had several
bad experiences with, for example, object relational mapping frameworks
(performance, complexity) and I'm still a fan of the entity engine.
_7. Web-apps design problem__:_ A big problem (in my opinion) in the
design is around having multiple web-apps, one for each component (or
more). This interrupts the flow of the app and makes it difficult to
come up with smooth solutions that work across the various screens and
instead we need to pass a key between the components (and face other
problems around session management). This restricts you in what you
can offer as a user experience.*
*
Do you have a solution in mind to solve this and also keep those
applications separated (see above)? Single Sign On?
_8. Problematic UI design__:_ Another/big/problem (in my opinion) is
having the UI show everything (reference implementation). I never
understood the value of showing/everything/! If I want to understand
the data model, I will just look at the data model. The purpose of the
UI is not to show everything, but to implement specific useful
use-cases. The reference implementation we have is not doing that, and
not offering something very useful, instead it is showing everything!
To improve this, we need to switch from data-model based interface
(party, order, etc ...) to use-case based interface (project
management, e-commerce, ERP, CRM, etc ...)*
*
That's indeed a big issue and part of the discussions with every new
user who wants to utilize the existing web apps. On one hand it is great
to be able to show the potential of the ecosystem, on the other hand
it's too complex, especially for new users.
IMO this is not so much a problem of the framework in general, which
does not prevent us from developing better UI and workflows. Personally,
I see OFBiz as an ERP framework which has to be customized to match the
target users's needs and not so much as an ready-to-use ERP.
But yes, this can and should be improved with additional web apps or
even ONE web app which tightly integrates all core applications into a
one-stop ERP solution which is more user-friendly than the base
applications. On the other hand, this will make separation more difficult...
_9. Limited client rendering:_Because it is difficult to refactor the
framework, it's hard to pin-point where to go exactly to refactor all
the code needed to support more client-rendering technology (react,
vue, angular, etc ...) and make it interact smoothly with the back-end
(drop-downs, complex forms, etc ...). A lot of work needs to happen
both to the /themes and to /framework to allow and provide more
flexibility (also moving things to DB). If this step is not done you
will be stuck with your UI capabilities and won't catch up (unless you
develop your own UI which makes the UI of the community redundant and
you let go of the whole widget system anyway).*
*
I guess it would help a lot if we have REST integrated into the core
framework and expose a proper REST API. For the HTML based web UI, I
have not much to change as the widget/freemarker templating gives us
much flexibility.
What do you miss exactly / in which direction do you want to do changes?
_10. Security is difficult to implement:_One of my favorite features
in moqui is having security baked in the artifacts themselves to avoid
sprinkling all screens / services with security logic. I never even
worry about writing security related stuff until way close to the end
of delivering the project. So there needs to be a whole security layer
that sits below all the artifacts such that we don't even need to
write security code in the various screens and services and around
certain entities.*
*
Can you elaborate more on how you want to achieve fine grained
permissions for screens and services if not tied to the screen/service
configurations (having in mind that they should be reusable)?
_11. Limited Integration Solutions__:_ I know we have now a REST
component, but this is not the same as being a core feature of the
framework (essential for any modern web framework). Why should it be
tightly integrated? Many reasons: security, localization, mapping to
service engine, mapping to entities, etc ...*
*
I'm currently working on the rest-api plugin and I agree that it would
be great to have it in the framework instead of being a plugin.
_12. No Specialization:_Because of the big repository, and having
everything point to everything, it's hard to specialize. Now compare
this to the approach we have in our work which is easier to specialize
in. For example, I have a component for payment gateway integration,
another one for twilio (SMS integration), and I have application
components for each solution in its own repository (HR, insurance
stuff, many others) and each is living in its own git repo yet they
work together just fine and without touching one line of code in the
framework or even core components (despite depending on them and using
pieces of them). I can jump into new versions of the framework
everyday without breaking a sweat. When I work on component X, I'm
only thinking of that and nothing else. I don't have to worry about
changing something that would crash another component or bring the
system down (happened to me many times in the past).
For our projects, we were going the other way: we have framework +
plugins in the same repository instead of handling different
repositories. I think it would be a real pain to split the project in
dozens of different repositories only to bring them together to
integrate during development or deployment again.
Can you describe how you are handling those different repositories in an
efficient way to get a better picture?
As long as your work in a component/application/plugin and don't touch
the framework code I also do not see why you should break something for
other components? Of course, there ARE applications which are more
fundamental than other (e.g. party, security) and if those are changed
they might break other functionality. I see those applications belonging
to the framework more than being independent applications.
I agree that we should try to identify false or unnecessary dependencies
though.
Solutions
Whatever solutions we come up with I think they're going to be
difficult and would require community buy-in and support. I'm breaking
this down into things to keep, update, add and remove as follows:
Keep it
- XSD signatures (so entity definitions, service definitions, XML
actions though should expand)
- Basic constructs of what is a component (with basic directory
structure)
- Data model
- Existing services (core data-model services)
- Java / groovy code that doesn't use core framework constructs or
goes too deep into the internals.
- Most used and useful APIs (Making queries, calling services, etc
...) although it should all be converted to interfaces with separate
implementations where it's lacking.
Agree to all above.
Improve It
- Break the project to smaller git repositories.
- framework
- data model
- services
- common UI
- use-case applications (ERP, CRM, e-commerce, etc ...)
- plugins
- entire framework directory needs refurbishment and big chunks
probably require a delete / rewrite
- the web architecture (get rid of web-apps)
- the tests (lots of them need to be deleted anyway)
- Routing needs to be programmatic with ability to inject code.
Add It
- Configuration override mechanism to remove the need to touch the
core. Overrides include everything including routing, component
registration, etc ...
- REST integration into the core
- Artifact based security that maps users / groups to artifacts
without the need to inject security code into every screen
- A super-set of XML actions that can be applied in all contexts and
can embed groovy code
- New, simple use-case based user interface. It needs to be designed
such that it is easy to extend to avoid the all-or-nothing UI that
currently exists.
Remove It / Deprecate It / Overwrite It
- Most of the existing UI (needs to be replaced with simpler use-case
based interface)
- Lots of the core framework code (needs major rewrites, needs
interfaces, import libraries for things like transactions, security,
concurrency, etc ...)
- Probably some of the existing plugins might need to be deprecated.
See my questions above for some of the topics. I will add more details
and my point-of-view once we've split up the topics.
The overall change would be huge and we should paint a clear picture of
what we want to achieve and which way to go in detail.
We'll have to prioritize with a good cost/effort consideration and do
implementations step-by-step with a proper migration plan. Maybe having
feature branches for a longer time to be able to put out releases if the
development gets stuck or lasts longer.
Quality Control
Investing in such a huge effort requires that we change the way we
view the project after / while work is being completed. Code should be
approached with greater care and a lot of emphasis should be put on
high quality, readability, avoiding anti-patterns, enforcing good
patterns and best-practices. Some of the daily workflows need to
improve / change, more thorough auditing and peer-reviews, and so on.
A big +1 for quality control. This will be a real challenge for the
project as it requires dedication and enough ressources for review.
Tradeoffs
It goes without saying that the above is a _lot_ of work and
commitment and requires a lot of effort. That's why I hesitated in
starting this thread, but I thought maybe showing the tradeoffs would
make it easier to weigh things out and decide on a direction to push.
Pros
- Pretty much most of the above mentioned problems will be resolved.
- The framework would be resilient and future proof.
- We will have more options and capabilities in the future, it would
be easier to incorporate new technologies as they emerge
- Development would go faster
- Technical Debt will be reduced
Cons
- Huge (maybe too huge) upfront costs in time and effort to make the
switch
- Extra effort needed to ensure a smooth upgrade path for existing
users / adopters / system integrators. A whole layer of code needs to
be written to ensure the ability to upgrade without massive rewrites.
- Risk of failure, exhaustion, inability to push through to the end
- Inconvenience to those with existing systems / solutions
Summary
Change is the only constant. Inability to adapt to the future means
not surviving. I'm not confident what is the best change possible or
at what pace, but I do know that lack of change is a death sentence to
any project. Maybe it's time to reflect and think of where we'd like
to go and this thread is sort of a brain-dump of possible ideas.
I look forward to hearing your thoughts.
Thanks, Taher, for the thoughtful write-up of your view on the project
and the fields of change you see to be tackled. I do not agree with all
of them and I have a sensible view on cost/effort, thorough planning and
the need to always have existing users/projects in mind. But I hope that
we/the project will find a way to constantly work on improvements.
Again, I would propose to split up the most important topics into
threads with accompanying documentation in the wiki to help avoid confusion.
I am curious what others have to say and if we can activate enough
interest to improve.
Cheers,
Taher Alkhateeb
[1] Example mixed XML actions and groovy from moqui
<if condition="settlementTerm?.orderPmtServiceRegisterId">
<then>
<service-call name="update#mantle.order.OrderPart"
in-map="[orderId:orderId,
orderPartSeqId:orderPartSeqId, settlementTermId:settlementTermId]"/>
<entity-find-one entity-name="moqui.service.ServiceRegister"
value-field="serviceRegister">
<field-map field-name="serviceRegisterId"
from="settlementTerm.orderPmtServiceRegisterId"/>
</entity-find-one>
<script><![CDATA[
try {
ec.service.sync().name(serviceRegister.serviceName).parameters(context).softValidate(true).disableAuthz().call()
if (ec.message.hasError()) return
} catch (Throwable t) {
ec.logger.log(300, "Error calling auto order payments
service for SettlementTerm ${settlementTermId}", t)
}
]]></script>
</then>
<else>
<service-call
name="mantle.account.PaymentServices.create#Payment" out-map="context"
in-map="context + [amount:amount,
amountUomId:amountUomId, fromPartyId:fromPartyId, toPartyId:toPartyId]"/>
</else>
</if>
[2] https://ci.apache.org/projects/ofbiz/site/trunk/javadocs/