Dear Taher/Michael,
There are two schools of thoughts when it comes to upgrades.
1. Oneside, Developers who have put so much of an effort on the product
will be reluctant to change (until we see logical,visible and/or immediate
benifits).
2. Another side, Who wants to be on the cutting edge. to take advanced
features.
I have been part of some large scale migration. My experience teaches me
that none of the above are wrong and at the same time none of the above are
correct too. Might look a bit confusing but it is the reality. Before we go
to success aspects of migration, let me first put the failures i have
scenes with these approaches
1. Projects that provided services for a long time become relevant in a
short span of time due advanced and useful new competitors.
2. Projects that went over ambitious with cutting edge technology,
messed up the whole product due to poor execution.
Having said that, no software can stay away from migration or upgrades
forever. It has to happen today/tomorrow or later. What is in our hands is
to make it planned one or forced one. One common aspect that I have seen in
successful migration is that most of them are not done in a day or two. The
plan and execution happens as part of everyday work.
We shall start with small but effective steps first. May be a dedicated
page (if not available already) and collect the wishlist from developers
and users of ofbiz.
Thoughts ?!
Regards,
Sathish Kumar Thiyagarajan
On Mon, Dec 27, 2021 at 7:51 PM Michael Brohl <michael.br...@ecomify.de>
wrote:
> Hi Taher,
>
> thanks for bringing this up! I will try to briefly explain my views on
> the raised topics.
>
> My perspective is from nearly 20 years of intensive work with OFBiz,
> both for internal and external projects. Those projects have/had a
> volume reaching from about 50 to over 1000 project days and we have
> active customer projects which are running for almost 20 years now,
> still being developed further with the business. OFBiz often became a
> crucial part of the business ecosystem of those users. It is also the
> central toolbox for our services around development of web based
> software solutions and system integration in our own company since 7
> years now. I consider myself both a user of OFBiz and a service
> provider/integrator.
>
> After so many years of working with OFBiz if have learned that - from a
> business users perspective - the functionality and integration aspects
> along with the powerful datamodel, services etc. as well as flexibility,
> performance and scalability is way more important than detailed
> technical aspects (in the sense of “modern” or “newest technology”).
>
> As a service provider for OFBiz, we are feeling responsible to provide a
> reliable, constant and upgradeable software ecosystem which is why we
> are on a more "conservative" path regarding big or long running changes
> in the project. I have seen several new, ambititious contributors who
> wanted to change everything, using the newest frameworks and tools
> around etc., but often did not have an extensive background of real
> world projects or who are not able to explain why they want to use other
> tools and mechanisms and which user problems they really want to solve.
>
> Most of these people are not active anymore or unfinished solutions are
> put into the codebase which is another reason why I am really careful
> when it comes to massive changes. I think the project needs good
> planning, a good migration path and constant support when those changes
> are decided to take place.
>
> All in all, I do not feel such a great pain as you seem to feel but
> there is definetely much room for improvement, for sure.
>
> I will try to give my views on the different topics inline. They are
> brief and need/deserve more in-depth preparation from my side which I
> cannot spare at the moment.
>
> For further discussions I recommend to split those topics in different
> threads because they would become unreadable soon with more
> participants. Also writing up the outcome of the discussions and the
> further roadmap in the wiki might help.
>
> Let's see how it works.
>
> Michael Brohl
>
> ecomify GmbH - www.ecomify.de
>
>
> Am 22.12.21 um 18:23 schrieb Taher Alkhateeb:
> > Problems
> ...
> > _1. Lack of Code Isolation_:It is unlikely to do anything substantial
> > in OFBiz without touching the core repositories. Compare this with
> > moqui or some other web frameworks like say Ruby-On-Rails or django
> > and you will see that you can work independently of the development
> > path in these frameworks. This makes upgrades quite painfulwith
> > patches and manual interventions. Some in the community even
> > introduced a whole patching framework in the past.
>
> We do not have this experience, at least not as much as you seem to
> have. Customer projects are developed completely in plugins, using the
> framework and avoiding changes there. There ARE some changes tagged as
> mods but not too much to be a real pain during a vendor update. Some
> mods are also contributed back to OFBiz so these mods disappear during
> the next vendor import.
>
> We are not directly using the original framework but rather use the
> original as an upstream branch for our own projects. This makes it easy
> to track changes and merge them into our codebase.
>
> It would be helpful to have some specific examples on where you are
> having those painful changes.
>
> >
> > _2. All or nothing:_In OFBiz, if you want to use the system, either
> > take the whole thing or take nothing. What if I don't want any user
> > interface for example, and just want to utilize the entities and
> > services?
>
> We are running projects which do not use the original user interface or
> are being headless / just REST interfaces. You are not forced to use the
> UI and you can switch off the web apps, do we really have a problem there?
>
> >
> > _3. Limited Re-usability:_Because of the way current components are
> > designed, it's difficult to take advantage of components in other
> > components. Sharing artifacts is pretty much limited to entities and
> > services. Even then they're not very shareable because they're
> > designed in a non-usable way for a certain context and certain
> > settings and database parameters.*
>
> Datamodel and services, the core business artifacts, are shareable and
> in my view that's what's needed mostly. Can you be more specific what's
> missing or designed in a way that it is not (re)usable ?
>
>
> > *
> >
> > _4. Fragmented Actions:_**Instead of having unified XML actions, you
> > have fragmentation across different contexts (screens, services,
> > SECAs, etc). Furthermore, you cannot mix XML actions with groovy or
> > other artifacts except by calling into external scripts and things
> > like that. In comparison to moqui which is more flexible in XML
> > actions I find it restrictive to work with OFBiz actions and it's hard
> > to review, debug, or modify. I'm listing something below as an example
> > for comparison [1]*
>
> I will have to investigate more how Moqui does those things. I'm not
> sure what the example shows. Is it part of the controller logic?
>
> Generally, I think that it is a strength to have these tasks separated
> in an MVC pattern and not being able to mix controller logic with
> business logic.
>
> When it comes to debugging: how do you debug the shown XML code?
>
>
> > *
> >
> > _5. Limited Routing capability:_**Routing is very minimal
> > (controller.xml). It cannot contain business logic and is limited
> > either to chaining requests or rendering screens. Sophisticated
> > routing logic is important for making rich systems that react to
> > differing circumstances.
> > **
>
> See above: the controller part should NOT contain too much business
> logic. If the business logic would be mixed with the controller logic,
> how would you then make the business logic more shareable (see point 3
> above)?
>
> Maybe I'm missing something and some examples would help me understand
> the point.
>
>
> >
> > _6. Core framework difficult to change:_**OFBiz core was written
> > a/long/time ago, and back then you didn't have many libraries for
> > caching, transactions, DB pooling, Concurrency stuff, and others, so a
> > lot was hand crafted, and we have a lots of interconnected API that is
> > difficult to change, and most of these APIs depend on static util
> > files that just kept piling and piling over the years. Where is the
> > service engine implementation exactly? What about the entity engine?
> > What about widgets? They are all over the place! Pieces for XML
> > parsing, others for DOM models, and so on. The code is scattered and
> > fixing it means going through a maze of many classes that are
> > interconnected and all referencing each other. It's also hard to
> > identify the signature or figure out the "big picture" since you have
> > essentially minimal interfaces and no clear public API that you can
> > refactor against. Pretty much the whole thing is public API**[2].
> > **
>
> That's a big field and I think we should break this into subtopics to
> discuss. Can you be more specific, for me this is too broad to discuss.
> What is the real problem, what would be a better solution, ...?
>
> In general, a framework like OFBiz IS complex, it will never be easy to
> read/understand. And of course, there is code which should be refactored
> and better maintained (deprecation etc.). It would maybe be a good start
> to identify those and do refactorings there first to reduce complexity
> and pave the path for bigger refactorings?
>
> From my experience, e.g. the caching capabilities of OFBiz are a big
> strength and to change those parts, we need some really good proof of
> concepts to check if other solutions are really better. I've had several
> bad experiences with, for example, object relational mapping frameworks
> (performance, complexity) and I'm still a fan of the entity engine.
>
>
> >
> > _7. Web-apps design problem__:_ A big problem (in my opinion) in the
> > design is around having multiple web-apps, one for each component (or
> > more). This interrupts the flow of the app and makes it difficult to
> > come up with smooth solutions that work across the various screens and
> > instead we need to pass a key between the components (and face other
> > problems around session management). This restricts you in what you
> > can offer as a user experience.*
> > *
>
> Do you have a solution in mind to solve this and also keep those
> applications separated (see above)? Single Sign On?
>
>
> >
> > _8. Problematic UI design__:_ Another/big/problem (in my opinion) is
> > having the UI show everything (reference implementation). I never
> > understood the value of showing/everything/! If I want to understand
> > the data model, I will just look at the data model. The purpose of the
> > UI is not to show everything, but to implement specific useful
> > use-cases. The reference implementation we have is not doing that, and
> > not offering something very useful, instead it is showing everything!
> > To improve this, we need to switch from data-model based interface
> > (party, order, etc ...) to use-case based interface (project
> > management, e-commerce, ERP, CRM, etc ...)*
> > *
>
> That's indeed a big issue and part of the discussions with every new
> user who wants to utilize the existing web apps. On one hand it is great
> to be able to show the potential of the ecosystem, on the other hand
> it's too complex, especially for new users.
>
> IMO this is not so much a problem of the framework in general, which
> does not prevent us from developing better UI and workflows. Personally,
> I see OFBiz as an ERP framework which has to be customized to match the
> target users's needs and not so much as an ready-to-use ERP.
>
> But yes, this can and should be improved with additional web apps or
> even ONE web app which tightly integrates all core applications into a
> one-stop ERP solution which is more user-friendly than the base
> applications. On the other hand, this will make separation more
> difficult...
>
>
> >
> > _9. Limited client rendering:_Because it is difficult to refactor the
> > framework, it's hard to pin-point where to go exactly to refactor all
> > the code needed to support more client-rendering technology (react,
> > vue, angular, etc ...) and make it interact smoothly with the back-end
> > (drop-downs, complex forms, etc ...). A lot of work needs to happen
> > both to the /themes and to /framework to allow and provide more
> > flexibility (also moving things to DB). If this step is not done you
> > will be stuck with your UI capabilities and won't catch up (unless you
> > develop your own UI which makes the UI of the community redundant and
> > you let go of the whole widget system anyway).*
> > *
>
> I guess it would help a lot if we have REST integrated into the core
> framework and expose a proper REST API. For the HTML based web UI, I
> have not much to change as the widget/freemarker templating gives us
> much flexibility.
>
> What do you miss exactly / in which direction do you want to do changes?
>
>
> >
> > _10. Security is difficult to implement:_One of my favorite features
> > in moqui is having security baked in the artifacts themselves to avoid
> > sprinkling all screens / services with security logic. I never even
> > worry about writing security related stuff until way close to the end
> > of delivering the project. So there needs to be a whole security layer
> > that sits below all the artifacts such that we don't even need to
> > write security code in the various screens and services and around
> > certain entities.*
> > *
>
> Can you elaborate more on how you want to achieve fine grained
> permissions for screens and services if not tied to the screen/service
> configurations (having in mind that they should be reusable)?
>
>
> >
> > _11. Limited Integration Solutions__:_ I know we have now a REST
> > component, but this is not the same as being a core feature of the
> > framework (essential for any modern web framework). Why should it be
> > tightly integrated? Many reasons: security, localization, mapping to
> > service engine, mapping to entities, etc ...*
> > *
>
> I'm currently working on the rest-api plugin and I agree that it would
> be great to have it in the framework instead of being a plugin.
>
>
> > _12. No Specialization:_Because of the big repository, and having
> > everything point to everything, it's hard to specialize. Now compare
> > this to the approach we have in our work which is easier to specialize
> > in. For example, I have a component for payment gateway integration,
> > another one for twilio (SMS integration), and I have application
> > components for each solution in its own repository (HR, insurance
> > stuff, many others) and each is living in its own git repo yet they
> > work together just fine and without touching one line of code in the
> > framework or even core components (despite depending on them and using
> > pieces of them). I can jump into new versions of the framework
> > everyday without breaking a sweat. When I work on component X, I'm
> > only thinking of that and nothing else. I don't have to worry about
> > changing something that would crash another component or bring the
> > system down (happened to me many times in the past).
>
> For our projects, we were going the other way: we have framework +
> plugins in the same repository instead of handling different
> repositories. I think it would be a real pain to split the project in
> dozens of different repositories only to bring them together to
> integrate during development or deployment again.
>
> Can you describe how you are handling those different repositories in an
> efficient way to get a better picture?
>
> As long as your work in a component/application/plugin and don't touch
> the framework code I also do not see why you should break something for
> other components? Of course, there ARE applications which are more
> fundamental than other (e.g. party, security) and if those are changed
> they might break other functionality. I see those applications belonging
> to the framework more than being independent applications.
>
> I agree that we should try to identify false or unnecessary dependencies
> though.
>
>
> >
> >
> > Solutions
> >
> > Whatever solutions we come up with I think they're going to be
> > difficult and would require community buy-in and support. I'm breaking
> > this down into things to keep, update, add and remove as follows:
> >
> >
> > Keep it
> >
> > - XSD signatures (so entity definitions, service definitions, XML
> > actions though should expand)
> > - Basic constructs of what is a component (with basic directory
> > structure)
> > - Data model
> > - Existing services (core data-model services)
> > - Java / groovy code that doesn't use core framework constructs or
> > goes too deep into the internals.
> > - Most used and useful APIs (Making queries, calling services, etc
> > ...) although it should all be converted to interfaces with separate
> > implementations where it's lacking.
> >
>
> Agree to all above.
>
>
> >
> > Improve It
> >
> > - Break the project to smaller git repositories.
> > - framework
> > - data model
> > - services
> > - common UI
> > - use-case applications (ERP, CRM, e-commerce, etc ...)
> > - plugins
> > - entire framework directory needs refurbishment and big chunks
> > probably require a delete / rewrite
> > - the web architecture (get rid of web-apps)
> > - the tests (lots of them need to be deleted anyway)
> > - Routing needs to be programmatic with ability to inject code.
> >
> >
> > Add It
> >
> > - Configuration override mechanism to remove the need to touch the
> > core. Overrides include everything including routing, component
> > registration, etc ...
> > - REST integration into the core
> > - Artifact based security that maps users / groups to artifacts
> > without the need to inject security code into every screen
> > - A super-set of XML actions that can be applied in all contexts and
> > can embed groovy code
> > - New, simple use-case based user interface. It needs to be designed
> > such that it is easy to extend to avoid the all-or-nothing UI that
> > currently exists.
> >
> >
> > Remove It / Deprecate It / Overwrite It
> >
> > - Most of the existing UI (needs to be replaced with simpler use-case
> > based interface)
> > - Lots of the core framework code (needs major rewrites, needs
> > interfaces, import libraries for things like transactions, security,
> > concurrency, etc ...)
> > - Probably some of the existing plugins might need to be deprecated.
>
> See my questions above for some of the topics. I will add more details
> and my point-of-view once we've split up the topics.
>
> The overall change would be huge and we should paint a clear picture of
> what we want to achieve and which way to go in detail.
>
> We'll have to prioritize with a good cost/effort consideration and do
> implementations step-by-step with a proper migration plan. Maybe having
> feature branches for a longer time to be able to put out releases if the
> development gets stuck or lasts longer.
>
>
> >
> >
> > Quality Control
> >
> > Investing in such a huge effort requires that we change the way we
> > view the project after / while work is being completed. Code should be
> > approached with greater care and a lot of emphasis should be put on
> > high quality, readability, avoiding anti-patterns, enforcing good
> > patterns and best-practices. Some of the daily workflows need to
> > improve / change, more thorough auditing and peer-reviews, and so on.
>
> A big +1 for quality control. This will be a real challenge for the
> project as it requires dedication and enough ressources for review.
>
>
> >
> >
> > Tradeoffs
> >
> > It goes without saying that the above is a _lot_ of work and
> > commitment and requires a lot of effort. That's why I hesitated in
> > starting this thread, but I thought maybe showing the tradeoffs would
> > make it easier to weigh things out and decide on a direction to push.
> >
> >
> > Pros
> >
> > - Pretty much most of the above mentioned problems will be resolved.
> > - The framework would be resilient and future proof.
> > - We will have more options and capabilities in the future, it would
> > be easier to incorporate new technologies as they emerge
> > - Development would go faster
> > - Technical Debt will be reduced
> >
> >
> > Cons
> >
> > - Huge (maybe too huge) upfront costs in time and effort to make the
> > switch
> > - Extra effort needed to ensure a smooth upgrade path for existing
> > users / adopters / system integrators. A whole layer of code needs to
> > be written to ensure the ability to upgrade without massive rewrites.
> > - Risk of failure, exhaustion, inability to push through to the end
> > - Inconvenience to those with existing systems / solutions
> >
> >
> > Summary
> >
> > Change is the only constant. Inability to adapt to the future means
> > not surviving. I'm not confident what is the best change possible or
> > at what pace, but I do know that lack of change is a death sentence to
> > any project. Maybe it's time to reflect and think of where we'd like
> > to go and this thread is sort of a brain-dump of possible ideas.
> >
> > I look forward to hearing your thoughts.
>
> Thanks, Taher, for the thoughtful write-up of your view on the project
> and the fields of change you see to be tackled. I do not agree with all
> of them and I have a sensible view on cost/effort, thorough planning and
> the need to always have existing users/projects in mind. But I hope that
> we/the project will find a way to constantly work on improvements.
>
> Again, I would propose to split up the most important topics into
> threads with accompanying documentation in the wiki to help avoid
> confusion.
>
> I am curious what others have to say and if we can activate enough
> interest to improve.
>
>
> >
> > Cheers,
> >
> > Taher Alkhateeb
> >
> >
> > [1] Example mixed XML actions and groovy from moqui
> >
> > <if condition="settlementTerm?.orderPmtServiceRegisterId">
> > <then>
> > <service-call name="update#mantle.order.OrderPart"
> > in-map="[orderId:orderId,
> > orderPartSeqId:orderPartSeqId, settlementTermId:settlementTermId]"/>
> >
> > <entity-find-one entity-name="moqui.service.ServiceRegister"
> > value-field="serviceRegister">
> > <field-map field-name="serviceRegisterId"
> > from="settlementTerm.orderPmtServiceRegisterId"/>
> > </entity-find-one>
> > <script><![CDATA[
> > try {
> >
> ec.service.sync().name(serviceRegister.serviceName).parameters(context).softValidate(true).disableAuthz().call()
> > if (ec.message.hasError()) return
> > } catch (Throwable t) {
> > ec.logger.log(300, "Error calling auto order payments
> > service for SettlementTerm ${settlementTermId}", t)
> > }
> > ]]></script>
> > </then>
> > <else>
> > <service-call
> > name="mantle.account.PaymentServices.create#Payment" out-map="context"
> > in-map="context + [amount:amount,
> > amountUomId:amountUomId, fromPartyId:fromPartyId, toPartyId:toPartyId]"/>
> > </else>
> > </if>
> >
> > [2] https://ci.apache.org/projects/ofbiz/site/trunk/javadocs/
> >
>
