Re: Why A Framework Rewrite Is Necessary

[Jacques Le Roux]

I have not much time to comment at the moment, but at large this seems wise to 
me (in other words I kinda agree)

Jacques

Le 16/10/2015 02:48, Scott Gray a écrit :

I don't think this community is large enough to attempt to rewrite all of
our applications to suit Moqui while at the same time still supporting them
under the existing framework.  That may well be the case for a homegrown
framework rewrite as well depending on how the applications would be
affected.  This is the main reason why I tend to prefer incremental
improvements to the existing framework.  But if people within the community
wanted to try for either of the "new framework" options, I certainly don't
have a problem with it and am happy to see some infrastructure donated to
the cause.

There's simply no need to form a consensus on any major decision in the
early stages, proponents just need to be willing to accept that their
endeavors might not become part of the project.  David certainly didn't
wait until he had a community backing him to go ahead and start work on
Moqui.  The best way to get something done is to get started on it, if no
one follows then it's either a bad idea or you're doing a bad job of
selling it.

By the way, I'm surprised I haven't seen any efforts within the Moqui
community to rewrite any of the OFBiz apps.  I don't have any ideas as to
why that hasn't happened, but I'm curious about it.  Surely it would be
easier for the Moqui community who already know the framework and are
probably also familiar with OFBiz to try this than us.

Regards
Scott


On 16 October 2015 at 12:20, Hans Bakker <h.bak...@antwebsystems.com> wrote:

Well said: +1


On 15/10/15 22:13, Al Byers wrote:

Fair enough, Adrian, but is this a widely held position on OFBiz that as a
group you could produce a product that is so much better than Moqui that
it
would be worth the time and risk (the risk of burning cycles that you will
never get back only to have the venture never succeed.) Have all the
people
who would like to do a rewrite done the same gap analysis on Moqui vs what
you would like to see OFBiz be? Does everyone understand and agree that
the
deficiencies that you see in Moqui are important? My concern is that the
many people who understandably want a rewrite of OFBiz will be swayed by
your arguments without there being due diligence being done. I think that
there are a lot of veterans of OFBiz who have confidence in David's
development skills, but who are not pressing for a move to Moqui or for a
rewrite because they have so much invested in the current state of OFBiz
and recognized what a long shot a rewrite would be and how much work it
would require of them to move over. But anybody who is open to a rewrite
would be foolish not to consider the work done by the principal architect
of OFBiz; work that was meant to address deficiencies in OFBiz.

On Thu, Oct 15, 2015 at 7:58 AM, Adrian Crum <
adrian.c...@sandglass-software.com> wrote:

Keep in mind that much of David's code in OFBiz has been rewritten. So

yes, we CAN do a better job than him. Also keep in mind that Moqui
duplicates some of the problems I listed - so by using Moqui, we keep the
same problems instead of fixing them. On the other hand, it is David's
responsibility to fix them, not ours.

If we created a sub-project, then we would have the opportunity to review
committer permissions and perhaps restrict access to the new code.

Adrian Crum
Sandglass Software
www.sandglass-software.com

On 10/15/2015 6:34 AM, Al Byers wrote:

I was waiting for someone to bring this up. David Jones created Moqui

with
the same end in mind that the rewrite is meant to accomplish. Do you
think
that you will do a better job than him? Even if you could, would it be
so
much better that it warrants the effort that it would take?

Is this a political thing? I don't know that David would give up control
of
Moqui Core and I, for one, would not want him to. So many of OFBiz's
problems are the result of ineptitude on the part of committers who did
not
know what they were doing (like me.) But I don't know that the Core will
change all that much going forward and there should be places in the
Mantle
to contribute and, most certainly, in the Crust.

I know that there were some valid questions about licensing and project
management, but I would think that they could be worked out.

On Thu, Oct 15, 2015 at 4:46 AM, Hans Bakker <
h.bak...@antwebsystems.com>
wrote:

Why not skip this step, using moqui which is already up and running and

then as Adrian states as a next step: applications could be pulled down
and
adapted to it.

Hans



On 15/10/15 16:51, Jacques Le Roux wrote:

I'm in the same mood than Paul and Scott. So a sub-project could indeed

be a solution.

Jacques

Le 15/10/2015 03:11, Adrian Crum a écrit :

I agree that a sub-project would be nice to have. Once the new
framework

is up and running, applications could be pulled down and adapted to
it.

Adrian Crum
Sandglass Software
www.sandglass-software.com

On 10/14/2015 5:53 PM, Ron Wheeler wrote:

This seems to be very good advice.

A completely separate sub-project under OFBiz with its own mailing
lists
would keep the people together yet all allow the new framework the
flexibility to move forward.

Ron
On 14/10/2015 8:27 PM, Scott Gray wrote:

My advice is as follows:

1. If people are interested, then get them together and start
working
on it.
2. Find somewhere to do the work.  I don't think a branch is
appropriate
because it's completely new development rather than a
refactoring.  I
don't
have any objections to it being done under the ASF OFBiz umbrella
(although
I don't really see the need either).
3. Set up a separate mailing list for discussion.  Generally I'd
try
and
keep quiet about it in the early stages on the dev/user lists or
other
marketing channels because it could potentially harm adoption of
our
existing framework (think AngularJS 2.0).

There really isn't any need to get early stage sign-off from the
PMC
or
anyone else in the community.  You only need enough PMC approval to
get the
required infrastructure sorted, which I don't think would be an
issue.
>From there, it's really up to the community to decide whether or
not
the
thing will fly.

Regards
Scott


On 15 October 2015 at 08:21, Adrian Crum
<adrian.c...@sandglass-software.com

wrote:

I understand that Sharan brought up the framework rewrite subject
at
ApacheCon, and some attendees felt that the framework is fine and
no
action
needs to be taken.

In this message, I will try to give a detailed explanation of why
a
framework rewrite is necessary. I don't plan to take any further
action on
this subject, because I've brought it up before without success,
and
I'm
tired of discussing it. It is my hope that the light bulb will
click
on in
someone's head and they will take action.

My Background
-------------

I became a member of the OFBiz community in 2004. I immediately
started
making contributions to the project by supplying patches to the
issue
tracker. In 2007, I became a committer. Most of my initial work
was
on the
UI and some work in the applications (mainly Asset Maintenance and
Work
Effort). I stayed away from touching the framework code because it
was
deep, dark, and scary.

Eventually, I started to understand how the framework code works
and I
made some minor modifications. As my understanding grew, I
progressed
to
rewriting large swaths of framework code - making it thread-safe,
fault
tolerant, efficient, and easier to use.

I will list some of my contributions here, so everyone can have a
clear
understanding of my experience with the framework code:

        New Features

            User Preferences

            Visual Themes

            Custom UI Label XML File Format

            Temporal Expressions

            Data Type Conversion Framework

            Screen Widget Boundary Comments

            Metrics

        Integrations

            UEL

            iCalendar

            JSR 223

            WebDAV

            LDAP

        Refactorings/Improvements

            FlexibleStringExpander

            FlexibleMapExpander

            FOP Integration

            FreeMarkerWorker

            Date-Time Handling

            Mini-language

            Job Scheduler

In addition, I have performed innumerable framework bug fixes.

So, the contents of this message come from years of experience
mucking
about in the framework code.

Okay, let's get started...

Initial Problem Statement
-------------------------

In 2009, David Jones started a framework rewrite in a branch:




https://svn.apache.org/repos/asf/ofbiz/branches/executioncontext20090716

At the time, there was some agreement that a rewrite was
necessary,
but
there was disagreement as to how the rewrite should be
incorporated
into
the project:





https://mail-archives.apache.org/mod_mbox/ofbiz-dev/200908.mbox/%3c455601.62605...@web63102.mail.re1.yahoo.com%3E


There were concerns that a rewrite would break backward
compatibility.
Work on the rewrite branch stopped. Eventually, Jacopo suggested
the
community be more accepting of backward-incompatible changes:





https://mail-archives.apache.org/mod_mbox/ofbiz-dev/201004.mbox/%3cd24f129d-4f9f-444e-84af-aca46f499...@hotwaxmedia.com%3e


Despite an effort to convince David to proceed with the framework
rewrite,
he ended up doing it in a separate project:





http://mail-archives.apache.org/mod_mbox/ofbiz-dev/201104.mbox/%3c07565c88-4023-4d24-93a3-a4906e86f...@me.com%3E


This page describes differences between OFBiz and Moqui, and
within
it you
can extract information on the problems David was trying to solve:

http://sourceforge.net/p/moqui/discussion/1086127/thread/4c52f240/

There was an email he sent out on the OFBiz dev list where he
listed
the
problems he saw in the framework, but I can't find it. The rest of
this
message will include the issues he mentioned (the ones I
remember).
I
was
in agreement with him at the time, and I still agree that a
framework
rewrite is necessary.

The Problems
------------

Code is scattered everywhere - due to an initial effort to make
the
framework modular. This causes serious problems. The mere fact
that
components like entityext and securityext EXIST makes it clear
that
there
are problems - those components should not be there. Also, we run
into the
recurring problem of circular dependencies (component A will not
build
unless component B is built, and component B will not build unless
component A is built).

Bad separation of concerns. There are far too many examples of
classes
that try to be everything to everyone. This makes debugging
difficult, and
it makes maintenance/improvements a nightmare. [Using an analogy,
consider
an automobile design where a spark plug is not separate from a
transmission. Instead, the automobile uses a
spark-plug-transmission.
So
when the engine is running rough because the spark plug is bad,
you
have to
replace the spark plug AND the transmission.] A good framework
example can
be found in my rewrite of the mini-language code. Originally, the
models
AND the script execution context both contained script behaviors -
making
debugging/improvements difficult. I changed it so only the models
contain
script behavior and the script execution context contains only the
script
execution state.

Lack of good OO design. There are many places where a bit of
framework
functionality is contained in a single method that is hundreds or
thousands
of lines long. There is a term for that: Brittle Code. Code isn't
reused.
Instead, it is copy-and-pasted all over - so when a problem is
found
in the
C&P code, it has to be fixed in many places instead of one.

Fail-slow design. There are a lot of places in low-level code
where
an
error condition is encountered, but instead of throwing an
exception,
the
error is ignored and maybe it is logged, or the code tries to
"guess"
at a
solution and then provide an arbitrary default behavior. I've seen
many
developers struggle with debugging a problem because they didn't
look
at
the logs, or because the error was not logged and there is no way
of
knowing what caused it. They end up spending hours single-stepping
through
code until it reaches the error.

Out-of-date code. A good example is the use of Javolution. That
library
was beneficial in the Java 1.4 days, but it is not necessary today
because
of improved garbage collection. Another good example is DCL code.
DCL
was
used extensively in OFBiz, but it is clearly documented to be an
unreliable
design (I can get it to fail 90% of the time). Some DCL code has
been
replaced, but a lot of it is still there.

Portions of the API are overly complicated. Some methods require a
collection of user-specified artifacts/arguments, which makes
client
code
complicated and verbose. (David solved that problem with his
Execution
Context.) Portions of the API are cluttered with unnecessary
"convenience
methods" - making the API harder to learn and memorize. In some
places, a
domain-specific API is spread across instance methods and static
methods
and across different classes - making the API hard to understand
and
use.
Yes, there can be good designs that require something like that,
but
in the
OFBiz framework, it exists because of a bad design, not a good
one.

Use of thread-local variables. This makes multi-threaded design
impossible. The J2EE specification and the Servlet API require one
thread
per request (and most J2EE libraries depend on that behavior), so
the
current design makes sense from a J2EE perspective, but what if I
don't
want to run the framework in a J2EE container? Which leads to the
next
problem...

Dependence on J2EE designs/APIs/libraries. There are developers in
the
Java community (myself included) who are beginning to question if
J2EE is
really necessary to run web applications. The folks at Atomikos
are
a
good
example. OFBiz does not use EJBs, so tying the framework to J2EE
does
not
make sense. It would be better if the framework was designed to
run
outside
a J2EE container, and then have container integration as an
option.

Configuration files are scattered everywhere. Anyone who has
deployed
OFBiz in a production environment will agree this is a problem.
Try
changing the HTTP/HTTPS and port settings - it is a nightmare.
Some
configuration settings are in nonsensical places.

An abysmal lack of unit testing. I don't have an exact figure for
code
coverage, but my gut feeling is coverage is less than 10%.
Basically,
we
all have our fingers crossed - hoping that the framework code
works
as
expected. This was made painfully obvious a while back when I was
looking
at some entity caching code and thought to myself "this code can't
work."
So I wrote some entity cache unit tests and confirmed that the
entity
cache
had serious problems. Think about that - years passed with no
entity
cache
unit tests and consequently we had no idea it wasn't working.

Fix Versus Rewrite
------------------

Jira issues could be created for these problems and teams of
developers
could work to fix them.

Or, we could create a branch and start over from scratch. This
time
around, there should be less push-back from people concerned about
backwards compatibility. A rewrite offers the advantage of
reconsidering
everything - like API design, general problem solving, and new
features.

I created a Wiki page for a framework design:





https://cwiki.apache.org/confluence/display/OFBADMIN/Another+Framework+Vision


but there hasn't been much interest in it. If the community
decides
to go
ahead with a rewrite, then please feel free to use the Wiki pages
as a
guide.

Sandglass Foundation
--------------------

Like David, I came to the conclusion that a framework rewrite
would
be
easier outside the OFBiz community. So, I created my own library
called
Foundation:





http://www.sandglass-software.com/products/sandglass/documents/Foundation_Brochure.pdf


(PDF)

and I only mention it here to stress how wonderful it can be to
start
with
a clean slate and design an API that is concise yet powerful.
(Please
do
not discuss Foundation here, contact me privately if you want more
information.)

Some examples of what can be done with a rewrite:

        A single configuration file
        Use ANSI/ISO SQL SELECT statement strings instead of
constructing
complicated Java structures
        Simultaneous asynchronous queries
        Relational integrity across multiple datasources
        Multi-table SELECT across multiple datasources
        Automatic and transparent row version control
        Automatic and transparent multi-language datasource support
        Abstract entities (similar to SQL user types)
        Service engine throttling (protects against server
over-utilization)
        Simplified security (authorization) (



https://cwiki.apache.org/confluence/display/OFBIZ/OFBiz+Security+Redesign
)


        Pure interface-based API - so developers are free to modify
framework
behavior by using decorators
        Thorough unit tests

Benefits of a rewrite:

        Reduced resource requirements (lower hosting fees)
        Reduce application development time - due to a simplified
API
        Easier framework code maintenance
        Better reliability

Conclusion
----------

Like I said at the start, this is all I will say about the
subject.
I'm
done trying to convince everyone. I hope someone agrees with me
and
they
are able to build support for the idea.

--
Adrian Crum
Sandglass Software
www.sandglass-software.com