I understand that Sharan brought up the framework rewrite subject at
ApacheCon, and some attendees felt that the framework is fine and no
action needs to be taken.
In this message, I will try to give a detailed explanation of why a
framework rewrite is necessary. I don't plan to take any further action
on this subject, because I've brought it up before without success, and
I'm tired of discussing it. It is my hope that the light bulb will click
on in someone's head and they will take action.
My Background
-------------
I became a member of the OFBiz community in 2004. I immediately started
making contributions to the project by supplying patches to the issue
tracker. In 2007, I became a committer. Most of my initial work was on
the UI and some work in the applications (mainly Asset Maintenance and
Work Effort). I stayed away from touching the framework code because it
was deep, dark, and scary.
Eventually, I started to understand how the framework code works and I
made some minor modifications. As my understanding grew, I progressed to
rewriting large swaths of framework code - making it thread-safe, fault
tolerant, efficient, and easier to use.
I will list some of my contributions here, so everyone can have a clear
understanding of my experience with the framework code:
New Features
User Preferences
Visual Themes
Custom UI Label XML File Format
Temporal Expressions
Data Type Conversion Framework
Screen Widget Boundary Comments
Metrics
Integrations
UEL
iCalendar
JSR 223
WebDAV
LDAP
Refactorings/Improvements
FlexibleStringExpander
FlexibleMapExpander
FOP Integration
FreeMarkerWorker
Date-Time Handling
Mini-language
Job Scheduler
In addition, I have performed innumerable framework bug fixes.
So, the contents of this message come from years of experience mucking
about in the framework code.
Okay, let's get started...
Initial Problem Statement
-------------------------
In 2009, David Jones started a framework rewrite in a branch:
https://svn.apache.org/repos/asf/ofbiz/branches/executioncontext20090716
At the time, there was some agreement that a rewrite was necessary, but
there was disagreement as to how the rewrite should be incorporated into
the project:
https://mail-archives.apache.org/mod_mbox/ofbiz-dev/200908.mbox/%3c455601.62605...@web63102.mail.re1.yahoo.com%3E
There were concerns that a rewrite would break backward compatibility.
Work on the rewrite branch stopped. Eventually, Jacopo suggested the
community be more accepting of backward-incompatible changes:
https://mail-archives.apache.org/mod_mbox/ofbiz-dev/201004.mbox/%3cd24f129d-4f9f-444e-84af-aca46f499...@hotwaxmedia.com%3e
Despite an effort to convince David to proceed with the framework
rewrite, he ended up doing it in a separate project:
http://mail-archives.apache.org/mod_mbox/ofbiz-dev/201104.mbox/%3c07565c88-4023-4d24-93a3-a4906e86f...@me.com%3E
This page describes differences between OFBiz and Moqui, and within it
you can extract information on the problems David was trying to solve:
http://sourceforge.net/p/moqui/discussion/1086127/thread/4c52f240/
There was an email he sent out on the OFBiz dev list where he listed the
problems he saw in the framework, but I can't find it. The rest of this
message will include the issues he mentioned (the ones I remember). I
was in agreement with him at the time, and I still agree that a
framework rewrite is necessary.
The Problems
------------
Code is scattered everywhere - due to an initial effort to make the
framework modular. This causes serious problems. The mere fact that
components like entityext and securityext EXIST makes it clear that
there are problems - those components should not be there. Also, we run
into the recurring problem of circular dependencies (component A will
not build unless component B is built, and component B will not build
unless component A is built).
Bad separation of concerns. There are far too many examples of classes
that try to be everything to everyone. This makes debugging difficult,
and it makes maintenance/improvements a nightmare. [Using an analogy,
consider an automobile design where a spark plug is not separate from a
transmission. Instead, the automobile uses a spark-plug-transmission. So
when the engine is running rough because the spark plug is bad, you have
to replace the spark plug AND the transmission.] A good framework
example can be found in my rewrite of the mini-language code.
Originally, the models AND the script execution context both contained
script behaviors - making debugging/improvements difficult. I changed it
so only the models contain script behavior and the script execution
context contains only the script execution state.
Lack of good OO design. There are many places where a bit of framework
functionality is contained in a single method that is hundreds or
thousands of lines long. There is a term for that: Brittle Code. Code
isn't reused. Instead, it is copy-and-pasted all over - so when a
problem is found in the C&P code, it has to be fixed in many places
instead of one.
Fail-slow design. There are a lot of places in low-level code where an
error condition is encountered, but instead of throwing an exception,
the error is ignored and maybe it is logged, or the code tries to
"guess" at a solution and then provide an arbitrary default behavior.
I've seen many developers struggle with debugging a problem because they
didn't look at the logs, or because the error was not logged and there
is no way of knowing what caused it. They end up spending hours
single-stepping through code until it reaches the error.
Out-of-date code. A good example is the use of Javolution. That library
was beneficial in the Java 1.4 days, but it is not necessary today
because of improved garbage collection. Another good example is DCL
code. DCL was used extensively in OFBiz, but it is clearly documented to
be an unreliable design (I can get it to fail 90% of the time). Some DCL
code has been replaced, but a lot of it is still there.
Portions of the API are overly complicated. Some methods require a
collection of user-specified artifacts/arguments, which makes client
code complicated and verbose. (David solved that problem with his
Execution Context.) Portions of the API are cluttered with unnecessary
"convenience methods" - making the API harder to learn and memorize. In
some places, a domain-specific API is spread across instance methods and
static methods and across different classes - making the API hard to
understand and use. Yes, there can be good designs that require
something like that, but in the OFBiz framework, it exists because of a
bad design, not a good one.
Use of thread-local variables. This makes multi-threaded design
impossible. The J2EE specification and the Servlet API require one
thread per request (and most J2EE libraries depend on that behavior), so
the current design makes sense from a J2EE perspective, but what if I
don't want to run the framework in a J2EE container? Which leads to the
next problem...
Dependence on J2EE designs/APIs/libraries. There are developers in the
Java community (myself included) who are beginning to question if J2EE
is really necessary to run web applications. The folks at Atomikos are a
good example. OFBiz does not use EJBs, so tying the framework to J2EE
does not make sense. It would be better if the framework was designed to
run outside a J2EE container, and then have container integration as an
option.
Configuration files are scattered everywhere. Anyone who has deployed
OFBiz in a production environment will agree this is a problem. Try
changing the HTTP/HTTPS and port settings - it is a nightmare. Some
configuration settings are in nonsensical places.
An abysmal lack of unit testing. I don't have an exact figure for code
coverage, but my gut feeling is coverage is less than 10%. Basically, we
all have our fingers crossed - hoping that the framework code works as
expected. This was made painfully obvious a while back when I was
looking at some entity caching code and thought to myself "this code
can't work." So I wrote some entity cache unit tests and confirmed that
the entity cache had serious problems. Think about that - years passed
with no entity cache unit tests and consequently we had no idea it
wasn't working.
Fix Versus Rewrite
------------------
Jira issues could be created for these problems and teams of developers
could work to fix them.
Or, we could create a branch and start over from scratch. This time
around, there should be less push-back from people concerned about
backwards compatibility. A rewrite offers the advantage of reconsidering
everything - like API design, general problem solving, and new features.
I created a Wiki page for a framework design:
https://cwiki.apache.org/confluence/display/OFBADMIN/Another+Framework+Vision
but there hasn't been much interest in it. If the community decides to
go ahead with a rewrite, then please feel free to use the Wiki pages as
a guide.
Sandglass Foundation
--------------------
Like David, I came to the conclusion that a framework rewrite would be
easier outside the OFBiz community. So, I created my own library called
Foundation:
http://www.sandglass-software.com/products/sandglass/documents/Foundation_Brochure.pdf
(PDF)
and I only mention it here to stress how wonderful it can be to start
with a clean slate and design an API that is concise yet powerful.
(Please do not discuss Foundation here, contact me privately if you want
more information.)
Some examples of what can be done with a rewrite:
A single configuration file
Use ANSI/ISO SQL SELECT statement strings instead of constructing
complicated Java structures
Simultaneous asynchronous queries
Relational integrity across multiple datasources
Multi-table SELECT across multiple datasources
Automatic and transparent row version control
Automatic and transparent multi-language datasource support
Abstract entities (similar to SQL user types)
Service engine throttling (protects against server over-utilization)
Simplified security (authorization)
(https://cwiki.apache.org/confluence/display/OFBIZ/OFBiz+Security+Redesign)
Pure interface-based API - so developers are free to modify
framework behavior by using decorators
Thorough unit tests
Benefits of a rewrite:
Reduced resource requirements (lower hosting fees)
Reduce application development time - due to a simplified API
Easier framework code maintenance
Better reliability
Conclusion
----------
Like I said at the start, this is all I will say about the subject. I'm
done trying to convince everyone. I hope someone agrees with me and they
are able to build support for the idea.
--
Adrian Crum
Sandglass Software
www.sandglass-software.com
