I understand that Sharan brought up the framework rewrite subject at
ApacheCon, and some attendees felt that the framework is fine and no
action needs to be taken.
In this message, I will try to give a detailed explanation of why a
framework rewrite is necessary. I don't plan to take any further
action on this subject, because I've brought it up before without
success, and I'm tired of discussing it. It is my hope that the light
bulb will click on in someone's head and they will take action.
My Background
-------------
I became a member of the OFBiz community in 2004. I immediately
started making contributions to the project by supplying patches to
the issue tracker. In 2007, I became a committer. Most of my initial
work was on the UI and some work in the applications (mainly Asset
Maintenance and Work Effort). I stayed away from touching the
framework code because it was deep, dark, and scary.
Eventually, I started to understand how the framework code works and I
made some minor modifications. As my understanding grew, I progressed
to rewriting large swaths of framework code - making it thread-safe,
fault tolerant, efficient, and easier to use.
I will list some of my contributions here, so everyone can have a
clear understanding of my experience with the framework code:
New Features
User Preferences
Visual Themes
Custom UI Label XML File Format
Temporal Expressions
Data Type Conversion Framework
Screen Widget Boundary Comments
Metrics
Integrations
UEL
iCalendar
JSR 223
WebDAV
LDAP
Refactorings/Improvements
FlexibleStringExpander
FlexibleMapExpander
FOP Integration
FreeMarkerWorker
Date-Time Handling
Mini-language
Job Scheduler
In addition, I have performed innumerable framework bug fixes.
So, the contents of this message come from years of experience mucking
about in the framework code.
Okay, let's get started...
Initial Problem Statement
-------------------------
In 2009, David Jones started a framework rewrite in a branch:
https://svn.apache.org/repos/asf/ofbiz/branches/executioncontext20090716
At the time, there was some agreement that a rewrite was necessary,
but there was disagreement as to how the rewrite should be
incorporated into the project:
https://mail-archives.apache.org/mod_mbox/ofbiz-dev/200908.mbox/%3c455601.62605...@web63102.mail.re1.yahoo.com%3E
There were concerns that a rewrite would break backward compatibility.
Work on the rewrite branch stopped. Eventually, Jacopo suggested the
community be more accepting of backward-incompatible changes:
https://mail-archives.apache.org/mod_mbox/ofbiz-dev/201004.mbox/%3cd24f129d-4f9f-444e-84af-aca46f499...@hotwaxmedia.com%3e
Despite an effort to convince David to proceed with the framework
rewrite, he ended up doing it in a separate project:
http://mail-archives.apache.org/mod_mbox/ofbiz-dev/201104.mbox/%3c07565c88-4023-4d24-93a3-a4906e86f...@me.com%3E
This page describes differences between OFBiz and Moqui, and within it
you can extract information on the problems David was trying to solve:
http://sourceforge.net/p/moqui/discussion/1086127/thread/4c52f240/
There was an email he sent out on the OFBiz dev list where he listed
the problems he saw in the framework, but I can't find it. The rest of
this message will include the issues he mentioned (the ones I
remember). I was in agreement with him at the time, and I still agree
that a framework rewrite is necessary.
The Problems
------------
Code is scattered everywhere - due to an initial effort to make the
framework modular. This causes serious problems. The mere fact that
components like entityext and securityext EXIST makes it clear that
there are problems - those components should not be there. Also, we
run into the recurring problem of circular dependencies (component A
will not build unless component B is built, and component B will not
build unless component A is built).
Bad separation of concerns. There are far too many examples of classes
that try to be everything to everyone. This makes debugging difficult,
and it makes maintenance/improvements a nightmare. [Using an analogy,
consider an automobile design where a spark plug is not separate from
a transmission. Instead, the automobile uses a
spark-plug-transmission. So when the engine is running rough because
the spark plug is bad, you have to replace the spark plug AND the
transmission.] A good framework example can be found in my rewrite of
the mini-language code. Originally, the models AND the script
execution context both contained script behaviors - making
debugging/improvements difficult. I changed it so only the models
contain script behavior and the script execution context contains only
the script execution state.
Lack of good OO design. There are many places where a bit of framework
functionality is contained in a single method that is hundreds or
thousands of lines long. There is a term for that: Brittle Code. Code
isn't reused. Instead, it is copy-and-pasted all over - so when a
problem is found in the C&P code, it has to be fixed in many places
instead of one.
Fail-slow design. There are a lot of places in low-level code where an
error condition is encountered, but instead of throwing an exception,
the error is ignored and maybe it is logged, or the code tries to
"guess" at a solution and then provide an arbitrary default behavior.
I've seen many developers struggle with debugging a problem because
they didn't look at the logs, or because the error was not logged and
there is no way of knowing what caused it. They end up spending hours
single-stepping through code until it reaches the error.
Out-of-date code. A good example is the use of Javolution. That
library was beneficial in the Java 1.4 days, but it is not necessary
today because of improved garbage collection. Another good example is
DCL code. DCL was used extensively in OFBiz, but it is clearly
documented to be an unreliable design (I can get it to fail 90% of the
time). Some DCL code has been replaced, but a lot of it is still there.
Portions of the API are overly complicated. Some methods require a
collection of user-specified artifacts/arguments, which makes client
code complicated and verbose. (David solved that problem with his
Execution Context.) Portions of the API are cluttered with unnecessary
"convenience methods" - making the API harder to learn and memorize.
In some places, a domain-specific API is spread across instance
methods and static methods and across different classes - making the
API hard to understand and use. Yes, there can be good designs that
require something like that, but in the OFBiz framework, it exists
because of a bad design, not a good one.
Use of thread-local variables. This makes multi-threaded design
impossible. The J2EE specification and the Servlet API require one
thread per request (and most J2EE libraries depend on that behavior),
so the current design makes sense from a J2EE perspective, but what if
I don't want to run the framework in a J2EE container? Which leads to
the next problem...
Dependence on J2EE designs/APIs/libraries. There are developers in the
Java community (myself included) who are beginning to question if J2EE
is really necessary to run web applications. The folks at Atomikos are
a good example. OFBiz does not use EJBs, so tying the framework to
J2EE does not make sense. It would be better if the framework was
designed to run outside a J2EE container, and then have container
integration as an option.
Configuration files are scattered everywhere. Anyone who has deployed
OFBiz in a production environment will agree this is a problem. Try
changing the HTTP/HTTPS and port settings - it is a nightmare. Some
configuration settings are in nonsensical places.
An abysmal lack of unit testing. I don't have an exact figure for code
coverage, but my gut feeling is coverage is less than 10%. Basically,
we all have our fingers crossed - hoping that the framework code works
as expected. This was made painfully obvious a while back when I was
looking at some entity caching code and thought to myself "this code
can't work." So I wrote some entity cache unit tests and confirmed
that the entity cache had serious problems. Think about that - years
passed with no entity cache unit tests and consequently we had no idea
it wasn't working.
Fix Versus Rewrite
------------------
Jira issues could be created for these problems and teams of
developers could work to fix them.
Or, we could create a branch and start over from scratch. This time
around, there should be less push-back from people concerned about
backwards compatibility. A rewrite offers the advantage of
reconsidering everything - like API design, general problem solving,
and new features.
I created a Wiki page for a framework design:
https://cwiki.apache.org/confluence/display/OFBADMIN/Another+Framework+Vision
but there hasn't been much interest in it. If the community decides to
go ahead with a rewrite, then please feel free to use the Wiki pages
as a guide.
Sandglass Foundation
--------------------
Like David, I came to the conclusion that a framework rewrite would be
easier outside the OFBiz community. So, I created my own library
called Foundation:
http://www.sandglass-software.com/products/sandglass/documents/Foundation_Brochure.pdf
(PDF)
and I only mention it here to stress how wonderful it can be to start
with a clean slate and design an API that is concise yet powerful.
(Please do not discuss Foundation here, contact me privately if you
want more information.)
Some examples of what can be done with a rewrite:
A single configuration file
Use ANSI/ISO SQL SELECT statement strings instead of constructing
complicated Java structures
Simultaneous asynchronous queries
Relational integrity across multiple datasources
Multi-table SELECT across multiple datasources
Automatic and transparent row version control
Automatic and transparent multi-language datasource support
Abstract entities (similar to SQL user types)
Service engine throttling (protects against server
over-utilization)
Simplified security (authorization)
(https://cwiki.apache.org/confluence/display/OFBIZ/OFBiz+Security+Redesign)
Pure interface-based API - so developers are free to modify
framework behavior by using decorators
Thorough unit tests
Benefits of a rewrite:
Reduced resource requirements (lower hosting fees)
Reduce application development time - due to a simplified API
Easier framework code maintenance
Better reliability
Conclusion
----------
Like I said at the start, this is all I will say about the subject.
I'm done trying to convince everyone. I hope someone agrees with me
and they are able to build support for the idea.
