As with anything, the law of diminishing returns also applies to OFBiz and tests. This is not true for unit tests and system integration tests, but also for user acceptance and performance tests.
Nevertheless, the work done up to now is a good start and - I feel confident - appreciated. And unit tests are certainly valuable in the framework stack. How it will be for functions (regarding components in application and special purpose stack) needs to be addressed when we reach that bridge. Best regards, Pierre Smits ORRTIZ.COM <http://www.orrtiz.com> OFBiz based solutions & services OFBiz Extensions Marketplace http://oem.ofbizci.net/oci-2/ On Tue, Jul 19, 2016 at 8:22 AM, Taher Alkhateeb <slidingfilame...@gmail.com > wrote: > Hi Scott, > > Well spoken thank you. Okay may I suggest that for any such work we will > discuss it here see its Merit and if it makes sense then we take it in. It > is a little early to discuss it right now because we did not yet go to the > higher-level components. Once we do I'll be sure to have a conversation > about this in here and would appreciate your input to it. > > Regards, > > Taher Alkhateeb > > On Jul 19, 2016 9:00 AM, "Scott Gray" <scott.g...@hotwaxsystems.com> > wrote: > > > Yeah I'm sure unit tests probably worked well for the start component > given > > it is the lowest level component in the system and closest to a basic > java > > app. I just think the value proposition might decrease the further up > the > > stack we move with them. I'm not against unit tests when they prove > > useful, but further up the stack I think we should prove the case for > them > > before doing much work to support the mocking that will be required to > keep > > them inline with 'best practices'. > > > > In OFBiz "bad or unwanted" tends to come mostly in the way of increasing > > complexity from features that require more effort to maintain than the > > value they provide to the community. I think there's a chance attempting > > to mock service tests could fall into that bucket. I could be wrong, > but I > > think it's worth looking into before we declare that unit tests are the > > best form of testing for OFBiz. > > > > Regards > > Scott > > > > On 19 July 2016 at 17:37, Taher Alkhateeb <slidingfilame...@gmail.com> > > wrote: > > > > > Hi Scott, > > > > > > Thank you for the feedback. To be focused exactly on integration vs > > unit, I > > > already mentioned above that at least for me the main objective is to > > > confidently and quickly run the tests in short bursts of red-green > > > refactor. This allows me to refactor code without waiting in front of > the > > > screen in between test cycles thus giving me immediate feedback on any > > > errors I made. Perhaps my intro was too long so this is the squeezed > out > > > version of it. > > > > > > I already had one round of testing in the start component which was > much > > > faster that way and had an immediate impact. Oh and by the way, you > > cannot > > > test the start component with integration tests for example unless you > do > > > it from an external component which cannot access package protected > > items. > > > > > > This style of coding is applicable I think to any software project > > > inclusive of OFBiz. Maybe in certain components more than others or > > certain > > > areas more than others. But I can't see how it could be bad or > unwanted. > > > > > > Taher Alkhateeb > > > > > > On Jul 19, 2016 8:18 AM, "Scott Gray" <scott.g...@hotwaxsystems.com> > > > wrote: > > > > > > > Thanks Ron and Taher for your responses, I appreciate them but I > don't > > > hear > > > > much in the conversation that speaks to the value of unit tests over > > > > integration tests. Most of the thoughts shared speaks more to the > > value > > > of > > > > tests rather than differences in the type of tests. > > > > > > > > Speed: At an application level we have ~685 tests that run in 35 > > seconds > > > > (excluding build and data load). Another point is that there isn't > > much > > > > reason why tests can't be run in parallel rather than sequentially. > > > > TDD: Integration tests in OFBiz are as simple if not simpler to write > > > than > > > > unit tests and just as useful for TDD. > > > > I'm not sure if I missed any other points raised regarding > integration > > > vs. > > > > unit tests? > > > > > > > > I'm not looking to start a big long debate on the topic, I just > wanted > > to > > > > speak out that someone out there (me) doesn't think unit tests are > the > > > best > > > > solution for testing OFBiz applications. > > > > > > > > Regards > > > > Scott > > > > > > > > On 19 July 2016 at 16:52, Taher Alkhateeb < > slidingfilame...@gmail.com> > > > > wrote: > > > > > > > > > Hi Scott, > > > > > > > > > > Thank you for your input. Your ideas are thought provoking and > > > enriching > > > > to > > > > > the conversation. > > > > > > > > > > The way I look at it, the general rule is usually many unit tests, > > less > > > > > integration tests, lesser functional tests. So we are not excluding > > any > > > > > types of test, all of them are important in certain areas for > certain > > > > > purposes with certain quantity. Usually integration tests are less > > > > because > > > > > as you said they just grab more. I like the picture below as a > > general > > > > > guideline > > > > > > > > > > http://i.stack.imgur.com/fjQvQ.png > > > > > > > > > > As you already mentioned unit tests are useful for the framework. I > > > > > discovered errors in code that I wrote which I was very very > careful > > > > with. > > > > > I immediately learned the lesson that humans are not designed to > > code, > > > > and > > > > > TDD gives you confidence as you build your code with those short > > 30-60 > > > > > second red-green refactors. I feel much much safer and more > confident > > > > > writing code that way, the test also documents how to use the api, > > > > > refactoring and feature change becomes less terrifying. Also messy > > code > > > > is > > > > > usually not test friendly thus refactoring and unit tests go > > > hand-in-hand > > > > > for improving the code base. > > > > > > > > > > Also, I am sorry to say that the framework code is rather messy and > > > > brittle > > > > > in many places. I think you probably encountered this yourself. The > > > same > > > > is > > > > > said for the applications. Now If we start refactoring without unit > > > tests > > > > > then we are back to scary business. So much can go wrong so fast > and > > > > break > > > > > things you never expected to break. The framework with all > > applications > > > > > require heavy refactoring of things like massive ugly methods, > > > sandwiched > > > > > logic, heavy shared mutable state, hidden dependencies, poor > > > > > interfaces, and much more. Every time I touch something I get > shocked > > > at > > > > > how bad it looks, spaghetti logic to no end. If you refactor with > > > > > integration tests instead of unit tests then you will come down to > a > > > > > screeching halt as you wait between these test cycles. My computer > > > > actually > > > > > takes 10 minutes or more for a full clean load data and testing. > > > > > > > > > > Now talking about mocking you raise some interesting points, what > can > > > you > > > > > mock vs utilizing integration tests? This is an excellent question > > that > > > > > does not have a simple answer, and you mostly learn as you code, > > > > difficult > > > > > to envision without coding. However simple things like Java > services > > > can > > > > be > > > > > mocked with a standard mocking class that you use everywhere. > > > > Transactions > > > > > are difficult to mock and better left to integration tests I think. > > > SECAs > > > > > might be mocked by simply passing output to input as chaining > > methods. > > > > Mind > > > > > you I am not 100% sure of all of this, coding is the ultimate guide > > to > > > > what > > > > > can or cannot be done. > > > > > > > > > > So I am not suggesting unit tests as a best-practice (even though > it > > > is). > > > > > Instead I suggest it as something that I and others did and got a > > huge > > > > > psychological relief and confidence and comfort from. Swimming in > > code > > > > > without tests is a terrifying business, made more terrifying if the > > > code > > > > is > > > > > bad. Short bursts of red/green/red/green makes you feel good as you > > > build > > > > > up your logic. And I don't have an exact vision of how to do it, > > > because > > > > > the details always win. > > > > > > > > > > I look forward to hearing your thoughts and thank you for enriching > > > this > > > > > conversation. > > > > > > > > > > Taher Alkhateeb > > > > > > > > > > On Tuesday, 19 July 2016, Scott Gray <scott.g...@hotwaxsystems.com > > > > > > wrote: > > > > > > > > > > > I know I'm late to the party here, but I just want to say that I > > > think > > > > > > integration tests have far greater value to OFBiz than unit > tests. > > > > > Mostly > > > > > > because we tend to have quite a low number of tests and > integration > > > > tests > > > > > > give us much better coverage per line of test code and the tests > > are > > > > much > > > > > > closer to the real world scenarios the application might > encounter. > > > > > > > > > > > > I don't really see how unit tests could be applied to > non-framework > > > > > testing > > > > > > in a useful manner, could you expand on your vision in that > regard? > > > I > > > > > mean > > > > > > would we be testing something smaller than a service as the > 'unit'? > > > > What > > > > > > would we mock? Would transaction management still be active? What > > > > happens > > > > > > when the service calls another service, I guess we mock the > > response > > > > from > > > > > > that service (how)? > > > > > > > > > > > > It just seems a very complicated method to achieve a less > thorough > > > but > > > > > > albeit faster (maybe) test result. > > > > > > > > > > > > A build, data load and full test run takes 4m 9s on my laptop. > > > > Excluding > > > > > > build, data load and framework tests: the application level tests > > > take > > > > > 35s, > > > > > > not very expensive IMO. The data load time can be reduced to > > > > practically > > > > > > nothing by copying a clean slate database into runtime for each > > run. > > > > > > > > > > > > I'm mostly just suggesting we be wary of adding complicated > testing > > > > > > procedures in the hope of achieving some 'best practice' result > > which > > > > in > > > > > > reality will provide minimal benefits. > > > > > > > > > > > > Regards > > > > > > Scott > > > > > > > > > > > > On 18 July 2016 at 18:57, Taher Alkhateeb < > > > slidingfilame...@gmail.com > > > > > > <javascript:;>> > > > > > > wrote: > > > > > > > > > > > > > Hello Akash, > > > > > > > > > > > > > > Fantastic, I have a few unit tests almost done to be included > in > > > the > > > > > > start > > > > > > > component. I will create a new subtask under OFBIZ-1463 to > commit > > > the > > > > > > tests > > > > > > > so you can use them as a reference if you like to. > > > > > > > > > > > > > > I also recommend that you follow the same directory structure > > > between > > > > > the > > > > > > > test code and production code. So for example: > > > > > > > > > > > > > > Production code: > > > > > > framework/start/src/main/java/org/apache/ofbiz/base/start > > > > > > > Test code: > > > framework/start/src/test/java/org/apache/ofbiz/base/start > > > > > > > > > > > > > > The benefit of this hierarchy is that you can access non-public > > > > > (package > > > > > > > protected) methods for testing. This is in fact exactly what I > > > needed > > > > > to > > > > > > be > > > > > > > able to apply some of the tests. > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > Taher Alkhateeb > > > > > > > > > > > > > > On Mon, Jul 18, 2016 at 9:22 AM, Akash Jain < > > > > > > akash.j...@hotwaxsystems.com <javascript:;>> > > > > > > > wrote: > > > > > > > > > > > > > > > Thanks Taher for nice initiative! > > > > > > > > > > > > > > > > We are planning to written unit tests to all components under > > > > > > OFBIZ-1463 > > > > > > > > > > > > > > > > Thanks and Regards > > > > > > > > -- > > > > > > > > Akash Jain > > > > > > > > > > > > > > > > On Mon, Jul 18, 2016 at 10:36 AM, Taher Alkhateeb < > > > > > > > > slidingfilame...@gmail.com <javascript:;>> wrote: > > > > > > > > > > > > > > > > > Hello Everyone, > > > > > > > > > > > > > > > > > > In reference to this thread and the Jira OFBIZ-7254, I'm > very > > > > happy > > > > > > to > > > > > > > > > announce that OFBiz is now ready for applying unit tests > with > > > > only > > > > > 8 > > > > > > > new > > > > > > > > > lines of code in the build script (r1753143) :) > > > > > > > > > > > > > > > > > > I recommend we do the following moving forward: > > > > > > > > > > > > > > > > > > 1- Introduce unit tests as much as we can to all components > > > > > > > > > 2- Migrate most of the integration tests we currently have > to > > > > unit > > > > > > > tests > > > > > > > > > (since they are designed to do very little integration). > > > > > > > > > > > > > > > > > > This is a great chance for us to practice real TDD in OFBiz > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > > > > > Taher Alkhateeb > > > > > > > > > > > > > > > > > > On Mon, Jun 13, 2016 at 7:43 AM, Pranay Pandey < > > > > > > > > > pranay.pan...@hotwaxsystems.com <javascript:;>> wrote: > > > > > > > > > > > > > > > > > > > +1 > > > > > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > > > > > > > > > > Pranay Pandey > > > > > > > > > > HotWax Systems > > > > > > > > > > http://www.hotwaxsystems.com/ > > > > > > > > > > > > > > > > > > > > On Fri, Jun 10, 2016 at 7:46 PM, Taher Alkhateeb < > > > > > > > > > > slidingfilame...@gmail.com <javascript:;> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hello Everyone, > > > > > > > > > > > > > > > > > > > > > > I was able to get a few tests running and this is very > > > > doable. > > > > > > But > > > > > > > I > > > > > > > > > > faced > > > > > > > > > > > a big problem in designing the testing framework > because > > of > > > > > ANT. > > > > > > > > > > > > > > > > > > > > > > The problem > > > > > > > > > > > ---------------- > > > > > > > > > > > The way the build scripts are designed in OFBiz are > very > > > > > > complex. A > > > > > > > > > > master > > > > > > > > > > > file calls other files which call other files. And in > the > > > > > middle > > > > > > > you > > > > > > > > > have > > > > > > > > > > > external libraries (ant-contrib) and macros, and > > variables, > > > > and > > > > > > > class > > > > > > > > > > path > > > > > > > > > > > declarations, and and and .... > > > > > > > > > > > > > > > > > > > > > > I cannot declare the tests programmatically (with JUnit > > > test > > > > > > > suites) > > > > > > > > > > > because this means lower level components would depend > on > > > > > higher > > > > > > > > level > > > > > > > > > > > components. So I have to do it in ANT, by navigating > this > > > > maze > > > > > of > > > > > > > > build > > > > > > > > > > > scripts, and it was a headache for me just to read > them, > > > let > > > > > > alone > > > > > > > > > modify > > > > > > > > > > > them to create a testing framework. > > > > > > > > > > > > > > > > > > > > > > Suggested Solution > > > > > > > > > > > ------------------------ > > > > > > > > > > > I suggest to implement the testing framework in Gradle, > > and > > > > > > simply > > > > > > > > call > > > > > > > > > > it > > > > > > > > > > > from within ant. This is a middle solution that > sustains > > > ant > > > > > for > > > > > > > now, > > > > > > > > > but > > > > > > > > > > > can allow us to switch out later. > > > > > > > > > > > > > > > > > > > > > > This means I will just add one more file called > > > build.gradle > > > > in > > > > > > the > > > > > > > > top > > > > > > > > > > > level directory, and figure out the business logic for > > > > calling > > > > > > the > > > > > > > > test > > > > > > > > > > > suites from that file > > > > > > > > > > > > > > > > > > > > > > I look forward to your feedback. > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > > > > > > > > > > Taher Alkhateeb > > > > > > > > > > > > > > > > > > > > > > On Wed, Jun 8, 2016 at 6:00 PM, Taher Alkhateeb < > > > > > > > > > > > slidingfilame...@gmail.com <javascript:;>> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Hi Everyone, > > > > > > > > > > > > > > > > > > > > > > > > Thank you all for your support, JIRA created in > > > > > > > > > > > > https://issues.apache.org/jira/browse/OFBIZ-7254 > > > > > > > > > > > > > > > > > > > > > > > > I will start working on it and try to implement ASAP > to > > > get > > > > > my > > > > > > > > focus > > > > > > > > > > back > > > > > > > > > > > > on refactoring. > > > > > > > > > > > > > > > > > > > > > > > > Cheers! > > > > > > > > > > > > > > > > > > > > > > > > Taher Alkhateeb > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Jun 8, 2016 at 4:58 PM, Deepak Dixit < > > > > > > > > > > > > deepak.di...@hotwaxsystems.com <javascript:;>> > wrote: > > > > > > > > > > > > > > > > > > > > > > > >> +1 > > > > > > > > > > > >> > > > > > > > > > > > >> Thanks & Regards > > > > > > > > > > > >> -- > > > > > > > > > > > >> Deepak Dixit > > > > > > > > > > > >> www.hotwaxsystems.com > > > > > > > > > > > >> > > > > > > > > > > > >> On Wed, Jun 8, 2016 at 7:12 PM, Mridul Pathak < > > > > > > > > > > > >> mridul.pat...@hotwaxsystems.com <javascript:;>> > > wrote: > > > > > > > > > > > >> > > > > > > > > > > > >> > +1 > > > > > > > > > > > >> > > > > > > > > > > > > >> > Makes perfect sense. > > > > > > > > > > > >> > > > > > > > > > > > > >> > -- > > > > > > > > > > > >> > Thanks & Regards, > > > > > > > > > > > >> > Mridul Pathak > > > > > > > > > > > >> > Senior Manager > > > > > > > > > > > >> > HotWax Systems > > > > > > > > > > > >> > http://www.hotwaxsystems.com > > > > > > > > > > > >> > > > > > > > > > > > > >> > > On Jun 8, 2016, at 2:41 PM, Taher Alkhateeb < > > > > > > > > > > > >> slidingfilame...@gmail.com <javascript:;>> > > > > > > > > > > > >> > wrote: > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > Hello Everyone, > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > After refactoring the start component and while > > > > starting > > > > > > on > > > > > > > > the > > > > > > > > > > base > > > > > > > > > > > >> > > component I realized that the testing framework > > for > > > > > OFBiz > > > > > > is > > > > > > > > not > > > > > > > > > > > good. > > > > > > > > > > > >> > You > > > > > > > > > > > >> > > cannot do real test driven development or > > > > > > red-green-refactor > > > > > > > > > with > > > > > > > > > > > the > > > > > > > > > > > >> > > current setup, hence my proposal to change it. I > > > > explain > > > > > > > > below: > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > Problem with current design > > > > > > > > > > > >> > > ---------------------------------------- > > > > > > > > > > > >> > > - What we have right now is not unit tests, it's > > > > really > > > > > > > > > > integration > > > > > > > > > > > >> > tests. > > > > > > > > > > > >> > > You have to start the framework, the database, > the > > > > > service > > > > > > > > > engine, > > > > > > > > > > > the > > > > > > > > > > > >> > > entity engine and pretty much everything. > > > > > > > > > > > >> > > - Testing is very slow, because it's an > > integration > > > > test > > > > > > as > > > > > > > I > > > > > > > > > > > >> mentioned > > > > > > > > > > > >> > > above. 10 minutes on a good computer! > > > > > > > > > > > >> > > - There is zero mocking! We actually have to > > > > --load-data > > > > > > for > > > > > > > > > > things > > > > > > > > > > > to > > > > > > > > > > > >> > > work. Again, these are integration tests. > > > > > > > > > > > >> > > - Too complex: Integration tests by their nature > > are > > > > > > > grabbing > > > > > > > > > too > > > > > > > > > > > >> much. > > > > > > > > > > > >> > > Mind you, I am not objecting to integration > tests > > (I > > > > > > > actually > > > > > > > > > like > > > > > > > > > > > >> them) > > > > > > > > > > > >> > > but I am objecting to not having real > unit-tests. > > > Unit > > > > > > tests > > > > > > > > > > should > > > > > > > > > > > >> all > > > > > > > > > > > >> > run > > > > > > > > > > > >> > > in a few seconds. > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > Proposed solution > > > > > > > > > > > >> > > -------------------------- > > > > > > > > > > > >> > > - We keep what is considered real integration > > tests > > > > the > > > > > > way > > > > > > > > they > > > > > > > > > > are > > > > > > > > > > > >> > right > > > > > > > > > > > >> > > now and keep using them > > > > > > > > > > > >> > > - We move what should be unit tests into simple > > > JUnit > > > > > > > classes, > > > > > > > > > and > > > > > > > > > > > we > > > > > > > > > > > >> do > > > > > > > > > > > >> > > not run them using java -jar ofbiz.jar --test, > but > > > > > instead > > > > > > > run > > > > > > > > > > them > > > > > > > > > > > >> > > directly from the build.xml script, so these > files > > > are > > > > > not > > > > > > > > > > > identified > > > > > > > > > > > >> in > > > > > > > > > > > >> > > any XML document, but are simply called > > immediately > > > > from > > > > > > the > > > > > > > > > build > > > > > > > > > > > >> > scripts. > > > > > > > > > > > >> > > - We clearly mark the difference between > > integration > > > > > tests > > > > > > > and > > > > > > > > > > unit > > > > > > > > > > > >> tests > > > > > > > > > > > >> > > (inside the source files or in the suite > > > > declarations). > > > > > > > > > > > >> > > - We change the run-tests target in build.xml to > > run > > > > > both > > > > > > > unit > > > > > > > > > > tests > > > > > > > > > > > >> and > > > > > > > > > > > >> > > integration tests. > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > I intend to heavily refactor the framework and I > > > would > > > > > > feel > > > > > > > > > better > > > > > > > > > > > >> about > > > > > > > > > > > >> > > introducing this change while refactoring. What > do > > > you > > > > > > guys > > > > > > > > > think? > > > > > > > > > > > >> Ideas? > > > > > > > > > > > >> > > Suggestions? Approvals and thumbs up? > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > Regards, > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > Taher Alkhateeb > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
