Thanks Ron and Taher for your responses, I appreciate them but I don't hear much in the conversation that speaks to the value of unit tests over integration tests. Most of the thoughts shared speaks more to the value of tests rather than differences in the type of tests.
Speed: At an application level we have ~685 tests that run in 35 seconds (excluding build and data load). Another point is that there isn't much reason why tests can't be run in parallel rather than sequentially. TDD: Integration tests in OFBiz are as simple if not simpler to write than unit tests and just as useful for TDD. I'm not sure if I missed any other points raised regarding integration vs. unit tests? I'm not looking to start a big long debate on the topic, I just wanted to speak out that someone out there (me) doesn't think unit tests are the best solution for testing OFBiz applications. Regards Scott On 19 July 2016 at 16:52, Taher Alkhateeb <slidingfilame...@gmail.com> wrote: > Hi Scott, > > Thank you for your input. Your ideas are thought provoking and enriching to > the conversation. > > The way I look at it, the general rule is usually many unit tests, less > integration tests, lesser functional tests. So we are not excluding any > types of test, all of them are important in certain areas for certain > purposes with certain quantity. Usually integration tests are less because > as you said they just grab more. I like the picture below as a general > guideline > > http://i.stack.imgur.com/fjQvQ.png > > As you already mentioned unit tests are useful for the framework. I > discovered errors in code that I wrote which I was very very careful with. > I immediately learned the lesson that humans are not designed to code, and > TDD gives you confidence as you build your code with those short 30-60 > second red-green refactors. I feel much much safer and more confident > writing code that way, the test also documents how to use the api, > refactoring and feature change becomes less terrifying. Also messy code is > usually not test friendly thus refactoring and unit tests go hand-in-hand > for improving the code base. > > Also, I am sorry to say that the framework code is rather messy and brittle > in many places. I think you probably encountered this yourself. The same is > said for the applications. Now If we start refactoring without unit tests > then we are back to scary business. So much can go wrong so fast and break > things you never expected to break. The framework with all applications > require heavy refactoring of things like massive ugly methods, sandwiched > logic, heavy shared mutable state, hidden dependencies, poor > interfaces, and much more. Every time I touch something I get shocked at > how bad it looks, spaghetti logic to no end. If you refactor with > integration tests instead of unit tests then you will come down to a > screeching halt as you wait between these test cycles. My computer actually > takes 10 minutes or more for a full clean load data and testing. > > Now talking about mocking you raise some interesting points, what can you > mock vs utilizing integration tests? This is an excellent question that > does not have a simple answer, and you mostly learn as you code, difficult > to envision without coding. However simple things like Java services can be > mocked with a standard mocking class that you use everywhere. Transactions > are difficult to mock and better left to integration tests I think. SECAs > might be mocked by simply passing output to input as chaining methods. Mind > you I am not 100% sure of all of this, coding is the ultimate guide to what > can or cannot be done. > > So I am not suggesting unit tests as a best-practice (even though it is). > Instead I suggest it as something that I and others did and got a huge > psychological relief and confidence and comfort from. Swimming in code > without tests is a terrifying business, made more terrifying if the code is > bad. Short bursts of red/green/red/green makes you feel good as you build > up your logic. And I don't have an exact vision of how to do it, because > the details always win. > > I look forward to hearing your thoughts and thank you for enriching this > conversation. > > Taher Alkhateeb > > On Tuesday, 19 July 2016, Scott Gray <scott.g...@hotwaxsystems.com> wrote: > > > I know I'm late to the party here, but I just want to say that I think > > integration tests have far greater value to OFBiz than unit tests. > Mostly > > because we tend to have quite a low number of tests and integration tests > > give us much better coverage per line of test code and the tests are much > > closer to the real world scenarios the application might encounter. > > > > I don't really see how unit tests could be applied to non-framework > testing > > in a useful manner, could you expand on your vision in that regard? I > mean > > would we be testing something smaller than a service as the 'unit'? What > > would we mock? Would transaction management still be active? What happens > > when the service calls another service, I guess we mock the response from > > that service (how)? > > > > It just seems a very complicated method to achieve a less thorough but > > albeit faster (maybe) test result. > > > > A build, data load and full test run takes 4m 9s on my laptop. Excluding > > build, data load and framework tests: the application level tests take > 35s, > > not very expensive IMO. The data load time can be reduced to practically > > nothing by copying a clean slate database into runtime for each run. > > > > I'm mostly just suggesting we be wary of adding complicated testing > > procedures in the hope of achieving some 'best practice' result which in > > reality will provide minimal benefits. > > > > Regards > > Scott > > > > On 18 July 2016 at 18:57, Taher Alkhateeb <slidingfilame...@gmail.com > > <javascript:;>> > > wrote: > > > > > Hello Akash, > > > > > > Fantastic, I have a few unit tests almost done to be included in the > > start > > > component. I will create a new subtask under OFBIZ-1463 to commit the > > tests > > > so you can use them as a reference if you like to. > > > > > > I also recommend that you follow the same directory structure between > the > > > test code and production code. So for example: > > > > > > Production code: > > framework/start/src/main/java/org/apache/ofbiz/base/start > > > Test code: framework/start/src/test/java/org/apache/ofbiz/base/start > > > > > > The benefit of this hierarchy is that you can access non-public > (package > > > protected) methods for testing. This is in fact exactly what I needed > to > > be > > > able to apply some of the tests. > > > > > > Cheers, > > > > > > Taher Alkhateeb > > > > > > On Mon, Jul 18, 2016 at 9:22 AM, Akash Jain < > > akash.j...@hotwaxsystems.com <javascript:;>> > > > wrote: > > > > > > > Thanks Taher for nice initiative! > > > > > > > > We are planning to written unit tests to all components under > > OFBIZ-1463 > > > > > > > > Thanks and Regards > > > > -- > > > > Akash Jain > > > > > > > > On Mon, Jul 18, 2016 at 10:36 AM, Taher Alkhateeb < > > > > slidingfilame...@gmail.com <javascript:;>> wrote: > > > > > > > > > Hello Everyone, > > > > > > > > > > In reference to this thread and the Jira OFBIZ-7254, I'm very happy > > to > > > > > announce that OFBiz is now ready for applying unit tests with only > 8 > > > new > > > > > lines of code in the build script (r1753143) :) > > > > > > > > > > I recommend we do the following moving forward: > > > > > > > > > > 1- Introduce unit tests as much as we can to all components > > > > > 2- Migrate most of the integration tests we currently have to unit > > > tests > > > > > (since they are designed to do very little integration). > > > > > > > > > > This is a great chance for us to practice real TDD in OFBiz > > > > > > > > > > Cheers, > > > > > > > > > > Taher Alkhateeb > > > > > > > > > > On Mon, Jun 13, 2016 at 7:43 AM, Pranay Pandey < > > > > > pranay.pan...@hotwaxsystems.com <javascript:;>> wrote: > > > > > > > > > > > +1 > > > > > > > > > > > > Best regards, > > > > > > > > > > > > Pranay Pandey > > > > > > HotWax Systems > > > > > > http://www.hotwaxsystems.com/ > > > > > > > > > > > > On Fri, Jun 10, 2016 at 7:46 PM, Taher Alkhateeb < > > > > > > slidingfilame...@gmail.com <javascript:;> > > > > > > > wrote: > > > > > > > > > > > > > Hello Everyone, > > > > > > > > > > > > > > I was able to get a few tests running and this is very doable. > > But > > > I > > > > > > faced > > > > > > > a big problem in designing the testing framework because of > ANT. > > > > > > > > > > > > > > The problem > > > > > > > ---------------- > > > > > > > The way the build scripts are designed in OFBiz are very > > complex. A > > > > > > master > > > > > > > file calls other files which call other files. And in the > middle > > > you > > > > > have > > > > > > > external libraries (ant-contrib) and macros, and variables, and > > > class > > > > > > path > > > > > > > declarations, and and and .... > > > > > > > > > > > > > > I cannot declare the tests programmatically (with JUnit test > > > suites) > > > > > > > because this means lower level components would depend on > higher > > > > level > > > > > > > components. So I have to do it in ANT, by navigating this maze > of > > > > build > > > > > > > scripts, and it was a headache for me just to read them, let > > alone > > > > > modify > > > > > > > them to create a testing framework. > > > > > > > > > > > > > > Suggested Solution > > > > > > > ------------------------ > > > > > > > I suggest to implement the testing framework in Gradle, and > > simply > > > > call > > > > > > it > > > > > > > from within ant. This is a middle solution that sustains ant > for > > > now, > > > > > but > > > > > > > can allow us to switch out later. > > > > > > > > > > > > > > This means I will just add one more file called build.gradle in > > the > > > > top > > > > > > > level directory, and figure out the business logic for calling > > the > > > > test > > > > > > > suites from that file > > > > > > > > > > > > > > I look forward to your feedback. > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > > Taher Alkhateeb > > > > > > > > > > > > > > On Wed, Jun 8, 2016 at 6:00 PM, Taher Alkhateeb < > > > > > > > slidingfilame...@gmail.com <javascript:;>> > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Everyone, > > > > > > > > > > > > > > > > Thank you all for your support, JIRA created in > > > > > > > > https://issues.apache.org/jira/browse/OFBIZ-7254 > > > > > > > > > > > > > > > > I will start working on it and try to implement ASAP to get > my > > > > focus > > > > > > back > > > > > > > > on refactoring. > > > > > > > > > > > > > > > > Cheers! > > > > > > > > > > > > > > > > Taher Alkhateeb > > > > > > > > > > > > > > > > On Wed, Jun 8, 2016 at 4:58 PM, Deepak Dixit < > > > > > > > > deepak.di...@hotwaxsystems.com <javascript:;>> wrote: > > > > > > > > > > > > > > > >> +1 > > > > > > > >> > > > > > > > >> Thanks & Regards > > > > > > > >> -- > > > > > > > >> Deepak Dixit > > > > > > > >> www.hotwaxsystems.com > > > > > > > >> > > > > > > > >> On Wed, Jun 8, 2016 at 7:12 PM, Mridul Pathak < > > > > > > > >> mridul.pat...@hotwaxsystems.com <javascript:;>> wrote: > > > > > > > >> > > > > > > > >> > +1 > > > > > > > >> > > > > > > > > >> > Makes perfect sense. > > > > > > > >> > > > > > > > > >> > -- > > > > > > > >> > Thanks & Regards, > > > > > > > >> > Mridul Pathak > > > > > > > >> > Senior Manager > > > > > > > >> > HotWax Systems > > > > > > > >> > http://www.hotwaxsystems.com > > > > > > > >> > > > > > > > > >> > > On Jun 8, 2016, at 2:41 PM, Taher Alkhateeb < > > > > > > > >> slidingfilame...@gmail.com <javascript:;>> > > > > > > > >> > wrote: > > > > > > > >> > > > > > > > > > >> > > Hello Everyone, > > > > > > > >> > > > > > > > > > >> > > After refactoring the start component and while starting > > on > > > > the > > > > > > base > > > > > > > >> > > component I realized that the testing framework for > OFBiz > > is > > > > not > > > > > > > good. > > > > > > > >> > You > > > > > > > >> > > cannot do real test driven development or > > red-green-refactor > > > > > with > > > > > > > the > > > > > > > >> > > current setup, hence my proposal to change it. I explain > > > > below: > > > > > > > >> > > > > > > > > > >> > > Problem with current design > > > > > > > >> > > ---------------------------------------- > > > > > > > >> > > - What we have right now is not unit tests, it's really > > > > > > integration > > > > > > > >> > tests. > > > > > > > >> > > You have to start the framework, the database, the > service > > > > > engine, > > > > > > > the > > > > > > > >> > > entity engine and pretty much everything. > > > > > > > >> > > - Testing is very slow, because it's an integration test > > as > > > I > > > > > > > >> mentioned > > > > > > > >> > > above. 10 minutes on a good computer! > > > > > > > >> > > - There is zero mocking! We actually have to --load-data > > for > > > > > > things > > > > > > > to > > > > > > > >> > > work. Again, these are integration tests. > > > > > > > >> > > - Too complex: Integration tests by their nature are > > > grabbing > > > > > too > > > > > > > >> much. > > > > > > > >> > > Mind you, I am not objecting to integration tests (I > > > actually > > > > > like > > > > > > > >> them) > > > > > > > >> > > but I am objecting to not having real unit-tests. Unit > > tests > > > > > > should > > > > > > > >> all > > > > > > > >> > run > > > > > > > >> > > in a few seconds. > > > > > > > >> > > > > > > > > > >> > > Proposed solution > > > > > > > >> > > -------------------------- > > > > > > > >> > > - We keep what is considered real integration tests the > > way > > > > they > > > > > > are > > > > > > > >> > right > > > > > > > >> > > now and keep using them > > > > > > > >> > > - We move what should be unit tests into simple JUnit > > > classes, > > > > > and > > > > > > > we > > > > > > > >> do > > > > > > > >> > > not run them using java -jar ofbiz.jar --test, but > instead > > > run > > > > > > them > > > > > > > >> > > directly from the build.xml script, so these files are > not > > > > > > > identified > > > > > > > >> in > > > > > > > >> > > any XML document, but are simply called immediately from > > the > > > > > build > > > > > > > >> > scripts. > > > > > > > >> > > - We clearly mark the difference between integration > tests > > > and > > > > > > unit > > > > > > > >> tests > > > > > > > >> > > (inside the source files or in the suite declarations). > > > > > > > >> > > - We change the run-tests target in build.xml to run > both > > > unit > > > > > > tests > > > > > > > >> and > > > > > > > >> > > integration tests. > > > > > > > >> > > > > > > > > > >> > > I intend to heavily refactor the framework and I would > > feel > > > > > better > > > > > > > >> about > > > > > > > >> > > introducing this change while refactoring. What do you > > guys > > > > > think? > > > > > > > >> Ideas? > > > > > > > >> > > Suggestions? Approvals and thumbs up? > > > > > > > >> > > > > > > > > > >> > > Regards, > > > > > > > >> > > > > > > > > > >> > > Taher Alkhateeb > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
