Hi Scott, Thank you for the feedback. To be focused exactly on integration vs unit, I already mentioned above that at least for me the main objective is to confidently and quickly run the tests in short bursts of red-green refactor. This allows me to refactor code without waiting in front of the screen in between test cycles thus giving me immediate feedback on any errors I made. Perhaps my intro was too long so this is the squeezed out version of it.
I already had one round of testing in the start component which was much faster that way and had an immediate impact. Oh and by the way, you cannot test the start component with integration tests for example unless you do it from an external component which cannot access package protected items. This style of coding is applicable I think to any software project inclusive of OFBiz. Maybe in certain components more than others or certain areas more than others. But I can't see how it could be bad or unwanted. Taher Alkhateeb On Jul 19, 2016 8:18 AM, "Scott Gray" <scott.g...@hotwaxsystems.com> wrote: > Thanks Ron and Taher for your responses, I appreciate them but I don't hear > much in the conversation that speaks to the value of unit tests over > integration tests. Most of the thoughts shared speaks more to the value of > tests rather than differences in the type of tests. > > Speed: At an application level we have ~685 tests that run in 35 seconds > (excluding build and data load). Another point is that there isn't much > reason why tests can't be run in parallel rather than sequentially. > TDD: Integration tests in OFBiz are as simple if not simpler to write than > unit tests and just as useful for TDD. > I'm not sure if I missed any other points raised regarding integration vs. > unit tests? > > I'm not looking to start a big long debate on the topic, I just wanted to > speak out that someone out there (me) doesn't think unit tests are the best > solution for testing OFBiz applications. > > Regards > Scott > > On 19 July 2016 at 16:52, Taher Alkhateeb <slidingfilame...@gmail.com> > wrote: > > > Hi Scott, > > > > Thank you for your input. Your ideas are thought provoking and enriching > to > > the conversation. > > > > The way I look at it, the general rule is usually many unit tests, less > > integration tests, lesser functional tests. So we are not excluding any > > types of test, all of them are important in certain areas for certain > > purposes with certain quantity. Usually integration tests are less > because > > as you said they just grab more. I like the picture below as a general > > guideline > > > > http://i.stack.imgur.com/fjQvQ.png > > > > As you already mentioned unit tests are useful for the framework. I > > discovered errors in code that I wrote which I was very very careful > with. > > I immediately learned the lesson that humans are not designed to code, > and > > TDD gives you confidence as you build your code with those short 30-60 > > second red-green refactors. I feel much much safer and more confident > > writing code that way, the test also documents how to use the api, > > refactoring and feature change becomes less terrifying. Also messy code > is > > usually not test friendly thus refactoring and unit tests go hand-in-hand > > for improving the code base. > > > > Also, I am sorry to say that the framework code is rather messy and > brittle > > in many places. I think you probably encountered this yourself. The same > is > > said for the applications. Now If we start refactoring without unit tests > > then we are back to scary business. So much can go wrong so fast and > break > > things you never expected to break. The framework with all applications > > require heavy refactoring of things like massive ugly methods, sandwiched > > logic, heavy shared mutable state, hidden dependencies, poor > > interfaces, and much more. Every time I touch something I get shocked at > > how bad it looks, spaghetti logic to no end. If you refactor with > > integration tests instead of unit tests then you will come down to a > > screeching halt as you wait between these test cycles. My computer > actually > > takes 10 minutes or more for a full clean load data and testing. > > > > Now talking about mocking you raise some interesting points, what can you > > mock vs utilizing integration tests? This is an excellent question that > > does not have a simple answer, and you mostly learn as you code, > difficult > > to envision without coding. However simple things like Java services can > be > > mocked with a standard mocking class that you use everywhere. > Transactions > > are difficult to mock and better left to integration tests I think. SECAs > > might be mocked by simply passing output to input as chaining methods. > Mind > > you I am not 100% sure of all of this, coding is the ultimate guide to > what > > can or cannot be done. > > > > So I am not suggesting unit tests as a best-practice (even though it is). > > Instead I suggest it as something that I and others did and got a huge > > psychological relief and confidence and comfort from. Swimming in code > > without tests is a terrifying business, made more terrifying if the code > is > > bad. Short bursts of red/green/red/green makes you feel good as you build > > up your logic. And I don't have an exact vision of how to do it, because > > the details always win. > > > > I look forward to hearing your thoughts and thank you for enriching this > > conversation. > > > > Taher Alkhateeb > > > > On Tuesday, 19 July 2016, Scott Gray <scott.g...@hotwaxsystems.com> > wrote: > > > > > I know I'm late to the party here, but I just want to say that I think > > > integration tests have far greater value to OFBiz than unit tests. > > Mostly > > > because we tend to have quite a low number of tests and integration > tests > > > give us much better coverage per line of test code and the tests are > much > > > closer to the real world scenarios the application might encounter. > > > > > > I don't really see how unit tests could be applied to non-framework > > testing > > > in a useful manner, could you expand on your vision in that regard? I > > mean > > > would we be testing something smaller than a service as the 'unit'? > What > > > would we mock? Would transaction management still be active? What > happens > > > when the service calls another service, I guess we mock the response > from > > > that service (how)? > > > > > > It just seems a very complicated method to achieve a less thorough but > > > albeit faster (maybe) test result. > > > > > > A build, data load and full test run takes 4m 9s on my laptop. > Excluding > > > build, data load and framework tests: the application level tests take > > 35s, > > > not very expensive IMO. The data load time can be reduced to > practically > > > nothing by copying a clean slate database into runtime for each run. > > > > > > I'm mostly just suggesting we be wary of adding complicated testing > > > procedures in the hope of achieving some 'best practice' result which > in > > > reality will provide minimal benefits. > > > > > > Regards > > > Scott > > > > > > On 18 July 2016 at 18:57, Taher Alkhateeb <slidingfilame...@gmail.com > > > <javascript:;>> > > > wrote: > > > > > > > Hello Akash, > > > > > > > > Fantastic, I have a few unit tests almost done to be included in the > > > start > > > > component. I will create a new subtask under OFBIZ-1463 to commit the > > > tests > > > > so you can use them as a reference if you like to. > > > > > > > > I also recommend that you follow the same directory structure between > > the > > > > test code and production code. So for example: > > > > > > > > Production code: > > > framework/start/src/main/java/org/apache/ofbiz/base/start > > > > Test code: framework/start/src/test/java/org/apache/ofbiz/base/start > > > > > > > > The benefit of this hierarchy is that you can access non-public > > (package > > > > protected) methods for testing. This is in fact exactly what I needed > > to > > > be > > > > able to apply some of the tests. > > > > > > > > Cheers, > > > > > > > > Taher Alkhateeb > > > > > > > > On Mon, Jul 18, 2016 at 9:22 AM, Akash Jain < > > > akash.j...@hotwaxsystems.com <javascript:;>> > > > > wrote: > > > > > > > > > Thanks Taher for nice initiative! > > > > > > > > > > We are planning to written unit tests to all components under > > > OFBIZ-1463 > > > > > > > > > > Thanks and Regards > > > > > -- > > > > > Akash Jain > > > > > > > > > > On Mon, Jul 18, 2016 at 10:36 AM, Taher Alkhateeb < > > > > > slidingfilame...@gmail.com <javascript:;>> wrote: > > > > > > > > > > > Hello Everyone, > > > > > > > > > > > > In reference to this thread and the Jira OFBIZ-7254, I'm very > happy > > > to > > > > > > announce that OFBiz is now ready for applying unit tests with > only > > 8 > > > > new > > > > > > lines of code in the build script (r1753143) :) > > > > > > > > > > > > I recommend we do the following moving forward: > > > > > > > > > > > > 1- Introduce unit tests as much as we can to all components > > > > > > 2- Migrate most of the integration tests we currently have to > unit > > > > tests > > > > > > (since they are designed to do very little integration). > > > > > > > > > > > > This is a great chance for us to practice real TDD in OFBiz > > > > > > > > > > > > Cheers, > > > > > > > > > > > > Taher Alkhateeb > > > > > > > > > > > > On Mon, Jun 13, 2016 at 7:43 AM, Pranay Pandey < > > > > > > pranay.pan...@hotwaxsystems.com <javascript:;>> wrote: > > > > > > > > > > > > > +1 > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > > > > Pranay Pandey > > > > > > > HotWax Systems > > > > > > > http://www.hotwaxsystems.com/ > > > > > > > > > > > > > > On Fri, Jun 10, 2016 at 7:46 PM, Taher Alkhateeb < > > > > > > > slidingfilame...@gmail.com <javascript:;> > > > > > > > > wrote: > > > > > > > > > > > > > > > Hello Everyone, > > > > > > > > > > > > > > > > I was able to get a few tests running and this is very > doable. > > > But > > > > I > > > > > > > faced > > > > > > > > a big problem in designing the testing framework because of > > ANT. > > > > > > > > > > > > > > > > The problem > > > > > > > > ---------------- > > > > > > > > The way the build scripts are designed in OFBiz are very > > > complex. A > > > > > > > master > > > > > > > > file calls other files which call other files. And in the > > middle > > > > you > > > > > > have > > > > > > > > external libraries (ant-contrib) and macros, and variables, > and > > > > class > > > > > > > path > > > > > > > > declarations, and and and .... > > > > > > > > > > > > > > > > I cannot declare the tests programmatically (with JUnit test > > > > suites) > > > > > > > > because this means lower level components would depend on > > higher > > > > > level > > > > > > > > components. So I have to do it in ANT, by navigating this > maze > > of > > > > > build > > > > > > > > scripts, and it was a headache for me just to read them, let > > > alone > > > > > > modify > > > > > > > > them to create a testing framework. > > > > > > > > > > > > > > > > Suggested Solution > > > > > > > > ------------------------ > > > > > > > > I suggest to implement the testing framework in Gradle, and > > > simply > > > > > call > > > > > > > it > > > > > > > > from within ant. This is a middle solution that sustains ant > > for > > > > now, > > > > > > but > > > > > > > > can allow us to switch out later. > > > > > > > > > > > > > > > > This means I will just add one more file called build.gradle > in > > > the > > > > > top > > > > > > > > level directory, and figure out the business logic for > calling > > > the > > > > > test > > > > > > > > suites from that file > > > > > > > > > > > > > > > > I look forward to your feedback. > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > > > > Taher Alkhateeb > > > > > > > > > > > > > > > > On Wed, Jun 8, 2016 at 6:00 PM, Taher Alkhateeb < > > > > > > > > slidingfilame...@gmail.com <javascript:;>> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Everyone, > > > > > > > > > > > > > > > > > > Thank you all for your support, JIRA created in > > > > > > > > > https://issues.apache.org/jira/browse/OFBIZ-7254 > > > > > > > > > > > > > > > > > > I will start working on it and try to implement ASAP to get > > my > > > > > focus > > > > > > > back > > > > > > > > > on refactoring. > > > > > > > > > > > > > > > > > > Cheers! > > > > > > > > > > > > > > > > > > Taher Alkhateeb > > > > > > > > > > > > > > > > > > On Wed, Jun 8, 2016 at 4:58 PM, Deepak Dixit < > > > > > > > > > deepak.di...@hotwaxsystems.com <javascript:;>> wrote: > > > > > > > > > > > > > > > > > >> +1 > > > > > > > > >> > > > > > > > > >> Thanks & Regards > > > > > > > > >> -- > > > > > > > > >> Deepak Dixit > > > > > > > > >> www.hotwaxsystems.com > > > > > > > > >> > > > > > > > > >> On Wed, Jun 8, 2016 at 7:12 PM, Mridul Pathak < > > > > > > > > >> mridul.pat...@hotwaxsystems.com <javascript:;>> wrote: > > > > > > > > >> > > > > > > > > >> > +1 > > > > > > > > >> > > > > > > > > > >> > Makes perfect sense. > > > > > > > > >> > > > > > > > > > >> > -- > > > > > > > > >> > Thanks & Regards, > > > > > > > > >> > Mridul Pathak > > > > > > > > >> > Senior Manager > > > > > > > > >> > HotWax Systems > > > > > > > > >> > http://www.hotwaxsystems.com > > > > > > > > >> > > > > > > > > > >> > > On Jun 8, 2016, at 2:41 PM, Taher Alkhateeb < > > > > > > > > >> slidingfilame...@gmail.com <javascript:;>> > > > > > > > > >> > wrote: > > > > > > > > >> > > > > > > > > > > >> > > Hello Everyone, > > > > > > > > >> > > > > > > > > > > >> > > After refactoring the start component and while > starting > > > on > > > > > the > > > > > > > base > > > > > > > > >> > > component I realized that the testing framework for > > OFBiz > > > is > > > > > not > > > > > > > > good. > > > > > > > > >> > You > > > > > > > > >> > > cannot do real test driven development or > > > red-green-refactor > > > > > > with > > > > > > > > the > > > > > > > > >> > > current setup, hence my proposal to change it. I > explain > > > > > below: > > > > > > > > >> > > > > > > > > > > >> > > Problem with current design > > > > > > > > >> > > ---------------------------------------- > > > > > > > > >> > > - What we have right now is not unit tests, it's > really > > > > > > > integration > > > > > > > > >> > tests. > > > > > > > > >> > > You have to start the framework, the database, the > > service > > > > > > engine, > > > > > > > > the > > > > > > > > >> > > entity engine and pretty much everything. > > > > > > > > >> > > - Testing is very slow, because it's an integration > test > > > as > > > > I > > > > > > > > >> mentioned > > > > > > > > >> > > above. 10 minutes on a good computer! > > > > > > > > >> > > - There is zero mocking! We actually have to > --load-data > > > for > > > > > > > things > > > > > > > > to > > > > > > > > >> > > work. Again, these are integration tests. > > > > > > > > >> > > - Too complex: Integration tests by their nature are > > > > grabbing > > > > > > too > > > > > > > > >> much. > > > > > > > > >> > > Mind you, I am not objecting to integration tests (I > > > > actually > > > > > > like > > > > > > > > >> them) > > > > > > > > >> > > but I am objecting to not having real unit-tests. Unit > > > tests > > > > > > > should > > > > > > > > >> all > > > > > > > > >> > run > > > > > > > > >> > > in a few seconds. > > > > > > > > >> > > > > > > > > > > >> > > Proposed solution > > > > > > > > >> > > -------------------------- > > > > > > > > >> > > - We keep what is considered real integration tests > the > > > way > > > > > they > > > > > > > are > > > > > > > > >> > right > > > > > > > > >> > > now and keep using them > > > > > > > > >> > > - We move what should be unit tests into simple JUnit > > > > classes, > > > > > > and > > > > > > > > we > > > > > > > > >> do > > > > > > > > >> > > not run them using java -jar ofbiz.jar --test, but > > instead > > > > run > > > > > > > them > > > > > > > > >> > > directly from the build.xml script, so these files are > > not > > > > > > > > identified > > > > > > > > >> in > > > > > > > > >> > > any XML document, but are simply called immediately > from > > > the > > > > > > build > > > > > > > > >> > scripts. > > > > > > > > >> > > - We clearly mark the difference between integration > > tests > > > > and > > > > > > > unit > > > > > > > > >> tests > > > > > > > > >> > > (inside the source files or in the suite > declarations). > > > > > > > > >> > > - We change the run-tests target in build.xml to run > > both > > > > unit > > > > > > > tests > > > > > > > > >> and > > > > > > > > >> > > integration tests. > > > > > > > > >> > > > > > > > > > > >> > > I intend to heavily refactor the framework and I would > > > feel > > > > > > better > > > > > > > > >> about > > > > > > > > >> > > introducing this change while refactoring. What do you > > > guys > > > > > > think? > > > > > > > > >> Ideas? > > > > > > > > >> > > Suggestions? Approvals and thumbs up? > > > > > > > > >> > > > > > > > > > > >> > > Regards, > > > > > > > > >> > > > > > > > > > > >> > > Taher Alkhateeb > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
