Hi Shawn Thanks very much for your feedback. Please see my comments inline below
Am 03.12.14 um 00:10 schrieb Shawn Heisey: > On 12/2/2014 3:31 PM, Michael Wechner wrote: >> I am trying to setup a testing environment, which is running tests >> incrementally based on changed files, instead of running whole builds. >> >> I would like to try this for Lucene/Solr, hence I am trying to >> understand better the relationship between individual code changes and >> associated tests. For example I have noticed to the following change: >> >> git diff --name-only 641ed06..cb4fc89 >> lucene/core/src/java/org/apache/lucene/codecs/compressing/CompressingStoredFieldsFormat.java >> >> and if I understand correctly, then the following tests would have to be >> executed: >> >> lucene/core/src/test/org/apache/lucene/codecs/compressing/TestCompressingStoredFieldsFormat.java >> lucene/test-framework/src/java/org/apache/lucene/codecs/compressing/CompressingCodec.java >> >> is that correct? > The test dependency tree for this particular class is *EXTENSIVE* and > not very easy to track down. > > CompressingStoredFieldsFormat is the parent class for > Lucene50StoredFieldsFormat. It is directly referenced by three classes > in addition to the direct descendant. Here's a screenshot showing all > direct references in branch_5x: > > https://www.dropbox.com/s/3pb4kljvdbd6xug/CompressingStoredFieldsFormat-eclipse.png?dl=0 > > Lucene50StoredFieldsFormat is used by a few classes, one of which is > Lucene50Codec: > > https://www.dropbox.com/s/0ibnw87m6rveinv/Lucene50StoredFieldsFormat.png?dl=0 > > Lucene50Codec is extremely central to almost everything the unreleased > Lucene 5.x does if it is using default settings, so a very large number > of tests would be affected by a change in the compressed stored field > format. Just for giggles, I grabbed a screenshot of the classes that > directly reference Lucene50Codec: > > https://www.dropbox.com/s/eqq30uvckc8b19d/Lucene50Codec.png?dl=0 thank you very much for your insights. I did not realize that this particular case has such extensive dependencies > > The build system includes the ability to use Clover if it is present ... > I don't know much about it, except that it's related to determining test > coverage for parts of the software. Perhaps a clover report could give > you the info you need? > > https://www.atlassian.com/software/clover/overview I have heard about Clover, but never used it. I will have a closer look at it now > > Aside from that ... running the entire test suite multiple times > whenever you change anything is the best option. The test system > randomizes a large number of options on each test run, so running the > tests multiple times will test many different combinations. A very > large number of bugs have been discovered and fixed because of that > randomization. I think you are right that in certain cases, like for example in the case of CompressingStoredFieldsFormat it makes a lot of sense to run the entire test suit (even multiple times considering randomization), but I would argue that there are probably quite a few other cases where incremental testing would be sufficient, at least as a first shot. I am saying this, because I think for larger projects running entire test suits do not scale well and that it is rather unefficient and inconvenient for developers to run the entire test suite every time, before they commit/push a tiny change. Since I am not familiar enough with the code of Lucene/Solr I cannot say whether all code is covered by tests, but I would assume it is not. Also it would be interesting to know whether developers do manual testing before they commit/push or whether they just rely on running the entire test suit. I guess they are doing manual tests as well, but probably only test things which they think are related to the changes which they made. Hence I think it would help developers to have a system which supports incremental testing. I really do not mean to say that it should replace the entire suite testing approach, but it would be additionally. I will think about it some more :-) Thanks Michael > > Thanks, > Shawn > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
