> On 9 Apr 2018, at 15:34, Thomas Mortagne <[email protected]> wrote: > > On Mon, Apr 9, 2018 at 1:38 PM, Vincent Massol <[email protected]> wrote: >> Hi Thomas, >> >>> On 6 Apr 2018, at 11:57, Thomas Mortagne <[email protected]> wrote: >>> >>> Not sure what to think of those examples. >>> >>> The theory sounds nice and it probably produce nice result when >>> talking about one method with a few inputs and an output but for >>> anything more complex (like anything involving components) I feel you >>> end up with a generated test which is a lot harder to read than simply >>> looking at the code. >> >> As I said in my original mail, I think that really depends about who handles >> the issue (a long time committer who’s being working for years on the code >> and knows it very well) or someone new who doesn’t know the code well. >> >> Also, it really depends on the context. Imagine the following example:you >> have a NPE stack trace in production and the line where the NPE is has >> several variables that can be null. Just looking at the code will not tell >> you what the problem is. But the test will and you can even put a breakpoint >> to know which variable is actually null. >> >>> * https://jira.xwiki.org/browse/XWIKI-13031: does not produce the same >>> error at all. Looks like evocash just try to produce any >>> ClassCastException but does not care about the actual types in it >>> which is quite useless (plus reading that test requires a lot of >>> effort just to conclude that the type of the field in the document is >>> not String…) >> >> I’ve not tested them all but all the ones I’ve tested do reproduce correctly >> the stack trace (that’s the point of evocrash so there’s no way it’ll >> generate a test that doesn’t reproduce the stack trace! ;)). >> >> To give you more info, you run evocrash by giving it a stacktrace and a >> frame number (the number of stack trace lines you want it to reproduce). >> >> For ex, on: >> >> Caused by: java.lang.ClassCastException: org.apache.lucene.document.Field >> cannot be cast to java.lang.String >> at >> org.xwiki.search.solr.internal.reference.SolrEntityReferenceResolver.getWikiReference(SolrEntityReferenceResolver.java:93) >> ~[xwiki-platform-search-solr-api-7.4.jar:na] >> at >> org.xwiki.search.solr.internal.reference.SolrEntityReferenceResolver.getEntityReference(SolrEntityReferenceResolver.java:70) >> ~[xwiki-platform-search-solr-api-7.4.jar:na] >> at >> org.xwiki.search.solr.internal.reference.SolrEntityReferenceResolver.resolve(SolrEntityReferenceResolver.java:63) >> ~[xwiki-platform-search-solr-api-7.4.jar:na] >> at >> org.xwiki.search.solr.internal.reference.SolrEntityReferenceResolver.resolve(SolrEntityReferenceResolver.java:46) >> ~[xwiki-platform-search-solr-api-7.4.jar:na] >> at >> org.xwiki.search.solr.internal.reference.SolrDocumentReferenceResolver.resolve(SolrDocumentReferenceResolver.java:48) >> ~[xwiki-platform-search-solr-api-7.4.jar:na] >> at >> org.xwiki.search.solr.internal.reference.SolrDocumentReferenceResolver.resolve(SolrDocumentReferenceResolver.java:38) >> ~[xwiki-platform-search-solr-api-7.4.jar:na] >> […] >> >> If you run evocrash with frame = 5, it’ll generate a test that, when >> executed lead to: >> >> Caused by: java.lang.ClassCastException: org.apache.lucene.document.Field >> cannot be cast to java.lang.String >> at >> org.xwiki.search.solr.internal.reference.SolrEntityReferenceResolver.getWikiReference(SolrEntityReferenceResolver.java:93) >> ~[xwiki-platform-search-solr-api-7.4.jar:na] >> at >> org.xwiki.search.solr.internal.reference.SolrEntityReferenceResolver.getEntityReference(SolrEntityReferenceResolver.java:70) >> ~[xwiki-platform-search-solr-api-7.4.jar:na] >> at >> org.xwiki.search.solr.internal.reference.SolrEntityReferenceResolver.resolve(SolrEntityReferenceResolver.java:63) >> ~[xwiki-platform-search-solr-api-7.4.jar:na] >> at >> org.xwiki.search.solr.internal.reference.SolrEntityReferenceResolver.resolve(SolrEntityReferenceResolver.java:46) >> ~[xwiki-platform-search-solr-api-7.4.jar:na] >> at >> org.xwiki.search.solr.internal.reference.SolrDocumentReferenceResolver.resolve(SolrDocumentReferenceResolver.java:48) >> ~[xwiki-platform-search-solr-api-7.4.jar:na] > > If you execute the test indicated in > https://jira.xwiki.org/browse/XWIKI-13031?focusedCommentId=98082&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-98082 > you will get the following exception: > > java.lang.ClassCastException: > org.xwiki.model.internal.reference.ExplicitStringEntityReferenceResolver > cannot be cast to java.lang.String > > Sure you get the same stack trace from Exception type and methods > calls point of view but what the message describe is a very different > use case.
As I said evocrash replicates exactly the same stack trace, including the message. If you get something different it’s either because you’re executing on different code or whoever generated the test made a mistake copying it. Thanks -Vincent > >> >>> * https://jira.xwiki.org/browse/XWIKI-14475: totally unreadable >> >> Actually this test was not that hard to read for me. It just sets up some >> mocks. Evocrash can make it even simpler by presenting the mocks in a >> simpler way but that’s cosmetic and not the focus of evocrash just right now >> (it’s not the hard part). I’ve also raised something a bit related: >> https://github.com/STAMP-project/EvoCrash/issues/7 >> >> Feel free to raise new issues at >> https://github.com/STAMP-project/EvoCrash/issues/ >> >>> * https://jira.xwiki.org/browse/XRENDERING-422: not really useful but >>> I doubt evocrash can do much about this kind of issue >> >> Yes I agree. >> >>> * https://jira.xwiki.org/browse/XWIKI-13196: this one could help a bit >>> I guess but not sure it's really faster to execute evocash than >>> looking at the code :) >> >> Sure but it also give you the test which is nice. And depending on the line >> generating the NPE, it’s not always easy to know which variable is null by >> looking at the code. >> >>> * https://jira.xwiki.org/browse/XWIKI-13916: that's a more complex way >>> than it should to express that we call addTextAreaField for an already >>> existing field yes it's useful >> >> Thanks for the feedback. >> >> There are plenty of limitations/issues with evocrash ATM. It’s good if we >> can identify cases where it generates stuff that could be useful. One >> direction that the devs are pursuing is looking at existing test classes to >> mimick how they’re written to produce the new tests. I don’t know how they’d >> be able to pull this off (that seems really hard to me ;)) but I know >> they’re working in this direction. >> >> Thanks >> -Vincent >> >>> >>> On Fri, Apr 6, 2018 at 10:44 AM, Vincent Massol <[email protected]> wrote: >>>> Hi devs, >>>> >>>> Context >>>> ====== >>>> >>>> Evocrash is a tool developed as part of the STAMP research project >>>> (https://www.stamp-project.eu/). Its goal is to take a stack trace (the >>>> use case is production stack traces) and generate a test that, when >>>> executed, reproduces the stack trace ;) >>>> >>>> It’s using Guided Genetic Algorithm to simplify the search space (which is >>>> pretty cool, isn’t it? ;)). More info at >>>> https://github.com/STAMP-project/EvoCrash and http://www.evocrash.org/ >>>> >>>> Results >>>> ====== >>>> >>>> So far evocrash was able to generate tests for the following issues (I’ve >>>> pasted the generated test code as comment): >>>> * https://jira.xwiki.org/browse/XRENDERING-422 >>>> * https://jira.xwiki.org/browse/XWIKI-14475 >>>> * https://jira.xwiki.org/browse/XWIKI-13031 >>>> * https://jira.xwiki.org/browse/XWIKI-13196 >>>> * https://jira.xwiki.org/browse/XWIKI-13916 >>>> >>>> Using >>>> ===== >>>> >>>> The use case I see is the following: >>>> * A user reports a problem when using XWiki and create a jira issue with a >>>> stack trace >>>> * An XWiki dev finds it and runs evocrash to generate a test reproducing >>>> the problem >>>> * The XWiki dev uses the test to understand the reason (breakpoints can be >>>> put in the IDE) of the problem and writes a better test (which can be >>>> inspired by the test generated by evocrash or be completely different). >>>> >>>> Questions >>>> ======== >>>> >>>> The question we need to answer is whether we think it coud be useful or >>>> not. Are there cases where it can be useful? >>>> >>>> For example if we imagine someone not knowing the xwiki code base well, >>>> maybe this can help him/her understand the bug in a simpler way than just >>>> having to read/find where the problem is in the code base? >>>> >>>> I’ve started the work of trying to evaluate how useful it could be on the >>>> XWiki project by analyzing the examples above. I've only analyzed 2 ATM >>>> (see the links in the jira comments) and I’d be interested to have your >>>> analysis too. >>>> >>>> WDYT? >>>> >>>> Thanks >>>> -Vincent >>> >>> >>> >>> -- >>> Thomas Mortagne >> > > > > -- > Thomas Mortagne
