[jira] [Comment Edited] (LUCENE-4332) Integrate PiTest mutation coverage tool into build
[
https://issues.apache.org/jira/browse/LUCENE-4332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13448527#comment-13448527
]
Uwe Schindler edited comment on LUCENE-4332 at 9/5/12 5:58 PM:
---
{code}
permission java.security.SecurityPermission *, read,write;
{code}
This makes no sense, as SecurityPermission has no action, so read,write
should be ignored. I was restricting SecurityPermission with something in mind
(see the last 2 lines that allowed only the BouncyCastle installed by TIKA -
now everything is allowed). What fails if I remove that line? I have no time to
run the whole pitest suite
bq. rather than as a full force prevention of malicious code
The idea was to find places (especially in TIKA) that do things they should not
do (like enabling security providers), which makes the configuration of J2EE
container hosting Solr hard. So we should limit all this, to see when somebody
adds a new feature to Solr that needs additional permissions.
I am already working on restricting RuntimePermission more, so only things
like reflection and property access is allowed.
was (Author: thetaphi):
{code}
permission java.security.SecurityPermission *, read,write;
{code}
This makes no sense, as SecurityPermission has no action, so read,write
should be ignored. I was restricting SecurityPermission with something in mind
(see the last 2 lines that allowed only the BouncyCastle installed by TIKA -
now everything is allowed). What fails if I remove that line? I have no time to
run the whole pitest suite
bq. rather than as a full force prevention of malicious code
The idea was to find places (especially in TIKA) that do things they should not
do (like enabling security providers), which makes the configuration of J2EE
container hosting Solr hard. So we should limit all this, to see when somebody
adds a new feature to Solr that needs additional permissions.
I already working on restricting RuntimePermission at all, so only things
like reflection and property access is allowed.
Integrate PiTest mutation coverage tool into build
--
Key: LUCENE-4332
URL: https://issues.apache.org/jira/browse/LUCENE-4332
Project: Lucene - Core
Issue Type: Improvement
Affects Versions: 4.1, 5.0
Reporter: Greg Bowyer
Assignee: Greg Bowyer
Labels: build
Attachments:
LUCENE-4332-Integrate-PiTest-mutation-coverage-tool-into-build.patch,
LUCENE-4332-Integrate-PiTest-mutation-coverage-tool-into-build.patch,
LUCENE-4332-Integrate-PiTest-mutation-coverage-tool-into-build.patch,
LUCENE-4332-Integrate-PiTest-mutation-coverage-tool-into-build.patch
As discussed briefly on the mailing list, this patch is an attempt to
integrate the PiTest mutation coverage tool into the lucene build
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-4332) Integrate PiTest mutation coverage tool into build
[
https://issues.apache.org/jira/browse/LUCENE-4332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13444739#comment-13444739
]
Uwe Schindler edited comment on LUCENE-4332 at 8/30/12 5:58 PM:
bq. Will this require people to put rhino in ant's lib/ folder? If so, I'd just
use an ugly property value=-D..=${} -D..=${}... Yours is very nice but an
additional step is required which seems an overkill?
We already have a lot of script tasks in our build files (e.g. generating
website using ). Javascript is shipped with every known JVM. - And pitest is
only run by few people, much more call tasks like root/lucene$ ant
documentation or root$ ant svn-check-working-copy, so this is no issue here.
was (Author: thetaphi):
bq. Will this require people to put rhino in ant's lib/ folder? If so, I'd
just use an ugly property value=-D..=${} -D..=${}... Yours is very nice but
an additional step is required which seems an overkill?
We already have a lot of script tasks in our build files (e.g. generating
website using ). Javascript is shipped with every known JVM.
Integrate PiTest mutation coverage tool into build
--
Key: LUCENE-4332
URL: https://issues.apache.org/jira/browse/LUCENE-4332
Project: Lucene - Core
Issue Type: Improvement
Affects Versions: 4.1, 5.0
Reporter: Greg Bowyer
Assignee: Greg Bowyer
Labels: build
Attachments:
LUCENE-4332-Integrate-PiTest-mutation-coverage-tool-into-build.patch,
LUCENE-4332-Integrate-PiTest-mutation-coverage-tool-into-build.patch,
LUCENE-4332-Integrate-PiTest-mutation-coverage-tool-into-build.patch
As discussed briefly on the mailing list, this patch is an attempt to
integrate the PiTest mutation coverage tool into the lucene build
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-4332) Integrate PiTest mutation coverage tool into build
[
https://issues.apache.org/jira/browse/LUCENE-4332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13443304#comment-13443304
]
Greg Bowyer edited comment on LUCENE-4332 at 8/29/12 4:51 AM:
--
Wow lot of interest .
I will try to answer some of the salient points
Core was missing until today as one test (TestLuceneConstantVersion) didn't run
correctly as it was lacking the Lucene version system property. Currently pit
refuses to run unless the underlying suite is all green (a good thing IMHO) so
I didn't have core from my first run (its there now).
This takes a long time to run, all of the ancillary Lucene packages take
roughly 4 hours to run on the largest CPU ec2 instance, core takes 8 hours
(this was the other reason core was missing, I was waiting for it to finish
crunching)
As to the random seed, I completely agree and it was one of the things I
mentioned on the mailing list that makes the output of this tool not perfect. I
do feel that the tests that are randomised typically do a better job at gaining
coverage, but its a good idea to stabilise the seed.
Jars and build.xml, I have no problems changing this to whatever people think
fits best into the build. My impression was that clover is handled the way it
is because it is not technically opensource and as a result has screwball
licensing concerns, essentially I didn't know any better :S I will try to get a
chance to make it use the ivy:cachepath approach.
Regarding the risks posed by mutations, I cannot prove or say there are no
risks; however mutation testing is not random in the mutations applied, they
are formulaic and quite simple. It will not permute arguments nor will it
mutate complex objects (it can and does mess with object references turning
references in arguments to nulls). I can conceive of ways in which it could
screwup mutated code making it possible to delete random files but I don't
think they are going to be extremely likely situations. FWIW I would be less
worried about this deleting something on the filesystem and far more worried
about it accidentally leaving corpses of undeleted files.
Sandboxing it could solve that issue, if that is too much effort another
approach might be to work with the pitest team and build a security manager
that is militant about file access, disallowing anything that canonicalises
outside of a given path.
Oh and as Robert suggested we can always point it away from key things.
At the end of the day its a tool like any other, I have exactly the same
feelings as Robert on this
{quote}
This is cool. I'd say lets get it up and going on jenkins (even weekly
or something). why worry about the imperfections in any of these
coverage tools, whats way more important is when the results find
situations where you thought you were testing something, but really
arent, etc (here was a recent one found by clover
http://svn.apache.org/viewvc?rev=1376722view=rev).
so imo just another tool to be able to identify serious gaps/test-bugs
after things are up and running. and especially looking at deltas from
line coverage to identify stuff thats 'executing' but not actually
being tested.
{quote}
was (Author: gbow...@fastmail.co.uk):
Wow lot of interest .
I will try to answer some of the salient points
Core was missing until today as one test (TestLuceneConstantVersion) didn't run
correctly as it was lacking the Lucene version system property. Currently pit
refuses to run unless the underlying suite is all green (a good thing IMHO) so
I didn't have core from my first run (its there now).
This takes a long time to run, all of the ancillary Lucene packages take
roughly 4 hours to run on the largest CPU ec2 instance, core takes 8 hours
(this was the other reason core was missing, I was waiting for it to finish
crunching)
As to the random seed, I completely agree and it was one of the things I
mentioned on the mailing list that makes the output of this tool not perfect. I
do feel that the tests that are randomised typically do a better job at gaining
coverage, but its a good idea to stabilise the seed.
Jars and build.xml, I have no problems changing this to whatever people think
fits best into the build. My impression was that clover is handled the way it
is because it is not technically opensource and as a result has screwball
licensing concerns, essentially I didn't know any better :S I will try to get a
chance to make it use the ivy:cachepath approach.
Regarding the risks posed by mutations, I cannot prove or say there are no
risks; however mutation testing is not random in the mutations applied, they
are formulaic and quite simple. It will not permute arguments nor will it
mutate complex objects (it can and does mess with object references turning
references in arguments to nulls). I can conceive of ways in which it could
[jira] [Comment Edited] (LUCENE-4332) Integrate PiTest mutation coverage tool into build
[
https://issues.apache.org/jira/browse/LUCENE-4332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13443449#comment-13443449
]
Greg Bowyer edited comment on LUCENE-4332 at 8/29/12 6:48 AM:
--
Following up it turns out to be *very* simple to do the security manager trick
{code:java}
import java.io.File;
public class Test {
public static void main(String... args) {
System.setSecurityManager(new SecurityManager() {
public void checkDelete(String file) throws SecurityException {
File fp = new File(file);
String path = fp.getAbsolutePath();
if (!path.startsWith(/tmp)) {
throw new SecurityException(Bang!);
}
}
});
new File(/home/greg/test).delete();
}
}
{code}
{code}
Exception in thread main java.lang.SecurityException: Bang!
at Test$1.checkDelete(Test.java:12)
at java.io.File.delete(File.java:971)
at Test.main(Test.java:17)
{code}
There is a lot of scope here if you want to abuse checking for all sorts of
things (files, sockets, threads etc)
was (Author: gbow...@fastmail.co.uk):
Following up it turns out to be *very* simple to do the security manager
trick
{code:java}
import java.io.File;
public class Test {
public static void main(String... args) {
System.setSecurityManager(new SecurityManager() {
public void checkDelete(String file) throws SecurityException {
File fp = new File(file);
String path = fp.getAbsolutePath();
if (!path.startsWith(/tmp)) {
throw new SecurityException(Bang!);
}
}
});
new File(/home/greg/test).delete();
}
}
{code}
{code}
Exception in thread main java.lang.SecurityException: Bang!
at Test$1.checkDelete(Test.java:12)
at java.io.File.delete(File.java:971)
at Test.main(Test.java:17)
{code}
Integrate PiTest mutation coverage tool into build
--
Key: LUCENE-4332
URL: https://issues.apache.org/jira/browse/LUCENE-4332
Project: Lucene - Core
Issue Type: Improvement
Affects Versions: 4.1, 5.0
Reporter: Greg Bowyer
Assignee: Greg Bowyer
Labels: build
Attachments:
LUCENE-4332-Integrate-PiTest-mutation-coverage-tool-into-build.patch,
LUCENE-4332-Integrate-PiTest-mutation-coverage-tool-into-build.patch
As discussed briefly on the mailing list, this patch is an attempt to
integrate the PiTest mutation coverage tool into the lucene build
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
