Re: benchmarks (was Re: Progress on a Classpath mauve suite?)
Chris Pickett wrote: Chris Pickett wrote: Perhaps a replacement for DaCapo and SPECjvm98 is not in order: this would make comparison with other papers in the literature difficult and just confuse the issue. So, how about a good *companion* suite then? By the way, Ashes was started as a free testing framework for Java benchmarks, so if we do anything, it should be examined closely as a starting point, in my opinion. It needs a bit of cleaning up, but at least the overhead of building a testing framework has mostly been thought about and taken care of. Of course Mauve or something related may be suitable/better for these purposes. Sorry, I looked on our homepage, and this link isn't obviously there (the software isn't exactly maintained at this point): http://www.sable.mcgill.ca/~bdufou1/ashes2/ I promise to stop spamming the CP list now... Chris ___ Classpath mailing list Classpath@gnu.org http://lists.gnu.org/mailman/listinfo/classpath
Re: benchmarks (was Re: Progress on a Classpath mauve suite?)
Chris Pickett wrote: Perhaps a replacement for DaCapo and SPECjvm98 is not in order: this would make comparison with other papers in the literature difficult and just confuse the issue. So, how about a good *companion* suite then? By the way, Ashes was started as a free testing framework for Java benchmarks, so if we do anything, it should be examined closely as a starting point, in my opinion. It needs a bit of cleaning up, but at least the overhead of building a testing framework has mostly been thought about and taken care of. Of course Mauve or something related may be suitable/better for these purposes. OK, I really have to go and run SPECjvm98 now, so I might drop out of this conversation somewhat. Chris ___ Classpath mailing list Classpath@gnu.org http://lists.gnu.org/mailman/listinfo/classpath
Re: benchmarks (was Re: Progress on a Classpath mauve suite?)
David P Grove wrote: I think you need to understand the point of the benchmark suite. The whole goal is reproducible science, so if someone doesn't cite the exact version of the benchmarks, then it isn't useful (in an academic sense). The license forces that plus proper academic credit (ie a citation) for the benchmark suite, which personally I think is quite fair given how much work was put into putting it together (much more than the typical academic paper). Actually I have the same opinion: we need rigourous benchmark suites in academics. I'm not diametrically opposed to using non-free ones, and I do understand the value (as I said, I use a non-free suite all the time in my work, so politics in practice don't stop me from using non-free software (although it's nice to avoid it)). Still: why can't there be high quality free Java suites for GNU Classpath VM's to test against? Furthermore, why can't academics use and distribute free software? It would seem to me that they're particularly in a good position to do so. I know some of the people who put together the benchmark suite fairly well and it was a massive effort (think person-years, not person-weeks; good, usable, portable, benchmark suites are a lot of work). It would be a shame if your politics prevented you from using it, but that's life I suppose. I don't mean to start a political tirade here, but if you look at all the free VM's and all of GNU Classpath, that's required a *much* larger effort than putting together one benchmark suite (composed of free or at least public domain software at that). So I don't really see why it should be a problem for interested parties to accomplish, although it obviously won't happen overnight. Perhaps a replacement for DaCapo and SPECjvm98 is not in order: this would make comparison with other papers in the literature difficult and just confuse the issue. So, how about a good *companion* suite then? Chris ___ Classpath mailing list Classpath@gnu.org http://lists.gnu.org/mailman/listinfo/classpath
Re: benchmarks (was Re: Progress on a Classpath mauve suite?)
David P Grove wrote: I think you need to understand the point of the benchmark suite. The whole goal is reproducible science, so if someone doesn't cite the exact version of the benchmarks, then it isn't useful (in an academic sense). That can be achieved without the DaCapo restrictions. For example TeX has fairly strict requirements on naming and versioning. But DaCapo license is extremely restrictive. It's not clear that a company could use DaCapo to benchmark their products - even open-source products, since that might be defined as "for-profit". Forbidding one to "alter, modify, improve, decompile, disassemble or otherwise reverse-engineer" has nothing to do with "reproducible science", and in fact is contrary to acadmic openness and "reproducible sciece". The license forces that plus proper academic credit (ie a citation) for the benchmark suite, which personally I think is quite fair given how much work was put into putting it together (much more than the typical academic paper). A requirement for credit isn't inherently incompatible with Free Software: The old BSD license with the "notorious advertising clause" may have been awkward and GPL-incompatible, but it was accepted as Free. I know some of the people who put together the benchmark suite fairly well and it was a massive effort (think person-years, not person-weeks; good, usable, portable, benchmark suites are a lot of work). It would be a shame if your politics prevented you from using it, but that's life I suppose. You're rather missing the point, methinks. I know some of the people at Redmond who've spent person-decades to put together quite a nice operating system. I also know some people in Santa Clara who've spent person-decades putting together a very nice Java implementation. It's a shame that politics are preventing us from using either. -- --Per Bothner [EMAIL PROTECTED] http://per.bothner.com/ ___ Classpath mailing list Classpath@gnu.org http://lists.gnu.org/mailman/listinfo/classpath
Re: benchmarks (was Re: Progress on a Classpath mauve suite?)
I think you need to understand the point of the benchmark suite. The whole goal is reproducible science, so if someone doesn't cite the exact version of the benchmarks, then it isn't useful (in an academic sense). The license forces that plus proper academic credit (ie a citation) for the benchmark suite, which personally I think is quite fair given how much work was put into putting it together (much more than the typical academic paper). I know some of the people who put together the benchmark suite fairly well and it was a massive effort (think person-years, not person-weeks; good, usable, portable, benchmark suites are a lot of work). It would be a shame if your politics prevented you from using it, but that's life I suppose. --dave___ Classpath mailing list Classpath@gnu.org http://lists.gnu.org/mailman/listinfo/classpath
benchmarks (was Re: Progress on a Classpath mauve suite?)
Hi David, That's a non-free license. http://osl-www.cs.umass.edu/DaCapo/gcbm.html#License_and_copyright This is generally what's accepted to define free software: http://www.gnu.org/philosophy/free-sw.html Maybe we need a replacement for DaCapo as well. Thanks for the suggestion though, it's always nice to know about new benchmark suites from an academic perspective (I still use SPECjvm98 actively, so I may as well look into DaCapo at some point). It's somewhat surprising, I think at least some of those component programs are actually free (this may be the case for SPEC too, but I'm not sure). We also have an AspectJ benchmark suite at: http://www.sable.mcgill.ca/benchmarks/ but I don't know if any free VM is actually capable of running this code (still, that means it's a good suite, right?). SableVM users certainly had problems with AspectJ in the optimizing compiler class I'm the TA for. I'd be interested in hearing from other VM developers if they can run it. Then there's Ashes, from Sable as well, but it's somewhat older. This should be freely redistributable. There's the JOlden set of benchmarks, often used as a good small example for harder problems like pointer analysis, but there doesn't appear to be any license for them, i.e. just copyright applies (we may try to get that cleared up though). Then there's the Java Grande Forum Suite, but I don't think it's free either (still, it can break our VM at the commandline so it's a good test). Cheers, Chris David P Grove wrote: It depends on what your definition of free is, but you night want to checkout the DaCapo benchmarks (http://osl-www.cs.umass.edu/DaCapo/gcbm.html). The main obligation of the license is that if you use the benchmarks you need to cite them correctly and report the version number of the benchmark suite. These also have the advantage of almost all running on Jikes RVM + classpath (the exceptions are due to incomplete AWT support in classpath). --dave ___ Classpath mailing list Classpath@gnu.org http://lists.gnu.org/mailman/listinfo/classpath ___ Classpath mailing list Classpath@gnu.org http://lists.gnu.org/mailman/listinfo/classpath
Re: Progress on a Classpath mauve suite?
It depends on what your definition of free is, but you night want to checkout the DaCapo benchmarks (http://osl-www.cs.umass.edu/DaCapo/gcbm.html). The main obligation of the license is that if you use the benchmarks you need to cite them correctly and report the version number of the benchmark suite. These also have the advantage of almost all running on Jikes RVM + classpath (the exceptions are due to incomplete AWT support in classpath). --dave ___ Classpath mailing list Classpath@gnu.org http://lists.gnu.org/mailman/listinfo/classpath
Re: Progress on a Classpath mauve suite?
Chris Pickett wrote: I think this kind of automated testing/regression testing for Classpath VM's is a great idea. s/automated testing/automated compatability/ If it's possible, some performance numbers would be interesting so we find discrepancies between there VM's there as well. s/between there VM's/between VM's/ Lack of sleep and increasing brain loss seems to have this weird effect on the quality of my writing. Chris ___ Classpath mailing list Classpath@gnu.org http://lists.gnu.org/mailman/listinfo/classpath
Re: Progress on a Classpath mauve suite?
Roman Kennke wrote: Am Dienstag, den 08.03.2005, 10:34 -0600 schrieb Archie Cobbs: Roman Kennke wrote: Ah, now I'm understanding. I think maybe you mean the mauve HTML pages that I once put together? This basically tested different runtimes against the Mauve suite and formatted to Mauve output nicely into HTML for displaying in a browser, so it is easily visible which tests fails (red) and which don't (green). Is that what you mean? If yes, then I can tell you that the nessecary scripts are not online ATM due to lack of resources on my side, but probably soon will be available again on developer.c.o. At least I hope so... Something like that :-) Ok, these are good suggestions for improving my mauve scripts. If we're going to make these HTML pages the "official" place for VM implementors to look for this information, that's OK. I'd just like us to decide and then do it, keep them updated, etc. This can perhaps be automated, etc. They actually are (or were) automated. Mauve has been run in a daily cron job and the rendering of the results has been dynamic (on-demand). A comparison of the failures/successes between the VMs should not be that difficult to add. I think this kind of automated testing/regression testing for Classpath VM's is a great idea. If it's possible, some performance numbers would be interesting so we find discrepancies between there VM's there as well. Having a VM that passes all passing Mauve tests (and fails all failing ones, although it seems highly unlikely that tests that are meant to fail might "accidentally" pass) could allow you to declare, "100% compatible with GNU Classpath v. 0.14," which I think is a cool kind of certification. We discussed on IRC the need for a free benchmark suite (in particular, a free equivalent to SPECjvm98), and this is also something that one might envision built into a cross-VM testing framework. Cheers, Chris ___ Classpath mailing list Classpath@gnu.org http://lists.gnu.org/mailman/listinfo/classpath
Re: Progress on a Classpath mauve suite?
On Tue, Mar 08, 2005 at 05:07:08PM +0100, Roman Kennke wrote: > Am Dienstag, den 08.03.2005, 09:11 -0600 schrieb Archie Cobbs: > > Roman Kennke wrote: > > >>http://lists.gnu.org/archive/html/classpath/2004-12/threads.html#00137 > > >> > > >>we talked about putting together somehow a list of Mauve tests that > > >>all Classpath-based VMs should expect to pass, along with another > > >>(complementary) list of those tests a Classpath-based VM should > > >>expect to fail. > > > > > > Sorry if I am completely naive here, but why should there be tests, that > > > a Classpath-based VM should be expected to fail?? Isn't the whole point > > > to (ideally) pass all tests? If there would be such a test, I would > > > rather rewrite the harness.check(cond) into harness.check(!cond) so that > > > the (expected) failure passes... > > > > Ideally there should be no expected failures. In reality there always > > are, simply because some feature is unimplemented or some bug has not > > been tracked down yet. Obviously, any expected failures are good > > starting points for anyone looking to improve Classpath... another > > benefit of having (or being able to easily generate) such a list. > > Ah, now I'm understanding. I think maybe you mean the mauve HTML pages > that I once put together? This basically tested different runtimes > against the Mauve suite and formatted to Mauve output nicely into HTML > for displaying in a browser, so it is easily visible which tests fails > (red) and which don't (green). At the FOSDEM meeting I brought up this issue since I have been working on a new mauve-runner (based on the work from David Gilbert) which indeed generates this kind of output and is written in such a way that a more structured output of a mauve run is also possible, allowing a 3rth package to cross-reference different runs. Where each run may be on a different JVM, for example. When the processing-power becomes available (developer.classpath.org is probably best) we intend to move this there and put online the results in various easy to parse ways. See also this thread (on the mauve mailinglist) http://sources.redhat.com/ml/mauve-discuss/2005-q1/msg00020.html -- Thomas Zander ___ Classpath mailing list Classpath@gnu.org http://lists.gnu.org/mailman/listinfo/classpath
UrlConnection and images
It seems that UrlConnection.getContent() for image/gif should return an
ImageProducer object. I guess doing something like...
package gnu.java.net.content.image;
import java.net.ContentHandler;
import javax.imageio.ImageIO;
public class gif extends ContentHandler{
public Object getContent(URLConnection urlc) throws IOException{
return ImageIO.read(urlc.getInputStream()).getSource();
}
}
...might work. I guess there is more to it than this? It seems too
simple to be missing. Are the ImageIO classes working? I'm trying to use
Apache FOP and it fails because of this. I tried to compile classpath
myself but I havent been able to. Can anyone check it out?
___
Classpath mailing list
Classpath@gnu.org
http://lists.gnu.org/mailman/listinfo/classpath
Re: Progress on a Classpath mauve suite?
Roman Kennke wrote: If we're going to make these HTML pages the "official" place for VM implementors to look for this information, that's OK. I'd just like us to decide and then do it, keep them updated, etc. This can perhaps be automated, etc. They actually are (or were) automated. Mauve has been run in a daily cron job and the rendering of the results has been dynamic (on-demand). A comparison of the failures/successes between the VMs should not be that difficult to add. Great! Looking forward to seeing your stuff back up on the web. -Archie __ Archie Cobbs *CTO, Awarix* http://www.awarix.com ___ Classpath mailing list Classpath@gnu.org http://lists.gnu.org/mailman/listinfo/classpath
Re: Progress on a Classpath mauve suite?
Am Dienstag, den 08.03.2005, 10:34 -0600 schrieb Archie Cobbs: > Roman Kennke wrote: > > Ah, now I'm understanding. I think maybe you mean the mauve HTML pages > > that I once put together? This basically tested different runtimes > > against the Mauve suite and formatted to Mauve output nicely into HTML > > for displaying in a browser, so it is easily visible which tests fails > > (red) and which don't (green). Is that what you mean? If yes, then I can > > tell you that the nessecary scripts are not online ATM due to lack of > > resources on my side, but probably soon will be available again on > > developer.c.o. At least I hope so... > > Something like that :-) Ok, these are good suggestions for improving my mauve scripts. > If we're going to make these HTML pages the "official" place for > VM implementors to look for this information, that's OK. I'd just > like us to decide and then do it, keep them updated, etc. This can > perhaps be automated, etc. They actually are (or were) automated. Mauve has been run in a daily cron job and the rendering of the results has been dynamic (on-demand). A comparison of the failures/successes between the VMs should not be that difficult to add. /Roman signature.asc Description: Dies ist ein digital signierter Nachrichtenteil ___ Classpath mailing list Classpath@gnu.org http://lists.gnu.org/mailman/listinfo/classpath
Re: Progress on a Classpath mauve suite?
Roman Kennke wrote: Ah, now I'm understanding. I think maybe you mean the mauve HTML pages that I once put together? This basically tested different runtimes against the Mauve suite and formatted to Mauve output nicely into HTML for displaying in a browser, so it is easily visible which tests fails (red) and which don't (green). Is that what you mean? If yes, then I can tell you that the nessecary scripts are not online ATM due to lack of resources on my side, but probably soon will be available again on developer.c.o. At least I hope so... Something like that :-) Basically, here's my problem. When I run Mauve against my VM, there are some test failures. Some are caused by Classpath and some are caused by my VM. Right now I don't know which is which. But if I did then that information would be very valuable, because it would save me a lot of time in tracking down the problem. I'm sure all the other VM implementors face the same situation. Sometimes I'm working on my particular VM and want to fix its bugs. Other times I'm working on Classpath and want to fix its bugs. So it's nice to know which bugs are which. This information does exist, we just haven't compiled it and made it easily accessible. Your HTML pages are a nice step in that direction. They still don't completely tell us whether a test failure is because of Classpath itself or not, i.e., it could be that all the tested VM's exhibit the same problem, but presumably that's pretty rare, so not a big deal. If we're going to make these HTML pages the "official" place for VM implementors to look for this information, that's OK. I'd just like us to decide and then do it, keep them updated, etc. This can perhaps be automated, etc. -Archie __ Archie Cobbs *CTO, Awarix* http://www.awarix.com ___ Classpath mailing list Classpath@gnu.org http://lists.gnu.org/mailman/listinfo/classpath
Re: Progress on a Classpath mauve suite?
Am Dienstag, den 08.03.2005, 09:11 -0600 schrieb Archie Cobbs: > Roman Kennke wrote: > >>http://lists.gnu.org/archive/html/classpath/2004-12/threads.html#00137 > >> > >>we talked about putting together somehow a list of Mauve tests that > >>all Classpath-based VMs should expect to pass, along with another > >>(complementary) list of those tests a Classpath-based VM should > >>expect to fail. > > > > Sorry if I am completely naive here, but why should there be tests, that > > a Classpath-based VM should be expected to fail?? Isn't the whole point > > to (ideally) pass all tests? If there would be such a test, I would > > rather rewrite the harness.check(cond) into harness.check(!cond) so that > > the (expected) failure passes... > > Ideally there should be no expected failures. In reality there always > are, simply because some feature is unimplemented or some bug has not > been tracked down yet. Obviously, any expected failures are good > starting points for anyone looking to improve Classpath... another > benefit of having (or being able to easily generate) such a list. Ah, now I'm understanding. I think maybe you mean the mauve HTML pages that I once put together? This basically tested different runtimes against the Mauve suite and formatted to Mauve output nicely into HTML for displaying in a browser, so it is easily visible which tests fails (red) and which don't (green). Is that what you mean? If yes, then I can tell you that the nessecary scripts are not online ATM due to lack of resources on my side, but probably soon will be available again on developer.c.o. At least I hope so... /Roman signature.asc Description: Dies ist ein digital signierter Nachrichtenteil ___ Classpath mailing list Classpath@gnu.org http://lists.gnu.org/mailman/listinfo/classpath
Re: Progress on a Classpath mauve suite?
Roman Kennke wrote: http://lists.gnu.org/archive/html/classpath/2004-12/threads.html#00137 we talked about putting together somehow a list of Mauve tests that all Classpath-based VMs should expect to pass, along with another (complementary) list of those tests a Classpath-based VM should expect to fail. Sorry if I am completely naive here, but why should there be tests, that a Classpath-based VM should be expected to fail?? Isn't the whole point to (ideally) pass all tests? If there would be such a test, I would rather rewrite the harness.check(cond) into harness.check(!cond) so that the (expected) failure passes... Ideally there should be no expected failures. In reality there always are, simply because some feature is unimplemented or some bug has not been tracked down yet. Obviously, any expected failures are good starting points for anyone looking to improve Classpath... another benefit of having (or being able to easily generate) such a list. You wouldn't want to change the test, because the test is written to the spec, not Classpath's (or any VM)'s particualr implementation. I.e., Mauve itself is independent of Classpath or any particular VM. It is supposed to represent what "run anywhere" actually looks like. -Archie __ Archie Cobbs *CTO, Awarix* http://www.awarix.com ___ Classpath mailing list Classpath@gnu.org http://lists.gnu.org/mailman/listinfo/classpath
Re: apidocs not showing "throws Throwable"
Hi Robert, Robert Schuster wrote: while wading throught our fine apidoc on developer.classpath.org/doc I wondered why "java.lang.reflect.InvocationHandler.invoke()" does not show that the method can throw a Throwable. At first I thought it is a bug in our API implementation but a short glimpse at the source code revealed that the method is properly declared. Nice catch! The thrown exception immediately preceding the closing semicolon of an interface method declaration was ignored. This is fixed in CVS now. Thanks, Julian ___ Classpath mailing list Classpath@gnu.org http://lists.gnu.org/mailman/listinfo/classpath
apidocs not showing "throws Throwable"
Hi, while wading throught our fine apidoc on developer.classpath.org/doc I wondered why "java.lang.reflect.InvocationHandler.invoke()" does not show that the method can throw a Throwable. At first I thought it is a bug in our API implementation but a short glimpse at the source code revealed that the method is properly declared. Is this a problem in gjdoc or is there another reason why this is not shown? cu Robert ___ Classpath mailing list Classpath@gnu.org http://lists.gnu.org/mailman/listinfo/classpath
Re: Progress on a Classpath mauve suite?
Am Montag, den 07.03.2005, 21:49 -0600 schrieb Archie Cobbs: > In this thread: > > http://lists.gnu.org/archive/html/classpath/2004-12/threads.html#00137 > > we talked about putting together somehow a list of Mauve tests that > all Classpath-based VMs should expect to pass, along with another > (complementary) list of those tests a Classpath-based VM should > expect to fail. Sorry if I am completely naive here, but why should there be tests, that a Classpath-based VM should be expected to fail?? Isn't the whole point to (ideally) pass all tests? If there would be such a test, I would rather rewrite the harness.check(cond) into harness.check(!cond) so that the (expected) failure passes... /Roman signature.asc Description: Dies ist ein digital signierter Nachrichtenteil ___ Classpath mailing list Classpath@gnu.org http://lists.gnu.org/mailman/listinfo/classpath
