RE: svn commit: r1141510 - /lucene/dev/trunk/modules/facet/src/java/org/apache/lucene/util/UnsafeByteArrayOutputStream.java

2011-07-01 Thread Uwe Schindler
Hi, I don't understand the whole discussion here, so please compare these two implementations and tell me which one is faster. Please don't hurt me, if you don't want to see src.jar code from OpenJDK Java6 - just delete this mail if you don’t want to (the code here is licensed under GPL):

Re: svn commit: r1141510 - /lucene/dev/trunk/modules/facet/src/java/org/apache/lucene/util/UnsafeByteArrayOutputStream.java

2011-07-01 Thread Michael McCandless
On Fri, Jul 1, 2011 at 2:33 AM, Uwe Schindler u...@thetaphi.de wrote: Hi, I don't understand the whole discussion here, so please compare these two implementations and tell me which one is faster. Please don't hurt me, if you don't want to see src.jar code from OpenJDK Java6 - just delete

Re: svn commit: r1141510 - /lucene/dev/trunk/modules/facet/src/java/org/apache/lucene/util/UnsafeByteArrayOutputStream.java

2011-07-01 Thread Shai Erera
About the encoders package - there are several encoders there besides VInt, so I wouldn't dispose of it so quickly. That said, I think we should definitely explore consolidating VInt with the core classes, and maybe write an encoder which delegate to them. Or, come up w/ a different approach for

Re: svn commit: r1141510 - /lucene/dev/trunk/modules/facet/src/java/org/apache/lucene/util/UnsafeByteArrayOutputStream.java

2011-06-30 Thread Simon Willnauer
hmm are you concerned about the extra Math.min that happens in the copyOf method? I don't how that relates to intrinsic and java 1.7 I don't have strong feelings here just checking if you mix something up in the comment you put there... I am happy to keep the old and now current code simon On

RE: svn commit: r1141510 - /lucene/dev/trunk/modules/facet/src/java/org/apache/lucene/util/UnsafeByteArrayOutputStream.java

2011-06-30 Thread Uwe Schindler
Hi Robert, you reverted a use of Arrays.copyOf() on native types which is *exactly* implemented like this in Arrays.java code! The slow ones are T T[] copyOf(T[] array, int newlen) (because they use reflection). Uwe - Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen

Re: svn commit: r1141510 - /lucene/dev/trunk/modules/facet/src/java/org/apache/lucene/util/UnsafeByteArrayOutputStream.java

2011-06-30 Thread Robert Muir
because on windows 32bit at least, -client is still the default on most jres out there. i realize people don't care about -client, but i will police foo[].clone() / arrays.copyOf etc to prevent problems. There are comments about this stuff on the relevant bug reports (oracle's site is down,

Re: svn commit: r1141510 - /lucene/dev/trunk/modules/facet/src/java/org/apache/lucene/util/UnsafeByteArrayOutputStream.java

2011-06-30 Thread Simon Willnauer
Robert I agree but doesn't that apply to Arrays.copyOf(Object[],int) only? here we use a specialized primitive version? simon On Thu, Jun 30, 2011 at 3:05 PM, Robert Muir rcm...@gmail.com wrote: because on windows 32bit at least, -client is still the default on most jres out there. i realize

Re: svn commit: r1141510 - /lucene/dev/trunk/modules/facet/src/java/org/apache/lucene/util/UnsafeByteArrayOutputStream.java

2011-06-30 Thread Dawid Weiss
Arrays.copyOf(primitive) is actually System.arraycopy by default. If intrinsics are used it can only get faster. For object types it will probably be a bit slower for -client because of a runtime check for the component type. Dawid On Thu, Jun 30, 2011 at 3:05 PM, Robert Muir rcm...@gmail.com

RE: svn commit: r1141510 - /lucene/dev/trunk/modules/facet/src/java/org/apache/lucene/util/UnsafeByteArrayOutputStream.java

2011-06-30 Thread Uwe Schindler
Robert, as noted in my other eMail, ist only slow for the generic Object[] method (as it uses j.l.reflect.Array.newInstance(Class componentType)). We are talking here about byte[], and the Arrays method is implemented with the same 3 lines of code, Simon replaced. The only difference is a

RE: svn commit: r1141510 - /lucene/dev/trunk/modules/facet/src/java/org/apache/lucene/util/UnsafeByteArrayOutputStream.java

2011-06-30 Thread Uwe Schindler
We had an issue about this with FST's array growing in Mike's code, in facts ist *much* slower for generic Arrays' T[] copyOf(T[]...), with T extends Object (uses slow reflection). For primitives it can only get faster in later JVMs, this is why we want to change all ArrayUtils.grow() to use

Re: svn commit: r1141510 - /lucene/dev/trunk/modules/facet/src/java/org/apache/lucene/util/UnsafeByteArrayOutputStream.java

2011-06-30 Thread Simon Willnauer
On Thu, Jun 30, 2011 at 3:26 PM, Uwe Schindler u...@thetaphi.de wrote: We had an issue about this with FST's array growing in Mike's code, in facts ist *much* slower for generic Arrays' T[] copyOf(T[]...), with T extends Object (uses slow reflection). For primitives it can only get faster

Re: svn commit: r1141510 - /lucene/dev/trunk/modules/facet/src/java/org/apache/lucene/util/UnsafeByteArrayOutputStream.java

2011-06-30 Thread Robert Muir
I think Dawid is correct here... so we should change this back? still personally when I see array clone() or copyOf() it makes me concerned, I know these are as fast as arraycopy in recent versions, but depending on which variant is used, and whether you use -server, can be slower... in general I

Re: svn commit: r1141510 - /lucene/dev/trunk/modules/facet/src/java/org/apache/lucene/util/UnsafeByteArrayOutputStream.java

2011-06-30 Thread Simon Willnauer
On Thu, Jun 30, 2011 at 4:44 PM, Robert Muir rcm...@gmail.com wrote: I think Dawid is correct here... so we should change this back? still personally when I see array clone() or copyOf() it makes me concerned, I know these are as fast as arraycopy in recent versions, but depending on which

Re: svn commit: r1141510 - /lucene/dev/trunk/modules/facet/src/java/org/apache/lucene/util/UnsafeByteArrayOutputStream.java

2011-06-30 Thread Dawid Weiss
I don't seen any evidence that this is any slower though. You need to run with -client (if the machine is a beast this is tricky because x64 will pick -server regardless of the command-line setting) and you need to be copying generic arrays. I think this can be shown -- a caliper benchmark would

Re: svn commit: r1141510 - /lucene/dev/trunk/modules/facet/src/java/org/apache/lucene/util/UnsafeByteArrayOutputStream.java

2011-06-30 Thread Michael McCandless
I think it's important Lucene keeps good performance on ordinary machines/envs. It's really quite dangerous that the active Lucene devs all use beasts for development/testing. We draw false conclusions. So we really should be testing with -client and if indeed generified Arrays.copyOf (and

Re: svn commit: r1141510 - /lucene/dev/trunk/modules/facet/src/java/org/apache/lucene/util/UnsafeByteArrayOutputStream.java

2011-06-30 Thread Dawid Weiss
I think it's important Lucene keeps good performance on ordinary machines/envs. Not that this voice will help in anything, but I think the above is virtually impossible to achieve unless you have a bunch of machines, OSs and VMs to continually test on and a consistent set of benchmarks plotted

Re: svn commit: r1141510 - /lucene/dev/trunk/modules/facet/src/java/org/apache/lucene/util/UnsafeByteArrayOutputStream.java

2011-06-30 Thread Michael McCandless
Fair enough, and I agree. Though the least we could do is rotate in a Windows env, where Java runs with -client, to our Jenkins. But simple-to-follow rules like Don't use Arrays.copyOf; use System.arraycopy instead (if indeed System.arraycopy seems to generally not be slower) seem like a

Re: svn commit: r1141510 - /lucene/dev/trunk/modules/facet/src/java/org/apache/lucene/util/UnsafeByteArrayOutputStream.java

2011-06-30 Thread Simon Willnauer
On Thu, Jun 30, 2011 at 8:50 PM, Dawid Weiss dawid.we...@cs.put.poznan.pl wrote: I don't seen any evidence that this is any slower though. You need to run with -client (if the machine is a beast this is tricky because x64 will pick -server regardless of the command-line setting) and you need

Re: svn commit: r1141510 - /lucene/dev/trunk/modules/facet/src/java/org/apache/lucene/util/UnsafeByteArrayOutputStream.java

2011-06-30 Thread Michael McCandless
On Thu, Jun 30, 2011 at 4:45 PM, Simon Willnauer simon.willna...@googlemail.com wrote: On Thu, Jun 30, 2011 at 8:50 PM, Dawid Weiss dawid.we...@cs.put.poznan.pl wrote: I don't seen any evidence that this is any slower though. You need to run with -client (if the machine is a beast this is

Re: svn commit: r1141510 - /lucene/dev/trunk/modules/facet/src/java/org/apache/lucene/util/UnsafeByteArrayOutputStream.java

2011-06-30 Thread Simon Willnauer
On Fri, Jul 1, 2011 at 12:19 AM, Michael McCandless luc...@mikemccandless.com wrote: On Thu, Jun 30, 2011 at 4:45 PM, Simon Willnauer simon.willna...@googlemail.com wrote: On Thu, Jun 30, 2011 at 8:50 PM, Dawid Weiss dawid.we...@cs.put.poznan.pl wrote: I don't seen any evidence that this is