Re: Shouldn't InputStream/Files::readAllBytes throw something other than OutOfMemoryError?
On 03/13/2017 12:33 PM, Chris Hegarty wrote: Anthony, Many of the Collection types throw OOME if requested to grow greater than ~2GB. Likewise some operations of String and StringBuilder. Though this behavior is not strictly part of the current specification, I suspect that it is the defacto standard ( since the implementation has always behaved this way ). The java.lang.module.ModuleReader::read method is another method that specifies the behavior if the returned type is not capable of supporting very large amounts of data. I agree that the use of OOME here is somewhat overloaded, but it appears that we already well down this path, best to make it clear and consistent in the spec. What about (talking about JDK10 of course) creating OutOfMemoryError subclasses to cover cases that don't pertain to Java heap memory? Regards, Peter -Chris. On 12/03/17 14:24, Anthony Vanelverdinghe wrote: Files::readAllBytes is specified to throw an OutOfMemoryError "if an array of the required size cannot be allocated, for example the file is larger that 2G". Now in Java 9, InputStream::readAllBytes does the same. However, this overloads the meaning of OutOfMemoryError: either "the JVM is out of memory" or "the resultant array would require long-based indices". In my opinion, this overloading is problematic, because: - OutOfMemoryError has very clear semantics, and I don't see the link between OOME and the fact that a resultant byte[] would need to be >2G. If I have 5G of free heap space, and try to read a 3G file, I'd expect something like an UnsupportedOperationException, but definitely not an OutOfMemoryError. - the former meaning is an actual Error, whereas the latter is an Exception from which the application can recover. - developers might be tempted to catch the OOME and retry to read the file/input stream in chunks, no matter the cause of the OOME. What was the rationale for using OutOfMemory here? And would it still be possible to change this before Rampdown Phase 2? Kind regards, Anthony
Re: Shouldn't InputStream/Files::readAllBytes throw something other than OutOfMemoryError?
Anthony, Many of the Collection types throw OOME if requested to grow greater than ~2GB. Likewise some operations of String and StringBuilder. Though this behavior is not strictly part of the current specification, I suspect that it is the defacto standard ( since the implementation has always behaved this way ). The java.lang.module.ModuleReader::read method is another method that specifies the behavior if the returned type is not capable of supporting very large amounts of data. I agree that the use of OOME here is somewhat overloaded, but it appears that we already well down this path, best to make it clear and consistent in the spec. -Chris. On 12/03/17 14:24, Anthony Vanelverdinghe wrote: Files::readAllBytes is specified to throw an OutOfMemoryError "if an array of the required size cannot be allocated, for example the file is larger that 2G". Now in Java 9, InputStream::readAllBytes does the same. However, this overloads the meaning of OutOfMemoryError: either "the JVM is out of memory" or "the resultant array would require long-based indices". In my opinion, this overloading is problematic, because: - OutOfMemoryError has very clear semantics, and I don't see the link between OOME and the fact that a resultant byte[] would need to be >2G. If I have 5G of free heap space, and try to read a 3G file, I'd expect something like an UnsupportedOperationException, but definitely not an OutOfMemoryError. - the former meaning is an actual Error, whereas the latter is an Exception from which the application can recover. - developers might be tempted to catch the OOME and retry to read the file/input stream in chunks, no matter the cause of the OOME. What was the rationale for using OutOfMemory here? And would it still be possible to change this before Rampdown Phase 2? Kind regards, Anthony
Re: Shouldn't InputStream/Files::readAllBytes throw something other than OutOfMemoryError?
Hey Anthony, Fair point though. Guess it might be just for consistency, with existing behavior. Anyhow your explanation makes sense, let’s hope it’ll be a temporary limitation as we’re seeing Arrays 2.0 on the horizon and most most most probably will offer 64bit array indexes :-) Chris > On 12. Mar 2017, at 18:31, Anthony Vanelverdinghe > wrote: > > Hi Chris > > Point well taken, but being unable to create a native thread is definitely a > VirtualMachineError, and personally I don't care whether the JVM throws an > OOME or any other kind of VirtualMachineError in that case. > > My point was that I don't see how being unable to return a result because of > a language limitation (i.e. arrays being indexed by integers, and thus > limited to 2G for byte[]), has anything to do with OutOfMemoryError. I > believe it would be much more logical to throw a recoverable RuntimeException > in this case (e.g. java.lang.ArrayOverflowException, as an analog of > java.nio.BufferOverflowException). > > Anthony > > On 12/03/2017 15:53, Christoph Engelbert wrote: >> Hey Anthony, >> >> The meaning is already overloaded, as "Cannot create native thread" >> is also an OutOfMemoryError and in like 99% of the cases means >> "Linux ran out of filehandles". The chance the OS really couldn't >> allocate a thread for the reason of no main memory available is very >> narrow :) >> >> Chris >> >> Am 3/12/2017 um 3:24 PM schrieb Anthony Vanelverdinghe: >>> Files::readAllBytes is specified to throw an OutOfMemoryError "if >>> an array of the required size cannot be allocated, for example the >>> file is larger that 2G". Now in Java 9, InputStream::readAllBytes >>> does the same. >>> >>> However, this overloads the meaning of OutOfMemoryError: either >>> "the JVM is out of memory" or "the resultant array would require >>> long-based indices". >>> >>> In my opinion, this overloading is problematic, because: >>> - OutOfMemoryError has very clear semantics, and I don't see the >>> link between OOME and the fact that a resultant byte[] would need >>> to be >2G. If I have 5G of free heap space, and try to read a 3G >>> file, I'd expect something like an UnsupportedOperationException, >>> but definitely not an OutOfMemoryError. >>> - the former meaning is an actual Error, whereas the latter is an >>> Exception from which the application can recover. >>> - developers might be tempted to catch the OOME and retry to read >>> the file/input stream in chunks, no matter the cause of the OOME. >>> >>> What was the rationale for using OutOfMemory here? And would it >>> still be possible to change this before Rampdown Phase 2? >>> >>> Kind regards, >>> Anthony >>> >>> >
Re: Shouldn't InputStream/Files::readAllBytes throw something other than OutOfMemoryError?
Hi Chris Point well taken, but being unable to create a native thread is definitely a VirtualMachineError, and personally I don't care whether the JVM throws an OOME or any other kind of VirtualMachineError in that case. My point was that I don't see how being unable to return a result because of a language limitation (i.e. arrays being indexed by integers, and thus limited to 2G for byte[]), has anything to do with OutOfMemoryError. I believe it would be much more logical to throw a recoverable RuntimeException in this case (e.g. java.lang.ArrayOverflowException, as an analog of java.nio.BufferOverflowException). Anthony On 12/03/2017 15:53, Christoph Engelbert wrote: Hey Anthony, The meaning is already overloaded, as "Cannot create native thread" is also an OutOfMemoryError and in like 99% of the cases means "Linux ran out of filehandles". The chance the OS really couldn't allocate a thread for the reason of no main memory available is very narrow :) Chris Am 3/12/2017 um 3:24 PM schrieb Anthony Vanelverdinghe: Files::readAllBytes is specified to throw an OutOfMemoryError "if an array of the required size cannot be allocated, for example the file is larger that 2G". Now in Java 9, InputStream::readAllBytes does the same. However, this overloads the meaning of OutOfMemoryError: either "the JVM is out of memory" or "the resultant array would require long-based indices". In my opinion, this overloading is problematic, because: - OutOfMemoryError has very clear semantics, and I don't see the link between OOME and the fact that a resultant byte[] would need to be >2G. If I have 5G of free heap space, and try to read a 3G file, I'd expect something like an UnsupportedOperationException, but definitely not an OutOfMemoryError. - the former meaning is an actual Error, whereas the latter is an Exception from which the application can recover. - developers might be tempted to catch the OOME and retry to read the file/input stream in chunks, no matter the cause of the OOME. What was the rationale for using OutOfMemory here? And would it still be possible to change this before Rampdown Phase 2? Kind regards, Anthony
Re: Shouldn't InputStream/Files::readAllBytes throw something other than OutOfMemoryError?
Hey Anthony, The meaning is already overloaded, as "Cannot create native thread" is also an OutOfMemoryError and in like 99% of the cases means "Linux ran out of filehandles". The chance the OS really couldn't allocate a thread for the reason of no main memory available is very narrow :) Chris Am 3/12/2017 um 3:24 PM schrieb Anthony Vanelverdinghe: > Files::readAllBytes is specified to throw an OutOfMemoryError "if > an array of the required size cannot be allocated, for example the > file is larger that 2G". Now in Java 9, InputStream::readAllBytes > does the same. > > However, this overloads the meaning of OutOfMemoryError: either > "the JVM is out of memory" or "the resultant array would require > long-based indices". > > In my opinion, this overloading is problematic, because: > - OutOfMemoryError has very clear semantics, and I don't see the > link between OOME and the fact that a resultant byte[] would need > to be >2G. If I have 5G of free heap space, and try to read a 3G > file, I'd expect something like an UnsupportedOperationException, > but definitely not an OutOfMemoryError. > - the former meaning is an actual Error, whereas the latter is an > Exception from which the application can recover. > - developers might be tempted to catch the OOME and retry to read > the file/input stream in chunks, no matter the cause of the OOME. > > What was the rationale for using OutOfMemory here? And would it > still be possible to change this before Rampdown Phase 2? > > Kind regards, > Anthony > >
