[jira] [Work logged] (COMPRESS-584) IOUtils.readRange() can read more from a channel than asked for
[
https://issues.apache.org/jira/browse/COMPRESS-584?focusedWorklogId=633968&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-633968
]
ASF GitHub Bot logged work on COMPRESS-584:
---
Author: ASF GitHub Bot
Created on: 05/Aug/21 06:46
Start Date: 05/Aug/21 06:46
Worklog Time Spent: 10m
Work Description: PeterAlfredLee commented on pull request #214:
URL: https://github.com/apache/commons-compress/pull/214#issuecomment-893209493
This current implemention looks good to me now.
For the "no copy buffer" version, it looks more efficient. But I'm not quite
sure about this. I will try to make some benchmark tests if I got time. And I'd
like to hear ideas from someone else.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 633968)
Time Spent: 4h 10m (was: 4h)
> IOUtils.readRange() can read more from a channel than asked for
> ---
>
> Key: COMPRESS-584
> URL: https://issues.apache.org/jira/browse/COMPRESS-584
> Project: Commons Compress
> Issue Type: Bug
> Components: Archivers
>Affects Versions: 1.21
>Reporter: Matthijs Laan
>Priority: Major
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> When {{IOUtils.readRange(ReadableByteChannel,int)}} gets less than the number
> of bytes asked for in the first read call it does not reduce the buffer size
> for the next read call and may read more than asked for.
> This situation is rare when using a {{FileChannel}} but I wrote a
> {{SeekableByteChannel}} backed by a URI using HTTP range requests and reading
> from a socket can often return less bytes than asked for. When I used this
> channel to read a {{ZipFile}} it only read the ZIP central directory
> partially sometimes because {{IOUtils.readRange()}} called from
> {{ZipFile.readCentralDirectoryEntry()}} read more bytes than asked for and it
> stopped parsing directory entries.
> Fix: [https://github.com/apache/commons-compress/pull/214]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [commons-compress] PeterAlfredLee commented on pull request #214: [COMPRESS-584] Fix IOUtils.readRange() can read more from a channel than asked for
PeterAlfredLee commented on pull request #214: URL: https://github.com/apache/commons-compress/pull/214#issuecomment-893209493 This current implemention looks good to me now. For the "no copy buffer" version, it looks more efficient. But I'm not quite sure about this. I will try to make some benchmark tests if I got time. And I'd like to hear ideas from someone else. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (VFS-807) LocalFile migration to NIO is not done correctly
[
https://issues.apache.org/jira/browse/VFS-807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17393510#comment-17393510
]
Gary D. Gregory commented on VFS-807:
-
Hi [~klv_m72]
Please provide your diff as a PR on GitHub such that the GitHub Actions builds
kick in for all of the OSs and JDKs we check.
> LocalFile migration to NIO is not done correctly
>
>
> Key: VFS-807
> URL: https://issues.apache.org/jira/browse/VFS-807
> Project: Commons VFS
> Issue Type: Bug
>Affects Versions: 2.9.0
>Reporter: L
>Priority: Blocker
> Fix For: 2.9.1
>
> Attachments: TestJdkWriteFile.java, vfs-807.diff
>
>
> According to release notes
> ([https://archive.apache.org/dist/commons/vfs/RELEASE-NOTES.txt),] version
> 2.9.0 includes the following change:
> Replace construction of FileInputStream and FileOutputStream objects with
> Files NIO APIs. #164. Thanks to Arturo Bernal.
> Strangely, it was not completed: LocalFlie.java has the following now, even
> in master:
> {code:java}
> /**
> * Creates an input stream to read the content from.
> */
> @Override
> protected InputStream doGetInputStream(final int bufferSize) throws
> Exception {
> return new FileInputStream(file);
> }
> {code}
>
> But what is worse, doGetOutputStream is *completely* broken:
> {code:java}
> /**
> * Creates an output stream to write the file content to.
> */
> @Override
> protected OutputStream doGetOutputStream(final boolean bAppend) throws
> Exception {
> return Files.newOutputStream(file.toPath(), bAppend ?
> StandardOpenOption.APPEND : StandardOpenOption.CREATE);
> } {code}
>
> This can be demonstrated with a small java program, see attachment. The
> program just tries to write to a local file (into the same directory), using
> different ways to do it:
> # Java IO, how it was done before VFS 2.9.0
> # Java NIO the way VFS 2.9.0 does it
> # Java NIO the way it must be done
> What is important, the program also verifies the result.
> VFS 2.9.0 fails in 2 cases:
> # When the file does not exist and append must be performed:
> java.nio.file.NoSuchFileException is thrown in this case.
> # When the file exists and no append must be performed: the file is not
> truncated, so whatever is written overwrites the beginning of the file,
> keeping the rest.
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (LANG-1593) Common behaviour for StringUtils join APIs when called with char or String delimiter
[
https://issues.apache.org/jira/browse/LANG-1593?focusedWorklogId=633890&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-633890
]
ASF GitHub Bot logged work on LANG-1593:
Author: ASF GitHub Bot
Created on: 05/Aug/21 00:07
Start Date: 05/Aug/21 00:07
Worklog Time Spent: 10m
Work Description: coveralls commented on pull request #784:
URL: https://github.com/apache/commons-lang/pull/784#issuecomment-893057654
[](https://coveralls.io/builds/41915872)
Coverage increased (+0.009%) to 95.003% when pulling
**ed64d9ca588414e1aa5bca026d41a1775e39c0d2 on
HubertWo:fix/LANG-1593-join-string-delimiter** into
**7c658527094083b2037d362916adf8eb2493ea65 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 633890)
Time Spent: 2h 40m (was: 2.5h)
> Common behaviour for StringUtils join APIs when called with char or String
> delimiter
>
>
> Key: LANG-1593
> URL: https://issues.apache.org/jira/browse/LANG-1593
> Project: Commons Lang
> Issue Type: Improvement
>Affects Versions: 3.4, 3.11
>Reporter: Kiruahxh
>Priority: Minor
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> For now, join(int[], char) is working well.
> However, the same join method called with a string delimiter behaves
> differently : it returns a single memory address which is not the desired
> behavior.
> I think that, for coherence, calling StringUtils with a char or String
> delimiter should return the exact same value.
> Ex :
> {code:java}
> CLASSPATH="./commons-lang3-3.11.jar" jshell
> | Welcome to JShell -- Version 11.0.8
> jshell> import org.apache.commons.lang3.StringUtils
> jshell> int[] arr = {1, 2, 3, 4, 5, 6, 7};
> jshell> String result = StringUtils.join(arr, '-');
> result ==> "1-2-3-4-5-6-7"
> jshell> String result = StringUtils.join(arr, "-");
> result ==> "[I@69663380-"
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [commons-lang] coveralls commented on pull request #784: [LANG-1593] Common behavior for StringUtils join APIs when called with char or String delimiter
coveralls commented on pull request #784: URL: https://github.com/apache/commons-lang/pull/784#issuecomment-893057654 [](https://coveralls.io/builds/41915872) Coverage increased (+0.009%) to 95.003% when pulling **ed64d9ca588414e1aa5bca026d41a1775e39c0d2 on HubertWo:fix/LANG-1593-join-string-delimiter** into **7c658527094083b2037d362916adf8eb2493ea65 on apache:master**. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Work logged] (LANG-1593) Common behaviour for StringUtils join APIs when called with char or String delimiter
[
https://issues.apache.org/jira/browse/LANG-1593?focusedWorklogId=633874&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-633874
]
ASF GitHub Bot logged work on LANG-1593:
Author: ASF GitHub Bot
Created on: 04/Aug/21 23:10
Start Date: 04/Aug/21 23:10
Worklog Time Spent: 10m
Work Description: garydgregory commented on a change in pull request #784:
URL: https://github.com/apache/commons-lang/pull/784#discussion_r683017648
##
File path: src/main/java/org/apache/commons/lang3/StringUtils.java
##
@@ -4818,13 +5052,48 @@ public static String join(final short[] array, final
char delimiter) {
* @since 3.2
*/
public static String join(final short[] array, final char delimiter, final
int startIndex, final int endIndex) {
+return join(array, String.valueOf(delimiter), startIndex, endIndex);
+}
+
+/**
+ *
Review comment:
The 1st para of a Javadoc is automatically in a paragraph IIRC.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 633874)
Time Spent: 2h 20m (was: 2h 10m)
> Common behaviour for StringUtils join APIs when called with char or String
> delimiter
>
>
> Key: LANG-1593
> URL: https://issues.apache.org/jira/browse/LANG-1593
> Project: Commons Lang
> Issue Type: Improvement
>Affects Versions: 3.4, 3.11
>Reporter: Kiruahxh
>Priority: Minor
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> For now, join(int[], char) is working well.
> However, the same join method called with a string delimiter behaves
> differently : it returns a single memory address which is not the desired
> behavior.
> I think that, for coherence, calling StringUtils with a char or String
> delimiter should return the exact same value.
> Ex :
> {code:java}
> CLASSPATH="./commons-lang3-3.11.jar" jshell
> | Welcome to JShell -- Version 11.0.8
> jshell> import org.apache.commons.lang3.StringUtils
> jshell> int[] arr = {1, 2, 3, 4, 5, 6, 7};
> jshell> String result = StringUtils.join(arr, '-');
> result ==> "1-2-3-4-5-6-7"
> jshell> String result = StringUtils.join(arr, "-");
> result ==> "[I@69663380-"
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (LANG-1593) Common behaviour for StringUtils join APIs when called with char or String delimiter
[
https://issues.apache.org/jira/browse/LANG-1593?focusedWorklogId=633875&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-633875
]
ASF GitHub Bot logged work on LANG-1593:
Author: ASF GitHub Bot
Created on: 04/Aug/21 23:10
Start Date: 04/Aug/21 23:10
Worklog Time Spent: 10m
Work Description: garydgregory commented on a change in pull request #784:
URL: https://github.com/apache/commons-lang/pull/784#discussion_r683017853
##
File path: src/main/java/org/apache/commons/lang3/StringUtils.java
##
@@ -4307,13 +4481,48 @@ public static String join(final int[] array, final char
separator) {
* @since 3.2
*/
public static String join(final int[] array, final char delimiter, final
int startIndex, final int endIndex) {
+return join(array, String.valueOf(delimiter), startIndex, endIndex);
+}
+
+/**
+ *
+ * Joins the elements of the provided array into a single String
containing the provided list of elements.
+ *
+ *
+ *
+ * No delimiter is added before or after the list. Null objects or empty
strings within the array are represented
+ * by empty strings.
+ *
+ *
+ *
+ * StringUtils.join(null, *) = null
+ * StringUtils.join([], *) = ""
+ * StringUtils.join([null], *) = ""
+ * StringUtils.join([1, 2, 3], ';') = "1;2;3"
+ * StringUtils.join([1, 2, 3], null) = "123"
+ *
+ *
+ * @param array
+ *the array of values to join together, may be null
+ * @param delimiter
+ *the separator String to use
+ * @param startIndex
+ *the first index to start joining from. It is an error to
pass in a start index past the end of the
+ *array
+ * @param endIndex
+ *the index to stop joining from (exclusive). It is an error
to pass in an end index past the end of
+ *the array
+ * @return the joined String, {@code null} if null array input
+ * @since 3.13.0
+ */
+public static String join(final int[] array, final String delimiter, final
int startIndex, final int endIndex) {
if (array == null) {
return null;
}
if (endIndex - startIndex <= 0) {
return EMPTY;
}
-final StringJoiner joiner = newStringJoiner(delimiter);
+final StringJoiner joiner = new
StringJoiner(toStringOrEmpty(delimiter));
Review comment:
Why did you remove the private helper method?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 633875)
Time Spent: 2.5h (was: 2h 20m)
> Common behaviour for StringUtils join APIs when called with char or String
> delimiter
>
>
> Key: LANG-1593
> URL: https://issues.apache.org/jira/browse/LANG-1593
> Project: Commons Lang
> Issue Type: Improvement
>Affects Versions: 3.4, 3.11
>Reporter: Kiruahxh
>Priority: Minor
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> For now, join(int[], char) is working well.
> However, the same join method called with a string delimiter behaves
> differently : it returns a single memory address which is not the desired
> behavior.
> I think that, for coherence, calling StringUtils with a char or String
> delimiter should return the exact same value.
> Ex :
> {code:java}
> CLASSPATH="./commons-lang3-3.11.jar" jshell
> | Welcome to JShell -- Version 11.0.8
> jshell> import org.apache.commons.lang3.StringUtils
> jshell> int[] arr = {1, 2, 3, 4, 5, 6, 7};
> jshell> String result = StringUtils.join(arr, '-');
> result ==> "1-2-3-4-5-6-7"
> jshell> String result = StringUtils.join(arr, "-");
> result ==> "[I@69663380-"
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [commons-lang] garydgregory commented on a change in pull request #784: [LANG-1593] Common behavior for StringUtils join APIs when called with char or String delimiter
garydgregory commented on a change in pull request #784:
URL: https://github.com/apache/commons-lang/pull/784#discussion_r683017853
##
File path: src/main/java/org/apache/commons/lang3/StringUtils.java
##
@@ -4307,13 +4481,48 @@ public static String join(final int[] array, final char
separator) {
* @since 3.2
*/
public static String join(final int[] array, final char delimiter, final
int startIndex, final int endIndex) {
+return join(array, String.valueOf(delimiter), startIndex, endIndex);
+}
+
+/**
+ *
+ * Joins the elements of the provided array into a single String
containing the provided list of elements.
+ *
+ *
+ *
+ * No delimiter is added before or after the list. Null objects or empty
strings within the array are represented
+ * by empty strings.
+ *
+ *
+ *
+ * StringUtils.join(null, *) = null
+ * StringUtils.join([], *) = ""
+ * StringUtils.join([null], *) = ""
+ * StringUtils.join([1, 2, 3], ';') = "1;2;3"
+ * StringUtils.join([1, 2, 3], null) = "123"
+ *
+ *
+ * @param array
+ *the array of values to join together, may be null
+ * @param delimiter
+ *the separator String to use
+ * @param startIndex
+ *the first index to start joining from. It is an error to
pass in a start index past the end of the
+ *array
+ * @param endIndex
+ *the index to stop joining from (exclusive). It is an error
to pass in an end index past the end of
+ *the array
+ * @return the joined String, {@code null} if null array input
+ * @since 3.13.0
+ */
+public static String join(final int[] array, final String delimiter, final
int startIndex, final int endIndex) {
if (array == null) {
return null;
}
if (endIndex - startIndex <= 0) {
return EMPTY;
}
-final StringJoiner joiner = newStringJoiner(delimiter);
+final StringJoiner joiner = new
StringJoiner(toStringOrEmpty(delimiter));
Review comment:
Why did you remove the private helper method?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [commons-lang] garydgregory commented on a change in pull request #784: [LANG-1593] Common behavior for StringUtils join APIs when called with char or String delimiter
garydgregory commented on a change in pull request #784:
URL: https://github.com/apache/commons-lang/pull/784#discussion_r683017648
##
File path: src/main/java/org/apache/commons/lang3/StringUtils.java
##
@@ -4818,13 +5052,48 @@ public static String join(final short[] array, final
char delimiter) {
* @since 3.2
*/
public static String join(final short[] array, final char delimiter, final
int startIndex, final int endIndex) {
+return join(array, String.valueOf(delimiter), startIndex, endIndex);
+}
+
+/**
+ *
Review comment:
The 1st para of a Javadoc is automatically in a paragraph IIRC.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Work logged] (LANG-1593) Common behaviour for StringUtils join APIs when called with char or String delimiter
[
https://issues.apache.org/jira/browse/LANG-1593?focusedWorklogId=633873&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-633873
]
ASF GitHub Bot logged work on LANG-1593:
Author: ASF GitHub Bot
Created on: 04/Aug/21 23:09
Start Date: 04/Aug/21 23:09
Worklog Time Spent: 10m
Work Description: garydgregory commented on a change in pull request #784:
URL: https://github.com/apache/commons-lang/pull/784#discussion_r683017414
##
File path: src/main/java/org/apache/commons/lang3/StringUtils.java
##
@@ -4858,6 +5127,38 @@ public static String join(final short[] array, final
char delimiter, final int s
return join(elements, null);
}
+public static String join(byte[] elements, String separator) {
Review comment:
All public methods should be Javadoc'd.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 633873)
Time Spent: 2h 10m (was: 2h)
> Common behaviour for StringUtils join APIs when called with char or String
> delimiter
>
>
> Key: LANG-1593
> URL: https://issues.apache.org/jira/browse/LANG-1593
> Project: Commons Lang
> Issue Type: Improvement
>Affects Versions: 3.4, 3.11
>Reporter: Kiruahxh
>Priority: Minor
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> For now, join(int[], char) is working well.
> However, the same join method called with a string delimiter behaves
> differently : it returns a single memory address which is not the desired
> behavior.
> I think that, for coherence, calling StringUtils with a char or String
> delimiter should return the exact same value.
> Ex :
> {code:java}
> CLASSPATH="./commons-lang3-3.11.jar" jshell
> | Welcome to JShell -- Version 11.0.8
> jshell> import org.apache.commons.lang3.StringUtils
> jshell> int[] arr = {1, 2, 3, 4, 5, 6, 7};
> jshell> String result = StringUtils.join(arr, '-');
> result ==> "1-2-3-4-5-6-7"
> jshell> String result = StringUtils.join(arr, "-");
> result ==> "[I@69663380-"
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [commons-lang] garydgregory commented on a change in pull request #784: [LANG-1593] Common behavior for StringUtils join APIs when called with char or String delimiter
garydgregory commented on a change in pull request #784:
URL: https://github.com/apache/commons-lang/pull/784#discussion_r683017414
##
File path: src/main/java/org/apache/commons/lang3/StringUtils.java
##
@@ -4858,6 +5127,38 @@ public static String join(final short[] array, final
char delimiter, final int s
return join(elements, null);
}
+public static String join(byte[] elements, String separator) {
Review comment:
All public methods should be Javadoc'd.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [commons-lang] garydgregory commented on a change in pull request #784: [LANG-1593] Common behavior for StringUtils join APIs when called with char or String delimiter
garydgregory commented on a change in pull request #784:
URL: https://github.com/apache/commons-lang/pull/784#discussion_r683017191
##
File path: src/main/java/org/apache/commons/lang3/StringUtils.java
##
@@ -4858,6 +5127,38 @@ public static String join(final short[] array, final
char delimiter, final int s
return join(elements, null);
}
+public static String join(byte[] elements, String separator) {
+return (elements == null) ? null : join(elements, separator, 0,
elements.length);
+}
+
+public static String join(boolean[] elements, String separator) {
+return (elements == null) ? null : join(elements, separator, 0,
elements.length);
+}
+
+public static String join(char[] elements, String separator) {
+return (elements == null) ? null : join(elements, separator, 0,
elements.length);
+}
+
+public static String join(double[] elements, String separator) {
+return (elements == null) ? null : join(elements, separator, 0,
elements.length);
+}
+
+public static String join(float[] elements, String separator) {
+return (elements == null) ? null : join(elements, separator, 0,
elements.length);
+}
+
+public static String join(int[] elements, String separator) {
+return (elements == null) ? null : join(elements, separator, 0,
elements.length);
+}
+
+public static String join(long[] elements, String separator) {
+return (elements == null) ? null : join(elements, separator, 0,
elements.length);
+}
+
+public static String join(short[] elements, String separator) {
+return (elements == null) ? null : join(elements, separator, 0,
elements.length);
+}
Review comment:
Remove useless parentheses please.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Work logged] (LANG-1593) Common behaviour for StringUtils join APIs when called with char or String delimiter
[
https://issues.apache.org/jira/browse/LANG-1593?focusedWorklogId=633872&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-633872
]
ASF GitHub Bot logged work on LANG-1593:
Author: ASF GitHub Bot
Created on: 04/Aug/21 23:08
Start Date: 04/Aug/21 23:08
Worklog Time Spent: 10m
Work Description: garydgregory commented on a change in pull request #784:
URL: https://github.com/apache/commons-lang/pull/784#discussion_r683017191
##
File path: src/main/java/org/apache/commons/lang3/StringUtils.java
##
@@ -4858,6 +5127,38 @@ public static String join(final short[] array, final
char delimiter, final int s
return join(elements, null);
}
+public static String join(byte[] elements, String separator) {
+return (elements == null) ? null : join(elements, separator, 0,
elements.length);
+}
+
+public static String join(boolean[] elements, String separator) {
+return (elements == null) ? null : join(elements, separator, 0,
elements.length);
+}
+
+public static String join(char[] elements, String separator) {
+return (elements == null) ? null : join(elements, separator, 0,
elements.length);
+}
+
+public static String join(double[] elements, String separator) {
+return (elements == null) ? null : join(elements, separator, 0,
elements.length);
+}
+
+public static String join(float[] elements, String separator) {
+return (elements == null) ? null : join(elements, separator, 0,
elements.length);
+}
+
+public static String join(int[] elements, String separator) {
+return (elements == null) ? null : join(elements, separator, 0,
elements.length);
+}
+
+public static String join(long[] elements, String separator) {
+return (elements == null) ? null : join(elements, separator, 0,
elements.length);
+}
+
+public static String join(short[] elements, String separator) {
+return (elements == null) ? null : join(elements, separator, 0,
elements.length);
+}
Review comment:
Remove useless parentheses please.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 633872)
Time Spent: 2h (was: 1h 50m)
> Common behaviour for StringUtils join APIs when called with char or String
> delimiter
>
>
> Key: LANG-1593
> URL: https://issues.apache.org/jira/browse/LANG-1593
> Project: Commons Lang
> Issue Type: Improvement
>Affects Versions: 3.4, 3.11
>Reporter: Kiruahxh
>Priority: Minor
> Time Spent: 2h
> Remaining Estimate: 0h
>
> For now, join(int[], char) is working well.
> However, the same join method called with a string delimiter behaves
> differently : it returns a single memory address which is not the desired
> behavior.
> I think that, for coherence, calling StringUtils with a char or String
> delimiter should return the exact same value.
> Ex :
> {code:java}
> CLASSPATH="./commons-lang3-3.11.jar" jshell
> | Welcome to JShell -- Version 11.0.8
> jshell> import org.apache.commons.lang3.StringUtils
> jshell> int[] arr = {1, 2, 3, 4, 5, 6, 7};
> jshell> String result = StringUtils.join(arr, '-');
> result ==> "1-2-3-4-5-6-7"
> jshell> String result = StringUtils.join(arr, "-");
> result ==> "[I@69663380-"
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (VFS-807) LocalFile migration to NIO is not done correctly
[
https://issues.apache.org/jira/browse/VFS-807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17393448#comment-17393448
]
Gary D. Gregory commented on VFS-807:
-
Hi [~klv_m72]
Please don't delete comments ("// TODO Reuse Apache Commons IO") in your patch
because that's where the implementation will be coming from as soon as Apache
Commons IO is released, you can see git master for Commons IO which I just
updated. My plan is to release Commons IO and then to reuse its
PathUtils.newOutputStream() method from here.
> LocalFile migration to NIO is not done correctly
>
>
> Key: VFS-807
> URL: https://issues.apache.org/jira/browse/VFS-807
> Project: Commons VFS
> Issue Type: Bug
>Affects Versions: 2.9.0
>Reporter: L
>Priority: Blocker
> Fix For: 2.9.1
>
> Attachments: TestJdkWriteFile.java, vfs-807.diff
>
>
> According to release notes
> ([https://archive.apache.org/dist/commons/vfs/RELEASE-NOTES.txt),] version
> 2.9.0 includes the following change:
> Replace construction of FileInputStream and FileOutputStream objects with
> Files NIO APIs. #164. Thanks to Arturo Bernal.
> Strangely, it was not completed: LocalFlie.java has the following now, even
> in master:
> {code:java}
> /**
> * Creates an input stream to read the content from.
> */
> @Override
> protected InputStream doGetInputStream(final int bufferSize) throws
> Exception {
> return new FileInputStream(file);
> }
> {code}
>
> But what is worse, doGetOutputStream is *completely* broken:
> {code:java}
> /**
> * Creates an output stream to write the file content to.
> */
> @Override
> protected OutputStream doGetOutputStream(final boolean bAppend) throws
> Exception {
> return Files.newOutputStream(file.toPath(), bAppend ?
> StandardOpenOption.APPEND : StandardOpenOption.CREATE);
> } {code}
>
> This can be demonstrated with a small java program, see attachment. The
> program just tries to write to a local file (into the same directory), using
> different ways to do it:
> # Java IO, how it was done before VFS 2.9.0
> # Java NIO the way VFS 2.9.0 does it
> # Java NIO the way it must be done
> What is important, the program also verifies the result.
> VFS 2.9.0 fails in 2 cases:
> # When the file does not exist and append must be performed:
> java.nio.file.NoSuchFileException is thrown in this case.
> # When the file exists and no append must be performed: the file is not
> truncated, so whatever is written overwrites the beginning of the file,
> keeping the rest.
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [commons-io] garydgregory merged pull request #255: Bump maven-enforcer-plugin from 3.0.0-M3 to 3.0.0
garydgregory merged pull request #255: URL: https://github.com/apache/commons-io/pull/255 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (LANG-1593) Common behaviour for StringUtils join APIs when called with char or String delimiter
[
https://issues.apache.org/jira/browse/LANG-1593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17393438#comment-17393438
]
Hubert Wojciechowski commented on LANG-1593:
Draft PR created: https://github.com/apache/commons-lang/pull/784
Solution is ready. The only missing thing is Javadoc for some methods.
Looks like build for Java 16 and Java 17 is broken - please check main branch
build or any PR.
> Common behaviour for StringUtils join APIs when called with char or String
> delimiter
>
>
> Key: LANG-1593
> URL: https://issues.apache.org/jira/browse/LANG-1593
> Project: Commons Lang
> Issue Type: Improvement
>Affects Versions: 3.4, 3.11
>Reporter: Kiruahxh
>Priority: Minor
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> For now, join(int[], char) is working well.
> However, the same join method called with a string delimiter behaves
> differently : it returns a single memory address which is not the desired
> behavior.
> I think that, for coherence, calling StringUtils with a char or String
> delimiter should return the exact same value.
> Ex :
> {code:java}
> CLASSPATH="./commons-lang3-3.11.jar" jshell
> | Welcome to JShell -- Version 11.0.8
> jshell> import org.apache.commons.lang3.StringUtils
> jshell> int[] arr = {1, 2, 3, 4, 5, 6, 7};
> jshell> String result = StringUtils.join(arr, '-');
> result ==> "1-2-3-4-5-6-7"
> jshell> String result = StringUtils.join(arr, "-");
> result ==> "[I@69663380-"
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (EMAIL-197) How about a release? :)
[ https://issues.apache.org/jira/browse/EMAIL-197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17393343#comment-17393343 ] Gary D. Gregory commented on EMAIL-197: --- This is a topic for the mailing list, not jira. > How about a release? :) > --- > > Key: EMAIL-197 > URL: https://issues.apache.org/jira/browse/EMAIL-197 > Project: Commons Email > Issue Type: Wish >Affects Versions: 1.5 >Reporter: Cott Lang >Priority: Major > > It's been what, 4 years since a release? > We just ran into a bug around accented characters, which was fixed years ago > in javax.mail 1.6.0. We're overriding the dependency to fix it, but it > appears commons email is already updated, just never released? > > Thanks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (EMAIL-197) How about a release? :)
Cott Lang created EMAIL-197: --- Summary: How about a release? :) Key: EMAIL-197 URL: https://issues.apache.org/jira/browse/EMAIL-197 Project: Commons Email Issue Type: Wish Affects Versions: 1.5 Reporter: Cott Lang It's been what, 4 years since a release? We just ran into a bug around accented characters, which was fixed years ago in javax.mail 1.6.0. We're overriding the dependency to fix it, but it appears commons email is already updated, just never released? Thanks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (COMPRESS-584) IOUtils.readRange() can read more from a channel than asked for
[
https://issues.apache.org/jira/browse/COMPRESS-584?focusedWorklogId=633511&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-633511
]
ASF GitHub Bot logged work on COMPRESS-584:
---
Author: ASF GitHub Bot
Created on: 04/Aug/21 12:58
Start Date: 04/Aug/21 12:58
Worklog Time Spent: 10m
Work Description: coveralls edited a comment on pull request #214:
URL: https://github.com/apache/commons-compress/pull/214#issuecomment-891708225
[](https://coveralls.io/builds/41898328)
Coverage increased (+0.0005%) to 86.355% when pulling
**cf5adfba2e3f8a1eebb51af817234bd50e99f734 on B3Partners:COMPRESS-584** into
**8287bfdb8adbeae433e0a1205c6e4fb180b9d3be on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 633511)
Time Spent: 4h (was: 3h 50m)
> IOUtils.readRange() can read more from a channel than asked for
> ---
>
> Key: COMPRESS-584
> URL: https://issues.apache.org/jira/browse/COMPRESS-584
> Project: Commons Compress
> Issue Type: Bug
> Components: Archivers
>Affects Versions: 1.21
>Reporter: Matthijs Laan
>Priority: Major
> Time Spent: 4h
> Remaining Estimate: 0h
>
> When {{IOUtils.readRange(ReadableByteChannel,int)}} gets less than the number
> of bytes asked for in the first read call it does not reduce the buffer size
> for the next read call and may read more than asked for.
> This situation is rare when using a {{FileChannel}} but I wrote a
> {{SeekableByteChannel}} backed by a URI using HTTP range requests and reading
> from a socket can often return less bytes than asked for. When I used this
> channel to read a {{ZipFile}} it only read the ZIP central directory
> partially sometimes because {{IOUtils.readRange()}} called from
> {{ZipFile.readCentralDirectoryEntry()}} read more bytes than asked for and it
> stopped parsing directory entries.
> Fix: [https://github.com/apache/commons-compress/pull/214]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [commons-compress] coveralls edited a comment on pull request #214: [COMPRESS-584] Fix IOUtils.readRange() can read more from a channel than asked for
coveralls edited a comment on pull request #214: URL: https://github.com/apache/commons-compress/pull/214#issuecomment-891708225 [](https://coveralls.io/builds/41898328) Coverage increased (+0.0005%) to 86.355% when pulling **cf5adfba2e3f8a1eebb51af817234bd50e99f734 on B3Partners:COMPRESS-584** into **8287bfdb8adbeae433e0a1205c6e4fb180b9d3be on apache:master**. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (DBCP-509) Not all methods are consistent in PerUserPooldataSource and InstanceKeyDataSource
[ https://issues.apache.org/jira/browse/DBCP-509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17393121#comment-17393121 ] Gary D. Gregory commented on DBCP-509: -- Hi [~kinow] Is this still an issue? Do you want to provide a PR for one of these settings so we can get a better grasp of the issue and solution? > Not all methods are consistent in PerUserPooldataSource and > InstanceKeyDataSource > - > > Key: DBCP-509 > URL: https://issues.apache.org/jira/browse/DBCP-509 > Project: Commons DBCP > Issue Type: Improvement >Affects Versions: 2.4.0 >Reporter: Bruno P. Kinoshita >Priority: Major > Fix For: 2.9.1 > > Time Spent: 40m > Remaining Estimate: 0h > > While working on DBCP-504 tests, I wrote a few files to help me automating > some of the tests (e.g. > https://gist.github.com/kinow/053b6d1f293fdc208a2a14571f246786). > In PerUserPooldataSource, I realized I had to change some tests that were > failing to handle null values. But not all methods. So I had a look at the > methods, and the majority was following a pattern > * using primitives > * default'ing to the class/parent method getDefaultPropertyZ() whenever Z > property was null > But three values were using objects instead of primitives, and allowing > null's: > * defaultAutoCommit > * defaultReadOnly > * perUserDefaultAutoCommit > I prepared a pull request that falls back to the default method's values. > It's more of a discussion issue, just to document what I found during > DBCP-504. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (DBCP-509) Not all methods are consistent in PerUserPooldataSource and InstanceKeyDataSource
[ https://issues.apache.org/jira/browse/DBCP-509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary D. Gregory updated DBCP-509: - Fix Version/s: (was: 2.9.0) 2.9.1 > Not all methods are consistent in PerUserPooldataSource and > InstanceKeyDataSource > - > > Key: DBCP-509 > URL: https://issues.apache.org/jira/browse/DBCP-509 > Project: Commons DBCP > Issue Type: Improvement >Affects Versions: 2.4.0 >Reporter: Bruno P. Kinoshita >Priority: Major > Fix For: 2.9.1 > > Time Spent: 40m > Remaining Estimate: 0h > > While working on DBCP-504 tests, I wrote a few files to help me automating > some of the tests (e.g. > https://gist.github.com/kinow/053b6d1f293fdc208a2a14571f246786). > In PerUserPooldataSource, I realized I had to change some tests that were > failing to handle null values. But not all methods. So I had a look at the > methods, and the majority was following a pattern > * using primitives > * default'ing to the class/parent method getDefaultPropertyZ() whenever Z > property was null > But three values were using objects instead of primitives, and allowing > null's: > * defaultAutoCommit > * defaultReadOnly > * perUserDefaultAutoCommit > I prepared a pull request that falls back to the default method's values. > It's more of a discussion issue, just to document what I found during > DBCP-504. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (COMPRESS-584) IOUtils.readRange() can read more from a channel than asked for
[
https://issues.apache.org/jira/browse/COMPRESS-584?focusedWorklogId=633452&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-633452
]
ASF GitHub Bot logged work on COMPRESS-584:
---
Author: ASF GitHub Bot
Created on: 04/Aug/21 11:56
Start Date: 04/Aug/21 11:56
Worklog Time Spent: 10m
Work Description: matthijsln edited a comment on pull request #214:
URL: https://github.com/apache/commons-compress/pull/214#issuecomment-892573540
The "no copy buffer" version can be more efficient by only calling
`Arrays.copyOf()` when hitting EOF. This avoids copying the byte array when it
is completely filled:
```
public static byte[] readRange(final ReadableByteChannel input, final
int len) throws IOException {
final ByteBuffer b = ByteBuffer.allocate(len);
while (b.hasRemaining()) {
if (input.read(b) <= 0) {
break;
}
}
if (b.hasRemaining()) {
// We hit EOF and read less bytes, return a smaller array than
len
return Arrays.copyOf(b.array(), b.position());
} else {
return b.array();
}
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 633452)
Time Spent: 3h 50m (was: 3h 40m)
> IOUtils.readRange() can read more from a channel than asked for
> ---
>
> Key: COMPRESS-584
> URL: https://issues.apache.org/jira/browse/COMPRESS-584
> Project: Commons Compress
> Issue Type: Bug
> Components: Archivers
>Affects Versions: 1.21
>Reporter: Matthijs Laan
>Priority: Major
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> When {{IOUtils.readRange(ReadableByteChannel,int)}} gets less than the number
> of bytes asked for in the first read call it does not reduce the buffer size
> for the next read call and may read more than asked for.
> This situation is rare when using a {{FileChannel}} but I wrote a
> {{SeekableByteChannel}} backed by a URI using HTTP range requests and reading
> from a socket can often return less bytes than asked for. When I used this
> channel to read a {{ZipFile}} it only read the ZIP central directory
> partially sometimes because {{IOUtils.readRange()}} called from
> {{ZipFile.readCentralDirectoryEntry()}} read more bytes than asked for and it
> stopped parsing directory entries.
> Fix: [https://github.com/apache/commons-compress/pull/214]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [commons-compress] matthijsln edited a comment on pull request #214: [COMPRESS-584] Fix IOUtils.readRange() can read more from a channel than asked for
matthijsln edited a comment on pull request #214:
URL: https://github.com/apache/commons-compress/pull/214#issuecomment-892573540
The "no copy buffer" version can be more efficient by only calling
`Arrays.copyOf()` when hitting EOF. This avoids copying the byte array when it
is completely filled:
```
public static byte[] readRange(final ReadableByteChannel input, final
int len) throws IOException {
final ByteBuffer b = ByteBuffer.allocate(len);
while (b.hasRemaining()) {
if (input.read(b) <= 0) {
break;
}
}
if (b.hasRemaining()) {
// We hit EOF and read less bytes, return a smaller array than
len
return Arrays.copyOf(b.array(), b.position());
} else {
return b.array();
}
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (NET-703) Include example sources in binary bundle
Sebb created NET-703: Summary: Include example sources in binary bundle Key: NET-703 URL: https://issues.apache.org/jira/browse/NET-703 Project: Commons Net Issue Type: Improvement Reporter: Sebb The binary bundle (zip and tar.gz) includes the main sources jar. However this does not include the examples. It would be more useful to include the example sources instead. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (COMPRESS-584) IOUtils.readRange() can read more from a channel than asked for
[
https://issues.apache.org/jira/browse/COMPRESS-584?focusedWorklogId=633446&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-633446
]
ASF GitHub Bot logged work on COMPRESS-584:
---
Author: ASF GitHub Bot
Created on: 04/Aug/21 11:14
Start Date: 04/Aug/21 11:14
Worklog Time Spent: 10m
Work Description: matthijsln commented on pull request #214:
URL: https://github.com/apache/commons-compress/pull/214#issuecomment-892573540
Wrapping a byte array in a `ByteBuffer` is the most efficient. In the
following implementation an array copy is only needed when hitting EOF:
```
public static byte[] readRange(final ReadableByteChannel input, final
int len) throws IOException {
final byte[] out = new byte[len];
final ByteBuffer b = ByteBuffer.wrap(out);
while (b.hasRemaining()) {
if (input.read(b) <= 0) {
break;
}
}
if (b.hasRemaining()) {
// We hit EOF and read less bytes, return a smaller array than
len
return Arrays.copyOf(b.array(), b.position());
} else {
return out;
}
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 633446)
Time Spent: 3h 40m (was: 3.5h)
> IOUtils.readRange() can read more from a channel than asked for
> ---
>
> Key: COMPRESS-584
> URL: https://issues.apache.org/jira/browse/COMPRESS-584
> Project: Commons Compress
> Issue Type: Bug
> Components: Archivers
>Affects Versions: 1.21
>Reporter: Matthijs Laan
>Priority: Major
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> When {{IOUtils.readRange(ReadableByteChannel,int)}} gets less than the number
> of bytes asked for in the first read call it does not reduce the buffer size
> for the next read call and may read more than asked for.
> This situation is rare when using a {{FileChannel}} but I wrote a
> {{SeekableByteChannel}} backed by a URI using HTTP range requests and reading
> from a socket can often return less bytes than asked for. When I used this
> channel to read a {{ZipFile}} it only read the ZIP central directory
> partially sometimes because {{IOUtils.readRange()}} called from
> {{ZipFile.readCentralDirectoryEntry()}} read more bytes than asked for and it
> stopped parsing directory entries.
> Fix: [https://github.com/apache/commons-compress/pull/214]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [commons-compress] matthijsln commented on pull request #214: [COMPRESS-584] Fix IOUtils.readRange() can read more from a channel than asked for
matthijsln commented on pull request #214:
URL: https://github.com/apache/commons-compress/pull/214#issuecomment-892573540
Wrapping a byte array in a `ByteBuffer` is the most efficient. In the
following implementation an array copy is only needed when hitting EOF:
```
public static byte[] readRange(final ReadableByteChannel input, final
int len) throws IOException {
final byte[] out = new byte[len];
final ByteBuffer b = ByteBuffer.wrap(out);
while (b.hasRemaining()) {
if (input.read(b) <= 0) {
break;
}
}
if (b.hasRemaining()) {
// We hit EOF and read less bytes, return a smaller array than
len
return Arrays.copyOf(b.array(), b.position());
} else {
return out;
}
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Work logged] (COMPRESS-584) IOUtils.readRange() can read more from a channel than asked for
[
https://issues.apache.org/jira/browse/COMPRESS-584?focusedWorklogId=633400&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-633400
]
ASF GitHub Bot logged work on COMPRESS-584:
---
Author: ASF GitHub Bot
Created on: 04/Aug/21 09:56
Start Date: 04/Aug/21 09:56
Worklog Time Spent: 10m
Work Description: PeterAlfredLee commented on pull request #214:
URL: https://github.com/apache/commons-compress/pull/214#issuecomment-892526508
I prefer the one with a copy buffer.
I'm not professional in IO but the implemention with buffer is pretty common
in IO implementions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 633400)
Time Spent: 3.5h (was: 3h 20m)
> IOUtils.readRange() can read more from a channel than asked for
> ---
>
> Key: COMPRESS-584
> URL: https://issues.apache.org/jira/browse/COMPRESS-584
> Project: Commons Compress
> Issue Type: Bug
> Components: Archivers
>Affects Versions: 1.21
>Reporter: Matthijs Laan
>Priority: Major
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> When {{IOUtils.readRange(ReadableByteChannel,int)}} gets less than the number
> of bytes asked for in the first read call it does not reduce the buffer size
> for the next read call and may read more than asked for.
> This situation is rare when using a {{FileChannel}} but I wrote a
> {{SeekableByteChannel}} backed by a URI using HTTP range requests and reading
> from a socket can often return less bytes than asked for. When I used this
> channel to read a {{ZipFile}} it only read the ZIP central directory
> partially sometimes because {{IOUtils.readRange()}} called from
> {{ZipFile.readCentralDirectoryEntry()}} read more bytes than asked for and it
> stopped parsing directory entries.
> Fix: [https://github.com/apache/commons-compress/pull/214]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [commons-compress] PeterAlfredLee commented on pull request #214: [COMPRESS-584] Fix IOUtils.readRange() can read more from a channel than asked for
PeterAlfredLee commented on pull request #214: URL: https://github.com/apache/commons-compress/pull/214#issuecomment-892526508 I prefer the one with a copy buffer. I'm not professional in IO but the implemention with buffer is pretty common in IO implementions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (COMPRESS-585) ZipFile fails to read a zipfile with a comment or extra data longer than 8024 bytes
[
https://issues.apache.org/jira/browse/COMPRESS-585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17392942#comment-17392942
]
Matthijs Laan edited comment on COMPRESS-585 at 8/4/21, 9:53 AM:
-
Yes, see the following test case:
{code:java}
@Test
public void testCommentLongerThanCopyBufSize() throws Exception {
final Field COPY_BUF_SIZE = IOUtils.class.getDeclaredField("COPY_BUF_SIZE");
COPY_BUF_SIZE.setAccessible(true);
final int copyBufSize = (int)COPY_BUF_SIZE.get(null);
String repeatingString = "...a comment that is longer than the copy buf
size";
// Repeat the comment string until we are sure to exceed the COPY_BUF_SIZE
int numberOfRepeats = (copyBufSize / repeatingString.length()) + 1;
String comment = String.join("", Collections.nCopies(numberOfRepeats,
repeatingString));
final File f =
File.createTempFile("commons-compress-zipfiletest-longcomment", ".zip");
f.deleteOnExit();
try (final OutputStream o = Files.newOutputStream(f.toPath());
final ZipArchiveOutputStream zo = new ZipArchiveOutputStream(o)) {
ZipArchiveEntry ze = new ZipArchiveEntry("foo");
ze.setComment(comment);
zo.putArchiveEntry(ze);
zo.write("bar".getBytes(UTF_8));
zo.closeArchiveEntry();
}
try(final ZipFile zip = new ZipFile(FileChannel.open(f.toPath(),
StandardOpenOption.READ))) {
ZipArchiveEntry entry = zip.getEntry("foo");
assertNotNull(entry);
assertEquals(comment, entry.getComment());
}
}{code}
My understanding of {{IOUtils.readRange()}} was incorrect, it can read more
instead of less.
This testcase creates a zip file with a 8050 length comment. In my case
{{IOUtils.readRange()}} returns 8072 bytes when reading the comment, although
this may depend on the platform.
The cause is the same as COMPRESS-584, see
[https://github.com/apache/commons-compress/pull/214]. With this
{{IOUtils.readRange()}} fix it works:
{code:java}
public static byte[] readRange(final ReadableByteChannel input, final int len)
throws IOException {
final ByteArrayOutputStream output = new ByteArrayOutputStream();
final ByteBuffer b = ByteBuffer.allocate(Math.min(len, COPY_BUF_SIZE));
int read = 0;
while (read < len) {
// Make sure we never read more than len bytes
b.limit(Math.min(len - read, b.remaining()));
final int readNow = input.read(b);
if (readNow <= 0) {
break;
}
output.write(b.array(), 0, readNow);
b.rewind();
read += readNow;
}{code}
The testcase can be made platform independent by reading from memory and
trickling bytes explicitly like the
{{readRangeFromChannelDoesntReadMoreThanAskedForWhenItGotLessInFirstReadCall()}}
testcase in the PR for COMPRESS-584.
was (Author: matthijsln):
Yes, see the following test case:
{code:java}
@Test
public void testCommentLongerThanCopyBufSize() throws Exception {
final Field COPY_BUF_SIZE = IOUtils.class.getDeclaredField("COPY_BUF_SIZE");
COPY_BUF_SIZE.setAccessible(true);
final int copyBufSize = (int)COPY_BUF_SIZE.get(null);
String repeatingString = "...a comment that is longer than the copy buf
size";
// Repeat the comment string until we are sure to exceed the COPY_BUF_SIZE
int numberOfRepeats = (copyBufSize / repeatingString.length()) + 1;
String comment = String.join("", Collections.nCopies(numberOfRepeats,
repeatingString));
final File f =
File.createTempFile("commons-compress-zipfiletest-longcomment", ".zip");
f.deleteOnExit();
try (final OutputStream o = Files.newOutputStream(f.toPath());
final ZipArchiveOutputStream zo = new ZipArchiveOutputStream(o)) {
ZipArchiveEntry ze = new ZipArchiveEntry("foo");
ze.setComment(comment);
zo.putArchiveEntry(ze);
zo.write("bar".getBytes(UTF_8));
zo.closeArchiveEntry();
}
try(final ZipFile zip = new ZipFile(FileChannel.open(f.toPath(),
StandardOpenOption.READ))) {
ZipArchiveEntry entry = zip.getEntry("foo");
assertNotNull(entry);
assertEquals(comment, entry.getComment());
}
}{code}
My understanding of {{IOUtils.readRange()}} was incorrect, it can read more
instead of less.
This testcase creates a zip file with a 8050 length comment. In my case
{{IOUtils.readRange()}} returns 8072 bytes instead, although this may depend on
the platform.
The cause is the same as COMPRESS-584, see
[https://github.com/apache/commons-compress/pull/214]. With this
{{IOUtils.readRange()}} fix it works:
{code:java}
public static byte[] readRange(final ReadableByteChannel input, final int len)
throws IOException {
final ByteArrayOutputStream output = new ByteArrayOutputStream();
final ByteBuffer b = ByteBuffer.allocate(Math.min(len, COPY_BUF_SIZE));
int read = 0;
while
[jira] [Comment Edited] (COMPRESS-585) ZipFile fails to read a zipfile with a comment or extra data longer than 8024 bytes
[
https://issues.apache.org/jira/browse/COMPRESS-585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17392942#comment-17392942
]
Matthijs Laan edited comment on COMPRESS-585 at 8/4/21, 9:52 AM:
-
Yes, see the following test case:
{code:java}
@Test
public void testCommentLongerThanCopyBufSize() throws Exception {
final Field COPY_BUF_SIZE = IOUtils.class.getDeclaredField("COPY_BUF_SIZE");
COPY_BUF_SIZE.setAccessible(true);
final int copyBufSize = (int)COPY_BUF_SIZE.get(null);
String repeatingString = "...a comment that is longer than the copy buf
size";
// Repeat the comment string until we are sure to exceed the COPY_BUF_SIZE
int numberOfRepeats = (copyBufSize / repeatingString.length()) + 1;
String comment = String.join("", Collections.nCopies(numberOfRepeats,
repeatingString));
final File f =
File.createTempFile("commons-compress-zipfiletest-longcomment", ".zip");
f.deleteOnExit();
try (final OutputStream o = Files.newOutputStream(f.toPath());
final ZipArchiveOutputStream zo = new ZipArchiveOutputStream(o)) {
ZipArchiveEntry ze = new ZipArchiveEntry("foo");
ze.setComment(comment);
zo.putArchiveEntry(ze);
zo.write("bar".getBytes(UTF_8));
zo.closeArchiveEntry();
}
try(final ZipFile zip = new ZipFile(FileChannel.open(f.toPath(),
StandardOpenOption.READ))) {
ZipArchiveEntry entry = zip.getEntry("foo");
assertNotNull(entry);
assertEquals(comment, entry.getComment());
}
}{code}
My understanding of {{IOUtils.readRange()}} was incorrect, it can read more
instead of less.
This testcase creates a zip file with a 8050 length comment. In my case
{{IOUtils.readRange()}} returns 8072 bytes instead, although this may depend on
the platform.
The cause is the same as COMPRESS-584, see
[https://github.com/apache/commons-compress/pull/214]. With this
{{IOUtils.readRange()}} fix it works:
{code:java}
public static byte[] readRange(final ReadableByteChannel input, final int len)
throws IOException {
final ByteArrayOutputStream output = new ByteArrayOutputStream();
final ByteBuffer b = ByteBuffer.allocate(Math.min(len, COPY_BUF_SIZE));
int read = 0;
while (read < len) {
// Make sure we never read more than len bytes
b.limit(Math.min(len - read, b.remaining()));
final int readNow = input.read(b);
if (readNow <= 0) {
break;
}
output.write(b.array(), 0, readNow);
b.rewind();
read += readNow;
}{code}
The testcase can be made platform independent by reading from memory and
trickling bytes explicitly like the
{{readRangeFromChannelDoesntReadMoreThanAskedForWhenItGotLessInFirstReadCall()}}
testcase in the PR for COMPRESS-584.
was (Author: matthijsln):
Yes, see the following test case:
{code:java}
@Test
public void testCommentLongerThanCopyBufSize() throws Exception {
final Field COPY_BUF_SIZE = IOUtils.class.getDeclaredField("COPY_BUF_SIZE");
COPY_BUF_SIZE.setAccessible(true);
final int copyBufSize = (int)COPY_BUF_SIZE.get(null);
String repeatingString = "...a comment that is longer than the copy buf
size";
// Repeat the comment string until we are sure to exceed the COPY_BUF_SIZE
int numberOfRepeats = (copyBufSize / repeatingString.length()) + 1;
String comment = String.join("", Collections.nCopies(numberOfRepeats,
repeatingString));
final File f =
File.createTempFile("commons-compress-zipfiletest-longcomment", ".zip");
f.deleteOnExit();
try (final OutputStream o = Files.newOutputStream(f.toPath());
final ZipArchiveOutputStream zo = new ZipArchiveOutputStream(o)) {
ZipArchiveEntry ze = new ZipArchiveEntry("foo");
ze.setComment(comment);
zo.putArchiveEntry(ze);
zo.write("bar".getBytes(UTF_8));
zo.closeArchiveEntry();
}
try(final ZipFile zip = new ZipFile(FileChannel.open(f.toPath(),
StandardOpenOption.READ))) {
ZipArchiveEntry entry = zip.getEntry("foo");
assertNotNull(entry);
assertEquals(comment, entry.getComment());
}
}{code}
My understanding of {{IOUtils.readRange()}} was incorrect, it can read more
instead of less.
This testcase creates a zip file with a 8050 length comment. In my case
{{IOUtils.readRange()}} returns 8072 bytes instead, although this may depend on
the OS.
The cause is the same as COMPRESS-584, see
[https://github.com/apache/commons-compress/pull/214]. With this
{{IOUtils.readRange()}} fix it works:
{code:java}
public static byte[] readRange(final ReadableByteChannel input, final int len)
throws IOException {
final ByteArrayOutputStream output = new ByteArrayOutputStream();
final ByteBuffer b = ByteBuffer.allocate(Math.min(len, COPY_BUF_SIZE));
int read = 0;
while (read < len) {
[jira] [Comment Edited] (COMPRESS-585) ZipFile fails to read a zipfile with a comment or extra data longer than 8024 bytes
[
https://issues.apache.org/jira/browse/COMPRESS-585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17392942#comment-17392942
]
Matthijs Laan edited comment on COMPRESS-585 at 8/4/21, 9:50 AM:
-
Yes, see the following test case:
{code:java}
@Test
public void testCommentLongerThanCopyBufSize() throws Exception {
final Field COPY_BUF_SIZE = IOUtils.class.getDeclaredField("COPY_BUF_SIZE");
COPY_BUF_SIZE.setAccessible(true);
final int copyBufSize = (int)COPY_BUF_SIZE.get(null);
String repeatingString = "...a comment that is longer than the copy buf
size";
// Repeat the comment string until we are sure to exceed the COPY_BUF_SIZE
int numberOfRepeats = (copyBufSize / repeatingString.length()) + 1;
String comment = String.join("", Collections.nCopies(numberOfRepeats,
repeatingString));
final File f =
File.createTempFile("commons-compress-zipfiletest-longcomment", ".zip");
f.deleteOnExit();
try (final OutputStream o = Files.newOutputStream(f.toPath());
final ZipArchiveOutputStream zo = new ZipArchiveOutputStream(o)) {
ZipArchiveEntry ze = new ZipArchiveEntry("foo");
ze.setComment(comment);
zo.putArchiveEntry(ze);
zo.write("bar".getBytes(UTF_8));
zo.closeArchiveEntry();
}
try(final ZipFile zip = new ZipFile(FileChannel.open(f.toPath(),
StandardOpenOption.READ))) {
ZipArchiveEntry entry = zip.getEntry("foo");
assertNotNull(entry);
assertEquals(comment, entry.getComment());
}
}{code}
My understanding of {{IOUtils.readRange()}} was incorrect, it can read more
instead of less.
This testcase creates a zip file with a 8050 length comment. In my case
{{IOUtils.readRange()}} returns 8072 bytes instead, although this may depend on
the OS.
The cause is the same as COMPRESS-584, see
[https://github.com/apache/commons-compress/pull/214]. With this
{{IOUtils.readRange()}} fix it works:
{code:java}
public static byte[] readRange(final ReadableByteChannel input, final int len)
throws IOException {
final ByteArrayOutputStream output = new ByteArrayOutputStream();
final ByteBuffer b = ByteBuffer.allocate(Math.min(len, COPY_BUF_SIZE));
int read = 0;
while (read < len) {
// Make sure we never read more than len bytes
b.limit(Math.min(len - read, b.remaining()));
final int readNow = input.read(b);
if (readNow <= 0) {
break;
}
output.write(b.array(), 0, readNow);
b.rewind();
read += readNow;
}{code}
was (Author: matthijsln):
Yes, see the following test case:
{code:java}
@Test
public void testCommentLongerThanCopyBufSize() throws Exception {
final Field COPY_BUF_SIZE = IOUtils.class.getDeclaredField("COPY_BUF_SIZE");
COPY_BUF_SIZE.setAccessible(true);
final int copyBufSize = (int)COPY_BUF_SIZE.get(null);
String repeatingString = "...a comment that is longer than the copy buf
size";
// Repeat the comment string until we are sure to exceed the COPY_BUF_SIZE
int numberOfRepeats = (copyBufSize / repeatingString.length()) + 1;
String comment = String.join("", Collections.nCopies(numberOfRepeats,
repeatingString));
final File f =
File.createTempFile("commons-compress-zipfiletest-longcomment", ".zip");
f.deleteOnExit();
try (final OutputStream o = Files.newOutputStream(f.toPath());
final ZipArchiveOutputStream zo = new ZipArchiveOutputStream(o)) {
ZipArchiveEntry ze = new ZipArchiveEntry("foo");
ze.setComment(comment);
zo.putArchiveEntry(ze);
zo.write("bar".getBytes(UTF_8));
zo.closeArchiveEntry();
}
try(final ZipFile zip = new ZipFile(FileChannel.open(f.toPath(),
StandardOpenOption.READ))) {
ZipArchiveEntry entry = zip.getEntry("foo");
assertNotNull(entry);
assertEquals(comment, entry.getComment());
}
}{code}
My understanding of {{IOUtils.readRange()}} was incorrect, it reads more
instead of less.
This testcase creates a zip file with a 8050 length comment. In my case
{{IOUtils.readRange()}} returns 8072 bytes instead, although this may depend on
the OS.
The cause is the same as COMPRESS-584, see
[https://github.com/apache/commons-compress/pull/214]. With this
{{IOUtils.readRange()}} fix it works:
{code:java}
public static byte[] readRange(final ReadableByteChannel input, final int len)
throws IOException {
final ByteArrayOutputStream output = new ByteArrayOutputStream();
final ByteBuffer b = ByteBuffer.allocate(Math.min(len, COPY_BUF_SIZE));
int read = 0;
while (read < len) {
// Make sure we never read more than len bytes
b.limit(Math.min(len - read, b.remaining()));
final int readNow = input.read(b);
if (readNow <= 0) {
break;
}
output.write(b.array(), 0,
[jira] [Comment Edited] (COMPRESS-585) ZipFile fails to read a zipfile with a comment or extra data longer than 8024 bytes
[
https://issues.apache.org/jira/browse/COMPRESS-585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17392942#comment-17392942
]
Matthijs Laan edited comment on COMPRESS-585 at 8/4/21, 9:49 AM:
-
Yes, see the following test case:
{code:java}
@Test
public void testCommentLongerThanCopyBufSize() throws Exception {
final Field COPY_BUF_SIZE = IOUtils.class.getDeclaredField("COPY_BUF_SIZE");
COPY_BUF_SIZE.setAccessible(true);
final int copyBufSize = (int)COPY_BUF_SIZE.get(null);
String repeatingString = "...a comment that is longer than the copy buf
size";
// Repeat the comment string until we are sure to exceed the COPY_BUF_SIZE
int numberOfRepeats = (copyBufSize / repeatingString.length()) + 1;
String comment = String.join("", Collections.nCopies(numberOfRepeats,
repeatingString));
final File f =
File.createTempFile("commons-compress-zipfiletest-longcomment", ".zip");
f.deleteOnExit();
try (final OutputStream o = Files.newOutputStream(f.toPath());
final ZipArchiveOutputStream zo = new ZipArchiveOutputStream(o)) {
ZipArchiveEntry ze = new ZipArchiveEntry("foo");
ze.setComment(comment);
zo.putArchiveEntry(ze);
zo.write("bar".getBytes(UTF_8));
zo.closeArchiveEntry();
}
try(final ZipFile zip = new ZipFile(FileChannel.open(f.toPath(),
StandardOpenOption.READ))) {
ZipArchiveEntry entry = zip.getEntry("foo");
assertNotNull(entry);
assertEquals(comment, entry.getComment());
}
}{code}
My understanding of {{IOUtils.readRange()}} was incorrect, it reads more
instead of less.
This testcase creates a zip file with a 8050 length comment. In my case
{{IOUtils.readRange()}} returns 8072 bytes instead, although this may depend on
the OS.
The cause is the same as COMPRESS-584, see
[https://github.com/apache/commons-compress/pull/214]. With this
{{IOUtils.readRange()}} fix it works:
{code:java}
public static byte[] readRange(final ReadableByteChannel input, final int len)
throws IOException {
final ByteArrayOutputStream output = new ByteArrayOutputStream();
final ByteBuffer b = ByteBuffer.allocate(Math.min(len, COPY_BUF_SIZE));
int read = 0;
while (read < len) {
// Make sure we never read more than len bytes
b.limit(Math.min(len - read, b.remaining()));
final int readNow = input.read(b);
if (readNow <= 0) {
break;
}
output.write(b.array(), 0, readNow);
b.rewind();
read += readNow;
}{code}
was (Author: matthijsln):
Yes, see the following test case:
{code:java}
@Test
public void testCommentLongerThanCopyBufSize() throws Exception {
final Field COPY_BUF_SIZE = IOUtils.class.getDeclaredField("COPY_BUF_SIZE");
COPY_BUF_SIZE.setAccessible(true);
final int copyBufSize = (int)COPY_BUF_SIZE.get(null);
String repeatingString = "...a comment that is longer than the copy buf
size";
// Repeat the comment string until we are sure to exceed the COPY_BUF_SIZE
int numberOfRepeats = (copyBufSize / repeatingString.length()) + 1;
String comment = String.join("", Collections.nCopies(numberOfRepeats,
repeatingString));
final File f =
File.createTempFile("commons-compress-zipfiletest-longcomment", ".zip");
f.deleteOnExit();
try (final OutputStream o = Files.newOutputStream(f.toPath());
final ZipArchiveOutputStream zo = new ZipArchiveOutputStream(o)) {
ZipArchiveEntry ze = new ZipArchiveEntry("foo");
ze.setComment(comment);
zo.putArchiveEntry(ze);
zo.write("bar".getBytes(UTF_8));
zo.closeArchiveEntry();
}
try(final ZipFile zip = new ZipFile(FileChannel.open(f.toPath(),
StandardOpenOption.READ))) {
ZipArchiveEntry entry = zip.getEntry("foo");
assertNotNull(entry);
assertEquals(comment, entry.getComment());
}
}{code}
My understanding of {{IOUtils.readRange()}} was incorrect, it reads more
instead of less.
This testcase creates a zip file with a 8050 length comment. In my case
`IOUtils.readRange()` returns 8072 bytes instead, although this may depend on
the `FileChannel` implementation.
The cause is the same as COMPRESS-584, see
[https://github.com/apache/commons-compress/pull/214]. With this
{{IOUtils.readRange()}} fix it works:
{code:java}
public static byte[] readRange(final ReadableByteChannel input, final int len)
throws IOException {
final ByteArrayOutputStream output = new ByteArrayOutputStream();
final ByteBuffer b = ByteBuffer.allocate(Math.min(len, COPY_BUF_SIZE));
int read = 0;
while (read < len) {
// Make sure we never read more than len bytes
b.limit(Math.min(len - read, b.remaining()));
final int readNow = input.read(b);
if (readNow <= 0) {
break;
}
output
[jira] [Commented] (COMPRESS-585) ZipFile fails to read a zipfile with a comment or extra data longer than 8024 bytes
[
https://issues.apache.org/jira/browse/COMPRESS-585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17392942#comment-17392942
]
Matthijs Laan commented on COMPRESS-585:
Yes, see the following test case:
{code:java}
@Test
public void testCommentLongerThanCopyBufSize() throws Exception {
final Field COPY_BUF_SIZE = IOUtils.class.getDeclaredField("COPY_BUF_SIZE");
COPY_BUF_SIZE.setAccessible(true);
final int copyBufSize = (int)COPY_BUF_SIZE.get(null);
String repeatingString = "...a comment that is longer than the copy buf
size";
// Repeat the comment string until we are sure to exceed the COPY_BUF_SIZE
int numberOfRepeats = (copyBufSize / repeatingString.length()) + 1;
String comment = String.join("", Collections.nCopies(numberOfRepeats,
repeatingString));
final File f =
File.createTempFile("commons-compress-zipfiletest-longcomment", ".zip");
f.deleteOnExit();
try (final OutputStream o = Files.newOutputStream(f.toPath());
final ZipArchiveOutputStream zo = new ZipArchiveOutputStream(o)) {
ZipArchiveEntry ze = new ZipArchiveEntry("foo");
ze.setComment(comment);
zo.putArchiveEntry(ze);
zo.write("bar".getBytes(UTF_8));
zo.closeArchiveEntry();
}
try(final ZipFile zip = new ZipFile(FileChannel.open(f.toPath(),
StandardOpenOption.READ))) {
ZipArchiveEntry entry = zip.getEntry("foo");
assertNotNull(entry);
assertEquals(comment, entry.getComment());
}
}{code}
My understanding of {{IOUtils.readRange()}} was incorrect, it reads more
instead of less.
This testcase creates a zip file with a 8050 length comment. In my case
`IOUtils.readRange()` returns 8072 bytes instead, although this may depend on
the `FileChannel` implementation.
The cause is the same as COMPRESS-584, see
[https://github.com/apache/commons-compress/pull/214]. With this
{{IOUtils.readRange()}} fix it works:
{code:java}
public static byte[] readRange(final ReadableByteChannel input, final int len)
throws IOException {
final ByteArrayOutputStream output = new ByteArrayOutputStream();
final ByteBuffer b = ByteBuffer.allocate(Math.min(len, COPY_BUF_SIZE));
int read = 0;
while (read < len) {
// Make sure we never read more than len bytes
b.limit(Math.min(len - read, b.remaining()));
final int readNow = input.read(b);
if (readNow <= 0) {
break;
}
output.write(b.array(), 0, readNow);
b.rewind();
read += readNow;
}{code}
> ZipFile fails to read a zipfile with a comment or extra data longer than 8024
> bytes
> ---
>
> Key: COMPRESS-585
> URL: https://issues.apache.org/jira/browse/COMPRESS-585
> Project: Commons Compress
> Issue Type: Bug
> Components: Archivers
>Affects Versions: 1.21
>Reporter: Matthijs Laan
>Priority: Minor
> Attachments: COMPRESS-585-test.patch
>
>
> See the attached patch for a unit test demonstrating the issue with a long
> comment.
> The cause is that {{ZipFile.readCentralDirectoryEntry()}} calls
> {{IOUtils.readRange()}} and assumes if it returns less than the length it
> asked for that the EOF is reached, however this is not how that method works:
> it returns max COPY_BUF_SIZE bytes (8024), even if EOF has not been reached.
> Besides comments and extra data (in the central directory or local file
> header) longer than 8024 bytes the only other place {{readRange()}} is called
> is reading filenames, but that seems like a remote edge case and an EOF
> exception is fine.
> The IOUtils.readRange() JavaDoc does not specify how it communicates EOF.
> With a blocking channel this would be when it returns a zero length array. It
> could throw an exception when {{Channel.read()}} returns 0 bytes, because
> that only happens on non-blocking channels.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (COMPRESS-584) IOUtils.readRange() can read more from a channel than asked for
[
https://issues.apache.org/jira/browse/COMPRESS-584?focusedWorklogId=633393&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-633393
]
ASF GitHub Bot logged work on COMPRESS-584:
---
Author: ASF GitHub Bot
Created on: 04/Aug/21 09:38
Start Date: 04/Aug/21 09:38
Worklog Time Spent: 10m
Work Description: matthijsln edited a comment on pull request #214:
URL: https://github.com/apache/commons-compress/pull/214#issuecomment-892504852
This would work without a copy buffer:
```
public static byte[] readRange(final ReadableByteChannel input, final
int len) throws IOException {
final ByteBuffer b = ByteBuffer.allocate(len);
while (b.hasRemaining()) {
if (input.read(b) <= 0) {
break;
}
}
return Arrays.copyOf(b.array(), b.position());
}
```
With a copy buffer:
```
public static byte[] readRange(final ReadableByteChannel input, final
int len) throws IOException {
final ByteArrayOutputStream output = new ByteArrayOutputStream();
final ByteBuffer b = ByteBuffer.allocate(Math.min(len,
COPY_BUF_SIZE));
int read = 0;
while (read < len) {
// Make sure we never read more than len bytes
b.limit(Math.min(len - read, b.capacity()));
final int readNow = input.read(b);
if (readNow <= 0) {
break;
}
output.write(b.array(), 0, readNow);
b.rewind();
read += readNow;
}
return output.toByteArray();
}
```
Additional test case for reading more than the copy buffer length (of course
COPY_BUF_SIZE can be made more accessible instead of using reflection):
```
@Test
public void readRangeMoreThanCopyBufferSize() throws Exception {
final Field COPY_BUF_SIZE =
IOUtils.class.getDeclaredField("COPY_BUF_SIZE");
COPY_BUF_SIZE.setAccessible(true);
final int copyBufSize = (int)COPY_BUF_SIZE.get(null);
// Make an input that requires two read loops
byte[] input = new byte[copyBufSize + 10];
try (SeekableByteChannel in = new
SeekableInMemoryByteChannel(input)) {
// Ask for less than the input length, but more than the buffer
size
int toRead = copyBufSize + 1;
byte[] read = IOUtils.readRange(in, toRead);
assertEquals(toRead, read.length);
assertEquals(toRead, in.position());
}
}
```
Hope this is correct :) What version is preferable? I don't think a copy
buffer in itself is needed and maximum memory requirements with a copy buffer
is at least `len * 2 + min(len,COPY_BUF_SIZE)` bytes as
`ByteArrayOutputStream.toByteArray()` eventually calls
`Arrays.copyOf(byte[],int)` which allocates a new byte array. The no copy
buffer version only requires `len * 2` bytes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 633393)
Time Spent: 3h 20m (was: 3h 10m)
> IOUtils.readRange() can read more from a channel than asked for
> ---
>
> Key: COMPRESS-584
> URL: https://issues.apache.org/jira/browse/COMPRESS-584
> Project: Commons Compress
> Issue Type: Bug
> Components: Archivers
>Affects Versions: 1.21
>Reporter: Matthijs Laan
>Priority: Major
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> When {{IOUtils.readRange(ReadableByteChannel,int)}} gets less than the number
> of bytes asked for in the first read call it does not reduce the buffer size
> for the next read call and may read more than asked for.
> This situation is rare when using a {{FileChannel}} but I wrote a
> {{SeekableByteChannel}} backed by a URI using HTTP range requests and reading
> from a socket can often return less bytes than asked for. When I used this
> channel to read a {{ZipFile}} it only read the ZIP central directory
> partially sometimes because {{IOUtils.readRange()}} called from
> {{ZipFile.readCentralDirectoryEntry()}} read more bytes than asked for and it
> stopped parsing directory entries.
> Fix: [https://github.com/apache/commons-compress/pull/214]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [commons-compress] matthijsln edited a comment on pull request #214: [COMPRESS-584] Fix IOUtils.readRange() can read more from a channel than asked for
matthijsln edited a comment on pull request #214:
URL: https://github.com/apache/commons-compress/pull/214#issuecomment-892504852
This would work without a copy buffer:
```
public static byte[] readRange(final ReadableByteChannel input, final
int len) throws IOException {
final ByteBuffer b = ByteBuffer.allocate(len);
while (b.hasRemaining()) {
if (input.read(b) <= 0) {
break;
}
}
return Arrays.copyOf(b.array(), b.position());
}
```
With a copy buffer:
```
public static byte[] readRange(final ReadableByteChannel input, final
int len) throws IOException {
final ByteArrayOutputStream output = new ByteArrayOutputStream();
final ByteBuffer b = ByteBuffer.allocate(Math.min(len,
COPY_BUF_SIZE));
int read = 0;
while (read < len) {
// Make sure we never read more than len bytes
b.limit(Math.min(len - read, b.capacity()));
final int readNow = input.read(b);
if (readNow <= 0) {
break;
}
output.write(b.array(), 0, readNow);
b.rewind();
read += readNow;
}
return output.toByteArray();
}
```
Additional test case for reading more than the copy buffer length (of course
COPY_BUF_SIZE can be made more accessible instead of using reflection):
```
@Test
public void readRangeMoreThanCopyBufferSize() throws Exception {
final Field COPY_BUF_SIZE =
IOUtils.class.getDeclaredField("COPY_BUF_SIZE");
COPY_BUF_SIZE.setAccessible(true);
final int copyBufSize = (int)COPY_BUF_SIZE.get(null);
// Make an input that requires two read loops
byte[] input = new byte[copyBufSize + 10];
try (SeekableByteChannel in = new
SeekableInMemoryByteChannel(input)) {
// Ask for less than the input length, but more than the buffer
size
int toRead = copyBufSize + 1;
byte[] read = IOUtils.readRange(in, toRead);
assertEquals(toRead, read.length);
assertEquals(toRead, in.position());
}
}
```
Hope this is correct :) What version is preferable? I don't think a copy
buffer in itself is needed and maximum memory requirements with a copy buffer
is at least `len * 2 + min(len,COPY_BUF_SIZE)` bytes as
`ByteArrayOutputStream.toByteArray()` eventually calls
`Arrays.copyOf(byte[],int)` which allocates a new byte array. The no copy
buffer version only requires `len * 2` bytes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Work logged] (COMPRESS-584) IOUtils.readRange() can read more from a channel than asked for
[
https://issues.apache.org/jira/browse/COMPRESS-584?focusedWorklogId=633390&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-633390
]
ASF GitHub Bot logged work on COMPRESS-584:
---
Author: ASF GitHub Bot
Created on: 04/Aug/21 09:34
Start Date: 04/Aug/21 09:34
Worklog Time Spent: 10m
Work Description: matthijsln edited a comment on pull request #214:
URL: https://github.com/apache/commons-compress/pull/214#issuecomment-892504852
This would work without a copy buffer:
```
public static byte[] readRange(final ReadableByteChannel input, final
int len) throws IOException {
final ByteBuffer b = ByteBuffer.allocate(len);
while (b.hasRemaining()) {
if (input.read(b) <= 0) {
break;
}
}
return Arrays.copyOf(b.array(), b.position());
}
```
With a copy buffer:
```
public static byte[] readRange(final ReadableByteChannel input, final
int len) throws IOException {
final ByteArrayOutputStream output = new ByteArrayOutputStream();
final ByteBuffer b = ByteBuffer.allocate(Math.min(len,
COPY_BUF_SIZE));
int read = 0;
while (read < len) {
// Make sure we never read more than len bytes
b.limit(Math.min(len - read, b.capacity()));
final int readNow = input.read(b);
if (readNow <= 0) {
break;
}
output.write(b.array(), 0, readNow);
b.rewind();
read += readNow;
}
return output.toByteArray();
}
```
Additional test case for reading more than the copy buffer length (of course
COPY_BUF_SIZE can be made more accessible instead of using reflection):
```
@Test
public void readRangeMoreThanCopyBufferSize() throws Exception {
final Field COPY_BUF_SIZE =
IOUtils.class.getDeclaredField("COPY_BUF_SIZE");
COPY_BUF_SIZE.setAccessible(true);
final int copyBufSize = (int)COPY_BUF_SIZE.get(null);
// Make an input that requires two read loops
byte[] input = new byte[copyBufSize + 10];
try (SeekableByteChannel in = new
SeekableInMemoryByteChannel(input)) {
// Ask for less than the input length, but more than the buffer
size
int toRead = copyBufSize + 1;
byte[] read = IOUtils.readRange(in, toRead);
assertEquals(toRead, read.length);
assertEquals(toRead, in.position());
}
}
```
Hope this is correct :) What version is preferable? I don't think a copy
buffer in itself is needed and maximum memory requirements with a copy buffer
is at least `len * 2 + COPY_BUF_SIZE` bytes as
`ByteArrayOutputStream.toByteArray()` eventually calls
`Arrays.copyOf(byte[],int)` which allocates a new byte array. The no copy
buffer version only requires `len * 2` bytes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 633390)
Time Spent: 3h 10m (was: 3h)
> IOUtils.readRange() can read more from a channel than asked for
> ---
>
> Key: COMPRESS-584
> URL: https://issues.apache.org/jira/browse/COMPRESS-584
> Project: Commons Compress
> Issue Type: Bug
> Components: Archivers
>Affects Versions: 1.21
>Reporter: Matthijs Laan
>Priority: Major
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> When {{IOUtils.readRange(ReadableByteChannel,int)}} gets less than the number
> of bytes asked for in the first read call it does not reduce the buffer size
> for the next read call and may read more than asked for.
> This situation is rare when using a {{FileChannel}} but I wrote a
> {{SeekableByteChannel}} backed by a URI using HTTP range requests and reading
> from a socket can often return less bytes than asked for. When I used this
> channel to read a {{ZipFile}} it only read the ZIP central directory
> partially sometimes because {{IOUtils.readRange()}} called from
> {{ZipFile.readCentralDirectoryEntry()}} read more bytes than asked for and it
> stopped parsing directory entries.
> Fix: [https://github.com/apache/commons-compress/pull/214]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [commons-compress] matthijsln edited a comment on pull request #214: [COMPRESS-584] Fix IOUtils.readRange() can read more from a channel than asked for
matthijsln edited a comment on pull request #214:
URL: https://github.com/apache/commons-compress/pull/214#issuecomment-892504852
This would work without a copy buffer:
```
public static byte[] readRange(final ReadableByteChannel input, final
int len) throws IOException {
final ByteBuffer b = ByteBuffer.allocate(len);
while (b.hasRemaining()) {
if (input.read(b) <= 0) {
break;
}
}
return Arrays.copyOf(b.array(), b.position());
}
```
With a copy buffer:
```
public static byte[] readRange(final ReadableByteChannel input, final
int len) throws IOException {
final ByteArrayOutputStream output = new ByteArrayOutputStream();
final ByteBuffer b = ByteBuffer.allocate(Math.min(len,
COPY_BUF_SIZE));
int read = 0;
while (read < len) {
// Make sure we never read more than len bytes
b.limit(Math.min(len - read, b.capacity()));
final int readNow = input.read(b);
if (readNow <= 0) {
break;
}
output.write(b.array(), 0, readNow);
b.rewind();
read += readNow;
}
return output.toByteArray();
}
```
Additional test case for reading more than the copy buffer length (of course
COPY_BUF_SIZE can be made more accessible instead of using reflection):
```
@Test
public void readRangeMoreThanCopyBufferSize() throws Exception {
final Field COPY_BUF_SIZE =
IOUtils.class.getDeclaredField("COPY_BUF_SIZE");
COPY_BUF_SIZE.setAccessible(true);
final int copyBufSize = (int)COPY_BUF_SIZE.get(null);
// Make an input that requires two read loops
byte[] input = new byte[copyBufSize + 10];
try (SeekableByteChannel in = new
SeekableInMemoryByteChannel(input)) {
// Ask for less than the input length, but more than the buffer
size
int toRead = copyBufSize + 1;
byte[] read = IOUtils.readRange(in, toRead);
assertEquals(toRead, read.length);
assertEquals(toRead, in.position());
}
}
```
Hope this is correct :) What version is preferable? I don't think a copy
buffer in itself is needed and maximum memory requirements with a copy buffer
is at least `len * 2 + COPY_BUF_SIZE` bytes as
`ByteArrayOutputStream.toByteArray()` eventually calls
`Arrays.copyOf(byte[],int)` which allocates a new byte array. The no copy
buffer version only requires `len * 2` bytes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Work logged] (COMPRESS-584) IOUtils.readRange() can read more from a channel than asked for
[
https://issues.apache.org/jira/browse/COMPRESS-584?focusedWorklogId=633388&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-633388
]
ASF GitHub Bot logged work on COMPRESS-584:
---
Author: ASF GitHub Bot
Created on: 04/Aug/21 09:29
Start Date: 04/Aug/21 09:29
Worklog Time Spent: 10m
Work Description: matthijsln edited a comment on pull request #214:
URL: https://github.com/apache/commons-compress/pull/214#issuecomment-892504852
This would work without a copy buffer:
```
public static byte[] readRange(final ReadableByteChannel input, final
int len) throws IOException {
final ByteBuffer b = ByteBuffer.allocate(len);
while (b.hasRemaining()) {
if (input.read(b) <= 0) {
break;
}
}
return Arrays.copyOf(b.array(), b.position());
}
```
With a copy buffer:
```
public static byte[] readRange(final ReadableByteChannel input, final
int len) throws IOException {
final ByteArrayOutputStream output = new ByteArrayOutputStream();
final ByteBuffer b = ByteBuffer.allocate(Math.min(len,
COPY_BUF_SIZE));
int read = 0;
while (read < len) {
// Make sure we never read more than len bytes
b.limit(Math.min(len - read, b.capacity()));
final int readNow = input.read(b);
if (readNow <= 0) {
break;
}
output.write(b.array(), 0, readNow);
b.rewind();
read += readNow;
}
return output.toByteArray();
}
```
Additional test case for reading more than the copy buffer length (of course
COPY_BUF_SIZE can be made more accessible instead of using reflection):
```
@Test
public void readRangeMoreThanCopyBufferSize() throws Exception {
final Field COPY_BUF_SIZE =
IOUtils.class.getDeclaredField("COPY_BUF_SIZE");
COPY_BUF_SIZE.setAccessible(true);
final int copyBufSize = (int)COPY_BUF_SIZE.get(null);
// Make an input that requires two read loops
byte[] input = new byte[copyBufSize + 10];
try (SeekableByteChannel in = new
SeekableInMemoryByteChannel(input)) {
// Ask for less than the input length, but more than the buffer
size
int toRead = copyBufSize + 1;
byte[] read = IOUtils.readRange(in, toRead);
assertEquals(toRead, read.length);
assertEquals(toRead, in.position());
}
}
```
Hope this is correct :) What version is preferable? I don't think a copy
buffer in itself is needed, although the maximum memory requirement is `len *
2` instead of `len + COPY_BUF_SIZE`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 633388)
Time Spent: 3h (was: 2h 50m)
> IOUtils.readRange() can read more from a channel than asked for
> ---
>
> Key: COMPRESS-584
> URL: https://issues.apache.org/jira/browse/COMPRESS-584
> Project: Commons Compress
> Issue Type: Bug
> Components: Archivers
>Affects Versions: 1.21
>Reporter: Matthijs Laan
>Priority: Major
> Time Spent: 3h
> Remaining Estimate: 0h
>
> When {{IOUtils.readRange(ReadableByteChannel,int)}} gets less than the number
> of bytes asked for in the first read call it does not reduce the buffer size
> for the next read call and may read more than asked for.
> This situation is rare when using a {{FileChannel}} but I wrote a
> {{SeekableByteChannel}} backed by a URI using HTTP range requests and reading
> from a socket can often return less bytes than asked for. When I used this
> channel to read a {{ZipFile}} it only read the ZIP central directory
> partially sometimes because {{IOUtils.readRange()}} called from
> {{ZipFile.readCentralDirectoryEntry()}} read more bytes than asked for and it
> stopped parsing directory entries.
> Fix: [https://github.com/apache/commons-compress/pull/214]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [commons-compress] matthijsln edited a comment on pull request #214: [COMPRESS-584] Fix IOUtils.readRange() can read more from a channel than asked for
matthijsln edited a comment on pull request #214:
URL: https://github.com/apache/commons-compress/pull/214#issuecomment-892504852
This would work without a copy buffer:
```
public static byte[] readRange(final ReadableByteChannel input, final
int len) throws IOException {
final ByteBuffer b = ByteBuffer.allocate(len);
while (b.hasRemaining()) {
if (input.read(b) <= 0) {
break;
}
}
return Arrays.copyOf(b.array(), b.position());
}
```
With a copy buffer:
```
public static byte[] readRange(final ReadableByteChannel input, final
int len) throws IOException {
final ByteArrayOutputStream output = new ByteArrayOutputStream();
final ByteBuffer b = ByteBuffer.allocate(Math.min(len,
COPY_BUF_SIZE));
int read = 0;
while (read < len) {
// Make sure we never read more than len bytes
b.limit(Math.min(len - read, b.capacity()));
final int readNow = input.read(b);
if (readNow <= 0) {
break;
}
output.write(b.array(), 0, readNow);
b.rewind();
read += readNow;
}
return output.toByteArray();
}
```
Additional test case for reading more than the copy buffer length (of course
COPY_BUF_SIZE can be made more accessible instead of using reflection):
```
@Test
public void readRangeMoreThanCopyBufferSize() throws Exception {
final Field COPY_BUF_SIZE =
IOUtils.class.getDeclaredField("COPY_BUF_SIZE");
COPY_BUF_SIZE.setAccessible(true);
final int copyBufSize = (int)COPY_BUF_SIZE.get(null);
// Make an input that requires two read loops
byte[] input = new byte[copyBufSize + 10];
try (SeekableByteChannel in = new
SeekableInMemoryByteChannel(input)) {
// Ask for less than the input length, but more than the buffer
size
int toRead = copyBufSize + 1;
byte[] read = IOUtils.readRange(in, toRead);
assertEquals(toRead, read.length);
assertEquals(toRead, in.position());
}
}
```
Hope this is correct :) What version is preferable? I don't think a copy
buffer in itself is needed, although the maximum memory requirement is `len *
2` instead of `len + COPY_BUF_SIZE`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Work logged] (COMPRESS-584) IOUtils.readRange() can read more from a channel than asked for
[
https://issues.apache.org/jira/browse/COMPRESS-584?focusedWorklogId=633386&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-633386
]
ASF GitHub Bot logged work on COMPRESS-584:
---
Author: ASF GitHub Bot
Created on: 04/Aug/21 09:23
Start Date: 04/Aug/21 09:23
Worklog Time Spent: 10m
Work Description: matthijsln edited a comment on pull request #214:
URL: https://github.com/apache/commons-compress/pull/214#issuecomment-892504852
This would work without a copy buffer:
```
public static byte[] readRange(final ReadableByteChannel input, final
int len) throws IOException {
final ByteBuffer b = ByteBuffer.allocate(len);
while (b.hasRemaining()) {
if (input.read(b) <= 0) {
break;
}
}
return Arrays.copyOfRange(b.array(), 0, b.position());
}
```
With a copy buffer:
```
public static byte[] readRange(final ReadableByteChannel input, final
int len) throws IOException {
final ByteArrayOutputStream output = new ByteArrayOutputStream();
final ByteBuffer b = ByteBuffer.allocate(Math.min(len,
COPY_BUF_SIZE));
int read = 0;
while (read < len) {
// Make sure we never read more than len bytes
b.limit(Math.min(len - read, b.capacity()));
final int readNow = input.read(b);
if (readNow <= 0) {
break;
}
output.write(b.array(), 0, readNow);
b.rewind();
read += readNow;
}
return output.toByteArray();
}
```
Additional test case for reading more than the copy buffer length (of course
COPY_BUF_SIZE can be made more accessible instead of using reflection):
```
@Test
public void readRangeMoreThanCopyBufferSize() throws Exception {
final Field COPY_BUF_SIZE =
IOUtils.class.getDeclaredField("COPY_BUF_SIZE");
COPY_BUF_SIZE.setAccessible(true);
final int copyBufSize = (int)COPY_BUF_SIZE.get(null);
// Make an input that requires two read loops
byte[] input = new byte[copyBufSize + 10];
try (SeekableByteChannel in = new
SeekableInMemoryByteChannel(input)) {
// Ask for less than the input length, but more than the buffer
size
int toRead = copyBufSize + 1;
byte[] read = IOUtils.readRange(in, toRead);
assertEquals(toRead, read.length);
assertEquals(toRead, in.position());
}
}
```
Hope this is correct :) What version is preferable? I don't think a copy
buffer in itself is needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 633386)
Time Spent: 2h 50m (was: 2h 40m)
> IOUtils.readRange() can read more from a channel than asked for
> ---
>
> Key: COMPRESS-584
> URL: https://issues.apache.org/jira/browse/COMPRESS-584
> Project: Commons Compress
> Issue Type: Bug
> Components: Archivers
>Affects Versions: 1.21
>Reporter: Matthijs Laan
>Priority: Major
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> When {{IOUtils.readRange(ReadableByteChannel,int)}} gets less than the number
> of bytes asked for in the first read call it does not reduce the buffer size
> for the next read call and may read more than asked for.
> This situation is rare when using a {{FileChannel}} but I wrote a
> {{SeekableByteChannel}} backed by a URI using HTTP range requests and reading
> from a socket can often return less bytes than asked for. When I used this
> channel to read a {{ZipFile}} it only read the ZIP central directory
> partially sometimes because {{IOUtils.readRange()}} called from
> {{ZipFile.readCentralDirectoryEntry()}} read more bytes than asked for and it
> stopped parsing directory entries.
> Fix: [https://github.com/apache/commons-compress/pull/214]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [commons-compress] matthijsln edited a comment on pull request #214: [COMPRESS-584] Fix IOUtils.readRange() can read more from a channel than asked for
matthijsln edited a comment on pull request #214:
URL: https://github.com/apache/commons-compress/pull/214#issuecomment-892504852
This would work without a copy buffer:
```
public static byte[] readRange(final ReadableByteChannel input, final
int len) throws IOException {
final ByteBuffer b = ByteBuffer.allocate(len);
while (b.hasRemaining()) {
if (input.read(b) <= 0) {
break;
}
}
return Arrays.copyOfRange(b.array(), 0, b.position());
}
```
With a copy buffer:
```
public static byte[] readRange(final ReadableByteChannel input, final
int len) throws IOException {
final ByteArrayOutputStream output = new ByteArrayOutputStream();
final ByteBuffer b = ByteBuffer.allocate(Math.min(len,
COPY_BUF_SIZE));
int read = 0;
while (read < len) {
// Make sure we never read more than len bytes
b.limit(Math.min(len - read, b.capacity()));
final int readNow = input.read(b);
if (readNow <= 0) {
break;
}
output.write(b.array(), 0, readNow);
b.rewind();
read += readNow;
}
return output.toByteArray();
}
```
Additional test case for reading more than the copy buffer length (of course
COPY_BUF_SIZE can be made more accessible instead of using reflection):
```
@Test
public void readRangeMoreThanCopyBufferSize() throws Exception {
final Field COPY_BUF_SIZE =
IOUtils.class.getDeclaredField("COPY_BUF_SIZE");
COPY_BUF_SIZE.setAccessible(true);
final int copyBufSize = (int)COPY_BUF_SIZE.get(null);
// Make an input that requires two read loops
byte[] input = new byte[copyBufSize + 10];
try (SeekableByteChannel in = new
SeekableInMemoryByteChannel(input)) {
// Ask for less than the input length, but more than the buffer
size
int toRead = copyBufSize + 1;
byte[] read = IOUtils.readRange(in, toRead);
assertEquals(toRead, read.length);
assertEquals(toRead, in.position());
}
}
```
Hope this is correct :) What version is preferable? I don't think a copy
buffer in itself is needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Work logged] (COMPRESS-584) IOUtils.readRange() can read more from a channel than asked for
[
https://issues.apache.org/jira/browse/COMPRESS-584?focusedWorklogId=633385&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-633385
]
ASF GitHub Bot logged work on COMPRESS-584:
---
Author: ASF GitHub Bot
Created on: 04/Aug/21 09:22
Start Date: 04/Aug/21 09:22
Worklog Time Spent: 10m
Work Description: matthijsln commented on pull request #214:
URL: https://github.com/apache/commons-compress/pull/214#issuecomment-892504852
This would work without a copy buffer:
```
public static byte[] readRange(final ReadableByteChannel input, final
int len) throws IOException {
final ByteBuffer b = ByteBuffer.allocate(len);
while (b.hasRemaining()) {
if (input.read(b) <= 0) {
break;
}
}
return Arrays.copyOfRange(b.array(), 0, b.position());
}
```
With a copy buffer:
```
public static byte[] readRange(final ReadableByteChannel input, final
int len) throws IOException {
final ByteArrayOutputStream output = new ByteArrayOutputStream();
final ByteBuffer b = ByteBuffer.allocate(Math.min(len,
COPY_BUF_SIZE));
int read = 0;
while (read < len) {
// Make sure we never read more than len bytes
b.limit(Math.min(len - read, b.remaining()));
final int readNow = input.read(b);
if (readNow <= 0) {
break;
}
output.write(b.array(), 0, readNow);
b.rewind();
read += readNow;
}
return output.toByteArray();
}
```
Additional test case for reading more than the copy buffer length (of course
COPY_BUF_SIZE can be made more accessible instead of using reflection):
```
@Test
public void readRangeMoreThanCopyBufferSize() throws Exception {
final Field COPY_BUF_SIZE =
IOUtils.class.getDeclaredField("COPY_BUF_SIZE");
COPY_BUF_SIZE.setAccessible(true);
final int copyBufSize = (int)COPY_BUF_SIZE.get(null);
// Make an input that requires two read loops
byte[] input = new byte[copyBufSize + 10];
try (SeekableByteChannel in = new
SeekableInMemoryByteChannel(input)) {
// Ask for less than the input length, but more than the buffer
size
int toRead = copyBufSize + 1;
byte[] read = IOUtils.readRange(in, toRead);
assertEquals(toRead, read.length);
assertEquals(toRead, in.position());
}
}
```
Hope this is correct :) What version is preferable? I don't think a copy
buffer in itself is needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 633385)
Time Spent: 2h 40m (was: 2.5h)
> IOUtils.readRange() can read more from a channel than asked for
> ---
>
> Key: COMPRESS-584
> URL: https://issues.apache.org/jira/browse/COMPRESS-584
> Project: Commons Compress
> Issue Type: Bug
> Components: Archivers
>Affects Versions: 1.21
>Reporter: Matthijs Laan
>Priority: Major
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> When {{IOUtils.readRange(ReadableByteChannel,int)}} gets less than the number
> of bytes asked for in the first read call it does not reduce the buffer size
> for the next read call and may read more than asked for.
> This situation is rare when using a {{FileChannel}} but I wrote a
> {{SeekableByteChannel}} backed by a URI using HTTP range requests and reading
> from a socket can often return less bytes than asked for. When I used this
> channel to read a {{ZipFile}} it only read the ZIP central directory
> partially sometimes because {{IOUtils.readRange()}} called from
> {{ZipFile.readCentralDirectoryEntry()}} read more bytes than asked for and it
> stopped parsing directory entries.
> Fix: [https://github.com/apache/commons-compress/pull/214]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [commons-compress] matthijsln commented on pull request #214: [COMPRESS-584] Fix IOUtils.readRange() can read more from a channel than asked for
matthijsln commented on pull request #214:
URL: https://github.com/apache/commons-compress/pull/214#issuecomment-892504852
This would work without a copy buffer:
```
public static byte[] readRange(final ReadableByteChannel input, final
int len) throws IOException {
final ByteBuffer b = ByteBuffer.allocate(len);
while (b.hasRemaining()) {
if (input.read(b) <= 0) {
break;
}
}
return Arrays.copyOfRange(b.array(), 0, b.position());
}
```
With a copy buffer:
```
public static byte[] readRange(final ReadableByteChannel input, final
int len) throws IOException {
final ByteArrayOutputStream output = new ByteArrayOutputStream();
final ByteBuffer b = ByteBuffer.allocate(Math.min(len,
COPY_BUF_SIZE));
int read = 0;
while (read < len) {
// Make sure we never read more than len bytes
b.limit(Math.min(len - read, b.remaining()));
final int readNow = input.read(b);
if (readNow <= 0) {
break;
}
output.write(b.array(), 0, readNow);
b.rewind();
read += readNow;
}
return output.toByteArray();
}
```
Additional test case for reading more than the copy buffer length (of course
COPY_BUF_SIZE can be made more accessible instead of using reflection):
```
@Test
public void readRangeMoreThanCopyBufferSize() throws Exception {
final Field COPY_BUF_SIZE =
IOUtils.class.getDeclaredField("COPY_BUF_SIZE");
COPY_BUF_SIZE.setAccessible(true);
final int copyBufSize = (int)COPY_BUF_SIZE.get(null);
// Make an input that requires two read loops
byte[] input = new byte[copyBufSize + 10];
try (SeekableByteChannel in = new
SeekableInMemoryByteChannel(input)) {
// Ask for less than the input length, but more than the buffer
size
int toRead = copyBufSize + 1;
byte[] read = IOUtils.readRange(in, toRead);
assertEquals(toRead, read.length);
assertEquals(toRead, in.position());
}
}
```
Hope this is correct :) What version is preferable? I don't think a copy
buffer in itself is needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Work logged] (COMPRESS-584) IOUtils.readRange() can read more from a channel than asked for
[
https://issues.apache.org/jira/browse/COMPRESS-584?focusedWorklogId=633365&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-633365
]
ASF GitHub Bot logged work on COMPRESS-584:
---
Author: ASF GitHub Bot
Created on: 04/Aug/21 08:53
Start Date: 04/Aug/21 08:53
Worklog Time Spent: 10m
Work Description: matthijsln commented on pull request #214:
URL: https://github.com/apache/commons-compress/pull/214#issuecomment-892485686
> > Note that for
[COMPRESS-585](https://issues.apache.org/jira/browse/COMPRESS-585) I propose
removing the limit to the max bytes read to COPY_BUF_SIZE altogether (the same
as the `IOUtils.readRange(InputStream,int)` method) because the calling code
does not expect the method to return less than `len` bytes unless EOF is
encountered.
> > With the limit removed we can use the
[f8587ff](https://github.com/apache/commons-compress/commit/f8587ff16469bcda58db57baafabc9e4639e7f93)
version.
>
> No. The `COPY_BUF_SIZE` is not a limit. It's just a limit of the buffer's
size.
> Note that the `ByteBuffer b` is just a buffer for copying the bytes.
That's why the `rewind` is nessary here - we can not always ensure the `len` is
less than `COPY_BUF_SIZE`.
Ah, I think I get it :) The problem in COMPRESS-585 is the same as
COMPRESS-584 then. The original intent of the method is to return a byte array
of `len` bytes or less if EOF was encountered. I mistakenly thought it was
designed to return max COPY_BUF_SIZE even of no EOF.
To summarize:
- If the first `read()` call returned less than `len`, for the next `read()`
call it could read more than `len`. This can happen reading from a socket.
- If `len` is more than COPY_BUF_SIZE, it would read in more loops and may
read more than `len`. This can happen with an entry comment longer than
COPY_BUF_SIZE.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 633365)
Time Spent: 2.5h (was: 2h 20m)
> IOUtils.readRange() can read more from a channel than asked for
> ---
>
> Key: COMPRESS-584
> URL: https://issues.apache.org/jira/browse/COMPRESS-584
> Project: Commons Compress
> Issue Type: Bug
> Components: Archivers
>Affects Versions: 1.21
>Reporter: Matthijs Laan
>Priority: Major
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> When {{IOUtils.readRange(ReadableByteChannel,int)}} gets less than the number
> of bytes asked for in the first read call it does not reduce the buffer size
> for the next read call and may read more than asked for.
> This situation is rare when using a {{FileChannel}} but I wrote a
> {{SeekableByteChannel}} backed by a URI using HTTP range requests and reading
> from a socket can often return less bytes than asked for. When I used this
> channel to read a {{ZipFile}} it only read the ZIP central directory
> partially sometimes because {{IOUtils.readRange()}} called from
> {{ZipFile.readCentralDirectoryEntry()}} read more bytes than asked for and it
> stopped parsing directory entries.
> Fix: [https://github.com/apache/commons-compress/pull/214]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [commons-compress] matthijsln commented on pull request #214: [COMPRESS-584] Fix IOUtils.readRange() can read more from a channel than asked for
matthijsln commented on pull request #214: URL: https://github.com/apache/commons-compress/pull/214#issuecomment-892485686 > > Note that for [COMPRESS-585](https://issues.apache.org/jira/browse/COMPRESS-585) I propose removing the limit to the max bytes read to COPY_BUF_SIZE altogether (the same as the `IOUtils.readRange(InputStream,int)` method) because the calling code does not expect the method to return less than `len` bytes unless EOF is encountered. > > With the limit removed we can use the [f8587ff](https://github.com/apache/commons-compress/commit/f8587ff16469bcda58db57baafabc9e4639e7f93) version. > > No. The `COPY_BUF_SIZE` is not a limit. It's just a limit of the buffer's size. > Note that the `ByteBuffer b` is just a buffer for copying the bytes. That's why the `rewind` is nessary here - we can not always ensure the `len` is less than `COPY_BUF_SIZE`. Ah, I think I get it :) The problem in COMPRESS-585 is the same as COMPRESS-584 then. The original intent of the method is to return a byte array of `len` bytes or less if EOF was encountered. I mistakenly thought it was designed to return max COPY_BUF_SIZE even of no EOF. To summarize: - If the first `read()` call returned less than `len`, for the next `read()` call it could read more than `len`. This can happen reading from a socket. - If `len` is more than COPY_BUF_SIZE, it would read in more loops and may read more than `len`. This can happen with an entry comment longer than COPY_BUF_SIZE. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (COMPRESS-585) ZipFile fails to read a zipfile with a comment or extra data longer than 8024 bytes
[
https://issues.apache.org/jira/browse/COMPRESS-585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17392876#comment-17392876
]
Peter Lee commented on COMPRESS-585:
_> The cause is that {{ZipFile.readCentralDirectoryEntry()}} calls
{{IOUtils.readRange()}} and assumes if it returns less than the length it asked
for that the EOF is reached, however this is not how that method works: it
returns max COPY_BUF_SIZE bytes (8024), even if EOF has not been reached._
Can this problem be reproduced?
The _COPY_BUF_SIZE_ is just the limit of buffer' size. With _rewind_ called,
the buffer can be reused to copy more than _COPY_BUF_SIZE_ bytes of data.
> ZipFile fails to read a zipfile with a comment or extra data longer than 8024
> bytes
> ---
>
> Key: COMPRESS-585
> URL: https://issues.apache.org/jira/browse/COMPRESS-585
> Project: Commons Compress
> Issue Type: Bug
> Components: Archivers
>Affects Versions: 1.21
>Reporter: Matthijs Laan
>Priority: Minor
> Attachments: COMPRESS-585-test.patch
>
>
> See the attached patch for a unit test demonstrating the issue with a long
> comment.
> The cause is that {{ZipFile.readCentralDirectoryEntry()}} calls
> {{IOUtils.readRange()}} and assumes if it returns less than the length it
> asked for that the EOF is reached, however this is not how that method works:
> it returns max COPY_BUF_SIZE bytes (8024), even if EOF has not been reached.
> Besides comments and extra data (in the central directory or local file
> header) longer than 8024 bytes the only other place {{readRange()}} is called
> is reading filenames, but that seems like a remote edge case and an EOF
> exception is fine.
> The IOUtils.readRange() JavaDoc does not specify how it communicates EOF.
> With a blocking channel this would be when it returns a zero length array. It
> could throw an exception when {{Channel.read()}} returns 0 bytes, because
> that only happens on non-blocking channels.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (COMPRESS-584) IOUtils.readRange() can read more from a channel than asked for
[
https://issues.apache.org/jira/browse/COMPRESS-584?focusedWorklogId=633364&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-633364
]
ASF GitHub Bot logged work on COMPRESS-584:
---
Author: ASF GitHub Bot
Created on: 04/Aug/21 08:41
Start Date: 04/Aug/21 08:41
Worklog Time Spent: 10m
Work Description: PeterAlfredLee commented on pull request #214:
URL: https://github.com/apache/commons-compress/pull/214#issuecomment-892478291
> Note that for
[COMPRESS-585](https://issues.apache.org/jira/browse/COMPRESS-585) I propose
removing the limit to the max bytes read to COPY_BUF_SIZE altogether (the same
as the `IOUtils.readRange(InputStream,int)` method) because the calling code
does not expect the method to return less than `len` bytes unless EOF is
encountered.
>
> With the limit removed we can use the
[f8587ff](https://github.com/apache/commons-compress/commit/f8587ff16469bcda58db57baafabc9e4639e7f93)
version.
No. The `COPY_BUF_SIZE` is not a limit. It's just a limit of the buffer's
size.
Note that the `ByteBuffer b` is just a buffer for copying the bytes. That's
why the `rewind` is nessary here - we can not always ensure the `len` is less
than `COPY_BUF_SIZE`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 633364)
Time Spent: 2h 20m (was: 2h 10m)
> IOUtils.readRange() can read more from a channel than asked for
> ---
>
> Key: COMPRESS-584
> URL: https://issues.apache.org/jira/browse/COMPRESS-584
> Project: Commons Compress
> Issue Type: Bug
> Components: Archivers
>Affects Versions: 1.21
>Reporter: Matthijs Laan
>Priority: Major
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> When {{IOUtils.readRange(ReadableByteChannel,int)}} gets less than the number
> of bytes asked for in the first read call it does not reduce the buffer size
> for the next read call and may read more than asked for.
> This situation is rare when using a {{FileChannel}} but I wrote a
> {{SeekableByteChannel}} backed by a URI using HTTP range requests and reading
> from a socket can often return less bytes than asked for. When I used this
> channel to read a {{ZipFile}} it only read the ZIP central directory
> partially sometimes because {{IOUtils.readRange()}} called from
> {{ZipFile.readCentralDirectoryEntry()}} read more bytes than asked for and it
> stopped parsing directory entries.
> Fix: [https://github.com/apache/commons-compress/pull/214]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [commons-compress] PeterAlfredLee commented on pull request #214: [COMPRESS-584] Fix IOUtils.readRange() can read more from a channel than asked for
PeterAlfredLee commented on pull request #214: URL: https://github.com/apache/commons-compress/pull/214#issuecomment-892478291 > Note that for [COMPRESS-585](https://issues.apache.org/jira/browse/COMPRESS-585) I propose removing the limit to the max bytes read to COPY_BUF_SIZE altogether (the same as the `IOUtils.readRange(InputStream,int)` method) because the calling code does not expect the method to return less than `len` bytes unless EOF is encountered. > > With the limit removed we can use the [f8587ff](https://github.com/apache/commons-compress/commit/f8587ff16469bcda58db57baafabc9e4639e7f93) version. No. The `COPY_BUF_SIZE` is not a limit. It's just a limit of the buffer's size. Note that the `ByteBuffer b` is just a buffer for copying the bytes. That's why the `rewind` is nessary here - we can not always ensure the `len` is less than `COPY_BUF_SIZE`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Work logged] (COMPRESS-584) IOUtils.readRange() can read more from a channel than asked for
[
https://issues.apache.org/jira/browse/COMPRESS-584?focusedWorklogId=633363&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-633363
]
ASF GitHub Bot logged work on COMPRESS-584:
---
Author: ASF GitHub Bot
Created on: 04/Aug/21 08:35
Start Date: 04/Aug/21 08:35
Worklog Time Spent: 10m
Work Description: matthijsln edited a comment on pull request #214:
URL: https://github.com/apache/commons-compress/pull/214#issuecomment-892469732
> Hi @matthijsln
> Please have a look at my reply :-)
>
> > When the len of bytes to read is bigger than COPY_BUF_SIZE, there are
some problems. You can reproduce this by simply set the COPY_BUF_SIZE to a
small value.
Indeed it did not handle a small COPY_BUF_SIZE correctly. If I set it to 1,
the test cases fail because they do not expect it, but with the e50dedf version
it never reads more than COPY_BUF_SIZE and returns correctly.
Note that for
[COMPRESS-585](https://issues.apache.org/jira/browse/COMPRESS-585) I propose
removing the COPY_BUF_SIZE limit to the max bytes to read altogether (the same
as the `IOUtils.readRange(InputStream,int)` method) because the calling code
does not expect the method to return less than `len` bytes unless EOF is
encountered.
With the limit removed we can use the f8587ff version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 633363)
Time Spent: 2h 10m (was: 2h)
> IOUtils.readRange() can read more from a channel than asked for
> ---
>
> Key: COMPRESS-584
> URL: https://issues.apache.org/jira/browse/COMPRESS-584
> Project: Commons Compress
> Issue Type: Bug
> Components: Archivers
>Affects Versions: 1.21
>Reporter: Matthijs Laan
>Priority: Major
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> When {{IOUtils.readRange(ReadableByteChannel,int)}} gets less than the number
> of bytes asked for in the first read call it does not reduce the buffer size
> for the next read call and may read more than asked for.
> This situation is rare when using a {{FileChannel}} but I wrote a
> {{SeekableByteChannel}} backed by a URI using HTTP range requests and reading
> from a socket can often return less bytes than asked for. When I used this
> channel to read a {{ZipFile}} it only read the ZIP central directory
> partially sometimes because {{IOUtils.readRange()}} called from
> {{ZipFile.readCentralDirectoryEntry()}} read more bytes than asked for and it
> stopped parsing directory entries.
> Fix: [https://github.com/apache/commons-compress/pull/214]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [commons-compress] matthijsln edited a comment on pull request #214: [COMPRESS-584] Fix IOUtils.readRange() can read more from a channel than asked for
matthijsln edited a comment on pull request #214: URL: https://github.com/apache/commons-compress/pull/214#issuecomment-892469732 > Hi @matthijsln > Please have a look at my reply :-) > > > When the len of bytes to read is bigger than COPY_BUF_SIZE, there are some problems. You can reproduce this by simply set the COPY_BUF_SIZE to a small value. Indeed it did not handle a small COPY_BUF_SIZE correctly. If I set it to 1, the test cases fail because they do not expect it, but with the e50dedf version it never reads more than COPY_BUF_SIZE and returns correctly. Note that for [COMPRESS-585](https://issues.apache.org/jira/browse/COMPRESS-585) I propose removing the COPY_BUF_SIZE limit to the max bytes to read altogether (the same as the `IOUtils.readRange(InputStream,int)` method) because the calling code does not expect the method to return less than `len` bytes unless EOF is encountered. With the limit removed we can use the f8587ff version. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Work logged] (COMPRESS-584) IOUtils.readRange() can read more from a channel than asked for
[
https://issues.apache.org/jira/browse/COMPRESS-584?focusedWorklogId=633361&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-633361
]
ASF GitHub Bot logged work on COMPRESS-584:
---
Author: ASF GitHub Bot
Created on: 04/Aug/21 08:31
Start Date: 04/Aug/21 08:31
Worklog Time Spent: 10m
Work Description: matthijsln edited a comment on pull request #214:
URL: https://github.com/apache/commons-compress/pull/214#issuecomment-892469732
> Hi @matthijsln
> Please have a look at my reply :-)
>
> > When the len of bytes to read is bigger than COPY_BUF_SIZE, there are
some problems. You can reproduce this by simply set the COPY_BUF_SIZE to a
small value.
Indeed it did not handle a small COPY_BUF_SIZE correctly. If I set it to 1,
the test cases fail because they do not expect it, but with the e50dedf version
it never reads more than COPY_BUF_SIZE and returns correctly.
Note that for
[COMPRESS-585](https://issues.apache.org/jira/browse/COMPRESS-585) I propose
removing the limit to the max bytes read to COPY_BUF_SIZE altogether (the same
as the `IOUtils.readRange(InputStream,int)` method) because the calling code
does not expect the method to return less than `len` bytes unless EOF is
encountered.
With the limit removed we can use the f8587ff version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 633361)
Time Spent: 2h (was: 1h 50m)
> IOUtils.readRange() can read more from a channel than asked for
> ---
>
> Key: COMPRESS-584
> URL: https://issues.apache.org/jira/browse/COMPRESS-584
> Project: Commons Compress
> Issue Type: Bug
> Components: Archivers
>Affects Versions: 1.21
>Reporter: Matthijs Laan
>Priority: Major
> Time Spent: 2h
> Remaining Estimate: 0h
>
> When {{IOUtils.readRange(ReadableByteChannel,int)}} gets less than the number
> of bytes asked for in the first read call it does not reduce the buffer size
> for the next read call and may read more than asked for.
> This situation is rare when using a {{FileChannel}} but I wrote a
> {{SeekableByteChannel}} backed by a URI using HTTP range requests and reading
> from a socket can often return less bytes than asked for. When I used this
> channel to read a {{ZipFile}} it only read the ZIP central directory
> partially sometimes because {{IOUtils.readRange()}} called from
> {{ZipFile.readCentralDirectoryEntry()}} read more bytes than asked for and it
> stopped parsing directory entries.
> Fix: [https://github.com/apache/commons-compress/pull/214]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [commons-compress] matthijsln edited a comment on pull request #214: [COMPRESS-584] Fix IOUtils.readRange() can read more from a channel than asked for
matthijsln edited a comment on pull request #214: URL: https://github.com/apache/commons-compress/pull/214#issuecomment-892469732 > Hi @matthijsln > Please have a look at my reply :-) > > > When the len of bytes to read is bigger than COPY_BUF_SIZE, there are some problems. You can reproduce this by simply set the COPY_BUF_SIZE to a small value. Indeed it did not handle a small COPY_BUF_SIZE correctly. If I set it to 1, the test cases fail because they do not expect it, but with the e50dedf version it never reads more than COPY_BUF_SIZE and returns correctly. Note that for [COMPRESS-585](https://issues.apache.org/jira/browse/COMPRESS-585) I propose removing the limit to the max bytes read to COPY_BUF_SIZE altogether (the same as the `IOUtils.readRange(InputStream,int)` method) because the calling code does not expect the method to return less than `len` bytes unless EOF is encountered. With the limit removed we can use the f8587ff version. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Work logged] (COMPRESS-584) IOUtils.readRange() can read more from a channel than asked for
[
https://issues.apache.org/jira/browse/COMPRESS-584?focusedWorklogId=633360&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-633360
]
ASF GitHub Bot logged work on COMPRESS-584:
---
Author: ASF GitHub Bot
Created on: 04/Aug/21 08:29
Start Date: 04/Aug/21 08:29
Worklog Time Spent: 10m
Work Description: matthijsln edited a comment on pull request #214:
URL: https://github.com/apache/commons-compress/pull/214#issuecomment-892469732
> Hi @matthijsln
> Please have a look at my reply :-)
>
> > When the len of bytes to read is bigger than COPY_BUF_SIZE, there are
some problems. You can reproduce this by simply set the COPY_BUF_SIZE to a
small value.
Indeed it did not handle a small COPY_BUF_SIZE correctly. If I set it to 1,
the test cases fail because they do not expect it, but with the e50dedf version
it never reads more than COPY_BUF_SIZE and returns correctly.
Note that for
[COMPRESS-585](https://issues.apache.org/jira/browse/COMPRESS-585) I propose
removing the limit to the max bytes read to COPY_BUF_SIZE altogether (the same
as the `IOUtils.readRange(InputStream,int)` method) because the calling code
does not expect the method to return less than `len` bytes unless EOF is
encountered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 633360)
Time Spent: 1h 50m (was: 1h 40m)
> IOUtils.readRange() can read more from a channel than asked for
> ---
>
> Key: COMPRESS-584
> URL: https://issues.apache.org/jira/browse/COMPRESS-584
> Project: Commons Compress
> Issue Type: Bug
> Components: Archivers
>Affects Versions: 1.21
>Reporter: Matthijs Laan
>Priority: Major
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> When {{IOUtils.readRange(ReadableByteChannel,int)}} gets less than the number
> of bytes asked for in the first read call it does not reduce the buffer size
> for the next read call and may read more than asked for.
> This situation is rare when using a {{FileChannel}} but I wrote a
> {{SeekableByteChannel}} backed by a URI using HTTP range requests and reading
> from a socket can often return less bytes than asked for. When I used this
> channel to read a {{ZipFile}} it only read the ZIP central directory
> partially sometimes because {{IOUtils.readRange()}} called from
> {{ZipFile.readCentralDirectoryEntry()}} read more bytes than asked for and it
> stopped parsing directory entries.
> Fix: [https://github.com/apache/commons-compress/pull/214]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [commons-compress] matthijsln edited a comment on pull request #214: [COMPRESS-584] Fix IOUtils.readRange() can read more from a channel than asked for
matthijsln edited a comment on pull request #214: URL: https://github.com/apache/commons-compress/pull/214#issuecomment-892469732 > Hi @matthijsln > Please have a look at my reply :-) > > > When the len of bytes to read is bigger than COPY_BUF_SIZE, there are some problems. You can reproduce this by simply set the COPY_BUF_SIZE to a small value. Indeed it did not handle a small COPY_BUF_SIZE correctly. If I set it to 1, the test cases fail because they do not expect it, but with the e50dedf version it never reads more than COPY_BUF_SIZE and returns correctly. Note that for [COMPRESS-585](https://issues.apache.org/jira/browse/COMPRESS-585) I propose removing the limit to the max bytes read to COPY_BUF_SIZE altogether (the same as the `IOUtils.readRange(InputStream,int)` method) because the calling code does not expect the method to return less than `len` bytes unless EOF is encountered. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Work logged] (COMPRESS-584) IOUtils.readRange() can read more from a channel than asked for
[
https://issues.apache.org/jira/browse/COMPRESS-584?focusedWorklogId=633358&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-633358
]
ASF GitHub Bot logged work on COMPRESS-584:
---
Author: ASF GitHub Bot
Created on: 04/Aug/21 08:28
Start Date: 04/Aug/21 08:28
Worklog Time Spent: 10m
Work Description: matthijsln edited a comment on pull request #214:
URL: https://github.com/apache/commons-compress/pull/214#issuecomment-892469732
> Hi @matthijsln
> Please have a look at my reply :-)
>
> > When the len of bytes to read is bigger than COPY_BUF_SIZE, there are
some problems. You can reproduce this by simply set the COPY_BUF_SIZE to a
small value.
Indeed it did not handle a small COPY_BUF_SIZE correctly. If I set it to 1,
the test cases fail because they do not expect it, but with the e50dedf version
it never reads more than COPY_BUF_SIZE and returns correctly.
Note that in
[COMPRESS-585](https://issues.apache.org/jira/browse/COMPRESS-585) I propose
limiting the number of bytes read to COPY_BUF_SIZE altogether (the same as the
`IOUtils.readRange(InputStream,int)` method) because the calling code does not
expect the method to return less than `len` bytes unless EOF is encountered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 633358)
Time Spent: 1h 40m (was: 1.5h)
> IOUtils.readRange() can read more from a channel than asked for
> ---
>
> Key: COMPRESS-584
> URL: https://issues.apache.org/jira/browse/COMPRESS-584
> Project: Commons Compress
> Issue Type: Bug
> Components: Archivers
>Affects Versions: 1.21
>Reporter: Matthijs Laan
>Priority: Major
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> When {{IOUtils.readRange(ReadableByteChannel,int)}} gets less than the number
> of bytes asked for in the first read call it does not reduce the buffer size
> for the next read call and may read more than asked for.
> This situation is rare when using a {{FileChannel}} but I wrote a
> {{SeekableByteChannel}} backed by a URI using HTTP range requests and reading
> from a socket can often return less bytes than asked for. When I used this
> channel to read a {{ZipFile}} it only read the ZIP central directory
> partially sometimes because {{IOUtils.readRange()}} called from
> {{ZipFile.readCentralDirectoryEntry()}} read more bytes than asked for and it
> stopped parsing directory entries.
> Fix: [https://github.com/apache/commons-compress/pull/214]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (COMPRESS-584) IOUtils.readRange() can read more from a channel than asked for
[
https://issues.apache.org/jira/browse/COMPRESS-584?focusedWorklogId=633357&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-633357
]
ASF GitHub Bot logged work on COMPRESS-584:
---
Author: ASF GitHub Bot
Created on: 04/Aug/21 08:28
Start Date: 04/Aug/21 08:28
Worklog Time Spent: 10m
Work Description: matthijsln commented on pull request #214:
URL: https://github.com/apache/commons-compress/pull/214#issuecomment-892469732
> Hi @matthijsln
> Please have a look at my reply :-)
>
> > When the len of bytes to read is bigger than COPY_BUF_SIZE, there are
some problems. You can reproduce this by simply set the COPY_BUF_SIZE to a
small value.
Indeed it did not handle a small COPY_BUF_SIZE correctly. If I set it to 1,
the test cases fail because they do not expect it, but with the e50dedf version
it never reads more than COPY_BUF_SIZE.
Note that in
[COMPRESS-585](https://issues.apache.org/jira/browse/COMPRESS-585) I propose
limiting the number of bytes read to COPY_BUF_SIZE altogether (the same as the
`IOUtils.readRange(InputStream,int)` method) because the calling code does not
expect the method to return less than `len` bytes unless EOF is encountered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 633357)
Time Spent: 1.5h (was: 1h 20m)
> IOUtils.readRange() can read more from a channel than asked for
> ---
>
> Key: COMPRESS-584
> URL: https://issues.apache.org/jira/browse/COMPRESS-584
> Project: Commons Compress
> Issue Type: Bug
> Components: Archivers
>Affects Versions: 1.21
>Reporter: Matthijs Laan
>Priority: Major
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> When {{IOUtils.readRange(ReadableByteChannel,int)}} gets less than the number
> of bytes asked for in the first read call it does not reduce the buffer size
> for the next read call and may read more than asked for.
> This situation is rare when using a {{FileChannel}} but I wrote a
> {{SeekableByteChannel}} backed by a URI using HTTP range requests and reading
> from a socket can often return less bytes than asked for. When I used this
> channel to read a {{ZipFile}} it only read the ZIP central directory
> partially sometimes because {{IOUtils.readRange()}} called from
> {{ZipFile.readCentralDirectoryEntry()}} read more bytes than asked for and it
> stopped parsing directory entries.
> Fix: [https://github.com/apache/commons-compress/pull/214]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [commons-compress] matthijsln edited a comment on pull request #214: [COMPRESS-584] Fix IOUtils.readRange() can read more from a channel than asked for
matthijsln edited a comment on pull request #214: URL: https://github.com/apache/commons-compress/pull/214#issuecomment-892469732 > Hi @matthijsln > Please have a look at my reply :-) > > > When the len of bytes to read is bigger than COPY_BUF_SIZE, there are some problems. You can reproduce this by simply set the COPY_BUF_SIZE to a small value. Indeed it did not handle a small COPY_BUF_SIZE correctly. If I set it to 1, the test cases fail because they do not expect it, but with the e50dedf version it never reads more than COPY_BUF_SIZE and returns correctly. Note that in [COMPRESS-585](https://issues.apache.org/jira/browse/COMPRESS-585) I propose limiting the number of bytes read to COPY_BUF_SIZE altogether (the same as the `IOUtils.readRange(InputStream,int)` method) because the calling code does not expect the method to return less than `len` bytes unless EOF is encountered. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [commons-compress] matthijsln commented on pull request #214: [COMPRESS-584] Fix IOUtils.readRange() can read more from a channel than asked for
matthijsln commented on pull request #214: URL: https://github.com/apache/commons-compress/pull/214#issuecomment-892469732 > Hi @matthijsln > Please have a look at my reply :-) > > > When the len of bytes to read is bigger than COPY_BUF_SIZE, there are some problems. You can reproduce this by simply set the COPY_BUF_SIZE to a small value. Indeed it did not handle a small COPY_BUF_SIZE correctly. If I set it to 1, the test cases fail because they do not expect it, but with the e50dedf version it never reads more than COPY_BUF_SIZE. Note that in [COMPRESS-585](https://issues.apache.org/jira/browse/COMPRESS-585) I propose limiting the number of bytes read to COPY_BUF_SIZE altogether (the same as the `IOUtils.readRange(InputStream,int)` method) because the calling code does not expect the method to return less than `len` bytes unless EOF is encountered. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Work logged] (COMPRESS-584) IOUtils.readRange() can read more from a channel than asked for
[
https://issues.apache.org/jira/browse/COMPRESS-584?focusedWorklogId=633355&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-633355
]
ASF GitHub Bot logged work on COMPRESS-584:
---
Author: ASF GitHub Bot
Created on: 04/Aug/21 08:23
Start Date: 04/Aug/21 08:23
Worklog Time Spent: 10m
Work Description: PeterAlfredLee commented on pull request #214:
URL: https://github.com/apache/commons-compress/pull/214#issuecomment-892466407
Hi @matthijsln
Please have a look at my reply :-)
> When the len of bytes to read is bigger than COPY_BUF_SIZE, there are some
problems. You can reproduce this by simply set the COPY_BUF_SIZE to a small
value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 633355)
Time Spent: 1h 20m (was: 1h 10m)
> IOUtils.readRange() can read more from a channel than asked for
> ---
>
> Key: COMPRESS-584
> URL: https://issues.apache.org/jira/browse/COMPRESS-584
> Project: Commons Compress
> Issue Type: Bug
> Components: Archivers
>Affects Versions: 1.21
>Reporter: Matthijs Laan
>Priority: Major
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> When {{IOUtils.readRange(ReadableByteChannel,int)}} gets less than the number
> of bytes asked for in the first read call it does not reduce the buffer size
> for the next read call and may read more than asked for.
> This situation is rare when using a {{FileChannel}} but I wrote a
> {{SeekableByteChannel}} backed by a URI using HTTP range requests and reading
> from a socket can often return less bytes than asked for. When I used this
> channel to read a {{ZipFile}} it only read the ZIP central directory
> partially sometimes because {{IOUtils.readRange()}} called from
> {{ZipFile.readCentralDirectoryEntry()}} read more bytes than asked for and it
> stopped parsing directory entries.
> Fix: [https://github.com/apache/commons-compress/pull/214]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [commons-compress] PeterAlfredLee commented on pull request #214: [COMPRESS-584] Fix IOUtils.readRange() can read more from a channel than asked for
PeterAlfredLee commented on pull request #214: URL: https://github.com/apache/commons-compress/pull/214#issuecomment-892466407 Hi @matthijsln Please have a look at my reply :-) > When the len of bytes to read is bigger than COPY_BUF_SIZE, there are some problems. You can reproduce this by simply set the COPY_BUF_SIZE to a small value. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Work logged] (COMPRESS-584) IOUtils.readRange() can read more from a channel than asked for
[
https://issues.apache.org/jira/browse/COMPRESS-584?focusedWorklogId=633347&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-633347
]
ASF GitHub Bot logged work on COMPRESS-584:
---
Author: ASF GitHub Bot
Created on: 04/Aug/21 08:13
Start Date: 04/Aug/21 08:13
Worklog Time Spent: 10m
Work Description: matthijsln commented on pull request #214:
URL: https://github.com/apache/commons-compress/pull/214#issuecomment-892460047
> Unfortunately my original fix had a bug when looping more than 2 times.
Revisiting this method I came up with a much simpler implementation that should
be better. I can squash the previous version and force-push if this looks
better (or it can be squash-merged).
Third time's a charm...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 633347)
Time Spent: 1h 10m (was: 1h)
> IOUtils.readRange() can read more from a channel than asked for
> ---
>
> Key: COMPRESS-584
> URL: https://issues.apache.org/jira/browse/COMPRESS-584
> Project: Commons Compress
> Issue Type: Bug
> Components: Archivers
>Affects Versions: 1.21
>Reporter: Matthijs Laan
>Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> When {{IOUtils.readRange(ReadableByteChannel,int)}} gets less than the number
> of bytes asked for in the first read call it does not reduce the buffer size
> for the next read call and may read more than asked for.
> This situation is rare when using a {{FileChannel}} but I wrote a
> {{SeekableByteChannel}} backed by a URI using HTTP range requests and reading
> from a socket can often return less bytes than asked for. When I used this
> channel to read a {{ZipFile}} it only read the ZIP central directory
> partially sometimes because {{IOUtils.readRange()}} called from
> {{ZipFile.readCentralDirectoryEntry()}} read more bytes than asked for and it
> stopped parsing directory entries.
> Fix: [https://github.com/apache/commons-compress/pull/214]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (COMPRESS-584) IOUtils.readRange() can read more from a channel than asked for
[
https://issues.apache.org/jira/browse/COMPRESS-584?focusedWorklogId=633345&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-633345
]
ASF GitHub Bot logged work on COMPRESS-584:
---
Author: ASF GitHub Bot
Created on: 04/Aug/21 08:13
Start Date: 04/Aug/21 08:13
Worklog Time Spent: 10m
Work Description: PeterAlfredLee commented on pull request #214:
URL: https://github.com/apache/commons-compress/pull/214#issuecomment-892459986
The new implemention also has some problems :
1. The test `readRangeFromChannelStopsIfThereIsNothingToReadAnymore` is
failed
2. When the `len` of bytes to read is bigger than `COPY_BUF_SIZE`, there are
some problems. You can reproduce this by simply set the `COPY_BUF_SIZE` to a
small value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 633345)
Time Spent: 1h (was: 50m)
> IOUtils.readRange() can read more from a channel than asked for
> ---
>
> Key: COMPRESS-584
> URL: https://issues.apache.org/jira/browse/COMPRESS-584
> Project: Commons Compress
> Issue Type: Bug
> Components: Archivers
>Affects Versions: 1.21
>Reporter: Matthijs Laan
>Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
> When {{IOUtils.readRange(ReadableByteChannel,int)}} gets less than the number
> of bytes asked for in the first read call it does not reduce the buffer size
> for the next read call and may read more than asked for.
> This situation is rare when using a {{FileChannel}} but I wrote a
> {{SeekableByteChannel}} backed by a URI using HTTP range requests and reading
> from a socket can often return less bytes than asked for. When I used this
> channel to read a {{ZipFile}} it only read the ZIP central directory
> partially sometimes because {{IOUtils.readRange()}} called from
> {{ZipFile.readCentralDirectoryEntry()}} read more bytes than asked for and it
> stopped parsing directory entries.
> Fix: [https://github.com/apache/commons-compress/pull/214]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [commons-compress] matthijsln commented on pull request #214: [COMPRESS-584] Fix IOUtils.readRange() can read more from a channel than asked for
matthijsln commented on pull request #214: URL: https://github.com/apache/commons-compress/pull/214#issuecomment-892460047 > Unfortunately my original fix had a bug when looping more than 2 times. Revisiting this method I came up with a much simpler implementation that should be better. I can squash the previous version and force-push if this looks better (or it can be squash-merged). Third time's a charm... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [commons-compress] PeterAlfredLee commented on pull request #214: [COMPRESS-584] Fix IOUtils.readRange() can read more from a channel than asked for
PeterAlfredLee commented on pull request #214: URL: https://github.com/apache/commons-compress/pull/214#issuecomment-892459986 The new implemention also has some problems : 1. The test `readRangeFromChannelStopsIfThereIsNothingToReadAnymore` is failed 2. When the `len` of bytes to read is bigger than `COPY_BUF_SIZE`, there are some problems. You can reproduce this by simply set the `COPY_BUF_SIZE` to a small value. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (COMPRESS-586) commons-compress/security-reports.html lost 1.21 fix information
[ https://issues.apache.org/jira/browse/COMPRESS-586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] swxEmily updated COMPRESS-586: -- Attachment: image-2021-08-04-16-08-42-786.png Component/s: Documentation Affects Version/s: 1.21 Description: Just the other day, I saw the 1.21 fix information:CVE-2021-35515, CVE-2021-35516, CVE-2021-35517, CVE-2021-36090 in [https://commons.apache.org/proper/commons-compress/security-reports.html,] but now it disappeared. Why delete the information? !image-2021-08-04-16-08-42-786.png! was: Just the other day, I saw the 1.21 fix information:CVE-2021-35515, CVE-2021-35516, CVE-2021-35517, CVE-2021-36090 in [https://commons.apache.org/proper/commons-compress/security-reports.html,] but now it disappeared. Why delete the information? Labels: documentation (was: ) > commons-compress/security-reports.html lost 1.21 fix information > > > Key: COMPRESS-586 > URL: https://issues.apache.org/jira/browse/COMPRESS-586 > Project: Commons Compress > Issue Type: Wish > Components: Documentation >Affects Versions: 1.21 >Reporter: swxEmily >Priority: Major > Labels: documentation > Attachments: image-2021-08-04-16-08-42-786.png > > > Just the other day, I saw the 1.21 fix information:CVE-2021-35515, > CVE-2021-35516, CVE-2021-35517, CVE-2021-36090 in > [https://commons.apache.org/proper/commons-compress/security-reports.html,] > but now it disappeared. Why delete the information? > !image-2021-08-04-16-08-42-786.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (COMPRESS-586) commons-compress/security-reports.html lost 1.21 fix information
[ https://issues.apache.org/jira/browse/COMPRESS-586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] swxEmily updated COMPRESS-586: -- Description: Just the other day, I saw the 1.21 fix information:CVE-2021-35515, CVE-2021-35516, CVE-2021-35517, CVE-2021-36090 in [https://commons.apache.org/proper/commons-compress/security-reports.html,] but now it disappeared. Why delete the information? was:Just the other day, I saw the 1.21 fix information in [https://commons.apache.org/proper/commons-compress/security-reports.html,] but now it disappeared. Why delete the information? > commons-compress/security-reports.html lost 1.21 fix information > > > Key: COMPRESS-586 > URL: https://issues.apache.org/jira/browse/COMPRESS-586 > Project: Commons Compress > Issue Type: Wish >Reporter: swxEmily >Priority: Major > > Just the other day, I saw the 1.21 fix information:CVE-2021-35515, > CVE-2021-35516, CVE-2021-35517, CVE-2021-36090 in > [https://commons.apache.org/proper/commons-compress/security-reports.html,] > but now it disappeared. Why delete the information? > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (COMPRESS-586) commons-compress/security-reports.html lost 1.21 fix information
swxEmily created COMPRESS-586: - Summary: commons-compress/security-reports.html lost 1.21 fix information Key: COMPRESS-586 URL: https://issues.apache.org/jira/browse/COMPRESS-586 Project: Commons Compress Issue Type: Wish Reporter: swxEmily Just the other day, I saw the 1.21 fix information in [https://commons.apache.org/proper/commons-compress/security-reports.html,] but now it disappeared. Why delete the information? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (COMPRESS-585) ZipFile fails to read a zipfile with a comment or extra data longer than 8024 bytes
[
https://issues.apache.org/jira/browse/COMPRESS-585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17392831#comment-17392831
]
Matthijs Laan commented on COMPRESS-585:
We could just allocate {{len}} bytes instead of {{Math.min(len,
COPY_BUF_SIZE)}} as the analogous {{IOUtils.readRange(InputStream,int)}} method
also does not enforce a limit to the {{ByteArrayOutputStream}} size.
> ZipFile fails to read a zipfile with a comment or extra data longer than 8024
> bytes
> ---
>
> Key: COMPRESS-585
> URL: https://issues.apache.org/jira/browse/COMPRESS-585
> Project: Commons Compress
> Issue Type: Bug
> Components: Archivers
>Affects Versions: 1.21
>Reporter: Matthijs Laan
>Priority: Minor
> Attachments: COMPRESS-585-test.patch
>
>
> See the attached patch for a unit test demonstrating the issue with a long
> comment.
> The cause is that {{ZipFile.readCentralDirectoryEntry()}} calls
> {{IOUtils.readRange()}} and assumes if it returns less than the length it
> asked for that the EOF is reached, however this is not how that method works:
> it returns max COPY_BUF_SIZE bytes (8024), even if EOF has not been reached.
> Besides comments and extra data (in the central directory or local file
> header) longer than 8024 bytes the only other place {{readRange()}} is called
> is reading filenames, but that seems like a remote edge case and an EOF
> exception is fine.
> The IOUtils.readRange() JavaDoc does not specify how it communicates EOF.
> With a blocking channel this would be when it returns a zero length array. It
> could throw an exception when {{Channel.read()}} returns 0 bytes, because
> that only happens on non-blocking channels.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (COMPRESS-584) IOUtils.readRange() can read more from a channel than asked for
[
https://issues.apache.org/jira/browse/COMPRESS-584?focusedWorklogId=65&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-65
]
ASF GitHub Bot logged work on COMPRESS-584:
---
Author: ASF GitHub Bot
Created on: 04/Aug/21 07:59
Start Date: 04/Aug/21 07:59
Worklog Time Spent: 10m
Work Description: matthijsln commented on pull request #214:
URL: https://github.com/apache/commons-compress/pull/214#issuecomment-892450747
Unfortunately my original fix had a bug when looping more than 2 times.
Revisiting this method I came up with a much simpler implementation that should
be better. I can squash the previous version and force-push if this looks
better (or it can be squash-merged).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 65)
Time Spent: 50m (was: 40m)
> IOUtils.readRange() can read more from a channel than asked for
> ---
>
> Key: COMPRESS-584
> URL: https://issues.apache.org/jira/browse/COMPRESS-584
> Project: Commons Compress
> Issue Type: Bug
> Components: Archivers
>Affects Versions: 1.21
>Reporter: Matthijs Laan
>Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
> When {{IOUtils.readRange(ReadableByteChannel,int)}} gets less than the number
> of bytes asked for in the first read call it does not reduce the buffer size
> for the next read call and may read more than asked for.
> This situation is rare when using a {{FileChannel}} but I wrote a
> {{SeekableByteChannel}} backed by a URI using HTTP range requests and reading
> from a socket can often return less bytes than asked for. When I used this
> channel to read a {{ZipFile}} it only read the ZIP central directory
> partially sometimes because {{IOUtils.readRange()}} called from
> {{ZipFile.readCentralDirectoryEntry()}} read more bytes than asked for and it
> stopped parsing directory entries.
> Fix: [https://github.com/apache/commons-compress/pull/214]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [commons-compress] matthijsln commented on pull request #214: [COMPRESS-584] Fix IOUtils.readRange() can read more from a channel than asked for
matthijsln commented on pull request #214: URL: https://github.com/apache/commons-compress/pull/214#issuecomment-892450747 Unfortunately my original fix had a bug when looping more than 2 times. Revisiting this method I came up with a much simpler implementation that should be better. I can squash the previous version and force-push if this looks better (or it can be squash-merged). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Work logged] (COMPRESS-584) IOUtils.readRange() can read more from a channel than asked for
[
https://issues.apache.org/jira/browse/COMPRESS-584?focusedWorklogId=633326&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-633326
]
ASF GitHub Bot logged work on COMPRESS-584:
---
Author: ASF GitHub Bot
Created on: 04/Aug/21 07:13
Start Date: 04/Aug/21 07:13
Worklog Time Spent: 10m
Work Description: PeterAlfredLee commented on a change in pull request
#214:
URL: https://github.com/apache/commons-compress/pull/214#discussion_r682349198
##
File path: src/test/java/org/apache/commons/compress/utils/IOUtilsTest.java
##
@@ -148,6 +148,25 @@ public void
readRangeFromChannelDoesntReadMoreThanAskedFor() throws IOException
}
}
+@Test
+public void
readRangeFromChannelDoesntReadMoreThanAskedForWhenItGotLessInFirstReadCall()
throws IOException {
+try (ReadableByteChannel in = new SeekableInMemoryByteChannel(new
byte[] { 1, 2, 3, 4, 5 }) {
+@Override
+public int read(ByteBuffer buf) throws IOException {
+// Read max 3 bytes at a time
+ByteBuffer temp = ByteBuffer.allocate(Math.min(3,
buf.remaining()));
Review comment:
I think we can have a `final` here.
Even through this is just a test, it's always appreciated if `final` is
always used if possible.
##
File path: src/test/java/org/apache/commons/compress/utils/IOUtilsTest.java
##
@@ -148,6 +148,25 @@ public void
readRangeFromChannelDoesntReadMoreThanAskedFor() throws IOException
}
}
+@Test
+public void
readRangeFromChannelDoesntReadMoreThanAskedForWhenItGotLessInFirstReadCall()
throws IOException {
+try (ReadableByteChannel in = new SeekableInMemoryByteChannel(new
byte[] { 1, 2, 3, 4, 5 }) {
+@Override
+public int read(ByteBuffer buf) throws IOException {
+// Read max 3 bytes at a time
+ByteBuffer temp = ByteBuffer.allocate(Math.min(3,
buf.remaining()));
+int read = super.read(temp);
+if (read > 0) {
+buf.put(temp.array(), 0, read);
+}
+return read;
+}
+}) {
+byte[] read = IOUtils.readRange(in, 4);
Review comment:
Same as above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 633326)
Time Spent: 40m (was: 0.5h)
> IOUtils.readRange() can read more from a channel than asked for
> ---
>
> Key: COMPRESS-584
> URL: https://issues.apache.org/jira/browse/COMPRESS-584
> Project: Commons Compress
> Issue Type: Bug
> Components: Archivers
>Affects Versions: 1.21
>Reporter: Matthijs Laan
>Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> When {{IOUtils.readRange(ReadableByteChannel,int)}} gets less than the number
> of bytes asked for in the first read call it does not reduce the buffer size
> for the next read call and may read more than asked for.
> This situation is rare when using a {{FileChannel}} but I wrote a
> {{SeekableByteChannel}} backed by a URI using HTTP range requests and reading
> from a socket can often return less bytes than asked for. When I used this
> channel to read a {{ZipFile}} it only read the ZIP central directory
> partially sometimes because {{IOUtils.readRange()}} called from
> {{ZipFile.readCentralDirectoryEntry()}} read more bytes than asked for and it
> stopped parsing directory entries.
> Fix: [https://github.com/apache/commons-compress/pull/214]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [commons-compress] PeterAlfredLee commented on a change in pull request #214: [COMPRESS-584] Fix IOUtils.readRange() can read more from a channel than asked for
PeterAlfredLee commented on a change in pull request #214:
URL: https://github.com/apache/commons-compress/pull/214#discussion_r682349198
##
File path: src/test/java/org/apache/commons/compress/utils/IOUtilsTest.java
##
@@ -148,6 +148,25 @@ public void
readRangeFromChannelDoesntReadMoreThanAskedFor() throws IOException
}
}
+@Test
+public void
readRangeFromChannelDoesntReadMoreThanAskedForWhenItGotLessInFirstReadCall()
throws IOException {
+try (ReadableByteChannel in = new SeekableInMemoryByteChannel(new
byte[] { 1, 2, 3, 4, 5 }) {
+@Override
+public int read(ByteBuffer buf) throws IOException {
+// Read max 3 bytes at a time
+ByteBuffer temp = ByteBuffer.allocate(Math.min(3,
buf.remaining()));
Review comment:
I think we can have a `final` here.
Even through this is just a test, it's always appreciated if `final` is
always used if possible.
##
File path: src/test/java/org/apache/commons/compress/utils/IOUtilsTest.java
##
@@ -148,6 +148,25 @@ public void
readRangeFromChannelDoesntReadMoreThanAskedFor() throws IOException
}
}
+@Test
+public void
readRangeFromChannelDoesntReadMoreThanAskedForWhenItGotLessInFirstReadCall()
throws IOException {
+try (ReadableByteChannel in = new SeekableInMemoryByteChannel(new
byte[] { 1, 2, 3, 4, 5 }) {
+@Override
+public int read(ByteBuffer buf) throws IOException {
+// Read max 3 bytes at a time
+ByteBuffer temp = ByteBuffer.allocate(Math.min(3,
buf.remaining()));
+int read = super.read(temp);
+if (read > 0) {
+buf.put(temp.array(), 0, read);
+}
+return read;
+}
+}) {
+byte[] read = IOUtils.readRange(in, 4);
Review comment:
Same as above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [commons-jcs] dependabot[bot] opened a new pull request #75: Bump commons-dbcp2 from 2.8.0 to 2.9.0
dependabot[bot] opened a new pull request #75: URL: https://github.com/apache/commons-jcs/pull/75 Bumps commons-dbcp2 from 2.8.0 to 2.9.0. [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Work logged] (COMPRESS-584) IOUtils.readRange() can read more from a channel than asked for
[
https://issues.apache.org/jira/browse/COMPRESS-584?focusedWorklogId=633321&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-633321
]
ASF GitHub Bot logged work on COMPRESS-584:
---
Author: ASF GitHub Bot
Created on: 04/Aug/21 07:07
Start Date: 04/Aug/21 07:07
Worklog Time Spent: 10m
Work Description: PeterAlfredLee commented on pull request #214:
URL: https://github.com/apache/commons-compress/pull/214#issuecomment-892419298
Good catch!
This looks good to me.
Thanks for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 633321)
Time Spent: 0.5h (was: 20m)
> IOUtils.readRange() can read more from a channel than asked for
> ---
>
> Key: COMPRESS-584
> URL: https://issues.apache.org/jira/browse/COMPRESS-584
> Project: Commons Compress
> Issue Type: Bug
> Components: Archivers
>Affects Versions: 1.21
>Reporter: Matthijs Laan
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> When {{IOUtils.readRange(ReadableByteChannel,int)}} gets less than the number
> of bytes asked for in the first read call it does not reduce the buffer size
> for the next read call and may read more than asked for.
> This situation is rare when using a {{FileChannel}} but I wrote a
> {{SeekableByteChannel}} backed by a URI using HTTP range requests and reading
> from a socket can often return less bytes than asked for. When I used this
> channel to read a {{ZipFile}} it only read the ZIP central directory
> partially sometimes because {{IOUtils.readRange()}} called from
> {{ZipFile.readCentralDirectoryEntry()}} read more bytes than asked for and it
> stopped parsing directory entries.
> Fix: [https://github.com/apache/commons-compress/pull/214]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [commons-compress] PeterAlfredLee commented on pull request #214: [COMPRESS-584] Fix IOUtils.readRange() can read more from a channel than asked for
PeterAlfredLee commented on pull request #214: URL: https://github.com/apache/commons-compress/pull/214#issuecomment-892419298 Good catch! This looks good to me. Thanks for your contributions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
