[ https://issues.apache.org/jira/browse/CODEC-265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986436#comment-16986436 ]
Alex Herbert commented on CODEC-265: ------------------------------------ Smarter memory allocation has been implemented for the Base32/Base64 codecs. They can now allocate up to the system limit for arrays. Please check against master. [Snapshots|https://repository.apache.org/content/repositories/snapshots/commons-codec/commons-codec/1.14-SNAPSHOT/] are now deployed so you can include the 1.14 snapshot dependency for testing. In the source there is a new test Base64Test.testCodec265 that demonstrates encoding a 1 gigabyte array. The test is memory hungry. In a real world use case it may be worth looking at the streaming API using Base64OutputStream to write chunks of byte[] data to your target output. Even if this is a ByteArrayOutputStream you will have lower memory usage as you will not have to read the entire allocated random file to memory and you can convert the ByteArrayOutputStream to a String directly. Memory usage will still peak at around 4GiB as the String will have 2 bytes for each character: {code:java} String filePath = "/tmp/1gb.zip"; Path path = Paths.get(filePath); String contentToBeSaved; try (InputStream in = Files.newInputStream(path)) { // Allocate 1.5 GiB to avoid buffer reallocation final ByteArrayOutputStream out = new ByteArrayOutputStream(3 * (1 << 29)); // Encode with no line separator final Base64OutputStream base64 = new Base64OutputStream(out, true, 0, null); final byte[] buffer = new byte[8192]; int read = in.read(buffer); while (read >= 0) { base64.write(buffer, 0, read); read = in.read(buffer); } base64.close(); // This will require 2.67GiB ~ 2 * 1GiB * 4/3 contentToBeSaved = out.toString(); } {code} There will be a fair bit more memory usage for the JSONObject in your example. Hopefully codec is not a blocker any more and you can continue your testing. > java.lang.NegativeArraySizeException > -------------------------------------- > > Key: CODEC-265 > URL: https://issues.apache.org/jira/browse/CODEC-265 > Project: Commons Codec > Issue Type: Bug > Affects Versions: 1.13 > Environment: Linux = Ubuntu 18.04.3 LTS > JDK = 1.8 > > Reporter: Ingimar > Assignee: Alex Herbert > Priority: Critical > Attachments: NewClientEncodePost.java, Util.java, pom.xml > > Time Spent: 20m > Remaining Estimate: 0h > > Hi, > trying to encode a file that is 1GB of size. > ( linux : > {code:java} > fallocate -l 1GB 1gb.zip{code} > ) > I want to post that file to a RESTful-service, package in JSON. > *here is the code* > > > {code:java} > String filePath = "/tmp/1gb.zip"; > System.out.println("\t Post to : ".concat(URL)); > System.out.println("\t file : ".concat(filePath)); > Path path = Paths.get(filePath); > byte[] bArray = Files.readAllBytes(path); > // testing commons codec 1.16 (2019-11-05) > byte[] encodeBase64 = Base64.encodeBase64(bArray); > final String contentToBeSaved = new String(encodeBase64); > HttpClient client = HttpClientBuilder.create().build(); > HttpResponse response = null; > JSONObject metadata = new JSONObject(); > metadata.put("owner", "Ingo"); > metadata.put("access", "public"); > metadata.put("licenseType", "CC BY"); > metadata.put("fileName", "fileName"); > metadata.put("fileDataBase64", contentToBeSaved); > String metadataFormatted = > StringEscapeUtils.unescapeJavaScript(metadata.toString()); > StringEntity entity = new StringEntity(metadataFormatted, > ContentType.APPLICATION_JSON); > HttpPost post = new HttpPost(URL); > post.setEntity(entity); > response = client.execute(post); > HttpEntity responseEntity = response.getEntity(); > String responseFromMediaserver = EntityUtils.toString(responseEntity, > "UTF-8"); > System.out.println("\n****"); > System.out.println("Response is : " + responseFromMediaserver); > JSONObject json = new JSONObject(responseFromMediaserver); > String uuid = json.getString("uuid"); > System.out.println("UUID is " + uuid); > {code} > > > # mvn clean package > # java -Xms512m -Xmx20480m -jar target/mediaClient.jar > The crasch is in > > {code:java} > byte[] encodeBase64 = Base64.encodeBase64(bArray);{code} > > the stacktrace is : > {code:java} > > Starting NewClientEncodePost > Post to : http://127.0.0.1:8080/MediaServerResteasy/media > file : /tmp/1gb.zip > Exception in thread "main" java.lang.NegativeArraySizeException > at > org.apache.commons.codec.binary.BaseNCodec.resizeBuffer(BaseNCodec.java:253) > at > org.apache.commons.codec.binary.BaseNCodec.ensureBufferSize(BaseNCodec.java:269) > at org.apache.commons.codec.binary.Base64.encode(Base64.java:380) > at org.apache.commons.codec.binary.BaseNCodec.encode(BaseNCodec.java:451) > at org.apache.commons.codec.binary.BaseNCodec.encode(BaseNCodec.java:430) > at org.apache.commons.codec.binary.Base64.encodeBase64(Base64.java:679) > at org.apache.commons.codec.binary.Base64.encodeBase64(Base64.java:642) > at org.apache.commons.codec.binary.Base64.encodeBase64(Base64.java:623) > at org.apache.commons.codec.binary.Base64.encodeBase64(Base64.java:556) > at > se.nrm.bio.mediaserver.testing.base64.NewClientEncodePost.posting(NewClientEncodePost.java:55) > at > se.nrm.bio.mediaserver.testing.base64.NewClientEncodePost.main(NewClientEncodePost.java:38) > > {code} > > -- This message was sent by Atlassian Jira (v8.3.4#803005)