On Wed, Mar 15, 2023 at 5:41 PM Oliver Hancock <oliverwhanc...@gmail.com> wrote: > > I am taking in a JSON file that is an array of objects. I am trying to unnest > it and serialize it however memory management is very import, meaning that > the json being read and unested needs to be done through through a stream. > The given input would look like an array of json objects looking like this: > > {"a": "some \"string\" here"} > > > With the desired outcome looking like > > {"data": "{\"a\":\"some \\\"string\\\" here\"}"} > > > for the reading and writing of the data object i am using the jackson > streaming library as it is the least memory intensive library fitting with my > requirements. To serialize the data i am use an chararray library called > fastUtils however the problem persists without this so it is not the library > or the serialization itself. I have also increased the buffer size of both > the pipedReader object and the char buffer but again there is no benefit. > > Current code: > > Reader reader = new InputStreamReader(inputStream, StandardCharsets.UTF_8); > Writer writer = new OutputStreamWriter(outputStream, StandardCharsets.UTF_8); > > final JsonFactory jsonFactory = new JsonFactory(); > final JsonGenerator jsonGenerator = jsonFactory.createGenerator(writer); > final JsonParser jsonParser = jsonFactory.createParser(reader); > > jsonParser.nextValue(); > > jsonGenerator.writeStartObject(); > jsonGenerator.writeFieldName("dataItem"); > jsonGenerator.flush(); > writer.flush(); > writer.write(58) > > PipedReader pipedReader = new PipedReader(4096); > PipedWriter pipedWriter = new PipedWriter(pipedReader); > JsonGenerator jsonGeneratorPiped = jsonFactory.createGenerator(pipedWriter); > > new Thread( > new Runnable(){ > public void run() { > jsonGeneratorPiped.copyCurrentStructure(jsonParser); > jsonGeneratorPiped.flush(); > pipedWriter.close(); > } > } > ).start(); > > char[] buffer = new char[2048]; > int len = reader.read(buffer); > char quote = '"'; > char escape = '\\'; > CharArrayList listBuffer; > //listBuffer = new CharArrayList(); > while (len != -1) { > listBuffer = new CharArrayList(buffer,0,len); > //listBuffer.addElements(0,buffer,0,len); > for (CharListIterator iter = listBuffer.iterator(); iter.hasNext();) { > if (iter.nextChar() == quote) { > iter.set(escape); > iter.add(quote); > } > } > > writer.write(listBuffer.elements(), 0, listBuffer.size()); > > //toCharArray uses lot of memory > //writer.write(listBuffer.toCharArray(), 0, listBuffer.size()); > > //listBuffer.trim(); > len = reader.read(buffer); > } > > writer.flush(); > pipedReader.close(); > jsonGeneratorPiped.close(); > > jsonGenerator.writeEndObject(); > jsonGenerator.flush(); > > jsonGenerator.close(); > jsonParser.close(); > > > that is the current code i have implemented and it works fine for files that > are small, however with larger files >10mb from my testing, the stream closes > mid way through resulting in this error > > java.io.IOexception: stream is closed > > Any help would be greatly appreciated
Since Jackson does not do any such limits checking currently, I'd guess use of PipedReader / PipedWriter is problematic. Your Runnable may be throwing an exception that you are not catching Alternatively if you are reading from/writing to some network resource (reading from a server via URL), it's likely some limit by that system. 10 megs sounds like the kind of limit many systems impose for maximum requests/responses. I hope this helps, -+ Tatu +- -- You received this message because you are subscribed to the Google Groups "jackson-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to jackson-user+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jackson-user/CAL4a10jO7wdZHPP4zZqfy%2BvdxG2651PTvmxTHFe-XHKXUMqV8Q%40mail.gmail.com.