On Wed, Mar 15, 2023 at 5:41 PM Oliver Hancock wrote:
>
> I am taking in a JSON file that is an array of objects. I am trying to unnest
> it and serialize it however memory management is very import, meaning that
> the json being read and unested needs to be done through through a stream.
> The given input would look like an array of json objects looking like this:
>
> {"a": "some \"string\" here"}
>
>
> With the desired outcome looking like
>
> {"data": "{\"a\":\"some \\\"string\\\" here\"}"}
>
>
> for the reading and writing of the data object i am using the jackson
> streaming library as it is the least memory intensive library fitting with my
> requirements. To serialize the data i am use an chararray library called
> fastUtils however the problem persists without this so it is not the library
> or the serialization itself. I have also increased the buffer size of both
> the pipedReader object and the char buffer but again there is no benefit.
>
> Current code:
>
> Reader reader = new InputStreamReader(inputStream, StandardCharsets.UTF_8);
> Writer writer = new OutputStreamWriter(outputStream, StandardCharsets.UTF_8);
>
> final JsonFactory jsonFactory = new JsonFactory();
> final JsonGenerator jsonGenerator = jsonFactory.createGenerator(writer);
> final JsonParser jsonParser = jsonFactory.createParser(reader);
>
> jsonParser.nextValue();
>
> jsonGenerator.writeStartObject();
> jsonGenerator.writeFieldName("dataItem");
> jsonGenerator.flush();
> writer.flush();
> writer.write(58)
>
> PipedReader pipedReader = new PipedReader(4096);
> PipedWriter pipedWriter = new PipedWriter(pipedReader);
> JsonGenerator jsonGeneratorPiped = jsonFactory.createGenerator(pipedWriter);
>
> new Thread(
> new Runnable(){
> public void run() {
> jsonGeneratorPiped.copyCurrentStructure(jsonParser);
> jsonGeneratorPiped.flush();
> pipedWriter.close();
> }
> }
> ).start();
>
> char[] buffer = new char[2048];
> int len = reader.read(buffer);
> char quote = '"';
> char escape = '\\';
> CharArrayList listBuffer;
> //listBuffer = new CharArrayList();
> while (len != -1) {
> listBuffer = new CharArrayList(buffer,0,len);
> //listBuffer.addElements(0,buffer,0,len);
> for (CharListIterator iter = listBuffer.iterator(); iter.hasNext();) {
> if (iter.nextChar() == quote) {
> iter.set(escape);
> iter.add(quote);
> }
> }
>
> writer.write(listBuffer.elements(), 0, listBuffer.size());
>
> //toCharArray uses lot of memory
> //writer.write(listBuffer.toCharArray(), 0, listBuffer.size());
>
> //listBuffer.trim();
> len = reader.read(buffer);
> }
>
> writer.flush();
> pipedReader.close();
> jsonGeneratorPiped.close();
>
> jsonGenerator.writeEndObject();
> jsonGenerator.flush();
>
> jsonGenerator.close();
> jsonParser.close();
>
>
> that is the current code i have implemented and it works fine for files that
> are small, however with larger files >10mb from my testing, the stream closes
> mid way through resulting in this error
>
> java.io.IOexception: stream is closed
>
> Any help would be greatly appreciated
Since Jackson does not do any such limits checking currently, I'd
guess use of PipedReader / PipedWriter is problematic.
Your Runnable may be throwing an exception that you are not catching
Alternatively if you are reading from/writing to some network resource
(reading from a server via URL), it's likely some limit by that
system. 10 megs sounds like the kind of limit many systems impose for
maximum requests/responses.
I hope this helps,
-+ Tatu +-
--
You received this message because you are subscribed to the Google Groups
"jackson-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to jackson-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/jackson-user/CAL4a10jO7wdZHPP4zZqfy%2BvdxG2651PTvmxTHFe-XHKXUMqV8Q%40mail.gmail.com.
