Stream Huge JSON File

Hi Camel Users, I have a 4G json array file need to load into database. How can I leverage Camel to stream the file and split into json? I had tried to use "stream:file" but it was reading line by line not split into json.  I leveraged gson streaming to read and insert into the database in java

Re: Stream Huge JSON File

>From personal experience any activities related to file component tend to try to load entire file into memory. You could perhaps fiddle around by converting it to an input stream but then you get into an issue of making sure that you don't read an entire file into memory before actually converting

Re: Stream Huge JSON File

Hi, >From my experience - got the same issue with xml years ago now, the simplest is to do a custom component (like jsonstreaming:) and handle the split in this component. It will likely require a jsonpointer to find an array to emit array item one by one to next processor. At the end the rout

Re: Stream Huge JSON File

I am thinking to have a processor to read json file in streaming and produce as batch to outbound. However, I am not sure how to batch messages out during streaming read. try (JsonReader reader = new JsonReader(new FileReader(fileName))) {             reader.beginArray();             Gson gson

Re: Stream Huge JSON File

Send each element as a medsage via producer template. On Fri, Oct 23, 2020, 18:44 Site Register wrote: > I am thinking to have a processor to read json file in streaming and > produce as batch to outbound. However, I am not sure how to batch messages > out during streaming read. > > try (JsonRe

Re: Stream Huge JSON File

Hi Just a note about splitting big xml files. Then there is camel-stax and also the xtokenizer language you can use in splitter with streaming. We have some blogs and articles about that. For json then the jsonpath project have a ticket about adding support for streaming, but the maintainers of t

Re: Stream Huge JSON File

Ah there is a nice jsonsurfer project that adds streaming support. I have created JIRA: https://issues.apache.org/jira/browse/CAMEL-15746 On Fri, Oct 23, 2020 at 5:59 PM Claus Ibsen wrote: > > Hi > > Just a note about splitting big xml files. Then there is camel-stax > and also the xtokenizer la

Re: Stream Huge JSON File

Doing the same than sax/jaxb with jsonp sounds very feasible too - that's the impl i had in mind. Romain Manni-Bucau @rmannibucau | Blog | Old Blog | Github |

Re: Stream Huge JSON File

Thank you for all the suggestions.  For now, I am using gson streaming manually read the huge json file and using  producer template to produce the messages. On Friday, October 23, 2020, 12:05:26 PM EDT, Claus Ibsen wrote: Ah there is a nice jsonsurfer project that adds streaming suppo

Re: Camel CXF with SOAP 1.1 and SOAP 1.2

Everybody was in the same boat, no worries. Best place to start (if replies in the email list doesn't do it) is Camel online documentation, here is for CXF component https://camel.apache.org/components/latest/cxf-component.html BTW, it has serviceClass mentioned 10 times(!) Also, have a look a