Ah there is a nice jsonsurfer project that adds streaming support. I have created JIRA: https://issues.apache.org/jira/browse/CAMEL-15746
On Fri, Oct 23, 2020 at 5:59 PM Claus Ibsen <claus.ib...@gmail.com> wrote: > > Hi > > Just a note about splitting big xml files. Then there is camel-stax > and also the xtokenizer language you can use in splitter with > streaming. > We have some blogs and articles about that. > > For json then the jsonpath project have a ticket about adding support > for streaming, but the maintainers of that project is no very active > so I doubt they get around working on that. A PR was in the works that > is stalled: https://github.com/json-path/JsonPath/pull/93 > > But from camel you can split anything in streaming mode by passing in > something that can be iterated - then its splitted in chunks. > > If your json file is structured the same way - then you could write an > iterator that reads the content and emit chunks when you find where > each "array" ends. > > We may consider adding an jtokenizer (json tokenizer) to camel that > works a bit the same. > > On Fri, Oct 23, 2020 at 4:39 PM Romain Manni-Bucau > <rmannibu...@gmail.com> wrote: > > > > Hi, > > > > From my experience - got the same issue with xml years ago now, the > > simplest is to do a custom component (like jsonstreaming:....) and handle > > the split in this component. > > It will likely require a jsonpointer to find an array to emit array item > > one by one to next processor. > > At the end the route could look like something like: > > > > from("jsonstreaming:/path/file.json?pointer=/items") > > .to("bean:processUser"); > > > > Romain Manni-Bucau > > @rmannibucau <https://twitter.com/rmannibucau> | Blog > > <https://rmannibucau.metawerx.net/> | Old Blog > > <http://rmannibucau.wordpress.com> | Github > > <https://github.com/rmannibucau> | > > LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book > > <https://www.packtpub.com/application-development/java-ee-8-high-performance> > > > > > > Le ven. 23 oct. 2020 à 16:16, Mantas Gridinas <mgridi...@gmail.com> a > > écrit : > > > > > From personal experience any activities related to file component tend to > > > try to load entire file into memory. You could perhaps fiddle around by > > > converting it to an input stream but then you get into an issue of making > > > sure that you don't read an entire file into memory before actually > > > converting it. > > > > > > I'd suggest avoiding using camel here at all for the sake of retaining > > > fine > > > grained control over streaming process. At most you could wrap it in a > > > processor and be done with it. Otherwise you'll start programming with > > > routes, which is something you'd want to avoid. > > > > > > > > > On Fri, Oct 23, 2020, 17:05 Site Register <site.regis...@ymail.com > > > .invalid> > > > wrote: > > > > > > > Hi Camel Users, > > > > I have a 4G json array file need to load into database. How can I > > > leverage > > > > Camel to stream the file and split into json? > > > > I had tried to use "stream:file" but it was reading line by line not > > > split > > > > into json. > > > > I leveraged gson streaming to read and insert into the database in java > > > > application which took about 3 minutes. However, I would like to check > > > > if > > > > there is a way to leverage camel pipeline for the same purpose. > > > > Thank you, > > > > > > > -- > Claus Ibsen > ----------------- > http://davsclaus.com @davsclaus > Camel in Action 2: https://www.manning.com/ibsen2 -- Claus Ibsen ----------------- http://davsclaus.com @davsclaus Camel in Action 2: https://www.manning.com/ibsen2
