A while ago, I implemented XML and Json input formats. However, having proper split support for structured formats without sync markers is not that easy. Any split that has a random start offset need to figure out the start of the next record on its own, which is fragile by definition. That's why supporting jsonl files is much easier; you just need to look for the next newline. For the same reason, supporting json or xml in Kafka is fairly straightforward: records are already split.
It would be easier to support XML and Json if we can get of splits. @Flavio would you expect to get inner file parallelism or would you be fine with processing only the files in parallel? Best, Arvid On Fri, Nov 29, 2019 at 3:26 PM Chesnay Schepler <ches...@apache.org> wrote: > I know that at least the Table API > <https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/table/connect.html#csv-format> > can read json, but I don't know how well this translates into other APIs. > > On 29/11/2019 12:09, Flavio Pompermaier wrote: > > Hi to all, > is there any out-of-the-box option to read multiline JSON or XML like in > Spark? > It would be awesome to have something like > > spark.read .option("multiline", true) .json("/path/to/user.json") > > Best, > Flavio > > >
