Hello dear driller,

Before starting the topic, I would like to do a simple survey :

1. Did you know that Drill already supports XML format?

2. If yes, what is the maximum size for the XML files you normally read? 1MB, 
10MB or 100MB

3. Do you expect that reading XML will be as easy as JSON (Schema Discovery)?

Thank you for responding to those questions.

XML is different from the JSON file, and if we rely solely on the Drill drive 
to deduce the structure of the data. (or called SCHEMA), the code will get very 
complex and delicate.

For example, inferring array structure and numeric range. So, "provided schema" 
or "TO_JSON" may be good medicine :

Provided Schema

We can add the DTD or XML Schema (XSD) support for the XML. It can build all 
value vectors (Writer) before reading data, solving the fields, types, and 
complex nested.

However, a definition file is actually a rule validator that allows elements to 
appear 0 or more times. As a result, it is not possible to know if all elements 
exist until the data is read.

Therefore, avoid creating a large number of value vectors that do not actually 
exist before reading the data.

We can build the top schema at the initial stage and add new value vectors as 
needed during the reading phase.

TO_JSON

Read and convert XML directly to JSON, using the JSON Reader for data 
resolution.

It makes it easier for us to query the XML data such as JSON, but requires 
reading the whole XML file in memory.

I think the two can be done, so I look forward to your spirited discussion.

Thanks.

- luoc

Reply via email to