How much memory is allocated to the Drill environment?
Embedded or in a cluster?
I don’t think there is a particular limit, but a single JSON file will be read
by a single minor fragment, in general it is better to match the number/size of
files to the Drill environment.
In the short term try to bump up planner.memory.max_query_memory_per_node in
the options and see if that works for you.
--Andries
On 11/2/17, 7:46 AM, "Yun Liu" <y....@castsoftware.com> wrote:
Hi,
I've been using Apache Drill actively and just wondering what is the
capacity of Drill? I have a json file which is 390MB and it keeps throwing me
an DATA_READ ERROR. I have another json file with exact same format but only
150MB and it's processing fine. When I did a *select* on the large json, it
returns successfully for some of the fields. None of these errors really apply
to me. So I am trying to understand the capacity of the json files Drill
supports up to. Or if there's something else I missed.
Thanks,
Yun Liu
Solutions Delivery Consultant
321 West 44th St | Suite 501 | New York, NY 10036
+1 212.871.8355 office | +1 646.752.4933 mobile
CAST, Leader in Software Analysis and Measurement
Achieve Insight. Deliver Excellence.
Join the discussion http://blog.castsoftware.com/
LinkedIn<http://www.linkedin.com/companies/162909> |
Twitter<http://twitter.com/onquality> |
Facebook<http://www.facebook.com/pages/CAST/105668942817177>
