First, congratulations on vastly improving the speed and size of JSON nested array parsing! I have run into a new problem when I try to flatten those large arrays now: Query Failed: An Error Occurred org.apache.drill.exec.rpc.RpcException: RemoteRpcException: Failure while running fragment., Incoming batch of org.apache.drill.exec.physical.impl.flatten.FlattenRecordBatch has size 69153, which is beyond the limit of 65536 [ 82af4f1d-c185-41dc-a348-5385c54aed04 on quickstart.cloudera:31010 ] [ 82af4f1d-c185-41dc-a348-5385c54aed04 on quickstart.cloudera:31010 ]
Is there a plan to improve the flatten function so that it can handle the large array sizes now allowed? Confidentiality Warning: This e-mail contains information intended only for the use of the individual or entity named above. If the reader of this e-mail is not the intended recipient or the employee or agent responsible for delivering it to the intended recipient, any dissemination, publication or copying of this e-mail is strictly prohibited. The sender does not accept any responsibility for any loss, disruption or damage to your data or computer system that may occur while using data contained in, or transmitted with, this e-mail. If you have received this e-mail in error, please immediately notify us by return e-mail. Thank you.
