Hi,
Is there any reason why InMemoryTableScanExec.inputRDD does not use the
buffers local value [1] for the non-batch case [2]? Just curious as I ran
into it and thought I'd do a tiny refactoring.
[1]
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/executi
I don't know anything about it directly, but seems like it would have been
caused by
https://github.com/apache/spark/commit/e3201e165e41f076ec72175af246d12c0da529cf
The "?" in fromClause is what's generating the warning, and it may be
ignorable.
On Mon, May 14, 2018 at 12:38 AM Reynold Xin wrote:
IIUC, since the `lateral View*` matches an empty string in optional blocks,
antlr shows such a warning;
fromClause
: FROM relation (',' relation)* (pivotClause | lateralView*)?
;
http://www.antlr.org/api/JavaTool/org/antlr/v4/tool/ErrorType.html#EPSILON_OPTIONAL
On Mon, May 14, 2018 at
Yes Takeshi, I agree, I think we can easily fix the warning replacing the *
with +, since the two options are not required.
I will test this fix and create a PR when it is ready.
Thanks,
Marco
2018-05-14 15:08 GMT+02:00 Takeshi Yamamuro :
> IIUC, since the `lateral View*` matches an empty string
Hi,
>I agree, it would be great if we could make the errors more clear about where
>the error happened (user code or in Spark code) and what assumption was
>violated. The problem is that this is a really hard thing to do generally,
>like Reynold said. I think we should look for individual cases
Thanks for starting this discussion, I'd also like to see some improvements
in this area and glad to hear that the Pandas UDFs / Arrow functionality
might be useful. I'm wondering if from your initial investigations you
found anything lacking from the Arrow format or possible improvements that
wou
