As it turns out, I also commented on the same Jira (and forgot about it until just now).
On Wed, May 19, 2021 at 8:32 AM Shubham Chaurasia <shubh.chaura...@gmail.com> wrote: > > Hi, > > I remember creating one for a similar scenario in the past - > https://issues.apache.org/jira/browse/SPARK-29372. > > Thanks, > Shubham > > On Wed, May 19, 2021 at 5:34 PM Takeshi Yamamuro <linguin....@gmail.com> > wrote: >> >> hi, Andrew, >> >> Welcome any improvement proposal for that. >> Could you file an issue in jira first to show us your idea and an example >> query >> to reproduce the issue you described? >> >> Bests, >> Takeshi >> >> On Wed, May 19, 2021 at 11:38 AM Andrew Melo <andrew.m...@gmail.com> wrote: >>> >>> Hello, >>> >>> When reading a very wide (> 1000 cols) input, WholeStageCodeGen blows >>> past the 64kB source limit and fails. Looking at the generated code, a >>> big part of the code is simply the DSv2 convention that the codegen'd >>> variable names are the same as the columns instead of something more >>> compact like 'c1', 'c2', etc.. >>> >>> Would there be any interest in accepting a patch that shortens these >>> variable names to try and stay under the limit? >>> >>> Thanks >>> Andrew >>> >>> --------------------------------------------------------------------- >>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>> >> >> >> -- >> --- >> Takeshi Yamamuro --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
