Hi. I'm trying to use the new feature but I can't use it with a big dataset (about 5 million rows).
I tried increasing executor memory, driver memory, partition number, but any solution can help me to solve the problem. One of the executor task increase the shufle memory until fails. Error is arrow generated: unable to expand the buffer. Any idea?
