+1 I’d prefer to see more of the end goal and how that could be achieved (such as ETL or SPARK-24579). However given the rounds and months of discussions we have come down to just the public API.
If the community thinks a new set of public API is maintainable, I don’t see any problem with that. ________________________________ From: Tom Graves <tgraves...@yahoo.com.INVALID> Sent: Sunday, May 26, 2019 8:22:59 AM To: hol...@pigscanfly.ca; Reynold Xin Cc: Bobby Evans; DB Tsai; Dongjoon Hyun; Imran Rashid; Jason Lowe; Matei Zaharia; Thomas graves; Xiangrui Meng; Xiangrui Meng; dev Subject: Re: [VOTE][SPARK-27396] SPIP: Public APIs for extended Columnar Processing Support More feedback would be great, this has been open a long time though, let's extend til Wednesday the 29th and see where we are at. Tom Sent from Yahoo Mail on Android<https://go.onelink.me/107872968?pid=InProduct&c=Global_Internal_YGrowth_AndroidEmailSig__AndroidUsers&af_wl=ym&af_sub1=Internal&af_sub2=Global_YGrowth&af_sub3=EmailSignature> On Sat, May 25, 2019 at 6:28 PM, Holden Karau <hol...@pigscanfly.ca> wrote: Same I meant to catch up after kubecon but had some unexpected travels. On Sat, May 25, 2019 at 10:56 PM Reynold Xin <r...@databricks.com<mailto:r...@databricks.com>> wrote: Can we push this to June 1st? I have been meaning to read it but unfortunately keeps traveling... On Sat, May 25, 2019 at 8:31 PM Dongjoon Hyun <dongjoon.h...@gmail.com<mailto:dongjoon.h...@gmail.com>> wrote: +1 Thanks, Dongjoon. On Fri, May 24, 2019 at 17:03 DB Tsai <dbt...@dbtsai.com.invalid> wrote: +1 on exposing the APIs for columnar processing support. I understand that the scope of this SPIP doesn't cover AI / ML use-cases. But I saw a good performance gain when I converted data from rows to columns to leverage on SIMD architectures in a POC ML application. With the exposed columnar processing support, I can imagine that the heavy lifting parts of ML applications (such as computing the objective functions) can be written as columnar expressions that leverage on SIMD architectures to get a good speedup. Sincerely, DB Tsai ---------------------------------------------------------- Web: https://www.dbtsai.com PGP Key ID: 42E5B25A8F7A82C1 On Wed, May 15, 2019 at 2:59 PM Bobby Evans <reva...@gmail.com<mailto:reva...@gmail.com>> wrote: > > It would allow for the columnar processing to be extended through the > shuffle. So if I were doing say an FPGA accelerated extension it could > replace the ShuffleExechangeExec with one that can take a ColumnarBatch as > input instead of a Row. The extended version of the ShuffleExchangeExec could > then do the partitioning on the incoming batch and instead of producing a > ShuffleRowRDD for the exchange they could produce something like a > ShuffleBatchRDD that would let the serializing and deserializing happen in a > column based format for a faster exchange, assuming that columnar processing > is also happening after the exchange. This is just like providing a columnar > version of any other catalyst operator, except in this case it is a bit more > complex of an operator. > > On Wed, May 15, 2019 at 12:15 PM Imran Rashid <iras...@cloudera.com.invalid> > wrote: >> >> sorry I am late to the discussion here -- the jira mentions using this >> extensions for dealing with shuffles, can you explain that part? I don't >> see how you would use this to change shuffle behavior at all. >> >> On Tue, May 14, 2019 at 10:59 AM Thomas graves >> <tgra...@apache.org<mailto:tgra...@apache.org>> wrote: >>> >>> Thanks for replying, I'll extend the vote til May 26th to allow your >>> and other people feedback who haven't had time to look at it. >>> >>> Tom >>> >>> On Mon, May 13, 2019 at 4:43 PM Holden Karau >>> <hol...@pigscanfly.ca<mailto:hol...@pigscanfly.ca>> wrote: >>> > >>> > I’d like to ask this vote period to be extended, I’m interested but I >>> > don’t have the cycles to review it in detail and make an informed vote >>> > until the 25th. >>> > >>> > On Tue, May 14, 2019 at 1:49 AM Xiangrui Meng >>> > <m...@databricks.com<mailto:m...@databricks.com>> wrote: >>> >> >>> >> My vote is 0. Since the updated SPIP focuses on ETL use cases, I don't >>> >> feel strongly about it. I would still suggest doing the following: >>> >> >>> >> 1. Link the POC mentioned in Q4. So people can verify the POC result. >>> >> 2. List public APIs we plan to expose in Appendix A. I did a quick >>> >> check. Beside ColumnarBatch and ColumnarVector, we also need to make the >>> >> following public. People who are familiar with SQL internals should help >>> >> assess the risk. >>> >> * ColumnarArray >>> >> * ColumnarMap >>> >> * unsafe.types.CaledarInterval >>> >> * ColumnarRow >>> >> * UTF8String >>> >> * ArrayData >>> >> * ... >>> >> 3. I still feel using Pandas UDF as the mid-term success doesn't match >>> >> the purpose of this SPIP. It does make some code cleaner. But I guess >>> >> for ETL use cases, it won't bring much value. >>> >> >>> > -- >>> > Twitter: https://twitter.com/holdenkarau >>> > Books (Learning Spark, High Performance Spark, etc.): >>> > https://amzn.to/2MaRAG9 >>> > YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>> >>> --------------------------------------------------------------------- >>> To unsubscribe e-mail: >>> dev-unsubscr...@spark.apache.org<mailto:dev-unsubscr...@spark.apache.org> >>> --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org<mailto:dev-unsubscr...@spark.apache.org> -- Twitter: https://twitter.com/holdenkarau Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
