Re: Slack for PySpark users

2023-04-04 Thread Mich Talebzadeh
Hi Shani, I believe I am an admin so that is fine by me. Hi Dongioon, With regard to summarising the discussion etc, no need, It is like flogging the dead horse, we have already discussed it enough. I don't see the point of it. HTH Mich Talebzadeh, Lead Solutions Architect/Engineering Lead Pal

Re: Slack for PySpark users

2023-04-04 Thread Bjørn Jørgensen
One of the things that I don't like about this slack solution is that questions and answers disappear after 90 days. Today's maillist solution is indexed by search engines and when one day you wonder about something, you can find solutions with the help of just searching the web. Another question t

Re: Slack for PySpark users

2023-04-04 Thread Mich Talebzadeh
That 3 months retention is just a soft setting. For low volume traffic, it can be negotiated to a year’s retention. Let me see what we can do about it. HTH On Tue, 4 Apr 2023 at 09:31, Bjørn Jørgensen wrote: > One of the things that I don't like about this slack solution is that > questions and

Re: Creating InMemory relations with data in ColumnarBatches

2023-04-04 Thread Bobby Evans
This is not going to work without changes to Spark. InMemoryTableScanExec supports columnar output, but not columnar input. You would have to write code to support that in Spark itself. The second part is that there are only a handful of operators that support columnar output. Really it is just t