The PR also touches ML4All eventually. I added the additional bits into the PR :)
—Alex Sent from my mobile, please excuse typos > On Dec 15, 2025, at 5:09 PM, Zoi Kaoudi via dev <[email protected]> wrote: > > I think moving the Spark platform to dataframes makes a lot of sense. > Some newer operators, like reading parquet files, use the dataframes with > wrap/unwrapping to RDD. We just need to be sure if there is a mapping from > each of our operators to a Spark dataframe operator. > Is that what you meant Alex? > Best > -- > Zoi > Στις Δευτέρα 15 Δεκεμβρίου 2025 στις 07:55:39 π.μ. GMT-6, ο χρήστης > Alexander Alten <[email protected]> έγραψε: > > Hey team, > > I have been working for a while on data frames support. Right now, I’m pretty > confident that a PR can be requested in the next few days. I noticed a lot of > scaffolding around RDDs to use DataFrames. Multiple existing operators > already wrap temporary DataFrames. I decided to refactor those too. The idea > is that a user sets a switch which is either RDD or DataFrames. > > In-depth ParquetSource now carries a preferDatasetOutput flag with > `preferDatasetOutput(boolean)/isDatasetOutputPreferred()` so we can request > Dataset-backed execution from higher APIs. > > Thoughts? > > —Alex > > -- > Alexander Alten > CTO & co-founder > Scalytics - We Connect the World’s Data > > Subscribe to our newsletter at LinkedIn > > e: [email protected] > ln: www.linkedin.com/in/alexanderalten/ > Book a meeting! > > Disclaimer: Human written, please excuse typos. > > > -- > *Scalytics Connect* > The foundation for secure, scalable, and transparent > AI. > www.scalytics.io > <https://www.google.com/url?q=http://www.scalytics.io&source=gmail-imap&ust=1766419743000000&usg=AOvVaw3_dnJ2-97ub5j_s5coQAqx> > > -- Please consider the > environment before printing this email -- > > Disclaimer: > The content of this > message is confidential. If you have received it by mistake, please inform > us by an email reply and then delete the message. It is forbidden to copy, > forward, or in any way reveal the contents of this message to anyone. The > integrity and security of this email cannot be guaranteed over the > Internet. Therefore, the sender will not be held liable for any damage > caused by the message. -- *Scalytics Connect* The foundation for secure, scalable, and transparent AI. www.scalytics.io <http://www.scalytics.io> -- Please consider the environment before printing this email -- Disclaimer: The content of this message is confidential. If you have received it by mistake, please inform us by an email reply and then delete the message. It is forbidden to copy, forward, or in any way reveal the contents of this message to anyone. The integrity and security of this email cannot be guaranteed over the Internet. Therefore, the sender will not be held liable for any damage caused by the message.
