You can take a look at PartitionReaderFactory.
It's created at the driver side, serialized and sent to the executor side.
For the write side, there is a similar channel: DataWriterFactory
On Wed, May 19, 2021 at 4:37 AM Andrew Melo wrote:
> Hello,
>
> When implementing a DSv2 datasource, where
Hello,
When implementing a DSv2 datasource, where is an appropriate place to
store/transmit secrets from the driver to the executors? Is there
built-in spark functionality for that, or is my best bet to stash it
as a member variable in one of the classes that gets sent to the
executors?
Thanks!
