[ https://issues.apache.org/jira/browse/SPARK-45600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Allison Wang updated SPARK-45600: --------------------------------- Description: Currently, registered data sources are stored in `sharedState` and can be accessed across multiple sessions. This, however, will not work with Spark Connect. We should make this registration session level, and support static registration (e.g. using pip install) in the future. (was: Currently we have added a few instance variables to store information for Python data source reader. We should have a dedicated reader class for Python data source to make the current DataFrameReader clean.) > Make Python data source registration session level > -------------------------------------------------- > > Key: SPARK-45600 > URL: https://issues.apache.org/jira/browse/SPARK-45600 > Project: Spark > Issue Type: Sub-task > Components: PySpark > Affects Versions: 4.0.0 > Reporter: Allison Wang > Priority: Major > > Currently, registered data sources are stored in `sharedState` and can be > accessed across multiple sessions. This, however, will not work with Spark > Connect. We should make this registration session level, and support static > registration (e.g. using pip install) in the future. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org