Re: Szehon's question: Currently there is no plan to include Java/Scala connectors. There are already established DSv2 integrations maintained by other communities, such as the MongoDB Spark Connector <https://github.com/mongodb/mongo-spark>.
I would prefer to keep Java/Scala separate from Python since their ecosystems and tooling are very different. We could consider a separate repository for them later. On Wed, Aug 19, 2026 at 12:10 PM Allison Wang <[email protected]> wrote: > Hi Nicholas, the main lesson for me is that sustainability must be a core > part of the design. Community-contributed integrations require ongoing > ownership, reviews, updates, testing, and releases. One difference is > that Spark Packages is a broad index of independently maintained > projects. The scope can quickly become too large to maintain, and it is > hard to know the quality or maintenance status of each package. This > proposal has a much narrower focus PySpark data sources and place the > implementations in one Apache github repository similar to > spark-connect-go <https://github.com/apache/spark-connect-go>. > > I also think today’s automation and AI-assisted development tools create > an opportunity to reduce the maintenance load. We could design the > repository to be AI-friendly from the beginning. These tools would not > replace maintainers or human review, but they could make contributions and > routine updates much easier. > > Love to hear others' thoughts on this. > > On Tue, Aug 18, 2026 at 8:02 PM Nicholas Chammas < > [email protected]> wrote: > >> Just to be clear, I mean "broadly similar” as in they are both >> community-maintained libraries for Spark. Allison’s proposal is, of course, >> much more focused! I would just like to raise the point of comparison since >> I think it’s relevant. >> >> >> On Aug 18, 2026, at 10:51 PM, Nicholas Chammas < >> [email protected]> wrote: >> >> We had something broadly similar from ~10 years ago, but not limited to >> data sources: https://spark-packages.org/ >> >> My impression of Spark Packages is that it is inactive and unmaintained. >> If so, it might be useful to have a brief post mortem as a community to >> understand why it didn’t stand the test of time, and how to avoid that fate >> for this new proposal. >> >> >> On Aug 18, 2026, at 7:45 PM, Allison Wang <[email protected]> wrote: >> >> Hi all, >> >> I would like to discuss whether Apache Spark should provide an experimental, >> community-maintained home for the ecosystem around the PySpark Data >> Source API >> <https://spark.apache.org/docs/latest/api/python/tutorial/sql/python_data_source.html> >> : >> >> Proposed repository: apache/spark-python-datasources >> Existing implementation: allisonwang-db/pyspark-data-sources >> <https://github.com/allisonwang-db/pyspark-data-sources> >> >> The intent would not be to make these data sources part of Spark core. >> Instead, the repository would provide an Apache-governed place where the >> community can collaborate on reusable implementations of the public Python >> Data Source API, share practical examples, and grow the ecosystem around >> the API without expanding Spark core itself. >> >> The existing project can serve as the initial contribution. It contains >> batch and streaming readers and writers built with the public PySpark Data >> Source API, covering a range of external systems and use cases. >> >> I propose starting with a deliberately lightweight model: >> >> - >> >> The repository would be experimental and community-supported. >> - It would focus specifically on implementations built on the public >> PySpark >> Data Source API. >> >> <https://spark.apache.org/docs/latest/api/python/tutorial/sql/python_data_source.html> >> - Code in this repository would remain separate from Spark core and >> would not carry the same compatibility or support guarantees as Spark >> itself. >> - New contributions would go through community review, with >> maintainability, dependencies, licensing, testing, and security taken into >> consideration. >> - Implementations that become unmaintained or no longer meet the >> repository's requirements could be deprecated or removed through the >> normal >> community process. >> - >> >> The repository would be governed by the Apache Spark community and >> follow ASF policies. >> >> The existing project is currently published on PyPI as >> pyspark-data-sources using the pyspark_datasources import namespace. For >> continuity, I would prefer to retain those names if they are compatible >> with ASF release and branding requirements, but the package naming is not >> essential to this proposal. I am willing to help maintain the >> repository, review contributions, and support the release process. >> The main question I would like feedback on is whether the Spark community >> thinks it is useful to provide this kind of lightweight, experimental >> home for extensions built on a public Spark API, while keeping those >> integrations explicitly outside Spark core. >> >> If the community supports this proposal, I will work with the Spark PMC >> on the required JIRA and ASF IP-clearance steps, move the approved code to >> the Apache repository, update the package metadata, and transfer PyPI >> publishing to an ASF-controlled release process. Existing PyPI releases and >> installation commands would remain unchanged. >> >> I would appreciate any feedback on this proposal. >> >> Thanks, >> Allison >> >> >> >>
