This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push: new 08866c280f87 [SPARK-47439][PYTHON] Document Python Data Source API in API reference page 08866c280f87 is described below commit 08866c280f877ce27d5c5305c7a09add76c86774 Author: Hyukjin Kwon <gurwls...@apache.org> AuthorDate: Mon Mar 18 20:08:22 2024 +0900 [SPARK-47439][PYTHON] Document Python Data Source API in API reference page ### What changes were proposed in this pull request? This PR proposes to document Python Data Source API in Python API reference page. ### Why are the changes needed? For users/developers to know how to use them. ### Does this PR introduce _any_ user-facing change? Yes, it documents Python Data Source API. ### How was this patch tested? Manually checked the output from Python API reference build ```bash cd python/docs make clean html open build/html/index.html ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #45561 from HyukjinKwon/SPARK-47439. Authored-by: Hyukjin Kwon <gurwls...@apache.org> Signed-off-by: Hyukjin Kwon <gurwls...@apache.org> --- .../source/reference/pyspark.sql/core_classes.rst | 7 ++++ .../{core_classes.rst => datasource.rst} | 44 +++++++++++----------- python/docs/source/reference/pyspark.sql/index.rst | 1 + .../source/reference/pyspark.sql/spark_session.rst | 1 + 4 files changed, 31 insertions(+), 22 deletions(-) diff --git a/python/docs/source/reference/pyspark.sql/core_classes.rst b/python/docs/source/reference/pyspark.sql/core_classes.rst index 3cf19686cdd8..65096da21de5 100644 --- a/python/docs/source/reference/pyspark.sql/core_classes.rst +++ b/python/docs/source/reference/pyspark.sql/core_classes.rst @@ -42,3 +42,10 @@ Core Classes UDTFRegistration udf.UserDefinedFunction udtf.UserDefinedTableFunction + datasource.DataSource + datasource.DataSourceReader + datasource.DataSourceStreamReader + datasource.DataSourceWriter + datasource.DataSourceRegistration + datasource.InputPartition + datasource.WriterCommitMessage diff --git a/python/docs/source/reference/pyspark.sql/core_classes.rst b/python/docs/source/reference/pyspark.sql/datasource.rst similarity index 58% copy from python/docs/source/reference/pyspark.sql/core_classes.rst copy to python/docs/source/reference/pyspark.sql/datasource.rst index 3cf19686cdd8..b92db7a28858 100644 --- a/python/docs/source/reference/pyspark.sql/core_classes.rst +++ b/python/docs/source/reference/pyspark.sql/datasource.rst @@ -16,29 +16,29 @@ under the License. -============ -Core Classes -============ -.. currentmodule:: pyspark.sql +================== +Python Data Source +================== + +.. currentmodule:: pyspark.sql.datasource .. autosummary:: :toctree: api/ - SparkSession - Catalog - DataFrame - Column - Observation - Row - GroupedData - PandasCogroupedOps - DataFrameNaFunctions - DataFrameStatFunctions - Window - DataFrameReader - DataFrameWriter - DataFrameWriterV2 - UDFRegistration - UDTFRegistration - udf.UserDefinedFunction - udtf.UserDefinedTableFunction + DataSource.name + DataSource.reader + DataSource.schema + DataSource.streamReader + DataSource.writer + DataSourceReader.partitions + DataSourceReader.read + DataSourceRegistration.register + DataSourceStreamReader.commit + DataSourceStreamReader.initialOffset + DataSourceStreamReader.latestOffset + DataSourceStreamReader.partitions + DataSourceStreamReader.read + DataSourceStreamReader.stop + DataSourceWriter.abort + DataSourceWriter.commit + DataSourceWriter.write diff --git a/python/docs/source/reference/pyspark.sql/index.rst b/python/docs/source/reference/pyspark.sql/index.rst index 233c8b238a6d..9322a91fba25 100644 --- a/python/docs/source/reference/pyspark.sql/index.rst +++ b/python/docs/source/reference/pyspark.sql/index.rst @@ -42,3 +42,4 @@ This page gives an overview of all public Spark SQL API. udf udtf protobuf + datasource diff --git a/python/docs/source/reference/pyspark.sql/spark_session.rst b/python/docs/source/reference/pyspark.sql/spark_session.rst index 4be343c52140..ea71249e292e 100644 --- a/python/docs/source/reference/pyspark.sql/spark_session.rst +++ b/python/docs/source/reference/pyspark.sql/spark_session.rst @@ -47,6 +47,7 @@ See also :class:`SparkSession`. SparkSession.catalog SparkSession.conf SparkSession.createDataFrame + SparkSession.dataSource SparkSession.getActiveSession SparkSession.newSession SparkSession.profile --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org