This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.3 by this push:
new e24859be740 [SPARK-39253][DOCS][PYTHON][3.3] Improve PySpark API
reference to be more readable
e24859be740 is described below
commit e24859be7407123018a07a23ec0a78e386bb7398
Author: itholic <haejoon....@databricks.com>
AuthorDate: Thu May 26 19:35:35 2022 +0900
[SPARK-39253][DOCS][PYTHON][3.3] Improve PySpark API reference to be more
readable
### What changes were proposed in this pull request?
Hotfix https://github.com/apache/spark/pull/36647 for branch-3.3.
### Why are the changes needed?
The improvement of document readability will also improve the usability for
PySpark.
### Does this PR introduce _any_ user-facing change?
Yes, now the documentation is categorized by its class or their own purpose
more clearly as below:
<img width="270" alt="Screen Shot 2022-05-24 at 1 50 23 PM"
src="https://user-images.githubusercontent.com/44108233/169951517-f8b9cb72-7408-46d6-8cd7-15ae890a7a7f.png";>
### How was this patch tested?
The existing test should cover.
Closes #36685 from itholic/SPARK-39253-3.3.
Authored-by: itholic <haejoon....@databricks.com>
Signed-off-by: Hyukjin Kwon <gurwls...@apache.org>
---
python/docs/source/reference/index.rst | 2 +-
python/docs/source/reference/pyspark.sql.rst | 663 ---------------------
.../reference/{index.rst => pyspark.sql/avro.rst} | 25 +-
.../{index.rst => pyspark.sql/catalog.rst} | 49 +-
.../{index.rst => pyspark.sql/column.rst} | 59 +-
.../{index.rst => pyspark.sql/configuration.rst} | 20 +-
.../{index.rst => pyspark.sql/core_classes.rst} | 40 +-
.../{index.rst => pyspark.sql/data_types.rst} | 47 +-
.../source/reference/pyspark.sql/dataframe.rst | 133 +++++
.../source/reference/pyspark.sql/functions.rst | 343 +++++++++++
.../{index.rst => pyspark.sql/grouping.rst} | 39 +-
.../source/reference/{ => pyspark.sql}/index.rst | 36 +-
python/docs/source/reference/pyspark.sql/io.rst | 54 ++
.../{index.rst => pyspark.sql/observation.rst} | 24 +-
.../reference/{index.rst => pyspark.sql/row.rst} | 24 +-
.../source/reference/pyspark.sql/spark_session.rst | 53 ++
.../{index.rst => pyspark.sql/window.rst} | 39 +-
17 files changed, 790 insertions(+), 860 deletions(-)
diff --git a/python/docs/source/reference/index.rst
b/python/docs/source/reference/index.rst
index f023b5a8c99..1d2db3f4a15 100644
--- a/python/docs/source/reference/index.rst
+++ b/python/docs/source/reference/index.rst
@@ -27,7 +27,7 @@ Pandas API on Spark follows the API specifications of pandas
1.3.
.. toctree::
:maxdepth: 2
- pyspark.sql
+ pyspark.sql/index
pyspark.pandas/index
pyspark.ss
pyspark.ml
diff --git a/python/docs/source/reference/pyspark.sql.rst
b/python/docs/source/reference/pyspark.sql.rst
deleted file mode 100644
index adc1958822e..00000000000
--- a/python/docs/source/reference/pyspark.sql.rst
+++ /dev/null
@@ -1,663 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
-.. http://www.apache.org/licenses/LICENSE-2.0
-
-.. Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
-
-
-=========
-Spark SQL
-=========
-
-Core Classes
-------------
-
-.. currentmodule:: pyspark.sql
-
-.. autosummary::
- :toctree: api/
-
- SparkSession
- Catalog
- DataFrame
- Column
- Observation
- Row
- GroupedData
- PandasCogroupedOps
- DataFrameNaFunctions
- DataFrameStatFunctions
- Window
-
-
-Spark Session APIs
-------------------
-
-.. currentmodule:: pyspark.sql
-
-The entry point to programming Spark with the Dataset and DataFrame API.
-To create a Spark session, you should use ``SparkSession.builder`` attribute.
-See also :class:`SparkSession`.
-
-.. autosummary::
- :toctree: api/
-
- SparkSession.builder.appName
- SparkSession.builder.config
- SparkSession.builder.enableHiveSupport
- SparkSession.builder.getOrCreate
- SparkSession.builder.master
- SparkSession.catalog
- SparkSession.conf
- SparkSession.createDataFrame
- SparkSession.getActiveSession
- SparkSession.newSession
- SparkSession.range
- SparkSession.read
- SparkSession.readStream
- SparkSession.sparkContext
- SparkSession.sql
- SparkSession.stop
- SparkSession.streams
- SparkSession.table
- SparkSession.udf
- SparkSession.version
-
-
-Configuration
--------------
-
-.. currentmodule:: pyspark.sql.conf
-
-.. autosummary::
- :toctree: api/
-
- RuntimeConfig
-
-
-Input and Output
-----------------
-
-.. currentmodule:: pyspark.sql
-
-.. autosummary::
- :toctree: api/
-
- DataFrameReader.csv
- DataFrameReader.format
- DataFrameReader.jdbc
- DataFrameReader.json
- DataFrameReader.load
- DataFrameReader.option
- DataFrameReader.options
- DataFrameReader.orc
- DataFrameReader.parquet
- DataFrameReader.schema
- DataFrameReader.table
- DataFrameWriter.bucketBy
- DataFrameWriter.csv
- DataFrameWriter.format
- DataFrameWriter.insertInto
- DataFrameWriter.jdbc
- DataFrameWriter.json
- DataFrameWriter.mode
- DataFrameWriter.option
- DataFrameWriter.options
- DataFrameWriter.orc
- DataFrameWriter.parquet
- DataFrameWriter.partitionBy
- DataFrameWriter.save
- DataFrameWriter.saveAsTable
- DataFrameWriter.sortBy
- DataFrameWriter.text
-
-
-DataFrame APIs
---------------
-
-.. currentmodule:: pyspark.sql
-
-.. autosummary::
- :toctree: api/
-
- DataFrame.agg
- DataFrame.alias
- DataFrame.approxQuantile
- DataFrame.cache
- DataFrame.checkpoint
- DataFrame.coalesce
- DataFrame.colRegex
- DataFrame.collect
- DataFrame.columns
- DataFrame.corr
- DataFrame.count
- DataFrame.cov
- DataFrame.createGlobalTempView
- DataFrame.createOrReplaceGlobalTempView
- DataFrame.createOrReplaceTempView
- DataFrame.createTempView
- DataFrame.crossJoin
- DataFrame.crosstab
- DataFrame.cube
- DataFrame.describe
- DataFrame.distinct
- DataFrame.drop
- DataFrame.dropDuplicates
- DataFrame.drop_duplicates
- DataFrame.dropna
- DataFrame.dtypes
- DataFrame.exceptAll
- DataFrame.explain
- DataFrame.fillna
- DataFrame.filter
- DataFrame.first
- DataFrame.foreach
- DataFrame.foreachPartition
- DataFrame.freqItems
- DataFrame.groupBy
- DataFrame.head
- DataFrame.hint
- DataFrame.inputFiles
- DataFrame.intersect
- DataFrame.intersectAll
- DataFrame.isEmpty
- DataFrame.isLocal
- DataFrame.isStreaming
- DataFrame.join
- DataFrame.limit
- DataFrame.localCheckpoint
- DataFrame.mapInPandas
- DataFrame.mapInArrow
- DataFrame.na
- DataFrame.observe
- DataFrame.orderBy
- DataFrame.persist
- DataFrame.printSchema
- DataFrame.randomSplit
- DataFrame.rdd
- DataFrame.registerTempTable
- DataFrame.repartition
- DataFrame.repartitionByRange
- DataFrame.replace
- DataFrame.rollup
- DataFrame.sameSemantics
- DataFrame.sample
- DataFrame.sampleBy
- DataFrame.schema
- DataFrame.select
- DataFrame.selectExpr
- DataFrame.semanticHash
- DataFrame.show
- DataFrame.sort
- DataFrame.sortWithinPartitions
- DataFrame.sparkSession
- DataFrame.stat
- DataFrame.storageLevel
- DataFrame.subtract
- DataFrame.summary
- DataFrame.tail
- DataFrame.take
- DataFrame.toDF
- DataFrame.toJSON
- DataFrame.toLocalIterator
- DataFrame.toPandas
- DataFrame.transform
- DataFrame.union
- DataFrame.unionAll
- DataFrame.unionByName
- DataFrame.unpersist
- DataFrame.where
- DataFrame.withColumn
- DataFrame.withColumnRenamed
- DataFrame.withWatermark
- DataFrame.write
- DataFrame.writeStream
- DataFrame.writeTo
- DataFrame.pandas_api
- DataFrameNaFunctions.drop
- DataFrameNaFunctions.fill
- DataFrameNaFunctions.replace
- DataFrameStatFunctions.approxQuantile
- DataFrameStatFunctions.corr
- DataFrameStatFunctions.cov
- DataFrameStatFunctions.crosstab
- DataFrameStatFunctions.freqItems
- DataFrameStatFunctions.sampleBy
-
-Column APIs
------------
-
-.. currentmodule:: pyspark.sql
-
-.. autosummary::
- :toctree: api/
-
- Column.alias
- Column.asc
- Column.asc_nulls_first
- Column.asc_nulls_last
- Column.astype
- Column.between
- Column.bitwiseAND
- Column.bitwiseOR
- Column.bitwiseXOR
- Column.cast
- Column.contains
- Column.desc
- Column.desc_nulls_first
- Column.desc_nulls_last
- Column.dropFields
- Column.endswith
- Column.eqNullSafe
- Column.getField
- Column.getItem
- Column.ilike
- Column.isNotNull
- Column.isNull
- Column.isin
- Column.like
- Column.name
- Column.otherwise
- Column.over
- Column.rlike
- Column.startswith
- Column.substr
- Column.when
- Column.withField
-
-Data Types
-----------
-
-.. currentmodule:: pyspark.sql.types
-
-.. autosummary::
- :template: autosummary/class_with_docs.rst
- :toctree: api/
-
- ArrayType
- BinaryType
- BooleanType
- ByteType
- DataType
- DateType
- DecimalType
- DoubleType
- FloatType
- IntegerType
- LongType
- MapType
- NullType
- ShortType
- StringType
- StructField
- StructType
- TimestampType
- DayTimeIntervalType
-
-
-Observation
------------
-
-.. currentmodule:: pyspark.sql
-
-.. autosummary::
- :toctree: api/
-
- Observation.get
-
-
-Row
----
-
-.. currentmodule:: pyspark.sql
-
-.. autosummary::
- :toctree: api/
-
- Row.asDict
-
-
-Functions
----------
-
-.. currentmodule:: pyspark.sql.functions
-
-.. autosummary::
- :toctree: api/
-
- abs
- acos
- acosh
- add_months
- aggregate
- approxCountDistinct
- approx_count_distinct
- array
- array_contains
- array_distinct
- array_except
- array_intersect
- array_join
- array_max
- array_min
- array_position
- array_remove
- array_repeat
- array_sort
- array_union
- arrays_overlap
- arrays_zip
- asc
- asc_nulls_first
- asc_nulls_last
- ascii
- asin
- asinh
- assert_true
- atan
- atanh
- atan2
- avg
- base64
- bin
- bit_length
- bitwise_not
- bitwiseNOT
- broadcast
- bround
- bucket
- cbrt
- ceil
- coalesce
- col
- collect_list
- collect_set
- column
- concat
- concat_ws
- conv
- corr
- cos
- cosh
- cot
- count
- count_distinct
- countDistinct
- covar_pop
- covar_samp
- crc32
- create_map
- csc
- cume_dist
- current_date
- current_timestamp
- date_add
- date_format
- date_sub
- date_trunc
- datediff
- dayofmonth
- dayofweek
- dayofyear
- days
- decode
- degrees
- dense_rank
- desc
- desc_nulls_first
- desc_nulls_last
- element_at
- encode
- exists
- exp
- explode
- explode_outer
- expm1
- expr
- factorial
- filter
- first
- flatten
- floor
- forall
- format_number
- format_string
- from_csv
- from_json
- from_unixtime
- from_utc_timestamp
- get_json_object
- greatest
- grouping
- grouping_id
- hash
- hex
- hour
- hours
- hypot
- initcap
- input_file_name
- instr
- isnan
- isnull
- json_tuple
- kurtosis
- lag
- last
- last_day
- lead
- least
- length
- levenshtein
- lit
- locate
- log
- log10
- log1p
- log2
- lower
- lpad
- ltrim
- make_date
- map_concat
- map_entries
- map_filter
- map_from_arrays
- map_from_entries
- map_keys
- map_values
- map_zip_with
- max
- max_by
- md5
- mean
- min
- min_by
- minute
- monotonically_increasing_id
- month
- months
- months_between
- nanvl
- next_day
- nth_value
- ntile
- octet_length
- overlay
- pandas_udf
- percent_rank
- percentile_approx
- posexplode
- posexplode_outer
- pow
- product
- quarter
- radians
- raise_error
- rand
- randn
- rank
- regexp_extract
- regexp_replace
- repeat
- reverse
- rint
- round
- row_number
- rpad
- rtrim
- schema_of_csv
- schema_of_json
- sec
- second
- sentences
- sequence
- session_window
- sha1
- sha2
- shiftleft
- shiftright
- shiftrightunsigned
- shuffle
- signum
- sin
- sinh
- size
- skewness
- slice
- sort_array
- soundex
- spark_partition_id
- split
- sqrt
- stddev
- stddev_pop
- stddev_samp
- struct
- substring
- substring_index
- sum
- sum_distinct
- sumDistinct
- tan
- tanh
- timestamp_seconds
- toDegrees
- toRadians
- to_csv
- to_date
- to_json
- to_timestamp
- to_utc_timestamp
- transform
- transform_keys
- transform_values
- translate
- trim
- trunc
- udf
- unbase64
- unhex
- unix_timestamp
- upper
- var_pop
- var_samp
- variance
- weekofyear
- when
- window
- xxhash64
- year
- years
- zip_with
-
-
-.. currentmodule:: pyspark.sql.avro.functions
-
-.. autosummary::
- :toctree: api/
-
- from_avro
- to_avro
-
-Window
-------
-
-.. currentmodule:: pyspark.sql
-
-.. autosummary::
- :toctree: api/
-
- Window.currentRow
- Window.orderBy
- Window.partitionBy
- Window.rangeBetween
- Window.rowsBetween
- Window.unboundedFollowing
- Window.unboundedPreceding
- WindowSpec.orderBy
- WindowSpec.partitionBy
- WindowSpec.rangeBetween
- WindowSpec.rowsBetween
-
-Grouping
---------
-
-.. currentmodule:: pyspark.sql
-
-.. autosummary::
- :toctree: api/
-
- GroupedData.agg
- GroupedData.apply
- GroupedData.applyInPandas
- GroupedData.avg
- GroupedData.cogroup
- GroupedData.count
- GroupedData.max
- GroupedData.mean
- GroupedData.min
- GroupedData.pivot
- GroupedData.sum
- PandasCogroupedOps.applyInPandas
-
-Catalog APIs
-------------
-
-.. currentmodule:: pyspark.sql
-
-.. autosummary::
- :toctree: api/
-
- Catalog.cacheTable
- Catalog.clearCache
- Catalog.createExternalTable
- Catalog.createTable
- Catalog.currentDatabase
- Catalog.databaseExists
- Catalog.dropGlobalTempView
- Catalog.dropTempView
- Catalog.functionExists
- Catalog.isCached
- Catalog.listColumns
- Catalog.listDatabases
- Catalog.listFunctions
- Catalog.listTables
- Catalog.recoverPartitions
- Catalog.refreshByPath
- Catalog.refreshTable
- Catalog.registerFunction
- Catalog.setCurrentDatabase
- Catalog.tableExists
- Catalog.uncacheTable
diff --git a/python/docs/source/reference/index.rst
b/python/docs/source/reference/pyspark.sql/avro.rst
similarity index 69%
copy from python/docs/source/reference/index.rst
copy to python/docs/source/reference/pyspark.sql/avro.rst
index f023b5a8c99..b6de88deef1 100644
--- a/python/docs/source/reference/index.rst
+++ b/python/docs/source/reference/pyspark.sql/avro.rst
@@ -16,22 +16,13 @@
under the License.
-=============
-API Reference
-=============
+====
+Avro
+====
+.. currentmodule:: pyspark.sql.avro.functions
-This page lists an overview of all public PySpark modules, classes, functions
and methods.
+.. autosummary::
+ :toctree: api/
-Pandas API on Spark follows the API specifications of pandas 1.3.
-
-.. toctree::
- :maxdepth: 2
-
- pyspark.sql
- pyspark.pandas/index
- pyspark.ss
- pyspark.ml
- pyspark.streaming
- pyspark.mllib
- pyspark
- pyspark.resource
+ from_avro
+ to_avro
diff --git a/python/docs/source/reference/index.rst
b/python/docs/source/reference/pyspark.sql/catalog.rst
similarity index 56%
copy from python/docs/source/reference/index.rst
copy to python/docs/source/reference/pyspark.sql/catalog.rst
index f023b5a8c99..8267e06410e 100644
--- a/python/docs/source/reference/index.rst
+++ b/python/docs/source/reference/pyspark.sql/catalog.rst
@@ -16,22 +16,33 @@
under the License.
-=============
-API Reference
-=============
-
-This page lists an overview of all public PySpark modules, classes, functions
and methods.
-
-Pandas API on Spark follows the API specifications of pandas 1.3.
-
-.. toctree::
- :maxdepth: 2
-
- pyspark.sql
- pyspark.pandas/index
- pyspark.ss
- pyspark.ml
- pyspark.streaming
- pyspark.mllib
- pyspark
- pyspark.resource
+=======
+Catalog
+=======
+
+.. currentmodule:: pyspark.sql
+
+.. autosummary::
+ :toctree: api/
+
+ Catalog.cacheTable
+ Catalog.clearCache
+ Catalog.createExternalTable
+ Catalog.createTable
+ Catalog.currentDatabase
+ Catalog.databaseExists
+ Catalog.dropGlobalTempView
+ Catalog.dropTempView
+ Catalog.functionExists
+ Catalog.isCached
+ Catalog.listColumns
+ Catalog.listDatabases
+ Catalog.listFunctions
+ Catalog.listTables
+ Catalog.recoverPartitions
+ Catalog.refreshByPath
+ Catalog.refreshTable
+ Catalog.registerFunction
+ Catalog.setCurrentDatabase
+ Catalog.tableExists
+ Catalog.uncacheTable
diff --git a/python/docs/source/reference/index.rst
b/python/docs/source/reference/pyspark.sql/column.rst
similarity index 53%
copy from python/docs/source/reference/index.rst
copy to python/docs/source/reference/pyspark.sql/column.rst
index f023b5a8c99..b5f39d299c1 100644
--- a/python/docs/source/reference/index.rst
+++ b/python/docs/source/reference/pyspark.sql/column.rst
@@ -16,22 +16,43 @@
under the License.
-=============
-API Reference
-=============
-
-This page lists an overview of all public PySpark modules, classes, functions
and methods.
-
-Pandas API on Spark follows the API specifications of pandas 1.3.
-
-.. toctree::
- :maxdepth: 2
-
- pyspark.sql
- pyspark.pandas/index
- pyspark.ss
- pyspark.ml
- pyspark.streaming
- pyspark.mllib
- pyspark
- pyspark.resource
+======
+Column
+======
+.. currentmodule:: pyspark.sql
+
+.. autosummary::
+ :toctree: api/
+
+ Column.alias
+ Column.asc
+ Column.asc_nulls_first
+ Column.asc_nulls_last
+ Column.astype
+ Column.between
+ Column.bitwiseAND
+ Column.bitwiseOR
+ Column.bitwiseXOR
+ Column.cast
+ Column.contains
+ Column.desc
+ Column.desc_nulls_first
+ Column.desc_nulls_last
+ Column.dropFields
+ Column.endswith
+ Column.eqNullSafe
+ Column.getField
+ Column.getItem
+ Column.ilike
+ Column.isNotNull
+ Column.isNull
+ Column.isin
+ Column.like
+ Column.name
+ Column.otherwise
+ Column.over
+ Column.rlike
+ Column.startswith
+ Column.substr
+ Column.when
+ Column.withField
diff --git a/python/docs/source/reference/index.rst
b/python/docs/source/reference/pyspark.sql/configuration.rst
similarity index 71%
copy from python/docs/source/reference/index.rst
copy to python/docs/source/reference/pyspark.sql/configuration.rst
index f023b5a8c99..7a5c10400de 100644
--- a/python/docs/source/reference/index.rst
+++ b/python/docs/source/reference/pyspark.sql/configuration.rst
@@ -17,21 +17,11 @@
=============
-API Reference
+Configuration
=============
+.. currentmodule:: pyspark.sql.conf
-This page lists an overview of all public PySpark modules, classes, functions
and methods.
+.. autosummary::
+ :toctree: api/
-Pandas API on Spark follows the API specifications of pandas 1.3.
-
-.. toctree::
- :maxdepth: 2
-
- pyspark.sql
- pyspark.pandas/index
- pyspark.ss
- pyspark.ml
- pyspark.streaming
- pyspark.mllib
- pyspark
- pyspark.resource
+ RuntimeConfig
diff --git a/python/docs/source/reference/index.rst
b/python/docs/source/reference/pyspark.sql/core_classes.rst
similarity index 69%
copy from python/docs/source/reference/index.rst
copy to python/docs/source/reference/pyspark.sql/core_classes.rst
index f023b5a8c99..72f9ca122a9 100644
--- a/python/docs/source/reference/index.rst
+++ b/python/docs/source/reference/pyspark.sql/core_classes.rst
@@ -16,22 +16,24 @@
under the License.
-=============
-API Reference
-=============
-
-This page lists an overview of all public PySpark modules, classes, functions
and methods.
-
-Pandas API on Spark follows the API specifications of pandas 1.3.
-
-.. toctree::
- :maxdepth: 2
-
- pyspark.sql
- pyspark.pandas/index
- pyspark.ss
- pyspark.ml
- pyspark.streaming
- pyspark.mllib
- pyspark
- pyspark.resource
+============
+Core Classes
+============
+.. currentmodule:: pyspark.sql
+
+.. autosummary::
+ :toctree: api/
+
+ SparkSession
+ Catalog
+ DataFrame
+ Column
+ Observation
+ Row
+ GroupedData
+ PandasCogroupedOps
+ DataFrameNaFunctions
+ DataFrameStatFunctions
+ Window
+ DataFrameReader
+ DataFrameWriter
diff --git a/python/docs/source/reference/index.rst
b/python/docs/source/reference/pyspark.sql/data_types.rst
similarity index 65%
copy from python/docs/source/reference/index.rst
copy to python/docs/source/reference/pyspark.sql/data_types.rst
index f023b5a8c99..d146c640477 100644
--- a/python/docs/source/reference/index.rst
+++ b/python/docs/source/reference/pyspark.sql/data_types.rst
@@ -16,22 +16,31 @@
under the License.
-=============
-API Reference
-=============
-
-This page lists an overview of all public PySpark modules, classes, functions
and methods.
-
-Pandas API on Spark follows the API specifications of pandas 1.3.
-
-.. toctree::
- :maxdepth: 2
-
- pyspark.sql
- pyspark.pandas/index
- pyspark.ss
- pyspark.ml
- pyspark.streaming
- pyspark.mllib
- pyspark
- pyspark.resource
+==========
+Data Types
+==========
+.. currentmodule:: pyspark.sql.types
+
+.. autosummary::
+ :template: autosummary/class_with_docs.rst
+ :toctree: api/
+
+ ArrayType
+ BinaryType
+ BooleanType
+ ByteType
+ DataType
+ DateType
+ DecimalType
+ DoubleType
+ FloatType
+ IntegerType
+ LongType
+ MapType
+ NullType
+ ShortType
+ StringType
+ StructField
+ StructType
+ TimestampType
+ DayTimeIntervalType
diff --git a/python/docs/source/reference/pyspark.sql/dataframe.rst
b/python/docs/source/reference/pyspark.sql/dataframe.rst
new file mode 100644
index 00000000000..5b6e704ba48
--- /dev/null
+++ b/python/docs/source/reference/pyspark.sql/dataframe.rst
@@ -0,0 +1,133 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+=========
+DataFrame
+=========
+
+.. currentmodule:: pyspark.sql
+
+.. autosummary::
+ :toctree: api/
+
+ DataFrame.agg
+ DataFrame.alias
+ DataFrame.approxQuantile
+ DataFrame.cache
+ DataFrame.checkpoint
+ DataFrame.coalesce
+ DataFrame.colRegex
+ DataFrame.collect
+ DataFrame.columns
+ DataFrame.corr
+ DataFrame.count
+ DataFrame.cov
+ DataFrame.createGlobalTempView
+ DataFrame.createOrReplaceGlobalTempView
+ DataFrame.createOrReplaceTempView
+ DataFrame.createTempView
+ DataFrame.crossJoin
+ DataFrame.crosstab
+ DataFrame.cube
+ DataFrame.describe
+ DataFrame.distinct
+ DataFrame.drop
+ DataFrame.dropDuplicates
+ DataFrame.drop_duplicates
+ DataFrame.dropna
+ DataFrame.dtypes
+ DataFrame.exceptAll
+ DataFrame.explain
+ DataFrame.fillna
+ DataFrame.filter
+ DataFrame.first
+ DataFrame.foreach
+ DataFrame.foreachPartition
+ DataFrame.freqItems
+ DataFrame.groupBy
+ DataFrame.head
+ DataFrame.hint
+ DataFrame.inputFiles
+ DataFrame.intersect
+ DataFrame.intersectAll
+ DataFrame.isEmpty
+ DataFrame.isLocal
+ DataFrame.isStreaming
+ DataFrame.join
+ DataFrame.limit
+ DataFrame.localCheckpoint
+ DataFrame.mapInPandas
+ DataFrame.mapInArrow
+ DataFrame.na
+ DataFrame.observe
+ DataFrame.orderBy
+ DataFrame.persist
+ DataFrame.printSchema
+ DataFrame.randomSplit
+ DataFrame.rdd
+ DataFrame.registerTempTable
+ DataFrame.repartition
+ DataFrame.repartitionByRange
+ DataFrame.replace
+ DataFrame.rollup
+ DataFrame.sameSemantics
+ DataFrame.sample
+ DataFrame.sampleBy
+ DataFrame.schema
+ DataFrame.select
+ DataFrame.selectExpr
+ DataFrame.semanticHash
+ DataFrame.show
+ DataFrame.sort
+ DataFrame.sortWithinPartitions
+ DataFrame.sparkSession
+ DataFrame.stat
+ DataFrame.storageLevel
+ DataFrame.subtract
+ DataFrame.summary
+ DataFrame.tail
+ DataFrame.take
+ DataFrame.toDF
+ DataFrame.toJSON
+ DataFrame.toLocalIterator
+ DataFrame.toPandas
+ DataFrame.to_pandas_on_spark
+ DataFrame.transform
+ DataFrame.union
+ DataFrame.unionAll
+ DataFrame.unionByName
+ DataFrame.unpersist
+ DataFrame.where
+ DataFrame.withColumn
+ DataFrame.withColumns
+ DataFrame.withColumnRenamed
+ DataFrame.withMetadata
+ DataFrame.withWatermark
+ DataFrame.write
+ DataFrame.writeStream
+ DataFrame.writeTo
+ DataFrame.pandas_api
+ DataFrameNaFunctions.drop
+ DataFrameNaFunctions.fill
+ DataFrameNaFunctions.replace
+ DataFrameStatFunctions.approxQuantile
+ DataFrameStatFunctions.corr
+ DataFrameStatFunctions.cov
+ DataFrameStatFunctions.crosstab
+ DataFrameStatFunctions.freqItems
+ DataFrameStatFunctions.sampleBy
diff --git a/python/docs/source/reference/pyspark.sql/functions.rst
b/python/docs/source/reference/pyspark.sql/functions.rst
new file mode 100644
index 00000000000..390d7d768ca
--- /dev/null
+++ b/python/docs/source/reference/pyspark.sql/functions.rst
@@ -0,0 +1,343 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+=========
+Functions
+=========
+.. currentmodule:: pyspark.sql.functions
+
+Normal Functions
+----------------
+.. autosummary::
+ :toctree: api/
+
+ col
+ column
+ create_map
+ lit
+ array
+ map_from_arrays
+ broadcast
+ coalesce
+ input_file_name
+ isnan
+ isnull
+ monotonically_increasing_id
+ nanvl
+ rand
+ randn
+ spark_partition_id
+ struct
+ when
+ bitwise_not
+ bitwiseNOT
+ expr
+ greatest
+ least
+
+
+Math Functions
+--------------
+.. autosummary::
+ :toctree: api/
+
+ sqrt
+ abs
+ acos
+ acosh
+ asin
+ asinh
+ atan
+ atanh
+ atan2
+ bin
+ cbrt
+ ceil
+ conv
+ cos
+ cosh
+ cot
+ csc
+ exp
+ expm1
+ factorial
+ floor
+ hex
+ unhex
+ hypot
+ log
+ log10
+ log1p
+ log2
+ pow
+ rint
+ round
+ bround
+ sec
+ shiftleft
+ shiftright
+ shiftrightunsigned
+ signum
+ sin
+ sinh
+ tan
+ tanh
+ toDegrees
+ degrees
+ toRadians
+ radians
+
+
+Datetime Functions
+------------------
+.. autosummary::
+ :toctree: api/
+
+ add_months
+ current_date
+ current_timestamp
+ date_add
+ date_format
+ date_sub
+ date_trunc
+ datediff
+ dayofmonth
+ dayofweek
+ dayofyear
+ second
+ weekofyear
+ year
+ quarter
+ month
+ last_day
+ minute
+ months_between
+ next_day
+ hour
+ make_date
+ from_unixtime
+ unix_timestamp
+ to_timestamp
+ to_date
+ trunc
+ from_utc_timestamp
+ to_utc_timestamp
+ window
+ session_window
+ timestamp_seconds
+
+
+Collection Functions
+--------------------
+.. autosummary::
+ :toctree: api/
+
+ array_contains
+ arrays_overlap
+ slice
+ array_join
+ concat
+ array_position
+ element_at
+ array_sort
+ array_remove
+ array_distinct
+ array_intersect
+ array_union
+ array_except
+ transform
+ exists
+ forall
+ filter
+ aggregate
+ zip_with
+ transform_keys
+ transform_values
+ map_filter
+ map_zip_with
+ explode
+ explode_outer
+ posexplode
+ posexplode_outer
+ get_json_object
+ json_tuple
+ from_json
+ schema_of_json
+ to_json
+ size
+ sort_array
+ array_max
+ array_min
+ shuffle
+ reverse
+ flatten
+ sequence
+ array_repeat
+ map_keys
+ map_values
+ map_entries
+ map_from_entries
+ arrays_zip
+ map_concat
+ from_csv
+ schema_of_csv
+ to_csv
+
+
+Partition Transformation Functions
+----------------------------------
+.. autosummary::
+ :toctree: api/
+
+ years
+ months
+ days
+ hours
+ bucket
+
+
+Aggregate Functions
+-------------------
+.. autosummary::
+ :toctree: api/
+
+ approxCountDistinct
+ approx_count_distinct
+ avg
+ collect_list
+ collect_set
+ corr
+ count
+ count_distinct
+ countDistinct
+ covar_pop
+ covar_samp
+ first
+ grouping
+ grouping_id
+ kurtosis
+ last
+ max
+ max_by
+ mean
+ min
+ min_by
+ percentile_approx
+ product
+ skewness
+ stddev
+ stddev_pop
+ stddev_samp
+ sum
+ sum_distinct
+ sumDistinct
+ var_pop
+ var_samp
+ variance
+
+
+Window Functions
+----------------
+.. autosummary::
+ :toctree: api/
+
+ cume_dist
+ dense_rank
+ lag
+ lead
+ nth_value
+ ntile
+ percent_rank
+ rank
+ row_number
+
+
+Sort Functions
+--------------
+.. autosummary::
+ :toctree: api/
+
+ asc
+ asc_nulls_first
+ asc_nulls_last
+ desc
+ desc_nulls_first
+ desc_nulls_last
+
+
+String Functions
+----------------
+.. autosummary::
+ :toctree: api/
+
+ ascii
+ base64
+ bit_length
+ concat_ws
+ decode
+ encode
+ format_number
+ format_string
+ initcap
+ instr
+ length
+ lower
+ levenshtein
+ locate
+ lpad
+ ltrim
+ octet_length
+ regexp_extract
+ regexp_replace
+ unbase64
+ rpad
+ repeat
+ rtrim
+ soundex
+ split
+ substring
+ substring_index
+ overlay
+ sentences
+ translate
+ trim
+ upper
+
+
+UDF
+---
+.. autosummary::
+ :toctree: api/
+
+ pandas_udf
+ udf
+
+Misc Functions
+--------------
+.. autosummary::
+ :toctree: api/
+
+ md5
+ sha1
+ sha2
+ crc32
+ hash
+ xxhash64
+ assert_true
+ raise_error
+
diff --git a/python/docs/source/reference/index.rst
b/python/docs/source/reference/pyspark.sql/grouping.rst
similarity index 68%
copy from python/docs/source/reference/index.rst
copy to python/docs/source/reference/pyspark.sql/grouping.rst
index f023b5a8c99..459ef572756 100644
--- a/python/docs/source/reference/index.rst
+++ b/python/docs/source/reference/pyspark.sql/grouping.rst
@@ -16,22 +16,23 @@
under the License.
-=============
-API Reference
-=============
-
-This page lists an overview of all public PySpark modules, classes, functions
and methods.
-
-Pandas API on Spark follows the API specifications of pandas 1.3.
-
-.. toctree::
- :maxdepth: 2
-
- pyspark.sql
- pyspark.pandas/index
- pyspark.ss
- pyspark.ml
- pyspark.streaming
- pyspark.mllib
- pyspark
- pyspark.resource
+========
+Grouping
+========
+.. currentmodule:: pyspark.sql
+
+.. autosummary::
+ :toctree: api/
+
+ GroupedData.agg
+ GroupedData.apply
+ GroupedData.applyInPandas
+ GroupedData.avg
+ GroupedData.cogroup
+ GroupedData.count
+ GroupedData.max
+ GroupedData.mean
+ GroupedData.min
+ GroupedData.pivot
+ GroupedData.sum
+ PandasCogroupedOps.applyInPandas
diff --git a/python/docs/source/reference/index.rst
b/python/docs/source/reference/pyspark.sql/index.rst
similarity index 70%
copy from python/docs/source/reference/index.rst
copy to python/docs/source/reference/pyspark.sql/index.rst
index f023b5a8c99..a8b52f4a1b5 100644
--- a/python/docs/source/reference/index.rst
+++ b/python/docs/source/reference/pyspark.sql/index.rst
@@ -16,22 +16,26 @@
under the License.
-=============
-API Reference
-=============
+=========
+Spark SQL
+=========
-This page lists an overview of all public PySpark modules, classes, functions
and methods.
-
-Pandas API on Spark follows the API specifications of pandas 1.3.
+This page gives an overview of all public Spark SQL API.
.. toctree::
- :maxdepth: 2
-
- pyspark.sql
- pyspark.pandas/index
- pyspark.ss
- pyspark.ml
- pyspark.streaming
- pyspark.mllib
- pyspark
- pyspark.resource
+ :maxdepth: 2
+
+ core_classes
+ spark_session
+ configuration
+ io
+ dataframe
+ column
+ data_types
+ row
+ functions
+ window
+ grouping
+ catalog
+ observation
+ avro
diff --git a/python/docs/source/reference/pyspark.sql/io.rst
b/python/docs/source/reference/pyspark.sql/io.rst
new file mode 100644
index 00000000000..52e4593eead
--- /dev/null
+++ b/python/docs/source/reference/pyspark.sql/io.rst
@@ -0,0 +1,54 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+============
+Input/Output
+============
+.. currentmodule:: pyspark.sql
+
+.. autosummary::
+ :toctree: api/
+
+ DataFrameReader.csv
+ DataFrameReader.format
+ DataFrameReader.jdbc
+ DataFrameReader.json
+ DataFrameReader.load
+ DataFrameReader.option
+ DataFrameReader.options
+ DataFrameReader.orc
+ DataFrameReader.parquet
+ DataFrameReader.schema
+ DataFrameReader.table
+ DataFrameReader.text
+ DataFrameWriter.bucketBy
+ DataFrameWriter.csv
+ DataFrameWriter.format
+ DataFrameWriter.insertInto
+ DataFrameWriter.jdbc
+ DataFrameWriter.json
+ DataFrameWriter.mode
+ DataFrameWriter.option
+ DataFrameWriter.options
+ DataFrameWriter.orc
+ DataFrameWriter.parquet
+ DataFrameWriter.partitionBy
+ DataFrameWriter.save
+ DataFrameWriter.saveAsTable
+ DataFrameWriter.sortBy
+ DataFrameWriter.text
diff --git a/python/docs/source/reference/index.rst
b/python/docs/source/reference/pyspark.sql/observation.rst
similarity index 69%
copy from python/docs/source/reference/index.rst
copy to python/docs/source/reference/pyspark.sql/observation.rst
index f023b5a8c99..52867eda109 100644
--- a/python/docs/source/reference/index.rst
+++ b/python/docs/source/reference/pyspark.sql/observation.rst
@@ -16,22 +16,12 @@
under the License.
-=============
-API Reference
-=============
+===========
+Observation
+===========
+.. currentmodule:: pyspark.sql
-This page lists an overview of all public PySpark modules, classes, functions
and methods.
+.. autosummary::
+ :toctree: api/
-Pandas API on Spark follows the API specifications of pandas 1.3.
-
-.. toctree::
- :maxdepth: 2
-
- pyspark.sql
- pyspark.pandas/index
- pyspark.ss
- pyspark.ml
- pyspark.streaming
- pyspark.mllib
- pyspark
- pyspark.resource
+ Observation.get
diff --git a/python/docs/source/reference/index.rst
b/python/docs/source/reference/pyspark.sql/row.rst
similarity index 69%
copy from python/docs/source/reference/index.rst
copy to python/docs/source/reference/pyspark.sql/row.rst
index f023b5a8c99..1234b8d92ae 100644
--- a/python/docs/source/reference/index.rst
+++ b/python/docs/source/reference/pyspark.sql/row.rst
@@ -16,22 +16,12 @@
under the License.
-=============
-API Reference
-=============
+===
+Row
+===
+.. currentmodule:: pyspark.sql
-This page lists an overview of all public PySpark modules, classes, functions
and methods.
+.. autosummary::
+ :toctree: api/
-Pandas API on Spark follows the API specifications of pandas 1.3.
-
-.. toctree::
- :maxdepth: 2
-
- pyspark.sql
- pyspark.pandas/index
- pyspark.ss
- pyspark.ml
- pyspark.streaming
- pyspark.mllib
- pyspark
- pyspark.resource
+ Row.asDict
diff --git a/python/docs/source/reference/pyspark.sql/spark_session.rst
b/python/docs/source/reference/pyspark.sql/spark_session.rst
new file mode 100644
index 00000000000..d4fb7270a77
--- /dev/null
+++ b/python/docs/source/reference/pyspark.sql/spark_session.rst
@@ -0,0 +1,53 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+=============
+Spark Session
+=============
+.. currentmodule:: pyspark.sql
+
+.. autosummary::
+ :toctree: api/
+
+The entry point to programming Spark with the Dataset and DataFrame API.
+To create a Spark session, you should use ``SparkSession.builder`` attribute.
+See also :class:`SparkSession`.
+
+.. autosummary::
+ :toctree: api/
+
+ SparkSession.builder.appName
+ SparkSession.builder.config
+ SparkSession.builder.enableHiveSupport
+ SparkSession.builder.getOrCreate
+ SparkSession.builder.master
+ SparkSession.catalog
+ SparkSession.conf
+ SparkSession.createDataFrame
+ SparkSession.getActiveSession
+ SparkSession.newSession
+ SparkSession.range
+ SparkSession.read
+ SparkSession.readStream
+ SparkSession.sparkContext
+ SparkSession.sql
+ SparkSession.stop
+ SparkSession.streams
+ SparkSession.table
+ SparkSession.udf
+ SparkSession.version
diff --git a/python/docs/source/reference/index.rst
b/python/docs/source/reference/pyspark.sql/window.rst
similarity index 69%
copy from python/docs/source/reference/index.rst
copy to python/docs/source/reference/pyspark.sql/window.rst
index f023b5a8c99..3625164d0a0 100644
--- a/python/docs/source/reference/index.rst
+++ b/python/docs/source/reference/pyspark.sql/window.rst
@@ -16,22 +16,23 @@
under the License.
-=============
-API Reference
-=============
-
-This page lists an overview of all public PySpark modules, classes, functions
and methods.
-
-Pandas API on Spark follows the API specifications of pandas 1.3.
-
-.. toctree::
- :maxdepth: 2
-
- pyspark.sql
- pyspark.pandas/index
- pyspark.ss
- pyspark.ml
- pyspark.streaming
- pyspark.mllib
- pyspark
- pyspark.resource
+======
+Window
+======
+
+.. currentmodule:: pyspark.sql
+
+.. autosummary::
+ :toctree: api/
+
+ Window.currentRow
+ Window.orderBy
+ Window.partitionBy
+ Window.rangeBetween
+ Window.rowsBetween
+ Window.unboundedFollowing
+ Window.unboundedPreceding
+ WindowSpec.orderBy
+ WindowSpec.partitionBy
+ WindowSpec.rangeBetween
+ WindowSpec.rowsBetween
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
![https://user-images.githubusercontent.com/44108233/169951517-f8b9cb72-7408-46d6-8cd7-15ae890a7a7f.png"](https://user-images.githubusercontent.com/44108233/169951517-f8b9cb72-7408-46d6-8cd7-15ae890a7a7f.png"){kind=link}