[jira] [Resolved] (SPARK-36887) Inline type hints for python/pyspark/sql/conf.py

2021-10-06 Thread dch nguyen (Jira) 


 [ 
https://issues.apache.org/jira/browse/SPARK-36887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

dch nguyen resolved SPARK-36887.

Resolution: Resolved

This issue is resolved by https://issues.apache.org/jira/browse/SPARK-36906

> Inline type hints for python/pyspark/sql/conf.py
> 
>
> Key: SPARK-36887
> URL: https://issues.apache.org/jira/browse/SPARK-36887
> Project: Spark
>  Issue Type: Sub-task
>  Components: PySpark
>Affects Versions: 3.3.0
>Reporter: dgd_contributor
>Priority: Major
>
> Inline type hints for python/pyspark/sql/session.py from Inline type hints 
> for python/pyspark/sql/conf.pyi.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-36930) Support ps.MultiIndex.dtypes

2021-10-04 Thread dch nguyen (Jira) 


 [ 
https://issues.apache.org/jira/browse/SPARK-36930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

dch nguyen updated SPARK-36930:
---
Description: 
when MultiIndex.dtypes is supported, we can use:


{code:java}
>>> idx = pd.MultiIndex.from_arrays([[0, 1, 2, 3, 4, 5, 6, 7, 8], [1, 2, 3, 4, 
>>> 5, 6, 7, 8, 9]], names=("zero", "one"))
>>> pdf = pd.DataFrame(
... {"a": [1, 2, 3, 4, 5, 6, 7, 8, 9], "b": [4, 5, 6, 3, 2, 1, 0, 0, 0]},
... index=idx,
... )
>>> psdf = ps.from_pandas(pdf)
>>> ps.DataFrame[psdf.ipsdf.dtypes]
psdf.iat  psdf.idxmin(  psdf.indexpsdf.insert(  
psdf.isna(psdf.items(   psdf.iterrows(
psdf.idxmax(  psdf.iloc psdf.info(psdf.isin(
psdf.isnull(  psdf.iteritems(   psdf.itertuples(
>>> ps.DataFrame[psdf.index.dtypes, psdf.dtypes]
typing.Tuple[pyspark.pandas.typedef.typehints.IndexNameType, 
pyspark.pandas.typedef.typehints.IndexNameType, 
pyspark.pandas.typedef.typehints.NameType, 
pyspark.pandas.typedef.typehints.NameType]
{code}


> Support ps.MultiIndex.dtypes
> 
>
> Key: SPARK-36930
> URL: https://issues.apache.org/jira/browse/SPARK-36930
> Project: Spark
>  Issue Type: Sub-task
>  Components: PySpark
>Affects Versions: 3.3.0
>Reporter: dch nguyen
>Priority: Major
>
> when MultiIndex.dtypes is supported, we can use:
> {code:java}
> >>> idx = pd.MultiIndex.from_arrays([[0, 1, 2, 3, 4, 5, 6, 7, 8], [1, 2, 3, 
> >>> 4, 5, 6, 7, 8, 9]], names=("zero", "one"))
> >>> pdf = pd.DataFrame(
> ... {"a": [1, 2, 3, 4, 5, 6, 7, 8, 9], "b": [4, 5, 6, 3, 2, 1, 0, 0, 0]},
> ... index=idx,
> ... )
> >>> psdf = ps.from_pandas(pdf)
> >>> ps.DataFrame[psdf.ipsdf.dtypes]
> psdf.iat  psdf.idxmin(  psdf.indexpsdf.insert(  
> psdf.isna(psdf.items(   psdf.iterrows(
> psdf.idxmax(  psdf.iloc psdf.info(psdf.isin(
> psdf.isnull(  psdf.iteritems(   psdf.itertuples(
> >>> ps.DataFrame[psdf.index.dtypes, psdf.dtypes]
> typing.Tuple[pyspark.pandas.typedef.typehints.IndexNameType, 
> pyspark.pandas.typedef.typehints.IndexNameType, 
> pyspark.pandas.typedef.typehints.NameType, 
> pyspark.pandas.typedef.typehints.NameType]
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-36930) Support ps.MultiIndex.dtypes

2021-10-04 Thread dch nguyen (Jira) 


 [ 
https://issues.apache.org/jira/browse/SPARK-36930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

dch nguyen updated SPARK-36930:
---
Description: 
when MultiIndex.dtypes is supported, we can use:
{code:java}
>>> idx = pd.MultiIndex.from_arrays([[0, 1, 2, 3, 4, 5, 6, 7, 8], [1, 2, 3, 4, 
>>> 5, 6, 7, 8, 9]], names=("zero", "one"))
>>> pdf = pd.DataFrame(
... {"a": [1, 2, 3, 4, 5, 6, 7, 8, 9], "b": [4, 5, 6, 3, 2, 1, 0, 0, 0]},
... index=idx,
... )
>>> psdf = ps.from_pandas(pdf)

>>> ps.DataFrame[psdf.index.dtypes, psdf.dtypes]
typing.Tuple[pyspark.pandas.typedef.typehints.IndexNameType, 
pyspark.pandas.typedef.typehints.IndexNameType, 
pyspark.pandas.typedef.typehints.NameType, 
pyspark.pandas.typedef.typehints.NameType]
{code}

  was:
when MultiIndex.dtypes is supported, we can use:


{code:java}
>>> idx = pd.MultiIndex.from_arrays([[0, 1, 2, 3, 4, 5, 6, 7, 8], [1, 2, 3, 4, 
>>> 5, 6, 7, 8, 9]], names=("zero", "one"))
>>> pdf = pd.DataFrame(
... {"a": [1, 2, 3, 4, 5, 6, 7, 8, 9], "b": [4, 5, 6, 3, 2, 1, 0, 0, 0]},
... index=idx,
... )
>>> psdf = ps.from_pandas(pdf)
>>> ps.DataFrame[psdf.ipsdf.dtypes]
psdf.iat  psdf.idxmin(  psdf.indexpsdf.insert(  
psdf.isna(psdf.items(   psdf.iterrows(
psdf.idxmax(  psdf.iloc psdf.info(psdf.isin(
psdf.isnull(  psdf.iteritems(   psdf.itertuples(
>>> ps.DataFrame[psdf.index.dtypes, psdf.dtypes]
typing.Tuple[pyspark.pandas.typedef.typehints.IndexNameType, 
pyspark.pandas.typedef.typehints.IndexNameType, 
pyspark.pandas.typedef.typehints.NameType, 
pyspark.pandas.typedef.typehints.NameType]
{code}



> Support ps.MultiIndex.dtypes
> 
>
> Key: SPARK-36930
> URL: https://issues.apache.org/jira/browse/SPARK-36930
> Project: Spark
>  Issue Type: Sub-task
>  Components: PySpark
>Affects Versions: 3.3.0
>Reporter: dch nguyen
>Priority: Major
>
> when MultiIndex.dtypes is supported, we can use:
> {code:java}
> >>> idx = pd.MultiIndex.from_arrays([[0, 1, 2, 3, 4, 5, 6, 7, 8], [1, 2, 3, 
> >>> 4, 5, 6, 7, 8, 9]], names=("zero", "one"))
> >>> pdf = pd.DataFrame(
> ... {"a": [1, 2, 3, 4, 5, 6, 7, 8, 9], "b": [4, 5, 6, 3, 2, 1, 0, 0, 0]},
> ... index=idx,
> ... )
> >>> psdf = ps.from_pandas(pdf)
> >>> ps.DataFrame[psdf.index.dtypes, psdf.dtypes]
> typing.Tuple[pyspark.pandas.typedef.typehints.IndexNameType, 
> pyspark.pandas.typedef.typehints.IndexNameType, 
> pyspark.pandas.typedef.typehints.NameType, 
> pyspark.pandas.typedef.typehints.NameType]
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-36930) Support ps.MultiIndex.dtypes

2021-10-04 Thread dch nguyen (Jira) 


 [ 
https://issues.apache.org/jira/browse/SPARK-36930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

dch nguyen updated SPARK-36930:
---
Summary: Support ps.MultiIndex.dtypes  (was: Support ps.DataFrame.dtypes)

> Support ps.MultiIndex.dtypes
> 
>
> Key: SPARK-36930
> URL: https://issues.apache.org/jira/browse/SPARK-36930
> Project: Spark
>  Issue Type: Sub-task
>  Components: PySpark
>Affects Versions: 3.3.0
>Reporter: dch nguyen
>Priority: Major
>
> When DF.dtypes is supported, we can use
>  
> {code:java}
> >>> pdf = pd.DataFrame(
> ... {"a": [1, 2, 3, 4, 5, 6, 7, 8, 9], "b": [4, 5, 6, 3, 2, 1, 0, 0, 0]},
> ... )
> >>> psdf = ps.from_pandas(pdf)
> >>> psdf.dtypes
> aint64
> bint64
> dtype: object
> >>> ps.DataFrame[psdf.dtypes]
> typing.Tuple[pyspark.pandas.typedef.typehints.NameType, 
> pyspark.pandas.typedef.typehints.NameType]
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-36930) Support ps.MultiIndex.dtypes

2021-10-04 Thread dch nguyen (Jira) 


 [ 
https://issues.apache.org/jira/browse/SPARK-36930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

dch nguyen updated SPARK-36930:
---
Description: (was: When DF.dtypes is supported, we can use

 
{code:java}
>>> pdf = pd.DataFrame(
... {"a": [1, 2, 3, 4, 5, 6, 7, 8, 9], "b": [4, 5, 6, 3, 2, 1, 0, 0, 0]},
... )
>>> psdf = ps.from_pandas(pdf)
>>> psdf.dtypes
aint64
bint64
dtype: object
>>> ps.DataFrame[psdf.dtypes]
typing.Tuple[pyspark.pandas.typedef.typehints.NameType, 
pyspark.pandas.typedef.typehints.NameType]
{code})

> Support ps.MultiIndex.dtypes
> 
>
> Key: SPARK-36930
> URL: https://issues.apache.org/jira/browse/SPARK-36930
> Project: Spark
>  Issue Type: Sub-task
>  Components: PySpark
>Affects Versions: 3.3.0
>Reporter: dch nguyen
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-36930) Support ps.DataFrame.dtypes

2021-10-04 Thread dch nguyen (Jira) 


[ 
https://issues.apache.org/jira/browse/SPARK-36930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17424288#comment-17424288
 ] 

dch nguyen commented on SPARK-36930:


working on this.

> Support ps.DataFrame.dtypes
> ---
>
> Key: SPARK-36930
> URL: https://issues.apache.org/jira/browse/SPARK-36930
> Project: Spark
>  Issue Type: Sub-task
>  Components: PySpark
>Affects Versions: 3.3.0
>Reporter: dch nguyen
>Priority: Major
>
> When DF.dtypes is supported, we can use
>  
> {code:java}
> >>> pdf = pd.DataFrame(
> ... {"a": [1, 2, 3, 4, 5, 6, 7, 8, 9], "b": [4, 5, 6, 3, 2, 1, 0, 0, 0]},
> ... )
> >>> psdf = ps.from_pandas(pdf)
> >>> psdf.dtypes
> aint64
> bint64
> dtype: object
> >>> ps.DataFrame[psdf.dtypes]
> typing.Tuple[pyspark.pandas.typedef.typehints.NameType, 
> pyspark.pandas.typedef.typehints.NameType]
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-36930) Support ps.DataFrame.dtypes

2021-10-04 Thread dch nguyen (Jira) 


 [ 
https://issues.apache.org/jira/browse/SPARK-36930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

dch nguyen updated SPARK-36930:
---
Description: 
When DF.dtypes is supported, we can use

 
{code:java}
>>> pdf = pd.DataFrame(
... {"a": [1, 2, 3, 4, 5, 6, 7, 8, 9], "b": [4, 5, 6, 3, 2, 1, 0, 0, 0]},
... )
>>> psdf = ps.from_pandas(pdf)
>>> psdf.dtypes
aint64
bint64
dtype: object
>>> ps.DataFrame[psdf.dtypes]
typing.Tuple[pyspark.pandas.typedef.typehints.NameType, 
pyspark.pandas.typedef.typehints.NameType]
{code}

> Support ps.DataFrame.dtypes
> ---
>
> Key: SPARK-36930
> URL: https://issues.apache.org/jira/browse/SPARK-36930
> Project: Spark
>  Issue Type: Sub-task
>  Components: PySpark
>Affects Versions: 3.3.0
>Reporter: dch nguyen
>Priority: Major
>
> When DF.dtypes is supported, we can use
>  
> {code:java}
> >>> pdf = pd.DataFrame(
> ... {"a": [1, 2, 3, 4, 5, 6, 7, 8, 9], "b": [4, 5, 6, 3, 2, 1, 0, 0, 0]},
> ... )
> >>> psdf = ps.from_pandas(pdf)
> >>> psdf.dtypes
> aint64
> bint64
> dtype: object
> >>> ps.DataFrame[psdf.dtypes]
> typing.Tuple[pyspark.pandas.typedef.typehints.NameType, 
> pyspark.pandas.typedef.typehints.NameType]
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Created] (SPARK-36930) Support ps.DataFrame.dtypes

2021-10-04 Thread dch nguyen (Jira) 
dch nguyen created SPARK-36930:
--

 Summary: Support ps.DataFrame.dtypes
 Key: SPARK-36930
 URL: https://issues.apache.org/jira/browse/SPARK-36930
 Project: Spark
  Issue Type: Sub-task
  Components: PySpark
Affects Versions: 3.3.0
Reporter: dch nguyen






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

<    1   2