[jira] [Assigned] (SPARK-37417) Inline type hints for python/pyspark/ml/linalg/__init__.py

2022-02-06 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz reassigned SPARK-37417: -- Assignee: Maciej Szymkiewicz > Inline type hints for

[jira] [Assigned] (SPARK-36885) Inline type hints for python/pyspark/sql/dataframe.py

2022-02-05 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz reassigned SPARK-36885: -- Assignee: Apache Spark > Inline type hints for

[jira] [Assigned] (SPARK-36885) Inline type hints for python/pyspark/sql/dataframe.py

2022-02-05 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz reassigned SPARK-36885: -- Assignee: (was: Apache Spark) > Inline type hints for

[jira] [Assigned] (SPARK-37408) Inline type hints for python/pyspark/ml/image.py

2022-02-04 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz reassigned SPARK-37408: -- Assignee: Maciej Szymkiewicz (was: Apache Spark) > Inline type hints for

[jira] [Updated] (SPARK-38082) Update minimum numpy version

2022-02-01 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-38082: --- Description: Currently, we use set numpy version in {{extras_require}} to be

[jira] [Updated] (SPARK-38082) Update minimum numpy version

2022-02-01 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-38082: --- Description: Currently, we use set numpy version in {{extras_require}} to be

[jira] [Updated] (SPARK-38082) Update minimum numpy version

2022-02-01 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-38082: --- Description: Currently, we use set numpy version in {{extras_require}} to be

[jira] [Updated] (SPARK-38082) Update minimum numpy version

2022-02-01 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-38082: --- Description: Currently, we use set numpy version in {{extras_require}} to be

[jira] [Commented] (SPARK-38082) Update minimum numpy version

2022-02-01 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17485327#comment-17485327 ] Maciej Szymkiewicz commented on SPARK-38082: cc [~hyukjin.kwon] [~WeichenXu123] [~huaxingao]

[jira] [Updated] (SPARK-38082) Update minimum numpy version

2022-02-01 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-38082: --- Description: Currently, we use set numpy version in {{extras_require}} to be

[jira] [Updated] (SPARK-38082) Update minimum numpy version

2022-02-01 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-38082: --- Description: Currently, we use set numpy version in {{extras_require}} to be

[jira] [Created] (SPARK-38082) Update minimum numpy version

2022-02-01 Thread Maciej Szymkiewicz (Jira)
Maciej Szymkiewicz created SPARK-38082: -- Summary: Update minimum numpy version Key: SPARK-38082 URL: https://issues.apache.org/jira/browse/SPARK-38082 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-38067) Inconsistent missing values handling in Pandas on Spark to_json

2022-02-01 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz reassigned SPARK-38067: -- Assignee: Bjørn Jørgensen > Inconsistent missing values handling in Pandas

[jira] [Resolved] (SPARK-38067) Inconsistent missing values handling in Pandas on Spark to_json

2022-02-01 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz resolved SPARK-38067. Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

[jira] [Updated] (SPARK-38067) Inconsistent missing values handling in Pandas on Spark to_json

2022-02-01 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-38067: --- Description: If {{ps.DataFrame.to_json}} is called without {{path}} argument,

[jira] [Updated] (SPARK-38067) Inconsistent missing values handling in Pandas on Spark to_json

2022-02-01 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-38067: --- Summary: Inconsistent missing values handling in Pandas on Spark to_json (was:

[jira] [Commented] (SPARK-38073) NameError: name 'sc' is not defined when running driver with IPyhon and Pyhon > 3.7

2022-01-30 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17484378#comment-17484378 ] Maciej Szymkiewicz commented on SPARK-38073: [~holden] FYI, since you questioned usage of

[jira] [Created] (SPARK-38073) NameError: name 'sc' is not defined when running driver with IPyhon and Pyhon > 3.7

2022-01-30 Thread Maciej Szymkiewicz (Jira)
Maciej Szymkiewicz created SPARK-38073: -- Summary: NameError: name 'sc' is not defined when running driver with IPyhon and Pyhon > 3.7 Key: SPARK-38073 URL: https://issues.apache.org/jira/browse/SPARK-38073

[jira] [Resolved] (SPARK-37288) Backport update pyspark.since annotation to 3.1 and 3.2

2022-01-24 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz resolved SPARK-37288. Assignee: Maciej Szymkiewicz Resolution: Fixed Issue resolved by pull

[jira] [Assigned] (SPARK-37153) Inline type hints for python/pyspark/profiler.py

2022-01-24 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz reassigned SPARK-37153: -- Assignee: dch nguyen > Inline type hints for python/pyspark/profiler.py >

[jira] [Resolved] (SPARK-37153) Inline type hints for python/pyspark/profiler.py

2022-01-24 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz resolved SPARK-37153. Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

[jira] [Created] (SPARK-37992) Restore mypy version check in dev/lint-python

2022-01-23 Thread Maciej Szymkiewicz (Jira)
Maciej Szymkiewicz created SPARK-37992: -- Summary: Restore mypy version check in dev/lint-python Key: SPARK-37992 URL: https://issues.apache.org/jira/browse/SPARK-37992 Project: Spark

[jira] [Resolved] (SPARK-37981) Deletes columns with all Null as default.

2022-01-21 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz resolved SPARK-37981. Resolution: Duplicate > Deletes columns with all Null as default. >

[jira] [Comment Edited] (SPARK-37981) Deletes columns with all Null as default.

2022-01-21 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17480291#comment-17480291 ] Maciej Szymkiewicz edited comment on SPARK-37981 at 1/21/22, 11:37 PM:

[jira] [Commented] (SPARK-37981) Deletes columns with all Null as default.

2022-01-21 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17480291#comment-17480291 ] Maciej Szymkiewicz commented on SPARK-37981: This doesn't seem valid.

[jira] [Resolved] (SPARK-37972) Typing incompatibilities with numpy==1.22.x

2022-01-21 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz resolved SPARK-37972. Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-37972) Typing incompatibilities with numpy==1.22.x

2022-01-21 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz reassigned SPARK-37972: -- Assignee: Maciej Szymkiewicz > Typing incompatibilities with numpy==1.22.x >

[jira] [Resolved] (SPARK-37083) Inline type hints for python/pyspark/accumulators.py

2022-01-20 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz resolved SPARK-37083. Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-37083) Inline type hints for python/pyspark/accumulators.py

2022-01-20 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz reassigned SPARK-37083: -- Assignee: dch nguyen > Inline type hints for python/pyspark/accumulators.py

[jira] [Updated] (SPARK-37972) Typing incompatibilities with numpy==1.22.x

2022-01-20 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-37972: --- Description: When type checked against {{numpy==1.22}} mypy detects following

[jira] [Updated] (SPARK-37972) Typing incompatibilities with numpy==1.22.x

2022-01-20 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-37972: --- Summary: Typing incompatibilities with numpy==1.22.x (was: Typing

[jira] [Created] (SPARK-37972) Typing incompatibilities with numpy~=1.22

2022-01-20 Thread Maciej Szymkiewicz (Jira)
Maciej Szymkiewicz created SPARK-37972: -- Summary: Typing incompatibilities with numpy~=1.22 Key: SPARK-37972 URL: https://issues.apache.org/jira/browse/SPARK-37972 Project: Spark Issue

[jira] [Commented] (SPARK-37498) test_reuse_worker_of_parallelize_range is flaky

2022-01-14 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17476385#comment-17476385 ] Maciej Szymkiewicz commented on SPARK-37498: ??maybe we should add eventually?? This might

[jira] [Assigned] (SPARK-37902) Update annotations to resolve issues detected with mypy==0.931

2022-01-14 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz reassigned SPARK-37902: -- Assignee: Maciej Szymkiewicz > Update annotations to resolve issues detected

[jira] [Resolved] (SPARK-37902) Update annotations to resolve issues detected with mypy==0.931

2022-01-14 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz resolved SPARK-37902. Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

[jira] [Resolved] (SPARK-37429) Inline type hints for python/pyspark/mllib/linalg/__init__.py

2022-01-14 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz resolved SPARK-37429. Fix Version/s: 3.3.0 Assignee: Maciej Szymkiewicz Resolution:

[jira] [Created] (SPARK-37909) Restore F401 checks

2022-01-14 Thread Maciej Szymkiewicz (Jira)
Maciej Szymkiewicz created SPARK-37909: -- Summary: Restore F401 checks Key: SPARK-37909 URL: https://issues.apache.org/jira/browse/SPARK-37909 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-37909) Restore F403 checks

2022-01-14 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-37909: --- Summary: Restore F403 checks (was: Restore F401 checks) > Restore F403 checks >

[jira] [Created] (SPARK-37902) Update annotations to resolve issues detected with mypy==0.931

2022-01-13 Thread Maciej Szymkiewicz (Jira)
Maciej Szymkiewicz created SPARK-37902: -- Summary: Update annotations to resolve issues detected with mypy==0.931 Key: SPARK-37902 URL: https://issues.apache.org/jira/browse/SPARK-37902 Project:

[jira] [Assigned] (SPARK-37686) Migrate remaining pyspark.sql.functions to _invoke_* style

2022-01-13 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz reassigned SPARK-37686: -- Assignee: Maciej Szymkiewicz > Migrate remaining pyspark.sql.functions to

[jira] [Resolved] (SPARK-37686) Migrate remaining pyspark.sql.functions to _invoke_* style

2022-01-13 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz resolved SPARK-37686. Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

[jira] [Resolved] (SPARK-37418) Inline type hints for python/pyspark/ml/param/__init__.py

2022-01-12 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz resolved SPARK-37418. Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-37418) Inline type hints for python/pyspark/ml/param/__init__.py

2022-01-12 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz reassigned SPARK-37418: -- Assignee: Maciej Szymkiewicz > Inline type hints for

[jira] [Created] (SPARK-37869) Update pytest-mypy-plugins to 1.9.3

2022-01-11 Thread Maciej Szymkiewicz (Jira)
Maciej Szymkiewicz created SPARK-37869: -- Summary: Update pytest-mypy-plugins to 1.9.3 Key: SPARK-37869 URL: https://issues.apache.org/jira/browse/SPARK-37869 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-37852) Enable flake's E741 rule in PySpark

2022-01-10 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz resolved SPARK-37852. Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-37852) Enable flake's E741 rule in PySpark

2022-01-10 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz reassigned SPARK-37852: -- Assignee: Hyukjin Kwon > Enable flake's E741 rule in PySpark >

[jira] [Assigned] (SPARK-37085) Missing overloads in functions accepting both varargs and single arg collection

2022-01-08 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz reassigned SPARK-37085: -- Assignee: Maciej Szymkiewicz > Missing overloads in functions accepting both

[jira] [Resolved] (SPARK-37085) Missing overloads in functions accepting both varargs and single arg collection

2022-01-08 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz resolved SPARK-37085. Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

[jira] [Commented] (SPARK-37417) Inline type hints for python/pyspark/ml/linalg/__init__.py

2022-01-07 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17470783#comment-17470783 ] Maciej Szymkiewicz commented on SPARK-37417: {{ml}} and {{mllib}} linalg types should be

[jira] [Commented] (SPARK-37415) Inline type hints for python/pyspark/ml/util.py

2022-01-07 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17470768#comment-17470768 ] Maciej Szymkiewicz commented on SPARK-37415: I am going to handle this one, once

[jira] [Commented] (SPARK-37416) Inline type hints for python/pyspark/ml/wrapper.py

2022-01-07 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17470767#comment-17470767 ] Maciej Szymkiewicz commented on SPARK-37416: I am going to handle this one, once

[jira] [Resolved] (SPARK-37419) Inline type hints for python/pyspark/ml/param/shared.py

2022-01-07 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz resolved SPARK-37419. Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-37419) Inline type hints for python/pyspark/ml/param/shared.py

2022-01-07 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz reassigned SPARK-37419: -- Assignee: Maciej Szymkiewicz > Inline type hints for

[jira] [Commented] (SPARK-37397) Inline type hints for python/pyspark/ml/base.py

2022-01-07 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17470654#comment-17470654 ] Maciej Szymkiewicz commented on SPARK-37397: I'll handle this once SPARK-37418 prerequisites

[jira] [Assigned] (SPARK-37397) Inline type hints for python/pyspark/ml/base.py

2022-01-07 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz reassigned SPARK-37397: -- Assignee: (was: Maciej Szymkiewicz) > Inline type hints for

[jira] [Assigned] (SPARK-37397) Inline type hints for python/pyspark/ml/base.py

2022-01-07 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz reassigned SPARK-37397: -- Assignee: Maciej Szymkiewicz > Inline type hints for

[jira] [Commented] (SPARK-37601) Could/should sql.DataFrame.transform accept function parameters?

2022-01-06 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17470304#comment-17470304 ] Maciej Szymkiewicz commented on SPARK-37601: Thanks for pointing this out [~dongjoon]! >

[jira] [Resolved] (SPARK-37788) ColumnOrName vs Column in PySpark Functions module

2022-01-05 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz resolved SPARK-37788. Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-37788) ColumnOrName vs Column in PySpark Functions module

2022-01-05 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz reassigned SPARK-37788: -- Assignee: Daniel Davies > ColumnOrName vs Column in PySpark Functions module

[jira] [Commented] (SPARK-37425) Inline type hints for python/pyspark/mllib/recommendation.py

2022-01-04 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17468609#comment-17468609 ] Maciej Szymkiewicz commented on SPARK-37425: Hi [~amirkdv]. For the whole {{pyspark.mllib}}

[jira] [Resolved] (SPARK-37601) Could/should sql.DataFrame.transform accept function parameters?

2021-12-30 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz resolved SPARK-37601. Fix Version/s: 3.3.0 Resolution: Fixed > Could/should

[jira] [Assigned] (SPARK-37738) PySpark date_add only accepts an integer as it's second parameter

2021-12-30 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz reassigned SPARK-37738: -- Assignee: Daniel Davies > PySpark date_add only accepts an integer as it's

[jira] [Assigned] (SPARK-37738) PySpark date_add only accepts an integer as it's second parameter

2021-12-30 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz reassigned SPARK-37738: -- Assignee: (was: Maciej Szymkiewicz) > PySpark date_add only accepts an

[jira] [Assigned] (SPARK-37738) PySpark date_add only accepts an integer as it's second parameter

2021-12-30 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz reassigned SPARK-37738: -- Assignee: Maciej Szymkiewicz > PySpark date_add only accepts an integer as

[jira] [Resolved] (SPARK-37738) PySpark date_add only accepts an integer as it's second parameter

2021-12-30 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz resolved SPARK-37738. Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-37754) Fix black version in dev/reformat-python

2021-12-28 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz reassigned SPARK-37754: -- Assignee: Chia-Ping Tsai > Fix black version in dev/reformat-python >

[jira] [Resolved] (SPARK-37754) Fix black version in dev/reformat-python

2021-12-28 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz resolved SPARK-37754. Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

[jira] [Created] (SPARK-37737) Update Black to 21.12.b0

2021-12-25 Thread Maciej Szymkiewicz (Jira)
Maciej Szymkiewicz created SPARK-37737: -- Summary: Update Black to 21.12.b0 Key: SPARK-37737 URL: https://issues.apache.org/jira/browse/SPARK-37737 Project: Spark Issue Type: Improvement

[jira] [Resolved] (SPARK-37407) Inline type hints for python/pyspark/ml/functions.py

2021-12-25 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz resolved SPARK-37407. Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-37407) Inline type hints for python/pyspark/ml/functions.py

2021-12-25 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz reassigned SPARK-37407: -- Assignee: dch nguyen > Inline type hints for python/pyspark/ml/functions.py

[jira] [Commented] (SPARK-34544) pyspark toPandas() should return pd.DataFrame

2021-12-22 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17464189#comment-17464189 ] Maciej Szymkiewicz commented on SPARK-34544: Thanks [~hyukjin.kwon]! > pyspark toPandas()

[jira] [Resolved] (SPARK-37704) Update mypy in tests to 0.920

2021-12-21 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz resolved SPARK-37704. Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-37704) Update mypy in tests to 0.920

2021-12-21 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz reassigned SPARK-37704: -- Assignee: Maciej Szymkiewicz > Update mypy in tests to 0.920 >

[jira] [Updated] (SPARK-37704) Update mypy in tests to 0.920

2021-12-21 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-37704: --- Component/s: Build (was: Project Infra) > Update mypy in tests

[jira] [Updated] (SPARK-37704) Update mypy in tests to 0.920

2021-12-21 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-37704: --- Component/s: (was: Build) > Update mypy in tests to 0.920 >

[jira] [Created] (SPARK-37704) Update mypy in tests to 0.920

2021-12-21 Thread Maciej Szymkiewicz (Jira)
Maciej Szymkiewicz created SPARK-37704: -- Summary: Update mypy in tests to 0.920 Key: SPARK-37704 URL: https://issues.apache.org/jira/browse/SPARK-37704 Project: Spark Issue Type:

[jira] [Created] (SPARK-37686) Migrare remaining pyspark.sql.functions to _invoke_* style

2021-12-19 Thread Maciej Szymkiewicz (Jira)
Maciej Szymkiewicz created SPARK-37686: -- Summary: Migrare remaining pyspark.sql.functions to _invoke_* style Key: SPARK-37686 URL: https://issues.apache.org/jira/browse/SPARK-37686 Project: Spark

[jira] [Updated] (SPARK-37686) Migrate remaining pyspark.sql.functions to _invoke_* style

2021-12-19 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-37686: --- Summary: Migrate remaining pyspark.sql.functions to _invoke_* style (was: Migrare

[jira] [Commented] (SPARK-37683) Nondeterministic type checker error in data tests

2021-12-18 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17461881#comment-17461881 ] Maciej Szymkiewicz commented on SPARK-37683: cc [~hyukjin.kwon] FYI > Nondeterministic type

[jira] [Created] (SPARK-37683) Nondeterministic type checker error in data tests

2021-12-18 Thread Maciej Szymkiewicz (Jira)
Maciej Szymkiewicz created SPARK-37683: -- Summary: Nondeterministic type checker error in data tests Key: SPARK-37683 URL: https://issues.apache.org/jira/browse/SPARK-37683 Project: Spark

[jira] [Commented] (SPARK-37678) Incorrect annotations in SeriesGroupBy._cleanup_and_return

2021-12-17 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17461679#comment-17461679 ] Maciej Szymkiewicz commented on SPARK-37678: ??It's an actual implementation of an abstract

[jira] [Resolved] (SPARK-37620) Use more precise types for SparkContext Optional fields (i.e. _gateway, _jvm)

2021-12-17 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz resolved SPARK-37620. Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-37620) Use more precise types for SparkContext Optional fields (i.e. _gateway, _jvm)

2021-12-17 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz reassigned SPARK-37620: -- Assignee: Byron Hsu > Use more precise types for SparkContext Optional

[jira] [Commented] (SPARK-37678) Incorrect annotations in SeriesGroupBy._cleanup_and_return

2021-12-17 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17461609#comment-17461609 ] Maciej Szymkiewicz commented on SPARK-37678: cc [~ueshin] Do you have any thoughts on this

[jira] [Created] (SPARK-37678) Incorrect annotations in SeriesGroupBy._cleanup_and_return

2021-12-17 Thread Maciej Szymkiewicz (Jira)
Maciej Szymkiewicz created SPARK-37678: -- Summary: Incorrect annotations in SeriesGroupBy._cleanup_and_return Key: SPARK-37678 URL: https://issues.apache.org/jira/browse/SPARK-37678 Project:

[jira] [Updated] (SPARK-37668) 'Index' object has no attribute 'levels' in pyspark.pandas.frame.DataFrame.insert

2021-12-16 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-37668: --- Summary: 'Index' object has no attribute 'levels' in

[jira] [Commented] (SPARK-37668) 'Index' object has no attribute 'levels'

2021-12-16 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17461039#comment-17461039 ] Maciej Szymkiewicz commented on SPARK-37668: cc [~hyukjin.kwon], [~itholic], [~ueshin],

[jira] [Created] (SPARK-37668) 'Index' object has no attribute 'levels'

2021-12-16 Thread Maciej Szymkiewicz (Jira)
Maciej Szymkiewicz created SPARK-37668: -- Summary: 'Index' object has no attribute 'levels' Key: SPARK-37668 URL: https://issues.apache.org/jira/browse/SPARK-37668 Project: Spark Issue

[jira] [Updated] (SPARK-37620) Use more precise types for SparkContext Optional fields (i.e. _gateway, _jvm)

2021-12-14 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-37620: --- Parent: SPARK-37094 Issue Type: Sub-task (was: Improvement) > Use more

[jira] [Resolved] (SPARK-37146) Inline type hints for python/pyspark/__init__.py

2021-12-14 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz resolved SPARK-37146. Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-37146) Inline type hints for python/pyspark/__init__.py

2021-12-14 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz reassigned SPARK-37146: -- Assignee: dch nguyen > Inline type hints for python/pyspark/__init__.py >

[jira] [Updated] (SPARK-37632) Drop code targetting Python < 3.7

2021-12-13 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-37632: --- Description: We should drop parts of code that target Python 3.6 (or less if these

[jira] [Created] (SPARK-37632) Drop code targetting Python < 3.7

2021-12-13 Thread Maciej Szymkiewicz (Jira)
Maciej Szymkiewicz created SPARK-37632: -- Summary: Drop code targetting Python < 3.7 Key: SPARK-37632 URL: https://issues.apache.org/jira/browse/SPARK-37632 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-36969) Inline type hints for SparkContext

2021-12-13 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz resolved SPARK-36969. Fix Version/s: 3.3.0 Resolution: Duplicate > Inline type hints for

[jira] [Commented] (SPARK-37620) Use more precise types for SparkContext Optional fields (i.e. _gateway, _jvm)

2021-12-12 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17458050#comment-17458050 ] Maciej Szymkiewicz commented on SPARK-37620: FYI [~ueshin] [~itholic] [~XinrongM]

[jira] [Created] (SPARK-37620) Use more precise types for SparkContext Optional fields (i.e. _gateway, _jvm)

2021-12-12 Thread Maciej Szymkiewicz (Jira)
Maciej Szymkiewicz created SPARK-37620: -- Summary: Use more precise types for SparkContext Optional fields (i.e. _gateway, _jvm) Key: SPARK-37620 URL: https://issues.apache.org/jira/browse/SPARK-37620

[jira] [Resolved] (SPARK-37152) Inline type hints for python/pyspark/context.py

2021-12-12 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz resolved SPARK-37152. Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-37152) Inline type hints for python/pyspark/context.py

2021-12-12 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz reassigned SPARK-37152: -- Assignee: Byron Hsu > Inline type hints for python/pyspark/context.py >

[jira] [Commented] (SPARK-37551) Argument 1 to "rename" of "DataFrame" has incompatible type in pandas.LocIndexerLike.__getitem__

2021-12-05 Thread Maciej Szymkiewicz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17453626#comment-17453626 ] Maciej Szymkiewicz commented on SPARK-37551: cc [~itholic] [~ueshin], since you're both

[jira] [Created] (SPARK-37551) Argument 1 to "rename" of "DataFrame" has incompatible type in pandas.LocIndexerLike.__getitem__

2021-12-05 Thread Maciej Szymkiewicz (Jira)
Maciej Szymkiewicz created SPARK-37551: -- Summary: Argument 1 to "rename" of "DataFrame" has incompatible type in pandas.LocIndexerLike.__getitem__ Key: SPARK-37551 URL:

<    1   2   3   4   5   6   7   >