[jira] [Updated] (SPARK-30542) Two Spark structured streaming jobs cannot write to same base path

[ https://issues.apache.org/jira/browse/SPARK-30542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sivakumar updated SPARK-30542: -- Description: Hi All, I have two structured streaming jobs which should write data to the same base

[jira] [Issue Comment Deleted] (SPARK-29231) Constraints should be inferred from cast equality constraint

[ https://issues.apache.org/jira/browse/SPARK-29231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-29231: Comment: was deleted (was: I'm working on.) > Constraints should be inferred from cast equality

[jira] [Created] (SPARK-30543) RandomForest add Param bootstrap to control sampling method

zhengruifeng created SPARK-30543: Summary: RandomForest add Param bootstrap to control sampling method Key: SPARK-30543 URL: https://issues.apache.org/jira/browse/SPARK-30543 Project: Spark

[jira] [Updated] (SPARK-30542) Two Spark structured streaming jobs cannot write to same base path

[ https://issues.apache.org/jira/browse/SPARK-30542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sivakumar updated SPARK-30542: -- Description: Hi All, I have two structured streaming jobs which should write data to the same base

[jira] [Created] (SPARK-30542) Two Spark structured streaming jobs cannot write to same base path

Sivakumar created SPARK-30542: - Summary: Two Spark structured streaming jobs cannot write to same base path Key: SPARK-30542 URL: https://issues.apache.org/jira/browse/SPARK-30542 Project: Spark

[jira] [Created] (SPARK-30541) Flaky test: org.apache.spark.sql.kafka010.KafkaDelegationTokenSuite

Jungtaek Lim created SPARK-30541: Summary: Flaky test: org.apache.spark.sql.kafka010.KafkaDelegationTokenSuite Key: SPARK-30541 URL: https://issues.apache.org/jira/browse/SPARK-30541 Project: Spark

[jira] [Created] (SPARK-30540) HistoryServer application link is incorrect when one application having multiple attempts

Junfan Zhang created SPARK-30540: Summary: HistoryServer application link is incorrect when one application having multiple attempts Key: SPARK-30540 URL: https://issues.apache.org/jira/browse/SPARK-30540

[jira] [Comment Edited] (SPARK-30325) markPartitionCompleted cause task status inconsistent

[ https://issues.apache.org/jira/browse/SPARK-30325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017758#comment-17017758 ] Dongjoon Hyun edited comment on SPARK-30325 at 1/17/20 7:15 AM: Hi, All.

[jira] [Commented] (SPARK-30325) markPartitionCompleted cause task status inconsistent

[ https://issues.apache.org/jira/browse/SPARK-30325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017758#comment-17017758 ] Dongjoon Hyun commented on SPARK-30325: --- Hi, All. Since there is no UT, it's difficult to

[jira] [Updated] (SPARK-30325) markPartitionCompleted cause task status inconsistent

[ https://issues.apache.org/jira/browse/SPARK-30325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30325: -- Affects Version/s: 2.4.1 2.4.2 2.4.3 >

[jira] [Resolved] (SPARK-29572) add v1 read fallback API in DS v2

[ https://issues.apache.org/jira/browse/SPARK-29572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-29572. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26231

[jira] [Updated] (SPARK-29231) Constraints should be inferred from cast equality constraint

[ https://issues.apache.org/jira/browse/SPARK-29231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-29231: Summary: Constraints should be inferred from cast equality constraint (was: Can not infer an

[jira] [Updated] (SPARK-30449) Introducing get_dummies method in pyspark

[ https://issues.apache.org/jira/browse/SPARK-30449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krishna Kumar Tiwari updated SPARK-30449: - Issue Type: New Feature (was: Task) > Introducing get_dummies method in

[jira] [Created] (SPARK-30539) DataFrame.tail in PySpark API

Hyukjin Kwon created SPARK-30539: Summary: DataFrame.tail in PySpark API Key: SPARK-30539 URL: https://issues.apache.org/jira/browse/SPARK-30539 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-30538) A not very elegant way to control ouput small file

angerszhu created SPARK-30538: - Summary: A not very elegant way to control ouput small file Key: SPARK-30538 URL: https://issues.apache.org/jira/browse/SPARK-30538 Project: Spark Issue Type:

[jira] [Created] (SPARK-30537) toPandas gets wrong dtypes when applied on empty DF when Arrow enabled

Hyukjin Kwon created SPARK-30537: Summary: toPandas gets wrong dtypes when applied on empty DF when Arrow enabled Key: SPARK-30537 URL: https://issues.apache.org/jira/browse/SPARK-30537 Project:

[jira] [Assigned] (SPARK-30499) Remove SQL config spark.sql.execution.pandas.respectSessionTimeZone

[ https://issues.apache.org/jira/browse/SPARK-30499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-30499: Assignee: Maxim Gekk > Remove SQL config

[jira] [Resolved] (SPARK-30499) Remove SQL config spark.sql.execution.pandas.respectSessionTimeZone

[ https://issues.apache.org/jira/browse/SPARK-30499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30499. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27218

[jira] [Created] (SPARK-30536) Sort-merge join operator spilling performance improvements

Sinisa Knezevic created SPARK-30536: --- Summary: Sort-merge join operator spilling performance improvements Key: SPARK-30536 URL: https://issues.apache.org/jira/browse/SPARK-30536 Project: Spark

[jira] [Commented] (SPARK-27868) Better document shuffle / RPC listen backlog

[ https://issues.apache.org/jira/browse/SPARK-27868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017615#comment-17017615 ] Dongjoon Hyun commented on SPARK-27868: --- Oh, do you want me revert back? Or, I can make a

[jira] [Assigned] (SPARK-30534) Use mvn in `dev/scalastyle`

[ https://issues.apache.org/jira/browse/SPARK-30534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-30534: - Assignee: Dongjoon Hyun > Use mvn in `dev/scalastyle` > --- >

[jira] [Resolved] (SPARK-30534) Use mvn in `dev/scalastyle`

[ https://issues.apache.org/jira/browse/SPARK-30534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-30534. --- Fix Version/s: 3.0.0 2.4.5 Resolution: Fixed Issue resolved by

[jira] [Commented] (SPARK-30420) Commands involved with namespace go thru the new resolution framework.

[ https://issues.apache.org/jira/browse/SPARK-30420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017558#comment-17017558 ] Terry Kim commented on SPARK-30420: --- This is resolved by: https://github.com/apache/spark/pull/27095

[jira] [Updated] (SPARK-30535) Migrate ALTER TABLE commands to the new resolution framework

[ https://issues.apache.org/jira/browse/SPARK-30535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Terry Kim updated SPARK-30535: -- Description: Migrate ALTER TABLE commands to the new resolution framework introduced in SPARK-30214

[jira] [Created] (SPARK-30535) Migrate ALTER TABLE commands to the new resolution framework

Terry Kim created SPARK-30535: - Summary: Migrate ALTER TABLE commands to the new resolution framework Key: SPARK-30535 URL: https://issues.apache.org/jira/browse/SPARK-30535 Project: Spark

[jira] [Created] (SPARK-30534) Use mvn in `dev/scalastyle`

Dongjoon Hyun created SPARK-30534: - Summary: Use mvn in `dev/scalastyle` Key: SPARK-30534 URL: https://issues.apache.org/jira/browse/SPARK-30534 Project: Spark Issue Type: Improvement

[jira] [Assigned] (SPARK-29950) Deleted excess executors can connect back to driver in K8S with dyn alloc on

[ https://issues.apache.org/jira/browse/SPARK-29950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Masiero Vanzin reassigned SPARK-29950: -- Assignee: Marcelo Masiero Vanzin > Deleted excess executors can

[jira] [Resolved] (SPARK-29950) Deleted excess executors can connect back to driver in K8S with dyn alloc on

[ https://issues.apache.org/jira/browse/SPARK-29950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Masiero Vanzin resolved SPARK-29950. Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull

[jira] [Commented] (SPARK-29708) Different answers in aggregates of duplicate grouping sets

[ https://issues.apache.org/jira/browse/SPARK-29708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017478#comment-17017478 ] Dongjoon Hyun commented on SPARK-29708: --- This is backported to branch-2.4 via

[jira] [Updated] (SPARK-29708) Different answers in aggregates of duplicate grouping sets

[ https://issues.apache.org/jira/browse/SPARK-29708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29708: -- Affects Version/s: 2.4.0 2.4.1 2.4.2

[jira] [Updated] (SPARK-30533) Add classes to represent Java Regressors and RegressionModels

[ https://issues.apache.org/jira/browse/SPARK-30533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-30533: --- Parent: SPARK-28958 Issue Type: Sub-task (was: Bug) > Add classes to

[jira] [Updated] (SPARK-29708) Different answers in aggregates of duplicate grouping sets

[ https://issues.apache.org/jira/browse/SPARK-29708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29708: -- Fix Version/s: 2.4.5 > Different answers in aggregates of duplicate grouping sets >

[jira] [Commented] (SPARK-27750) Standalone scheduler - ability to prioritize applications over drivers, many drivers act like Denial of Service

[ https://issues.apache.org/jira/browse/SPARK-27750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017448#comment-17017448 ] t oo commented on SPARK-27750: -- yes, I hit this. 'spark standalone' is the RM, it has no pools. I am

[jira] [Commented] (SPARK-30512) Use a dedicated boss event group loop in the netty pipeline for external shuffle service

[ https://issues.apache.org/jira/browse/SPARK-30512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017447#comment-17017447 ] Chandni Singh commented on SPARK-30512: --- [https://github.com/apache/spark/pull/27240] > Use a

[jira] [Issue Comment Deleted] (SPARK-30512) Use a dedicated boss event group loop in the netty pipeline for external shuffle service

[ https://issues.apache.org/jira/browse/SPARK-30512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chandni Singh updated SPARK-30512: -- Comment: was deleted (was: [https://github.com/apache/spark/pull/27240]) > Use a dedicated

[jira] [Commented] (SPARK-6235) Address various 2G limits

[ https://issues.apache.org/jira/browse/SPARK-6235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017438#comment-17017438 ] Samuel Shepard commented on SPARK-6235: --- [~irashid] I followed your suggestion of looking in the

[jira] [Created] (SPARK-30533) Add classes to represent Java Regressors and RegressionModels

Maciej Szymkiewicz created SPARK-30533: -- Summary: Add classes to represent Java Regressors and RegressionModels Key: SPARK-30533 URL: https://issues.apache.org/jira/browse/SPARK-30533 Project:

[jira] [Created] (SPARK-30532) DataFrameStatFunctions.approxQuantile doesn't work with TABLE.COLUMN syntax

Chris Suchanek created SPARK-30532: -- Summary: DataFrameStatFunctions.approxQuantile doesn't work with TABLE.COLUMN syntax Key: SPARK-30532 URL: https://issues.apache.org/jira/browse/SPARK-30532

[jira] [Updated] (SPARK-30491) Enable dependency audit files to tell dependency classifier

[ https://issues.apache.org/jira/browse/SPARK-30491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-30491: - Description: Current dependency audit files under `dev/deps` only show jar names. And there

[jira] [Updated] (SPARK-30511) Spark marks intentionally killed speculative tasks as pending leads to holding idle executors

[ https://issues.apache.org/jira/browse/SPARK-30511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zebing Lin updated SPARK-30511: --- Description: *TL;DR* When speculative tasks fail/get killed, they are still considered as pending

[jira] [Updated] (SPARK-30511) Spark marks intentionally killed speculative tasks as pending leads to holding idle executors

[ https://issues.apache.org/jira/browse/SPARK-30511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zebing Lin updated SPARK-30511: --- Summary: Spark marks intentionally killed speculative tasks as pending leads to holding idle

[jira] [Commented] (SPARK-30530) CSV load followed by "is null" filter produces incorrect results

[ https://issues.apache.org/jira/browse/SPARK-30530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017290#comment-17017290 ] Maxim Gekk commented on SPARK-30530: [~jlowe] Thank you for the bug report. I will take a look at

[jira] [Created] (SPARK-30531) Duplicate query plan on Spark UI SQL page

Enrico Minack created SPARK-30531: - Summary: Duplicate query plan on Spark UI SQL page Key: SPARK-30531 URL: https://issues.apache.org/jira/browse/SPARK-30531 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-27868) Better document shuffle / RPC listen backlog

[ https://issues.apache.org/jira/browse/SPARK-27868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017281#comment-17017281 ] Marcelo Masiero Vanzin commented on SPARK-27868: You shoudn't have reverted the whole

[jira] [Commented] (SPARK-30530) CSV load followed by "is null" filter produces incorrect results

[ https://issues.apache.org/jira/browse/SPARK-30530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017088#comment-17017088 ] Jason Darrell Lowe commented on SPARK-30530: The regressed behavior was introduced by this

[jira] [Commented] (SPARK-27750) Standalone scheduler - ability to prioritize applications over drivers, many drivers act like Denial of Service

[ https://issues.apache.org/jira/browse/SPARK-27750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017085#comment-17017085 ] Wenchen Fan commented on SPARK-27750: - do you hit this issue in the real world? > Standalone

[jira] [Created] (SPARK-30530) CSV load followed by "is null" filter produces incorrect results

Jason Darrell Lowe created SPARK-30530: -- Summary: CSV load followed by "is null" filter produces incorrect results Key: SPARK-30530 URL: https://issues.apache.org/jira/browse/SPARK-30530

[jira] [Created] (SPARK-30529) Improve error messages when Executor dies before registering with driver

Thomas Graves created SPARK-30529: - Summary: Improve error messages when Executor dies before registering with driver Key: SPARK-30529 URL: https://issues.apache.org/jira/browse/SPARK-30529 Project:

[jira] [Resolved] (SPARK-30524) Disable OptimizeSkewJoin rule if introducing additional shuffle.

[ https://issues.apache.org/jira/browse/SPARK-30524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-30524. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27226

[jira] [Assigned] (SPARK-30524) Disable OptimizeSkewJoin rule if introducing additional shuffle.

[ https://issues.apache.org/jira/browse/SPARK-30524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-30524: --- Assignee: Ke Jia > Disable OptimizeSkewJoin rule if introducing additional shuffle. >

[jira] [Commented] (SPARK-27750) Standalone scheduler - ability to prioritize applications over drivers, many drivers act like Denial of Service

[ https://issues.apache.org/jira/browse/SPARK-27750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016974#comment-17016974 ] Sean R. Owen commented on SPARK-27750: -- I don't think Spark or a resource manager can save you from

[jira] [Assigned] (SPARK-30507) TableCalalog reserved properties shoudn't be changed with options or tblpropeties

[ https://issues.apache.org/jira/browse/SPARK-30507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-30507: --- Assignee: Kent Yao > TableCalalog reserved properties shoudn't be changed with options or

[jira] [Resolved] (SPARK-30507) TableCalalog reserved properties shoudn't be changed with options or tblpropeties

[ https://issues.apache.org/jira/browse/SPARK-30507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-30507. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27197

[jira] [Resolved] (SPARK-30337) Convert case class with var to normal class in spark-sql-kafka module

[ https://issues.apache.org/jira/browse/SPARK-30337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-30337. -- Resolution: Invalid We just removed TODO. > Convert case class with var to normal class in

[jira] [Commented] (SPARK-30483) Job History does not show pool properties table

[ https://issues.apache.org/jira/browse/SPARK-30483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016873#comment-17016873 ] pavithra ramachandran commented on SPARK-30483: --- Issue is resolved in master and 2.4  

[jira] [Commented] (SPARK-30337) Convert case class with var to normal class in spark-sql-kafka module

[ https://issues.apache.org/jira/browse/SPARK-30337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016866#comment-17016866 ] Gabor Somogyi commented on SPARK-30337: --- [~hyukjin.kwon] [~kabhwan] shall we resolve this jira

[jira] [Updated] (SPARK-30528) DPP issues

[ https://issues.apache.org/jira/browse/SPARK-30528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayur Bhosale updated SPARK-30528: -- Description: In DPP, heuristics to decide if DPP is going to benefit relies on the sizes of

[jira] [Updated] (SPARK-30528) DPP issues

[ https://issues.apache.org/jira/browse/SPARK-30528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayur Bhosale updated SPARK-30528: -- Attachment: plan.png cases.png > DPP issues > -- > >

[jira] [Created] (SPARK-30528) DPP issues

Mayur Bhosale created SPARK-30528: - Summary: DPP issues Key: SPARK-30528 URL: https://issues.apache.org/jira/browse/SPARK-30528 Project: Spark Issue Type: Bug Components: Optimizer

[jira] [Commented] (SPARK-12312) JDBC connection to Kerberos secured databases fails on remote executors

[ https://issues.apache.org/jira/browse/SPARK-12312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016826#comment-17016826 ] Gabor Somogyi commented on SPARK-12312: --- The PR is discontinued for long time so I've picked this

[jira] [Resolved] (SPARK-30395) When one or more DISTINCT aggregate expressions operate on the same field, the DISTINCT aggregate expression allows the use of the FILTER clause

[ https://issues.apache.org/jira/browse/SPARK-30395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng resolved SPARK-30395. Resolution: Duplicate Duplicate with https://issues.apache.org/jira/browse/SPARK-30276 > When

[jira] [Resolved] (SPARK-30396) When there are multiple DISTINCT aggregate expressions acting on different fields, any DISTINCT aggregate expression allows the use of the FILTER clause

[ https://issues.apache.org/jira/browse/SPARK-30396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng resolved SPARK-30396. Resolution: Duplicate Duplicate with https://issues.apache.org/jira/browse/SPARK-30276 > When

[jira] [Created] (SPARK-30527) Add IsNotNull filter when use In, InSet and InSubQuery

ulysses you created SPARK-30527: --- Summary: Add IsNotNull filter when use In, InSet and InSubQuery Key: SPARK-30527 URL: https://issues.apache.org/jira/browse/SPARK-30527 Project: Spark Issue

[jira] [Reopened] (SPARK-23626) DAGScheduler blocked due to JobSubmitted event

[ https://issues.apache.org/jira/browse/SPARK-23626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajith S reopened SPARK-23626: - Old PR was closed due to inactivity. Reopening with a new PR and hence to conclude > DAGScheduler

[jira] [Created] (SPARK-30526) Can I translate Spark documents into Chinese ？

WangQiang Yang created SPARK-30526: -- Summary: Can I translate Spark documents into Chinese ？ Key: SPARK-30526 URL: https://issues.apache.org/jira/browse/SPARK-30526 Project: Spark Issue

[jira] [Updated] (SPARK-30525) HiveTableScanExec do not need to prune partitions again after pushing down to hive metastore

[ https://issues.apache.org/jira/browse/SPARK-30525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hu Fuwang updated SPARK-30525: -- Description: In HiveTableScanExec, it will push down to hive metastore for partition pruning if 

[jira] [Updated] (SPARK-30525) HiveTableScanExec do not need to prune partitions again after pushing down to hive metastore

[ https://issues.apache.org/jira/browse/SPARK-30525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hu Fuwang updated SPARK-30525: -- Description: In HiveTableScanExec, it will push down to hive metastore for partition pruning if 

[jira] [Created] (SPARK-30525) HiveTableScanExec do not need to prune partitions again after pushing down to hive metastore

Hu Fuwang created SPARK-30525: - Summary: HiveTableScanExec do not need to prune partitions again after pushing down to hive metastore Key: SPARK-30525 URL: https://issues.apache.org/jira/browse/SPARK-30525