[jira] [Commented] (SPARK-31783) Performance test on dense and sparse datasets

[ https://issues.apache.org/jira/browse/SPARK-31783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112818#comment-17112818 ] zhengruifeng commented on SPARK-31783: -- The test code is in 'blockify_total', and current result is

[jira] [Updated] (SPARK-31783) Performance test on dense and sparse datasets

[ https://issues.apache.org/jira/browse/SPARK-31783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng updated SPARK-31783: - Attachment: blockify_perf_20200521.xlsx > Performance test on dense and sparse datasets >

[jira] [Updated] (SPARK-31783) Performance test on dense and sparse datasets

[ https://issues.apache.org/jira/browse/SPARK-31783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng updated SPARK-31783: - Attachment: blockify_total > Performance test on dense and sparse datasets >

[jira] [Assigned] (SPARK-31783) Performance test on dense and sparse datasets

[ https://issues.apache.org/jira/browse/SPARK-31783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng reassigned SPARK-31783: Assignee: zhengruifeng > Performance test on dense and sparse datasets >

[jira] [Created] (SPARK-31783) Performance test on dense and sparse datasets

zhengruifeng created SPARK-31783: Summary: Performance test on dense and sparse datasets Key: SPARK-31783 URL: https://issues.apache.org/jira/browse/SPARK-31783 Project: Spark Issue Type:

[jira] [Created] (SPARK-31782) Performance test on dense and sparse datasets

zhengruifeng created SPARK-31782: Summary: Performance test on dense and sparse datasets Key: SPARK-31782 URL: https://issues.apache.org/jira/browse/SPARK-31782 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-31782) Performance test on dense and sparse datasets

[ https://issues.apache.org/jira/browse/SPARK-31782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng resolved SPARK-31782. -- Resolution: Duplicate > Performance test on dense and sparse datasets >

[jira] [Commented] (SPARK-31754) Spark Structured Streaming: NullPointerException in Stream Stream join

[ https://issues.apache.org/jira/browse/SPARK-31754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112816#comment-17112816 ] Jungtaek Lim commented on SPARK-31754: -- I can also take a look if the input and checkpoint are

[jira] [Updated] (SPARK-31761) Sql Div operator can result in incorrect output for int_min

[ https://issues.apache.org/jira/browse/SPARK-31761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-31761: - Target Version/s: (was: 3.1.0) > Sql Div operator can result in incorrect output for int_min

[jira] [Updated] (SPARK-31762) Fix perf regression of date/timestamp formatting in toHiveString

[ https://issues.apache.org/jira/browse/SPARK-31762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-31762: Parent: SPARK-31408 Issue Type: Sub-task (was: Bug) > Fix perf regression of

[jira] [Assigned] (SPARK-31762) Fix perf regression of date/timestamp formatting in toHiveString

[ https://issues.apache.org/jira/browse/SPARK-31762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-31762: --- Assignee: Maxim Gekk > Fix perf regression of date/timestamp formatting in toHiveString >

[jira] [Resolved] (SPARK-31762) Fix perf regression of date/timestamp formatting in toHiveString

[ https://issues.apache.org/jira/browse/SPARK-31762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-31762. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 28582

[jira] [Updated] (SPARK-31760) Simplification Based on Containment

[ https://issues.apache.org/jira/browse/SPARK-31760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-31760: Summary: Simplification Based on Containment (was: Consolidating Single Table Predicates) >

[jira] [Commented] (SPARK-31781) Move param k (number of clusters) to shared params

[ https://issues.apache.org/jira/browse/SPARK-31781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112716#comment-17112716 ] Apache Spark commented on SPARK-31781: -- User 'huaxingao' has created a pull request for this issue:

[jira] [Assigned] (SPARK-31781) Move param k (number of clusters) to shared params

[ https://issues.apache.org/jira/browse/SPARK-31781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31781: Assignee: Apache Spark > Move param k (number of clusters) to shared params >

[jira] [Commented] (SPARK-31781) Move param k (number of clusters) to shared params

[ https://issues.apache.org/jira/browse/SPARK-31781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112715#comment-17112715 ] Apache Spark commented on SPARK-31781: -- User 'huaxingao' has created a pull request for this issue:

[jira] [Assigned] (SPARK-31781) Move param k (number of clusters) to shared params

[ https://issues.apache.org/jira/browse/SPARK-31781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31781: Assignee: (was: Apache Spark) > Move param k (number of clusters) to shared params >

[jira] [Created] (SPARK-31781) Move param k (number of clusters) to shared params

Huaxin Gao created SPARK-31781: -- Summary: Move param k (number of clusters) to shared params Key: SPARK-31781 URL: https://issues.apache.org/jira/browse/SPARK-31781 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-31780) Add R test tag to exclude R K8s image building and test

[ https://issues.apache.org/jira/browse/SPARK-31780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-31780: - Assignee: Dongjoon Hyun > Add R test tag to exclude R K8s image building and test >

[jira] [Resolved] (SPARK-31780) Add R test tag to exclude R K8s image building and test

[ https://issues.apache.org/jira/browse/SPARK-31780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-31780. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 28594

[jira] [Updated] (SPARK-31780) Add R test tag to exclude R K8s image building and test

[ https://issues.apache.org/jira/browse/SPARK-31780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31780: -- Priority: Minor (was: Major) > Add R test tag to exclude R K8s image building and test >

[jira] [Updated] (SPARK-31780) Add R test tag to exclude R K8s image building and test

[ https://issues.apache.org/jira/browse/SPARK-31780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31780: -- Affects Version/s: (was: 3.1.0) 3.0.0 > Add R test tag to exclude

[jira] [Assigned] (SPARK-31780) Add R test tag to exclude R K8s image building and test

[ https://issues.apache.org/jira/browse/SPARK-31780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31780: Assignee: Apache Spark > Add R test tag to exclude R K8s image building and test >

[jira] [Commented] (SPARK-31780) Add R test tag to exclude R K8s image building and test

[ https://issues.apache.org/jira/browse/SPARK-31780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112662#comment-17112662 ] Apache Spark commented on SPARK-31780: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Assigned] (SPARK-31780) Add R test tag to exclude R K8s image building and test

[ https://issues.apache.org/jira/browse/SPARK-31780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31780: Assignee: (was: Apache Spark) > Add R test tag to exclude R K8s image building and

[jira] [Created] (SPARK-31780) Add R test tag to exclude R K8s image building and test

Dongjoon Hyun created SPARK-31780: - Summary: Add R test tag to exclude R K8s image building and test Key: SPARK-31780 URL: https://issues.apache.org/jira/browse/SPARK-31780 Project: Spark

[jira] [Created] (SPARK-31779) Redefining struct inside array incorrectly wraps child fields in array

Jeff Evans created SPARK-31779: -- Summary: Redefining struct inside array incorrectly wraps child fields in array Key: SPARK-31779 URL: https://issues.apache.org/jira/browse/SPARK-31779 Project: Spark

[jira] [Created] (SPARK-31778) Support cross-building docker images

Holden Karau created SPARK-31778: Summary: Support cross-building docker images Key: SPARK-31778 URL: https://issues.apache.org/jira/browse/SPARK-31778 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-31777) CrossValidator supports user-supplied folds

Xiangrui Meng created SPARK-31777: - Summary: CrossValidator supports user-supplied folds Key: SPARK-31777 URL: https://issues.apache.org/jira/browse/SPARK-31777 Project: Spark Issue Type:

[jira] [Updated] (SPARK-31776) Literal lit() supports lists and numpy arrays

[ https://issues.apache.org/jira/browse/SPARK-31776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-31776: -- Issue Type: Improvement (was: New Feature) > Literal lit() supports lists and numpy arrays >

[jira] [Created] (SPARK-31776) Literal lit() supports lists and numpy arrays

Xiangrui Meng created SPARK-31776: - Summary: Literal lit() supports lists and numpy arrays Key: SPARK-31776 URL: https://issues.apache.org/jira/browse/SPARK-31776 Project: Spark Issue Type:

[jira] [Created] (SPARK-31775) Support tensor type (TensorType) in Spark SQL/DataFrame

Xiangrui Meng created SPARK-31775: - Summary: Support tensor type (TensorType) in Spark SQL/DataFrame Key: SPARK-31775 URL: https://issues.apache.org/jira/browse/SPARK-31775 Project: Spark

[jira] [Assigned] (SPARK-31387) HiveThriftServer2Listener update methods fail with unknown operation/session id

[ https://issues.apache.org/jira/browse/SPARK-31387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-31387: - Assignee: Ali Smesseim > HiveThriftServer2Listener update methods fail with unknown

[jira] [Resolved] (SPARK-31387) HiveThriftServer2Listener update methods fail with unknown operation/session id

[ https://issues.apache.org/jira/browse/SPARK-31387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-31387. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 28544

[jira] [Created] (SPARK-31773) getting the Caused by: org.apache.spark.sql.catalyst.errors.package$TreeNodeException: Binding attribute, at org.apache.spark.sql.catalyst.errors.package$.attachTree(pac

Pankaj Tiwari created SPARK-31773: - Summary: getting the Caused by: org.apache.spark.sql.catalyst.errors.package$TreeNodeException: Binding attribute, at org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:56)

[jira] [Created] (SPARK-31774) getting the Caused by: org.apache.spark.sql.catalyst.errors.package$TreeNodeException: Binding attribute, at org.apache.spark.sql.catalyst.errors.package$.attachTree(pac

Pankaj Tiwari created SPARK-31774: - Summary: getting the Caused by: org.apache.spark.sql.catalyst.errors.package$TreeNodeException: Binding attribute, at org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:56)

[jira] [Created] (SPARK-31772) Json schema reading is not consistent between int and string types

yaniv oren created SPARK-31772: -- Summary: Json schema reading is not consistent between int and string types Key: SPARK-31772 URL: https://issues.apache.org/jira/browse/SPARK-31772 Project: Spark

[jira] [Comment Edited] (SPARK-31761) Sql Div operator can result in incorrect output for int_min

[ https://issues.apache.org/jira/browse/SPARK-31761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112368#comment-17112368 ] Sandeep Katta edited comment on SPARK-31761 at 5/20/20, 3:46 PM: -

[jira] [Comment Edited] (SPARK-31761) Sql Div operator can result in incorrect output for int_min

[ https://issues.apache.org/jira/browse/SPARK-31761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112368#comment-17112368 ] Sandeep Katta edited comment on SPARK-31761 at 5/20/20, 3:44 PM: -

[jira] [Commented] (SPARK-31761) Sql Div operator can result in incorrect output for int_min

[ https://issues.apache.org/jira/browse/SPARK-31761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112368#comment-17112368 ] Sandeep Katta commented on SPARK-31761: --- [~sowen] [~hyukjin.kwon]    I have executed the same

[jira] [Comment Edited] (SPARK-31754) Spark Structured Streaming: NullPointerException in Stream Stream join

[ https://issues.apache.org/jira/browse/SPARK-31754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112359#comment-17112359 ] Puviarasu edited comment on SPARK-31754 at 5/20/20, 3:36 PM: - Hello

[jira] [Commented] (SPARK-31754) Spark Structured Streaming: NullPointerException in Stream Stream join

[ https://issues.apache.org/jira/browse/SPARK-31754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112359#comment-17112359 ] Puviarasu commented on SPARK-31754: --- Hello [~kabhwan] ,  Please find below the comments in *bold*. #

[jira] [Updated] (SPARK-31754) Spark Structured Streaming: NullPointerException in Stream Stream join

[ https://issues.apache.org/jira/browse/SPARK-31754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Puviarasu updated SPARK-31754: -- Attachment: Logical-Plan.txt > Spark Structured Streaming: NullPointerException in Stream Stream join

[jira] [Updated] (SPARK-31766) Add Spark version prefix to K8s UUID test image tag

[ https://issues.apache.org/jira/browse/SPARK-31766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31766: -- Fix Version/s: (was: 3.1.0) 3.0.0 > Add Spark version prefix to K8s

[jira] [Updated] (SPARK-31766) Add Spark version prefix to K8s UUID test image tag

[ https://issues.apache.org/jira/browse/SPARK-31766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31766: -- Affects Version/s: (was: 3.1.0) 3.0.0 > Add Spark version prefix

[jira] [Commented] (SPARK-12312) JDBC connection to Kerberos secured databases fails on remote executors

[ https://issues.apache.org/jira/browse/SPARK-12312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112339#comment-17112339 ] Agrim Bansal commented on SPARK-12312: -- are we not solving this for hive database ? > JDBC

[jira] [Commented] (SPARK-31437) Try assigning tasks to existing executors by which required resources in ResourceProfile are satisfied

[ https://issues.apache.org/jira/browse/SPARK-31437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112245#comment-17112245 ] Thomas Graves commented on SPARK-31437: --- Originally when I thought about this briefly I was

[jira] [Resolved] (SPARK-31759) Support configurable max number of rotate logs for spark daemons

[ https://issues.apache.org/jira/browse/SPARK-31759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-31759. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 28580

[jira] [Assigned] (SPARK-31759) Support configurable max number of rotate logs for spark daemons

[ https://issues.apache.org/jira/browse/SPARK-31759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-31759: Assignee: Kent Yao > Support configurable max number of rotate logs for spark daemons >

[jira] [Commented] (SPARK-31710) result is the not the same when query and execute jobs

[ https://issues.apache.org/jira/browse/SPARK-31710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112021#comment-17112021 ] Apache Spark commented on SPARK-31710: -- User 'GuoPhilipse' has created a pull request for this

[jira] [Assigned] (SPARK-31771) Disable Narrow TextStyle for datetime pattern 'G/M/L/E/u/Q/q'

[ https://issues.apache.org/jira/browse/SPARK-31771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31771: Assignee: Apache Spark > Disable Narrow TextStyle for datetime pattern 'G/M/L/E/u/Q/q' >

[jira] [Commented] (SPARK-31771) Disable Narrow TextStyle for datetime pattern 'G/M/L/E/u/Q/q'

[ https://issues.apache.org/jira/browse/SPARK-31771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111985#comment-17111985 ] Apache Spark commented on SPARK-31771: -- User 'yaooqinn' has created a pull request for this issue:

[jira] [Assigned] (SPARK-31771) Disable Narrow TextStyle for datetime pattern 'G/M/L/E/u/Q/q'

[ https://issues.apache.org/jira/browse/SPARK-31771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-31771: Assignee: (was: Apache Spark) > Disable Narrow TextStyle for datetime pattern

[jira] [Comment Edited] (SPARK-31437) Try assigning tasks to existing executors by which required resources in ResourceProfile are satisfied

[ https://issues.apache.org/jira/browse/SPARK-31437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17084552#comment-17084552 ] Hongze Zhang edited comment on SPARK-31437 at 5/20/20, 9:14 AM: (edited)

[jira] [Created] (SPARK-31771) Disable Narrow TextStyle for datetime pattern 'G/M/L/E/u/Q/q'

Kent Yao created SPARK-31771: Summary: Disable Narrow TextStyle for datetime pattern 'G/M/L/E/u/Q/q' Key: SPARK-31771 URL: https://issues.apache.org/jira/browse/SPARK-31771 Project: Spark Issue

[jira] [Created] (SPARK-31770) spark double count issue when table is recreated from hive

Nithin created SPARK-31770: -- Summary: spark double count issue when table is recreated from hive Key: SPARK-31770 URL: https://issues.apache.org/jira/browse/SPARK-31770 Project: Spark Issue Type:

[jira] [Commented] (SPARK-31769) Add support of MDC in spark driver logs.

[ https://issues.apache.org/jira/browse/SPARK-31769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111955#comment-17111955 ] Izek Greenfield commented on SPARK-31769: - Hi [~cloud_fan] Following previous discussions on

[jira] [Created] (SPARK-31769) Add support of MDC in spark driver logs.

Izek Greenfield created SPARK-31769: --- Summary: Add support of MDC in spark driver logs. Key: SPARK-31769 URL: https://issues.apache.org/jira/browse/SPARK-31769 Project: Spark Issue Type:

[jira] [Comment Edited] (SPARK-31437) Try assigning tasks to existing executors by which required resources in ResourceProfile are satisfied

[ https://issues.apache.org/jira/browse/SPARK-31437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17084552#comment-17084552 ] Hongze Zhang edited comment on SPARK-31437 at 5/20/20, 8:43 AM: Thanks

[jira] [Comment Edited] (SPARK-31437) Try assigning tasks to existing executors by which required resources in ResourceProfile are satisfied

[ https://issues.apache.org/jira/browse/SPARK-31437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17084552#comment-17084552 ] Hongze Zhang edited comment on SPARK-31437 at 5/20/20, 8:43 AM: Thanks

[jira] [Comment Edited] (SPARK-31437) Try assigning tasks to existing executors by which required resources in ResourceProfile are satisfied

[ https://issues.apache.org/jira/browse/SPARK-31437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17084552#comment-17084552 ] Hongze Zhang edited comment on SPARK-31437 at 5/20/20, 8:43 AM: Thanks

[jira] [Assigned] (SPARK-8981) Set applicationId and appName in log4j MDC

[ https://issues.apache.org/jira/browse/SPARK-8981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-8981: -- Assignee: Izek Greenfield > Set applicationId and appName in log4j MDC >

[jira] [Resolved] (SPARK-8981) Set applicationId and appName in log4j MDC

[ https://issues.apache.org/jira/browse/SPARK-8981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-8981. Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 26624

[jira] [Commented] (SPARK-21784) Add ALTER TABLE ADD CONSTRANT DDL to support defining primary key and foreign keys

[ https://issues.apache.org/jira/browse/SPARK-21784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111866#comment-17111866 ] Takeshi Yamamuro commented on SPARK-21784: -- Not yet. IIUC there is no active work now for

[jira] [Commented] (SPARK-30689) Allow custom resource scheduling to work with YARN versions that don't support custom resource scheduling

[ https://issues.apache.org/jira/browse/SPARK-30689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111814#comment-17111814 ] Apache Spark commented on SPARK-30689: -- User 'Ngone51' has created a pull request for this issue:

[jira] [Commented] (SPARK-30689) Allow custom resource scheduling to work with YARN versions that don't support custom resource scheduling

[ https://issues.apache.org/jira/browse/SPARK-30689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111813#comment-17111813 ] Apache Spark commented on SPARK-30689: -- User 'Ngone51' has created a pull request for this issue:

[jira] [Commented] (SPARK-31693) Investigate AmpLab Jenkins server network issue

[ https://issues.apache.org/jira/browse/SPARK-31693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111807#comment-17111807 ] Hyukjin Kwon commented on SPARK-31693: -- Awesome!! > Investigate AmpLab Jenkins server network