[jira] [Commented] (SPARK-45282) Join loses records for cached datasets

2023-11-08 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17784115#comment-17784115 ] koert kuipers commented on SPARK-45282: --- it does look like same issue and partitioning being the

[jira] [Commented] (SPARK-45282) Join loses records for cached datasets

2023-10-30 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781171#comment-17781171 ] koert kuipers commented on SPARK-45282: --- last time i checked this issue was still present in

[jira] [Comment Edited] (SPARK-45282) Join loses records for cached datasets

2023-09-27 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17769893#comment-17769893 ] koert kuipers edited comment on SPARK-45282 at 9/28/23 4:07 AM: yes i

[jira] [Comment Edited] (SPARK-45282) Join loses records for cached datasets

2023-09-27 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17769893#comment-17769893 ] koert kuipers edited comment on SPARK-45282 at 9/28/23 4:06 AM: yes i

[jira] [Commented] (SPARK-45282) Join loses records for cached datasets

2023-09-27 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17769893#comment-17769893 ] koert kuipers commented on SPARK-45282: --- yes i can reproduce it. master branch on commit:  

[jira] [Comment Edited] (SPARK-45282) Join loses records for cached datasets

2023-09-22 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17768123#comment-17768123 ] koert kuipers edited comment on SPARK-45282 at 9/22/23 7:04 PM: after

[jira] [Updated] (SPARK-45282) Join loses records for cached datasets

2023-09-22 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] koert kuipers updated SPARK-45282: -- Description: we observed this issue on spark 3.4.1 but it is also present on 3.5.0. it is

[jira] [Updated] (SPARK-45282) Join loses records for cached datasets

2023-09-22 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] koert kuipers updated SPARK-45282: -- Description: we observed this issue on spark 3.4.1 but it is also present on 3.5.0. it is

[jira] [Commented] (SPARK-45282) Join loses records for cached datasets

2023-09-22 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17768123#comment-17768123 ] koert kuipers commented on SPARK-45282: --- after reverting SPARK-41048 the issue went away. so i

[jira] [Updated] (SPARK-45282) Join loses records for cached datasets

2023-09-22 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] koert kuipers updated SPARK-45282: -- Description: we observed this issue on spark 3.4.1 but it is also present on 3.5.0. it is

[jira] [Updated] (SPARK-45282) Join loses records for cached datasets

2023-09-22 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] koert kuipers updated SPARK-45282: -- Description: we observed this issue on spark 3.4.1 but it is also present on 3.5.0. it is

[jira] [Updated] (SPARK-45282) Join loses records for cached datasets

2023-09-22 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] koert kuipers updated SPARK-45282: -- Description: we observed this issue on spark 3.4.1 but it is also present on 3.5.0. it is

[jira] [Created] (SPARK-45282) Join loses records for cached datasets

2023-09-22 Thread koert kuipers (Jira)
koert kuipers created SPARK-45282: - Summary: Join loses records for cached datasets Key: SPARK-45282 URL: https://issues.apache.org/jira/browse/SPARK-45282 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-44323) Scala None shows up as null for Aggregator BUF or OUT

2023-07-09 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17741400#comment-17741400 ] koert kuipers commented on SPARK-44323: --- not sure why pullreq isnt getting linked automatically

[jira] [Comment Edited] (SPARK-37829) An outer-join using joinWith on DataFrames returns Rows with null fields instead of null values

2023-07-08 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17740726#comment-17740726 ] koert kuipers edited comment on SPARK-37829 at 7/8/23 4:19 PM: --- since this

[jira] [Commented] (SPARK-44323) Scala None shows up as null for Aggregator BUF or OUT

2023-07-06 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17740744#comment-17740744 ] koert kuipers commented on SPARK-44323: --- i think the issue is that Nones inside Tuples now become

[jira] [Comment Edited] (SPARK-44323) Scala None shows up as null for Aggregator BUF or OUT

2023-07-06 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17740744#comment-17740744 ] koert kuipers edited comment on SPARK-44323 at 7/6/23 6:28 PM: --- i think

[jira] [Comment Edited] (SPARK-37829) An outer-join using joinWith on DataFrames returns Rows with null fields instead of null values

2023-07-06 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17740726#comment-17740726 ] koert kuipers edited comment on SPARK-37829 at 7/6/23 5:32 PM: --- since this

[jira] [Comment Edited] (SPARK-37829) An outer-join using joinWith on DataFrames returns Rows with null fields instead of null values

2023-07-06 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17740726#comment-17740726 ] koert kuipers edited comment on SPARK-37829 at 7/6/23 5:23 PM: --- since this

[jira] [Commented] (SPARK-37829) An outer-join using joinWith on DataFrames returns Rows with null fields instead of null values

2023-07-06 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17740726#comment-17740726 ] koert kuipers commented on SPARK-37829: --- since this behavior of returning a Row with null values

[jira] [Updated] (SPARK-44323) Scala None shows up as null for Aggregator BUF or OUT

2023-07-06 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] koert kuipers updated SPARK-44323: -- Description: when doing an upgrade from spark 3.3.1 to spark 3.4.1 we suddenly started

[jira] [Created] (SPARK-44323) Scala None shows up as null for Aggregator BUF or OUT

2023-07-06 Thread koert kuipers (Jira)
koert kuipers created SPARK-44323: - Summary: Scala None shows up as null for Aggregator BUF or OUT Key: SPARK-44323 URL: https://issues.apache.org/jira/browse/SPARK-44323 Project: Spark

[jira] [Created] (SPARK-39770) Support Avro schema evolution

2022-07-13 Thread koert kuipers (Jira)
koert kuipers created SPARK-39770: - Summary: Support Avro schema evolution Key: SPARK-39770 URL: https://issues.apache.org/jira/browse/SPARK-39770 Project: Spark Issue Type: Improvement

[jira] [Comment Edited] (SPARK-31726) Make spark.files available in driver with cluster deploy mode on kubernetes

2022-01-23 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17480756#comment-17480756 ] koert kuipers edited comment on SPARK-31726 at 1/23/22, 11:22 PM: --

[jira] [Commented] (SPARK-31726) Make spark.files available in driver with cluster deploy mode on kubernetes

2022-01-23 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17480756#comment-17480756 ] koert kuipers commented on SPARK-31726: --- [~beregon87] about --jars, are you seeing that the jars

[jira] [Commented] (SPARK-31726) Make spark.files available in driver with cluster deploy mode on kubernetes

2022-01-16 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17476940#comment-17476940 ] koert kuipers commented on SPARK-31726: ---

[jira] [Commented] (SPARK-37476) udaf doesnt work with nullable (or option of) case class result

2021-12-01 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17452118#comment-17452118 ] koert kuipers commented on SPARK-37476: --- note that changing my case class to: {code:java} case

[jira] [Comment Edited] (SPARK-37476) udaf doesnt work with nullable (or option of) case class result

2021-12-01 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17452101#comment-17452101 ] koert kuipers edited comment on SPARK-37476 at 12/2/21, 12:58 AM: -- i

[jira] [Commented] (SPARK-37476) udaf doesnt work with nullable (or option of) case class result

2021-12-01 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17452101#comment-17452101 ] koert kuipers commented on SPARK-37476: --- i get that scala Double cannot be null. however i dont

[jira] [Commented] (SPARK-37476) udaf doesnt work with nullable (or option of) case class result

2021-12-01 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17451921#comment-17451921 ] koert kuipers commented on SPARK-37476: --- yes it works well     {code:java} import

[jira] [Comment Edited] (SPARK-37476) udaf doesnt work with nullable (or option of) case class result

2021-12-01 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17451921#comment-17451921 ] koert kuipers edited comment on SPARK-37476 at 12/1/21, 4:57 PM: - yes it

[jira] [Updated] (SPARK-37476) udaf doesnt work with nullable (or option of) case class result

2021-11-27 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] koert kuipers updated SPARK-37476: -- Description: i have a need to have a dataframe aggregation return a nullable case class.

[jira] [Updated] (SPARK-37476) udaf doesnt work with nullable (or option of) case class result

2021-11-27 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] koert kuipers updated SPARK-37476: -- Description: i have a need to have a dataframe aggregation return a nullable case class.

[jira] [Updated] (SPARK-37476) udaf doesnt work with nullable (or option of) case class result

2021-11-27 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] koert kuipers updated SPARK-37476: -- Description: i have a need to have a dataframe aggregation return a nullable case class.

[jira] [Updated] (SPARK-37476) udaf doesnt work with nullable (or option of) case class result

2021-11-27 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] koert kuipers updated SPARK-37476: -- Description: i have a need to have a dataframe aggregation return a nullable case class.

[jira] [Updated] (SPARK-37476) udaf doesnt work with nullable (or option of) case class result

2021-11-27 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] koert kuipers updated SPARK-37476: -- Description: i have a need to have a dataframe aggregation return a nullable case class.

[jira] [Updated] (SPARK-37476) udaf doesnt work with nullable (or option of) case class result

2021-11-27 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] koert kuipers updated SPARK-37476: -- Summary: udaf doesnt work with nullable (or option of) case class result (was: udaf doesnt

[jira] [Updated] (SPARK-37476) udaf doesnt work with nullable (or option of) case class result

2021-11-27 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] koert kuipers updated SPARK-37476: -- Description: i have a need to have a dataframe aggregation return a nullable case class.

[jira] [Updated] (SPARK-37476) udaf doesnt work with nullable case class result

2021-11-27 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] koert kuipers updated SPARK-37476: -- Description: i have a need to have a dataframe aggregation return a nullable case class.

[jira] [Updated] (SPARK-37476) udaf doesnt work with nullable case class result

2021-11-27 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] koert kuipers updated SPARK-37476: -- Description: i have a need to have a dataframe aggregation return a nullable case class.

[jira] [Created] (SPARK-37476) udaf doesnt work with nullable case class result

2021-11-27 Thread koert kuipers (Jira)
koert kuipers created SPARK-37476: - Summary: udaf doesnt work with nullable case class result Key: SPARK-37476 URL: https://issues.apache.org/jira/browse/SPARK-37476 Project: Spark Issue

[jira] [Commented] (SPARK-36681) Fail to load Snappy codec

2021-10-18 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17430303#comment-17430303 ] koert kuipers commented on SPARK-36681: --- hadoop Jira issue:

[jira] [Created] (SPARK-36210) Preserve column insertion order in Dataset.withColumns

2021-07-19 Thread koert kuipers (Jira)
koert kuipers created SPARK-36210: - Summary: Preserve column insertion order in Dataset.withColumns Key: SPARK-36210 URL: https://issues.apache.org/jira/browse/SPARK-36210 Project: Spark

[jira] [Commented] (SPARK-35079) Transform with udf gives incorrect result

2021-05-18 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17346970#comment-17346970 ] koert kuipers commented on SPARK-35079: --- looks to me like this is a duplicate of SPARK-34829 >

[jira] [Updated] (SPARK-35079) Transform with udf gives incorrect result

2021-04-14 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] koert kuipers updated SPARK-35079: -- Description: i think this is a correctness bug in spark 3.1.1 the behavior is correct in

[jira] [Created] (SPARK-35079) Transform with udf gives incorrect result

2021-04-14 Thread koert kuipers (Jira)
koert kuipers created SPARK-35079: - Summary: Transform with udf gives incorrect result Key: SPARK-35079 URL: https://issues.apache.org/jira/browse/SPARK-35079 Project: Spark Issue Type: Bug

[jira] [Comment Edited] (SPARK-13860) TPCDS query 39 returns wrong results compared to TPC official result set

2021-04-13 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-13860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17320450#comment-17320450 ] koert kuipers edited comment on SPARK-13860 at 4/13/21, 6:38 PM: - this

[jira] [Comment Edited] (SPARK-13860) TPCDS query 39 returns wrong results compared to TPC official result set

2021-04-13 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-13860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17320450#comment-17320450 ] koert kuipers edited comment on SPARK-13860 at 4/13/21, 6:33 PM: - this

[jira] [Commented] (SPARK-13860) TPCDS query 39 returns wrong results compared to TPC official result set

2021-04-13 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-13860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17320450#comment-17320450 ] koert kuipers commented on SPARK-13860: --- this is what i now see: {code:java} scala> Seq(1.0,

[jira] [Commented] (SPARK-32109) SQL hash function handling of nulls makes collision too likely

2020-06-28 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17147358#comment-17147358 ] koert kuipers commented on SPARK-32109: --- the issue is that Row here isnt really a sequence. it

[jira] [Comment Edited] (SPARK-32109) SQL hash function handling of nulls makes collision too likely

2020-06-28 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17147358#comment-17147358 ] koert kuipers edited comment on SPARK-32109 at 6/28/20, 2:58 PM: - the

[jira] [Updated] (SPARK-32109) SQL hash function handling of nulls makes collision too likely

2020-06-26 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] koert kuipers updated SPARK-32109: -- Description: this ticket is about org.apache.spark.sql.functions.hash and sparks handling of

[jira] [Created] (SPARK-32109) SQL hash function handling of nulls makes collision too likely

2020-06-26 Thread koert kuipers (Jira)
koert kuipers created SPARK-32109: - Summary: SQL hash function handling of nulls makes collision too likely Key: SPARK-32109 URL: https://issues.apache.org/jira/browse/SPARK-32109 Project: Spark

[jira] [Updated] (SPARK-32056) Repartition by key should support partition coalesce for AQE

2020-06-22 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] koert kuipers updated SPARK-32056: -- Priority: Minor (was: Major) > Repartition by key should support partition coalesce for AQE

[jira] [Updated] (SPARK-32056) Repartition by key should support partition coalesce for AQE

2020-06-22 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] koert kuipers updated SPARK-32056: -- Description: when adaptive query execution is enabled the following expression should

[jira] [Created] (SPARK-32056) Repartition by key should support partition coalesce for AQE

2020-06-22 Thread koert kuipers (Jira)
koert kuipers created SPARK-32056: - Summary: Repartition by key should support partition coalesce for AQE Key: SPARK-32056 URL: https://issues.apache.org/jira/browse/SPARK-32056 Project: Spark

[jira] [Commented] (SPARK-31841) Dataset.repartition leverage adaptive execution

2020-05-27 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118120#comment-17118120 ] koert kuipers commented on SPARK-31841: --- that is right. its a feature request i believe (unless i

[jira] [Created] (SPARK-31841) Dataset.repartition leverage adaptive execution

2020-05-27 Thread koert kuipers (Jira)
koert kuipers created SPARK-31841: - Summary: Dataset.repartition leverage adaptive execution Key: SPARK-31841 URL: https://issues.apache.org/jira/browse/SPARK-31841 Project: Spark Issue

[jira] [Created] (SPARK-31726) Make spark.files available in driver with cluster deploy mode on kubernetes

2020-05-15 Thread koert kuipers (Jira)
koert kuipers created SPARK-31726: - Summary: Make spark.files available in driver with cluster deploy mode on kubernetes Key: SPARK-31726 URL: https://issues.apache.org/jira/browse/SPARK-31726

[jira] [Updated] (SPARK-29302) dynamic partition overwrite with speculation enabled

2020-03-28 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] koert kuipers updated SPARK-29302: -- Affects Version/s: (was: 3.0.0) > dynamic partition overwrite with speculation enabled >

[jira] [Updated] (SPARK-29302) dynamic partition overwrite with speculation enabled

2020-03-28 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] koert kuipers updated SPARK-29302: -- Affects Version/s: 3.0.0 > dynamic partition overwrite with speculation enabled >

[jira] [Commented] (SPARK-30320) Insert overwrite to DataSource table with dynamic partition error when running multiple task attempts

2020-03-26 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17068005#comment-17068005 ] koert kuipers commented on SPARK-30320: --- i believe we are seeing this issue. it shows up in

[jira] [Comment Edited] (SPARK-30320) Insert overwrite to DataSource table with dynamic partition error when running multiple task attempts

2020-03-26 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17068005#comment-17068005 ] koert kuipers edited comment on SPARK-30320 at 3/26/20, 8:11 PM: - i

[jira] [Comment Edited] (SPARK-29302) dynamic partition overwrite with speculation enabled

2020-03-26 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067972#comment-17067972 ] koert kuipers edited comment on SPARK-29302 at 3/26/20, 7:23 PM: - i

[jira] [Commented] (SPARK-29302) dynamic partition overwrite with speculation enabled

2020-03-26 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067972#comment-17067972 ] koert kuipers commented on SPARK-29302: --- i believe we are seeing this issue. it shows up in

[jira] [Comment Edited] (SPARK-29302) dynamic partition overwrite with speculation enabled

2020-03-26 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067972#comment-17067972 ] koert kuipers edited comment on SPARK-29302 at 3/26/20, 7:16 PM: - i

[jira] [Comment Edited] (SPARK-31183) Incompatible Avro dates/timestamps with Spark 2.4

2020-03-22 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063725#comment-17063725 ] koert kuipers edited comment on SPARK-31183 at 3/22/20, 2:03 PM: - hi!

[jira] [Commented] (SPARK-31183) Incompatible Avro dates/timestamps with Spark 2.4

2020-03-20 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063725#comment-17063725 ] koert kuipers commented on SPARK-31183: --- hi! all spark tests pass for me except hive and avro.

[jira] [Updated] (SPARK-29906) Reading of csv file fails with adaptive execution turned on

2019-11-16 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] koert kuipers updated SPARK-29906: -- Priority: Minor (was: Major) > Reading of csv file fails with adaptive execution turned on >

[jira] [Updated] (SPARK-29906) Reading of csv file fails with adaptive execution turned on

2019-11-15 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] koert kuipers updated SPARK-29906: -- Labels: correctness (was: ) > Reading of csv file fails with adaptive execution turned on >

[jira] [Commented] (SPARK-29906) Reading of csv file fails with adaptive execution turned on

2019-11-15 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975286#comment-16975286 ] koert kuipers commented on SPARK-29906: --- note that with the default option for csv being

[jira] [Commented] (SPARK-29906) Reading of csv file fails with adaptive execution turned on

2019-11-14 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16974836#comment-16974836 ] koert kuipers commented on SPARK-29906: --- i added a bit of debug logging: {code:java} $ git diff

[jira] [Updated] (SPARK-29906) Reading of csv file fails with adaptive execution turned on

2019-11-14 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] koert kuipers updated SPARK-29906: -- Description: we observed an issue where spark seems to confuse a data line (not the first

[jira] [Updated] (SPARK-29906) Reading of csv file fails with adaptive execution turned on

2019-11-14 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] koert kuipers updated SPARK-29906: -- Description: we observed an issue where spark seems to confuse a data line (not the first

[jira] [Created] (SPARK-29906) Reading of csv file fails with adaptive execution turned on

2019-11-14 Thread koert kuipers (Jira)
koert kuipers created SPARK-29906: - Summary: Reading of csv file fails with adaptive execution turned on Key: SPARK-29906 URL: https://issues.apache.org/jira/browse/SPARK-29906 Project: Spark

[jira] [Commented] (SPARK-28945) Allow concurrent writes to different partitions with dynamic partition overwrite

2019-11-13 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973715#comment-16973715 ] koert kuipers commented on SPARK-28945: --- i understand there is a great deal of complexity in the

[jira] [Comment Edited] (SPARK-29435) Spark 3 doesnt work with older shuffle service

2019-10-12 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16950122#comment-16950122 ] koert kuipers edited comment on SPARK-29435 at 10/12/19 11:50 PM: -- i

[jira] [Commented] (SPARK-29435) Spark 3 doesnt work with older shuffle service

2019-10-12 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16950122#comment-16950122 ] koert kuipers commented on SPARK-29435: --- i checked and it works with dynamic execution using spark

[jira] [Comment Edited] (SPARK-29435) Spark 3 doesnt work with older shuffle service

2019-10-10 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949095#comment-16949095 ] koert kuipers edited comment on SPARK-29435 at 10/11/19 3:20 AM: -

[jira] [Commented] (SPARK-29435) Spark 3 doesnt work with older shuffle service

2019-10-10 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949095#comment-16949095 ] koert kuipers commented on SPARK-29435: --- actually, it doesnt matter if i use spark 2 or spark 3

[jira] [Commented] (SPARK-29435) Spark 3 doesnt work with older shuffle service

2019-10-10 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948989#comment-16948989 ] koert kuipers commented on SPARK-29435: --- [~vanzin] sorry i should have been more clear, i did set

[jira] [Created] (SPARK-29435) Spark 3 doesnt work with older shuffle service

2019-10-10 Thread koert kuipers (Jira)
koert kuipers created SPARK-29435: - Summary: Spark 3 doesnt work with older shuffle service Key: SPARK-29435 URL: https://issues.apache.org/jira/browse/SPARK-29435 Project: Spark Issue Type:

[jira] [Commented] (SPARK-27665) Split fetch shuffle blocks protocol from OpenBlocks

2019-10-10 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948979#comment-16948979 ] koert kuipers commented on SPARK-27665: --- i tried using spark.shuffle.useOldFetchProtocol=true

[jira] [Commented] (SPARK-27665) Split fetch shuffle blocks protocol from OpenBlocks

2019-09-18 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932833#comment-16932833 ] koert kuipers commented on SPARK-27665: --- oh wait i didnt realize there is a setting

[jira] [Commented] (SPARK-27665) Split fetch shuffle blocks protocol from OpenBlocks

2019-09-18 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932823#comment-16932823 ] koert kuipers commented on SPARK-27665: --- i am a little nervous that this got merged into master

[jira] [Commented] (SPARK-29027) KafkaDelegationTokenSuite fails

2019-09-11 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927912#comment-16927912 ] koert kuipers commented on SPARK-29027: --- [~gsomogyi] if you email me at koert at tresata dot com i

[jira] [Commented] (SPARK-29027) KafkaDelegationTokenSuite fails

2019-09-11 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927889#comment-16927889 ] koert kuipers commented on SPARK-29027: --- just for this one test debug logs is 62mb of kerberos and

[jira] [Commented] (SPARK-29027) KafkaDelegationTokenSuite fails

2019-09-11 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927785#comment-16927785 ] koert kuipers commented on SPARK-29027: --- [~gsomogyi] i can email you debug log file directly if

[jira] [Commented] (SPARK-29027) KafkaDelegationTokenSuite fails

2019-09-11 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927783#comment-16927783 ] koert kuipers commented on SPARK-29027: --- i get same error in sbt i think, plus i find sbt a lot

[jira] [Commented] (SPARK-29027) KafkaDelegationTokenSuite fails

2019-09-11 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927715#comment-16927715 ] koert kuipers commented on SPARK-29027: --- i renamed /etc/krb5.conf and it did not change anything.

[jira] [Commented] (SPARK-29027) KafkaDelegationTokenSuite fails

2019-09-11 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927646#comment-16927646 ] koert kuipers commented on SPARK-29027: --- let me try to get debug logs > KafkaDelegationTokenSuite

[jira] [Commented] (SPARK-29027) KafkaDelegationTokenSuite fails

2019-09-10 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926809#comment-16926809 ] koert kuipers commented on SPARK-29027: --- [~gsomogyi] do you use any services that require open

[jira] [Comment Edited] (SPARK-29027) KafkaDelegationTokenSuite fails

2019-09-10 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926707#comment-16926707 ] koert kuipers edited comment on SPARK-29027 at 9/10/19 2:53 PM: i tried

[jira] [Commented] (SPARK-29027) KafkaDelegationTokenSuite fails

2019-09-10 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926707#comment-16926707 ] koert kuipers commented on SPARK-29027: --- i tried doing tests in a virtual machine and they pass so

[jira] [Commented] (SPARK-29027) KafkaDelegationTokenSuite fails

2019-09-10 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926620#comment-16926620 ] koert kuipers commented on SPARK-29027: --- i am going to try running tests on a virtual machine to

[jira] [Comment Edited] (SPARK-29027) KafkaDelegationTokenSuite fails

2019-09-10 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926577#comment-16926577 ] koert kuipers edited comment on SPARK-29027 at 9/10/19 1:02 PM: i am

[jira] [Commented] (SPARK-29027) KafkaDelegationTokenSuite fails

2019-09-10 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926577#comment-16926577 ] koert kuipers commented on SPARK-29027: --- i am running test on my work laptop. it has kerberos

[jira] [Commented] (SPARK-29027) KafkaDelegationTokenSuite fails

2019-09-10 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926563#comment-16926563 ] koert kuipers commented on SPARK-29027: --- hey the command i run is: mvn clean test -fae i am not

[jira] [Created] (SPARK-29027) KafkaDelegationTokenSuite fails

2019-09-09 Thread koert kuipers (Jira)
koert kuipers created SPARK-29027: - Summary: KafkaDelegationTokenSuite fails Key: SPARK-29027 URL: https://issues.apache.org/jira/browse/SPARK-29027 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-28945) Allow concurrent writes to different partitions with dynamic partition overwrite

2019-09-01 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] koert kuipers updated SPARK-28945: -- Summary: Allow concurrent writes to different partitions with dynamic partition overwrite

  1   2   3   4   >