[jira] [Commented] (SPARK-28945) Allow concurrent writes to unrelated partitions with dynamic partition overwrite

[ https://issues.apache.org/jira/browse/SPARK-28945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16920602#comment-16920602 ] koert kuipers commented on SPARK-28945: --- See also:

[jira] [Created] (SPARK-28945) Allow concurrent writes to unrelated partitions with dynamic partition overwrite

koert kuipers created SPARK-28945: - Summary: Allow concurrent writes to unrelated partitions with dynamic partition overwrite Key: SPARK-28945 URL: https://issues.apache.org/jira/browse/SPARK-28945

[jira] [Comment Edited] (SPARK-27780) Shuffle server & client should be versioned to enable smoother upgrade

[ https://issues.apache.org/jira/browse/SPARK-27780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16875858#comment-16875858 ] koert kuipers edited comment on SPARK-27780 at 6/30/19 8:29 PM: fwiw i

[jira] [Commented] (SPARK-27780) Shuffle server & client should be versioned to enable smoother upgrade

[ https://issues.apache.org/jira/browse/SPARK-27780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16875858#comment-16875858 ] koert kuipers commented on SPARK-27780: --- fwiw i just ran into this since i am building and

[jira] [Commented] (SPARK-25783) Spark shell fails because of jline incompatibility

[ https://issues.apache.org/jira/browse/SPARK-25783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16844334#comment-16844334 ] koert kuipers commented on SPARK-25783: --- [~vanzin] i am using: {code:java} export

[jira] [Commented] (SPARK-27512) Decimal parsing leads to unexpected type inference

[ https://issues.apache.org/jira/browse/SPARK-27512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16823115#comment-16823115 ] koert kuipers commented on SPARK-27512: --- i agree it is better than having two different decimal

[jira] [Commented] (SPARK-27512) Decimal parsing leads to unexpected type inference

[ https://issues.apache.org/jira/browse/SPARK-27512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16822796#comment-16822796 ] koert kuipers commented on SPARK-27512: --- seems DecimalFormat.parse also simply ignores commas.

[jira] [Comment Edited] (SPARK-27512) Decimal parsing leads to unexpected type inference

[ https://issues.apache.org/jira/browse/SPARK-27512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16822790#comment-16822790 ] koert kuipers edited comment on SPARK-27512 at 4/21/19 11:03 PM: -

[jira] [Commented] (SPARK-27512) Decimal parsing leads to unexpected type inference

[ https://issues.apache.org/jira/browse/SPARK-27512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16822790#comment-16822790 ] koert kuipers commented on SPARK-27512: --- [~maxgekk] max do you know why getDecimalParser has that

[jira] [Commented] (SPARK-27513) Spark tarball with binaries should have files owned by uid 0

[ https://issues.apache.org/jira/browse/SPARK-27513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16822789#comment-16822789 ] koert kuipers commented on SPARK-27513: --- i think this can be closed as wont fix > Spark tarball

[jira] [Commented] (SPARK-27512) Decimal parsing leads to unexpected type inference

[ https://issues.apache.org/jira/browse/SPARK-27512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16822525#comment-16822525 ] koert kuipers commented on SPARK-27512: --- default locale is US, which now has this logic for

[jira] [Commented] (SPARK-27512) Decimal parsing leads to unexpected type inference

[ https://issues.apache.org/jira/browse/SPARK-27512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821954#comment-16821954 ] koert kuipers commented on SPARK-27512: --- {code:bash} $ hadoop fs -cat test.bsv x|y 1|1,2,3 2|4,5,6

[jira] [Updated] (SPARK-27513) Spark tarball with binaries should have files owned by uid 0

[ https://issues.apache.org/jira/browse/SPARK-27513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] koert kuipers updated SPARK-27513: -- Description: currently the tarball is created in dev/make-distribution.sh like this:

[jira] [Created] (SPARK-27513) Spark tarball with binaries should have files owned by uid 0

koert kuipers created SPARK-27513: - Summary: Spark tarball with binaries should have files owned by uid 0 Key: SPARK-27513 URL: https://issues.apache.org/jira/browse/SPARK-27513 Project: Spark

[jira] [Updated] (SPARK-27512) Decimal parsing leads to unexpected type inference

[ https://issues.apache.org/jira/browse/SPARK-27512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] koert kuipers updated SPARK-27512: -- Summary: Decimal parsing leads to unexpected type inference (was: Decimal parsing leading to

[jira] [Created] (SPARK-27512) Decimal parsing leading to unexpected type inference

koert kuipers created SPARK-27512: - Summary: Decimal parsing leading to unexpected type inference Key: SPARK-27512 URL: https://issues.apache.org/jira/browse/SPARK-27512 Project: Spark Issue

[jira] [Created] (SPARK-27477) Kafka token provider should have provided dependency on Spark

koert kuipers created SPARK-27477: - Summary: Kafka token provider should have provided dependency on Spark Key: SPARK-27477 URL: https://issues.apache.org/jira/browse/SPARK-27477 Project: Spark

[jira] [Created] (SPARK-26208) Empty dataframe does not roundtrip for csv with header

koert kuipers created SPARK-26208: - Summary: Empty dataframe does not roundtrip for csv with header Key: SPARK-26208 URL: https://issues.apache.org/jira/browse/SPARK-26208 Project: Spark

[jira] [Commented] (SPARK-25783) Spark shell fails because of jline incompatibility

[ https://issues.apache.org/jira/browse/SPARK-25783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16658442#comment-16658442 ] koert kuipers commented on SPARK-25783: --- [~viirya] what version of jline is loaded from scala

[jira] [Commented] (SPARK-25783) Spark shell fails because of jline incompatibility

[ https://issues.apache.org/jira/browse/SPARK-25783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16658334#comment-16658334 ] koert kuipers commented on SPARK-25783: --- [~viirya] are you saying that jline classes are loaded

[jira] [Commented] (SPARK-25783) Spark shell fails because of jline incompatibility

[ https://issues.apache.org/jira/browse/SPARK-25783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16657958#comment-16657958 ] koert kuipers commented on SPARK-25783: --- adding jline-2.14.6.jar to the spark jars folder solves

[jira] [Comment Edited] (SPARK-25783) Spark shell fails because of jline incompatibility

[ https://issues.apache.org/jira/browse/SPARK-25783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16657455#comment-16657455 ] koert kuipers edited comment on SPARK-25783 at 10/19/18 10:25 PM: -- i

[jira] [Comment Edited] (SPARK-25783) Spark shell fails because of jline incompatibility

[ https://issues.apache.org/jira/browse/SPARK-25783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16657455#comment-16657455 ] koert kuipers edited comment on SPARK-25783 at 10/19/18 9:25 PM: - i was

[jira] [Commented] (SPARK-25783) Spark shell fails because of jline incompatibility

[ https://issues.apache.org/jira/browse/SPARK-25783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16657455#comment-16657455 ] koert kuipers commented on SPARK-25783: --- i was just to same conclusion. i couldn't find jline

[jira] [Updated] (SPARK-25783) Spark shell fails because of jline incompatibility

[ https://issues.apache.org/jira/browse/SPARK-25783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] koert kuipers updated SPARK-25783: -- Environment: spark 2.4.0-rc3 on hadoop 2.6.0 (cdh 5.15.1) with -Phadoop-provided (was: spark

[jira] [Updated] (SPARK-25783) Spark shell fails because of jline incompatibility

[ https://issues.apache.org/jira/browse/SPARK-25783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] koert kuipers updated SPARK-25783: -- Environment: spark 2.4.0-rc3 on hadoop 2.6.0 (cdh 5.15.1) with (was: spark 2.4.0-rc3 which

[jira] [Updated] (SPARK-25783) Spark shell fails because of jline incompatibility

[ https://issues.apache.org/jira/browse/SPARK-25783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] koert kuipers updated SPARK-25783: -- Description: error i get when launching spark-shell is: {code:bash} Spark context Web UI

[jira] [Created] (SPARK-25783) Spark shell fails because of jline incompatibility

koert kuipers created SPARK-25783: - Summary: Spark shell fails because of jline incompatibility Key: SPARK-25783 URL: https://issues.apache.org/jira/browse/SPARK-25783 Project: Spark Issue

[jira] [Comment Edited] (SPARK-17916) CSV data source treats empty string as null no matter what nullValue option is

[ https://issues.apache.org/jira/browse/SPARK-17916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16584378#comment-16584378 ] koert kuipers edited comment on SPARK-17916 at 8/18/18 7:20 PM: my first

[jira] [Commented] (SPARK-17916) CSV data source treats empty string as null no matter what nullValue option is

[ https://issues.apache.org/jira/browse/SPARK-17916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16584393#comment-16584393 ] koert kuipers commented on SPARK-17916: --- now the particular unit test that broke for us, where

[jira] [Commented] (SPARK-17916) CSV data source treats empty string as null no matter what nullValue option is

[ https://issues.apache.org/jira/browse/SPARK-17916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16584381#comment-16584381 ] koert kuipers commented on SPARK-17916: --- we also use csv format to write files like for example

[jira] [Commented] (SPARK-17916) CSV data source treats empty string as null no matter what nullValue option is

[ https://issues.apache.org/jira/browse/SPARK-17916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16584378#comment-16584378 ] koert kuipers commented on SPARK-17916: --- my first observation is that if i do this: {code:scala}

[jira] [Commented] (SPARK-17916) CSV data source treats empty string as null no matter what nullValue option is

[ https://issues.apache.org/jira/browse/SPARK-17916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16584374#comment-16584374 ] koert kuipers commented on SPARK-17916: --- hi [~maxgekk] i saw your unit test for the old behavior.

[jira] [Comment Edited] (SPARK-17916) CSV data source treats empty string as null no matter what nullValue option is

[ https://issues.apache.org/jira/browse/SPARK-17916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16584324#comment-16584324 ] koert kuipers edited comment on SPARK-17916 at 8/17/18 8:05 PM: the

[jira] [Comment Edited] (SPARK-17916) CSV data source treats empty string as null no matter what nullValue option is

[ https://issues.apache.org/jira/browse/SPARK-17916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16584324#comment-16584324 ] koert kuipers edited comment on SPARK-17916 at 8/17/18 8:03 PM: the

[jira] [Comment Edited] (SPARK-17916) CSV data source treats empty string as null no matter what nullValue option is

[ https://issues.apache.org/jira/browse/SPARK-17916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16584324#comment-16584324 ] koert kuipers edited comment on SPARK-17916 at 8/17/18 7:54 PM: the

[jira] [Commented] (SPARK-17916) CSV data source treats empty string as null no matter what nullValue option is

[ https://issues.apache.org/jira/browse/SPARK-17916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16584324#comment-16584324 ] koert kuipers commented on SPARK-17916: --- the default behavior in 2.3.x for csv format is that when

[jira] [Created] (SPARK-25134) Csv column pruning with checking of headers throws incorrect error

koert kuipers created SPARK-25134: - Summary: Csv column pruning with checking of headers throws incorrect error Key: SPARK-25134 URL: https://issues.apache.org/jira/browse/SPARK-25134 Project: Spark

[jira] [Commented] (SPARK-15516) Schema merging in driver fails for parquet when merging LongType and IntegerType

[ https://issues.apache.org/jira/browse/SPARK-15516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16562632#comment-16562632 ] koert kuipers commented on SPARK-15516: --- we also ran into this on columns that are not key

[jira] [Created] (SPARK-24860) Expose dynamic partition overwrite per write operation

koert kuipers created SPARK-24860: - Summary: Expose dynamic partition overwrite per write operation Key: SPARK-24860 URL: https://issues.apache.org/jira/browse/SPARK-24860 Project: Spark

[jira] [Commented] (SPARK-20073) Unexpected Cartesian product when using eqNullSafe in join with a derived table

[ https://issues.apache.org/jira/browse/SPARK-20073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16018907#comment-16018907 ] koert kuipers commented on SPARK-20073: --- leftjoin of a dataframe x with results from an aggregation

[jira] [Created] (SPARK-20804) Join with null safe equality fails with AnalysisException

koert kuipers created SPARK-20804: - Summary: Join with null safe equality fails with AnalysisException Key: SPARK-20804 URL: https://issues.apache.org/jira/browse/SPARK-20804 Project: Spark

[jira] [Created] (SPARK-20359) Catalyst EliminateOuterJoin optimization can cause NPE

koert kuipers created SPARK-20359: - Summary: Catalyst EliminateOuterJoin optimization can cause NPE Key: SPARK-20359 URL: https://issues.apache.org/jira/browse/SPARK-20359 Project: Spark

[jira] [Commented] (SPARK-19536) Improve capability to merge SQL data types

[ https://issues.apache.org/jira/browse/SPARK-19536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860748#comment-15860748 ] koert kuipers commented on SPARK-19536: --- great, that only leaves maps and nested structs then i

[jira] [Created] (SPARK-19536) Improve capability to merge SQL data types

koert kuipers created SPARK-19536: - Summary: Improve capability to merge SQL data types Key: SPARK-19536 URL: https://issues.apache.org/jira/browse/SPARK-19536 Project: Spark Issue Type:

[jira] [Comment Edited] (SPARK-13801) DataFrame.col should return unresolved attribute

[ https://issues.apache.org/jira/browse/SPARK-13801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856289#comment-15856289 ] koert kuipers edited comment on SPARK-13801 at 2/7/17 4:30 PM: --- wenchen,

[jira] [Comment Edited] (SPARK-13801) DataFrame.col should return unresolved attribute

[ https://issues.apache.org/jira/browse/SPARK-13801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856289#comment-15856289 ] koert kuipers edited comment on SPARK-13801 at 2/7/17 4:29 PM: --- wenchen,

[jira] [Comment Edited] (SPARK-13801) DataFrame.col should return unresolved attribute

[ https://issues.apache.org/jira/browse/SPARK-13801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856289#comment-15856289 ] koert kuipers edited comment on SPARK-13801 at 2/7/17 4:30 PM: --- wenchen,

[jira] [Commented] (SPARK-13801) DataFrame.col should return unresolved attribute

[ https://issues.apache.org/jira/browse/SPARK-13801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856289#comment-15856289 ] koert kuipers commented on SPARK-13801: --- wenchen, you say: just get the name out and wrap it with

[jira] [Comment Edited] (SPARK-13801) DataFrame.col should return unresolved attribute

[ https://issues.apache.org/jira/browse/SPARK-13801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856289#comment-15856289 ] koert kuipers edited comment on SPARK-13801 at 2/7/17 4:29 PM: --- wenchen,

[jira] [Commented] (SPARK-11976) Support "." character in DataFrame column name

[ https://issues.apache.org/jira/browse/SPARK-11976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856267#comment-15856267 ] koert kuipers commented on SPARK-11976: --- do we really need this? this sounds as complex and maybe

[jira] [Commented] (SPARK-19493) Remove Java 7 support

[ https://issues.apache.org/jira/browse/SPARK-19493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856245#comment-15856245 ] koert kuipers commented on SPARK-19493: --- all for this now, after a year of deprecation > Remove

[jira] [Commented] (SPARK-19468) Dataset slow because of unnecessary shuffles

[ https://issues.apache.org/jira/browse/SPARK-19468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854242#comment-15854242 ] koert kuipers commented on SPARK-19468: --- so to summarize: RDD does what we would expect, DataFrame

[jira] [Commented] (SPARK-19468) Dataset slow because of unnecessary shuffles

[ https://issues.apache.org/jira/browse/SPARK-19468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854152#comment-15854152 ] koert kuipers commented on SPARK-19468: --- inserting unnecessary shuffles makes things very slow. and

[jira] [Created] (SPARK-19468) Dataset slow because of unnecessary shuffles

koert kuipers created SPARK-19468: - Summary: Dataset slow because of unnecessary shuffles Key: SPARK-19468 URL: https://issues.apache.org/jira/browse/SPARK-19468 Project: Spark Issue Type:

[jira] [Updated] (SPARK-19428) Ability to select first row of groupby

[ https://issues.apache.org/jira/browse/SPARK-19428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koert Kuipers updated SPARK-19428: -- if I remember correctly window function is like a canned secondary sort with limited

[jira] [Comment Edited] (SPARK-19428) Ability to select first row of groupby

[ https://issues.apache.org/jira/browse/SPARK-19428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852929#comment-15852929 ] koert kuipers edited comment on SPARK-19428 at 2/4/17 8:57 PM: ---

[jira] [Commented] (SPARK-19428) Ability to select first row of groupby

[ https://issues.apache.org/jira/browse/SPARK-19428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852929#comment-15852929 ] koert kuipers commented on SPARK-19428: --- generalizing to return top-x by some sorting is

[jira] [Issue Comment Deleted] (SPARK-19416) Dataset.schema is inconsistent with Dataset in handling columns with periods

[ https://issues.apache.org/jira/browse/SPARK-19416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] koert kuipers updated SPARK-19416: -- Comment: was deleted (was: would it be simpler to ban columns with a period in the name?) >

[jira] [Comment Edited] (SPARK-19428) Ability to select first row of groupby

[ https://issues.apache.org/jira/browse/SPARK-19428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852882#comment-15852882 ] koert kuipers edited comment on SPARK-19428 at 2/4/17 6:10 PM: --- getting a

[jira] [Commented] (SPARK-19428) Ability to select first row of groupby

[ https://issues.apache.org/jira/browse/SPARK-19428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852882#comment-15852882 ] koert kuipers commented on SPARK-19428: --- getting a first element for each group (which is somewhat

[jira] [Commented] (SPARK-19416) Dataset.schema is inconsistent with Dataset in handling columns with periods

[ https://issues.apache.org/jira/browse/SPARK-19416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852876#comment-15852876 ] koert kuipers commented on SPARK-19416: --- would it be simpler to ban columns with a period in the

[jira] [Commented] (SPARK-17977) DataFrameReader and DataStreamReader should have an ancestor class

[ https://issues.apache.org/jira/browse/SPARK-17977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851011#comment-15851011 ] koert kuipers commented on SPARK-17977: --- this would be great. and the same applies to

[jira] (SPARK-19323) Upgrade breeze to 0.13

Title: Message Title koert kuipers commented on SPARK-19323

[jira] [Created] (SPARK-19323) Upgrade breeze to 0.13

koert kuipers created SPARK-19323: - Summary: Upgrade breeze to 0.13 Key: SPARK-19323 URL: https://issues.apache.org/jira/browse/SPARK-19323 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-16494) Upgrade breeze version to 0.12

[ https://issues.apache.org/jira/browse/SPARK-16494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15785629#comment-15785629 ] koert kuipers commented on SPARK-16494: --- i just ran into an issue because of this when trying to

[jira] [Commented] (SPARK-17346) Kafka 0.10 support in Structured Streaming

[ https://issues.apache.org/jira/browse/SPARK-17346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15785560#comment-15785560 ] koert kuipers commented on SPARK-17346: --- this ticket mentions kafka 0.10-based sinks for structured

[jira] [Commented] (SPARK-18711) NPE in generated SpecificMutableProjection for Aggregator

[ https://issues.apache.org/jira/browse/SPARK-18711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15724115#comment-15724115 ] koert kuipers commented on SPARK-18711: --- confirmed it resolved the issue for me. thanks > NPE in

[jira] [Commented] (SPARK-18251) DataSet API | RuntimeException: Null value appeared in non-nullable field when holding Option Case Class

[ https://issues.apache.org/jira/browse/SPARK-18251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15721228#comment-15721228 ] koert kuipers commented on SPARK-18251: --- i think all of these arguments are very valid, however to

[jira] [Commented] (SPARK-18711) NPE in generated SpecificMutableProjection for Aggregator

[ https://issues.apache.org/jira/browse/SPARK-18711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15721175#comment-15721175 ] koert kuipers commented on SPARK-18711: --- simplified: {noformat} case class Holder(i: Int) val

[jira] [Updated] (SPARK-18711) NPE in generated SpecificMutableProjection for Aggregator

[ https://issues.apache.org/jira/browse/SPARK-18711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] koert kuipers updated SPARK-18711: -- Description: this is a bug in the branch-2.1, but i don't think it was in 2.1.0-rc1 code

[jira] [Updated] (SPARK-18711) NPE in generated SpecificMutableProjection for Aggregator

[ https://issues.apache.org/jira/browse/SPARK-18711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] koert kuipers updated SPARK-18711: -- Description: this is a bug in the branch-2.1, but i don't think it was in 2.1.0-rc1 code:

[jira] [Created] (SPARK-18711) NPE in generated SpecificMutableProjection for Aggregator

koert kuipers created SPARK-18711: - Summary: NPE in generated SpecificMutableProjection for Aggregator Key: SPARK-18711 URL: https://issues.apache.org/jira/browse/SPARK-18711 Project: Spark

[jira] [Commented] (SPARK-15810) Aggregator doesn't play nice with Option

[ https://issues.apache.org/jira/browse/SPARK-15810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15720610#comment-15720610 ] koert kuipers commented on SPARK-15810: --- here is an example where (None,) gets read back in as

[jira] [Comment Edited] (SPARK-15810) Aggregator doesn't play nice with Option

[ https://issues.apache.org/jira/browse/SPARK-15810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15720557#comment-15720557 ] koert kuipers edited comment on SPARK-15810 at 12/4/16 8:36 PM: when i

[jira] [Commented] (SPARK-15810) Aggregator doesn't play nice with Option

[ https://issues.apache.org/jira/browse/SPARK-15810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15720557#comment-15720557 ] koert kuipers commented on SPARK-15810: --- when i code Aggregator up generically as a wrapper for

[jira] [Commented] (SPARK-15810) Aggregator doesn't play nice with Option

[ https://issues.apache.org/jira/browse/SPARK-15810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15720404#comment-15720404 ] koert kuipers commented on SPARK-15810: --- similar test: {noformat} val agg1 = new Aggregator[Int,

[jira] [Commented] (SPARK-15810) Aggregator doesn't play nice with Option

[ https://issues.apache.org/jira/browse/SPARK-15810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15720397#comment-15720397 ] koert kuipers commented on SPARK-15810: --- to continue the previous example: wrapping all my

[jira] [Commented] (SPARK-15810) Aggregator doesn't play nice with Option

[ https://issues.apache.org/jira/browse/SPARK-15810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15720387#comment-15720387 ] koert kuipers commented on SPARK-15810: --- next, again with spark 2.1.x branch i wanted to test an

[jira] [Commented] (SPARK-15810) Aggregator doesn't play nice with Option

[ https://issues.apache.org/jira/browse/SPARK-15810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15720377#comment-15720377 ] koert kuipers commented on SPARK-15810: --- i am redoing tests with options and aggregators against

[jira] [Commented] (SPARK-15798) Secondary sort in Dataset/DataFrame

[ https://issues.apache.org/jira/browse/SPARK-15798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15717366#comment-15717366 ] koert kuipers commented on SPARK-15798: --- using the implementation in spark-sorted

[jira] [Commented] (SPARK-17583) Remove unused rowSeparator variable and set auto-expanding buffer as default for maxCharsPerColumn option in CSV

[ https://issues.apache.org/jira/browse/SPARK-17583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710871#comment-15710871 ] koert kuipers commented on SPARK-17583: --- i see. so you are saying in spark 2.0.x it fails when the

[jira] [Commented] (SPARK-17583) Remove unused rowSeparator variable and set auto-expanding buffer as default for maxCharsPerColumn option in CSV

[ https://issues.apache.org/jira/browse/SPARK-17583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710172#comment-15710172 ] koert kuipers commented on SPARK-17583: --- i just tested out inhouse unit test (which run against

[jira] [Comment Edited] (SPARK-18651) KeyValueGroupedDataset[K, V].reduceGroups cannot handle primitive for V

[ https://issues.apache.org/jira/browse/SPARK-18651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15709349#comment-15709349 ] koert kuipers edited comment on SPARK-18651 at 11/30/16 6:39 PM: - i

[jira] [Comment Edited] (SPARK-18651) KeyValueGroupedDataset[K, V].reduceGroups cannot handle primitive for V

[ https://issues.apache.org/jira/browse/SPARK-18651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15709352#comment-15709352 ] koert kuipers edited comment on SPARK-18651 at 11/30/16 6:38 PM: - Fixed

[jira] [Reopened] (SPARK-18651) KeyValueGroupedDataset[K, V].reduceGroups cannot handle primitive for V

[ https://issues.apache.org/jira/browse/SPARK-18651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] koert kuipers reopened SPARK-18651: --- > KeyValueGroupedDataset[K, V].reduceGroups cannot handle primitive for V >

[jira] [Resolved] (SPARK-18651) KeyValueGroupedDataset[K, V].reduceGroups cannot handle primitive for V

[ https://issues.apache.org/jira/browse/SPARK-18651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] koert kuipers resolved SPARK-18651. --- Resolution: Not A Bug Fixed in master, still issue in branch-2.0 >

[jira] [Commented] (SPARK-18651) KeyValueGroupedDataset[K, V].reduceGroups cannot handle primitive for V

[ https://issues.apache.org/jira/browse/SPARK-18651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15709349#comment-15709349 ] koert kuipers commented on SPARK-18651: --- i cannot reproduce the error in master. it seems to work

[jira] [Created] (SPARK-18651) KeyValueGroupedDataset[K, V].reduceGroups cannot handle primitive for V

koert kuipers created SPARK-18651: - Summary: KeyValueGroupedDataset[K, V].reduceGroups cannot handle primitive for V Key: SPARK-18651 URL: https://issues.apache.org/jira/browse/SPARK-18651 Project:

[jira] [Commented] (SPARK-18122) Fallback to Kryo for unknown classes in ExpressionEncoder

[ https://issues.apache.org/jira/browse/SPARK-18122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15699681#comment-15699681 ] koert kuipers commented on SPARK-18122: --- [~sowen] why won'f fix? what changed [~marmbrus] thought

[jira] [Commented] (SPARK-15798) Secondary sort in Dataset/DataFrame

[ https://issues.apache.org/jira/browse/SPARK-15798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15661918#comment-15661918 ] koert kuipers commented on SPARK-15798: --- turns out the operations needed for this are already

[jira] [Commented] (SPARK-15798) Secondary sort in Dataset/DataFrame

[ https://issues.apache.org/jira/browse/SPARK-15798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634352#comment-15634352 ] koert kuipers commented on SPARK-15798: --- looking at the code for Window operators it seems to me

[jira] [Commented] (SPARK-17668) Support representing structs with case classes and tuples in spark sql udf inputs

[ https://issues.apache.org/jira/browse/SPARK-17668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15626613#comment-15626613 ] koert kuipers commented on SPARK-17668: --- similar issues with tuples: {noformat} val df = Seq((1,

[jira] [Commented] (SPARK-18147) Broken Spark SQL Codegen

[ https://issues.apache.org/jira/browse/SPARK-18147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15613472#comment-15613472 ] koert kuipers commented on SPARK-18147: --- it also breaks with an option of a case class. like this:

[jira] [Updated] (SPARK-18147) Broken Spark SQL Codegen

[ https://issues.apache.org/jira/browse/SPARK-18147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] koert kuipers updated SPARK-18147: -- Description: this is me on purpose trying to break spark sql codegen to uncover potential

[jira] [Created] (SPARK-18147) Broken Spark SQL Codegen

koert kuipers created SPARK-18147: - Summary: Broken Spark SQL Codegen Key: SPARK-18147 URL: https://issues.apache.org/jira/browse/SPARK-18147 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-15769) Add Encoder for input type to Aggregator

[ https://issues.apache.org/jira/browse/SPARK-15769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15586654#comment-15586654 ] Koert Kuipers commented on SPARK-15769: --- what i really want is to be able to use Aggregator in

[jira] [Commented] (SPARK-15769) Add Encoder for input type to Aggregator

[ https://issues.apache.org/jira/browse/SPARK-15769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15585931#comment-15585931 ] koert kuipers commented on SPARK-15769: --- worth pointing out: * adding an encoder to the input type

[jira] [Updated] (SPARK-17668) Support representing structs with case classes and tuples in spark sql udf inputs

[ https://issues.apache.org/jira/browse/SPARK-17668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] koert kuipers updated SPARK-17668: -- Summary: Support representing structs with case classes and tuples in spark sql udf inputs

[jira] [Commented] (SPARK-17668) Support case classes and tuples to represent structs in spark sql udfs

[ https://issues.apache.org/jira/browse/SPARK-17668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15523989#comment-15523989 ] koert kuipers commented on SPARK-17668: --- original conversation is here:

<    1   2   3   4   >