[jira] [Assigned] (SPARK-22476) Add new function dayofweek in R

2017-11-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22476: Assignee: (was: Apache Spark) > Add new function dayofweek in R >

[jira] [Commented] (SPARK-22476) Add new function dayofweek in R

2017-11-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16245305#comment-16245305 ] Apache Spark commented on SPARK-22476: -- User 'HyukjinKwon' has created a pull reques

[jira] [Assigned] (SPARK-22476) Add new function dayofweek in R

2017-11-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22476: Assignee: Apache Spark > Add new function dayofweek in R > ---

[jira] [Created] (SPARK-22476) Add new function dayofweek in R

2017-11-08 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-22476: Summary: Add new function dayofweek in R Key: SPARK-22476 URL: https://issues.apache.org/jira/browse/SPARK-22476 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-22308) Support unit tests of spark code using ScalaTest using suites other than FunSuite

2017-11-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16245283#comment-16245283 ] Apache Spark commented on SPARK-22308: -- User 'nkronenfeld' has created a pull reques

[jira] [Assigned] (SPARK-22456) Add new function dayofweek

2017-11-08 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-22456: Assignee: Michael Styles > Add new function dayofweek > -- > >

[jira] [Commented] (SPARK-21657) Spark has exponential time complexity to explode(array of structs)

2017-11-08 Thread Ruslan Dautkhanov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16245240#comment-16245240 ] Ruslan Dautkhanov commented on SPARK-21657: --- Thank you [~uzadude] - great inves

[jira] [Resolved] (SPARK-22456) Add new function dayofweek

2017-11-08 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-22456. -- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19672 [https://git

[jira] [Assigned] (SPARK-22466) SPARK_CONF_DIR is not is set by Spark's launch scripts with default value

2017-11-08 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-22466: Assignee: Kent Yao > SPARK_CONF_DIR is not is set by Spark's launch scripts with default v

[jira] [Resolved] (SPARK-22466) SPARK_CONF_DIR is not is set by Spark's launch scripts with default value

2017-11-08 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-22466. -- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19688 [https://git

[jira] [Commented] (SPARK-22417) createDataFrame from a pandas.DataFrame reads datetime64 values as longs

2017-11-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16245204#comment-16245204 ] Apache Spark commented on SPARK-22417: -- User 'ueshin' has created a pull request for

[jira] [Commented] (SPARK-22403) StructuredKafkaWordCount example fails in YARN cluster mode

2017-11-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16245191#comment-16245191 ] Apache Spark commented on SPARK-22403: -- User 'wypoon' has created a pull request for

[jira] [Assigned] (SPARK-22403) StructuredKafkaWordCount example fails in YARN cluster mode

2017-11-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22403: Assignee: (was: Apache Spark) > StructuredKafkaWordCount example fails in YARN cluster

[jira] [Assigned] (SPARK-22403) StructuredKafkaWordCount example fails in YARN cluster mode

2017-11-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22403: Assignee: Apache Spark > StructuredKafkaWordCount example fails in YARN cluster mode > ---

[jira] [Resolved] (SPARK-22460) Spark De-serialization of Timestamp field is Incorrect

2017-11-08 Thread Saniya Tech (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saniya Tech resolved SPARK-22460. - Resolution: Not A Problem > Spark De-serialization of Timestamp field is Incorrect >

[jira] [Commented] (SPARK-22460) Spark De-serialization of Timestamp field is Incorrect

2017-11-08 Thread Saniya Tech (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16245163#comment-16245163 ] Saniya Tech commented on SPARK-22460: - Based on the feedback I am going to close this

[jira] [Comment Edited] (SPARK-22460) Spark De-serialization of Timestamp field is Incorrect

2017-11-08 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16245158#comment-16245158 ] Liang-Chi Hsieh edited comment on SPARK-22460 at 11/9/17 3:33 AM: -

[jira] [Comment Edited] (SPARK-22460) Spark De-serialization of Timestamp field is Incorrect

2017-11-08 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16245158#comment-16245158 ] Liang-Chi Hsieh edited comment on SPARK-22460 at 11/9/17 3:32 AM: -

[jira] [Commented] (SPARK-22460) Spark De-serialization of Timestamp field is Incorrect

2017-11-08 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16245158#comment-16245158 ] Liang-Chi Hsieh commented on SPARK-22460: - >From the output of {{print(s"${rawOut

[jira] [Assigned] (SPARK-10365) Support Parquet logical type TIMESTAMP_MICROS

2017-11-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10365: Assignee: Apache Spark > Support Parquet logical type TIMESTAMP_MICROS > -

[jira] [Assigned] (SPARK-10365) Support Parquet logical type TIMESTAMP_MICROS

2017-11-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10365: Assignee: (was: Apache Spark) > Support Parquet logical type TIMESTAMP_MICROS > --

[jira] [Commented] (SPARK-10365) Support Parquet logical type TIMESTAMP_MICROS

2017-11-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16245082#comment-16245082 ] Apache Spark commented on SPARK-10365: -- User 'cloud-fan' has created a pull request

[jira] [Commented] (SPARK-22211) LimitPushDown optimization for FullOuterJoin generates wrong results

2017-11-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16245063#comment-16245063 ] Apache Spark commented on SPARK-22211: -- User 'henryr' has created a pull request for

[jira] [Commented] (SPARK-22211) LimitPushDown optimization for FullOuterJoin generates wrong results

2017-11-08 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16245029#comment-16245029 ] Dongjoon Hyun commented on SPARK-22211: --- Hi, All. This breaks `branch-2.2`. - SBT:

[jira] [Created] (SPARK-22475) show histogram in DESC COLUMN command

2017-11-08 Thread Zhenhua Wang (JIRA)
Zhenhua Wang created SPARK-22475: Summary: show histogram in DESC COLUMN command Key: SPARK-22475 URL: https://issues.apache.org/jira/browse/SPARK-22475 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-17593) list files on s3 very slow

2017-11-08 Thread Nick Dimiduk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16244887#comment-16244887 ] Nick Dimiduk commented on SPARK-17593: -- So the fix in Hadoop 2.8 is for any variant

[jira] [Commented] (SPARK-22471) SQLListener consumes much memory causing OutOfMemoryError

2017-11-08 Thread Arseniy Tashoyan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16244836#comment-16244836 ] Arseniy Tashoyan commented on SPARK-22471: -- I am coming up with a simple fix, ba

[jira] [Assigned] (SPARK-22471) SQLListener consumes much memory causing OutOfMemoryError

2017-11-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22471: Assignee: (was: Apache Spark) > SQLListener consumes much memory causing OutOfMemoryEr

[jira] [Commented] (SPARK-22471) SQLListener consumes much memory causing OutOfMemoryError

2017-11-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16244821#comment-16244821 ] Apache Spark commented on SPARK-22471: -- User 'tashoyan' has created a pull request f

[jira] [Assigned] (SPARK-22471) SQLListener consumes much memory causing OutOfMemoryError

2017-11-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22471: Assignee: Apache Spark > SQLListener consumes much memory causing OutOfMemoryError > -

[jira] [Assigned] (SPARK-20648) Make Jobs and Stages pages use the new app state store

2017-11-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20648: Assignee: Apache Spark > Make Jobs and Stages pages use the new app state store >

[jira] [Assigned] (SPARK-20648) Make Jobs and Stages pages use the new app state store

2017-11-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20648: Assignee: (was: Apache Spark) > Make Jobs and Stages pages use the new app state store

[jira] [Commented] (SPARK-20648) Make Jobs and Stages pages use the new app state store

2017-11-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16244557#comment-16244557 ] Apache Spark commented on SPARK-20648: -- User 'vanzin' has created a pull request for

[jira] [Commented] (SPARK-22472) Datasets generate random values for null primitive types

2017-11-08 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16244466#comment-16244466 ] Sean Owen commented on SPARK-22472: --- This doesn't look right ... https://github.com/ap

[jira] [Commented] (SPARK-22222) Fix the ARRAY_MAX in BufferHolder and add a test

2017-11-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16244455#comment-16244455 ] Apache Spark commented on SPARK-2: -- User 'dongjoon-hyun' has created a pull requ

[jira] [Commented] (SPARK-22471) SQLListener consumes much memory causing OutOfMemoryError

2017-11-08 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16244452#comment-16244452 ] Marcelo Vanzin commented on SPARK-22471: My patch for SPARK-20652 greatly reduces

[jira] [Commented] (SPARK-22472) Datasets generate random values for null primitive types

2017-11-08 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16244438#comment-16244438 ] Kazuaki Ishizaki commented on SPARK-22472: -- >From [the initial >version|https:/

[jira] [Assigned] (SPARK-22133) Document Mesos reject offer duration configutations

2017-11-08 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-22133: - Assignee: windkithk > Document Mesos reject offer duration configutations >

[jira] [Updated] (SPARK-22133) Document Mesos reject offer duration configutations

2017-11-08 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-22133: -- Priority: Minor (was: Major) > Document Mesos reject offer duration configutations > -

[jira] [Resolved] (SPARK-22133) Document Mesos reject offer duration configutations

2017-11-08 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-22133. --- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19555 [https://github.co

[jira] [Comment Edited] (SPARK-22472) Datasets generate random values for null primitive types

2017-11-08 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16244327#comment-16244327 ] Kazuaki Ishizaki edited comment on SPARK-22472 at 11/8/17 5:30 PM:

[jira] [Comment Edited] (SPARK-22472) Datasets generate random values for null primitive types

2017-11-08 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16244327#comment-16244327 ] Kazuaki Ishizaki edited comment on SPARK-22472 at 11/8/17 5:22 PM:

[jira] [Updated] (SPARK-22474) cannot read a parquet file containing a Seq[Map[MyCaseClass, String]]

2017-11-08 Thread Mikael Valot (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikael Valot updated SPARK-22474: - Affects Version/s: 2.1.2 > cannot read a parquet file containing a Seq[Map[MyCaseClass, String]]

[jira] [Created] (SPARK-22474) cannot read a parquet file containing a Seq[Map[MyCaseClass, String]]

2017-11-08 Thread Mikael Valot (JIRA)
Mikael Valot created SPARK-22474: Summary: cannot read a parquet file containing a Seq[Map[MyCaseClass, String]] Key: SPARK-22474 URL: https://issues.apache.org/jira/browse/SPARK-22474 Project: Spark

[jira] [Comment Edited] (SPARK-22472) Datasets generate random values for null primitive types

2017-11-08 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16244327#comment-16244327 ] Kazuaki Ishizaki edited comment on SPARK-22472 at 11/8/17 5:11 PM:

[jira] [Commented] (SPARK-22472) Datasets generate random values for null primitive types

2017-11-08 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16244327#comment-16244327 ] Kazuaki Ishizaki commented on SPARK-22472: -- Thank you for reporting this behavio

[jira] [Comment Edited] (SPARK-22472) Datasets generate random values for null primitive types

2017-11-08 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16244327#comment-16244327 ] Kazuaki Ishizaki edited comment on SPARK-22472 at 11/8/17 5:10 PM:

[jira] [Commented] (SPARK-22472) Datasets generate random values for null primitive types

2017-11-08 Thread Vladislav Kuzemchik (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16244319#comment-16244319 ] Vladislav Kuzemchik commented on SPARK-22472: - I'm using Option[Long] as a wo

[jira] [Comment Edited] (SPARK-22472) Datasets generate random values for null primitive types

2017-11-08 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16244310#comment-16244310 ] Marco Gaido edited comment on SPARK-22472 at 11/8/17 4:58 PM: -

[jira] [Commented] (SPARK-22472) Datasets generate random values for null primitive types

2017-11-08 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16244310#comment-16244310 ] Marco Gaido commented on SPARK-22472: - Two things: 1 - if you use `as[Option[Long]]`

[jira] [Commented] (SPARK-22472) Datasets generate random values for null primitive types

2017-11-08 Thread Vladislav Kuzemchik (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16244273#comment-16244273 ] Vladislav Kuzemchik commented on SPARK-22472: - My assumption is that it is be

[jira] [Commented] (SPARK-22472) Datasets generate random values for null primitive types

2017-11-08 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16244263#comment-16244263 ] Sean Owen commented on SPARK-22472: --- Not sure it's random, but, somehow the value is be

[jira] [Assigned] (SPARK-22473) Replace deprecated AsyncAssertions.Waiter and methods of java.sql.Date

2017-11-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22473: Assignee: Apache Spark > Replace deprecated AsyncAssertions.Waiter and methods of java.sql

[jira] [Assigned] (SPARK-22473) Replace deprecated AsyncAssertions.Waiter and methods of java.sql.Date

2017-11-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22473: Assignee: (was: Apache Spark) > Replace deprecated AsyncAssertions.Waiter and methods

[jira] [Commented] (SPARK-22473) Replace deprecated AsyncAssertions.Waiter and methods of java.sql.Date

2017-11-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16244260#comment-16244260 ] Apache Spark commented on SPARK-22473: -- User 'mgaido91' has created a pull request f

[jira] [Created] (SPARK-22473) Replace deprecated AsyncAssertions.Waiter and methods of java.sql.Date

2017-11-08 Thread Marco Gaido (JIRA)
Marco Gaido created SPARK-22473: --- Summary: Replace deprecated AsyncAssertions.Waiter and methods of java.sql.Date Key: SPARK-22473 URL: https://issues.apache.org/jira/browse/SPARK-22473 Project: Spark

[jira] [Updated] (SPARK-22472) Datasets generate random values for null primitive types

2017-11-08 Thread Vladislav Kuzemchik (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladislav Kuzemchik updated SPARK-22472: Affects Version/s: 2.2.0 > Datasets generate random values for null primitive types

[jira] [Updated] (SPARK-22471) SQLListener consumes much memory causing OutOfMemoryError

2017-11-08 Thread Arseniy Tashoyan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arseniy Tashoyan updated SPARK-22471: - Attachment: SQLListener_retained_size.png > SQLListener consumes much memory causing OutO

[jira] [Created] (SPARK-22472) Datasets generate random values for null primitive types

2017-11-08 Thread Vladislav Kuzemchik (JIRA)
Vladislav Kuzemchik created SPARK-22472: --- Summary: Datasets generate random values for null primitive types Key: SPARK-22472 URL: https://issues.apache.org/jira/browse/SPARK-22472 Project: Spark

[jira] [Updated] (SPARK-22471) SQLListener consumes much memory causing OutOfMemoryError

2017-11-08 Thread Arseniy Tashoyan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arseniy Tashoyan updated SPARK-22471: - Attachment: SQLListener_stageIdToStageMetrics_retained_size.png > SQLListener consumes mu

[jira] [Created] (SPARK-22471) SQLListener consumes much memory causing OutOfMemoryError

2017-11-08 Thread Arseniy Tashoyan (JIRA)
Arseniy Tashoyan created SPARK-22471: Summary: SQLListener consumes much memory causing OutOfMemoryError Key: SPARK-22471 URL: https://issues.apache.org/jira/browse/SPARK-22471 Project: Spark

[jira] [Commented] (SPARK-22460) Spark De-serialization of Timestamp field is Incorrect

2017-11-08 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16243913#comment-16243913 ] Kazuaki Ishizaki commented on SPARK-22460: -- I run the similar code with json. It

[jira] [Commented] (SPARK-22460) Spark De-serialization of Timestamp field is Incorrect

2017-11-08 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16243834#comment-16243834 ] Liang-Chi Hsieh commented on SPARK-22460: - In Spark SQL, to cast a long field to

[jira] [Commented] (SPARK-22377) Maven nightly snapshot jenkins jobs are broken on multiple workers due to lsof

2017-11-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16243789#comment-16243789 ] Apache Spark commented on SPARK-22377: -- User 'HyukjinKwon' has created a pull reques

[jira] [Assigned] (SPARK-22377) Maven nightly snapshot jenkins jobs are broken on multiple workers due to lsof

2017-11-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22377: Assignee: (was: Apache Spark) > Maven nightly snapshot jenkins jobs are broken on mult

[jira] [Assigned] (SPARK-22377) Maven nightly snapshot jenkins jobs are broken on multiple workers due to lsof

2017-11-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22377: Assignee: Apache Spark > Maven nightly snapshot jenkins jobs are broken on multiple worker

[jira] [Resolved] (SPARK-22446) Optimizer causing StringIndexerModel's indexer UDF to throw "Unseen label" exception incorrectly for filtered data.

2017-11-08 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-22446. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19662 [https://githu

[jira] [Assigned] (SPARK-22446) Optimizer causing StringIndexerModel's indexer UDF to throw "Unseen label" exception incorrectly for filtered data.

2017-11-08 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-22446: --- Assignee: Liang-Chi Hsieh > Optimizer causing StringIndexerModel's indexer UDF to throw "Uns

[jira] [Updated] (SPARK-22468) subtract creating empty DataFrame that isn't initialised properly

2017-11-08 Thread James Porritt (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Porritt updated SPARK-22468: -- Description: I have an issue whereby a subtract between two DataFrames that will correctly end

[jira] [Updated] (SPARK-22468) subtract creating empty DataFrame that isn't initialised properly

2017-11-08 Thread James Porritt (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Porritt updated SPARK-22468: -- Description: I have an issue whereby a subtract between two DataFrames that will correctly end

[jira] [Updated] (SPARK-22468) subtract creating empty DataFrame that isn't initialised properly

2017-11-08 Thread James Porritt (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Porritt updated SPARK-22468: -- Description: I have an issue whereby a subtract between two DataFrames that will correctly end

[jira] [Updated] (SPARK-22468) subtract creating empty DataFrame that isn't initialised properly

2017-11-08 Thread James Porritt (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Porritt updated SPARK-22468: -- Description: I have an issue whereby a subtract between two DataFrames that will correctly end

[jira] [Updated] (SPARK-22468) subtract creating empty DataFrame that isn't initialised properly

2017-11-08 Thread James Porritt (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Porritt updated SPARK-22468: -- Description: I have an issue whereby a subtract between two DataFrames that will correctly end

[jira] [Assigned] (SPARK-22470) Doc that functions.hash is also used internally for shuffle and bucketing

2017-11-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22470: Assignee: Apache Spark > Doc that functions.hash is also used internally for shuffle and b

[jira] [Assigned] (SPARK-22470) Doc that functions.hash is also used internally for shuffle and bucketing

2017-11-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22470: Assignee: (was: Apache Spark) > Doc that functions.hash is also used internally for sh

[jira] [Commented] (SPARK-22470) Doc that functions.hash is also used internally for shuffle and bucketing

2017-11-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16243720#comment-16243720 ] Apache Spark commented on SPARK-22470: -- User 'ash211' has created a pull request for

[jira] [Created] (SPARK-22470) Doc that functions.hash is also used internally for shuffle and bucketing

2017-11-08 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-22470: -- Summary: Doc that functions.hash is also used internally for shuffle and bucketing Key: SPARK-22470 URL: https://issues.apache.org/jira/browse/SPARK-22470 Project: Spark

[jira] [Commented] (SPARK-22469) Accuracy problem in comparison with string and numeric

2017-11-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16243600#comment-16243600 ] Apache Spark commented on SPARK-22469: -- User 'liutang123' has created a pull request

[jira] [Assigned] (SPARK-22469) Accuracy problem in comparison with string and numeric

2017-11-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22469: Assignee: Apache Spark > Accuracy problem in comparison with string and numeric > ---

[jira] [Assigned] (SPARK-22469) Accuracy problem in comparison with string and numeric

2017-11-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22469: Assignee: (was: Apache Spark) > Accuracy problem in comparison with string and numeric

[jira] [Created] (SPARK-22469) Accuracy problem in comparison with string and numeric

2017-11-08 Thread Lijia Liu (JIRA)
Lijia Liu created SPARK-22469: - Summary: Accuracy problem in comparison with string and numeric Key: SPARK-22469 URL: https://issues.apache.org/jira/browse/SPARK-22469 Project: Spark Issue Type:

[jira] [Created] (SPARK-22468) subtract creating empty DataFrame that isn't initialised properly

2017-11-08 Thread James Porritt (JIRA)
James Porritt created SPARK-22468: - Summary: subtract creating empty DataFrame that isn't initialised properly Key: SPARK-22468 URL: https://issues.apache.org/jira/browse/SPARK-22468 Project: Spark

[jira] [Resolved] (SPARK-22281) Handle R method breaking signature changes

2017-11-08 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung resolved SPARK-22281. -- Resolution: Fixed Assignee: Felix Cheung Fix Version/s: 2.3.0

[jira] [Assigned] (SPARK-14922) Alter Table Drop Partition Using Predicate-based Partition Spec

2017-11-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-14922: Assignee: (was: Apache Spark) > Alter Table Drop Partition Using Predicate-based Parti

[jira] [Commented] (SPARK-14922) Alter Table Drop Partition Using Predicate-based Partition Spec

2017-11-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16243488#comment-16243488 ] Apache Spark commented on SPARK-14922: -- User 'DazhuangSu' has created a pull request

[jira] [Commented] (SPARK-22467) Added a switch to support whether `stdout_stream` and `stderr_stream` output to disk

2017-11-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16243481#comment-16243481 ] Apache Spark commented on SPARK-22467: -- User '10110346' has created a pull request f

[jira] [Assigned] (SPARK-22467) Added a switch to support whether `stdout_stream` and `stderr_stream` output to disk

2017-11-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22467: Assignee: Apache Spark > Added a switch to support whether `stdout_stream` and `stderr_str

[jira] [Created] (SPARK-22467) Added a switch to support whether `stdout_stream` and `stderr_stream` output to disk

2017-11-08 Thread liuxian (JIRA)
liuxian created SPARK-22467: --- Summary: Added a switch to support whether `stdout_stream` and `stderr_stream` output to disk Key: SPARK-22467 URL: https://issues.apache.org/jira/browse/SPARK-22467 Project: S

[jira] [Assigned] (SPARK-22467) Added a switch to support whether `stdout_stream` and `stderr_stream` output to disk

2017-11-08 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22467: Assignee: (was: Apache Spark) > Added a switch to support whether `stdout_stream` and