[jira] [Assigned] (SPARK-21451) HiveConf in SparkSQLCLIDriver doesn't respect spark.hadoop.some.hive.variables

2017-07-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21451: Assignee: Apache Spark > HiveConf in SparkSQLCLIDriver doesn't respect spark.hadoop.some.h

[jira] [Commented] (SPARK-21451) HiveConf in SparkSQLCLIDriver doesn't respect spark.hadoop.some.hive.variables

2017-07-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16091197#comment-16091197 ] Apache Spark commented on SPARK-21451: -- User 'yaooqinn' has created a pull request f

[jira] [Assigned] (SPARK-21451) HiveConf in SparkSQLCLIDriver doesn't respect spark.hadoop.some.hive.variables

2017-07-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21451: Assignee: (was: Apache Spark) > HiveConf in SparkSQLCLIDriver doesn't respect spark.ha

[jira] [Resolved] (SPARK-21450) List of NA is flattened inside a SparkR struct type

2017-07-17 Thread Hossein Falaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hossein Falaki resolved SPARK-21450. Resolution: Not A Bug > List of NA is flattened inside a SparkR struct type > -

[jira] [Commented] (SPARK-21450) List of NA is flattened inside a SparkR struct type

2017-07-17 Thread Hossein Falaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16091194#comment-16091194 ] Hossein Falaki commented on SPARK-21450: Thanks guys. I will close it as not an i

[jira] [Created] (SPARK-21451) HiveConf in SparkSQLCLIDriver doesn't respect spark.hadoop.some.hive.variables

2017-07-17 Thread Kent Yao (JIRA)
Kent Yao created SPARK-21451: Summary: HiveConf in SparkSQLCLIDriver doesn't respect spark.hadoop.some.hive.variables Key: SPARK-21451 URL: https://issues.apache.org/jira/browse/SPARK-21451 Project: Spark

[jira] [Comment Edited] (SPARK-21450) List of NA is flattened inside a SparkR struct type

2017-07-17 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16091180#comment-16091180 ] Hyukjin Kwon edited comment on SPARK-21450 at 7/18/17 6:35 AM:

[jira] [Commented] (SPARK-21450) List of NA is flattened inside a SparkR struct type

2017-07-17 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16091180#comment-16091180 ] Hyukjin Kwon commented on SPARK-21450: -- [~falaki], so, if this describes {code} >

[jira] [Commented] (SPARK-19908) Direct buffer memory OOM should not cause stage retries.

2017-07-17 Thread Bojan Babic (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16091168#comment-16091168 ] Bojan Babic commented on SPARK-19908: - [~skp33] got the same error while running RowM

[jira] [Commented] (SPARK-21450) List of NA is flattened inside a SparkR struct type

2017-07-17 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16091153#comment-16091153 ] Hyukjin Kwon commented on SPARK-21450: -- I have some time to test this on Scala/Pytho

[jira] [Commented] (SPARK-21450) List of NA is flattened inside a SparkR struct type

2017-07-17 Thread Hossein Falaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16091149#comment-16091149 ] Hossein Falaki commented on SPARK-21450: Yes, it was a copy/paste typo. Here is a

[jira] [Updated] (SPARK-21450) List of NA is flattened inside a SparkR struct type

2017-07-17 Thread Hossein Falaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hossein Falaki updated SPARK-21450: --- Description: Consider the following two cases copied from {{test_sparkSQL.R}}: {code} df <-

[jira] [Commented] (SPARK-21105) Useless empty files in hive table

2017-07-17 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16091137#comment-16091137 ] Hyukjin Kwon commented on SPARK-21105: -- Let me give a shot to clean up related JIRAs

[jira] [Commented] (SPARK-21450) List of NA is flattened inside a SparkR struct type

2017-07-17 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16091132#comment-16091132 ] Felix Cheung commented on SPARK-21450: -- [~hyukjin.kwon] If you follow the code in

[jira] [Resolved] (SPARK-21448) Hi dear guys, I have a question about aggregateByKey of pairrrd.

2017-07-17 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-21448. -- Resolution: Invalid Question should go to mailing list. I am resolving this. > Hi dear guys,

[jira] [Commented] (SPARK-21437) Java Keyword cannot be used in table schema

2017-07-17 Thread pin_zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16091100#comment-16091100 ] pin_zhang commented on SPARK-21437: --- Hive doesn't have such limitation, we can create

[jira] [Commented] (SPARK-21392) Unable to infer schema when loading large Parquet file

2017-07-17 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16091086#comment-16091086 ] Hyukjin Kwon commented on SPARK-21392: -- [~stuartreynolds], Are you running a cluster

[jira] [Resolved] (SPARK-21445) NotSerializableException thrown by UTF8String.IntWrapper

2017-07-17 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-21445. - Resolution: Fixed Fix Version/s: 2.3.0 2.2.1 Issue resolved by pull req

[jira] [Commented] (SPARK-21227) Unicode in Json field causes AnalysisException when selecting from Dataframe

2017-07-17 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16091076#comment-16091076 ] Hyukjin Kwon commented on SPARK-21227: -- [~sdia], I think you can but I am quite sure

[jira] [Updated] (SPARK-21332) Incorrect result type inferred for some decimal expressions

2017-07-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-21332: Labels: corr (was: ) > Incorrect result type inferred for some decimal expressions > -

[jira] [Resolved] (SPARK-21332) Incorrect result type inferred for some decimal expressions

2017-07-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-21332. - Resolution: Fixed Assignee: Anton Okolnychyi Fix Version/s: 2.3.0 > Incorrect result type

[jira] [Updated] (SPARK-21332) Incorrect result type inferred for some decimal expressions

2017-07-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-21332: Fix Version/s: 2.2.1 2.1.2 2.0.3 > Incorrect result type inferred for

[jira] [Updated] (SPARK-21332) Incorrect result type inferred for some decimal expressions

2017-07-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-21332: Labels: correctness (was: corr) > Incorrect result type inferred for some decimal expressions > --

[jira] [Resolved] (SPARK-21444) Fetch failure due to node reboot causes job failure

2017-07-17 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-21444. Resolution: Fixed Fix Version/s: 2.3.0 > Fetch failure due to node reboot causes job failure

[jira] [Updated] (SPARK-21450) List of NA is flattened inside a SparkR struct type

2017-07-17 Thread Hossein Falaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hossein Falaki updated SPARK-21450: --- Description: Consider the following two cases copied from {{test_sparkSQL.R}}: {code} df <-

[jira] [Updated] (SPARK-21450) List of NA is flattened inside a SparkR struct type

2017-07-17 Thread Hossein Falaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hossein Falaki updated SPARK-21450: --- Description: Consider the following two cases copied from {{test_sparkSQL.R}}: {code} df <-

[jira] [Created] (SPARK-21450) List of NA is flattened inside a SparkR struct type

2017-07-17 Thread Hossein Falaki (JIRA)
Hossein Falaki created SPARK-21450: -- Summary: List of NA is flattened inside a SparkR struct type Key: SPARK-21450 URL: https://issues.apache.org/jira/browse/SPARK-21450 Project: Spark Issue

[jira] [Assigned] (SPARK-21449) Hive client's SessionState was not closed properly in HiveExternalCatalog

2017-07-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21449: Assignee: (was: Apache Spark) > Hive client's SessionState was not closed properly in

[jira] [Assigned] (SPARK-21449) Hive client's SessionState was not closed properly in HiveExternalCatalog

2017-07-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21449: Assignee: Apache Spark > Hive client's SessionState was not closed properly in HiveExtern

[jira] [Commented] (SPARK-21449) Hive client's SessionState was not closed properly in HiveExternalCatalog

2017-07-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16091055#comment-16091055 ] Apache Spark commented on SPARK-21449: -- User 'yaooqinn' has created a pull request f

[jira] [Created] (SPARK-21449) Hive client's SessionState was not closed properly in HiveExternalCatalog

2017-07-17 Thread Kent Yao (JIRA)
Kent Yao created SPARK-21449: Summary: Hive client's SessionState was not closed properly in HiveExternalCatalog Key: SPARK-21449 URL: https://issues.apache.org/jira/browse/SPARK-21449 Project: Spark

[jira] [Assigned] (SPARK-21446) [SQL] JDBC Postgres fetchsize parameter ignored again

2017-07-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21446: Assignee: (was: Apache Spark) > [SQL] JDBC Postgres fetchsize parameter ignored again

[jira] [Assigned] (SPARK-21446) [SQL] JDBC Postgres fetchsize parameter ignored again

2017-07-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21446: Assignee: Apache Spark > [SQL] JDBC Postgres fetchsize parameter ignored again > -

[jira] [Commented] (SPARK-21446) [SQL] JDBC Postgres fetchsize parameter ignored again

2017-07-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16091044#comment-16091044 ] Apache Spark commented on SPARK-21446: -- User 'DFFuture' has created a pull request f

[jira] [Updated] (SPARK-21446) [SQL] JDBC Postgres fetchsize parameter ignored again

2017-07-17 Thread Albert Jang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Albert Jang updated SPARK-21446: Issue Type: Bug (was: Improvement) > [SQL] JDBC Postgres fetchsize parameter ignored again > -

[jira] [Commented] (SPARK-15703) Make ListenerBus event queue size configurable

2017-07-17 Thread Ruslan Dautkhanov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16091031#comment-16091031 ] Ruslan Dautkhanov commented on SPARK-15703: --- It breaks spark dynamic allocation

[jira] [Resolved] (SPARK-21409) Expose state store memory usage in SQL metrics and progress updates

2017-07-17 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-21409. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 18661 [https://g

[jira] [Comment Edited] (SPARK-15703) Make ListenerBus event queue size configurable

2017-07-17 Thread Carlos Bribiescas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090991#comment-16090991 ] Carlos Bribiescas edited comment on SPARK-15703 at 7/18/17 2:15 AM: ---

[jira] [Commented] (SPARK-15703) Make ListenerBus event queue size configurable

2017-07-17 Thread Carlos Bribiescas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090991#comment-16090991 ] Carlos Bribiescas commented on SPARK-15703: --- Does this only affect the UI or wi

[jira] [Assigned] (SPARK-21410) In RangePartitioner(partitions: Int, rdd: RDD[]), RangePartitioner.numPartitions is wrong if the number of elements in RDD (rdd.count()) is less than number of partitio

2017-07-17 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-21410: --- Assignee: APeng Zhang > In RangePartitioner(partitions: Int, rdd: RDD[]), > RangePartitione

[jira] [Resolved] (SPARK-21410) In RangePartitioner(partitions: Int, rdd: RDD[]), RangePartitioner.numPartitions is wrong if the number of elements in RDD (rdd.count()) is less than number of partitio

2017-07-17 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-21410. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18631 [https://githu

[jira] [Updated] (SPARK-21448) Hi dear guys, I have a question about aggregateByKey of pairrrd.

2017-07-17 Thread qihuagao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qihuagao updated SPARK-21448: - Description: java pair rrd has aggregateByKey, which can avoid full shuffle, so have impressive performa

[jira] [Created] (SPARK-21448) Hi dear guys, I have a question about aggregateByKey of pairrrd.

2017-07-17 Thread qihuagao (JIRA)
qihuagao created SPARK-21448: Summary: Hi dear guys, I have a question about aggregateByKey of pairrrd. Key: SPARK-21448 URL: https://issues.apache.org/jira/browse/SPARK-21448 Project: Spark Is

[jira] [Updated] (SPARK-19104) CompileException with Map and Case Class in Spark 2.1.0

2017-07-17 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-19104: Fix Version/s: 2.1.2 > CompileException with Map and Case Class in Spark 2.1.0 > -

[jira] [Updated] (SPARK-21446) [SQL] JDBC Postgres fetchsize parameter ignored again

2017-07-17 Thread Albert Jang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Albert Jang updated SPARK-21446: Description: This bug was fixed in https://github.com/apache/spark/pull/9861 but reappears in spark

[jira] [Comment Edited] (SPARK-21425) LongAccumulator, DoubleAccumulator not threadsafe

2017-07-17 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090918#comment-16090918 ] Shixiong Zhu edited comment on SPARK-21425 at 7/18/17 1:06 AM:

[jira] [Commented] (SPARK-21425) LongAccumulator, DoubleAccumulator not threadsafe

2017-07-17 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090918#comment-16090918 ] Shixiong Zhu commented on SPARK-21425: -- I remember that if making LongAccumulator, D

[jira] [Created] (SPARK-21447) Spark history server fails to render compressed inprogress history file in some cases.

2017-07-17 Thread Eric Vandenberg (JIRA)
Eric Vandenberg created SPARK-21447: --- Summary: Spark history server fails to render compressed inprogress history file in some cases. Key: SPARK-21447 URL: https://issues.apache.org/jira/browse/SPARK-21447

[jira] [Updated] (SPARK-21394) Reviving broken callable objects in UDF in PySpark

2017-07-17 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk updated SPARK-21394: Fix Version/s: 2.3.0 > Reviving broken callable objects in UDF in PySpark > ---

[jira] [Updated] (SPARK-21432) Reviving broken partial functions in UDF in PySpark

2017-07-17 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk updated SPARK-21432: Fix Version/s: 2.3.0 > Reviving broken partial functions in UDF in PySpark > --

[jira] [Assigned] (SPARK-21394) Reviving broken callable objects in UDF in PySpark

2017-07-17 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk reassigned SPARK-21394: --- Assignee: Hyukjin Kwon > Reviving broken callable objects in UDF in PySpark > --

[jira] [Resolved] (SPARK-21394) Reviving broken callable objects in UDF in PySpark

2017-07-17 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk resolved SPARK-21394. - Resolution: Fixed > Reviving broken callable objects in UDF in PySpark >

[jira] [Resolved] (SPARK-21432) Reviving broken partial functions in UDF in PySpark

2017-07-17 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk resolved SPARK-21432. - Resolution: Fixed > Reviving broken partial functions in UDF in PySpark > ---

[jira] [Assigned] (SPARK-21432) Reviving broken partial functions in UDF in PySpark

2017-07-17 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk reassigned SPARK-21432: --- Assignee: Hyukjin Kwon > Reviving broken partial functions in UDF in PySpark > -

[jira] [Updated] (SPARK-21394) Reviving broken callable objects in UDF in PySpark

2017-07-17 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk updated SPARK-21394: Affects Version/s: (was: 2.2.0) > Reviving broken callable objects in UDF in PySpark >

[jira] [Assigned] (SPARK-21375) Add date and timestamp support to ArrowConverters for toPandas() collection

2017-07-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21375: Assignee: Apache Spark > Add date and timestamp support to ArrowConverters for toPandas()

[jira] [Commented] (SPARK-21375) Add date and timestamp support to ArrowConverters for toPandas() collection

2017-07-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090898#comment-16090898 ] Apache Spark commented on SPARK-21375: -- User 'BryanCutler' has created a pull reques

[jira] [Assigned] (SPARK-21375) Add date and timestamp support to ArrowConverters for toPandas() collection

2017-07-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21375: Assignee: (was: Apache Spark) > Add date and timestamp support to ArrowConverters for

[jira] [Issue Comment Deleted] (SPARK-17565) Janino exception when calculating metrics for large generated class

2017-07-17 Thread Huan Dao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huan Dao updated SPARK-17565: - Comment: was deleted (was: I'm getting this error, any update for fixing this?) > Janino exception when

[jira] [Issue Comment Deleted] (SPARK-17565) Janino exception when calculating metrics for large generated class

2017-07-17 Thread Huan Dao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huan Dao updated SPARK-17565: - Comment: was deleted (was: I'm getting this error, any update for fixing this?) > Janino exception when

[jira] [Commented] (SPARK-17565) Janino exception when calculating metrics for large generated class

2017-07-17 Thread Huan Dao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090861#comment-16090861 ] Huan Dao commented on SPARK-17565: -- I'm getting this error, any update for fixing this?

[jira] [Commented] (SPARK-17565) Janino exception when calculating metrics for large generated class

2017-07-17 Thread Huan Dao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090859#comment-16090859 ] Huan Dao commented on SPARK-17565: -- I'm getting this error, any update for fixing this?

[jira] [Commented] (SPARK-20079) Re registration of AM hangs spark cluster in yarn-client mode

2017-07-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090854#comment-16090854 ] Apache Spark commented on SPARK-20079: -- User 'vanzin' has created a pull request for

[jira] [Assigned] (SPARK-9686) Spark Thrift server doesn't return correct JDBC metadata

2017-07-17 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian reassigned SPARK-9686: - Assignee: (was: Cheng Lian) > Spark Thrift server doesn't return correct JDBC metadata > ---

[jira] [Commented] (SPARK-21444) Fetch failure due to node reboot causes job failure

2017-07-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090834#comment-16090834 ] Apache Spark commented on SPARK-21444: -- User 'JoshRosen' has created a pull request

[jira] [Updated] (SPARK-21446) [SQL] JDBC Postgres fetchsize parameter ignored again

2017-07-17 Thread Albert Jang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Albert Jang updated SPARK-21446: Description: This bug is fixed in https://github.com/apache/spark/pull/9861 but reappears in spark

[jira] [Updated] (SPARK-21446) [SQL] JDBC Postgres fetchsize parameter ignored again

2017-07-17 Thread Albert Jang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Albert Jang updated SPARK-21446: Description: This bug is fixed in https://github.com/apache/spark/pull/9861 but reappears in spark

[jira] [Created] (SPARK-21446) [SQL] JDBC Postgres fetchsize parameter ignored again

2017-07-17 Thread Albert Jang (JIRA)
Albert Jang created SPARK-21446: --- Summary: [SQL] JDBC Postgres fetchsize parameter ignored again Key: SPARK-21446 URL: https://issues.apache.org/jira/browse/SPARK-21446 Project: Spark Issue Typ

[jira] [Commented] (SPARK-21409) Expose state store memory usage in SQL metrics and progress updates

2017-07-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090829#comment-16090829 ] Apache Spark commented on SPARK-21409: -- User 'tdas' has created a pull request for t

[jira] [Assigned] (SPARK-21445) NotSerializableException thrown by UTF8String.IntWrapper

2017-07-17 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz reassigned SPARK-21445: --- Assignee: Burak Yavuz > NotSerializableException thrown by UTF8String.IntWrapper > -

[jira] [Assigned] (SPARK-21444) Fetch failure due to node reboot causes job failure

2017-07-17 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen reassigned SPARK-21444: -- Assignee: Josh Rosen > Fetch failure due to node reboot causes job failure > -

[jira] [Commented] (SPARK-21444) Fetch failure due to node reboot causes job failure

2017-07-17 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090815#comment-16090815 ] Josh Rosen commented on SPARK-21444: I spot the problem: in the old code, we removed

[jira] [Assigned] (SPARK-21445) NotSerializableException thrown by UTF8String.IntWrapper

2017-07-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21445: Assignee: Apache Spark > NotSerializableException thrown by UTF8String.IntWrapper > --

[jira] [Assigned] (SPARK-21445) NotSerializableException thrown by UTF8String.IntWrapper

2017-07-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21445: Assignee: (was: Apache Spark) > NotSerializableException thrown by UTF8String.IntWrapp

[jira] [Commented] (SPARK-21445) NotSerializableException thrown by UTF8String.IntWrapper

2017-07-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090813#comment-16090813 ] Apache Spark commented on SPARK-21445: -- User 'brkyvz' has created a pull request for

[jira] [Created] (SPARK-21445) NotSerializableException thrown by UTF8String.IntWrapper

2017-07-17 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-21445: --- Summary: NotSerializableException thrown by UTF8String.IntWrapper Key: SPARK-21445 URL: https://issues.apache.org/jira/browse/SPARK-21445 Project: Spark Issue

[jira] [Commented] (SPARK-21444) Fetch failure due to node reboot causes job failure

2017-07-17 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090809#comment-16090809 ] Josh Rosen commented on SPARK-21444: I'm going to adjust the "affects versions" on th

[jira] [Updated] (SPARK-21444) Fetch failure due to node reboot causes job failure

2017-07-17 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-21444: --- Affects Version/s: (was: 2.0.2) 2.3.0 > Fetch failure due to node reboot c

[jira] [Commented] (SPARK-21444) Fetch failure due to node reboot causes job failure

2017-07-17 Thread Sital Kedia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090800#comment-16090800 ] Sital Kedia commented on SPARK-21444: - Any idea how to fix this issue? > Fetch failu

[jira] [Commented] (SPARK-21444) Fetch failure due to node reboot causes job failure

2017-07-17 Thread Sital Kedia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090799#comment-16090799 ] Sital Kedia commented on SPARK-21444: - cc - [~joshrosen] > Fetch failure due to node

[jira] [Created] (SPARK-21444) Fetch failure due to node reboot causes job failure

2017-07-17 Thread Sital Kedia (JIRA)
Sital Kedia created SPARK-21444: --- Summary: Fetch failure due to node reboot causes job failure Key: SPARK-21444 URL: https://issues.apache.org/jira/browse/SPARK-21444 Project: Spark Issue Type:

[jira] [Commented] (SPARK-20307) SparkR: pass on setHandleInvalid to spark.mllib functions that use StringIndexer

2017-07-17 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090792#comment-16090792 ] Yanbo Liang commented on SPARK-20307: - [~wangmiao1981] I think [~podongfeng] already

[jira] [Commented] (SPARK-21187) Complete support for remaining Spark data types in Arrow Converters

2017-07-17 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090624#comment-16090624 ] Bryan Cutler commented on SPARK-21187: -- NOTE - There was a bug fixed in Arrow 0.4.1

[jira] [Updated] (SPARK-21375) Add date and timestamp support to ArrowConverters for toPandas() collection

2017-07-17 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler updated SPARK-21375: - Issue Type: Sub-task (was: Improvement) Parent: SPARK-21187 > Add date and timestamp sup

[jira] [Assigned] (SPARK-21404) Simple Vectorized Python UDFs using Arrow

2017-07-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21404: Assignee: (was: Apache Spark) > Simple Vectorized Python UDFs using Arrow > --

[jira] [Commented] (SPARK-21404) Simple Vectorized Python UDFs using Arrow

2017-07-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090516#comment-16090516 ] Apache Spark commented on SPARK-21404: -- User 'BryanCutler' has created a pull reques

[jira] [Assigned] (SPARK-21404) Simple Vectorized Python UDFs using Arrow

2017-07-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21404: Assignee: Apache Spark > Simple Vectorized Python UDFs using Arrow > -

[jira] [Resolved] (SPARK-21377) Jars specified with --jars or --packages are not added into AM's system classpath

2017-07-17 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-21377. Resolution: Fixed Assignee: Saisai Shao Fix Version/s: 2.3.0 > Jars specifi

[jira] [Commented] (SPARK-15574) Python meta-algorithms in Scala

2017-07-17 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090459#comment-16090459 ] Joseph K. Bradley commented on SPARK-15574: --- (Just commented on the PR; I'm unc

[jira] [Commented] (SPARK-21443) Very long planning duration for queries with lots of operations

2017-07-17 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090242#comment-16090242 ] Kazuaki Ishizaki commented on SPARK-21443: -- These two optimizations {{InferFilte

[jira] [Commented] (SPARK-20307) SparkR: pass on setHandleInvalid to spark.mllib functions that use StringIndexer

2017-07-17 Thread Miao Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090213#comment-16090213 ] Miao Wang commented on SPARK-20307: --- [~yanboliang] Have you added the support on Python

[jira] [Resolved] (SPARK-21321) Spark very verbose on shutdown confusing users

2017-07-17 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-21321. --- Resolution: Fixed Assignee: Jong Yoon Lee Fix Version/s: 2.3.0

[jira] [Commented] (SPARK-11182) HDFS Delegation Token will be expired when calling "UserGroupInformation.getCurrentUser.addCredentials" in HA mode

2017-07-17 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090183#comment-16090183 ] John Zhuge commented on SPARK-11182: Is this issue fixed by HDFS-9276? > HDFS Delega

[jira] [Commented] (SPARK-21392) Unable to infer schema when loading large Parquet file

2017-07-17 Thread Stuart Reynolds (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090159#comment-16090159 ] Stuart Reynolds commented on SPARK-21392: - So trying to look at the csv was h

[jira] [Commented] (SPARK-20871) Only log Janino code in debug mode

2017-07-17 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090146#comment-16090146 ] Sean Owen commented on SPARK-20871: --- Got it, so it can fail in normal operation. I can

[jira] [Commented] (SPARK-21392) Unable to infer schema when loading large Parquet file

2017-07-17 Thread Stuart Reynolds (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090134#comment-16090134 ] Stuart Reynolds commented on SPARK-21392: - I've made the example self contained a

[jira] [Updated] (SPARK-21392) Unable to infer schema when loading large Parquet file

2017-07-17 Thread Stuart Reynolds (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stuart Reynolds updated SPARK-21392: Description: The following boring code works up until when I read in the parquet file. {co

[jira] [Updated] (SPARK-21392) Unable to infer schema when loading large Parquet file

2017-07-17 Thread Stuart Reynolds (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stuart Reynolds updated SPARK-21392: Description: The following boring code works up until when I read in the parquet file. {co

[jira] [Commented] (SPARK-20871) Only log Janino code in debug mode

2017-07-17 Thread PJ Fanning (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090125#comment-16090125 ] PJ Fanning commented on SPARK-20871: [~srowen] One of the main reasons for the Janino

[jira] [Updated] (SPARK-20871) Only log Janino code in debug mode

2017-07-17 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-20871: -- Priority: Trivial (was: Major) Fix Version/s: (was: 2.3.0) > Only log Janino code in debu

  1   2   >