[jira] [Commented] (SPARK-21450) List of NA is flattened inside a SparkR struct type

2017-07-17 Thread Hossein Falaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091149#comment-16091149 ] Hossein Falaki commented on SPARK-21450: Yes, it was a copy/paste typo. Here is an example:

[jira] [Updated] (SPARK-21450) List of NA is flattened inside a SparkR struct type

2017-07-17 Thread Hossein Falaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hossein Falaki updated SPARK-21450: --- Description: Consider the following two cases copied from {{test_sparkSQL.R}}: {code} df <-

[jira] [Commented] (SPARK-21105) Useless empty files in hive table

2017-07-17 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091137#comment-16091137 ] Hyukjin Kwon commented on SPARK-21105: -- Let me give a shot to clean up related JIRAs with a followup

[jira] [Commented] (SPARK-21450) List of NA is flattened inside a SparkR struct type

2017-07-17 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091132#comment-16091132 ] Felix Cheung commented on SPARK-21450: -- [~hyukjin.kwon] If you follow the code in test_sparkSQL.R,

[jira] [Resolved] (SPARK-21448) Hi dear guys, I have a question about aggregateByKey of pairrrd.

2017-07-17 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-21448. -- Resolution: Invalid Question should go to mailing list. I am resolving this. > Hi dear guys,

[jira] [Commented] (SPARK-21437) Java Keyword cannot be used in table schema

2017-07-17 Thread pin_zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091100#comment-16091100 ] pin_zhang commented on SPARK-21437: --- Hive doesn't have such limitation, we can create table with sql

[jira] [Commented] (SPARK-21392) Unable to infer schema when loading large Parquet file

2017-07-17 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091086#comment-16091086 ] Hyukjin Kwon commented on SPARK-21392: -- [~stuartreynolds], Are you running a cluster but using a

[jira] [Resolved] (SPARK-21445) NotSerializableException thrown by UTF8String.IntWrapper

2017-07-17 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-21445. - Resolution: Fixed Fix Version/s: 2.3.0 2.2.1 Issue resolved by pull

[jira] [Commented] (SPARK-21227) Unicode in Json field causes AnalysisException when selecting from Dataframe

2017-07-17 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091076#comment-16091076 ] Hyukjin Kwon commented on SPARK-21227: -- [~sdia], I think you can but I am quite sure that proper and

[jira] [Updated] (SPARK-21332) Incorrect result type inferred for some decimal expressions

2017-07-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-21332: Labels: corr (was: ) > Incorrect result type inferred for some decimal expressions >

[jira] [Resolved] (SPARK-21332) Incorrect result type inferred for some decimal expressions

2017-07-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-21332. - Resolution: Fixed Assignee: Anton Okolnychyi Fix Version/s: 2.3.0 > Incorrect result

[jira] [Updated] (SPARK-21332) Incorrect result type inferred for some decimal expressions

2017-07-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-21332: Fix Version/s: 2.2.1 2.1.2 2.0.3 > Incorrect result type inferred

[jira] [Updated] (SPARK-21332) Incorrect result type inferred for some decimal expressions

2017-07-17 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-21332: Labels: correctness (was: corr) > Incorrect result type inferred for some decimal expressions >

[jira] [Resolved] (SPARK-21444) Fetch failure due to node reboot causes job failure

2017-07-17 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-21444. Resolution: Fixed Fix Version/s: 2.3.0 > Fetch failure due to node reboot causes job

[jira] [Updated] (SPARK-21450) List of NA is flattened inside a SparkR struct type

2017-07-17 Thread Hossein Falaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hossein Falaki updated SPARK-21450: --- Description: Consider the following two cases copied from {{test_sparkSQL.R}}: {code} df <-

[jira] [Updated] (SPARK-21450) List of NA is flattened inside a SparkR struct type

2017-07-17 Thread Hossein Falaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hossein Falaki updated SPARK-21450: --- Description: Consider the following two cases copied from {{test_sparkSQL.R}}: {code} df <-

[jira] [Created] (SPARK-21450) List of NA is flattened inside a SparkR struct type

2017-07-17 Thread Hossein Falaki (JIRA)
Hossein Falaki created SPARK-21450: -- Summary: List of NA is flattened inside a SparkR struct type Key: SPARK-21450 URL: https://issues.apache.org/jira/browse/SPARK-21450 Project: Spark

[jira] [Assigned] (SPARK-21449) Hive client's SessionState was not closed properly in HiveExternalCatalog

2017-07-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21449: Assignee: (was: Apache Spark) > Hive client's SessionState was not closed properly

[jira] [Assigned] (SPARK-21449) Hive client's SessionState was not closed properly in HiveExternalCatalog

2017-07-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21449: Assignee: Apache Spark > Hive client's SessionState was not closed properly in

[jira] [Commented] (SPARK-21449) Hive client's SessionState was not closed properly in HiveExternalCatalog

2017-07-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091055#comment-16091055 ] Apache Spark commented on SPARK-21449: -- User 'yaooqinn' has created a pull request for this issue:

[jira] [Created] (SPARK-21449) Hive client's SessionState was not closed properly in HiveExternalCatalog

2017-07-17 Thread Kent Yao (JIRA)
Kent Yao created SPARK-21449: Summary: Hive client's SessionState was not closed properly in HiveExternalCatalog Key: SPARK-21449 URL: https://issues.apache.org/jira/browse/SPARK-21449 Project: Spark

[jira] [Assigned] (SPARK-21446) [SQL] JDBC Postgres fetchsize parameter ignored again

2017-07-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21446: Assignee: (was: Apache Spark) > [SQL] JDBC Postgres fetchsize parameter ignored again

[jira] [Assigned] (SPARK-21446) [SQL] JDBC Postgres fetchsize parameter ignored again

2017-07-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21446: Assignee: Apache Spark > [SQL] JDBC Postgres fetchsize parameter ignored again >

[jira] [Commented] (SPARK-21446) [SQL] JDBC Postgres fetchsize parameter ignored again

2017-07-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091044#comment-16091044 ] Apache Spark commented on SPARK-21446: -- User 'DFFuture' has created a pull request for this issue:

[jira] [Updated] (SPARK-21446) [SQL] JDBC Postgres fetchsize parameter ignored again

2017-07-17 Thread Albert Jang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Albert Jang updated SPARK-21446: Issue Type: Bug (was: Improvement) > [SQL] JDBC Postgres fetchsize parameter ignored again >

[jira] [Commented] (SPARK-15703) Make ListenerBus event queue size configurable

2017-07-17 Thread Ruslan Dautkhanov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091031#comment-16091031 ] Ruslan Dautkhanov commented on SPARK-15703: --- It breaks spark dynamic allocation too - Spark

[jira] [Resolved] (SPARK-21409) Expose state store memory usage in SQL metrics and progress updates

2017-07-17 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-21409. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 18661

[jira] [Comment Edited] (SPARK-15703) Make ListenerBus event queue size configurable

2017-07-17 Thread Carlos Bribiescas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090991#comment-16090991 ] Carlos Bribiescas edited comment on SPARK-15703 at 7/18/17 2:15 AM:

[jira] [Commented] (SPARK-15703) Make ListenerBus event queue size configurable

2017-07-17 Thread Carlos Bribiescas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090991#comment-16090991 ] Carlos Bribiescas commented on SPARK-15703: --- Does this only affect the UI or will jobs actually

[jira] [Assigned] (SPARK-21410) In RangePartitioner(partitions: Int, rdd: RDD[]), RangePartitioner.numPartitions is wrong if the number of elements in RDD (rdd.count()) is less than number of partitio

2017-07-17 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-21410: --- Assignee: APeng Zhang > In RangePartitioner(partitions: Int, rdd: RDD[]), >

[jira] [Resolved] (SPARK-21410) In RangePartitioner(partitions: Int, rdd: RDD[]), RangePartitioner.numPartitions is wrong if the number of elements in RDD (rdd.count()) is less than number of partitio

2017-07-17 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-21410. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18631

[jira] [Updated] (SPARK-21448) Hi dear guys, I have a question about aggregateByKey of pairrrd.

2017-07-17 Thread qihuagao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qihuagao updated SPARK-21448: - Description: java pair rrd has aggregateByKey, which can avoid full shuffle, so have impressive

[jira] [Created] (SPARK-21448) Hi dear guys, I have a question about aggregateByKey of pairrrd.

2017-07-17 Thread qihuagao (JIRA)
qihuagao created SPARK-21448: Summary: Hi dear guys, I have a question about aggregateByKey of pairrrd. Key: SPARK-21448 URL: https://issues.apache.org/jira/browse/SPARK-21448 Project: Spark

[jira] [Updated] (SPARK-19104) CompileException with Map and Case Class in Spark 2.1.0

2017-07-17 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-19104: Fix Version/s: 2.1.2 > CompileException with Map and Case Class in Spark 2.1.0 >

[jira] [Updated] (SPARK-21446) [SQL] JDBC Postgres fetchsize parameter ignored again

2017-07-17 Thread Albert Jang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Albert Jang updated SPARK-21446: Description: This bug was fixed in https://github.com/apache/spark/pull/9861 but reappears in

[jira] [Comment Edited] (SPARK-21425) LongAccumulator, DoubleAccumulator not threadsafe

2017-07-17 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090918#comment-16090918 ] Shixiong Zhu edited comment on SPARK-21425 at 7/18/17 1:06 AM: --- I remember

[jira] [Commented] (SPARK-21425) LongAccumulator, DoubleAccumulator not threadsafe

2017-07-17 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090918#comment-16090918 ] Shixiong Zhu commented on SPARK-21425: -- I remember that if making LongAccumulator, DoubleAccumulator

[jira] [Created] (SPARK-21447) Spark history server fails to render compressed inprogress history file in some cases.

2017-07-17 Thread Eric Vandenberg (JIRA)
Eric Vandenberg created SPARK-21447: --- Summary: Spark history server fails to render compressed inprogress history file in some cases. Key: SPARK-21447 URL: https://issues.apache.org/jira/browse/SPARK-21447

[jira] [Updated] (SPARK-21394) Reviving broken callable objects in UDF in PySpark

2017-07-17 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk updated SPARK-21394: Fix Version/s: 2.3.0 > Reviving broken callable objects in UDF in PySpark >

[jira] [Updated] (SPARK-21432) Reviving broken partial functions in UDF in PySpark

2017-07-17 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk updated SPARK-21432: Fix Version/s: 2.3.0 > Reviving broken partial functions in UDF in PySpark >

[jira] [Assigned] (SPARK-21394) Reviving broken callable objects in UDF in PySpark

2017-07-17 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk reassigned SPARK-21394: --- Assignee: Hyukjin Kwon > Reviving broken callable objects in UDF in PySpark >

[jira] [Resolved] (SPARK-21394) Reviving broken callable objects in UDF in PySpark

2017-07-17 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk resolved SPARK-21394. - Resolution: Fixed > Reviving broken callable objects in UDF in PySpark >

[jira] [Resolved] (SPARK-21432) Reviving broken partial functions in UDF in PySpark

2017-07-17 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk resolved SPARK-21432. - Resolution: Fixed > Reviving broken partial functions in UDF in PySpark >

[jira] [Assigned] (SPARK-21432) Reviving broken partial functions in UDF in PySpark

2017-07-17 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk reassigned SPARK-21432: --- Assignee: Hyukjin Kwon > Reviving broken partial functions in UDF in PySpark >

[jira] [Updated] (SPARK-21394) Reviving broken callable objects in UDF in PySpark

2017-07-17 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk updated SPARK-21394: Affects Version/s: (was: 2.2.0) > Reviving broken callable objects in UDF in PySpark >

[jira] [Assigned] (SPARK-21375) Add date and timestamp support to ArrowConverters for toPandas() collection

2017-07-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21375: Assignee: Apache Spark > Add date and timestamp support to ArrowConverters for toPandas()

[jira] [Commented] (SPARK-21375) Add date and timestamp support to ArrowConverters for toPandas() collection

2017-07-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090898#comment-16090898 ] Apache Spark commented on SPARK-21375: -- User 'BryanCutler' has created a pull request for this

[jira] [Assigned] (SPARK-21375) Add date and timestamp support to ArrowConverters for toPandas() collection

2017-07-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21375: Assignee: (was: Apache Spark) > Add date and timestamp support to ArrowConverters for

[jira] [Issue Comment Deleted] (SPARK-17565) Janino exception when calculating metrics for large generated class

2017-07-17 Thread Huan Dao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huan Dao updated SPARK-17565: - Comment: was deleted (was: I'm getting this error, any update for fixing this?) > Janino exception

[jira] [Issue Comment Deleted] (SPARK-17565) Janino exception when calculating metrics for large generated class

2017-07-17 Thread Huan Dao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huan Dao updated SPARK-17565: - Comment: was deleted (was: I'm getting this error, any update for fixing this?) > Janino exception when

[jira] [Commented] (SPARK-17565) Janino exception when calculating metrics for large generated class

2017-07-17 Thread Huan Dao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090861#comment-16090861 ] Huan Dao commented on SPARK-17565: -- I'm getting this error, any update for fixing this? > Janino

[jira] [Commented] (SPARK-17565) Janino exception when calculating metrics for large generated class

2017-07-17 Thread Huan Dao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090859#comment-16090859 ] Huan Dao commented on SPARK-17565: -- I'm getting this error, any update for fixing this? > Janino

[jira] [Commented] (SPARK-20079) Re registration of AM hangs spark cluster in yarn-client mode

2017-07-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090854#comment-16090854 ] Apache Spark commented on SPARK-20079: -- User 'vanzin' has created a pull request for this issue:

[jira] [Assigned] (SPARK-9686) Spark Thrift server doesn't return correct JDBC metadata

2017-07-17 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian reassigned SPARK-9686: - Assignee: (was: Cheng Lian) > Spark Thrift server doesn't return correct JDBC metadata >

[jira] [Commented] (SPARK-21444) Fetch failure due to node reboot causes job failure

2017-07-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090834#comment-16090834 ] Apache Spark commented on SPARK-21444: -- User 'JoshRosen' has created a pull request for this issue:

[jira] [Updated] (SPARK-21446) [SQL] JDBC Postgres fetchsize parameter ignored again

2017-07-17 Thread Albert Jang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Albert Jang updated SPARK-21446: Description: This bug is fixed in https://github.com/apache/spark/pull/9861 but reappears in spark

[jira] [Updated] (SPARK-21446) [SQL] JDBC Postgres fetchsize parameter ignored again

2017-07-17 Thread Albert Jang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Albert Jang updated SPARK-21446: Description: This bug is fixed in https://github.com/apache/spark/pull/9861 but reappears in spark

[jira] [Created] (SPARK-21446) [SQL] JDBC Postgres fetchsize parameter ignored again

2017-07-17 Thread Albert Jang (JIRA)
Albert Jang created SPARK-21446: --- Summary: [SQL] JDBC Postgres fetchsize parameter ignored again Key: SPARK-21446 URL: https://issues.apache.org/jira/browse/SPARK-21446 Project: Spark Issue

[jira] [Commented] (SPARK-21409) Expose state store memory usage in SQL metrics and progress updates

2017-07-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090829#comment-16090829 ] Apache Spark commented on SPARK-21409: -- User 'tdas' has created a pull request for this issue:

[jira] [Assigned] (SPARK-21445) NotSerializableException thrown by UTF8String.IntWrapper

2017-07-17 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz reassigned SPARK-21445: --- Assignee: Burak Yavuz > NotSerializableException thrown by UTF8String.IntWrapper >

[jira] [Assigned] (SPARK-21444) Fetch failure due to node reboot causes job failure

2017-07-17 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen reassigned SPARK-21444: -- Assignee: Josh Rosen > Fetch failure due to node reboot causes job failure >

[jira] [Commented] (SPARK-21444) Fetch failure due to node reboot causes job failure

2017-07-17 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090815#comment-16090815 ] Josh Rosen commented on SPARK-21444: I spot the problem: in the old code, we removed the broadcast

[jira] [Assigned] (SPARK-21445) NotSerializableException thrown by UTF8String.IntWrapper

2017-07-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21445: Assignee: Apache Spark > NotSerializableException thrown by UTF8String.IntWrapper >

[jira] [Commented] (SPARK-21445) NotSerializableException thrown by UTF8String.IntWrapper

2017-07-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090813#comment-16090813 ] Apache Spark commented on SPARK-21445: -- User 'brkyvz' has created a pull request for this issue:

[jira] [Assigned] (SPARK-21445) NotSerializableException thrown by UTF8String.IntWrapper

2017-07-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21445: Assignee: (was: Apache Spark) > NotSerializableException thrown by

[jira] [Created] (SPARK-21445) NotSerializableException thrown by UTF8String.IntWrapper

2017-07-17 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-21445: --- Summary: NotSerializableException thrown by UTF8String.IntWrapper Key: SPARK-21445 URL: https://issues.apache.org/jira/browse/SPARK-21445 Project: Spark Issue

[jira] [Commented] (SPARK-21444) Fetch failure due to node reboot causes job failure

2017-07-17 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090809#comment-16090809 ] Josh Rosen commented on SPARK-21444: I'm going to adjust the "affects versions" on this because it

[jira] [Updated] (SPARK-21444) Fetch failure due to node reboot causes job failure

2017-07-17 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-21444: --- Affects Version/s: (was: 2.0.2) 2.3.0 > Fetch failure due to node reboot

[jira] [Commented] (SPARK-21444) Fetch failure due to node reboot causes job failure

2017-07-17 Thread Sital Kedia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090800#comment-16090800 ] Sital Kedia commented on SPARK-21444: - Any idea how to fix this issue? > Fetch failure due to node

[jira] [Commented] (SPARK-21444) Fetch failure due to node reboot causes job failure

2017-07-17 Thread Sital Kedia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090799#comment-16090799 ] Sital Kedia commented on SPARK-21444: - cc - [~joshrosen] > Fetch failure due to node reboot causes

[jira] [Created] (SPARK-21444) Fetch failure due to node reboot causes job failure

2017-07-17 Thread Sital Kedia (JIRA)
Sital Kedia created SPARK-21444: --- Summary: Fetch failure due to node reboot causes job failure Key: SPARK-21444 URL: https://issues.apache.org/jira/browse/SPARK-21444 Project: Spark Issue

[jira] [Commented] (SPARK-20307) SparkR: pass on setHandleInvalid to spark.mllib functions that use StringIndexer

2017-07-17 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090792#comment-16090792 ] Yanbo Liang commented on SPARK-20307: - [~wangmiao1981] I think [~podongfeng] already added it in

[jira] [Commented] (SPARK-21187) Complete support for remaining Spark data types in Arrow Converters

2017-07-17 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090624#comment-16090624 ] Bryan Cutler commented on SPARK-21187: -- NOTE - There was a bug fixed in Arrow 0.4.1 for the Decimal

[jira] [Updated] (SPARK-21375) Add date and timestamp support to ArrowConverters for toPandas() collection

2017-07-17 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler updated SPARK-21375: - Issue Type: Sub-task (was: Improvement) Parent: SPARK-21187 > Add date and timestamp

[jira] [Assigned] (SPARK-21404) Simple Vectorized Python UDFs using Arrow

2017-07-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21404: Assignee: (was: Apache Spark) > Simple Vectorized Python UDFs using Arrow >

[jira] [Commented] (SPARK-21404) Simple Vectorized Python UDFs using Arrow

2017-07-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090516#comment-16090516 ] Apache Spark commented on SPARK-21404: -- User 'BryanCutler' has created a pull request for this

[jira] [Assigned] (SPARK-21404) Simple Vectorized Python UDFs using Arrow

2017-07-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21404: Assignee: Apache Spark > Simple Vectorized Python UDFs using Arrow >

[jira] [Resolved] (SPARK-21377) Jars specified with --jars or --packages are not added into AM's system classpath

2017-07-17 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-21377. Resolution: Fixed Assignee: Saisai Shao Fix Version/s: 2.3.0 > Jars

[jira] [Commented] (SPARK-15574) Python meta-algorithms in Scala

2017-07-17 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090459#comment-16090459 ] Joseph K. Bradley commented on SPARK-15574: --- (Just commented on the PR; I'm uncertain about the

[jira] [Commented] (SPARK-21443) Very long planning duration for queries with lots of operations

2017-07-17 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090242#comment-16090242 ] Kazuaki Ishizaki commented on SPARK-21443: -- These two optimizations

[jira] [Commented] (SPARK-20307) SparkR: pass on setHandleInvalid to spark.mllib functions that use StringIndexer

2017-07-17 Thread Miao Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090213#comment-16090213 ] Miao Wang commented on SPARK-20307: --- [~yanboliang] Have you added the support on Python side? I think

[jira] [Resolved] (SPARK-21321) Spark very verbose on shutdown confusing users

2017-07-17 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-21321. --- Resolution: Fixed Assignee: Jong Yoon Lee Fix Version/s: 2.3.0

[jira] [Commented] (SPARK-11182) HDFS Delegation Token will be expired when calling "UserGroupInformation.getCurrentUser.addCredentials" in HA mode

2017-07-17 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090183#comment-16090183 ] John Zhuge commented on SPARK-11182: Is this issue fixed by HDFS-9276? > HDFS Delegation Token will

[jira] [Commented] (SPARK-21392) Unable to infer schema when loading large Parquet file

2017-07-17 Thread Stuart Reynolds (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090159#comment-16090159 ] Stuart Reynolds commented on SPARK-21392: - So trying to look at the csv was helpful.

[jira] [Commented] (SPARK-20871) Only log Janino code in debug mode

2017-07-17 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090146#comment-16090146 ] Sean Owen commented on SPARK-20871: --- Got it, so it can fail in normal operation. I can see turning down

[jira] [Commented] (SPARK-21392) Unable to infer schema when loading large Parquet file

2017-07-17 Thread Stuart Reynolds (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090134#comment-16090134 ] Stuart Reynolds commented on SPARK-21392: - I've made the example self contained and sourced from

[jira] [Updated] (SPARK-21392) Unable to infer schema when loading large Parquet file

2017-07-17 Thread Stuart Reynolds (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stuart Reynolds updated SPARK-21392: Description: The following boring code works up until when I read in the parquet file.

[jira] [Updated] (SPARK-21392) Unable to infer schema when loading large Parquet file

2017-07-17 Thread Stuart Reynolds (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stuart Reynolds updated SPARK-21392: Description: The following boring code works up until when I read in the parquet file.

[jira] [Commented] (SPARK-20871) Only log Janino code in debug mode

2017-07-17 Thread PJ Fanning (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090125#comment-16090125 ] PJ Fanning commented on SPARK-20871: [~srowen] One of the main reasons for the Janino compile to fail

[jira] [Updated] (SPARK-20871) Only log Janino code in debug mode

2017-07-17 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-20871: -- Priority: Trivial (was: Major) Fix Version/s: (was: 2.3.0) > Only log Janino code in

[jira] [Commented] (SPARK-20871) Only log Janino code in debug mode

2017-07-17 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090106#comment-16090106 ] Sean Owen commented on SPARK-20871: --- Why would you not want this information, if compilation has

[jira] [Updated] (SPARK-20871) Only log Janino code in debug mode

2017-07-17 Thread PJ Fanning (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PJ Fanning updated SPARK-20871: --- Fix Version/s: 2.3.0 > Only log Janino code in debug mode > -- > >

[jira] [Updated] (SPARK-20871) Only log Janino code in debug mode

2017-07-17 Thread PJ Fanning (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PJ Fanning updated SPARK-20871: --- Component/s: (was: Spark Core) SQL > Only log Janino code in debug mode >

[jira] [Assigned] (SPARK-20871) Only log Janino code in debug mode

2017-07-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20871: Assignee: (was: Apache Spark) > Only log Janino code in debug mode >

[jira] [Assigned] (SPARK-20871) Only log Janino code in debug mode

2017-07-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20871: Assignee: Apache Spark > Only log Janino code in debug mode >

[jira] [Commented] (SPARK-20871) Only log Janino code in debug mode

2017-07-17 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090099#comment-16090099 ] Apache Spark commented on SPARK-20871: -- User 'pjfanning' has created a pull request for this issue:

[jira] [Resolved] (SPARK-21221) CrossValidator and TrainValidationSplit Persist Nested Estimators such as OneVsRest

2017-07-17 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley resolved SPARK-21221. --- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18428

[jira] [Commented] (SPARK-18085) SPIP: Better History Server scalability for many / large applications

2017-07-17 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090042#comment-16090042 ] Marcelo Vanzin commented on SPARK-18085:  > SPIP: Better History Server scalability for many /

[jira] [Commented] (SPARK-21433) Spark SQL should support higher version of Hive metastore

2017-07-17 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090035#comment-16090035 ] Marcelo Vanzin commented on SPARK-21433: I have no idea what is being asked here. Spark 2.2

[jira] [Commented] (SPARK-21425) LongAccumulator, DoubleAccumulator not threadsafe

2017-07-17 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090017#comment-16090017 ] Ryan Williams commented on SPARK-21425: --- Yea, static accumulators; I was able to reproduce it in

  1   2   >