[jira] [Commented] (SPARK-18941) Spark thrift server, Spark 2.0.2, The "drop table" command doesn't delete the directory associated with the Hive table (not EXTERNAL table) from the HDFS file system

[ https://issues.apache.org/jira/browse/SPARK-18941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15772224#comment-15772224 ] luat commented on SPARK-18941: -- Hi [~dongjoon], It is ok. But I think that this difference should be

[jira] [Created] (SPARK-18987) I am trying to disable Spark Stage progress logs on cluster.

vidit Singh created SPARK-18987: --- Summary: I am trying to disable Spark Stage progress logs on cluster. Key: SPARK-18987 URL: https://issues.apache.org/jira/browse/SPARK-18987 Project: Spark

[jira] [Updated] (SPARK-18974) FileInputDStream could not detected files which moved to the directory

[ https://issues.apache.org/jira/browse/SPARK-18974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Wang updated SPARK-18974: -- Description: FileInputDStream use mod time to find new files, but if a file was moved into the

[jira] [Commented] (SPARK-18978) Spark streaming ClassCastException

[ https://issues.apache.org/jira/browse/SPARK-18978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15772185#comment-15772185 ] Liang-Chi Hsieh commented on SPARK-18978: - Actually I can't reproduce this in master branch with

[jira] [Updated] (SPARK-18974) FileInputDStream could not detected files which moved to the directory

[ https://issues.apache.org/jira/browse/SPARK-18974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Wang updated SPARK-18974: -- Description: FileInputDStream use mod time to find new files, but if a file was moved into the

[jira] [Assigned] (SPARK-18986) ExternalAppendOnlyMap shouldn't fail when forced to spill before calling its iterator

[ https://issues.apache.org/jira/browse/SPARK-18986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18986: Assignee: Apache Spark > ExternalAppendOnlyMap shouldn't fail when forced to spill before

[jira] [Commented] (SPARK-18986) ExternalAppendOnlyMap shouldn't fail when forced to spill before calling its iterator

[ https://issues.apache.org/jira/browse/SPARK-18986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15772013#comment-15772013 ] Apache Spark commented on SPARK-18986: -- User 'viirya' has created a pull request for this issue:

[jira] [Commented] (SPARK-18199) Support appending to Parquet files

[ https://issues.apache.org/jira/browse/SPARK-18199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15772014#comment-15772014 ] Takeshi Yamamuro commented on SPARK-18199: -- Have you check this

[jira] [Assigned] (SPARK-18986) ExternalAppendOnlyMap shouldn't fail when forced to spill before calling its iterator

[ https://issues.apache.org/jira/browse/SPARK-18986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18986: Assignee: (was: Apache Spark) > ExternalAppendOnlyMap shouldn't fail when forced to

[jira] [Updated] (SPARK-18986) ExternalAppendOnlyMap shouldn't fail when forced to spill before calling its iterator

[ https://issues.apache.org/jira/browse/SPARK-18986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-18986: Component/s: (was: SQL) Spark Core > ExternalAppendOnlyMap shouldn't

[jira] [Created] (SPARK-18986) ExternalAppendOnlyMap shouldn't fail when forced to spill before calling its iterator

Liang-Chi Hsieh created SPARK-18986: --- Summary: ExternalAppendOnlyMap shouldn't fail when forced to spill before calling its iterator Key: SPARK-18986 URL: https://issues.apache.org/jira/browse/SPARK-18986

[jira] [Commented] (SPARK-9686) Spark Thrift server doesn't return correct JDBC metadata

[ https://issues.apache.org/jira/browse/SPARK-9686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771773#comment-15771773 ] David Rosenstrauch commented on SPARK-9686: --- Apologies, this actually wasn't the issue I was

[jira] [Updated] (SPARK-18981) The last job hung when speculation is on

[ https://issues.apache.org/jira/browse/SPARK-18981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] roncenzhao updated SPARK-18981: --- Description: CONF: spark.speculation true spark.dynamicAllocation.minExecutors0

[jira] [Updated] (SPARK-18981) The last job hung when speculation is on

[ https://issues.apache.org/jira/browse/SPARK-18981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] roncenzhao updated SPARK-18981: --- Description: CONF: spark.speculation true spark.dynamicAllocation.minExecutors0

[jira] [Updated] (SPARK-18981) The last job hung when speculation is on

[ https://issues.apache.org/jira/browse/SPARK-18981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] roncenzhao updated SPARK-18981: --- Attachment: run_scala.sh > The last job hung when speculation is on >

[jira] [Commented] (SPARK-18984) Concat with ds.write.text() throw exception if column contains null data

[ https://issues.apache.org/jira/browse/SPARK-18984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771706#comment-15771706 ] Hyukjin Kwon commented on SPARK-18984: -- [~tonythor], it seems you could reproduce this in Spark

[jira] [Updated] (SPARK-18981) The last job hung when speculation is on

[ https://issues.apache.org/jira/browse/SPARK-18981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] roncenzhao updated SPARK-18981: --- Attachment: Test.scala > The last job hung when speculation is on >

[jira] [Comment Edited] (SPARK-18984) Concat with ds.write.text() throw exception if column contains null data

[ https://issues.apache.org/jira/browse/SPARK-18984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771695#comment-15771695 ] Hyukjin Kwon edited comment on SPARK-18984 at 12/23/16 2:50 AM: Yes, I

[jira] [Commented] (SPARK-18984) Concat with ds.write.text() throw exception if column contains null data

[ https://issues.apache.org/jira/browse/SPARK-18984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771695#comment-15771695 ] Hyukjin Kwon commented on SPARK-18984: -- Yes, I can reproduce the same error with the codes as below:

[jira] [Assigned] (SPARK-18352) Parse normal, multi-line JSON files (not just JSON Lines)

[ https://issues.apache.org/jira/browse/SPARK-18352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18352: Assignee: (was: Apache Spark) > Parse normal, multi-line JSON files (not just JSON

[jira] [Assigned] (SPARK-18352) Parse normal, multi-line JSON files (not just JSON Lines)

[ https://issues.apache.org/jira/browse/SPARK-18352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18352: Assignee: Apache Spark > Parse normal, multi-line JSON files (not just JSON Lines) >

[jira] [Commented] (SPARK-18352) Parse normal, multi-line JSON files (not just JSON Lines)

[ https://issues.apache.org/jira/browse/SPARK-18352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771689#comment-15771689 ] Apache Spark commented on SPARK-18352: -- User 'NathanHowell' has created a pull request for this

[jira] [Comment Edited] (SPARK-18984) Concat with ds.write.text() throw exception if column contains null data

[ https://issues.apache.org/jira/browse/SPARK-18984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771675#comment-15771675 ] Hyukjin Kwon edited comment on SPARK-18984 at 12/23/16 2:41 AM: It seems

[jira] [Commented] (SPARK-18984) Concat with ds.write.text() throw exception if column contains null data

[ https://issues.apache.org/jira/browse/SPARK-18984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771675#comment-15771675 ] Hyukjin Kwon commented on SPARK-18984: -- It seems this is fixed together in SPARK-18658 in in master.

[jira] [Updated] (SPARK-18981) The last job hung when speculation is on

[ https://issues.apache.org/jira/browse/SPARK-18981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] roncenzhao updated SPARK-18981: --- Attachment: job_hang.png > The last job hung when speculation is on >

[jira] [Updated] (SPARK-18981) The last job hung when speculation is on

[ https://issues.apache.org/jira/browse/SPARK-18981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] roncenzhao updated SPARK-18981: --- Description: CONF: spark.speculation true spark.dynamicAllocation.minExecutors0

[jira] [Commented] (SPARK-18924) Improve collect/createDataFrame performance in SparkR

[ https://issues.apache.org/jira/browse/SPARK-18924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771602#comment-15771602 ] Shivaram Venkataraman commented on SPARK-18924: --- Yeah I think we could convert this JIRA

[jira] [Commented] (SPARK-18970) FileSource failure during file list refresh doesn't cause an application to fail, but stops further processing

[ https://issues.apache.org/jira/browse/SPARK-18970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771583#comment-15771583 ] Lev commented on SPARK-18970: - Actually this is exactly the behavior I want. My problem is that application

[jira] [Commented] (SPARK-18805) InternalMapWithStateDStream make java.lang.StackOverflowError

[ https://issues.apache.org/jira/browse/SPARK-18805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771504#comment-15771504 ] Vladimir Pchelko commented on SPARK-18805: -- I had faced with similar problem ... there are two

[jira] [Resolved] (SPARK-18972) Fix the netty thread names for RPC

[ https://issues.apache.org/jira/browse/SPARK-18972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-18972. -- Resolution: Fixed Fix Version/s: 2.2.0 2.1.1 2.0.3

[jira] [Updated] (SPARK-17807) Scalatest listed as compile dependency in spark-tags

[ https://issues.apache.org/jira/browse/SPARK-17807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-17807: --- Fix Version/s: 2.0.3 > Scalatest listed as compile dependency in spark-tags >

[jira] [Resolved] (SPARK-18985) Add missing @InterfaceStability.Evolving for Structured Streaming APIs

[ https://issues.apache.org/jira/browse/SPARK-18985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-18985. -- Resolution: Fixed Fix Version/s: 2.2.0 2.1.1 > Add missing

[jira] [Resolved] (SPARK-17807) Scalatest listed as compile dependency in spark-tags

[ https://issues.apache.org/jira/browse/SPARK-17807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-17807. Resolution: Fixed Fix Version/s: 2.1.1 > Scalatest listed as compile dependency in

[jira] [Commented] (SPARK-18618) SparkR GLM model predict should support type as a argument

[ https://issues.apache.org/jira/browse/SPARK-18618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771417#comment-15771417 ] Joseph K. Bradley commented on SPARK-18618: --- Note that [~yanboliang]'s PR from [SPARK-18291]

[jira] [Closed] (SPARK-18291) SparkR glm predict should output original label when family = "binomial"

[ https://issues.apache.org/jira/browse/SPARK-18291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley closed SPARK-18291. - Resolution: Duplicate Target Version/s: (was: 2.2.0) I'm closing this since

[jira] [Updated] (SPARK-18618) SparkR GLM model predict should support type as a argument

[ https://issues.apache.org/jira/browse/SPARK-18618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-18618: -- Description: SparkR GLM model {{predict}} should support {{type}} as a argument. This

[jira] [Updated] (SPARK-18618) SparkR GLM model predict should support type as a argument

[ https://issues.apache.org/jira/browse/SPARK-18618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-18618: -- Summary: SparkR GLM model predict should support type as a argument (was: SparkR

[jira] [Commented] (SPARK-9686) Spark Thrift server doesn't return correct JDBC metadata

[ https://issues.apache.org/jira/browse/SPARK-9686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771364#comment-15771364 ] David Rosenstrauch commented on SPARK-9686: --- Ditto. Any closer to a fix? > Spark Thrift server

[jira] [Commented] (SPARK-18970) FileSource failure during file list refresh doesn't cause an application to fail, but stops further processing

[ https://issues.apache.org/jira/browse/SPARK-18970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771304#comment-15771304 ] Shixiong Zhu commented on SPARK-18970: -- I see. FileStreamSource ignores FileNotFoundException when

[jira] [Updated] (SPARK-18970) FileSource failure during file list refresh doesn't cause an application to fail, but stops further processing

[ https://issues.apache.org/jira/browse/SPARK-18970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lev updated SPARK-18970: Attachment: sparkerror.log Here is the log file > FileSource failure during file list refresh doesn't cause an

[jira] [Commented] (SPARK-18970) FileSource failure during file list refresh doesn't cause an application to fail, but stops further processing

[ https://issues.apache.org/jira/browse/SPARK-18970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771221#comment-15771221 ] Shixiong Zhu commented on SPARK-18970: -- Where did you find the exception? If it's in the driver,

[jira] [Commented] (SPARK-18970) FileSource failure during file list refresh doesn't cause an application to fail, but stops further processing

[ https://issues.apache.org/jira/browse/SPARK-18970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771199#comment-15771199 ] Lev commented on SPARK-18970: - I am not sure whether task was retried or not, but Spark application never

[jira] [Updated] (SPARK-17344) Kafka 0.8 support for Structured Streaming

[ https://issues.apache.org/jira/browse/SPARK-17344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-17344: - Target Version/s: (was: 2.1.1) > Kafka 0.8 support for Structured Streaming >

[jira] [Commented] (SPARK-18985) Add missing @InterfaceStability.Evolving for Structured Streaming APIs

[ https://issues.apache.org/jira/browse/SPARK-18985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771175#comment-15771175 ] Apache Spark commented on SPARK-18985: -- User 'zsxwing' has created a pull request for this issue:

[jira] [Assigned] (SPARK-18985) Add missing @InterfaceStability.Evolving for Structured Streaming APIs

[ https://issues.apache.org/jira/browse/SPARK-18985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18985: Assignee: Shixiong Zhu (was: Apache Spark) > Add missing @InterfaceStability.Evolving

[jira] [Commented] (SPARK-18941) Spark thrift server, Spark 2.0.2, The "drop table" command doesn't delete the directory associated with the Hive table (not EXTERNAL table) from the HDFS file system

[ https://issues.apache.org/jira/browse/SPARK-18941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771176#comment-15771176 ] Dongjoon Hyun commented on SPARK-18941: --- After investigating, I found that it was reported by

[jira] [Assigned] (SPARK-18985) Add missing @InterfaceStability.Evolving for Structured Streaming APIs

[ https://issues.apache.org/jira/browse/SPARK-18985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18985: Assignee: Apache Spark (was: Shixiong Zhu) > Add missing @InterfaceStability.Evolving

[jira] [Created] (SPARK-18985) Add missing @InterfaceStability.Evolving for Structured Streaming APIs

Shixiong Zhu created SPARK-18985: Summary: Add missing @InterfaceStability.Evolving for Structured Streaming APIs Key: SPARK-18985 URL: https://issues.apache.org/jira/browse/SPARK-18985 Project:

[jira] [Updated] (SPARK-18537) Add a REST api to spark streaming

[ https://issues.apache.org/jira/browse/SPARK-18537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-18537: --- Assignee: Xing Shi > Add a REST api to spark streaming > - >

[jira] [Commented] (SPARK-18970) FileSource failure during file list refresh doesn't cause an application to fail, but stops further processing

[ https://issues.apache.org/jira/browse/SPARK-18970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771166#comment-15771166 ] Shixiong Zhu commented on SPARK-18970: -- Did the Spark task fail or not? Looks like the Spark task

[jira] [Resolved] (SPARK-18537) Add a REST api to spark streaming

[ https://issues.apache.org/jira/browse/SPARK-18537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-18537. Resolution: Fixed Fix Version/s: 2.2.0 > Add a REST api to spark streaming >

[jira] [Commented] (SPARK-18054) Unexpected error from UDF that gets an element of a vector: argument 1 requires vector type, however, '`_column_`' is of vector type

[ https://issues.apache.org/jira/browse/SPARK-18054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15770991#comment-15770991 ] Ilya Matiach commented on SPARK-18054: -- It looks like I can still repro the error with this code:

[jira] [Commented] (SPARK-18054) Unexpected error from UDF that gets an element of a vector: argument 1 requires vector type, however, '`_column_`' is of vector type

[ https://issues.apache.org/jira/browse/SPARK-18054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15770981#comment-15770981 ] Ilya Matiach commented on SPARK-18054: -- Actually, that error message above looks different. Maybe

[jira] [Resolved] (SPARK-18054) Unexpected error from UDF that gets an element of a vector: argument 1 requires vector type, however, '`_column_`' is of vector type

[ https://issues.apache.org/jira/browse/SPARK-18054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-18054. --- Resolution: Not A Problem > Unexpected error from UDF that gets an element of a vector: argument 1

[jira] [Commented] (SPARK-18054) Unexpected error from UDF that gets an element of a vector: argument 1 requires vector type, however, '`_column_`' is of vector type

[ https://issues.apache.org/jira/browse/SPARK-18054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15770972#comment-15770972 ] Ilya Matiach commented on SPARK-18054: -- It looks like this is already fixed in the latest version.

[jira] [Commented] (SPARK-18941) Spark thrift server, Spark 2.0.2, The "drop table" command doesn't delete the directory associated with the Hive table (not EXTERNAL table) from the HDFS file system

[ https://issues.apache.org/jira/browse/SPARK-18941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15770967#comment-15770967 ] Dongjoon Hyun commented on SPARK-18941: --- Hi, first of all. your case is correct. I can reproduce

[jira] [Comment Edited] (SPARK-18941) Spark thrift server, Spark 2.0.2, The "drop table" command doesn't delete the directory associated with the Hive table (not EXTERNAL table) from the HDFS file sys

[ https://issues.apache.org/jira/browse/SPARK-18941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15770967#comment-15770967 ] Dongjoon Hyun edited comment on SPARK-18941 at 12/22/16 8:05 PM: - Hi,

[jira] [Created] (SPARK-18984) Concat with ds.write.text() throw exception if column contains null data

Tony Fraser created SPARK-18984: --- Summary: Concat with ds.write.text() throw exception if column contains null data Key: SPARK-18984 URL: https://issues.apache.org/jira/browse/SPARK-18984 Project:

[jira] [Resolved] (SPARK-18975) Add an API to remove SparkListener from SparkContext

[ https://issues.apache.org/jira/browse/SPARK-18975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-18975. - Resolution: Fixed Assignee: Saisai Shao Fix Version/s: 2.2.0 > Add an API to

[jira] [Resolved] (SPARK-18983) Couldn't find leader offsets exception when the one of kafka cluster brokers is down

[ https://issues.apache.org/jira/browse/SPARK-18983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-18983. --- Resolution: Invalid Target Version/s: (was: 1.6.1) Please read

[jira] [Resolved] (SPARK-18982) Couldn't find leader offsets exception when the one of kafka cluster brokers is down

[ https://issues.apache.org/jira/browse/SPARK-18982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-18982. --- Resolution: Duplicate Target Version/s: (was: 1.6.1) > Couldn't find leader offsets

[jira] [Commented] (SPARK-18941) Spark thrift server, Spark 2.0.2, The "drop table" command doesn't delete the directory associated with the Hive table (not EXTERNAL table) from the HDFS file system

[ https://issues.apache.org/jira/browse/SPARK-18941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15770823#comment-15770823 ] Dongjoon Hyun commented on SPARK-18941: --- Thank you for the detail! I'll try that. > Spark thrift

[jira] [Created] (SPARK-18983) Couldn't find leader offsets exception when the one of kafka cluster brokers is down

kraken created SPARK-18983: -- Summary: Couldn't find leader offsets exception when the one of kafka cluster brokers is down Key: SPARK-18983 URL: https://issues.apache.org/jira/browse/SPARK-18983 Project:

[jira] [Created] (SPARK-18982) Couldn't find leader offsets exception when the one of kafka cluster brokers is down

kraken created SPARK-18982: -- Summary: Couldn't find leader offsets exception when the one of kafka cluster brokers is down Key: SPARK-18982 URL: https://issues.apache.org/jira/browse/SPARK-18982 Project:

[jira] [Updated] (SPARK-18970) FileSource failure during file list refresh doesn't cause an application to fail, but stops further processing

[ https://issues.apache.org/jira/browse/SPARK-18970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lev updated SPARK-18970: Affects Version/s: 2.0.2 > FileSource failure during file list refresh doesn't cause an application to > fail,

[jira] [Resolved] (SPARK-18973) Remove SortPartitions and RedistributeData

[ https://issues.apache.org/jira/browse/SPARK-18973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell resolved SPARK-18973. --- Resolution: Fixed Fix Version/s: 2.2.0 2.1.1 > Remove

[jira] [Updated] (SPARK-18031) Flaky test: org.apache.spark.streaming.scheduler.ExecutorAllocationManagerSuite basic functionality

[ https://issues.apache.org/jira/browse/SPARK-18031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18031: - Fix Version/s: 2.0.3 > Flaky test: >

[jira] [Updated] (SPARK-18031) Flaky test: org.apache.spark.streaming.scheduler.ExecutorAllocationManagerSuite basic functionality

[ https://issues.apache.org/jira/browse/SPARK-18031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18031: - Fix Version/s: 2.2.0 > Flaky test: >

[jira] [Updated] (SPARK-18054) Unexpected error from UDF that gets an element of a vector: argument 1 requires vector type, however, '`_column_`' is of vector type

[ https://issues.apache.org/jira/browse/SPARK-18054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-18054: -- Priority: Minor (was: Major) Component/s: Documentation Issue Type: Improvement (was:

[jira] [Commented] (SPARK-18054) Unexpected error from UDF that gets an element of a vector: argument 1 requires vector type, however, '`_column_`' is of vector type

[ https://issues.apache.org/jira/browse/SPARK-18054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15770690#comment-15770690 ] Ilya Matiach commented on SPARK-18054: -- I can try to repro this and add in a better error message.

[jira] [Updated] (SPARK-18738) Some Spark SQL queries has poor performance on HDFS Erasure Coding feature when enabling dynamic allocation.

[ https://issues.apache.org/jira/browse/SPARK-18738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-18738: -- Fix Version/s: (was: 2.2.0) > Some Spark SQL queries has poor performance on HDFS Erasure Coding

[jira] [Updated] (SPARK-18234) Update mode in structured streaming

[ https://issues.apache.org/jira/browse/SPARK-18234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-18234: -- Assignee: Tathagata Das > Update mode in structured streaming > --- >

[jira] [Updated] (SPARK-18281) toLocalIterator yields time out error on pyspark2

[ https://issues.apache.org/jira/browse/SPARK-18281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-18281: -- Assignee: Liang-Chi Hsieh > toLocalIterator yields time out error on pyspark2 >

[jira] [Resolved] (SPARK-17801) [ML]Random Forest Regression fails for large input

[ https://issues.apache.org/jira/browse/SPARK-17801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-17801. --- Resolution: Not A Problem I think this is just attributable to extremely high maxBins, and not a

[jira] [Commented] (SPARK-17801) [ML]Random Forest Regression fails for large input

[ https://issues.apache.org/jira/browse/SPARK-17801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15770463#comment-15770463 ] Ilya Matiach commented on SPARK-17801: -- Taking a look into the error > [ML]Random Forest Regression

[jira] [Commented] (SPARK-17975) EMLDAOptimizer fails with ClassCastException on YARN

[ https://issues.apache.org/jira/browse/SPARK-17975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15770446#comment-15770446 ] Ilya Matiach commented on SPARK-17975: -- Could you send a link to the repro dataset? I could work on

[jira] [Resolved] (SPARK-18922) Fix more resource-closing-related and path-related test failures in identified ones on Windows

[ https://issues.apache.org/jira/browse/SPARK-18922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-18922. --- Resolution: Fixed Assignee: Hyukjin Kwon Fix Version/s: 2.2.0 If we find more like

[jira] [Resolved] (SPARK-18878) Fix/investigate the more identified test failures in Java/Scala on Windows

[ https://issues.apache.org/jira/browse/SPARK-18878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-18878. --- Resolution: Done Let's look at reopening recent JIRAs and adding PRs if you find more changes of

[jira] [Updated] (SPARK-18981) The last job hung when speculation is on

[ https://issues.apache.org/jira/browse/SPARK-18981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] roncenzhao updated SPARK-18981: --- Description: related settings: spark.speculation true spark.dynamicAllocation.minExecutors0

[jira] [Created] (SPARK-18981) The last job hung when speculation is on

roncenzhao created SPARK-18981: -- Summary: The last job hung when speculation is on Key: SPARK-18981 URL: https://issues.apache.org/jira/browse/SPARK-18981 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-18199) Support appending to Parquet files

[ https://issues.apache.org/jira/browse/SPARK-18199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15770282#comment-15770282 ] Soubhik Chakraborty commented on SPARK-18199: - Can't we use PARQUET-382 feature that got

[jira] [Commented] (SPARK-18878) Fix/investigate the more identified test failures in Java/Scala on Windows

[ https://issues.apache.org/jira/browse/SPARK-18878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15770030#comment-15770030 ] Hyukjin Kwon commented on SPARK-18878: -- Thank you for guiding me. Let me try to follow it. >

[jira] [Commented] (SPARK-18896) Suppress ScalaCheck warning -- Unknown ScalaCheck args provided when executing tests using sbt

[ https://issues.apache.org/jira/browse/SPARK-18896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15769991#comment-15769991 ] PJ Fanning commented on SPARK-18896: I noticed from the pull request that you are looking at possibly

[jira] [Updated] (SPARK-18977) Heavy udf is not stopped by cancelJobGroup

[ https://issues.apache.org/jira/browse/SPARK-18977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vitaly Gerasimov updated SPARK-18977: - Summary: Heavy udf is not stopped by cancelJobGroup (was: Heavy udf in not stopped by

[jira] [Assigned] (SPARK-18980) implement Aggregator with TypedImperativeAggregate

[ https://issues.apache.org/jira/browse/SPARK-18980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18980: Assignee: Wenchen Fan (was: Apache Spark) > implement Aggregator with

[jira] [Assigned] (SPARK-18980) implement Aggregator with TypedImperativeAggregate

[ https://issues.apache.org/jira/browse/SPARK-18980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18980: Assignee: Apache Spark (was: Wenchen Fan) > implement Aggregator with

[jira] [Commented] (SPARK-18980) implement Aggregator with TypedImperativeAggregate

[ https://issues.apache.org/jira/browse/SPARK-18980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15769916#comment-15769916 ] Apache Spark commented on SPARK-18980: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Created] (SPARK-18980) implement Aggregator with TypedImperativeAggregate

Wenchen Fan created SPARK-18980: --- Summary: implement Aggregator with TypedImperativeAggregate Key: SPARK-18980 URL: https://issues.apache.org/jira/browse/SPARK-18980 Project: Spark Issue Type:

[jira] [Commented] (SPARK-18979) ShutdownHookManager：Exception while deleting Spark temp dir

[ https://issues.apache.org/jira/browse/SPARK-18979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15769827#comment-15769827 ] zuotingbing commented on SPARK-18979: - my SPARK_LOCAL_DIRS value is setted like this:

[jira] [Resolved] (SPARK-18979) ShutdownHookManager：Exception while deleting Spark temp dir

[ https://issues.apache.org/jira/browse/SPARK-18979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-18979. --- Resolution: Duplicate > ShutdownHookManager：Exception while deleting Spark temp dir >

[jira] [Created] (SPARK-18979) ShutdownHookManager：Exception while deleting Spark temp dir

zuotingbing created SPARK-18979: --- Summary: ShutdownHookManager：Exception while deleting Spark temp dir Key: SPARK-18979 URL: https://issues.apache.org/jira/browse/SPARK-18979 Project: Spark

[jira] [Commented] (SPARK-18977) Heavy udf in not stopped by cancelJobGroup

[ https://issues.apache.org/jira/browse/SPARK-18977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15769741#comment-15769741 ] Vitaly Gerasimov commented on SPARK-18977: -- Yeah.. You are right. But how do jobs stop in Spark,

[jira] [Comment Edited] (SPARK-18608) Spark ML algorithms that check RDD cache level for internal caching double-cache data

[ https://issues.apache.org/jira/browse/SPARK-18608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15769718#comment-15769718 ] zakaria hili edited comment on SPARK-18608 at 12/22/16 10:42 AM: -

[jira] [Comment Edited] (SPARK-18608) Spark ML algorithms that check RDD cache level for internal caching double-cache data

[ https://issues.apache.org/jira/browse/SPARK-18608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15769718#comment-15769718 ] zakaria hili edited comment on SPARK-18608 at 12/22/16 10:41 AM: -

[jira] [Comment Edited] (SPARK-18608) Spark ML algorithms that check RDD cache level for internal caching double-cache data

[ https://issues.apache.org/jira/browse/SPARK-18608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15769718#comment-15769718 ] zakaria hili edited comment on SPARK-18608 at 12/22/16 10:39 AM: - I

[jira] [Commented] (SPARK-18608) Spark ML algorithms that check RDD cache level for internal caching double-cache data

[ https://issues.apache.org/jira/browse/SPARK-18608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15769718#comment-15769718 ] zakaria hili commented on SPARK-18608: -- I understand now what you mean, your purpose is to optimize

[jira] [Commented] (SPARK-18964) HiveContext does not support Time Interval Literals

[ https://issues.apache.org/jira/browse/SPARK-18964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15769691#comment-15769691 ] Suhas Nalapure commented on SPARK-18964: My understanding is that both the features, namely

[jira] [Commented] (SPARK-18878) Fix/investigate the more identified test failures in Java/Scala on Windows

[ https://issues.apache.org/jira/browse/SPARK-18878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15769663#comment-15769663 ] Sean Owen commented on SPARK-18878: --- I understand why you need to fix some of these things in batches,

[jira] [Updated] (SPARK-18976) in standlone mode,executor expired by HeartbeanReceiver that still take up cores but no tasks assigned to

[ https://issues.apache.org/jira/browse/SPARK-18976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-18976: -- Target Version/s: (was: 1.6.1) Fix Version/s: (was: 1.6.1) [~liujianhui] please read

[jira] [Commented] (SPARK-18608) Spark ML algorithms that check RDD cache level for internal caching double-cache data

[ https://issues.apache.org/jira/browse/SPARK-18608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15769648#comment-15769648 ] Sean Owen commented on SPARK-18608: --- I don't think this has to do with Pyspark. The situation is

  1   2   >