[jira] [Comment Edited] (SPARK-18881) Spark never finishes jobs and stages, JobProgressListener fails

[ https://issues.apache.org/jira/browse/SPARK-18881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030025#comment-16030025 ] Mathieu D edited comment on SPARK-18881 at 5/30/17 7:52 PM: Just to mention a

[jira] [Commented] (SPARK-18881) Spark never finishes jobs and stages, JobProgressListener fails

[ https://issues.apache.org/jira/browse/SPARK-18881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030025#comment-16030025 ] Mathieu D commented on SPARK-18881: --- Just to mention a workaround for those experiencing the problem :

[jira] [Commented] (SPARK-18838) High latency of event processing for large jobs

[ https://issues.apache.org/jira/browse/SPARK-18838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16017528#comment-16017528 ] Mathieu D commented on SPARK-18838: --- I'm not very familiar with this part of Spark, but I'd like to

[jira] [Closed] (SPARK-20784) Spark hangs (v2.0) or Futures timed out (v2.1) after a joinWith() and cache() in YARN client mode

[ https://issues.apache.org/jira/browse/SPARK-20784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mathieu D closed SPARK-20784. - Resolution: Not A Bug Oh boy, it was an OOM on the driver. Most of the times, it was silent. I just

[jira] [Updated] (SPARK-20784) Spark hangs (v2.0) or Futures timed out (v2.1) after a joinWith() and cache() in YARN client mode

[ https://issues.apache.org/jira/browse/SPARK-20784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mathieu D updated SPARK-20784: -- Affects Version/s: 2.1.1 Description: Spark hangs and stop executing any job or task

[jira] [Commented] (SPARK-20784) Spark hangs forever after a joinWith() and cache()

[ https://issues.apache.org/jira/browse/SPARK-20784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16013911#comment-16013911 ] Mathieu D commented on SPARK-20784: --- Changed the title. It's noted for the self reproducer, although

[jira] [Updated] (SPARK-20784) Spark hangs forever after a joinWith() and cache()

[ https://issues.apache.org/jira/browse/SPARK-20784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mathieu D updated SPARK-20784: -- Summary: Spark hangs forever after a joinWith() and cache() (was: Spark hangs forever) > Spark hangs

[jira] [Updated] (SPARK-20784) Spark hangs forever

[ https://issues.apache.org/jira/browse/SPARK-20784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mathieu D updated SPARK-20784: -- Description: Spark hangs and stop executing any job or task. Web UI shows *0 active stages* and *0

[jira] [Commented] (SPARK-20784) Spark hangs forever

[ https://issues.apache.org/jira/browse/SPARK-20784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16013784#comment-16013784 ] Mathieu D commented on SPARK-20784: --- Well, Spark is blocked, though. I had a couple of occurrences of

[jira] [Updated] (SPARK-20784) Spark hangs forever / potential deadlock ?

[ https://issues.apache.org/jira/browse/SPARK-20784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mathieu D updated SPARK-20784: -- Description: Spark hangs and stop executing any job or task. Web UI shows 0 active task on executors.

[jira] [Updated] (SPARK-20784) Spark hangs forever

[ https://issues.apache.org/jira/browse/SPARK-20784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mathieu D updated SPARK-20784: -- Summary: Spark hangs forever (was: Spark hangs forever / potential deadlock ?) > Spark hangs forever

[jira] [Created] (SPARK-20784) Spark hangs forever / potential deadlock ?

Mathieu D created SPARK-20784: - Summary: Spark hangs forever / potential deadlock ? Key: SPARK-20784 URL: https://issues.apache.org/jira/browse/SPARK-20784 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-20082) Incremental update of LDA model, by adding initialModel as start point

[ https://issues.apache.org/jira/browse/SPARK-20082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958754#comment-15958754 ] Mathieu D commented on SPARK-20082: --- [~yuhaoyan] or [~josephkb] any feedback on this approach and PR ?

[jira] [Comment Edited] (SPARK-20082) Incremental update of LDA model, by adding initialModel as start point

[ https://issues.apache.org/jira/browse/SPARK-20082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945892#comment-15945892 ] Mathieu D edited comment on SPARK-20082 at 3/28/17 8:39 PM: [~yuhaoyan] would

[jira] [Commented] (SPARK-20082) Incremental update of LDA model, by adding initialModel as start point

[ https://issues.apache.org/jira/browse/SPARK-20082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945892#comment-15945892 ] Mathieu D commented on SPARK-20082: --- [~yuhaoyan] would you mind having a look to this PR. Right now, I

[jira] [Comment Edited] (SPARK-20082) Incremental update of LDA model, by adding initialModel as start point

[ https://issues.apache.org/jira/browse/SPARK-20082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945892#comment-15945892 ] Mathieu D edited comment on SPARK-20082 at 3/28/17 8:26 PM: [~yuhaoyan] would

[jira] [Comment Edited] (SPARK-20082) Incremental update of LDA model, by adding initialModel as start point

[ https://issues.apache.org/jira/browse/SPARK-20082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945892#comment-15945892 ] Mathieu D edited comment on SPARK-20082 at 3/28/17 8:27 PM: [~yuhaoyan] would

[jira] [Created] (SPARK-20082) Incremental update of LDA model, by adding initialModel as start point

Mathieu D created SPARK-20082: - Summary: Incremental update of LDA model, by adding initialModel as start point Key: SPARK-20082 URL: https://issues.apache.org/jira/browse/SPARK-20082 Project: Spark

[jira] [Comment Edited] (SPARK-17890) scala.ScalaReflectionException

[ https://issues.apache.org/jira/browse/SPARK-17890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15874360#comment-15874360 ] Mathieu D edited comment on SPARK-17890 at 2/20/17 11:01 AM: - We experience

[jira] [Commented] (SPARK-17890) scala.ScalaReflectionException

[ https://issues.apache.org/jira/browse/SPARK-17890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15874360#comment-15874360 ] Mathieu D commented on SPARK-17890: --- We experience the same issue. When running from our tests (no

[jira] [Updated] (SPARK-19136) Aggregator with case class as output type fails with ClassCastException

[ https://issues.apache.org/jira/browse/SPARK-19136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mathieu D updated SPARK-19136: -- Priority: Minor (was: Major) > Aggregator with case class as output type fails with

[jira] [Commented] (SPARK-19136) Aggregator with case class as output type fails with ClassCastException

[ https://issues.apache.org/jira/browse/SPARK-19136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816422#comment-15816422 ] Mathieu D commented on SPARK-19136: --- And... a RDD version based on treeAggregate is even quicker :-/ At

[jira] [Commented] (SPARK-19136) Aggregator with case class as output type fails with ClassCastException

[ https://issues.apache.org/jira/browse/SPARK-19136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816366#comment-15816366 ] Mathieu D commented on SPARK-19136: --- Both queries are not equivalent, the dummy group generate a

[jira] [Updated] (SPARK-19136) Aggregator with case class as output type fails with ClassCastException

[ https://issues.apache.org/jira/browse/SPARK-19136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mathieu D updated SPARK-19136: -- Summary: Aggregator with case class as output type fails with ClassCastException (was: Aggregator

[jira] [Created] (SPARK-19136) Aggregator fails with case class as output type

Mathieu D created SPARK-19136: - Summary: Aggregator fails with case class as output type Key: SPARK-19136 URL: https://issues.apache.org/jira/browse/SPARK-19136 Project: Spark Issue Type: Bug

[jira] [Issue Comment Deleted] (SPARK-17668) Support representing structs with case classes and tuples in spark sql udf inputs

[ https://issues.apache.org/jira/browse/SPARK-17668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mathieu D updated SPARK-17668: -- Comment: was deleted (was: I experience the same issue with a custom Aggregator having a case class

[jira] [Commented] (SPARK-17668) Support representing structs with case classes and tuples in spark sql udf inputs

[ https://issues.apache.org/jira/browse/SPARK-17668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15802399#comment-15802399 ] Mathieu D commented on SPARK-17668: --- I experience the same issue with a custom Aggregator having a case

[jira] [Updated] (SPARK-18881) Spark never finishes jobs and stages, JobProgressListener fails

[ https://issues.apache.org/jira/browse/SPARK-18881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mathieu D updated SPARK-18881: -- Description: We have a Spark application that process continuously a lot of incoming jobs. Several

[jira] [Commented] (SPARK-18883) FileNotFoundException on _temporary directory

[ https://issues.apache.org/jira/browse/SPARK-18883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15763747#comment-15763747 ] Mathieu D commented on SPARK-18883: --- The problem does not appear with

[jira] [Commented] (SPARK-18883) FileNotFoundException on _temporary directory

[ https://issues.apache.org/jira/browse/SPARK-18883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751582#comment-15751582 ] Mathieu D commented on SPARK-18883: --- as suggested by [~steve_l], I'm going to try the

[jira] [Commented] (SPARK-18512) FileNotFoundException on _temporary directory with Spark Streaming 2.0.1 and S3A

[ https://issues.apache.org/jira/browse/SPARK-18512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751577#comment-15751577 ] Mathieu D commented on SPARK-18512: --- [SPARK-18883] > FileNotFoundException on _temporary directory

[jira] [Created] (SPARK-18883) FileNotFoundException on _temporary directory

Mathieu D created SPARK-18883: - Summary: FileNotFoundException on _temporary directory Key: SPARK-18883 URL: https://issues.apache.org/jira/browse/SPARK-18883 Project: Spark Issue Type: Bug

[jira] [Comment Edited] (SPARK-18512) FileNotFoundException on _temporary directory with Spark Streaming 2.0.1 and S3A

[ https://issues.apache.org/jira/browse/SPARK-18512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751329#comment-15751329 ] Mathieu D edited comment on SPARK-18512 at 12/15/16 1:17 PM: - I'm

[jira] [Commented] (SPARK-18512) FileNotFoundException on _temporary directory with Spark Streaming 2.0.1 and S3A

[ https://issues.apache.org/jira/browse/SPARK-18512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751329#comment-15751329 ] Mathieu D commented on SPARK-18512: --- I'm experiencing the same problem, but we're using HDFS, not S3.

[jira] [Created] (SPARK-18881) Spark never finishes jobs and stages, JobProgressListener fails

Mathieu D created SPARK-18881: - Summary: Spark never finishes jobs and stages, JobProgressListener fails Key: SPARK-18881 URL: https://issues.apache.org/jira/browse/SPARK-18881 Project: Spark

[jira] [Commented] (SPARK-17168) CSV with header is incorrectly read if file is partitioned

[ https://issues.apache.org/jira/browse/SPARK-17168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15429443#comment-15429443 ] Mathieu D commented on SPARK-17168: --- This is error-prone, because the scenario I show will drop rows

[jira] [Created] (SPARK-17168) CSV with header is incorrectly read if file is partitioned

Mathieu D created SPARK-17168: - Summary: CSV with header is incorrectly read if file is partitioned Key: SPARK-17168 URL: https://issues.apache.org/jira/browse/SPARK-17168 Project: Spark Issue

[jira] [Created] (SPARK-16938) Cannot resolve column name after a join

Mathieu D created SPARK-16938: - Summary: Cannot resolve column name after a join Key: SPARK-16938 URL: https://issues.apache.org/jira/browse/SPARK-16938 Project: Spark Issue Type: Bug