[jira] [Comment Edited] (SPARK-18838) High latency of event processing for large jobs

[ https://issues.apache.org/jira/browse/SPARK-18838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129149#comment-16129149 ] Miles Crawford edited comment on SPARK-18838 at 8/16/17 5:57 PM: --

[jira] [Commented] (SPARK-18838) High latency of event processing for large jobs

[ https://issues.apache.org/jira/browse/SPARK-18838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129149#comment-16129149 ] Miles Crawford commented on SPARK-18838: [~imranr] and [~vanzin] Here is your log

[jira] [Commented] (SPARK-18838) High latency of event processing for large jobs

[ https://issues.apache.org/jira/browse/SPARK-18838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16126402#comment-16126402 ] Miles Crawford commented on SPARK-18838: [~imranr] We do runs tens of thousands o

[jira] [Comment Edited] (SPARK-18838) High latency of event processing for large jobs

[ https://issues.apache.org/jira/browse/SPARK-18838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16116916#comment-16116916 ] Miles Crawford edited comment on SPARK-18838 at 8/7/17 5:41 PM: ---

[jira] [Comment Edited] (SPARK-18838) High latency of event processing for large jobs

[ https://issues.apache.org/jira/browse/SPARK-18838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16116916#comment-16116916 ] Miles Crawford edited comment on SPARK-18838 at 8/7/17 5:41 PM: ---

[jira] [Commented] (SPARK-18838) High latency of event processing for large jobs

[ https://issues.apache.org/jira/browse/SPARK-18838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16116916#comment-16116916 ] Miles Crawford commented on SPARK-18838: Can I get specific direction on the logs

[jira] [Comment Edited] (SPARK-18838) High latency of event processing for large jobs

[ https://issues.apache.org/jira/browse/SPARK-18838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16116916#comment-16116916 ] Miles Crawford edited comment on SPARK-18838 at 8/7/17 5:40 PM: ---

[jira] [Commented] (SPARK-20863) Add metrics/instrumentation to LiveListenerBus

[ https://issues.apache.org/jira/browse/SPARK-20863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16116905#comment-16116905 ] Miles Crawford commented on SPARK-20863: How can I enable and view these metrics

[jira] [Commented] (SPARK-18838) High latency of event processing for large jobs

[ https://issues.apache.org/jira/browse/SPARK-18838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16116896#comment-16116896 ] Miles Crawford commented on SPARK-18838: We do not use dynamic allocation, and ou

[jira] [Commented] (SPARK-18838) High latency of event processing for large jobs

[ https://issues.apache.org/jira/browse/SPARK-18838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16116886#comment-16116886 ] Miles Crawford commented on SPARK-18838: This seems to be the core issue for a la

[jira] [Comment Edited] (SPARK-18838) High latency of event processing for large jobs

[ https://issues.apache.org/jira/browse/SPARK-18838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16113067#comment-16113067 ] Miles Crawford edited comment on SPARK-18838 at 8/3/17 5:13 PM: ---

[jira] [Commented] (SPARK-17669) Strange behavior using Datasets

[ https://issues.apache.org/jira/browse/SPARK-17669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16113083#comment-16113083 ] Miles Crawford commented on SPARK-17669: This UI behavior is caused by SPARK-1883

[jira] [Commented] (SPARK-18838) High latency of event processing for large jobs

[ https://issues.apache.org/jira/browse/SPARK-18838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16113067#comment-16113067 ] Miles Crawford commented on SPARK-18838: We have disabled our eventlog listener,

[jira] [Comment Edited] (SPARK-18838) High latency of event processing for large jobs

[ https://issues.apache.org/jira/browse/SPARK-18838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16113067#comment-16113067 ] Miles Crawford edited comment on SPARK-18838 at 8/3/17 4:51 PM: ---

[jira] [Commented] (SPARK-17669) Strange behavior using Datasets

[ https://issues.apache.org/jira/browse/SPARK-17669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15537420#comment-15537420 ] Miles Crawford commented on SPARK-17669: Was not able to reproduce using a variet

[jira] [Commented] (SPARK-17669) Strange behavior using Datasets

[ https://issues.apache.org/jira/browse/SPARK-17669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15533678#comment-15533678 ] Miles Crawford commented on SPARK-17669: I'm working on a minimal reproduction.

[jira] [Comment Edited] (SPARK-17669) Strange behavior using Datasets

[ https://issues.apache.org/jira/browse/SPARK-17669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15531133#comment-15531133 ] Miles Crawford edited comment on SPARK-17669 at 9/29/16 5:21 PM: --

[jira] [Commented] (SPARK-17669) Strange behavior using Datasets

[ https://issues.apache.org/jira/browse/SPARK-17669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15533394#comment-15533394 ] Miles Crawford commented on SPARK-17669: Unfortunately, I was wrong - I'm still g

[jira] [Commented] (SPARK-17669) Strange behavior using Datasets

[ https://issues.apache.org/jira/browse/SPARK-17669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15531133#comment-15531133 ] Miles Crawford commented on SPARK-17669: So, it looks like the issue was a versio

[jira] [Updated] (SPARK-17669) Strange behavior using Datasets

[ https://issues.apache.org/jira/browse/SPARK-17669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miles Crawford updated SPARK-17669: --- Summary: Strange behavior using Datasets (was: Strange UI behavior using Datasets) > Strang

[jira] [Created] (SPARK-17669) Strange UI behavior using Datasets

Miles Crawford created SPARK-17669: -- Summary: Strange UI behavior using Datasets Key: SPARK-17669 URL: https://issues.apache.org/jira/browse/SPARK-17669 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-15828) YARN is not aware of Spark's External Shuffle Service

[ https://issues.apache.org/jira/browse/SPARK-15828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15323043#comment-15323043 ] Miles Crawford commented on SPARK-15828: That's correct, on a cloud provider, AWS

[jira] [Commented] (SPARK-15828) YARN is not aware of Spark's External Shuffle Service

[ https://issues.apache.org/jira/browse/SPARK-15828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322876#comment-15322876 ] Miles Crawford commented on SPARK-15828: This is Hadoop's standard means of resiz

[jira] [Commented] (SPARK-15828) YARN is not aware of Spark's External Shuffle Service

[ https://issues.apache.org/jira/browse/SPARK-15828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15322827#comment-15322827 ] Miles Crawford commented on SPARK-15828: Possibly could happen without dynamic al

[jira] [Created] (SPARK-15828) YARN is not aware of Spark's External Shuffle Service

Miles Crawford created SPARK-15828: -- Summary: YARN is not aware of Spark's External Shuffle Service Key: SPARK-15828 URL: https://issues.apache.org/jira/browse/SPARK-15828 Project: Spark Iss

[jira] [Updated] (SPARK-15828) YARN is not aware of Spark's External Shuffle Service

[ https://issues.apache.org/jira/browse/SPARK-15828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miles Crawford updated SPARK-15828: --- Description: When using Spark with dynamic allocation, it is common for all containers on a p

[jira] [Commented] (SPARK-4563) Allow spark driver to bind to different ip then advertise ip

[ https://issues.apache.org/jira/browse/SPARK-4563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15294181#comment-15294181 ] Miles Crawford commented on SPARK-4563: --- Any chance we could boost the priority of t

[jira] [Commented] (SPARK-11293) Spillable collections leak shuffle memory

[ https://issues.apache.org/jira/browse/SPARK-11293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15276723#comment-15276723 ] Miles Crawford commented on SPARK-11293: Also biting us in 1.6.1 - we have to rep

[jira] [Commented] (SPARK-14209) Application failure during preemption.

[ https://issues.apache.org/jira/browse/SPARK-14209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15275780#comment-15275780 ] Miles Crawford commented on SPARK-14209: This issue looks a lot like https://issu

[jira] [Commented] (SPARK-14209) Application failure during preemption.

[ https://issues.apache.org/jira/browse/SPARK-14209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15240040#comment-15240040 ] Miles Crawford commented on SPARK-14209: Just had a very similar error when a hos

[jira] [Created] (SPARK-14561) History Server does not see new logs in S3

Miles Crawford created SPARK-14561: -- Summary: History Server does not see new logs in S3 Key: SPARK-14561 URL: https://issues.apache.org/jira/browse/SPARK-14561 Project: Spark Issue Type: Bu

[jira] [Commented] (SPARK-14561) History Server does not see new logs in S3

[ https://issues.apache.org/jira/browse/SPARK-14561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15237525#comment-15237525 ] Miles Crawford commented on SPARK-14561: Steve Loughran on the user list says: {q

[jira] [Commented] (SPARK-14209) Application failure during preemption.

[ https://issues.apache.org/jira/browse/SPARK-14209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15232598#comment-15232598 ] Miles Crawford commented on SPARK-14209: So, I have launched the exact same simpl

[jira] [Commented] (SPARK-14209) Application failure during preemption.

[ https://issues.apache.org/jira/browse/SPARK-14209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15229394#comment-15229394 ] Miles Crawford commented on SPARK-14209: I have not forgotten this, worked on get

[jira] [Commented] (SPARK-14209) Application failure during preemption.

[ https://issues.apache.org/jira/browse/SPARK-14209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15222412#comment-15222412 ] Miles Crawford commented on SPARK-14209: Yes. I'm still committed to helping you

[jira] [Commented] (SPARK-14209) Application failure during preemption.

[ https://issues.apache.org/jira/browse/SPARK-14209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15222396#comment-15222396 ] Miles Crawford commented on SPARK-14209: yes, we definitely use logback - if you

[jira] [Commented] (SPARK-14209) Application failure during preemption.

[ https://issues.apache.org/jira/browse/SPARK-14209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15222389#comment-15222389 ] Miles Crawford commented on SPARK-14209: Can you be a bit more specific about the

[jira] [Commented] (SPARK-14209) Application failure during preemption.

[ https://issues.apache.org/jira/browse/SPARK-14209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15218572#comment-15218572 ] Miles Crawford commented on SPARK-14209: Here's a tarball of all the container lo

[jira] [Commented] (SPARK-14209) Application failure during preemption.

[ https://issues.apache.org/jira/browse/SPARK-14209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15218356#comment-15218356 ] Miles Crawford commented on SPARK-14209: Anything I can do to help you investigat

[jira] [Commented] (SPARK-14209) Application failure during preemption.

[ https://issues.apache.org/jira/browse/SPARK-14209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15217170#comment-15217170 ] Miles Crawford commented on SPARK-14209: I don't think there's anything custom ab

[jira] [Commented] (SPARK-14209) Application failure during preemption.

[ https://issues.apache.org/jira/browse/SPARK-14209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15217089#comment-15217089 ] Miles Crawford commented on SPARK-14209: Here you go, from the same application r

[jira] [Commented] (SPARK-14209) Application failure during preemption.

[ https://issues.apache.org/jira/browse/SPARK-14209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15217085#comment-15217085 ] Miles Crawford commented on SPARK-14209: Here you go, from the same application r

[jira] [Issue Comment Deleted] (SPARK-14209) Application failure during preemption.

[ https://issues.apache.org/jira/browse/SPARK-14209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miles Crawford updated SPARK-14209: --- Comment: was deleted (was: Here you go, from the same application run: https://www.dropbox.c

[jira] [Commented] (SPARK-14209) Application failure during preemption.

[ https://issues.apache.org/jira/browse/SPARK-14209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15216899#comment-15216899 ] Miles Crawford commented on SPARK-14209: That's fantastic news! let me know if I

[jira] [Commented] (SPARK-14209) Application failure during preemption.

[ https://issues.apache.org/jira/browse/SPARK-14209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15216283#comment-15216283 ] Miles Crawford commented on SPARK-14209: It will be useful to know whether the is

[jira] [Comment Edited] (SPARK-14209) Application failure during preemption.

[ https://issues.apache.org/jira/browse/SPARK-14209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214732#comment-15214732 ] Miles Crawford edited comment on SPARK-14209 at 3/28/16 7:35 PM: --

[jira] [Commented] (SPARK-14209) Application failure during preemption.

[ https://issues.apache.org/jira/browse/SPARK-14209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214732#comment-15214732 ] Miles Crawford commented on SPARK-14209: Yes, I am certainly caching. > Applicat

[jira] [Commented] (SPARK-14209) Application failure during preemption.

[ https://issues.apache.org/jira/browse/SPARK-14209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214674#comment-15214674 ] Miles Crawford commented on SPARK-14209: Here is all the nodemanager logs related

[jira] [Commented] (SPARK-14209) Application failure during preemption.

[ https://issues.apache.org/jira/browse/SPARK-14209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214649#comment-15214649 ] Miles Crawford commented on SPARK-14209: Sorry, i was just showing that they were

[jira] [Commented] (SPARK-14209) Application failure during preemption.

[ https://issues.apache.org/jira/browse/SPARK-14209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214614#comment-15214614 ] Miles Crawford commented on SPARK-14209: spark.shuffle.service.enabled is shown

[jira] [Commented] (SPARK-14209) Application failure during preemption.

[ https://issues.apache.org/jira/browse/SPARK-14209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214595#comment-15214595 ] Miles Crawford commented on SPARK-14209: I am using an external shuffle service,

[jira] [Updated] (SPARK-14209) Application failure during preemption.

[ https://issues.apache.org/jira/browse/SPARK-14209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miles Crawford updated SPARK-14209: --- Description: We have a fair-sharing cluster set up, including the external shuffle service.

[jira] [Created] (SPARK-14209) Application failure during preemption.

Miles Crawford created SPARK-14209: -- Summary: Application failure during preemption. Key: SPARK-14209 URL: https://issues.apache.org/jira/browse/SPARK-14209 Project: Spark Issue Type: Bug