[GitHub] spark issue #21213: [SPARK-24120] Show `Jobs` page when `jobId` is missing
Github user jongyoul commented on the issue: https://github.com/apache/spark/pull/21213 @gengliangwang Thanks. But changing URI format might make any incompatibility with other versions, thus we need to consider other aspects more. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21213: [SPARK-24120] Show `Jobs` page when `jobId` is missing
Github user jongyoul commented on the issue: https://github.com/apache/spark/pull/21213 Thanks for reviewing this PR. Concerning your comments: 1. It makes sense. I'll prolong the time to wait for the current page. 1. Personally, I wondered if redirecting page would be better or not. I also need feedbacks for it. both look good. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21213: [SPARK-24120] Show `Jobs` page when `jobId` is missing
Github user jongyoul commented on the issue: https://github.com/apache/spark/pull/21213 Please review this PR --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21213: [SPARK-24120] Show `Jobs` page when `jobId` is missing
Github user jongyoul commented on the issue: https://github.com/apache/spark/pull/21213 test this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21213: [SPARK-24120] Show `Jobs` page when `jobId` is missing
Github user jongyoul commented on the issue: https://github.com/apache/spark/pull/21213 As I described in issue description, yarn filter couldn't pass query strings when redirecting to yarn application. It's already fixed with the latest version of yarn, but some old versions still have the same problem. This PR helps users not to see 404 page only. instead, it returns `Jobs` page to find proper information --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21213: [SPARK-24120]
GitHub user jongyoul opened a pull request: https://github.com/apache/spark/pull/21213 [SPARK-24120] ## What changes were proposed in this pull request? Change the case which doesn't have jobId as a parameter so that it will redirect to `Jobs` page. ## How was this patch tested? 1. Build with this PR 1. Access http://:4040/jobs/job without any specific `jobId` parameter 1. Check to be redirected to http://:4040/jobs You can merge this pull request into a Git repository by running: $ git pull https://github.com/jongyoul/spark SPARK-24120 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/21213.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #21213 commit 5e7d4a34cf4103300b4fd083ff709866edc8c2d4 Author: Jongyoul Lee Date: 2018-05-02T08:38:48Z Change the case which doesn't have jobId as a parameter so that it will redirect to Jobs page --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20860: [SPARK-23743][SQL] Changed a comparison logic from conta...
Github user jongyoul commented on the issue: https://github.com/apache/spark/pull/20860 @dongjoon-hyun Thanks. I'll do that next time. :-) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20860: [SPARK-23743][SQL] Changed a comparison logic from conta...
Github user jongyoul commented on the issue: https://github.com/apache/spark/pull/20860 @jerryshao Do I have anything to fix that failure? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20860: [SPARK-23743][SQL] Changed a comparison logic fro...
Github user jongyoul commented on a diff in the pull request:
https://github.com/apache/spark/pull/20860#discussion_r176654007
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
---
@@ -179,7 +179,7 @@ private[hive] class IsolatedClientLoader(
val isHadoopClass =
name.startsWith("org.apache.hadoop.") &&
!name.startsWith("org.apache.hadoop.hive.")
-name.contains("slf4j") ||
+name.startsWith("org.slf4j") ||
name.contains("log4j") ||
--- End diff --
Fixed
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20860: [SPARK-23743][SQL] Changed a comparison logic fro...
Github user jongyoul commented on a diff in the pull request:
https://github.com/apache/spark/pull/20860#discussion_r176652812
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
---
@@ -179,7 +179,7 @@ private[hive] class IsolatedClientLoader(
val isHadoopClass =
name.startsWith("org.apache.hadoop.") &&
!name.startsWith("org.apache.hadoop.hive.")
-name.contains("slf4j") ||
+name.startsWith("org.slf4j") ||
name.contains("log4j") ||
--- End diff --
Yes, It looks like a similar issue. I'll add it.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20860: [SPARK-23743][SQL] Changed a comparison logic from conta...
Github user jongyoul commented on the issue: https://github.com/apache/spark/pull/20860 Thank you for reviewing this PR, @dongjoon-hyun @HyukjinKwon --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20860: [SPARK-23743][SQL] Changed a comparison logic fro...
GitHub user jongyoul opened a pull request:
https://github.com/apache/spark/pull/20860
[SPARK-23743][SQL] Changed a comparison logic from containing 'slf4j' to
starting with 'org.slf4j'
## What changes were proposed in this pull request?
isSharedClass returns if some classes can/should be shared or not. It
checks if the classes names have some keywords or start with some names.
Following the logic, it can occur unintended behaviors when a custom package
has `slf4j` inside the package or class name. As I guess, the first intention
seems to figure out the class containing `org.slf4j`. It would be better to
change the comparison logic to `name.startsWith("org.slf4j")`
## How was this patch tested?
This patch should pass all of the current tests and keep all of the current
behaviors. In my case, I'm using ProtobufDeserializer to get a table schema
from hive tables. Thus some Protobuf packages and names have `slf4j` inside.
Without this patch, it cannot be resolved because of ClassCastException from
different classloaders.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jongyoul/spark SPARK-23743
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20860.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20860
commit 192ce305f05d4280c5c35b94a3666d313dab2733
Author: Jongyoul Lee
Date: 2018-03-20T01:45:44Z
Changed a comparison logic from containing 'slf4j' to starting with
'org.slf4j'
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-93898567 @andrewor14 I've fixed what you issue. Please review and merge this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-93872085 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-93731580 I've rebased it from current master at first. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-93206397 @andrewor14 Thanks for overall reviewing. I'll handle what you issue. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user jongyoul commented on a diff in the pull request: https://github.com/apache/spark/pull/5063#discussion_r28381413 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala --- @@ -220,10 +222,9 @@ private[spark] class MesosSchedulerBackend( val mem = getResource(o.getResourcesList, "mem") val cpus = getResource(o.getResourcesList, "cpus") val slaveId = o.getSlaveId.getValue -// TODO(pwendell): Should below be 1 + scheduler.CPUS_PER_TASK? (mem >= MemoryUtils.calculateTotalMemory(sc) && // need at least 1 for executor, 1 for task --- End diff -- @andrewor14 I've missed it. It doesn't need --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-88293005 @sryza Could you please merge this PR for spark 1.3.1? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-87544727 @sryza I hope this will be a final review :-) I'm sorry for making a typo again and again. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86841550 @sryza Review it again, please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user jongyoul commented on a diff in the pull request: https://github.com/apache/spark/pull/5063#discussion_r27275980 --- Diff: docs/running-on-mesos.md --- @@ -211,6 +211,14 @@ See the [configuration page](configuration.html) for information on Spark config + spark.mesos.executor.cores + 1.0 + +The setting, which can be a floating point number, controls the number of cores allocated --- End diff -- Ok, I see. It's not a big problem to edit docs. :-) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86795209 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86357586 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86357482 @sryza Who Do you know help this build issue? I just edited docs only. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86345441 I've rebased from master. Don't mind that I have a mistake to write a commit log which I've worked another issue. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86344972 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-86327232 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-85920174 @tnachen @elyast I've updated it to support fractional number of cores for executors. Please review this PR. I've tested my cluster. My setting is ``` AVA_HOME=/app/jdk HADOOP_HOME=/app/hdfs HADOOP_CONF_DIR=/app/hdfs/conf MESOS_NATIVE_JAVA_LIBRARY=/app/mesos-0.21.0/lib/libmesos.so SPARK_EXECUTOR_MEMORY=5G spark-1.4.0-SNAPSHOT-bin-2.5.0-cdh5.3.0/bin/spark-shell --master mesos://zk://dicc-m001:2181,dicc-m002:2181,dicc-m003:2181/mesos-0.21.0 --conf spark.executor.uri=hdfs:///app/spark/spark-1.4.0-SNAPSHOT-bin-2.5.0-cdh5.3.0.tgz --conf spark.io.compression.codec=lzf --conf spark.task.cpus=5 --conf spark.mesos.executor.cores=0.1 ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-85500449 @elyast @tnachen Do you think `CPUS_PER_TASK` also support fractional value? If it's not, I may be support executorCores as fractional value without huge changes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-85317877 @tnachen I see. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-84769305 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-84769168 I rebased this from master --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-84761674 Jekins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6453][Mesos] Some Mesos*Suite have a di...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5126#issuecomment-84614077 @srowen Yes, All tests passed in the test log. Do you know what the problem is? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6453][Mesos] Some Mesos*Suite have a di...
GitHub user jongyoul opened a pull request: https://github.com/apache/spark/pull/5126 [SPARK-6453][Mesos] Some Mesos*Suite have a different package with their classes - Moved Suites from o.a.s.s.mesos to o.a.s.s.cluster.mesos You can merge this pull request into a Git repository by running: $ git pull https://github.com/jongyoul/spark SPARK-6453 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/5126.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5126 commit 8ab149d21cc666fc26f973ff25686283029d7eba Author: Jongyoul Lee Date: 2015-03-22T09:37:38Z [SPARK-6453][Mesos] Some Mesos*Suite have a different package with their classes - Moved Suites from o.a.s.s.mesos to o.a.s.s.cluster.mesos commit 4f24a3ead1b2bdead48c2308c328cbec0e594e61 Author: Jongyoul Lee Date: 2015-03-22T09:41:06Z [SPARK-6453][Mesos] Some Mesos*Suite have a different package with their classes - Fixed imports orders --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-84258060 @elyast It's enough to explain why we can set this property. @sryza This feature exists only in Mesos mode now. How about setting this property specified in Mesos now, and changing it later if any of cluster will use this property? I don't thinks It's not generic now, but It is needed in Mesos absolutely. I recommend the property name of `spark.mesos.executor.freameworkCores`, or `spark.mesos.executor.cores` looks fine. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6423][Mesos] MemoryUtils should use mem...
Github user jongyoul commented on a diff in the pull request:
https://github.com/apache/spark/pull/5099#discussion_r26843969
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/cluster/mesos/MemoryUtilsSuite.scala
---
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.scheduler.cluster.mesos
+
+import org.apache.spark.{SparkConf, SparkContext}
+import org.mockito.Mockito._
--- End diff --
@srowen Yes, I've known that I should fit import orders, but I missed it.
And I will also update the description.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-83887545 I don't think parameter name is not a critical issue. It looks like that @sryza doesn't want a new configuration parameter used only in a specific cluster, but he wants new configuration to be more general. I, actually, don't know which is the best option. @sryza, Give me your opinion. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6423][Mesos] MemoryUtils should use mem...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5099#issuecomment-83861862 @srowen I made a PR for handling with unintended behaviour issued by #5065. Please review this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6423][Mesos] MemoryUtils should use mem...
GitHub user jongyoul opened a pull request: https://github.com/apache/spark/pull/5099 [SPARK-6423][Mesos] MemoryUtils should use memoryOverhead if it's set - Fixed calculateTotalMemory to use spark.mesos.executor.memoryOverhead - Added testCase You can merge this pull request into a Git repository by running: $ git pull https://github.com/jongyoul/spark SPARK-6423 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/5099.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5099 commit 453c5a2f31d1e474b8d785fde09d53f423d234ee Author: Jongyoul Lee Date: 2015-03-20T02:37:35Z [SPARK-6423][Mesos] MemoryUtils should use memoryOverhead if it's set - Fixed calculateTotalMemory to use spark.mesos.executor.memoryOverhead - Added testCase --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6286][Mesos][minor] Handle missing Meso...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5088#issuecomment-83619131 @srowen Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6286][Mesos][minor] Handle missing Meso...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5088#issuecomment-83440190 jenkins, test this, please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6286][Mesos][minor] Handle missing Meso...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5088#issuecomment-83433890 @srowen Review this PR which is related by #5000 for handling `TASK_ERROR`, please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6286][Mesos][minor] Handle missing Meso...
GitHub user jongyoul opened a pull request: https://github.com/apache/spark/pull/5088 [SPARK-6286][Mesos][minor] Handle missing Mesos case TASK_ERROR - Made TaskState.isFailed for handling TASK_LOST and TASK_ERROR and synchronizing CoarseMesosSchedulerBackend and MesosSchedulerBackend - This is related #5000 You can merge this pull request into a Git repository by running: $ git pull https://github.com/jongyoul/spark SPARK-6286-1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/5088.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5088 commit ac4336ae5988598a9ed663588606c410dd154480 Author: Jongyoul Lee Date: 2015-03-19T09:13:28Z [SPARK-6286][Mesos][minor] Handle missing Mesos case TASK_ERROR - Made TaskState.isFailed for handling TASK_LOST and TASK_ERROR and synchronizing CoarseMesosSchedulerBackend and MesosSchedulerBackend --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6286][minor] Handle missing Mesos case ...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5000#issuecomment-83276749 @dragos Yes, right. I'll make `TaskState.isFailed` method. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6286][minor] Handle missing Mesos case ...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5000#issuecomment-83027537 @srowen Two missing points remains. We should handle `MesosTaskState.TASK_LOST` in `MesosSchedulerBackend` and `CoarseMesosSchedulerBackend`. I will follow up this with another PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-6085 Part. 2 Increase default value for ...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5065#issuecomment-83030138 @srowen No, Mesos doc is not wrong, and I mean that Mesos also uses MB as same as MiB. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-82821343 @sryza What do you think of @tnachen's opinion? For avoiding this confusion, I added comments that this configuration is being used on mesos fine-grained mode. Is it enough? or needed to do something else? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-82812556 @sryza I mean the former one. I agree to your opinion of `spark.executor.frameworkCores`, and I have one more question. Do you think It's good to update configuration.md for adding this new configuration option not to make this specified on Mesos? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-6085 Part. 2 Increase default value for ...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5065#issuecomment-82701925 @srowen I'm sorry not for understanding your opinion fully - I'm not a native english -, do you mean that it's proper to use "MB" instead of "MiB"? I've found Mesos project used that "MB" has meant "MiB". And I'll follow up an unintentional difference in a separate issue, too. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-82694857 @sryza Thanks. Do you think it is reasonable to use `spark.executor.cores` as mesos executor cores? In fine-grained mode on mesos, each task has its own cores to be offered, so It's used only the number of cores of executor itself when executor's status is idle. Simply, if one task is launched on a executor, the total resources of cores is the sum of `spark.executor.cores` and `spark.task.cpus`. I think this make configurations confused. What do you think of it? @elyast Yes, if we use 'spark.executor.cores', we should update docs, too. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user jongyoul commented on a diff in the pull request:
https://github.com/apache/spark/pull/5063#discussion_r26556299
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala
---
@@ -67,6 +67,8 @@ private[spark] class MesosSchedulerBackend(
// The listener bus to publish executor added/removed events.
val listenerBus = sc.listenerBus
+
+ val executorCores = sc.conf.getInt("spark.mesos.executor.cores", 1)
--- End diff --
I thought spark.executor.cores means total number of cores on yarn mode,
and it's confused that one configuration makes different meaning whenever I
choose a different cluster manager. Do you think spark.executor.cores is
reasonable and clear? I also think less configuration options are better. I
just try not to confuse when I configure setting.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-6085 Part. 2 Increase default value for ...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5065#issuecomment-82193193 @srowen Review this PR, please. This is a second part of SPARK-6085 which you committed. I just fixed a small documentation, so I don't make another issue. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-6085 Part. 2 Increase default value for ...
GitHub user jongyoul opened a pull request: https://github.com/apache/spark/pull/5065 SPARK-6085 Part. 2 Increase default value for memory overhead - fixed a description of spark.mesos.executor.memoryOverhead from 7% to 10% - This is a second part of SPARK-6085 You can merge this pull request into a Git repository by running: $ git pull https://github.com/jongyoul/spark SPARK-6085-1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/5065.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5065 commit dbac1c05168bf9c3a0e0e7d10767a5600a6848d1 Author: Jongyoul Lee Date: 2015-03-17T07:57:17Z SPARK-6085 Part. 2 Increase default value for memory overhead - fixed a description of spark.mesos.executor.memoryOverhead from 7% to 10% --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-82189318 @elyast I submitted a PR based on #4170. @tnachen Could you please review my PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5063#issuecomment-82141199 Jenkins, test this, please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6350][Mesos] Make mesosExecutorCores co...
GitHub user jongyoul opened a pull request: https://github.com/apache/spark/pull/5063 [SPARK-6350][Mesos] Make mesosExecutorCores configurable in mesos "fine-grained" mode - Defined executorCores from "spark.mesos.executor.cores" - Changed the amount of mesosExecutor's cores to executorCores. - Added new configuration option on running-on-mesos.md You can merge this pull request into a Git repository by running: $ git pull https://github.com/jongyoul/spark SPARK-6350 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/5063.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5063 commit 4486a0e48e3a92d74685e57a310f07bbe169c23b Author: Jongyoul Lee Date: 2015-03-17T06:21:34Z [SPARK-6350][Mesos] Make mesosExecutorCores configurable in mesos "fine-grained" mode - Defined executorCores from "spark.mesos.executor.cores" - Changed the amount of mesosExecutor's cores to executorCores. - Added new configuration option on running-on-mesos.md --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-82031855 @tnachen @elyast I made a new issue about configuring mesos executor cores. https://issues.apache.org/jira/browse/SPARK-6350 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-80798498 @elyast Thanks for interesting this PR, which was about resources of cores and memory. I misunderstood how mesos works specially in memory side, so I closed this PR. However, I agree with you that executor cores are sometime too much. @tnachen we cannot adjust memory issue, but, changing executor cores is meaningful and should fix that executor has same cores with CPUS_PER_TASK initially. See TODO(pwendell) in MesosSchedulerBackend.scala. I want to fix this TODO to extract a configuration parameter. What do you think of it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3619] Part 2. Upgrade to Mesos 0.21 to ...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/4361#issuecomment-80797514 @srowen Master and Branch-1.3 have a dependency of Mesos 0.21.0. Is it ok to merge it into branch-1.3? And MESOS-1688 is resolved by Mesos 0.21.0. So Spark isn't influenced by that issue anymore. So I removed that part. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3619] Part 2. Upgrade to Mesos 0.21 to ...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/4361#issuecomment-78812902 @pwendell @srowen Review this PR, please. This is about SPARK-3619 which is already completed but has some missing documentation about ENV name. From mesos-0.21.0, MESOS_NATIVE_LIBRARY is deprecated and MESOS_NATIVE_JAVA_LIBRARY is introduced newly. we should fix a configuration on this. @tnachen completed reviews on this PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6286][minor] Handle missing Mesos case ...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/5000#issuecomment-78759592 @dragos You will find more codes which you could fix in *SchedulerBackend which handles MesosTaskState directly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3619] Part 2. Upgrade to Mesos 0.21 to ...
Github user jongyoul commented on a diff in the pull request: https://github.com/apache/spark/pull/4361#discussion_r25142832 --- Diff: conf/spark-env.sh.template --- @@ -15,7 +15,7 @@ # - SPARK_PUBLIC_DNS, to set the public DNS name of the driver program # - SPARK_CLASSPATH, default classpath entries to append # - SPARK_LOCAL_DIRS, storage directories to use on this node for shuffle and RDD data -# - MESOS_NATIVE_LIBRARY, to point to your libmesos.so if you use Mesos +# - MESOS_NATIVE_JAVA_LIBRARY, to point to your libmesos.so if you use Mesos --- End diff -- @tnachen Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3619] Part 2. Upgrade to Mesos 0.21 to ...
Github user jongyoul commented on a diff in the pull request: https://github.com/apache/spark/pull/4361#discussion_r25141239 --- Diff: conf/spark-env.sh.template --- @@ -15,7 +15,7 @@ # - SPARK_PUBLIC_DNS, to set the public DNS name of the driver program # - SPARK_CLASSPATH, default classpath entries to append # - SPARK_LOCAL_DIRS, storage directories to use on this node for shuffle and RDD data -# - MESOS_NATIVE_LIBRARY, to point to your libmesos.so if you use Mesos --- End diff -- @tnachen I've found only this setting for mesos. Do you think there is another comment? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3619] Part 2. Upgrade to Mesos 0.21 to ...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/4361#issuecomment-74405146 @tnachen I've already changed docs/running_on_mesos.md. Could you please tell me any other docs for changing description? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3619] Part 2. Upgrade to Mesos 0.21 to ...
GitHub user jongyoul opened a pull request: https://github.com/apache/spark/pull/4361 [SPARK-3619] Part 2. Upgrade to Mesos 0.21 to work around MESOS-1688 - MESOS_NATIVE_LIBRARY become deprecated - Chagned MESOS_NATIVE_LIBRARY to MESOS_NATIVE_JAVA_LIBRARY You can merge this pull request into a Git repository by running: $ git pull https://github.com/jongyoul/spark SPARK-3619-1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/4361.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4361 commit 0dace7b20cbe48e931d2d657ff4807ce54df0665 Author: Jongyoul Lee Date: 2015-02-04T08:44:32Z [SPARK-3619] Upgrade to Mesos 0.21 to work around MESOS-1688 - MESOS_NATIVE_LIBRARY become deprecated - Chagned MESOS_NATIVE_LIBRARY to MESOS_NATIVE_JAVA_LIBRARY --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5198][Mesos] Change executorId more uni...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/3994#issuecomment-71776175 I'll also close this PR. I've misunderstood mesos #4170 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5198][Mesos] Change executorId more uni...
Github user jongyoul closed the pull request at: https://github.com/apache/spark/pull/3994 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user jongyoul closed the pull request at: https://github.com/apache/spark/pull/4170 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-71776099 I'll close this PR. It's wrong approach. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-71775404 @tnachen @mateiz So sorry for taking up a lot of time. I've found that only one executor as a process runs at any time, and I understand executor can have multiple tasks at the same time. I've believed each executor is launched separately when driver launchTasks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-71773877 @tnachen Yes, I fully understand reusing executor while a framework is alive. However, we launch two task on a same executor? What you've answered is they are launched at the same time, isn't it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-71773477 I believed that when we launch mesos driver launchTasks, container run the command `bin/spark-class` everytime running task. And in my qna email for mesos, @tnachen answers that one container run multiple command simultaneously. And my some tests show two tasks runs simutaneously because they write a same log file at the same time. And my digging codes results no limit to launch task on a mesos container. However, @mateiz told me that one executor only runs a single JVM and launch a single task at any time. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-71772544 I don't know the behaviour in coarse-grained mode, but in fine-grained mode, we use multiple JVM for running tasks. we run spark-class by launcher. This means we launch JVM by running per task. Am I wrong? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-71770002 Sorry, I've shaw you my configuration. my configuraion is 5G for SPARK_EXECUTOR_MEMORY and 5 for spark.task.cpus. In my screenshot, we launch two tasks on the same machine. Don't you think It's good to offer task memory twice? My PR gives correct resource management information to mesos' master. For CPUs, I don't know proper value of executor cpus, but not CPUS_TASK_CPUS. Recommend this value, please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-71767996 @mateiz That's a sample screenshot.  --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-71765614 Thus, I believe that executor has executor cores and executor memory setting on ExecutorInfo, and task has its own cores and memories setting on TaskInfo while launching per task. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-71765405 @mateiz We agree with one executor and multi task is intended behaviour. In this situation, MesosScheduler offers CPUS_PER_TASK resources to executor when we launch separate task. If we launch two tasks, we offers 3 * CPUS_PER_TASK(= 1 for executor and 2 for tasks) for running only two tasks. @pwendell thinks that's too much resources and It's enough for executor to have one cores. In my PR, I enable to set executor's cores. In memory, we just offered memory to executor only. If we launch two tasks again, we offers 1 * calculateTotalMemory(sc) to all tasks. I think we offer one executor memory and two task memories. I agree that executor uses memory by itself, but we fix amount of those value --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-71411958 /cc @mateiz Could you please review this PR which is about offering resources to executor and task. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5058] Part 2. Typos and broken URL
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/4172#issuecomment-71161988 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5058] Part 2. Typos and broken URL
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/4172#issuecomment-71158869 /cc @tdas This PR is a second part of [SPARK-5058](https://issues.apache.org/jira/browse/SPARK-5058) which you already reviewed and resolved. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5058] Part 2. Typos and broken URL
GitHub user jongyoul opened a pull request: https://github.com/apache/spark/pull/4172 [SPARK-5058] Part 2. Typos and broken URL - Also fixed java link You can merge this pull request into a Git repository by running: $ git pull https://github.com/jongyoul/spark SPARK-FIXDOC Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/4172.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4172 commit 6be03e54416cf3046382dba678a5ab3a2c3e2598 Author: Jongyoul Lee Date: 2015-01-23T07:30:54Z [SPARK-5058] Part 2. Typos and broken URL - Also fixed java link --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user jongyoul commented on a diff in the pull request: https://github.com/apache/spark/pull/4170#discussion_r23433986 --- Diff: docs/configuration.md --- @@ -341,6 +341,13 @@ Apart from these, the following properties are also available, and may be useful `spark.mesos.executor.memoryOverhead` or 7% of `spark.executor.memory`. + + spark.mesos.executor.cpus + 1 + +This value is the amount of cores so that executor is running itself. --- End diff -- I agree. I'm also confused between cpus and cores. I think `cores` is more proper term. Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/4170#issuecomment-71150639 /cc @tnachen @pwendell This PR is about @pwendell 's todo. Review this, please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5376][Mesos] MesosExecutor should have ...
GitHub user jongyoul opened a pull request: https://github.com/apache/spark/pull/4170 [SPARK-5376][Mesos] MesosExecutor should have correct resources - Divided task and executor resources - Added `spark.mesos.executor.cpus` and fixed docs You can merge this pull request into a Git repository by running: $ git pull https://github.com/jongyoul/spark SPARK-5376 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/4170.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4170 commit 71703c808027b54970367af7d26f0d151b6a8003 Author: Jongyoul Lee Date: 2015-01-23T05:46:25Z [SPARK-5376][Mesos] MesosExecutor should have correct resources - Divided task and executor resources - Added `spark.mesos.executor.cpus` and fixed docs --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5333][Mesos] MesosTaskLaunchData occurs...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/4119#issuecomment-70639628 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5333][Mesos] MesosTaskLaunchData occurs...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/4119#issuecomment-70633473 I've added license information. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5333][Mesos] MesosTaskLaunchData occurs...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/4119#issuecomment-70633023 retest this, please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5333][Mesos] MesosTaskLaunchData occurs...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/4119#issuecomment-70632731 /cc @JoshRosen This is related by SPARK-4104 #3849. Review this PR, please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5333][Mesos] MesosTaskLaunchData occurs...
GitHub user jongyoul opened a pull request: https://github.com/apache/spark/pull/4119 [SPARK-5333][Mesos] MesosTaskLaunchData occurs BufferUnderflowException - Rewind ByteBuffer before making ByteString You can merge this pull request into a Git repository by running: $ git pull https://github.com/jongyoul/spark SPARK-5333 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/4119.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4119 commit b7f5517835e0fb6b9ffb89456fa4567bc8e0edc9 Author: Jongyoul Lee Date: 2015-01-20T10:18:25Z [SPARK-5333][Mesos] MesosTaskLaunchData occurs BufferUnderflowException - Rewind ByteBuffer before making ByteString --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5198][Mesos] Change executorId more uni...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/3994#issuecomment-70600230 @tnachen Ok, I see. It happened when executor couldn't get launched, doesn't it? I'll change that setting first. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5198][Mesos] Change executorId more uni...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/3994#issuecomment-70597055 @tnachen In my case - above logs -, task 34 and 63 are assigned to same executor and also same container on same node. Task 34 occurs error about registration timeout, and task 34 is terminated and task 63, which is on queue, is also exited because mesos_containerizer destroys that container which contain task 34 and 63. I think this is a bug. As you told, an executor is responsible to manage tasks and executor can terminate all tasks which are running or queueing. I think only one task is terminated if the task fails. I think a task must not be influenced by any other task. What do you think of this situation? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5198][Mesos] Change executorId more uni...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/3994#issuecomment-70594275 @tnachen Yes, I've also found what you told about timeout. I'll check it again by changing that value. But changing executorId is needed. If there are two tasks running on a same executor id - same node -, one task become failed, all tasks become failed. Don't you think this is a problem? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5088] Use spark-class for running execu...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/3897#issuecomment-70455615 @mateiz I've rebased this PR and finished tests successfully. Merge this, please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5088] Use spark-class for running execu...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/3897#issuecomment-70445944 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5088] Use spark-class for running execu...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/3897#issuecomment-70444764 Rebase is not finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5198][Mesos] Change executorId more uni...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/3994#issuecomment-70359713 @tnachen And slave's logs around task 34, 63. It looks like that if any task occurs error while running, the executor running that task is terminated. Check this, please. ``` I0117 17:21:43.678827 41388 slave.cpp:625] Got assigned task 34 for framework 20150117-171023-3391097354-60030-7325-0004 I0117 17:21:43.679612 41388 slave.cpp:734] Launching task 34 for framework 20150117-171023-3391097354-60030-7325-0004 I0117 17:21:43.721297 41388 slave.cpp:844] Queuing task '34' for executor 20141110-112437-3374320138-60030-57359-44 of framework '20150117-171023-3391097354-60030-7325-0004 I0117 17:21:43.775977 41388 slave.cpp:358] Successfully attached file '/data03/mesos/slaves/20141110-112437-3374320138-60030-57359-44/frameworks/20150117-171023-3391097354-60030-7325-0004/executors/20141110-112437-3374320138-60030-57359-44/runs/3fdbdd09-98cd-4197-954f-d95d9b3b4aee' I0117 17:21:43.721451 41386 mesos_containerizer.cpp:407] Starting container '3fdbdd09-98cd-4197-954f-d95d9b3b4aee' for executor '20141110-112437-3374320138-60030-57359-44' of framework '20150117-171023-3391097354-60030-7325-0004' I0117 17:21:43.777179 41386 mesos_containerizer.cpp:528] Fetching URIs for container '3fdbdd09-98cd-4197-954f-d95d9b3b4aee' using command '/usr/bin/env MESOS_EXECUTOR_URIS="hdfs:///app/spark/spark-1.3.0-SNAPSHOT-bin-2.3.0-cdh5.0.1.tgz+0X" MESOS_WORK_DIRECTORY=/data03/mesos/slaves/20141110-112437-3374320138-60030-57359-44/frameworks/20150117-171023-3391097354-60030-7325-0004/executors/20141110-112437-3374320138-60030-57359-44/runs/3fdbdd09-98cd-4197-954f-d95d9b3b4aee HADOOP_HOME=/app/hdfs/ /app/mesos-0.18.1/libexec/mesos/mesos-fetcher' I0117 17:22:28.863304 41374 slave.cpp:2523] Current usage 44.85%. Max allowed age: 3.160841566048842days I0117 17:22:38.472086 41384 slave.cpp:625] Got assigned task 63 for framework 20150117-171023-3391097354-60030-7325-0004 I0117 17:22:38.472584 41384 slave.cpp:734] Launching task 63 for framework 20150117-171023-3391097354-60030-7325-0004 I0117 17:22:38.472801 41384 slave.cpp:844] Queuing task '63' for executor 20141110-112437-3374320138-60030-57359-44 of framework '20150117-171023-3391097354-60030-7325-0004 I0117 17:22:43.721726 41370 slave.cpp:2475] Terminating executor 20141110-112437-3374320138-60030-57359-44 of framework 20150117-171023-3391097354-60030-7325-0004 because it did not register within 1mins I0117 17:22:43.722038 41378 mesos_containerizer.cpp:818] Destroying container '3fdbdd09-98cd-4197-954f-d95d9b3b4aee' I0117 17:22:43.722295 41378 slave.cpp:2052] Executor '20141110-112437-3374320138-60030-57359-44' of framework 20150117-171023-3391097354-60030-7325-0004 has terminated with unknown status E0117 17:22:43.722744 41376 slave.cpp:2332] Failed to unmonitor container for executor 20141110-112437-3374320138-60030-57359-44 of framework 20150117-171023-3391097354-60030-7325-0004: Not monitored I0117 17:22:43.737566 41378 slave.cpp:1669] Handling status update TASK_LOST (UUID: cc571b34-e161-4ec9-bcf1-033bd967209f) for task 34 of framework 20150117-171023-3391097354-60030-7325-0004 from @0.0.0.0:0 I0117 17:22:43.737829 41378 slave.cpp:3142] Terminating task 34 I0117 17:22:43.738701 41372 status_update_manager.cpp:315] Received status update TASK_LOST (UUID: cc571b34-e161-4ec9-bcf1-033bd967209f) for task 34 of framework 20150117-171023-3391097354-60030-7325-0004 I0117 17:22:43.739341 41378 slave.cpp:1669] Handling status update TASK_LOST (UUID: f198e879-a762-4cce-97ff-5261cb4ff820) for task 63 of framework 20150117-171023-3391097354-60030-7325-0004 from @0.0.0.0:0 I0117 17:22:43.739398 41372 status_update_manager.cpp:494] Creating StatusUpdate stream for task 34 of framework 20150117-171023-3391097354-60030-7325-0004 I0117 17:22:43.739542 41378 slave.cpp:3142] Terminating task 63 I0117 17:22:43.739869 41372 status_update_manager.cpp:368] Forwarding status update TASK_LOST (UUID: cc571b34-e161-4ec9-bcf1-033bd967209f) for task 34 of framework 20150117-171023-3391097354-60030-7325-0004 to master@10.10.32.202:60030 I0117 17:22:43.740393 41372 status_update_manager.cpp:315] Received status update TASK_LOST (UUID: f198e879-a762-4cce-97ff-5261cb4ff820) for task 63 of framework 20150117-171023-3391097354-60030-7325-0004 I0117 17:22:43.740411 41384 slave.cpp:1789] Status update manager successfully handled status update TASK_LOST (UUID: cc571b34-e161-4ec9-bcf1-033bd967209f) for task 34 of framework 20150117-171023-3391097354-60030-7325-0004 I0117 17:22:43.740573 41372 status_update_manager.cpp:494] Creating StatusUpdate stream for task 63 of framework 20150117-171023-3391097354-60030-7325-0004
[GitHub] spark pull request: [SPARK-5198][Mesos] Change executorId more uni...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/3994#issuecomment-70359541 @tnachen - Slave page  - Sandbox page  - stderr  --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5198][Mesos] Change executorId more uni...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/3994#issuecomment-70358767 @mateiz I think I don't know fine-grained mode how you intend to behave exactly. What help me to understand more? I don't know how multi executor break spark's intended behaviour. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5198][Mesos] Change executorId more uni...
Github user jongyoul commented on the pull request: https://github.com/apache/spark/pull/3994#issuecomment-70358682 @tnachen Yes, I used spark executor uri like `hdfs:///app/spark/test-spark.tgz`. I'll upload test log as soon as possible. At last, `GLOG_v=1` is a mesos slave setting, isn't it? And how can I find a setting about timeout that you said. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
