[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/5823


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[tgravescs]

Github user tgravescs commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98228814
  
changed to pass exception to logWarning on commit.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[vanzin]

Github user vanzin commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98219293
  
Yeah, they're all nits, with the exception of the exception (which you 
could easily add when pushing if you feel like it).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[tgravescs]

Github user tgravescs commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98218341
  
@vanzin thanks for taking a look. looks like those comments are nits, the 
exception would be nice but rather then pushing this out further and waiting 
for jenkins again, I think I'm going to commit this as is and we can file 
follow up to fix them up.  Let me know asap if you object.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[vanzin]

Github user vanzin commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98215083
  
Just gave the code a cursory view since the logic has been reviewed ad 
nauseum. LGTM.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[vanzin]

Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/5823#discussion_r29527629
  
--- Diff: 
yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorDelegationTokenUpdater.scala
 ---
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.deploy.yarn
+
+import java.util.concurrent.{Executors, TimeUnit}
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileSystem, Path}
+import org.apache.hadoop.security.{Credentials, UserGroupInformation}
+
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.{Logging, SparkConf}
+import org.apache.spark.util.{ThreadUtils, Utils}
+
+import scala.util.control.NonFatal
--- End diff --

super nit: out of order


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[vanzin]

Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/5823#discussion_r29527470
  
--- Diff: 
yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala ---
@@ -189,6 +200,7 @@ private[spark] class ApplicationMaster(
   logDebug("shutting down user thread")
   userClassThread.interrupt()
 }
+if (!inShutdown) delegationTokenRenewerOption.foreach(_.stop())
--- End diff --

super nit:

if (...) {
}



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[vanzin]

Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/5823#discussion_r29527345
  
--- Diff: 
yarn/src/main/scala/org/apache/spark/deploy/yarn/AMDelegationTokenRenewer.scala 
---
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.deploy.yarn
+
+import java.security.PrivilegedExceptionAction
+import java.util.concurrent.{Executors, TimeUnit}
+
+import scala.language.postfixOps
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileSystem, Path}
+import org.apache.hadoop.security.UserGroupInformation
+import org.apache.spark.deploy.SparkHadoopUtil
+
+import org.apache.spark.{Logging, SparkConf}
+import org.apache.spark.util.ThreadUtils
+
+/*
+ * The following methods are primarily meant to make sure long-running 
apps like Spark
+ * Streaming apps can run without interruption while writing to secure 
HDFS. The
+ * scheduleLoginFromKeytab method is called on the driver when the
+ * CoarseGrainedScheduledBackend starts up. This method wakes up a thread 
that logs into the KDC
+ * once 75% of the renewal interval of the original delegation tokens used 
for the container
+ * has elapsed. It then creates new delegation tokens and writes them to 
HDFS in a
+ * pre-specified location - the prefix of which is specified in the 
sparkConf by
+ * spark.yarn.credentials.file (so the file(s) would be named c-1, c-2 
etc. - each update goes
+ * to a new file, with a monotonically increasing suffix). After this, the 
credentials are
+ * updated once 75% of the new tokens renewal interval has elapsed.
+ *
+ * On the executor side, the updateCredentialsIfRequired method is called 
once 80% of the
+ * validity of the original tokens has elapsed. At that time the executor 
finds the
+ * credentials file with the latest timestamp and checks if it has read 
those credentials
+ * before (by keeping track of the suffix of the last file it read). If a 
new file has
+ * appeared, it will read the credentials and update the currently running 
UGI with it. This
+ * process happens again once 80% of the validity of this has expired.
+ */
+private[yarn] class AMDelegationTokenRenewer(
+sparkConf: SparkConf,
+hadoopConf: Configuration) extends Logging {
+
+  private var lastCredentialsFileSuffix = 0
+
+  private val delegationTokenRenewer =
+Executors.newSingleThreadScheduledExecutor(
+  ThreadUtils.namedThreadFactory("Delegation Token Refresh Thread"))
+
+  private val hadoopUtil = YarnSparkHadoopUtil.get
+
+  private val daysToKeepFiles = 
sparkConf.getInt("spark.yarn.credentials.file.retention.days", 5)
+  private val numFilesToKeep = 
sparkConf.getInt("spark.yarn.credentials.file.retention.count", 5)
+
+  /**
+   * Schedule a login from the keytab and principal set using the 
--principal and --keytab
+   * arguments to spark-submit. This login happens only when the 
credentials of the current user
+   * are about to expire. This method reads spark.yarn.principal and 
spark.yarn.keytab from
+   * SparkConf to do the login. This method is a no-op in non-YARN mode.
+   *
+   */
+  private[spark] def scheduleLoginFromKeytab(): Unit = {
+val principal = sparkConf.get("spark.yarn.principal")
+val keytab = sparkConf.get("spark.yarn.keytab")
+
+/**
+ * Schedule re-login and creation of new tokens. If tokens have 
already expired, this method
+ * will synchronously create new ones.
+ */
+def scheduleRenewal(runnable: Runnable): Unit = {
+  val credentials = UserGroupInformation.getCurrentUser.getCredentials
+  val renewalInterval = hadoopUtil.getTimeFromNowToRenewal(sparkConf, 
0.75, credentials)
+  // Run now!
+  if (renewalInterval <= 0) {
+logInfo("HDFS tokens have expired, creating new tokens now.")
+runnable.run()
+  } else {
+

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[vanzin]

Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/5823#discussion_r29527233
  
--- Diff: 
yarn/src/main/scala/org/apache/spark/deploy/yarn/AMDelegationTokenRenewer.scala 
---
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.deploy.yarn
+
+import java.security.PrivilegedExceptionAction
+import java.util.concurrent.{Executors, TimeUnit}
+
+import scala.language.postfixOps
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileSystem, Path}
+import org.apache.hadoop.security.UserGroupInformation
+import org.apache.spark.deploy.SparkHadoopUtil
+
+import org.apache.spark.{Logging, SparkConf}
+import org.apache.spark.util.ThreadUtils
+
+/*
+ * The following methods are primarily meant to make sure long-running 
apps like Spark
+ * Streaming apps can run without interruption while writing to secure 
HDFS. The
+ * scheduleLoginFromKeytab method is called on the driver when the
+ * CoarseGrainedScheduledBackend starts up. This method wakes up a thread 
that logs into the KDC
+ * once 75% of the renewal interval of the original delegation tokens used 
for the container
+ * has elapsed. It then creates new delegation tokens and writes them to 
HDFS in a
+ * pre-specified location - the prefix of which is specified in the 
sparkConf by
+ * spark.yarn.credentials.file (so the file(s) would be named c-1, c-2 
etc. - each update goes
+ * to a new file, with a monotonically increasing suffix). After this, the 
credentials are
+ * updated once 75% of the new tokens renewal interval has elapsed.
+ *
+ * On the executor side, the updateCredentialsIfRequired method is called 
once 80% of the
+ * validity of the original tokens has elapsed. At that time the executor 
finds the
+ * credentials file with the latest timestamp and checks if it has read 
those credentials
+ * before (by keeping track of the suffix of the last file it read). If a 
new file has
+ * appeared, it will read the credentials and update the currently running 
UGI with it. This
+ * process happens again once 80% of the validity of this has expired.
+ */
+private[yarn] class AMDelegationTokenRenewer(
+sparkConf: SparkConf,
+hadoopConf: Configuration) extends Logging {
+
+  private var lastCredentialsFileSuffix = 0
+
+  private val delegationTokenRenewer =
+Executors.newSingleThreadScheduledExecutor(
+  ThreadUtils.namedThreadFactory("Delegation Token Refresh Thread"))
+
+  private val hadoopUtil = YarnSparkHadoopUtil.get
+
+  private val daysToKeepFiles = 
sparkConf.getInt("spark.yarn.credentials.file.retention.days", 5)
+  private val numFilesToKeep = 
sparkConf.getInt("spark.yarn.credentials.file.retention.count", 5)
+
+  /**
+   * Schedule a login from the keytab and principal set using the 
--principal and --keytab
+   * arguments to spark-submit. This login happens only when the 
credentials of the current user
+   * are about to expire. This method reads spark.yarn.principal and 
spark.yarn.keytab from
+   * SparkConf to do the login. This method is a no-op in non-YARN mode.
+   *
+   */
+  private[spark] def scheduleLoginFromKeytab(): Unit = {
+val principal = sparkConf.get("spark.yarn.principal")
+val keytab = sparkConf.get("spark.yarn.keytab")
+
+/**
+ * Schedule re-login and creation of new tokens. If tokens have 
already expired, this method
+ * will synchronously create new ones.
+ */
+def scheduleRenewal(runnable: Runnable): Unit = {
+  val credentials = UserGroupInformation.getCurrentUser.getCredentials
+  val renewalInterval = hadoopUtil.getTimeFromNowToRenewal(sparkConf, 
0.75, credentials)
+  // Run now!
+  if (renewalInterval <= 0) {
+logInfo("HDFS tokens have expired, creating new tokens now.")
+runnable.run()
+  } else {
+

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[vanzin]

Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/5823#discussion_r29527137
  
--- Diff: 
yarn/src/main/scala/org/apache/spark/deploy/yarn/AMDelegationTokenRenewer.scala 
---
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.deploy.yarn
+
+import java.security.PrivilegedExceptionAction
+import java.util.concurrent.{Executors, TimeUnit}
+
+import scala.language.postfixOps
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileSystem, Path}
+import org.apache.hadoop.security.UserGroupInformation
+import org.apache.spark.deploy.SparkHadoopUtil
+
+import org.apache.spark.{Logging, SparkConf}
+import org.apache.spark.util.ThreadUtils
+
+/*
+ * The following methods are primarily meant to make sure long-running 
apps like Spark
+ * Streaming apps can run without interruption while writing to secure 
HDFS. The
+ * scheduleLoginFromKeytab method is called on the driver when the
+ * CoarseGrainedScheduledBackend starts up. This method wakes up a thread 
that logs into the KDC
+ * once 75% of the renewal interval of the original delegation tokens used 
for the container
+ * has elapsed. It then creates new delegation tokens and writes them to 
HDFS in a
+ * pre-specified location - the prefix of which is specified in the 
sparkConf by
+ * spark.yarn.credentials.file (so the file(s) would be named c-1, c-2 
etc. - each update goes
+ * to a new file, with a monotonically increasing suffix). After this, the 
credentials are
+ * updated once 75% of the new tokens renewal interval has elapsed.
+ *
+ * On the executor side, the updateCredentialsIfRequired method is called 
once 80% of the
+ * validity of the original tokens has elapsed. At that time the executor 
finds the
+ * credentials file with the latest timestamp and checks if it has read 
those credentials
+ * before (by keeping track of the suffix of the last file it read). If a 
new file has
+ * appeared, it will read the credentials and update the currently running 
UGI with it. This
+ * process happens again once 80% of the validity of this has expired.
+ */
+private[yarn] class AMDelegationTokenRenewer(
+sparkConf: SparkConf,
+hadoopConf: Configuration) extends Logging {
+
+  private var lastCredentialsFileSuffix = 0
+
+  private val delegationTokenRenewer =
+Executors.newSingleThreadScheduledExecutor(
+  ThreadUtils.namedThreadFactory("Delegation Token Refresh Thread"))
+
+  private val hadoopUtil = YarnSparkHadoopUtil.get
+
+  private val daysToKeepFiles = 
sparkConf.getInt("spark.yarn.credentials.file.retention.days", 5)
+  private val numFilesToKeep = 
sparkConf.getInt("spark.yarn.credentials.file.retention.count", 5)
+
+  /**
+   * Schedule a login from the keytab and principal set using the 
--principal and --keytab
+   * arguments to spark-submit. This login happens only when the 
credentials of the current user
+   * are about to expire. This method reads spark.yarn.principal and 
spark.yarn.keytab from
+   * SparkConf to do the login. This method is a no-op in non-YARN mode.
+   *
+   */
+  private[spark] def scheduleLoginFromKeytab(): Unit = {
+val principal = sparkConf.get("spark.yarn.principal")
+val keytab = sparkConf.get("spark.yarn.keytab")
+
+/**
+ * Schedule re-login and creation of new tokens. If tokens have 
already expired, this method
+ * will synchronously create new ones.
+ */
+def scheduleRenewal(runnable: Runnable): Unit = {
+  val credentials = UserGroupInformation.getCurrentUser.getCredentials
+  val renewalInterval = hadoopUtil.getTimeFromNowToRenewal(sparkConf, 
0.75, credentials)
+  // Run now!
+  if (renewalInterval <= 0) {
+logInfo("HDFS tokens have expired, creating new tokens now.")
+runnable.run()
+  } else {
+

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[vanzin]

Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/5823#discussion_r29527031
  
--- Diff: 
yarn/src/main/scala/org/apache/spark/deploy/yarn/AMDelegationTokenRenewer.scala 
---
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.deploy.yarn
+
+import java.security.PrivilegedExceptionAction
+import java.util.concurrent.{Executors, TimeUnit}
+
+import scala.language.postfixOps
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileSystem, Path}
+import org.apache.hadoop.security.UserGroupInformation
+import org.apache.spark.deploy.SparkHadoopUtil
--- End diff --

super nit: in wrong group


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[vanzin]

Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/5823#discussion_r29526864
  
--- Diff: 
core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala
 ---
@@ -20,6 +20,8 @@ package org.apache.spark.executor
 import java.net.URL
 import java.nio.ByteBuffer
 
+import org.apache.hadoop.conf.Configuration
--- End diff --

super nit: out of order


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98212621
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98212622
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31577/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98212605
  
  [Test build #31577 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31577/consoleFull)
 for   PR 5823 at commit 
[`3c86bba`](https://github.com/apache/spark/commit/3c86bbac8e0116575e5c632c91e77e4d2687dfc1).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[tgravescs]

Github user tgravescs commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98211536
  
+1 pending screwdriver


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98186407
  
  [Test build #31577 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31577/consoleFull)
 for   PR 5823 at commit 
[`3c86bba`](https://github.com/apache/spark/commit/3c86bbac8e0116575e5c632c91e77e4d2687dfc1).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98186110
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98186139
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[harishreedharan]

Github user harishreedharan commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98181074
  
Fixing this and the other imports

Tom Graves wrote:
>
> import scala.language.postfixOps



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[tgravescs]

Github user tgravescs commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98180910
  
its used elsewhere but I think everywhere its used it does:import 
scala.language.postfixOps

Can you simply add that to the imports so we get rid of the compiler 
warning.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[harishreedharan]

Github user harishreedharan commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98168394
  
That is the post fix operator warning. I have seen it used elsewhere in
spark but I can switch it out if you want. It trill is not a blocker I
think.

On Friday, May 1, 2015, Tom Graves  wrote:

> seeing the following warning while compiling:
>
> [warn]
> 
/home/tgraves/tgravescs_spark/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala:247:
> postfix operator hours should be enabled
> [warn] by making the implicit value scala.language.postfixOps visible.
> [warn] This can be achieved by adding the import clause 'import
> scala.language.postfixOps'
> [warn] or by setting the compiler option -language:postfixOps.
> [warn] See the Scala docs for value scala.language.postfixOps for a
> discussion
> [warn] why the feature should be explicitly enabled.
> [warn] sparkConf.getLong("spark.yarn.token.renewal.interval", (24
> hours).toMillis)
>
> —
> Reply to this email directly or view it on GitHub
> .
>


-- 

Thanks,
Hari



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[tgravescs]

Github user tgravescs commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98144960
  
seeing the following warning while compiling:

[warn] 
/home/tgraves/tgravescs_spark/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala:247:
 postfix operator hours should be enabled
[warn] by making the implicit value scala.language.postfixOps visible.
[warn] This can be achieved by adding the import clause 'import 
scala.language.postfixOps'
[warn] or by setting the compiler option -language:postfixOps.
[warn] See the Scala docs for value scala.language.postfixOps for a 
discussion
[warn] why the feature should be explicitly enabled.
[warn]   sparkConf.getLong("spark.yarn.token.renewal.interval", (24 
hours).toMillis)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[tgravescs]

Github user tgravescs commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98142852
  
very minor comments, not even really worth holding this up.  I'm going to 
run some tests and builds manually and if everything looks good commit it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[tgravescs]

Github user tgravescs commented on a diff in the pull request:

https://github.com/apache/spark/pull/5823#discussion_r29503120
  
--- Diff: 
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtil.scala ---
@@ -17,25 +17,26 @@
 
 package org.apache.spark.deploy.yarn
 
-import java.io.File
+import java.io._
 import java.util.regex.Matcher
 import java.util.regex.Pattern
 
 import scala.collection.mutable.HashMap
 import scala.util.Try
 
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs._
--- End diff --

same here specify each one please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[tgravescs]

Github user tgravescs commented on a diff in the pull request:

https://github.com/apache/spark/pull/5823#discussion_r29503113
  
--- Diff: 
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtil.scala ---
@@ -17,25 +17,26 @@
 
 package org.apache.spark.deploy.yarn
 
-import java.io.File
+import java.io._
--- End diff --

can you specify each one we are using instead of _


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[tgravescs]

Github user tgravescs commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98139247
  
looks like the addCredentials method was added in hadoop 2.0.2-alpha so we 
should be good to use it anywhere in yarn build since we only support 2.2 and 
on.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[harishreedharan]

Github user harishreedharan commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98084772
  
Verified that executor and AM failures before and after expiry still work 
fine.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98049198
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98049199
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31510/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98049185
  
  [Test build #31510 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31510/consoleFull)
 for   PR 5823 at commit 
[`4d04301`](https://github.com/apache/spark/commit/4d04301f04566b7e38f9ba4bc3bf5d591d30c224).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.
 * This patch does not change any dependencies.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98048164
  
  [Test build #31507 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31507/consoleFull)
 for   PR 5823 at commit 
[`b5e7a72`](https://github.com/apache/spark/commit/b5e7a728340d44ae96e3adf9817aa3a58365c466).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.
 * This patch does not change any dependencies.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98048198
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31507/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98048196
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98038728
  
  [Test build #31510 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31510/consoleFull)
 for   PR 5823 at commit 
[`4d04301`](https://github.com/apache/spark/commit/4d04301f04566b7e38f9ba4bc3bf5d591d30c224).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98038537
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98038473
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[harishreedharan]

Github user harishreedharan commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98038182
  
Diff from #4688 : 
https://github.com/harishreedharan/spark/compare/36eb8a956c357388e4fdf5858cb4f27236f26a9e...kerberos-longrunning


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98038097
  
  [Test build #31507 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31507/consoleFull)
 for   PR 5823 at commit 
[`b5e7a72`](https://github.com/apache/spark/commit/b5e7a728340d44ae96e3adf9817aa3a58365c466).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98038018
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98038007
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98036759
  
**[Test build #31486 timed 
out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31486/consoleFull)**
 for PR 5823 at commit 
[`50c369d`](https://github.com/apache/spark/commit/50c369dbd0beba74b780f7cc0b90ad5a165b37b7)
 after a configured wait of `150m`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98036762
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31486/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98036761
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98036172
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98036174
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31489/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98036165
  
  [Test build #31489 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31489/consoleFull)
 for   PR 5823 at commit 
[`7bff6e9`](https://github.com/apache/spark/commit/7bff6e9c3cb3184ca4b544d0c96f7d9fed85512d).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.
 * This patch **adds the following new dependencies:**
   * `jaxb-api-2.2.7.jar`
   * `jaxb-core-2.2.7.jar`
   * `jaxb-impl-2.2.7.jar`
   * `pmml-agent-1.1.15.jar`
   * `pmml-model-1.1.15.jar`
   * `pmml-schema-1.1.15.jar`
   * `spark-unsafe_2.10-1.4.0-SNAPSHOT.jar`

 * This patch **removes the following dependencies:**
   * `activation-1.1.jar`
   * `jaxb-api-2.2.2.jar`
   * `jaxb-impl-2.2.3-1.jar`



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98035877
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98035884
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31485/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98035849
  
**[Test build #31485 timed 
out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31485/consoleFull)**
 for PR 5823 at commit 
[`8269c7a`](https://github.com/apache/spark/commit/8269c7a5f8866d17eea85324f16a2989101c6e9d)
 after a configured wait of `150m`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[harishreedharan]

Github user harishreedharan commented on a diff in the pull request:

https://github.com/apache/spark/pull/5823#discussion_r29487432
  
--- Diff: 
core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala
 ---
@@ -168,6 +170,26 @@ private[spark] object CoarseGrainedExecutorBackend 
extends Logging {
   driverConf.set(key, value)
 }
   }
+  // Delegation Token Updater is not supported in Hadoop 1, so use 
reflection.
+  // Can't use Option[ExecutorDelegationTokenUpdater] because it is 
built only in YARN
+  // profile, so use Option[Any] since even the stop method call will 
be via reflection.
+  var tokenUpdaterOption: Option[Any] = None
+  var tokenUpdaterClass: Option[Class[_]] = None
--- End diff --

I didn't want to add more methods to `SparkHadoopUtil` and 
`YarnSparkHadoopUtil`, but since that gets rid of the reflection and also 
guarantees that `addCredentials` gets called only in YARN mode, I guess it is 
worth it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[harishreedharan]

Github user harishreedharan commented on a diff in the pull request:

https://github.com/apache/spark/pull/5823#discussion_r29487205
  
--- Diff: 
core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala
 ---
@@ -168,6 +170,26 @@ private[spark] object CoarseGrainedExecutorBackend 
extends Logging {
   driverConf.set(key, value)
 }
   }
+  // Delegation Token Updater is not supported in Hadoop 1, so use 
reflection.
+  // Can't use Option[ExecutorDelegationTokenUpdater] because it is 
built only in YARN
+  // profile, so use Option[Any] since even the stop method call will 
be via reflection.
+  var tokenUpdaterOption: Option[Any] = None
+  var tokenUpdaterClass: Option[Class[_]] = None
--- End diff --

We can keep the reflection out if we separate it out into different methods 
- that seems like a sensible thing to do, that way all of the runtime-related 
error-prone code can be gotten rid of. 

I will update this with something that does that.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[tgravescs]

Github user tgravescs commented on a diff in the pull request:

https://github.com/apache/spark/pull/5823#discussion_r29486248
  
--- Diff: 
core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala
 ---
@@ -168,6 +170,26 @@ private[spark] object CoarseGrainedExecutorBackend 
extends Logging {
   driverConf.set(key, value)
 }
   }
+  // Delegation Token Updater is not supported in Hadoop 1, so use 
reflection.
+  // Can't use Option[ExecutorDelegationTokenUpdater] because it is 
built only in YARN
+  // profile, so use Option[Any] since even the stop method call will 
be via reflection.
+  var tokenUpdaterOption: Option[Any] = None
+  var tokenUpdaterClass: Option[Class[_]] = None
--- End diff --

I'm wondering if we can create a new function in SparkHadoopUtil and 
YarnSparkHadoopUtil.  The SparkHadoopUtil version wouldn't do anything and the 
YarnSparkHadoopUtil would have this logic.   That way its only used on yarn.  I 
still need to look to see exactly which versions of hadoop support 
addCredentials and getCredentials from UserGroupInformation.  

haven't looked into exactly how that would work @harishreedharan  thoughts?

the downside is we would either have to keep the reference to to the 
tokenUpdater in YarnSparkhadoopUtil so we can do the stop on it or return it 
back here and still use reflection.  I was trying to keep to much stuff from 
going in there but it seems like it might be the most convenient place at this 
point. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[harishreedharan]

Github user harishreedharan commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98020639
  
It should be fixed. There are 3 builds running on this PrR for some reason,
so one of the following two should succeed (I think I had to force push one
commit to my branch which seems to have confused Jenkins)

On Thursday, April 30, 2015, Tom Graves  wrote:

> @harishreedharan  can you fix the
> style issues
>
> —
> Reply to this email directly or view it on GitHub
> .
>


-- 

Thanks,
Hari



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[tgravescs]

Github user tgravescs commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98020263
  
@harishreedharan can you fix the style issues


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98016108
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98016112
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31487/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98016103
  
  [Test build #31487 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31487/consoleFull)
 for   PR 5823 at commit 
[`600aa31`](https://github.com/apache/spark/commit/600aa31a069798cbcd440398604584a5c0a6f01f).
 * This patch **fails Scala style tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.
 * This patch does not change any dependencies.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98011601
  
  [Test build #31489 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31489/consoleFull)
 for   PR 5823 at commit 
[`7bff6e9`](https://github.com/apache/spark/commit/7bff6e9c3cb3184ca4b544d0c96f7d9fed85512d).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[harishreedharan]

Github user harishreedharan commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98011606
  
@tgravescs - Here is the actual diff with the last merge:

https://github.com/harishreedharan/spark/compare/36eb8a956c357388e4fdf5858cb4f27236f26a9e...kerberos-longrunning#diff-79391110e9f26657e415aa169a004998R184

Just using reflection to create the class instance, which is now in yarn 
module. There is no need to catch a `ClassNotFoundException` because if that 
parameter is present, we can assume YARN support was built into the assembly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98011578
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98011567
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98011148
  
  [Test build #31487 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31487/consoleFull)
 for   PR 5823 at commit 
[`600aa31`](https://github.com/apache/spark/commit/600aa31a069798cbcd440398604584a5c0a6f01f).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98010983
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31483/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98010962
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98010975
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98010863
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98009895
  
  [Test build #31486 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31486/consoleFull)
 for   PR 5823 at commit 
[`50c369d`](https://github.com/apache/spark/commit/50c369dbd0beba74b780f7cc0b90ad5a165b37b7).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98009821
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98009806
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98009482
  
  [Test build #31485 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31485/consoleFull)
 for   PR 5823 at commit 
[`8269c7a`](https://github.com/apache/spark/commit/8269c7a5f8866d17eea85324f16a2989101c6e9d).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98009432
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98009424
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98009020
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/5823#issuecomment-98009008
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[harishreedharan]

GitHub user harishreedharan opened a pull request:

https://github.com/apache/spark/pull/5823

[SPARK-5342][YARN] Allow long running Spark apps to run on secure YARN/HDFS

Take 2. Does the same thing as #4688, but fixes Hadoop-1 build.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/harishreedharan/spark kerberos-longrunning

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/5823.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5823


commit 77914dd74b3a4af3501bda7a72c658ffcdd0682f
Author: Hari Shreedharan 
Date:   2015-01-30T19:14:35Z

WIP: Add kerberos principal and keytab to YARN client.

commit ccba5bc3e7ceceb9b1f15072888454b88d1a2322
Author: Hari Shreedharan 
Date:   2015-02-02T23:06:30Z

WIP: More changes wrt kerberos

commit 2b0d745ec7b76c3dd992660c24ddac556ba1de6a
Author: Hari Shreedharan 
Date:   2015-02-19T05:47:14Z

[SPARK-5342][YARN] Allow long running Spark apps to run on secure YARN/HDFS.

Current Spark apps running on Secure YARN/HDFS would not be able to write 
data
to HDFS after 7 days, since delegation tokens cannot be renewed beyond 
that. This
means Spark Streaming apps will not be able to run on Secure YARN.

This commit adds basic functionality to fix this issue. In this patch:
- new parameters are added - principal and keytab, which can be used to 
login to a KDC
- the client logs in, and then get tokens to start the AM
- the keytab is copied to the staging directory
- the AM waits for 60% of the time till expiry of the tokens and then logs 
in using the keytab
- each time after 60% of the time, new tokens are created and sent to the 
executors

commit f8fe694efd117d707313748c02cef42240a3aec7
Author: Hari Shreedharan 
Date:   2015-02-19T18:46:46Z

Handle None if keytab-login is not scheduled.

commit bcfc3747e14ce6f1a242ba9cd71c03017b77f36e
Author: Hari Shreedharan 
Date:   2015-02-20T07:08:15Z

Fix Hadoop-1 build by adding no-op methods in SparkHadoopUtil, with impl in 
YarnSparkHadoopUtil.

commit d282d7a69d563604776f6985760bd2c367725d0d
Author: Hari Shreedharan 
Date:   2015-02-20T07:41:00Z

Fix ClientSuite to set YARN mode, so that the correct class is used in 
tests.

commit 41efde0ce0523f53f5d27c85e5c760b6d20fe0d6
Author: Hari Shreedharan 
Date:   2015-02-22T02:00:55Z

Merge branch 'master' into kerberos-longrunning

commit fb27f46f2f2b06cffe15a4728e828371f92e17a5
Author: Hari Shreedharan 
Date:   2015-02-24T01:38:51Z

Make sure principal and keytab are set before CoarseGrainedSchedulerBackend 
is started.
Also schedule re-logins in CoarseGrainedSchedulerBackend#start()

commit 8c6928a1cf966136847e6035a0610781b17e2769
Author: Hari Shreedharan 
Date:   2015-02-24T03:29:46Z

Fix issue caused by direct creation of Actor object.

commit d79b2b98532b1b5133026fe095104c5dc5f52af9
Author: Hari Shreedharan 
Date:   2015-02-24T19:36:00Z

Make sure correct credentials are passed to FileSystem#addDelegationTokens()

commit 0985b4e2fb1c51247eb993c28718c9b86f242563
Author: Hari Shreedharan 
Date:   2015-02-27T22:00:29Z

Write tokens to HDFS and read them back when required, rather than sending 
them over the wire.

commit b4cb917d8ed5e06b3470f43ec221dd7ecdba7ec8
Author: Hari Shreedharan 
Date:   2015-02-28T00:04:07Z

Send keytab to AM via DistributedCache rather than directly via HDFS

commit 5c11c3e348fecdd070f5ab471314bce94bb4b66e
Author: Hari Shreedharan 
Date:   2015-02-28T06:28:39Z

Move tests to YarnSparkHadoopUtil to fix compile issues.

commit f6954dab2c1d7ebc614093bbde80ae1ae59bf97e
Author: Hari Shreedharan 
Date:   2015-03-05T20:30:06Z

Got rid of Akka communication to renew, instead the executors check a known 
file's
modification time to read the credentials.

commit f0f54cba1e579a0ee320dd27d0683ff4cd458cc4
Author: Hari Shreedharan 
Date:   2015-03-05T21:19:09Z

Be more defensive when updating the credentials file.

commit af6d5f0b2ca70f46507c98f1c930b21591a35e59
Author: Hari Shreedharan 
Date:   2015-03-05T23:41:22Z

Cleaning up files where changes weren't required.

commit 2debcea367aa7e54d49af2109b47568ba61f829b
Author: Hari Shreedharan 
Date:   2015-03-06T08:49:38Z

Change the file structure for credentials files. I will push a followup 
patch which
adds a cleanup mechanism for old credentials files. The credentials files 
are small
and few enough for it to cause issues on HDFS.

commit f4fd711f44c24224178ca1ad9c7b4529bf62fa47
Author: Hari Shreedharan 
Date:   2015-03-06T19:26:11Z

Fix SparkConf usage.

commit 9ef5f1b7558731762435524636ff5bc87552a355
Author: Hari Shreedharan 
Date:   2015-03-07T00:41:15Z

Added explanation of how the credentials refresh works, some other minor 
fixes.

commit 55522e3733de3d6f1a9

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[harishreedharan]

Github user harishreedharan commented on the pull request:

https://github.com/apache/spark/pull/4688#issuecomment-97993995
  
This was reverted because it broke Hadoop-1. I am moving the 
`ExecutorDelegationTokenRenewer` to the `yarn` module and using reflection to 
use it the `CoarseGrainedExecutorBackend` class.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/4688


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[tgravescs]

Github user tgravescs commented on the pull request:

https://github.com/apache/spark/pull/4688#issuecomment-97899357
  
+1, looks good.  Thanks for testing the latest, I was about to ask that.  I 
ran a few things with the patch but didn't configure things to expire. 




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[harishreedharan]

Github user harishreedharan commented on the pull request:

https://github.com/apache/spark/pull/4688#issuecomment-97898669
  
Tested the latest update, still working fine


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4688#issuecomment-97832794
  
  [Test build #31419 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31419/consoleFull)
 for   PR 4688 at commit 
[`36eb8a9`](https://github.com/apache/spark/commit/36eb8a956c357388e4fdf5858cb4f27236f26a9e).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.
 * This patch does not change any dependencies.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4688#issuecomment-97832838
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4688#issuecomment-97832843
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31419/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4688#issuecomment-97790405
  
  [Test build #31419 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31419/consoleFull)
 for   PR 4688 at commit 
[`36eb8a9`](https://github.com/apache/spark/commit/36eb8a956c357388e4fdf5858cb4f27236f26a9e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4688#issuecomment-97788293
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4688#issuecomment-97788488
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[tgravescs]

Github user tgravescs commented on the pull request:

https://github.com/apache/spark/pull/4688#issuecomment-97785135
  
jenkins, test this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4688#issuecomment-97712537
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4688#issuecomment-97712538
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31395/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4688#issuecomment-97712526
  
  [Test build #31395 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31395/consoleFull)
 for   PR 4688 at commit 
[`36eb8a9`](https://github.com/apache/spark/commit/36eb8a956c357388e4fdf5858cb4f27236f26a9e).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.
 * This patch **removes the following dependencies:**
   * `RoaringBitmap-0.4.5.jar`
   * `activation-1.1.jar`
   * `akka-actor_2.10-2.3.4-spark.jar`
   * `akka-remote_2.10-2.3.4-spark.jar`
   * `akka-slf4j_2.10-2.3.4-spark.jar`
   * `aopalliance-1.0.jar`
   * `arpack_combined_all-0.1.jar`
   * `avro-1.7.7.jar`
   * `breeze-macros_2.10-0.11.2.jar`
   * `breeze_2.10-0.11.2.jar`
   * `chill-java-0.5.0.jar`
   * `chill_2.10-0.5.0.jar`
   * `commons-beanutils-1.7.0.jar`
   * `commons-beanutils-core-1.8.0.jar`
   * `commons-cli-1.2.jar`
   * `commons-codec-1.10.jar`
   * `commons-collections-3.2.1.jar`
   * `commons-compress-1.4.1.jar`
   * `commons-configuration-1.6.jar`
   * `commons-digester-1.8.jar`
   * `commons-httpclient-3.1.jar`
   * `commons-io-2.1.jar`
   * `commons-lang-2.5.jar`
   * `commons-lang3-3.3.2.jar`
   * `commons-math-2.1.jar`
   * `commons-math3-3.4.1.jar`
   * `commons-net-2.2.jar`
   * `compress-lzf-1.0.0.jar`
   * `config-1.2.1.jar`
   * `core-1.1.2.jar`
   * `curator-client-2.4.0.jar`
   * `curator-framework-2.4.0.jar`
   * `curator-recipes-2.4.0.jar`
   * `gmbal-api-only-3.0.0-b023.jar`
   * `grizzly-framework-2.1.2.jar`
   * `grizzly-http-2.1.2.jar`
   * `grizzly-http-server-2.1.2.jar`
   * `grizzly-http-servlet-2.1.2.jar`
   * `grizzly-rcm-2.1.2.jar`
   * `groovy-all-2.3.7.jar`
   * `guava-14.0.1.jar`
   * `guice-3.0.jar`
   * `hadoop-annotations-2.2.0.jar`
   * `hadoop-auth-2.2.0.jar`
   * `hadoop-client-2.2.0.jar`
   * `hadoop-common-2.2.0.jar`
   * `hadoop-hdfs-2.2.0.jar`
   * `hadoop-mapreduce-client-app-2.2.0.jar`
   * `hadoop-mapreduce-client-common-2.2.0.jar`
   * `hadoop-mapreduce-client-core-2.2.0.jar`
   * `hadoop-mapreduce-client-jobclient-2.2.0.jar`
   * `hadoop-mapreduce-client-shuffle-2.2.0.jar`
   * `hadoop-yarn-api-2.2.0.jar`
   * `hadoop-yarn-client-2.2.0.jar`
   * `hadoop-yarn-common-2.2.0.jar`
   * `hadoop-yarn-server-common-2.2.0.jar`
   * `ivy-2.4.0.jar`
   * `jackson-annotations-2.4.0.jar`
   * `jackson-core-2.4.4.jar`
   * `jackson-core-asl-1.8.8.jar`
   * `jackson-databind-2.4.4.jar`
   * `jackson-jaxrs-1.8.8.jar`
   * `jackson-mapper-asl-1.8.8.jar`
   * `jackson-module-scala_2.10-2.4.4.jar`
   * `jackson-xc-1.8.8.jar`
   * `jansi-1.4.jar`
   * `javax.inject-1.jar`
   * `javax.servlet-3.0.0.v201112011016.jar`
   * `javax.servlet-3.1.jar`
   * `javax.servlet-api-3.0.1.jar`
   * `jaxb-api-2.2.2.jar`
   * `jaxb-impl-2.2.3-1.jar`
   * `jcl-over-slf4j-1.7.10.jar`
   * `jersey-client-1.9.jar`
   * `jersey-core-1.9.jar`
   * `jersey-grizzly2-1.9.jar`
   * `jersey-guice-1.9.jar`
   * `jersey-json-1.9.jar`
   * `jersey-server-1.9.jar`
   * `jersey-test-framework-core-1.9.jar`
   * `jersey-test-framework-grizzly2-1.9.jar`
   * `jets3t-0.7.1.jar`
   * `jettison-1.1.jar`
   * `jetty-util-6.1.26.jar`
   * `jline-0.9.94.jar`
   * `jline-2.10.4.jar`
   * `jodd-core-3.6.3.jar`
   * `json4s-ast_2.10-3.2.10.jar`
   * `json4s-core_2.10-3.2.10.jar`
   * `json4s-jackson_2.10-3.2.10.jar`
   * `jsr305-1.3.9.jar`
   * `jtransforms-2.4.0.jar`
   * `jul-to-slf4j-1.7.10.jar`
   * `kryo-2.21.jar`
   * `log4j-1.2.17.jar`
   * `lz4-1.2.0.jar`
   * `management-api-3.0.0-b012.jar`
   * `mesos-0.21.0-shaded-protobuf.jar`
   * `metrics-core-3.1.0.jar`
   * `metrics-graphite-3.1.0.jar`
   * `metrics-json-3.1.0.jar`
   * `metrics-jvm-3.1.0.jar`
   * `minlog-1.2.jar`
   * `netty-3.8.0.Final.jar`
   * `netty-all-4.0.23.Final.jar`
   * `objenesis-1.2.jar`
   * `opencsv-2.3.jar`
   * `oro-2.0.8.jar`
   * `paranamer-2.6.jar`
   * `parquet-column-1.6.0rc3.jar`
   * `parquet-common-1.6.0rc3.jar`
   * `parquet-encoding-1.6.0rc3.jar`
   * `parquet-format-2.2.0-rc1.jar`
   * `parquet-generator-1.6.0rc3.jar`
   * `parquet-hadoop-1.6.0rc3.jar`
   * `parquet-jackson-1.6.0rc3.jar`
   * `protobuf-java-2.4.1.jar`
   * `protobuf-java-2.5.0-spark.jar`
   * `py4j-0.8.2.1.jar`
   * `pyrolite-2.0.1.jar`
   * `quasiquotes_2.10-2.0.1.jar`
   * `reflectasm-1.07-shaded.jar`
   * `scala-compiler-2.10.4.jar`
   * `scala-library-2.10.4.jar`
   * `scala

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/4688#issuecomment-97685626
  
  [Test build #31395 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31395/consoleFull)
 for   PR 4688 at commit 
[`36eb8a9`](https://github.com/apache/spark/commit/36eb8a956c357388e4fdf5858cb4f27236f26a9e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4688#issuecomment-97684819
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/4688#issuecomment-97684840
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[tgravescs]

Github user tgravescs commented on a diff in the pull request:

https://github.com/apache/spark/pull/4688#discussion_r29401037
  
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -395,7 +436,16 @@ private[spark] class Client(
 env("SPARK_YARN_MODE") = "true"
 env("SPARK_YARN_STAGING_DIR") = stagingDir
 env("SPARK_USER") = 
UserGroupInformation.getCurrentUser().getShortUserName()
-
+if (loginFromKeytab) {
+  val remoteFs = FileSystem.get(hadoopConf)
+  val stagingDirPath = new Path(remoteFs.getHomeDirectory, stagingDir)
+  val credentialsFile = "credentials-" + UUID.randomUUID().toString
+  sparkConf.set(
+"spark.yarn.credentials.file", new Path(stagingDirPath, 
credentialsFile).toString)
+  logInfo(s"Credentials file set to: $credentialsFile")
+  val renewalInterval = getTokenRenewalInterval(stagingDirPath)
+  sparkConf.set("spark.yarn.renewal.interval", 
renewalInterval.toString)
--- End diff --

lets be more specific with the name of this config, perhaps 
spark.yarn.token.renewal.interval


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[tgravescs]

Github user tgravescs commented on a diff in the pull request:

https://github.com/apache/spark/pull/4688#discussion_r29400173
  
--- Diff: 
yarn/src/main/scala/org/apache/spark/deploy/yarn/AMDelegationTokenRenewer.scala 
---
@@ -0,0 +1,202 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.deploy.yarn
+
+import java.security.PrivilegedExceptionAction
+import java.util.concurrent.{Executors, TimeUnit}
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileSystem, Path}
+import org.apache.hadoop.security.UserGroupInformation
+import org.apache.spark.deploy.SparkHadoopUtil
+
+import org.apache.spark.{Logging, SparkConf}
+import org.apache.spark.util.ThreadUtils
+
+/*
+ * The following methods are primarily meant to make sure long-running 
apps like Spark
+ * Streaming apps can run without interruption while writing to secure 
HDFS. The
+ * scheduleLoginFromKeytab method is called on the driver when the
+ * CoarseGrainedScheduledBackend starts up. This method wakes up a thread 
that logs into the KDC
+ * once 75% of the renewal interval of the original delegation tokens used 
for the container
+ * has elapsed. It then creates new delegation tokens and writes them to 
HDFS in a
+ * pre-specified location - the prefix of which is specified in the 
sparkConf by
+ * spark.yarn.credentials.file (so the file(s) would be named c-1, c-2 
etc. - each update goes
+ * to a new file, with a monotonically increasing suffix). After this, the 
credentials are
+ * updated once 75% of the new tokens renewal interval has elapsed.
+ *
+ * On the executor side, the updateCredentialsIfRequired method is called 
once 80% of the
+ * validity of the original tokens has elapsed. At that time the executor 
finds the
+ * credentials file with the latest timestamp and checks if it has read 
those credentials
+ * before (by keeping track of the suffix of the last file it read). If a 
new file has
+ * appeared, it will read the credentials and update the currently running 
UGI with it. This
+ * process happens again once 80% of the validity of this has expired.
+ */
+private[yarn] class AMDelegationTokenRenewer(
+sparkConf: SparkConf,
+hadoopConf: Configuration) extends Logging {
+
+  private var lastCredentialsFileSuffix = 0
+
+  private val delegationTokenRenewer =
+Executors.newSingleThreadScheduledExecutor(
+  ThreadUtils.namedThreadFactory("Delegation Token Refresh Thread"))
+
+  private val hadoopUtil = YarnSparkHadoopUtil.get
+
+  private val daysToKeepFiles = 
sparkConf.getInt("spark.yarn.credentials.file.retention.days", 5)
+  private val numFilesToKeep = 
sparkConf.getInt("spark.yarn.credentials.file.retention.count", 5)
+
+  /**
+   * Schedule a login from the keytab and principal set using the 
--principal and --keytab
+   * arguments to spark-submit. This login happens only when the 
credentials of the current user
+   * are about to expire. This method reads SPARK_PRINCIPAL and 
SPARK_KEYTAB from the environment
--- End diff --

comment needs to be updated since env variables not used


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[tgravescs]

Github user tgravescs commented on a diff in the pull request:

https://github.com/apache/spark/pull/4688#discussion_r29400077
  
--- Diff: 
core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala
 ---
@@ -182,6 +192,7 @@ private[spark] object CoarseGrainedExecutorBackend 
extends Logging {
   workerUrl.foreach { url =>
 env.rpcEnv.setupEndpoint("WorkerWatcher", new 
WorkerWatcher(env.rpcEnv, url))
   }
+  tokenUpdaterOption.foreach(_.stop())
--- End diff --

this is in the wrong order. you want to do this after the await right?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[harishreedharan]

Github user harishreedharan commented on the pull request:

https://github.com/apache/spark/pull/4688#issuecomment-97608683
  
Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[tgravescs]

Github user tgravescs commented on the pull request:

https://github.com/apache/spark/pull/4688#issuecomment-97589875
  
@harishreedharan sorry I've been swamped today, I'll look at this later 
tonight.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[harishreedharan]

Github user harishreedharan commented on the pull request:

https://github.com/apache/spark/pull/4688#issuecomment-97564223
  
@tgravescs - Jira for Hive and HBase token updates: 
https://issues.apache.org/jira/browse/SPARK-7252


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-5342][YARN] Allow long running Spark ap...

[harishreedharan]

Github user harishreedharan commented on the pull request:

https://github.com/apache/spark/pull/4688#issuecomment-97536131
  
@tgravescs - Do you have any more comments? I'd like to get this into 1.4 
if possible.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org