[jira] [Created] (SPARK-26522) CLONE - Add configurable auth secret source in k8s backend
Euijun created SPARK-26522: -- Summary: CLONE - Add configurable auth secret source in k8s backend Key: SPARK-26522 URL: https://issues.apache.org/jira/browse/SPARK-26522 Project: Spark Issue Type: New Feature Components: Kubernetes Affects Versions: 3.0.0 Reporter: Euijun Assignee: Matt Cheah Fix For: 3.0.0 This is a follow up to SPARK-26194, which aims to add auto-generated secrets similar to the YARN backend. There's a desire to support different ways to generate and propagate these auth secrets (e.g. using things like Vault). Need to investigate: - exposing configuration to support that - changing SecurityManager so that it can delegate some of the secret-handling logic to custom implementations - figuring out whether this can also be used in client-mode, where the driver is not created by the k8s backend in Spark. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-26522) Auth secret error in RBackendAuthHandler
[ https://issues.apache.org/jira/browse/SPARK-26522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Euijun updated SPARK-26522: --- Affects Version/s: (was: 3.0.0) 2.3.1 Priority: Minor (was: Major) Fix Version/s: (was: 3.0.0) Description: Hi expert, I try to use livy to connect sparkR backend. This is related to https://stackoverflow.com/questions/53900995/livy-spark-r-issue Error message is, {code:java} Error in sparkR.sparkContext(master, appName, sparkHome, sparkConfigMap, : Auth secret not provided in environment.{code} caused by, spark-2.3.1/R/pkg/R/sparkR.R {code:java} sparkR.sparkContext <- function( ... authSecret <- Sys.getenv("SPARKR_BACKEND_AUTH_SECRET") if (nchar(authSecret) == 0) { stop("Auth secret not provided in environment.") } ... ) {code} Best regard. was: This is a follow up to SPARK-26194, which aims to add auto-generated secrets similar to the YARN backend. There's a desire to support different ways to generate and propagate these auth secrets (e.g. using things like Vault). Need to investigate: - exposing configuration to support that - changing SecurityManager so that it can delegate some of the secret-handling logic to custom implementations - figuring out whether this can also be used in client-mode, where the driver is not created by the k8s backend in Spark. Component/s: (was: Kubernetes) SparkR Issue Type: Bug (was: New Feature) Summary: Auth secret error in RBackendAuthHandler (was: CLONE - Add configurable auth secret source in k8s backend) > Auth secret error in RBackendAuthHandler > > > Key: SPARK-26522 > URL: https://issues.apache.org/jira/browse/SPARK-26522 > Project: Spark > Issue Type: Bug > Components: SparkR >Affects Versions: 2.3.1 >Reporter: Euijun >Assignee: Matt Cheah >Priority: Minor > > Hi expert, > I try to use livy to connect sparkR backend. > This is related to > https://stackoverflow.com/questions/53900995/livy-spark-r-issue > > Error message is, > {code:java} > Error in sparkR.sparkContext(master, appName, sparkHome, sparkConfigMap, : > Auth secret not provided in environment.{code} > caused by, > spark-2.3.1/R/pkg/R/sparkR.R > > {code:java} > sparkR.sparkContext <- function( > ... > authSecret <- Sys.getenv("SPARKR_BACKEND_AUTH_SECRET") > if (nchar(authSecret) == 0) { > stop("Auth secret not provided in environment.") > } > ... > ) > {code} > > Best regard. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-26522) Auth secret error in RBackendAuthHandler
[ https://issues.apache.org/jira/browse/SPARK-26522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Euijun updated SPARK-26522: --- Shepherd: Apache Spark Labels: newbie (was: ) > Auth secret error in RBackendAuthHandler > > > Key: SPARK-26522 > URL: https://issues.apache.org/jira/browse/SPARK-26522 > Project: Spark > Issue Type: Bug > Components: SparkR >Affects Versions: 2.3.1 >Reporter: Euijun >Assignee: Matt Cheah >Priority: Minor > Labels: newbie > > Hi expert, > I try to use livy to connect sparkR backend. > This is related to > https://stackoverflow.com/questions/53900995/livy-spark-r-issue > > Error message is, > {code:java} > Error in sparkR.sparkContext(master, appName, sparkHome, sparkConfigMap, : > Auth secret not provided in environment.{code} > caused by, > spark-2.3.1/R/pkg/R/sparkR.R > > {code:java} > sparkR.sparkContext <- function( > ... > authSecret <- Sys.getenv("SPARKR_BACKEND_AUTH_SECRET") > if (nchar(authSecret) == 0) { > stop("Auth secret not provided in environment.") > } > ... > ) > {code} > > Best regard. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-26522) Auth secret error in RBackendAuthHandler
[ https://issues.apache.org/jira/browse/SPARK-26522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Euijun updated SPARK-26522: --- Description: Hi expert, I try to use livy to connect sparkR backend. This is related to [https://stackoverflow.com/questions/53900995/livy-spark-r-issue] Error message is, {code:java} Error in sparkR.sparkContext(master, appName, sparkHome, sparkConfigMap, : Auth secret not provided in environment.{code} caused by, spark-2.3.1/R/pkg/R/sparkR.R {code:java} sparkR.sparkContext <- function( ... authSecret <- Sys.getenv("SPARKR_BACKEND_AUTH_SECRET") if (nchar(authSecret) == 0) { stop("Auth secret not provided in environment.") } ... ) {code} Best regard. was: Hi expert, I try to use livy to connect sparkR backend. This is related to https://stackoverflow.com/questions/53900995/livy-spark-r-issue Error message is, {code:java} Error in sparkR.sparkContext(master, appName, sparkHome, sparkConfigMap, : Auth secret not provided in environment.{code} caused by, spark-2.3.1/R/pkg/R/sparkR.R {code:java} sparkR.sparkContext <- function( ... authSecret <- Sys.getenv("SPARKR_BACKEND_AUTH_SECRET") if (nchar(authSecret) == 0) { stop("Auth secret not provided in environment.") } ... ) {code} Best regard. > Auth secret error in RBackendAuthHandler > > > Key: SPARK-26522 > URL: https://issues.apache.org/jira/browse/SPARK-26522 > Project: Spark > Issue Type: Bug > Components: SparkR >Affects Versions: 2.3.1 >Reporter: Euijun >Assignee: Matt Cheah >Priority: Minor > Labels: newbie > > Hi expert, > I try to use livy to connect sparkR backend. > This is related to > [https://stackoverflow.com/questions/53900995/livy-spark-r-issue] > > Error message is, > {code:java} > Error in sparkR.sparkContext(master, appName, sparkHome, sparkConfigMap, : > Auth secret not provided in environment.{code} > > caused by, > spark-2.3.1/R/pkg/R/sparkR.R > {code:java} > sparkR.sparkContext <- function( > ... > authSecret <- Sys.getenv("SPARKR_BACKEND_AUTH_SECRET") > if (nchar(authSecret) == 0) { > stop("Auth secret not provided in environment.") > } > ... > ) > {code} > > Best regard. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org