[jira] [Updated] (SPARK-10649) Streaming jobs unexpectedly inherits job group, job descriptions from context starting thread
[ https://issues.apache.org/jira/browse/SPARK-10649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-10649: -- Fix Version/s: 1.5.1 > Streaming jobs unexpectedly inherits job group, job descriptions from context > starting thread > - > > Key: SPARK-10649 > URL: https://issues.apache.org/jira/browse/SPARK-10649 > Project: Spark > Issue Type: Bug > Components: Streaming >Affects Versions: 1.3.1, 1.4.1, 1.5.0 >Reporter: Tathagata Das >Assignee: Tathagata Das > Fix For: 1.6.0, 1.5.1 > > > The job group, and job descriptions information is passed through thread > local properties, and get inherited by child threads. In case of spark > streaming, the streaming jobs inherit these properties from the thread that > called streamingContext.start(). This may not make sense. > 1. Job group: This is mainly used for cancelling a group of jobs together. It > does not make sense to cancel streaming jobs like this, as the effect will be > unpredictable. And its not a valid usecase any way, to cancel a streaming > context, call streamingContext.stop() > 2. Job description: This is used to pass on nice text descriptions for jobs > to show up in the UI. The job description of the thread that calls > streamingContext.start() is not useful for all the streaming jobs, as it does > not make sense for all of the streaming jobs to have the same description, > and the description may or may not be related to streaming. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-10649) Streaming jobs unexpectedly inherits job group, job descriptions from context starting thread
[ https://issues.apache.org/jira/browse/SPARK-10649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-10649: -- Target Version/s: 1.6.0, 1.5.1 (was: 1.6.0) > Streaming jobs unexpectedly inherits job group, job descriptions from context > starting thread > - > > Key: SPARK-10649 > URL: https://issues.apache.org/jira/browse/SPARK-10649 > Project: Spark > Issue Type: Bug > Components: Streaming >Affects Versions: 1.3.1, 1.4.1, 1.5.0 >Reporter: Tathagata Das >Assignee: Tathagata Das > Fix For: 1.6.0 > > > The job group, and job descriptions information is passed through thread > local properties, and get inherited by child threads. In case of spark > streaming, the streaming jobs inherit these properties from the thread that > called streamingContext.start(). This may not make sense. > 1. Job group: This is mainly used for cancelling a group of jobs together. It > does not make sense to cancel streaming jobs like this, as the effect will be > unpredictable. And its not a valid usecase any way, to cancel a streaming > context, call streamingContext.stop() > 2. Job description: This is used to pass on nice text descriptions for jobs > to show up in the UI. The job description of the thread that calls > streamingContext.start() is not useful for all the streaming jobs, as it does > not make sense for all of the streaming jobs to have the same description, > and the description may or may not be related to streaming. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-10649) Streaming jobs unexpectedly inherits job group, job descriptions from context starting thread
[ https://issues.apache.org/jira/browse/SPARK-10649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-10649: -- Description: The job group, and job descriptions information is passed through thread local properties, and get inherited by child threads. In case of spark streaming, the streaming jobs inherit these properties from the thread that called streamingContext.start(). This may not make sense. 1. Job group: This is mainly used for cancelling a group of jobs together. It does not make sense to cancel streaming jobs like this, as the effect will be unpredictable. And its not a valid usecase any way, to cancel a streaming context, call streamingContext.stop() 2. Job description: This is used to pass on nice text descriptions for jobs to show up in the UI. The job description of the thread that calls streamingContext.start() is not useful for all the streaming jobs, as it does not make sense for all of the streaming jobs to have the same description, and the description may or may not be related to streaming. was: The job group, job descriptions and scheduler pool information is passed through thread local properties, and get inherited by child threads. In case of spark streaming, the streaming jobs inherit these properties from the thread that called streamingContext.start(). This may not make sense. 1. Job group: This is mainly used for cancelling a group of jobs together. It does not make sense to cancel streaming jobs like this, as the effect will be unpredictable. And its not a valid usecase any way, to cancel a streaming context, call streamingContext.stop() 2. Job description: This is used to pass on nice text descriptions for jobs to show up in the UI. The job description of the thread that calls streamingContext.start() is not useful for all the streaming jobs, as it does not make sense for all of the streaming jobs to have the same description, and the description may or may not be related to streaming. > Streaming jobs unexpectedly inherits job group, job descriptions from context > starting thread > - > > Key: SPARK-10649 > URL: https://issues.apache.org/jira/browse/SPARK-10649 > Project: Spark > Issue Type: Bug > Components: Streaming >Affects Versions: 1.3.1, 1.4.1, 1.5.0 >Reporter: Tathagata Das >Assignee: Tathagata Das > > The job group, and job descriptions information is passed through thread > local properties, and get inherited by child threads. In case of spark > streaming, the streaming jobs inherit these properties from the thread that > called streamingContext.start(). This may not make sense. > 1. Job group: This is mainly used for cancelling a group of jobs together. It > does not make sense to cancel streaming jobs like this, as the effect will be > unpredictable. And its not a valid usecase any way, to cancel a streaming > context, call streamingContext.stop() > 2. Job description: This is used to pass on nice text descriptions for jobs > to show up in the UI. The job description of the thread that calls > streamingContext.start() is not useful for all the streaming jobs, as it does > not make sense for all of the streaming jobs to have the same description, > and the description may or may not be related to streaming. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org