[jira] [Commented] (SPARK-14245) webUI should display the user
[ https://issues.apache.org/jira/browse/SPARK-14245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15971633#comment-15971633 ] Alex Bozarth commented on SPARK-14245: -- Given it's been a year since I fixed that PR I honestly can't remember the exact details of that race condition, and looking back I wish I had been more through in detailing it in my comment. I can remember how the race condition manifested as though: After first starting up a stand-alone master (using user1), I would then submit two apps simultaneously (or close enough for testing) using two users (user1 and user2) in this case both applications always listed user1, but this would only happen if the two were the first applications submitted after starting up the master (subsequent simultaneous submittals worked as expected). As I mentioned I can't remember how I tracked down and identified the race condition in the code though, feel free to dig into it. I would look at it, but I don't have time at the moment. > webUI should display the user > - > > Key: SPARK-14245 > URL: https://issues.apache.org/jira/browse/SPARK-14245 > Project: Spark > Issue Type: Improvement > Components: Web UI >Affects Versions: 1.6.1 >Reporter: Thomas Graves >Assignee: Alex Bozarth > Fix For: 2.0.0 > > > It would be nice if the Spark UI (both active and history) showed the user > who ran the application somewhere when you are in the application view. > Perhaps under the Jobs view by total uptime and scheduler mode. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-14245) webUI should display the user
[
https://issues.apache.org/jira/browse/SPARK-14245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15971536#comment-15971536
]
Imran Rashid commented on SPARK-14245:
--
thanks, I should have looked in the PR first, sorry.
But I have to admit -- I really do not understand what that comment is
referring to. Why is the master involved at all? Is there a separate
standalone-only issue, maybe a race in the master that we need to fix?
I'm mostly concerned about the difference between hadoop's
{{UserGroupInformation.getCurrentUser()}} vs. the "user.name" system property.
I imagine those are not always the same.
> webUI should display the user
> -
>
> Key: SPARK-14245
> URL: https://issues.apache.org/jira/browse/SPARK-14245
> Project: Spark
> Issue Type: Improvement
> Components: Web UI
>Affects Versions: 1.6.1
>Reporter: Thomas Graves
>Assignee: Alex Bozarth
> Fix For: 2.0.0
>
>
> It would be nice if the Spark UI (both active and history) showed the user
> who ran the application somewhere when you are in the application view.
> Perhaps under the Jobs view by total uptime and scheduler mode.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-14245) webUI should display the user
[ https://issues.apache.org/jira/browse/SPARK-14245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15971365#comment-15971365 ] Thomas Graves commented on SPARK-14245: --- see the commend in the PR, I think there was a race with the SparkContext version: I just pushed an update. I changed how I get the username so it uses the same method for both active and history. This method actually pulls the information from the same source as both sc.sparkUser and the API do at their root, the system env. Except with this method we make sure we always get the correct user; I noticed a race condition when using sc.sparkUser while finding this solution. After starting up the master when two spark applications are started at once, one by the user who started the master and one who didn't, both would appear to been started by the user who started the master. This would only happened on the first apps started, any apps started after would behave as expected. This race condition doesn't exist with my change. > webUI should display the user > - > > Key: SPARK-14245 > URL: https://issues.apache.org/jira/browse/SPARK-14245 > Project: Spark > Issue Type: Improvement > Components: Web UI >Affects Versions: 1.6.1 >Reporter: Thomas Graves >Assignee: Alex Bozarth > Fix For: 2.0.0 > > > It would be nice if the Spark UI (both active and history) showed the user > who ran the application somewhere when you are in the application view. > Perhaps under the Jobs view by total uptime and scheduler mode. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-14245) webUI should display the user
[
https://issues.apache.org/jira/browse/SPARK-14245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15971337#comment-15971337
]
Imran Rashid commented on SPARK-14245:
--
Hi [~ajbozarth] [~tgraves] -- I was just taking a look at this b/c of a new pr
to get the user added to the rest api:
https://github.com/apache/spark/pull/17656
but one thing puzzled me about this. Why is it using a different way of
getting the user than the history server uses? The HistoryServer takes the
user from the SparkListenerApplicationStart event, which originates with
[{{SparkContext.sparkUser}}|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SparkContext.scala#L295].
It seems like that is doing something slightly more general than just taking
"user.name". I haven't looked closely yet, but is there a reason to do one or
the other?
> webUI should display the user
> -
>
> Key: SPARK-14245
> URL: https://issues.apache.org/jira/browse/SPARK-14245
> Project: Spark
> Issue Type: Improvement
> Components: Web UI
>Affects Versions: 1.6.1
>Reporter: Thomas Graves
>Assignee: Alex Bozarth
> Fix For: 2.0.0
>
>
> It would be nice if the Spark UI (both active and history) showed the user
> who ran the application somewhere when you are in the application view.
> Perhaps under the Jobs view by total uptime and scheduler mode.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-14245) webUI should display the user
[ https://issues.apache.org/jira/browse/SPARK-14245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15224589#comment-15224589 ] Thomas Graves commented on SPARK-14245: --- ok, personally I would prefer people to assign jira to themselves so its very obvious they are working on it. If others want to still work on it in parallel they are welcome but at least they aren't wasting their time duplicating work unless they chose to. Looking back at our contributing guidelines doesn't mention either so i guess its fine either way. > webUI should display the user > - > > Key: SPARK-14245 > URL: https://issues.apache.org/jira/browse/SPARK-14245 > Project: Spark > Issue Type: Improvement > Components: Web UI >Affects Versions: 1.6.1 >Reporter: Thomas Graves >Assignee: Alex Bozarth > > It would be nice if the Spark UI (both active and history) showed the user > who ran the application somewhere when you are in the application view. > Perhaps under the Jobs view by total uptime and scheduler mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-14245) webUI should display the user
[ https://issues.apache.org/jira/browse/SPARK-14245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15224352#comment-15224352 ] Sean Owen commented on SPARK-14245: --- Yeah kind of a judgment call. I tend to not Assign JIRAs until something's merged, since we already see here open PRs. I don't see a need to further 'reserve' the JIRA for someone. You can just go ahead here. If someone's far along on a big change I think we might assign it just to make it clear it's likely that person's branch will be used. Until that happens I think it's open for other suggestions even if there's an open PR. > webUI should display the user > - > > Key: SPARK-14245 > URL: https://issues.apache.org/jira/browse/SPARK-14245 > Project: Spark > Issue Type: Improvement > Components: Web UI >Affects Versions: 1.6.1 >Reporter: Thomas Graves >Assignee: Alex Bozarth > > It would be nice if the Spark UI (both active and history) showed the user > who ran the application somewhere when you are in the application view. > Perhaps under the Jobs view by total uptime and scheduler mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-14245) webUI should display the user
[ https://issues.apache.org/jira/browse/SPARK-14245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15224308#comment-15224308 ] Alex Bozarth commented on SPARK-14245: -- Thank you for that info, when I first started working on Spark I was originally given the impression that Spark Jira's weren't assigned until a pr was merged. > webUI should display the user > - > > Key: SPARK-14245 > URL: https://issues.apache.org/jira/browse/SPARK-14245 > Project: Spark > Issue Type: Improvement > Components: Web UI >Affects Versions: 1.6.1 >Reporter: Thomas Graves >Assignee: Alex Bozarth > > It would be nice if the Spark UI (both active and history) showed the user > who ran the application somewhere when you are in the application view. > Perhaps under the Jobs view by total uptime and scheduler mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-14245) webUI should display the user
[ https://issues.apache.org/jira/browse/SPARK-14245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15224288#comment-15224288 ] Thomas Graves commented on SPARK-14245: --- [~ajbozarth] Please go ahead and assign yourself the jira when you start working on it so we don't get multiple people working on the same things. > webUI should display the user > - > > Key: SPARK-14245 > URL: https://issues.apache.org/jira/browse/SPARK-14245 > Project: Spark > Issue Type: Improvement > Components: Web UI >Affects Versions: 1.6.1 >Reporter: Thomas Graves > > It would be nice if the Spark UI (both active and history) showed the user > who ran the application somewhere when you are in the application view. > Perhaps under the Jobs view by total uptime and scheduler mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-14245) webUI should display the user
[ https://issues.apache.org/jira/browse/SPARK-14245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15222713#comment-15222713 ] Dongjoon Hyun commented on SPARK-14245: --- I see. Thank you! > webUI should display the user > - > > Key: SPARK-14245 > URL: https://issues.apache.org/jira/browse/SPARK-14245 > Project: Spark > Issue Type: Improvement > Components: Web UI >Affects Versions: 1.6.1 >Reporter: Thomas Graves > > It would be nice if the Spark UI (both active and history) showed the user > who ran the application somewhere when you are in the application view. > Perhaps under the Jobs view by total uptime and scheduler mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-14245) webUI should display the user
[ https://issues.apache.org/jira/browse/SPARK-14245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15222702#comment-15222702 ] Alex Bozarth commented on SPARK-14245: -- I don't know what you mean by "real" application submitter. My change shows the same user listed in the active/completed application lists on the driver Web UI, which is whichever system user the spark app is run by. > webUI should display the user > - > > Key: SPARK-14245 > URL: https://issues.apache.org/jira/browse/SPARK-14245 > Project: Spark > Issue Type: Improvement > Components: Web UI >Affects Versions: 1.6.1 >Reporter: Thomas Graves > > It would be nice if the Spark UI (both active and history) showed the user > who ran the application somewhere when you are in the application view. > Perhaps under the Jobs view by total uptime and scheduler mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-14245) webUI should display the user
[ https://issues.apache.org/jira/browse/SPARK-14245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15222697#comment-15222697 ] Dongjoon Hyun commented on SPARK-14245: --- Hi, [~ajbozarth]. I'm just curious. Does your PR show the real application submitter? Thanks in advance. > webUI should display the user > - > > Key: SPARK-14245 > URL: https://issues.apache.org/jira/browse/SPARK-14245 > Project: Spark > Issue Type: Improvement > Components: Web UI >Affects Versions: 1.6.1 >Reporter: Thomas Graves > > It would be nice if the Spark UI (both active and history) showed the user > who ran the application somewhere when you are in the application view. > Perhaps under the Jobs view by total uptime and scheduler mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-14245) webUI should display the user
[ https://issues.apache.org/jira/browse/SPARK-14245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15222560#comment-15222560 ] Apache Spark commented on SPARK-14245: -- User 'ajbozarth' has created a pull request for this issue: https://github.com/apache/spark/pull/12123 > webUI should display the user > - > > Key: SPARK-14245 > URL: https://issues.apache.org/jira/browse/SPARK-14245 > Project: Spark > Issue Type: Improvement > Components: Web UI >Affects Versions: 1.6.1 >Reporter: Thomas Graves > > It would be nice if the Spark UI (both active and history) showed the user > who ran the application somewhere when you are in the application view. > Perhaps under the Jobs view by total uptime and scheduler mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-14245) webUI should display the user
[ https://issues.apache.org/jira/browse/SPARK-14245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15218742#comment-15218742 ] Alex Bozarth commented on SPARK-14245: -- I'm looking into this in my free moments today, will probably have something in the next day or two > webUI should display the user > - > > Key: SPARK-14245 > URL: https://issues.apache.org/jira/browse/SPARK-14245 > Project: Spark > Issue Type: Improvement > Components: Web UI >Affects Versions: 1.6.1 >Reporter: Thomas Graves > > It would be nice if the Spark UI (both active and history) showed the user > who ran the application somewhere when you are in the application view. > Perhaps under the Jobs view by total uptime and scheduler mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
