[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[steveloughran]

Github user steveloughran commented on the pull request:

https://github.com/apache/spark/pull/11152#issuecomment-214245234
  
thanks for reviewing this, 
Steve


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/11152


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/11152#issuecomment-213919880
  
Merged to master


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11152#issuecomment-212547490
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11152#issuecomment-212547495
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56385/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11152#issuecomment-212547232
  
**[Test build #56385 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56385/consoleFull)**
 for PR 11152 at commit 
[`1ec238f`](https://github.com/apache/spark/commit/1ec238ff12c6a7b47cc4aa0993104cba5098df5c).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11152#issuecomment-212544067
  
**[Test build #56385 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56385/consoleFull)**
 for PR 11152 at commit 
[`1ec238f`](https://github.com/apache/spark/commit/1ec238ff12c6a7b47cc4aa0993104cba5098df5c).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[steveloughran]

Github user steveloughran commented on a diff in the pull request:

https://github.com/apache/spark/pull/11152#discussion_r60461183
  
--- Diff: docs/monitoring.md ---
@@ -253,10 +273,24 @@ for a running application, at 
`http://localhost:4040/api/v1`.
 Summary metrics of all tasks in the given stage attempt
   
   
+
+?quantiles summarize the metrics with the given 
quantiles.
+Example: ?quantiles=0.01,0.5,0.99
--- End diff --

In fact it is worse than that: the java8 javadoc engine will fail if a 
javadoc has an "illegal" closing tag in it. Which means that putting the 
closing tags in there breaks builds on java 8+. That's why I'm fairly cautious 
about closing the "classic" HTML tags these days.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/11152#discussion_r60459834
  
--- Diff: docs/monitoring.md ---
@@ -253,10 +273,24 @@ for a running application, at 
`http://localhost:4040/api/v1`.
 Summary metrics of all tasks in the given stage attempt
   
   
+
+?quantiles summarize the metrics with the given 
quantiles.
+Example: ?quantiles=0.01,0.5,0.99
--- End diff --

Whatever works, I thought HTML was technically supposed to act like XML and 
use the self-closing syntax for tags with no body.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[steveloughran]

Github user steveloughran commented on a diff in the pull request:

https://github.com/apache/spark/pull/11152#discussion_r60457665
  
--- Diff: docs/monitoring.md ---
@@ -253,10 +273,24 @@ for a running application, at 
`http://localhost:4040/api/v1`.
 Summary metrics of all tasks in the given stage attempt
   
   
+
+?quantiles summarize the metrics with the given 
quantiles.
+Example: ?quantiles=0.01,0.5,0.99
--- End diff --

Now that's interesting. in HTML 1.0-5.x it's like {{}}: you don't insert 
a closing element in , though its permitted. See: [The br 
element](http://w3c.github.io/html/textlevel-semantics.html#the-br-element).

If the output goes straight to HTML then, leaving it open is the correct 
action. 

If, alternatively, its going to go through a process pipeline that 
generates XML and then things from it, then the closing tag matters.

What is the pipeline that jekyll implements?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[steveloughran]

Github user steveloughran commented on a diff in the pull request:

https://github.com/apache/spark/pull/11152#discussion_r60452742
  
--- Diff: docs/monitoring.md ---
@@ -222,17 +222,33 @@ both running applications, and in the history server. 
 The endpoints are mounted
 for the history server, they would typically be accessible at 
`http://:18080/api/v1`, and
 for a running application, at `http://localhost:4040/api/v1`.
 
+In the API, an application is referenced by its application ID, `[app-id]`.
+When running on YARN, each application may have multiple attempts; each 
identified by their `[attempt-id]`.
+In the API listed below, `[app-id]` will actually be 
`[base-app-id]/[attempt-id]`,
+where `[base-app-id]` is the YARN application ID.
+
 
   EndpointMeaning
   
-/applications
-A list of all applications
+/applications
+A list of all applications
+
+?status=[completed|running] list only applications in the 
chosen state
+
+?minDate=[date] earliest date/time to list. Examples:
+?minDate=2015-02-10
+?minDate=2015-02-03T16:42:40.000GMT
+?maxDate=[date] latest date/time to list; uses same 
format as minDate
   
   
 /applications/[app-id]/jobs
 A list of all jobs for a given application
   
   
+
+?status=[complete|succeeded|failed] list only jobs in 
the state
--- End diff --

OK, I'll go with that and ``


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[steveloughran]

Github user steveloughran commented on a diff in the pull request:

https://github.com/apache/spark/pull/11152#discussion_r60452582
  
--- Diff: docs/monitoring.md ---
@@ -269,17 +303,24 @@ for a running application, at 
`http://localhost:4040/api/v1`.
 Details for the storage status of a given RDD
   
   
-/applications/[app-id]/logs
-Download the event logs for all attempts of the given application 
as a zip file
+/applications/[base-app-id]/logs
+Download the event logs for all attempts of the given application 
as files within
+a zip file.
+
   
   
-/applications/[app-id]/[attempt-id]/logs
-Download the event logs for the specified attempt of the given 
application as a zip file
+/applications/[base-app-id]/[attempt-id]/logs
+Download the event logs for a specific application as a zip 
file.
   
 
 
-When running on Yarn, each application has multiple attempts, so 
`[app-id]` is actually
-`[app-id]/[attempt-id]` in all cases.
+The number of jobs and stages which can retrieved is constrained by the 
same garbage
--- End diff --

true, it's a sort of meta-level GC: go over the limit and the UI just 
throws away things. How about:  "the retention policy of the standalone spark 
UI"


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/11152#discussion_r60387049
  
--- Diff: docs/monitoring.md ---
@@ -222,17 +222,33 @@ both running applications, and in the history server. 
 The endpoints are mounted
 for the history server, they would typically be accessible at 
`http://:18080/api/v1`, and
 for a running application, at `http://localhost:4040/api/v1`.
 
+In the API, an application is referenced by its application ID, `[app-id]`.
+When running on YARN, each application may have multiple attempts; each 
identified by their `[attempt-id]`.
+In the API listed below, `[app-id]` will actually be 
`[base-app-id]/[attempt-id]`,
+where `[base-app-id]` is the YARN application ID.
+
 
   EndpointMeaning
   
-/applications
-A list of all applications
+/applications
+A list of all applications
+
+?status=[completed|running] list only applications in the 
chosen state
+
+?minDate=[date] earliest date/time to list. Examples:
+?minDate=2015-02-10
+?minDate=2015-02-03T16:42:40.000GMT
+?maxDate=[date] latest date/time to list; uses same 
format as minDate
   
   
 /applications/[app-id]/jobs
 A list of all jobs for a given application
   
   
+
+?status=[complete|succeeded|failed] list only jobs in 
the state
--- End diff --

I do think the form with one `` looks meaningfully better. Otherwise 
these looks like hanging docs for some unspecified endpoint


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/11152#discussion_r60386964
  
--- Diff: docs/monitoring.md ---
@@ -253,10 +273,24 @@ for a running application, at 
`http://localhost:4040/api/v1`.
 Summary metrics of all tasks in the given stage attempt
   
   
+
+?quantiles summarize the metrics with the given 
quantiles.
+Example: ?quantiles=0.01,0.5,0.99
--- End diff --

Technically `` but no big deal


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/11152#discussion_r60386935
  
--- Diff: docs/monitoring.md ---
@@ -269,17 +303,24 @@ for a running application, at 
`http://localhost:4040/api/v1`.
 Details for the storage status of a given RDD
   
   
-/applications/[app-id]/logs
-Download the event logs for all attempts of the given application 
as a zip file
+/applications/[base-app-id]/logs
+Download the event logs for all attempts of the given application 
as files within
+a zip file.
+
   
   
-/applications/[app-id]/[attempt-id]/logs
-Download the event logs for the specified attempt of the given 
application as a zip file
+/applications/[base-app-id]/[attempt-id]/logs
+Download the event logs for a specific application as a zip 
file.
   
 
 
-When running on Yarn, each application has multiple attempts, so 
`[app-id]` is actually
-`[app-id]/[attempt-id]` in all cases.
+The number of jobs and stages which can retrieved is constrained by the 
same garbage
--- End diff --

I don't really mind the text here but it doesn't actually trigger GC right? 
this is just about how much stuff is retained in memory?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11152#issuecomment-211568620
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56132/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11152#issuecomment-211568618
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11152#issuecomment-211568486
  
**[Test build #56132 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56132/consoleFull)**
 for PR 11152 at commit 
[`d609c89`](https://github.com/apache/spark/commit/d609c8914f65740628fc210e77ca76a73e73b97b).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11152#issuecomment-211566158
  
**[Test build #56132 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56132/consoleFull)**
 for PR 11152 at commit 
[`d609c89`](https://github.com/apache/spark/commit/d609c8914f65740628fc210e77ca76a73e73b97b).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[steveloughran]

Github user steveloughran commented on the pull request:

https://github.com/apache/spark/pull/11152#issuecomment-211563858
  
OK: for the jekyll site all is well with and without vtable; its just the 
dev tools where things go wrong. I'll push up to github to see what decisions 
it makes. Note that the markdown editors both centre all the options in the 
spark configuration options table —I could add a valign there tag too


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[steveloughran]

Github user steveloughran commented on the pull request:

https://github.com/apache/spark/pull/11152#issuecomment-211560297
  
Here's a couple of screenshots of two MD editors: IntelliJ IDEA and Mou. 
Both of these are placing the /applications text in the centre of all the text, 
which really is ugly

That may not happen in github renderings, but its clear that some MD 
processors are reaching different conclusions about vertical text alignments.

To avoid that the table will need either {{valign=top}} or the td elements 
retained. Those two IDEs do take valign happily; I'll look at how the site 
builds go with  & valign.

![screen shot 2016-04-18 at 21 12 
07](https://cloud.githubusercontent.com/assets/162090/14618387/4c71ef56-05aa-11e6-9ece-5173c2083a6a.png)


![screen shot 2016-04-18 at 21 12 
13](https://cloud.githubusercontent.com/assets/162090/14618388/4c73e70c-05aa-11e6-91dd-aec59bb3b279.png)



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[steveloughran]

Github user steveloughran commented on the pull request:

https://github.com/apache/spark/pull/11152#issuecomment-211554153
  
OK, I'll move to {{
}} tags


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[steveloughran]

Github user steveloughran commented on a diff in the pull request:

https://github.com/apache/spark/pull/11152#discussion_r60124829
  
--- Diff: docs/monitoring.md ---
@@ -269,17 +310,23 @@ for a running application, at 
`http://localhost:4040/api/v1`.
 Details for the storage status of a given RDD
   
   
-/applications/[app-id]/logs
-Download the event logs for all attempts of the given application 
as a zip file
+/applications/[base-app-id]/logs
+Download the event logs for all attempts of the given application 
as a zip file.
+
   
   
-/applications/[app-id]/[attempt-id]/logs
-Download the event logs for the specified attempt of the given 
application as a zip file
+/applications/[base-app-id]/[attempt-id]/logs
+Download the event logs for a specific application as a zip 
file
--- End diff --

Looking a the the implementation in `FsHistoryProvider`, it states " If no 
attempt is specified, or there is no attemptId for attempts, return all 
attempts"

That is: the zip file contains the logs of *every* attempt. I'll add that 
to the text.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[squito]

Github user squito commented on a diff in the pull request:

https://github.com/apache/spark/pull/11152#discussion_r60086973
  
--- Diff: docs/monitoring.md ---
@@ -222,17 +222,40 @@ both running applications, and in the history server. 
 The endpoints are mounted
 for the history server, they would typically be accessible at 
`http://:18080/api/v1`, and
 for a running application, at `http://localhost:4040/api/v1`.
 
+In the API, an application is referenced by its application ID, `[app-id]`.
+When running on YARN, each application may have multiple attempts; each 
identified by their `[attempt-id]`.
+In the API listed below, `[app-id]` will actually be 
`[base-app-id]/[attempt-id]`,
+where `[base-app-id]` is the YARN application ID.
+
 
   EndpointMeaning
   
 /applications
 A list of all applications
   
   
+
+?status=[completed|running] list only applications in 
the chosen state
--- End diff --

yes, no changes to this in 2.0


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[squito]

Github user squito commented on the pull request:

https://github.com/apache/spark/pull/11152#issuecomment-211450907
  
Sorry I had forgotten about this.  Good question about just using line 
breaks instead of separate rows.  Here is a screen shot with separate rows:

https://cloud.githubusercontent.com/assets/71240/14611159/7d8bcb4c-0556-11e6-9b88-d52fb550786d.png";>

And here it is just with linebreaks:

https://cloud.githubusercontent.com/assets/71240/14611164/854a7fa4-0556-11e6-8e71-e00454216684.png";>

I don't really have a strong preference, so think maybe we should just go 
with `` for simplicity?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[squito]

Github user squito commented on a diff in the pull request:

https://github.com/apache/spark/pull/11152#discussion_r60086938
  
--- Diff: docs/monitoring.md ---
@@ -269,17 +310,23 @@ for a running application, at 
`http://localhost:4040/api/v1`.
 Details for the storage status of a given RDD
   
   
-/applications/[app-id]/logs
-Download the event logs for all attempts of the given application 
as a zip file
+/applications/[base-app-id]/logs
+Download the event logs for all attempts of the given application 
as a zip file.
+
   
   
-/applications/[app-id]/[attempt-id]/logs
-Download the event logs for the specified attempt of the given 
application as a zip file
+/applications/[base-app-id]/[attempt-id]/logs
+Download the event logs for a specific application as a zip 
file
--- End diff --

I think the idea is that in all other cases `[app-id]` *must* include the 
attempt-id when applicable as well.  However `/logs` is special, in that even 
if there are multiple attempts, there is still an end-point with just 
base-app-id for getting logs from all attempts.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/11152#discussion_r60034499
  
--- Diff: docs/monitoring.md ---
@@ -222,17 +222,40 @@ both running applications, and in the history server. 
 The endpoints are mounted
 for the history server, they would typically be accessible at 
`http://:18080/api/v1`, and
 for a running application, at `http://localhost:4040/api/v1`.
 
+In the API, an application is referenced by its application ID, `[app-id]`.
+When running on YARN, each application may have multiple attempts; each 
identified by their `[attempt-id]`.
+In the API listed below, `[app-id]` will actually be 
`[base-app-id]/[attempt-id]`,
+where `[base-app-id]` is the YARN application ID.
+
 
   EndpointMeaning
   
 /applications
 A list of all applications
   
   
+
+?status=[completed|running] list only applications in 
the chosen state
--- End diff --

And just sense-checking -- all these params are definitely already 
supported in 2.0, you're just documenting them?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/11152#discussion_r60034407
  
--- Diff: docs/monitoring.md ---
@@ -222,17 +222,40 @@ both running applications, and in the history server. 
 The endpoints are mounted
 for the history server, they would typically be accessible at 
`http://:18080/api/v1`, and
 for a running application, at `http://localhost:4040/api/v1`.
 
+In the API, an application is referenced by its application ID, `[app-id]`.
+When running on YARN, each application may have multiple attempts; each 
identified by their `[attempt-id]`.
+In the API listed below, `[app-id]` will actually be 
`[base-app-id]/[attempt-id]`,
+where `[base-app-id]` is the YARN application ID.
+
 
   EndpointMeaning
   
 /applications
 A list of all applications
   
   
+
+?status=[completed|running] list only applications in 
the chosen state
+  
+  
+
+?minDate=[date] earliest date/time to list. Examples:
--- End diff --

Formatting-wise, nit, can this be accomplished with `` or `` in one 
big `` rather than make extraneous rows, or does it just not look nice?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/11152#discussion_r60034324
  
--- Diff: docs/monitoring.md ---
@@ -269,17 +310,23 @@ for a running application, at 
`http://localhost:4040/api/v1`.
 Details for the storage status of a given RDD
   
   
-/applications/[app-id]/logs
-Download the event logs for all attempts of the given application 
as a zip file
+/applications/[base-app-id]/logs
+Download the event logs for all attempts of the given application 
as a zip file.
+
   
   
-/applications/[app-id]/[attempt-id]/logs
-Download the event logs for the specified attempt of the given 
application as a zip file
+/applications/[base-app-id]/[attempt-id]/logs
+Download the event logs for a specific application as a zip 
file
--- End diff --

Is this right, because in this form the attempt ID is specified?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[steveloughran]

Github user steveloughran commented on the pull request:

https://github.com/apache/spark/pull/11152#issuecomment-207759561
  
@squito have you had a chance to review this?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11152#issuecomment-195440805
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52927/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11152#issuecomment-195440802
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11152#issuecomment-195440652
  
**[Test build #52927 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52927/consoleFull)**
 for PR 11152 at commit 
[`2895a4f`](https://github.com/apache/spark/commit/2895a4f3da0cd9cc30e1db3dc1a7fdf00eff2149).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11152#issuecomment-195436124
  
**[Test build #52927 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52927/consoleFull)**
 for PR 11152 at commit 
[`2895a4f`](https://github.com/apache/spark/commit/2895a4f3da0cd9cc30e1db3dc1a7fdf00eff2149).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[steveloughran]

Github user steveloughran commented on the pull request:

https://github.com/apache/spark/pull/11152#issuecomment-195435446
  
Fixed up the endpoints, added some more detail on app-id vs app-attempt, 
using base-app-id for the log retrieval. Also mentioned that after job/stage 
GC, there are less to retrieve, and that you can increase the number and 
restart the HS, if you really want more



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[squito]

Github user squito commented on a diff in the pull request:

https://github.com/apache/spark/pull/11152#discussion_r55850527
  
--- Diff: docs/monitoring.md ---
@@ -273,8 +309,8 @@ for a running application, at 
`http://localhost:4040/api/v1`.
 Download the event logs for all attempts of the given application 
as a zip file
   
   
-/applications/[app-id]/[attempt-id]/logs
-Download the event logs for the specified attempt of the given 
application as a zip file
+/applications/[app-id]/logs
+Download the event logs for the application as a zip file
--- End diff --

ah good point.  So this case is a little different because both are 
supported and the one w/out `attemptId` is sometimes useful, to grab the logs 
for all attempts.  I thought the docs do actually spell this out, the first 
endpoint says it will download logs for all attempts.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[steveloughran]

Github user steveloughran commented on a diff in the pull request:

https://github.com/apache/spark/pull/11152#discussion_r55810810
  
--- Diff: docs/monitoring.md ---
@@ -273,8 +309,8 @@ for a running application, at 
`http://localhost:4040/api/v1`.
 Download the event logs for all attempts of the given application 
as a zip file
   
   
-/applications/[app-id]/[attempt-id]/logs
-Download the event logs for the specified attempt of the given 
application as a zip file
+/applications/[app-id]/logs
+Download the event logs for the application as a zip file
--- End diff --

OK, now i'm confusing, because the docs at the bottom say appId is 
`appid/attemptid` in yarn, which is true for every other element, which was why 
I adjusted this one. 

I'll try and clarify things more about appId vs attemptId, pull the details 
to the top and then have special mention here. As clearly it managed to confuse 
me.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[steveloughran]

Github user steveloughran commented on a diff in the pull request:

https://github.com/apache/spark/pull/11152#discussion_r55810591
  
--- Diff: docs/monitoring.md ---
@@ -229,10 +229,28 @@ for a running application, at 
`http://localhost:4040/api/v1`.
 A list of all applications
   
   
+ 
+ ?status=[completed|running] list only applications 
in the chosen state
+  
+  
+ 
+ ?minDate=[date] earliest date/time to list. Examples:
+ ?minDate=2015-02-10
--- End diff --

actually no, it MUST be `` without a tag, except in the specific case 
of XHTML, which nobody uses

1. [Mozilla](https://developer.mozilla.org/en/docs/Web/HTML/Element/br)
1. [MSFT](https://msdn.microsoft.com/en-us/library/ms535208(v=vs.85).aspx)

This is the same as the paragraph tag, {{}}.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[squito]

Github user squito commented on the pull request:

https://github.com/apache/spark/pull/11152#issuecomment-195015251
  
Sorry @steveloughran one more little issue!  I can update those too, but 
just want to make sure I wasn't missing some other intended change to the 
`logs` endpoints.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[squito]

Github user squito commented on a diff in the pull request:

https://github.com/apache/spark/pull/11152#discussion_r55737607
  
--- Diff: docs/monitoring.md ---
@@ -229,10 +229,28 @@ for a running application, at 
`http://localhost:4040/api/v1`.
 A list of all applications
   
   
+ 
+ ?status=[completed|running] list only applications 
in the chosen state
+  
+  
+ 
+ ?minDate=[date] earliest date/time to list. Examples:
+ ?minDate=2015-02-10
--- End diff --

super-nit: should these be ``?  I guess no browsers care about the `/` 
anymore, but might as well ...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[squito]

Github user squito commented on a diff in the pull request:

https://github.com/apache/spark/pull/11152#discussion_r55737357
  
--- Diff: docs/monitoring.md ---
@@ -273,8 +309,8 @@ for a running application, at 
`http://localhost:4040/api/v1`.
 Download the event logs for all attempts of the given application 
as a zip file
   
   
-/applications/[app-id]/[attempt-id]/logs
-Download the event logs for the specified attempt of the given 
application as a zip file
+/applications/[app-id]/logs
+Download the event logs for the application as a zip file
--- End diff --

this change looks wrong -- there are two endpoints, one for all attempts 
for the applications, and another for just one attempt.  The docs for the 
all-attempt endpoint is above this one.  with this change, its just duplicated.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[steveloughran]

Github user steveloughran commented on the pull request:

https://github.com/apache/spark/pull/11152#issuecomment-194948428
  
@squito : think this is ready for a merge now. I've been testing code 
against it and not found any inconsistencies


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11152#issuecomment-194429659
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11152#issuecomment-194429664
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52754/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11152#issuecomment-194429331
  
**[Test build #52754 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52754/consoleFull)**
 for PR 11152 at commit 
[`e4e5ae8`](https://github.com/apache/spark/commit/e4e5ae8bcde2fffb00752c71592ff049eb833e04).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11152#issuecomment-194424502
  
**[Test build #52754 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52754/consoleFull)**
 for PR 11152 at commit 
[`e4e5ae8`](https://github.com/apache/spark/commit/e4e5ae8bcde2fffb00752c71592ff049eb833e04).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11152#issuecomment-183342174
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11152#issuecomment-183342176
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51190/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11152#issuecomment-183342048
  
**[Test build #51190 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51190/consoleFull)**
 for PR 11152 at commit 
[`15c675d`](https://github.com/apache/spark/commit/15c675d8b9fbfaa47e2915ad0075ebe03f1b5bcf).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11152#issuecomment-183335489
  
**[Test build #51190 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51190/consoleFull)**
 for PR 11152 at commit 
[`15c675d`](https://github.com/apache/spark/commit/15c675d8b9fbfaa47e2915ad0075ebe03f1b5bcf).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[steveloughran]

Github user steveloughran commented on the pull request:

https://github.com/apache/spark/pull/11152#issuecomment-183331075
  

  EndpointMeaning
  
/applications
A list of all applications
  
  
 
 ?status=[completed|running] list only applications in 
the chosen state
  
  
 
 ?minDate=[date] earliest date/time to list. Examples:
 ?minDate=2015-02-10
 ?minDate=2015-02-03T16:42:40.000GMT
  
  
 
 ?maxDate=[date] latest date/time to list; uses same 
format as minDate
  
  
/applications/[app-id]/jobs
A list of all jobs for a given application
  
  
 
 ?status=[complete|succeeded|failed] list only jobs in 
the state
  
  
/applications/[app-id]/jobs/[job-id]
Details for the given job
  
  
/applications/[app-id]/stages
A list of all stages for a given application
  
  
/applications/[app-id]/stages/[stage-id]
A list of all attempts for the given stage
  
  
 
 ?status=[active|complete|pending|failed] list only 
stages in the state
  
  

/applications/[app-id]/stages/[stage-id]/[stage-attempt-id]
Details for the given stage attempt
  
  

/applications/[app-id]/stages/[stage-id]/[stage-attempt-id]/taskSummary
Summary metrics of all tasks in the given stage attempt
  
  
 
 ?quantiles summarize the metrics with the given 
quantiles.
 Example: ?quantiles=0.01,0.5,0.99
  
  

/applications/[app-id]/stages/[stage-id]/[stage-attempt-id]/taskList
A list of all tasks for the given stage attempt
  
  
 
 ?offset=[offset]&length=[len] list only tasks in 
the range.
  Example:  ?offset=10&length=50
  
  
 
 ?sortBy=[runtime|-runtime] sort the tasks
  
  
/applications/[app-id]/executors
A list of all executors for the given application
  
  
/applications/[app-id]/storage/rdd
A list of stored RDDs for the given application
  
  
/applications/[app-id]/storage/rdd/[rdd-id]
Details for the storage status of a given RDD
  
  
/applications/[app-id]/logs
Download the event logs for all attempts of the given application 
as a zip file
  
  
/applications/[app-id]/[attempt-id]/logs
Download the event logs for the specified attempt of the given 
application as a zip file
  




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[steveloughran]

Github user steveloughran commented on the pull request:

https://github.com/apache/spark/pull/11152#issuecomment-183330853
  
done. Also put the examples on new lines with a  tag


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[squito]

Github user squito commented on the pull request:

https://github.com/apache/spark/pull/11152#issuecomment-182957793
  
lgtm other than one minor comment


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[squito]

Github user squito commented on a diff in the pull request:

https://github.com/apache/spark/pull/11152#discussion_r52630690
  
--- Diff: docs/monitoring.md ---
@@ -189,10 +189,26 @@ for a running application, at 
`http://localhost:4040/api/v1`.
 A list of all applications
   
   
+ 
+ ?status=[completed|running] list only applications 
in the state
+  
+  
+ 
+ ?minDate=[date] earliest date to list. e.g 
?minDate=2015-02-10
+  
+  
+ 
+ ?maxDate=[date] latest date to list. e.g 
?maxDate=2015-02-03T16:42:40.000GMT
--- End diff --

maybe mention both date formats under `minDate`, and then just say this can 
take the same formats as `minDate`?  It should be pretty obvious that both 
versions work for either option, but just to be a little more clear.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[squito]

Github user squito commented on the pull request:

https://github.com/apache/spark/pull/11152#issuecomment-182956764
  
Here's a screenshot of what this looks like:

https://cloud.githubusercontent.com/assets/71240/12982980/e01be154-d0ac-11e5-94b1-3921c24298e5.png";>

Its not wonderful, but seems pretty reasonable without some bigger 
restructuring of the page.  (Eg. as one random example, see this: 
https://docs.atlassian.com/jira/REST/latest/#api/2/-getPermissions  But that is 
clearly a bigger change.)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11152#issuecomment-182480912
  
**[Test build #51037 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51037/consoleFull)**
 for PR 11152 at commit 
[`bf1a128`](https://github.com/apache/spark/commit/bf1a128de6f2af3f7bba2aef6d546fcca1aaa19b).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11152#issuecomment-182481088
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11152#issuecomment-182481089
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51037/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11152#issuecomment-182475512
  
**[Test build #51037 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51037/consoleFull)**
 for PR 11152 at commit 
[`bf1a128`](https://github.com/apache/spark/commit/bf1a128de6f2af3f7bba2aef6d546fcca1aaa19b).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13267] [Web UI] document the ?param arg...

[steveloughran]

GitHub user steveloughran opened a pull request:

https://github.com/apache/spark/pull/11152

[SPARK-13267] [Web UI] document the ?param arguments of the REST API; lift 
the…

Add to the REST API details on the ? args and examples from the test suite.

I've used the existing table, adding all the fields to the second table.

see [in the 
pr](https://github.com/steveloughran/spark/blob/history/SPARK-13267-doc-params/docs/monitoring.md).

There's a slightly more sophisticated option: make the table 3 columns 
wide, and for all existing entries, have the initial `td` span 2 columns. The 
new entries would then have an empty 1st column, param in 2nd and text in 3rd, 
with any examples after a `br` entry.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/steveloughran/spark 
history/SPARK-13267-doc-params

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/11152.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #11152


commit bf1a128de6f2af3f7bba2aef6d546fcca1aaa19b
Author: Steve Loughran 
Date:   2016-02-10T16:27:12Z

[SPARK-13267] document the ?param arguments of the REST API; lift the 
examples from the test suite




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org