[jira] [Commented] (SPARK-13241) add long--formatted timestamps to org.apache.spark.status.api.v1.ApplicationAttemptInfo
[
https://issues.apache.org/jira/browse/SPARK-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15159653#comment-15159653
]
Apache Spark commented on SPARK-13241:
--
User 'ajbozarth' has created a pull request for this issue:
https://github.com/apache/spark/pull/11326
> add long--formatted timestamps to
> org.apache.spark.status.api.v1.ApplicationAttemptInfo
> ---
>
> Key: SPARK-13241
> URL: https://issues.apache.org/jira/browse/SPARK-13241
> Project: Spark
> Issue Type: Improvement
> Components: Web UI
>Affects Versions: 1.6.0
>Reporter: Steve Loughran
>
> While writing tests to parse
> {{org.apache.spark.status.api.v1.ApplicationAttemptInfo}} coming off the
> history rest server, I can see that I can't actually unmarshall the
> timestamps without parsing the strings —and I fear that may be local specific.
> The problem here is that jersey is marshalling {{Date}} classes by calling
> {{Date.toString()}}, leaving an entry like
> {code}
> {
> "id": "application__",
> "name": "spark-demo",
> "attempts": [
> {
> "attemptId": "1",
> "startTime": "2016-02-08T20:12:20.825GMT",
> "endTime": "1970-01-01T00:00:00.000GMT",
> "lastUpdated": "2016-02-08T20:12:20.848GMT",
> "duration": 0,
> "sparkUser": "hdfs",
> "completed": false
> }
> ]
> }
> {code}
> This is good for humans and the web UI, bad for any code trying to use the
> API for its own purposes.
> I propose adding, alongside the existing values, a simple {{long}} value for
> each time, representing the time in millis from the start of the epoch. This
> is trivial to marshall/unmarshall.
> This is easy to add, provided everyone is happy that adding a new field to
> the returned JSON is considered compatible with the existing REST API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13241) add long--formatted timestamps to org.apache.spark.status.api.v1.ApplicationAttemptInfo
[
https://issues.apache.org/jira/browse/SPARK-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15158069#comment-15158069
]
Alex Bozarth commented on SPARK-13241:
--
I have a fix for this in the works, PR will come sometime tomorrow. We can
continue on discussion there unless someone wants to veto this now
> add long--formatted timestamps to
> org.apache.spark.status.api.v1.ApplicationAttemptInfo
> ---
>
> Key: SPARK-13241
> URL: https://issues.apache.org/jira/browse/SPARK-13241
> Project: Spark
> Issue Type: Improvement
> Components: Web UI
>Affects Versions: 1.6.0
>Reporter: Steve Loughran
>
> While writing tests to parse
> {{org.apache.spark.status.api.v1.ApplicationAttemptInfo}} coming off the
> history rest server, I can see that I can't actually unmarshall the
> timestamps without parsing the strings —and I fear that may be local specific.
> The problem here is that jersey is marshalling {{Date}} classes by calling
> {{Date.toString()}}, leaving an entry like
> {code}
> {
> "id": "application__",
> "name": "spark-demo",
> "attempts": [
> {
> "attemptId": "1",
> "startTime": "2016-02-08T20:12:20.825GMT",
> "endTime": "1970-01-01T00:00:00.000GMT",
> "lastUpdated": "2016-02-08T20:12:20.848GMT",
> "duration": 0,
> "sparkUser": "hdfs",
> "completed": false
> }
> ]
> }
> {code}
> This is good for humans and the web UI, bad for any code trying to use the
> API for its own purposes.
> I propose adding, alongside the existing values, a simple {{long}} value for
> each time, representing the time in millis from the start of the epoch. This
> is trivial to marshall/unmarshall.
> This is easy to add, provided everyone is happy that adding a new field to
> the returned JSON is considered compatible with the existing REST API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13241) add long--formatted timestamps to org.apache.spark.status.api.v1.ApplicationAttemptInfo
[
https://issues.apache.org/jira/browse/SPARK-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15154052#comment-15154052
]
Steve Loughran commented on SPARK-13241:
The REST API compatibility is described in
[monitoring|http://spark.apache.org/docs/latest/monitoring.html]. It explicitly
states
# Individual fields will never be removed for any given endpoint
# New fields may be added to existing endpoints
You can't change the existing field without breaking statement #1 and upsetting
people; but statement #2 provides the strategy for addressing this.
a {{long}} timestamp could have to go alongside the date. More of an issue than
the duplicate code is "what does a recipient do if they are different"; I'd go
for making the long normative, though even there there's the eternal ambiguity
over timezones, something probably ignored by now.
> add long--formatted timestamps to
> org.apache.spark.status.api.v1.ApplicationAttemptInfo
> ---
>
> Key: SPARK-13241
> URL: https://issues.apache.org/jira/browse/SPARK-13241
> Project: Spark
> Issue Type: Improvement
> Components: Web UI
>Affects Versions: 1.6.0
>Reporter: Steve Loughran
>
> While writing tests to parse
> {{org.apache.spark.status.api.v1.ApplicationAttemptInfo}} coming off the
> history rest server, I can see that I can't actually unmarshall the
> timestamps without parsing the strings —and I fear that may be local specific.
> The problem here is that jersey is marshalling {{Date}} classes by calling
> {{Date.toString()}}, leaving an entry like
> {code}
> {
> "id": "application__",
> "name": "spark-demo",
> "attempts": [
> {
> "attemptId": "1",
> "startTime": "2016-02-08T20:12:20.825GMT",
> "endTime": "1970-01-01T00:00:00.000GMT",
> "lastUpdated": "2016-02-08T20:12:20.848GMT",
> "duration": 0,
> "sparkUser": "hdfs",
> "completed": false
> }
> ]
> }
> {code}
> This is good for humans and the web UI, bad for any code trying to use the
> API for its own purposes.
> I propose adding, alongside the existing values, a simple {{long}} value for
> each time, representing the time in millis from the start of the epoch. This
> is trivial to marshall/unmarshall.
> This is easy to add, provided everyone is happy that adding a new field to
> the returned JSON is considered compatible with the existing REST API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13241) add long--formatted timestamps to org.apache.spark.status.api.v1.ApplicationAttemptInfo
[
https://issues.apache.org/jira/browse/SPARK-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153677#comment-15153677
]
Alex Bozarth commented on SPARK-13241:
--
Ok, that makes sense, I've made parameter additions to ExecutorSummary in the
API previously but I believe those only affected the web ui and not external
use cases.
> add long--formatted timestamps to
> org.apache.spark.status.api.v1.ApplicationAttemptInfo
> ---
>
> Key: SPARK-13241
> URL: https://issues.apache.org/jira/browse/SPARK-13241
> Project: Spark
> Issue Type: Improvement
> Components: Web UI
>Affects Versions: 1.6.0
>Reporter: Steve Loughran
>
> While writing tests to parse
> {{org.apache.spark.status.api.v1.ApplicationAttemptInfo}} coming off the
> history rest server, I can see that I can't actually unmarshall the
> timestamps without parsing the strings —and I fear that may be local specific.
> The problem here is that jersey is marshalling {{Date}} classes by calling
> {{Date.toString()}}, leaving an entry like
> {code}
> {
> "id": "application__",
> "name": "spark-demo",
> "attempts": [
> {
> "attemptId": "1",
> "startTime": "2016-02-08T20:12:20.825GMT",
> "endTime": "1970-01-01T00:00:00.000GMT",
> "lastUpdated": "2016-02-08T20:12:20.848GMT",
> "duration": 0,
> "sparkUser": "hdfs",
> "completed": false
> }
> ]
> }
> {code}
> This is good for humans and the web UI, bad for any code trying to use the
> API for its own purposes.
> I propose adding, alongside the existing values, a simple {{long}} value for
> each time, representing the time in millis from the start of the epoch. This
> is trivial to marshall/unmarshall.
> This is easy to add, provided everyone is happy that adding a new field to
> the returned JSON is considered compatible with the existing REST API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13241) add long--formatted timestamps to org.apache.spark.status.api.v1.ApplicationAttemptInfo
[
https://issues.apache.org/jira/browse/SPARK-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153674#comment-15153674
]
Reynold Xin commented on SPARK-13241:
-
I think this API is versioned (v1) because we want to make sure there is some
stability here for external tools.
> add long--formatted timestamps to
> org.apache.spark.status.api.v1.ApplicationAttemptInfo
> ---
>
> Key: SPARK-13241
> URL: https://issues.apache.org/jira/browse/SPARK-13241
> Project: Spark
> Issue Type: Improvement
> Components: Web UI
>Affects Versions: 1.6.0
>Reporter: Steve Loughran
>
> While writing tests to parse
> {{org.apache.spark.status.api.v1.ApplicationAttemptInfo}} coming off the
> history rest server, I can see that I can't actually unmarshall the
> timestamps without parsing the strings —and I fear that may be local specific.
> The problem here is that jersey is marshalling {{Date}} classes by calling
> {{Date.toString()}}, leaving an entry like
> {code}
> {
> "id": "application__",
> "name": "spark-demo",
> "attempts": [
> {
> "attemptId": "1",
> "startTime": "2016-02-08T20:12:20.825GMT",
> "endTime": "1970-01-01T00:00:00.000GMT",
> "lastUpdated": "2016-02-08T20:12:20.848GMT",
> "duration": 0,
> "sparkUser": "hdfs",
> "completed": false
> }
> ]
> }
> {code}
> This is good for humans and the web UI, bad for any code trying to use the
> API for its own purposes.
> I propose adding, alongside the existing values, a simple {{long}} value for
> each time, representing the time in millis from the start of the epoch. This
> is trivial to marshall/unmarshall.
> This is easy to add, provided everyone is happy that adding a new field to
> the returned JSON is considered compatible with the existing REST API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13241) add long--formatted timestamps to org.apache.spark.status.api.v1.ApplicationAttemptInfo
[
https://issues.apache.org/jira/browse/SPARK-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153671#comment-15153671
]
Alex Bozarth commented on SPARK-13241:
--
I'm still relatively new to this, but wouldn't we just need to update all the
internal calls to that class to include the new fields? Or does adding them
fundamentally break the usage of it in ways I haven't learned?
> add long--formatted timestamps to
> org.apache.spark.status.api.v1.ApplicationAttemptInfo
> ---
>
> Key: SPARK-13241
> URL: https://issues.apache.org/jira/browse/SPARK-13241
> Project: Spark
> Issue Type: Improvement
> Components: Web UI
>Affects Versions: 1.6.0
>Reporter: Steve Loughran
>
> While writing tests to parse
> {{org.apache.spark.status.api.v1.ApplicationAttemptInfo}} coming off the
> history rest server, I can see that I can't actually unmarshall the
> timestamps without parsing the strings —and I fear that may be local specific.
> The problem here is that jersey is marshalling {{Date}} classes by calling
> {{Date.toString()}}, leaving an entry like
> {code}
> {
> "id": "application__",
> "name": "spark-demo",
> "attempts": [
> {
> "attemptId": "1",
> "startTime": "2016-02-08T20:12:20.825GMT",
> "endTime": "1970-01-01T00:00:00.000GMT",
> "lastUpdated": "2016-02-08T20:12:20.848GMT",
> "duration": 0,
> "sparkUser": "hdfs",
> "completed": false
> }
> ]
> }
> {code}
> This is good for humans and the web UI, bad for any code trying to use the
> API for its own purposes.
> I propose adding, alongside the existing values, a simple {{long}} value for
> each time, representing the time in millis from the start of the epoch. This
> is trivial to marshall/unmarshall.
> This is easy to add, provided everyone is happy that adding a new field to
> the returned JSON is considered compatible with the existing REST API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13241) add long--formatted timestamps to org.apache.spark.status.api.v1.ApplicationAttemptInfo
[
https://issues.apache.org/jira/browse/SPARK-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153571#comment-15153571
]
Reynold Xin commented on SPARK-13241:
-
Won't this break the protocol?
> add long--formatted timestamps to
> org.apache.spark.status.api.v1.ApplicationAttemptInfo
> ---
>
> Key: SPARK-13241
> URL: https://issues.apache.org/jira/browse/SPARK-13241
> Project: Spark
> Issue Type: Improvement
> Components: Web UI
>Affects Versions: 1.6.0
>Reporter: Steve Loughran
>
> While writing tests to parse
> {{org.apache.spark.status.api.v1.ApplicationAttemptInfo}} coming off the
> history rest server, I can see that I can't actually unmarshall the
> timestamps without parsing the strings —and I fear that may be local specific.
> The problem here is that jersey is marshalling {{Date}} classes by calling
> {{Date.toString()}}, leaving an entry like
> {code}
> {
> "id": "application__",
> "name": "spark-demo",
> "attempts": [
> {
> "attemptId": "1",
> "startTime": "2016-02-08T20:12:20.825GMT",
> "endTime": "1970-01-01T00:00:00.000GMT",
> "lastUpdated": "2016-02-08T20:12:20.848GMT",
> "duration": 0,
> "sparkUser": "hdfs",
> "completed": false
> }
> ]
> }
> {code}
> This is good for humans and the web UI, bad for any code trying to use the
> API for its own purposes.
> I propose adding, alongside the existing values, a simple {{long}} value for
> each time, representing the time in millis from the start of the epoch. This
> is trivial to marshall/unmarshall.
> This is easy to add, provided everyone is happy that adding a new field to
> the returned JSON is considered compatible with the existing REST API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13241) add long--formatted timestamps to org.apache.spark.status.api.v1.ApplicationAttemptInfo
[
https://issues.apache.org/jira/browse/SPARK-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153040#comment-15153040
]
Alex Bozarth commented on SPARK-13241:
--
Looked into this and I could update the api to include this, but before I go
through the leg work, do we actually want it?
[~joshrosen] [~rxin] since you're in charge of the API, is adding the date
params as {{long}}s in {{ApplicationAttemptInfo}} something we actually want to
do?
It's definitely useful, but the duplicate data clusters up the api a bit.
> add long--formatted timestamps to
> org.apache.spark.status.api.v1.ApplicationAttemptInfo
> ---
>
> Key: SPARK-13241
> URL: https://issues.apache.org/jira/browse/SPARK-13241
> Project: Spark
> Issue Type: Improvement
> Components: Web UI
>Affects Versions: 1.6.0
>Reporter: Steve Loughran
>
> While writing tests to parse
> {{org.apache.spark.status.api.v1.ApplicationAttemptInfo}} coming off the
> history rest server, I can see that I can't actually unmarshall the
> timestamps without parsing the strings —and I fear that may be local specific.
> The problem here is that jersey is marshalling {{Date}} classes by calling
> {{Date.toString()}}, leaving an entry like
> {code}
> {
> "id": "application__",
> "name": "spark-demo",
> "attempts": [
> {
> "attemptId": "1",
> "startTime": "2016-02-08T20:12:20.825GMT",
> "endTime": "1970-01-01T00:00:00.000GMT",
> "lastUpdated": "2016-02-08T20:12:20.848GMT",
> "duration": 0,
> "sparkUser": "hdfs",
> "completed": false
> }
> ]
> }
> {code}
> This is good for humans and the web UI, bad for any code trying to use the
> API for its own purposes.
> I propose adding, alongside the existing values, a simple {{long}} value for
> each time, representing the time in millis from the start of the epoch. This
> is trivial to marshall/unmarshall.
> This is easy to add, provided everyone is happy that adding a new field to
> the returned JSON is considered compatible with the existing REST API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
