[jira] [Updated] (SPARK-31440) Improve SQL Rest API
[ https://issues.apache.org/jira/browse/SPARK-31440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-31440: Fix Version/s: (was: 3.0.0) 3.1.0 > Improve SQL Rest API > > > Key: SPARK-31440 > URL: https://issues.apache.org/jira/browse/SPARK-31440 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.1.0 >Reporter: Eren Avsarogullari >Assignee: Eren Avsarogullari >Priority: Major > Fix For: 3.1.0 > > Attachments: current_version.json, improved_version.json, > improved_version_May1th.json > > > SQL Rest API exposes query execution metrics as Public API. This Jira aims to > apply following improvements on SQL Rest API by aligning Spark-UI. > *Proposed Improvements:* > 1- Support Physical Operations and group metrics per physical operation by > aligning Spark UI. > 2- Support *wholeStageCodegenId* for Physical Operations > 3- *nodeId* can be useful for grouping metrics and sorting physical > operations (according to execution order) to differentiate same operators (if > used multiple times during the same query execution) and their metrics. > 4- Filter *empty* metrics by aligning with Spark UI - SQL Tab. Currently, > Spark UI does not show empty metrics. > 5- Remove line breakers(*\n*) from *metricValue*. > 6- *planDescription* can be *optional* Http parameter to avoid network cost > where there is specially complex jobs creating big-plans. > 7- *metrics* attribute needs to be exposed at the bottom order as > *metricDetails*. Specially, this can be useful for the user where > *metricDetails* array size is high. > 8- Reverse order on *metricDetails* aims to match with Spark UI by supporting > Physical Operators' execution order. > *Attachments:* > Please find both *current* and *improved* versions of the results as > attached for following SQL Rest Endpoint: > {code:java} > curl -X GET > http://localhost:4040/api/v1/applications/$appId/sql/$executionId?details=true{code} > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-31440) Improve SQL Rest API
[ https://issues.apache.org/jira/browse/SPARK-31440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-31440: - Fix Version/s: 3.0.0 > Improve SQL Rest API > > > Key: SPARK-31440 > URL: https://issues.apache.org/jira/browse/SPARK-31440 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.1.0 >Reporter: Eren Avsarogullari >Assignee: Eren Avsarogullari >Priority: Major > Fix For: 3.0.0 > > Attachments: current_version.json, improved_version.json, > improved_version_May1th.json > > > SQL Rest API exposes query execution metrics as Public API. This Jira aims to > apply following improvements on SQL Rest API by aligning Spark-UI. > *Proposed Improvements:* > 1- Support Physical Operations and group metrics per physical operation by > aligning Spark UI. > 2- Support *wholeStageCodegenId* for Physical Operations > 3- *nodeId* can be useful for grouping metrics and sorting physical > operations (according to execution order) to differentiate same operators (if > used multiple times during the same query execution) and their metrics. > 4- Filter *empty* metrics by aligning with Spark UI - SQL Tab. Currently, > Spark UI does not show empty metrics. > 5- Remove line breakers(*\n*) from *metricValue*. > 6- *planDescription* can be *optional* Http parameter to avoid network cost > where there is specially complex jobs creating big-plans. > 7- *metrics* attribute needs to be exposed at the bottom order as > *metricDetails*. Specially, this can be useful for the user where > *metricDetails* array size is high. > 8- Reverse order on *metricDetails* aims to match with Spark UI by supporting > Physical Operators' execution order. > *Attachments:* > Please find both *current* and *improved* versions of the results as > attached for following SQL Rest Endpoint: > {code:java} > curl -X GET > http://localhost:4040/api/v1/applications/$appId/sql/$executionId?details=true{code} > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-31440) Improve SQL Rest API
[ https://issues.apache.org/jira/browse/SPARK-31440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eren Avsarogullari updated SPARK-31440: --- Attachment: improved_version_May1th.json > Improve SQL Rest API > > > Key: SPARK-31440 > URL: https://issues.apache.org/jira/browse/SPARK-31440 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.1.0 >Reporter: Eren Avsarogullari >Priority: Major > Attachments: current_version.json, improved_version.json, > improved_version_May1th.json > > > SQL Rest API exposes query execution metrics as Public API. This Jira aims to > apply following improvements on SQL Rest API by aligning Spark-UI. > *Proposed Improvements:* > 1- Support Physical Operations and group metrics per physical operation by > aligning Spark UI. > 2- Support *wholeStageCodegenId* for Physical Operations > 3- *nodeId* can be useful for grouping metrics and sorting physical > operations (according to execution order) to differentiate same operators (if > used multiple times during the same query execution) and their metrics. > 4- Filter *empty* metrics by aligning with Spark UI - SQL Tab. Currently, > Spark UI does not show empty metrics. > 5- Remove line breakers(*\n*) from *metricValue*. > 6- *planDescription* can be *optional* Http parameter to avoid network cost > where there is specially complex jobs creating big-plans. > 7- *metrics* attribute needs to be exposed at the bottom order as > *metricDetails*. Specially, this can be useful for the user where > *metricDetails* array size is high. > 8- Reverse order on *metricDetails* aims to match with Spark UI by supporting > Physical Operators' execution order. > *Attachments:* > Please find both *current* and *improved* versions of the results as > attached for following SQL Rest Endpoint: > {code:java} > curl -X GET > http://localhost:4040/api/v1/applications/$appId/sql/$executionId?details=true{code} > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-31440) Improve SQL Rest API
[ https://issues.apache.org/jira/browse/SPARK-31440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eren Avsarogullari updated SPARK-31440: --- Attachment: (was: improved_version.json) > Improve SQL Rest API > > > Key: SPARK-31440 > URL: https://issues.apache.org/jira/browse/SPARK-31440 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.1.0 >Reporter: Eren Avsarogullari >Priority: Major > Attachments: current_version.json, improved_version.json > > > SQL Rest API exposes query execution metrics as Public API. This Jira aims to > apply following improvements on SQL Rest API by aligning Spark-UI. > *Proposed Improvements:* > 1- Support Physical Operations and group metrics per physical operation by > aligning Spark UI. > 2- Support *wholeStageCodegenId* for Physical Operations > 3- *nodeId* can be useful for grouping metrics and sorting physical > operations (according to execution order) to differentiate same operators (if > used multiple times during the same query execution) and their metrics. > 4- Filter *empty* metrics by aligning with Spark UI - SQL Tab. Currently, > Spark UI does not show empty metrics. > 5- Remove line breakers(*\n*) from *metricValue*. > 6- *planDescription* can be *optional* Http parameter to avoid network cost > where there is specially complex jobs creating big-plans. > 7- *metrics* attribute needs to be exposed at the bottom order as > *metricDetails*. Specially, this can be useful for the user where > *metricDetails* array size is high. > 8- Reverse order on *metricDetails* aims to match with Spark UI by supporting > Physical Operators' execution order. > *Attachments:* > Please find both *current* and *improved* versions of the results as > attached for following SQL Rest Endpoint: > {code:java} > curl -X GET > http://localhost:4040/api/v1/applications/$appId/sql/$executionId?details=true{code} > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-31440) Improve SQL Rest API
[ https://issues.apache.org/jira/browse/SPARK-31440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eren Avsarogullari updated SPARK-31440: --- Attachment: improved_version.json > Improve SQL Rest API > > > Key: SPARK-31440 > URL: https://issues.apache.org/jira/browse/SPARK-31440 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.1.0 >Reporter: Eren Avsarogullari >Priority: Major > Attachments: current_version.json, improved_version.json > > > SQL Rest API exposes query execution metrics as Public API. This Jira aims to > apply following improvements on SQL Rest API by aligning Spark-UI. > *Proposed Improvements:* > 1- Support Physical Operations and group metrics per physical operation by > aligning Spark UI. > 2- Support *wholeStageCodegenId* for Physical Operations > 3- *nodeId* can be useful for grouping metrics and sorting physical > operations (according to execution order) to differentiate same operators (if > used multiple times during the same query execution) and their metrics. > 4- Filter *empty* metrics by aligning with Spark UI - SQL Tab. Currently, > Spark UI does not show empty metrics. > 5- Remove line breakers(*\n*) from *metricValue*. > 6- *planDescription* can be *optional* Http parameter to avoid network cost > where there is specially complex jobs creating big-plans. > 7- *metrics* attribute needs to be exposed at the bottom order as > *metricDetails*. Specially, this can be useful for the user where > *metricDetails* array size is high. > 8- Reverse order on *metricDetails* aims to match with Spark UI by supporting > Physical Operators' execution order. > *Attachments:* > Please find both *current* and *improved* versions of the results as > attached for following SQL Rest Endpoint: > {code:java} > curl -X GET > http://localhost:4040/api/v1/applications/$appId/sql/$executionId?details=true{code} > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-31440) Improve SQL Rest API
[ https://issues.apache.org/jira/browse/SPARK-31440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eren Avsarogullari updated SPARK-31440: --- Description: SQL Rest API exposes query execution metrics as Public API. This Jira aims to apply following improvements on SQL Rest API by aligning Spark-UI. *Proposed Improvements:* 1- Support Physical Operations and group metrics per physical operation by aligning Spark UI. 2- Support *wholeStageCodegenId* for Physical Operations 3- *nodeId* can be useful for grouping metrics and sorting physical operations (according to execution order) to differentiate same operators (if used multiple times during the same query execution) and their metrics. 4- Filter *empty* metrics by aligning with Spark UI - SQL Tab. Currently, Spark UI does not show empty metrics. 5- Remove line breakers(*\n*) from *metricValue*. 6- *planDescription* can be *optional* Http parameter to avoid network cost where there is specially complex jobs creating big-plans. 7- *metrics* attribute needs to be exposed at the bottom order as *metricDetails*. Specially, this can be useful for the user where *metricDetails* array size is high. 8- Reverse order on *metricDetails* aims to match with Spark UI by supporting Physical Operators' execution order. *Attachments:* Please find both *current* and *improved* versions of the results as attached for following SQL Rest Endpoint: {code:java} curl -X GET http://localhost:4040/api/v1/applications/$appId/sql/$executionId?details=true{code} was: SQL Rest API exposes query execution metrics as Public API. This Jira aims to apply following improvements on SQL Rest API by aligning Spark-UI. *Proposed Improvements:* 1- Support Physical Operations and group metrics per operation by aligning Spark UI. 2- *nodeId* can be useful for grouping metrics as well as for sorting and to differentiate same operators and their metrics. 3- Filter *blank* metrics by aligning with Spark UI - SQL Tab 4- Remove *\n* from *metricValue(s)* 5- *planDescription* can be optional Http parameter to avoid network cost (specially for complex jobs creating big-plans). 6- *metrics* attribute needs to be exposed at the bottom order as *metricDetails*. This order matches with Spark UI by highlighting with execution order. *Attachments:* Please find both *current* and *improved* versions of the results as attached for following SQL Rest Endpoint: {code:java} curl -X GET http://localhost:4040/api/v1/applications/$appId/sql/$executionId?details=true{code} > Improve SQL Rest API > > > Key: SPARK-31440 > URL: https://issues.apache.org/jira/browse/SPARK-31440 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.1.0 >Reporter: Eren Avsarogullari >Priority: Major > Attachments: current_version.json, improved_version.json > > > SQL Rest API exposes query execution metrics as Public API. This Jira aims to > apply following improvements on SQL Rest API by aligning Spark-UI. > *Proposed Improvements:* > 1- Support Physical Operations and group metrics per physical operation by > aligning Spark UI. > 2- Support *wholeStageCodegenId* for Physical Operations > 3- *nodeId* can be useful for grouping metrics and sorting physical > operations (according to execution order) to differentiate same operators (if > used multiple times during the same query execution) and their metrics. > 4- Filter *empty* metrics by aligning with Spark UI - SQL Tab. Currently, > Spark UI does not show empty metrics. > 5- Remove line breakers(*\n*) from *metricValue*. > 6- *planDescription* can be *optional* Http parameter to avoid network cost > where there is specially complex jobs creating big-plans. > 7- *metrics* attribute needs to be exposed at the bottom order as > *metricDetails*. Specially, this can be useful for the user where > *metricDetails* array size is high. > 8- Reverse order on *metricDetails* aims to match with Spark UI by supporting > Physical Operators' execution order. > *Attachments:* > Please find both *current* and *improved* versions of the results as > attached for following SQL Rest Endpoint: > {code:java} > curl -X GET > http://localhost:4040/api/v1/applications/$appId/sql/$executionId?details=true{code} > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-31440) Improve SQL Rest API
[ https://issues.apache.org/jira/browse/SPARK-31440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eren Avsarogullari updated SPARK-31440: --- Description: SQL Rest API exposes query execution metrics as Public API. This Jira aims to apply following improvements on SQL Rest API by aligning Spark-UI. *Proposed Improvements:* 1- Support Physical Operations and group metrics per operation by aligning Spark UI. 2- *nodeId* can be useful for grouping metrics as well as for sorting and to differentiate same operators and their metrics. 3- Filter *blank* metrics by aligning with Spark UI - SQL Tab 4- Remove *\n* from *metricValue(s)* 5- *planDescription* can be optional Http parameter to avoid network cost (specially for complex jobs creating big-plans). 6- *metrics* attribute needs to be exposed at the bottom order as *metricDetails*. This order matches with Spark UI by highlighting with execution order. *Attachments:* Please find both *current* and *improved* versions of the results as attached for following SQL Rest Endpoint: {code:java} curl -X GET http://localhost:4040/api/v1/applications/$appId/sql/$executionId?details=true{code} was: SQL Rest API exposes query execution metrics as Public API. This Jira aims to apply following improvements on SQL Rest API by aligning Spark-UI. *Proposed Improvements:* 1- Support Physical Operations and group metrics per operation by aligning Spark UI. 2- *nodeId* can be useful for grouping metrics as well as for sorting and to differentiate same operators and their metrics. 3- Filter *blank* metrics by aligning with Spark UI - SQL Tab 4- Remove *\n* from *metricValue(s)* 5- *planDescription* can be optional Http parameter to avoid network cost (specially for complex jobs creating big-plans). 6- *metrics* attribute needs to be exposed at the bottom order as *metricDetails*. This order matches with Spark UI by highlighting with execution order. *Attachments:* Please find both *current* and *improved* versions of results as attached. > Improve SQL Rest API > > > Key: SPARK-31440 > URL: https://issues.apache.org/jira/browse/SPARK-31440 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.1.0 >Reporter: Eren Avsarogullari >Priority: Major > Attachments: current_version.json, improved_version.json > > > SQL Rest API exposes query execution metrics as Public API. This Jira aims to > apply following improvements on SQL Rest API by aligning Spark-UI. > *Proposed Improvements:* > 1- Support Physical Operations and group metrics per operation by aligning > Spark UI. > 2- *nodeId* can be useful for grouping metrics as well as for sorting and to > differentiate same operators and their metrics. > 3- Filter *blank* metrics by aligning with Spark UI - SQL Tab > 4- Remove *\n* from *metricValue(s)* > 5- *planDescription* can be optional Http parameter to avoid network cost > (specially for complex jobs creating big-plans). > 6- *metrics* attribute needs to be exposed at the bottom order as > *metricDetails*. This order matches with Spark UI by highlighting with > execution order. > *Attachments:* > Please find both *current* and *improved* versions of the results as > attached for following SQL Rest Endpoint: > {code:java} > curl -X GET > http://localhost:4040/api/v1/applications/$appId/sql/$executionId?details=true{code} > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-31440) Improve SQL Rest API
[ https://issues.apache.org/jira/browse/SPARK-31440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eren Avsarogullari updated SPARK-31440: --- Attachment: current_version.json > Improve SQL Rest API > > > Key: SPARK-31440 > URL: https://issues.apache.org/jira/browse/SPARK-31440 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.1.0 >Reporter: Eren Avsarogullari >Priority: Major > Attachments: current_version.json, improved_version.json > > > SQL Rest API exposes query execution metrics as Public API. This Jira aims to > apply following improvements on SQL Rest API by aligning Spark-UI. > *Proposed Improvements:* > 1- Support Physical Operations and group metrics per operation by aligning > Spark UI. > 2- *nodeId* can be useful for grouping metrics as well as for sorting and to > differentiate same operators and their metrics. > 3- Filter *blank* metrics by aligning with Spark UI - SQL Tab > 4- Remove *\n* from *metricValue(s)* > 5- *planDescription* can be optional Http parameter to avoid network cost > (specially for complex jobs creating big-plans). > 6- *metrics* attribute needs to be exposed at the bottom order as > *metricDetails*. This order matches with Spark UI by highlighting with > execution order. > *Attachments:* > Please find both *current* and *improved* versions of results as attached. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-31440) Improve SQL Rest API
[ https://issues.apache.org/jira/browse/SPARK-31440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eren Avsarogullari updated SPARK-31440: --- Attachment: improved_version.json > Improve SQL Rest API > > > Key: SPARK-31440 > URL: https://issues.apache.org/jira/browse/SPARK-31440 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.1.0 >Reporter: Eren Avsarogullari >Priority: Major > Attachments: current_version.json, improved_version.json > > > SQL Rest API exposes query execution metrics as Public API. This Jira aims to > apply following improvements on SQL Rest API by aligning Spark-UI. > *Proposed Improvements:* > 1- Support Physical Operations and group metrics per operation by aligning > Spark UI. > 2- *nodeId* can be useful for grouping metrics as well as for sorting and to > differentiate same operators and their metrics. > 3- Filter *blank* metrics by aligning with Spark UI - SQL Tab > 4- Remove *\n* from *metricValue(s)* > 5- *planDescription* can be optional Http parameter to avoid network cost > (specially for complex jobs creating big-plans). > 6- *metrics* attribute needs to be exposed at the bottom order as > *metricDetails*. This order matches with Spark UI by highlighting with > execution order. > *Attachments:* > Please find both *current* and *improved* versions of results as attached. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-31440) Improve SQL Rest API
[ https://issues.apache.org/jira/browse/SPARK-31440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eren Avsarogullari updated SPARK-31440: --- Description: SQL Rest API exposes query execution metrics as Public API. This Jira aims to apply following improvements on SQL Rest API by aligning Spark-UI. *Proposed Improvements:* 1- Support Physical Operations and group metrics per operation by aligning Spark UI. 2- *nodeId* can be useful for grouping metrics as well as for sorting and to differentiate same operators and their metrics. 3- Filter *blank* metrics by aligning with Spark UI - SQL Tab 4- Remove *\n* from *metricValue(s)* 5- *planDescription* can be optional Http parameter to avoid network cost (specially for complex jobs creating big-plans). 6- *metrics* attribute needs to be exposed at the bottom order as *metricDetails*. This order matches with Spark UI by highlighting with execution order. *Attachments:* Please find both *current* and *improved* versions of results as attached. was: SQL Rest API exposes query execution metrics as Public API. This Jira aims to apply following improvements on SQL Rest API by aligning Spark-UI. *Proposed Improvements:* 1- Support Physical Operations and group metrics per operation by aligning Spark UI. 2- `nodeId` can be useful for grouping metrics as well as for sorting and to differentiate same operators and their metrics. 3- Filter `blank` metrics by aligning with Spark UI - SQL Tab 4- Remove `\n` from `metricValue` 5- `planDescription` can be optional Http parameter to avoid network cost (specially for complex jobs creating big-plans). 6- `metrics` attribute needs to be exposed at the bottom order as `metricDetails`. This order matches with Spark UI by highlighting with execution order. *Attachments:* Please find both *current* and *improved* versions of results as attached. > Improve SQL Rest API > > > Key: SPARK-31440 > URL: https://issues.apache.org/jira/browse/SPARK-31440 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.1.0 >Reporter: Eren Avsarogullari >Priority: Major > > SQL Rest API exposes query execution metrics as Public API. This Jira aims to > apply following improvements on SQL Rest API by aligning Spark-UI. > *Proposed Improvements:* > 1- Support Physical Operations and group metrics per operation by aligning > Spark UI. > 2- *nodeId* can be useful for grouping metrics as well as for sorting and to > differentiate same operators and their metrics. > 3- Filter *blank* metrics by aligning with Spark UI - SQL Tab > 4- Remove *\n* from *metricValue(s)* > 5- *planDescription* can be optional Http parameter to avoid network cost > (specially for complex jobs creating big-plans). > 6- *metrics* attribute needs to be exposed at the bottom order as > *metricDetails*. This order matches with Spark UI by highlighting with > execution order. > *Attachments:* > Please find both *current* and *improved* versions of results as attached. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org