[jira] [Commented] (LIVY-633) session should not be gc-ed for long running queries
[ https://issues.apache.org/jira/browse/LIVY-633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16921215#comment-16921215 ] Yiheng Wang commented on LIVY-633: -- Yes, it's a different problem. It should be a bug in Livy. I'm working on a patch to fix it. > session should not be gc-ed for long running queries > > > Key: LIVY-633 > URL: https://issues.apache.org/jira/browse/LIVY-633 > Project: Livy > Issue Type: Bug > Components: Server >Affects Versions: 0.6.0 >Reporter: Liju >Priority: Major > > If you have set a relatively small session timeout eg 15 mins and query > execution is taking > 15 mins , the session gets gc-ed , which is incorrect > wrt user experience as the user was still active on session and waiting for > result -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (LIVY-588) Full support for Spark on Kubernetes
[
https://issues.apache.org/jira/browse/LIVY-588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aliaksandr Sasnouskikh updated LIVY-588:
Description:
Since Spark on Kubernetes has been released relatively long ago this can be a
good idea to include Kubernetes support to Livy project as well, as it can
solve much problems related to working with Spark on Kubernetes, it can fully
replace Yarn in case of working atop Kubernetes cluster:
- Livy UI has cached logs/diagnostics page
- Livy UI shows links to Spark UI and Spark History Server
- With Kubernetes Ingress resource Livy can be configured to serve as an
orchestrator of Spark Apps atop Kubernetes (PR includes Nginx Ingress support
option to create routes to Spark UI)
- Nginx Ingress solves `basePath` support for Spark UI and History Server as
well as has lots of auth integrations available:
https://github.com/kubernetes/ingress-nginx
- Livy UI can be integrated with Grafana Loki logs (PR provides solution for
that)
Dockerfiles repo: https://github.com/jahstreet/spark-on-kubernetes-docker
Helm charts: https://github.com/jahstreet/spark-on-kubernetes-helm
Associated JIRA: https://issues.apache.org/jira/browse/LIVY-588
Design concept:
https://github.com/jahstreet/spark-on-kubernetes-helm/blob/develop/README.md
## How was this patch tested?
Was tested manually on AKS cluster (Azure Kubernetes Services), Kubernetes
v1.11.8:
- Image: Spark 2.4.3 with Hadoop 3.2.0
(https://github.com/jahstreet/spark-on-kubernetes-docker)
- History Server:
https://github.com/helm/charts/tree/master/stable/spark-history-server
- Jupyter Notebook with Sparkmagic:
https://github.com/jahstreet/spark-on-kubernetes-helm/tree/master/charts/jupyter
Unit and/or integration tests are in progress.
was:
Since Spark on Kubernetes has been released relatively long ago this can be a
good idea to include Kubernetes support to Livy project as well, as it can
solve much problems related to working with Spark on Kubernetes, it can fully
replace Yarn in case of working atop Kubernetes cluster:
* Livy UI has cached logs/diagnostics page
* Livy UI shows links to Spark UI and Spark History Server
* With Kubernetes Ingress resource Livy can be configured to serve as an
orchestrator of Spark Apps atop Kubernetes (PR includes Nginx Ingress support
option to create routes to Spark UI)
* Nginx Ingress solves {{basePath}} support for Spark UI and History Server as
well as has lots of auth integrations available:
[https://github.com/kubernetes/ingress-nginx]
Dockerfiles repo: [https://github.com/jahstreet/spark-on-kubernetes-docker]
Helm charts: [https://github.com/jahstreet/spark-on-kubernetes-helm]
Associated PR: [https://github.com/apache/incubator-livy/pull/167]
Design concept doc:
[https://github.com/jahstreet/spark-on-kubernetes-helm/blob/master/README.md]
> Full support for Spark on Kubernetes
>
>
> Key: LIVY-588
> URL: https://issues.apache.org/jira/browse/LIVY-588
> Project: Livy
> Issue Type: New Feature
>Affects Versions: 0.7.0
> Environment: Kubernetes
>Reporter: Aliaksandr Sasnouskikh
>Priority: Major
>
>
> Since Spark on Kubernetes has been released relatively long ago this can be a
> good idea to include Kubernetes support to Livy project as well, as it can
> solve much problems related to working with Spark on Kubernetes, it can fully
> replace Yarn in case of working atop Kubernetes cluster:
> - Livy UI has cached logs/diagnostics page
> - Livy UI shows links to Spark UI and Spark History Server
> - With Kubernetes Ingress resource Livy can be configured to serve as an
> orchestrator of Spark Apps atop Kubernetes (PR includes Nginx Ingress support
> option to create routes to Spark UI)
> - Nginx Ingress solves `basePath` support for Spark UI and History Server as
> well as has lots of auth integrations available:
> https://github.com/kubernetes/ingress-nginx
> - Livy UI can be integrated with Grafana Loki logs (PR provides solution for
> that)
> Dockerfiles repo: https://github.com/jahstreet/spark-on-kubernetes-docker
> Helm charts: https://github.com/jahstreet/spark-on-kubernetes-helm
> Associated JIRA: https://issues.apache.org/jira/browse/LIVY-588
> Design concept:
> https://github.com/jahstreet/spark-on-kubernetes-helm/blob/develop/README.md
> ## How was this patch tested?
> Was tested manually on AKS cluster (Azure Kubernetes Services), Kubernetes
> v1.11.8:
> - Image: Spark 2.4.3 with Hadoop 3.2.0
> (https://github.com/jahstreet/spark-on-kubernetes-docker)
> - History Server:
> https://github.com/helm/charts/tree/master/stable/spark-history-server
> - Jupyter Notebook with Sparkmagic:
> https://github.com/jahstreet/spark-on-kubernetes-helm/tree/master/charts/jupyter
> Unit and/or integration tests are in progress.
--
This messa
[jira] [Updated] (LIVY-588) Full support for Spark on Kubernetes
[ https://issues.apache.org/jira/browse/LIVY-588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aliaksandr Sasnouskikh updated LIVY-588: Description: Since Spark on Kubernetes has been released relatively long ago this can be a good idea to include Kubernetes support to Livy project as well, as it can solve much problems related to working with Spark on Kubernetes, it can fully replace Yarn in case of working atop Kubernetes cluster: - Livy UI has cached logs/diagnostics page - Livy UI shows links to Spark UI and Spark History Server - With Kubernetes Ingress resource Livy can be configured to serve as an orchestrator of Spark Apps atop Kubernetes (PR includes Nginx Ingress support option to create routes to Spark UI) - Nginx Ingress solves `basePath` support for Spark UI and History Server as well as has lots of auth integrations available: [https://github.com/kubernetes/ingress-nginx] - Livy UI can be integrated with Grafana Loki logs (PR provides solution for that) Dockerfiles repo: [https://github.com/jahstreet/spark-on-kubernetes-docker] Helm charts: [https://github.com/jahstreet/spark-on-kubernetes-helm] Associated JIRA: https://issues.apache.org/jira/browse/LIVY-588 Design concept: [https://github.com/jahstreet/spark-on-kubernetes-helm/blob/develop/README.md] Was tested manually on AKS cluster (Azure Kubernetes Services), Kubernetes v1.11.8: - Image: Spark 2.4.3 with Hadoop 3.2.0 ([https://github.com/jahstreet/spark-on-kubernetes-docker]) - History Server: [https://github.com/helm/charts/tree/master/stable/spark-history-server] - Jupyter Notebook with Sparkmagic: [https://github.com/jahstreet/spark-on-kubernetes-helm/tree/master/charts/jupyter] Unit and/or integration tests are in progress. was: Since Spark on Kubernetes has been released relatively long ago this can be a good idea to include Kubernetes support to Livy project as well, as it can solve much problems related to working with Spark on Kubernetes, it can fully replace Yarn in case of working atop Kubernetes cluster: - Livy UI has cached logs/diagnostics page - Livy UI shows links to Spark UI and Spark History Server - With Kubernetes Ingress resource Livy can be configured to serve as an orchestrator of Spark Apps atop Kubernetes (PR includes Nginx Ingress support option to create routes to Spark UI) - Nginx Ingress solves `basePath` support for Spark UI and History Server as well as has lots of auth integrations available: https://github.com/kubernetes/ingress-nginx - Livy UI can be integrated with Grafana Loki logs (PR provides solution for that) Dockerfiles repo: https://github.com/jahstreet/spark-on-kubernetes-docker Helm charts: https://github.com/jahstreet/spark-on-kubernetes-helm Associated JIRA: https://issues.apache.org/jira/browse/LIVY-588 Design concept: https://github.com/jahstreet/spark-on-kubernetes-helm/blob/develop/README.md ## How was this patch tested? Was tested manually on AKS cluster (Azure Kubernetes Services), Kubernetes v1.11.8: - Image: Spark 2.4.3 with Hadoop 3.2.0 (https://github.com/jahstreet/spark-on-kubernetes-docker) - History Server: https://github.com/helm/charts/tree/master/stable/spark-history-server - Jupyter Notebook with Sparkmagic: https://github.com/jahstreet/spark-on-kubernetes-helm/tree/master/charts/jupyter Unit and/or integration tests are in progress. > Full support for Spark on Kubernetes > > > Key: LIVY-588 > URL: https://issues.apache.org/jira/browse/LIVY-588 > Project: Livy > Issue Type: New Feature >Affects Versions: 0.7.0 > Environment: Kubernetes >Reporter: Aliaksandr Sasnouskikh >Priority: Major > > > Since Spark on Kubernetes has been released relatively long ago this can be a > good idea to include Kubernetes support to Livy project as well, as it can > solve much problems related to working with Spark on Kubernetes, it can fully > replace Yarn in case of working atop Kubernetes cluster: > - Livy UI has cached logs/diagnostics page > - Livy UI shows links to Spark UI and Spark History Server > - With Kubernetes Ingress resource Livy can be configured to serve as an > orchestrator of Spark Apps atop Kubernetes (PR includes Nginx Ingress support > option to create routes to Spark UI) > - Nginx Ingress solves `basePath` support for Spark UI and History Server as > well as has lots of auth integrations available: > [https://github.com/kubernetes/ingress-nginx] > - Livy UI can be integrated with Grafana Loki logs (PR provides solution for > that) > Dockerfiles repo: [https://github.com/jahstreet/spark-on-kubernetes-docker] > Helm charts: [https://github.com/jahstreet/spark-on-kubernetes-helm] > Associated JIRA: https://issues.apache.org/jira/browse/LIVY-588 > Design concept: > [https://github.com/jahstreet/spark-on-k
[jira] [Resolved] (LIVY-652) Thrifserver doesn't set session name correctly
[ https://issues.apache.org/jira/browse/LIVY-652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-652. -- Fix Version/s: 0.7.0 Assignee: Jeffrey(Xilang) Yan Resolution: Fixed Issue resolved by pull request 218 https://github.com/apache/incubator-livy/pull/218 > Thrifserver doesn't set session name correctly > -- > > Key: LIVY-652 > URL: https://issues.apache.org/jira/browse/LIVY-652 > Project: Livy > Issue Type: Bug > Components: Thriftserver >Affects Versions: 0.6.0 >Reporter: Jeffrey(Xilang) Yan >Assignee: Jeffrey(Xilang) Yan >Priority: Major > Fix For: 0.7.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Thrift server doesn't set session name correctly, so not able to view session > name in Livy Web UI -- This message was sent by Atlassian Jira (v8.3.2#803003)
