[ https://issues.apache.org/jira/browse/LIVY-588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16960443#comment-16960443 ]
Aliaksandr Sasnouskikh commented on LIVY-588: --------------------------------------------- This epic fixes that ticket splitting it to multiple steps: https://issues.apache.org/jira/browse/LIVY-702 > Full support for Spark on Kubernetes > ------------------------------------ > > Key: LIVY-588 > URL: https://issues.apache.org/jira/browse/LIVY-588 > Project: Livy > Issue Type: New Feature > Affects Versions: 0.7.0 > Environment: Kubernetes > Reporter: Aliaksandr Sasnouskikh > Priority: Major > > > Since Spark on Kubernetes has been released relatively long ago this can be a > good idea to include Kubernetes support to Livy project as well, as it can > solve much problems related to working with Spark on Kubernetes, it can fully > replace Yarn in case of working atop Kubernetes cluster: > - Livy UI has cached logs/diagnostics page > - Livy UI shows links to Spark UI and Spark History Server > - With Kubernetes Ingress resource Livy can be configured to serve as an > orchestrator of Spark Apps atop Kubernetes (PR includes Nginx Ingress support > option to create routes to Spark UI) > - Nginx Ingress solves `basePath` support for Spark UI and History Server as > well as has lots of auth integrations available: > [https://github.com/kubernetes/ingress-nginx] > - Livy UI can be integrated with Grafana Loki logs (PR provides solution for > that) > Dockerfiles repo: [https://github.com/jahstreet/spark-on-kubernetes-docker] > Helm charts: [https://github.com/jahstreet/spark-on-kubernetes-helm] > Associated JIRA: https://issues.apache.org/jira/browse/LIVY-588 > Design concept: > [https://github.com/jahstreet/spark-on-kubernetes-helm/blob/develop/README.md] > Was tested manually on AKS cluster (Azure Kubernetes Services), Kubernetes > v1.11.8: > - Image: Spark 2.4.3 with Hadoop 3.2.0 > ([https://github.com/jahstreet/spark-on-kubernetes-docker]) > - History Server: > [https://github.com/helm/charts/tree/master/stable/spark-history-server] > - Jupyter Notebook with Sparkmagic: > [https://github.com/jahstreet/spark-on-kubernetes-helm/tree/master/charts/jupyter] > Unit and/or integration tests are in progress. -- This message was sent by Atlassian Jira (v8.3.4#803005)