Issue with Spark UI when running on AWS EMR is it requires ssh tunneling to be setup which requires private aws keys.
Our team is building a analytic platform on zeppelin for end-users who we obviously can't hand out these keys. Another issue is setting up correct port - Zeppelin tries to use 4040 for spark but during an interpreter restart 4040 could be used by an old still stuck paragraph. In that case Zeppelin simply tries the next port and so on. Static url for Spark can't handle this and hence requires some dynamic implementation. PS - As I write this a lightbulb goes on in my head. I guess we could also modify Zeppelin restart script to kill those rogue processes and make sure 4040 is always available? Thanks Ankit On Wed, Jan 24, 2018 at 6:10 PM, Jeff Zhang <zjf...@gmail.com> wrote: > > If Spark interpreter didn't give you the correct spark UI, this should be > a bug, you can file a ticket to fix it. Although you can make a custom > interpreter by extending the current spark interpreter, it is not a trivial > work. > > > ankit jain <ankitjain....@gmail.com>于2018年1月25日周四 上午8:07写道： > >> Hi fellow Zeppelin users, >> Has anyone tried to write a custom Spark Interpreter perhaps extending >> from the one that ships currently with zeppelin - >> spark/src/main/java/org/apache/zeppelin/spark/*SparkInterpreter.java?* >> >> We are coming across cases where we need the interpreter to do "more", eg >> change getSparkUIUrl() to directly load Yarn >> ResourceManager/proxy/application_id123 >> rather than a fixed web ui. >> >> If we directly modify Zeppelin source code, upgrading to new zeppelin >> versions will be a mess. >> >> Before we get too deep into it, wanted to get thoughts of the community. >> >> What is a "clean" way to do such changes? >> >> -- >> Thanks & Regards, >> Ankit. >> > -- Thanks & Regards, Ankit.
