We're using sc.addPyFile("path/to/mymodule.py") to load our own custom Python
modules from HDFS. We're frequently editing the .py files and so want to
update what's imported into the %pyspark interpreter of Zeppelin. However we
have not found a way to do this without restarting the livy interpreter.
reload(mymodule) doesn't work. Yes, we're using Python 2 right now but I'm
also interested in Python 3 solutions.
Has anyone else found a way to reload modules without restarting the
interpreter?
I came across a method at https://stackoverflow.com/a/44387776/1843329 for
reloading modules without restarting the PySpark shell. But it doesn't sound
too reliable. Also, Zeppelin injects the sc variable anyway and only our
admins are able to edit the settings for our interpreters (although we could
ask them to change the settings).
Thanks, Lucas.
