Not quiet sure, but try pointing the spark.history.fs.logDirectory to your
s3
Thanks
Best Regards
On Tue, Jun 16, 2015 at 6:26 PM, Gianluca Privitera
gianluca.privite...@studio.unibo.it wrote:
In Spark website it’s stated in the View After the Fact section (
In Spark website it’s stated in the View After the Fact section
(https://spark.apache.org/docs/latest/monitoring.html) that you can point the
start-history-server.sh script to a directory in order do view the Web UI using
the logs as data source.
Is it possible to point that script to S3?
It gives me an exception with org.apache.spark.deploy.history.FsHistoryProvider
, a problem with the file system. I can reproduce the exception if you want.
It perfectly works if I give a local path, I tested it in 1.3.0 version.
Gianluca
On 16 Jun 2015, at 15:08, Akhil Das
