Spark History Server pointing to S3

[Gianluca Privitera]

In Spark website it’s stated in the View After the Fact section 
(https://spark.apache.org/docs/latest/monitoring.html) that you can point the 
start-history-server.sh script to a directory in order do view the Web UI using 
the logs as data source.

Is it possible to point that script to S3? Maybe from a EC2 instance? 

Thanks,

Gianluca
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: Spark History Server pointing to S3

[Vladimir Grigor]

Gianluca,

have you had any luck with S3 storage of logs for history server?

Best,
Vladimir


On Tue, Jun 16, 2015 at 6:53 PM, Gianluca Privitera <
gianluca.privite...@studio.unibo.it> wrote:

> It gives me an exception with
> org.apache.spark.deploy.history.FsHistoryProvider , a problem with the
> file system. I can reproduce the exception if you want.
> It perfectly works if I give a local path, I tested it in 1.3.0 version.
>
> Gianluca
>
> On 16 Jun 2015, at 15:08, Akhil Das  wrote:
>
> Not quiet sure, but try pointing the spark.history.fs.logDirectory to your
> s3
>
> Thanks
> Best Regards
>
> On Tue, Jun 16, 2015 at 6:26 PM, Gianluca Privitera <
> gianluca.privite...@studio.unibo.it> wrote:
>
>> In Spark website it’s stated in the View After the Fact section (
>> https://spark.apache.org/docs/latest/monitoring.html) that you can point
>> the start-history-server.sh script to a directory in order do view the Web
>> UI using the logs as data source.
>>
>> Is it possible to point that script to S3? Maybe from a EC2 instance?
>>
>> Thanks,
>>
>> Gianluca
>> -
>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>> For additional commands, e-mail: user-h...@spark.apache.org
>>
>>
>
>

Re: Spark History Server pointing to S3

[Akhil Das]

Not quiet sure, but try pointing the spark.history.fs.logDirectory to your
s3

Thanks
Best Regards

On Tue, Jun 16, 2015 at 6:26 PM, Gianluca Privitera <
gianluca.privite...@studio.unibo.it> wrote:

> In Spark website it’s stated in the View After the Fact section (
> https://spark.apache.org/docs/latest/monitoring.html) that you can point
> the start-history-server.sh script to a directory in order do view the Web
> UI using the logs as data source.
>
> Is it possible to point that script to S3? Maybe from a EC2 instance?
>
> Thanks,
>
> Gianluca
> -
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>

Re: Spark History Server pointing to S3

[Gianluca Privitera]

It gives me an exception with org.apache.spark.deploy.history.FsHistoryProvider 
, a problem with the file system. I can reproduce the exception if you want.
It perfectly works if I give a local path, I tested it in 1.3.0 version.

Gianluca

On 16 Jun 2015, at 15:08, Akhil Das 
mailto:ak...@sigmoidanalytics.com>> wrote:

Not quiet sure, but try pointing the spark.history.fs.logDirectory to your s3

Thanks
Best Regards

On Tue, Jun 16, 2015 at 6:26 PM, Gianluca Privitera 
mailto:gianluca.privite...@studio.unibo.it>>
 wrote:
In Spark website it’s stated in the View After the Fact section 
(https://spark.apache.org/docs/latest/monitoring.html) that you can point the 
start-history-server.sh script to a directory in order do view the Web UI using 
the logs as data source.

Is it possible to point that script to S3? Maybe from a EC2 instance?

Thanks,

Gianluca
-
To unsubscribe, e-mail: 
user-unsubscr...@spark.apache.org
For additional commands, e-mail: 
user-h...@spark.apache.org