Re: Retrieve hadoop conf object from Python API

2015-07-01 Thread Ted Yu 
To close the loop.

This should work:
sc._jsc.hadoopConfiguration

See this method in JavaSparkContext :

  def hadoopConfiguration(): Configuration = {
sc.hadoopConfiguration

On Tue, Jun 30, 2015 at 5:52 PM, Ted Yu  wrote:

> Minor correction:
> It should be sc._jsc
>
> Cheers
>
> On Tue, Jun 30, 2015 at 4:23 PM, ayan guha  wrote:
>
>> There is a sc._jsc_ which you can access to get/set hadoop conf.
>>
>> On Wed, Jul 1, 2015 at 7:41 AM, Richard Ding  wrote:
>>
>>> Hi,
>>>
>>> I noticed that,  in Scala API, one can call hadoopConfiguration on
>>> SparkContext to retrieve the hadoop configuration object which is very
>>> handy in modifying certain hadoop properties at runtime. But there is no
>>> corresponding method in Python API.
>>>
>>> Will this method be added to Python API in a later release? In the mean
>>> time, what will be the work around (of setting hadoop properties at
>>> runtime) using Python API?
>>>
>>> Thanks,
>>> Richard
>>>
>>
>>
>>
>> --
>> Best Regards,
>> Ayan Guha
>>
>
>

Re: Retrieve hadoop conf object from Python API

2015-06-30 Thread Ted Yu 
Minor correction:
It should be sc._jsc

Cheers

On Tue, Jun 30, 2015 at 4:23 PM, ayan guha  wrote:

> There is a sc._jsc_ which you can access to get/set hadoop conf.
>
> On Wed, Jul 1, 2015 at 7:41 AM, Richard Ding  wrote:
>
>> Hi,
>>
>> I noticed that,  in Scala API, one can call hadoopConfiguration on
>> SparkContext to retrieve the hadoop configuration object which is very
>> handy in modifying certain hadoop properties at runtime. But there is no
>> corresponding method in Python API.
>>
>> Will this method be added to Python API in a later release? In the mean
>> time, what will be the work around (of setting hadoop properties at
>> runtime) using Python API?
>>
>> Thanks,
>> Richard
>>
>
>
>
> --
> Best Regards,
> Ayan Guha
>

Re: Retrieve hadoop conf object from Python API

2015-06-30 Thread ayan guha 
There is a sc._jsc_ which you can access to get/set hadoop conf.

On Wed, Jul 1, 2015 at 7:41 AM, Richard Ding  wrote:

> Hi,
>
> I noticed that,  in Scala API, one can call hadoopConfiguration on
> SparkContext to retrieve the hadoop configuration object which is very
> handy in modifying certain hadoop properties at runtime. But there is no
> corresponding method in Python API.
>
> Will this method be added to Python API in a later release? In the mean
> time, what will be the work around (of setting hadoop properties at
> runtime) using Python API?
>
> Thanks,
> Richard
>



-- 
Best Regards,
Ayan Guha

Retrieve hadoop conf object from Python API

2015-06-30 Thread Richard Ding 
Hi,

I noticed that,  in Scala API, one can call hadoopConfiguration on
SparkContext to retrieve the hadoop configuration object which is very
handy in modifying certain hadoop properties at runtime. But there is no
corresponding method in Python API.

Will this method be added to Python API in a later release? In the mean
time, what will be the work around (of setting hadoop properties at
runtime) using Python API?

Thanks,
Richard