To close the loop.
This should work:
sc._jsc.hadoopConfiguration
See this method in JavaSparkContext :
def hadoopConfiguration(): Configuration = {
sc.hadoopConfiguration
On Tue, Jun 30, 2015 at 5:52 PM, Ted Yu <yuzhih...@gmail.com> wrote:
> Minor correction:
> It should be sc._jsc
>
> Cheers
>
> On Tue, Jun 30, 2015 at 4:23 PM, ayan guha <guha.a...@gmail.com> wrote:
>
>> There is a sc._jsc_ which you can access to get/set hadoop conf.
>>
>> On Wed, Jul 1, 2015 at 7:41 AM, Richard Ding <pigu...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I noticed that, in Scala API, one can call hadoopConfiguration on
>>> SparkContext to retrieve the hadoop configuration object which is very
>>> handy in modifying certain hadoop properties at runtime. But there is no
>>> corresponding method in Python API.
>>>
>>> Will this method be added to Python API in a later release? In the mean
>>> time, what will be the work around (of setting hadoop properties at
>>> runtime) using Python API?
>>>
>>> Thanks,
>>> Richard
>>>
>>
>>
>>
>> --
>> Best Regards,
>> Ayan Guha
>>
>
>
