Re: Retrieve hadoop conf object from Python API
To close the loop. This should work: sc._jsc.hadoopConfiguration See this method in JavaSparkContext : def hadoopConfiguration(): Configuration = { sc.hadoopConfiguration On Tue, Jun 30, 2015 at 5:52 PM, Ted Yu wrote: > Minor correction: > It should be sc._jsc > > Cheers > > On Tue, Jun 30, 2015 at 4:23 PM, ayan guha wrote: > >> There is a sc._jsc_ which you can access to get/set hadoop conf. >> >> On Wed, Jul 1, 2015 at 7:41 AM, Richard Ding wrote: >> >>> Hi, >>> >>> I noticed that, in Scala API, one can call hadoopConfiguration on >>> SparkContext to retrieve the hadoop configuration object which is very >>> handy in modifying certain hadoop properties at runtime. But there is no >>> corresponding method in Python API. >>> >>> Will this method be added to Python API in a later release? In the mean >>> time, what will be the work around (of setting hadoop properties at >>> runtime) using Python API? >>> >>> Thanks, >>> Richard >>> >> >> >> >> -- >> Best Regards, >> Ayan Guha >> > >
Re: Retrieve hadoop conf object from Python API
Minor correction: It should be sc._jsc Cheers On Tue, Jun 30, 2015 at 4:23 PM, ayan guha wrote: > There is a sc._jsc_ which you can access to get/set hadoop conf. > > On Wed, Jul 1, 2015 at 7:41 AM, Richard Ding wrote: > >> Hi, >> >> I noticed that, in Scala API, one can call hadoopConfiguration on >> SparkContext to retrieve the hadoop configuration object which is very >> handy in modifying certain hadoop properties at runtime. But there is no >> corresponding method in Python API. >> >> Will this method be added to Python API in a later release? In the mean >> time, what will be the work around (of setting hadoop properties at >> runtime) using Python API? >> >> Thanks, >> Richard >> > > > > -- > Best Regards, > Ayan Guha >
Re: Retrieve hadoop conf object from Python API
There is a sc._jsc_ which you can access to get/set hadoop conf. On Wed, Jul 1, 2015 at 7:41 AM, Richard Ding wrote: > Hi, > > I noticed that, in Scala API, one can call hadoopConfiguration on > SparkContext to retrieve the hadoop configuration object which is very > handy in modifying certain hadoop properties at runtime. But there is no > corresponding method in Python API. > > Will this method be added to Python API in a later release? In the mean > time, what will be the work around (of setting hadoop properties at > runtime) using Python API? > > Thanks, > Richard > -- Best Regards, Ayan Guha