Re: PySpark TaskContext

I love working with the Python community & I've heard similar requests in the past few months so its good to have a solid reason to try and add this functionality :) Just to be clear though I'm not a Spark committer so when I work on stuff getting in it very much dependent on me finding a

Re: PySpark TaskContext

thank u so much for this! Great to see that u listen to the community. On Thu, Nov 24, 2016 at 12:10 PM, Holden Karau wrote: > https://issues.apache.org/jira/browse/SPARK-18576 > > On Thu, Nov 24, 2016 at 2:05 AM, Holden Karau > wrote: > >> Cool -

Re: PySpark TaskContext

https://issues.apache.org/jira/browse/SPARK-18576 On Thu, Nov 24, 2016 at 2:05 AM, Holden Karau wrote: > Cool - thanks. I'll circle back with the JIRA number once I've got it > created - will probably take awhile before it lands in a Spark release > (since 2.1 has already

Re: PySpark TaskContext

Cool - thanks. I'll circle back with the JIRA number once I've got it created - will probably take awhile before it lands in a Spark release (since 2.1 has already branched) but better debugging information for Python users is certainly important/useful. On Thu, Nov 24, 2016 at 2:03 AM, Ofer

Re: PySpark TaskContext

Since we can't work with log4j in pyspark executors we build our own logging infrastructure (based on logstash/elastic/kibana). Would help to have TID in the logs, so we can drill down accordingly. On Thu, Nov 24, 2016 at 11:48 AM, Holden Karau wrote: > Hi, > > The

Re: PySpark TaskContext

Hi, The TaskContext isn't currently exposed in PySpark but I've been meaning to look at exposing at least some of TaskContext for parity in PySpark. Is there a particular use case which you want this for? Would help with crafting the JIRA :) Cheers, Holden :) On Thu, Nov 24, 2016 at 1:39 AM,