I love working with the Python community & I've heard similar requests in the past few months so its good to have a solid reason to try and add this functionality :)
Just to be clear though I'm not a Spark committer so when I work on stuff getting in it very much dependent on me finding a committer who shares my view - but I've over a hundred commits so it happens more often than not :) On Thu, Nov 24, 2016 at 3:15 AM, Ofer Eliassaf <ofer.elias...@gmail.com> wrote: > thank u so much for this! Great to see that u listen to the community. > > On Thu, Nov 24, 2016 at 12:10 PM, Holden Karau <hol...@pigscanfly.ca> > wrote: > >> https://issues.apache.org/jira/browse/SPARK-18576 >> >> On Thu, Nov 24, 2016 at 2:05 AM, Holden Karau <hol...@pigscanfly.ca> >> wrote: >> >>> Cool - thanks. I'll circle back with the JIRA number once I've got it >>> created - will probably take awhile before it lands in a Spark release >>> (since 2.1 has already branched) but better debugging information for >>> Python users is certainly important/useful. >>> >>> On Thu, Nov 24, 2016 at 2:03 AM, Ofer Eliassaf <ofer.elias...@gmail.com> >>> wrote: >>> >>>> Since we can't work with log4j in pyspark executors we build our own >>>> logging infrastructure (based on logstash/elastic/kibana). >>>> Would help to have TID in the logs, so we can drill down accordingly. >>>> >>>> >>>> On Thu, Nov 24, 2016 at 11:48 AM, Holden Karau <hol...@pigscanfly.ca> >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> The TaskContext isn't currently exposed in PySpark but I've been >>>>> meaning to look at exposing at least some of TaskContext for parity in >>>>> PySpark. Is there a particular use case which you want this for? Would >>>>> help >>>>> with crafting the JIRA :) >>>>> >>>>> Cheers, >>>>> >>>>> Holden :) >>>>> >>>>> On Thu, Nov 24, 2016 at 1:39 AM, ofer <ofer.elias...@gmail.com> wrote: >>>>> >>>>>> Hi, >>>>>> Is there a way to get in PYSPARK something like TaskContext from a >>>>>> code >>>>>> running on executor like in scala spark? >>>>>> >>>>>> If not - how can i know my task id from inside the executors? >>>>>> >>>>>> Thanks! >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> View this message in context: http://apache-spark-user-list. >>>>>> 1001560.n3.nabble.com/PySpark-TaskContext-tp28125.html >>>>>> Sent from the Apache Spark User List mailing list archive at >>>>>> Nabble.com. >>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe e-mail: user-unsubscr...@spark.apache.org >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Cell : 425-233-8271 >>>>> Twitter: https://twitter.com/holdenkarau >>>>> >>>> >>>> >>>> >>>> -- >>>> Regards, >>>> Ofer Eliassaf >>>> >>> >>> >>> >>> -- >>> Cell : 425-233-8271 >>> Twitter: https://twitter.com/holdenkarau >>> >> >> >> >> -- >> Cell : 425-233-8271 >> Twitter: https://twitter.com/holdenkarau >> > > > > -- > Regards, > Ofer Eliassaf > -- Cell : 425-233-8271 Twitter: https://twitter.com/holdenkarau
