thank u so much for this! Great to see that u listen to the community. On Thu, Nov 24, 2016 at 12:10 PM, Holden Karau <hol...@pigscanfly.ca> wrote:
> https://issues.apache.org/jira/browse/SPARK-18576 > > On Thu, Nov 24, 2016 at 2:05 AM, Holden Karau <hol...@pigscanfly.ca> > wrote: > >> Cool - thanks. I'll circle back with the JIRA number once I've got it >> created - will probably take awhile before it lands in a Spark release >> (since 2.1 has already branched) but better debugging information for >> Python users is certainly important/useful. >> >> On Thu, Nov 24, 2016 at 2:03 AM, Ofer Eliassaf <ofer.elias...@gmail.com> >> wrote: >> >>> Since we can't work with log4j in pyspark executors we build our own >>> logging infrastructure (based on logstash/elastic/kibana). >>> Would help to have TID in the logs, so we can drill down accordingly. >>> >>> >>> On Thu, Nov 24, 2016 at 11:48 AM, Holden Karau <hol...@pigscanfly.ca> >>> wrote: >>> >>>> Hi, >>>> >>>> The TaskContext isn't currently exposed in PySpark but I've been >>>> meaning to look at exposing at least some of TaskContext for parity in >>>> PySpark. Is there a particular use case which you want this for? Would help >>>> with crafting the JIRA :) >>>> >>>> Cheers, >>>> >>>> Holden :) >>>> >>>> On Thu, Nov 24, 2016 at 1:39 AM, ofer <ofer.elias...@gmail.com> wrote: >>>> >>>>> Hi, >>>>> Is there a way to get in PYSPARK something like TaskContext from a code >>>>> running on executor like in scala spark? >>>>> >>>>> If not - how can i know my task id from inside the executors? >>>>> >>>>> Thanks! >>>>> >>>>> >>>>> >>>>> -- >>>>> View this message in context: http://apache-spark-user-list. >>>>> 1001560.n3.nabble.com/PySpark-TaskContext-tp28125.html >>>>> Sent from the Apache Spark User List mailing list archive at >>>>> Nabble.com. >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe e-mail: user-unsubscr...@spark.apache.org >>>>> >>>>> >>>> >>>> >>>> -- >>>> Cell : 425-233-8271 >>>> Twitter: https://twitter.com/holdenkarau >>>> >>> >>> >>> >>> -- >>> Regards, >>> Ofer Eliassaf >>> >> >> >> >> -- >> Cell : 425-233-8271 >> Twitter: https://twitter.com/holdenkarau >> > > > > -- > Cell : 425-233-8271 > Twitter: https://twitter.com/holdenkarau > -- Regards, Ofer Eliassaf
