https://issues.apache.org/jira/browse/SPARK-18576
On Thu, Nov 24, 2016 at 2:05 AM, Holden Karau <hol...@pigscanfly.ca> wrote: > Cool - thanks. I'll circle back with the JIRA number once I've got it > created - will probably take awhile before it lands in a Spark release > (since 2.1 has already branched) but better debugging information for > Python users is certainly important/useful. > > On Thu, Nov 24, 2016 at 2:03 AM, Ofer Eliassaf <ofer.elias...@gmail.com> > wrote: > >> Since we can't work with log4j in pyspark executors we build our own >> logging infrastructure (based on logstash/elastic/kibana). >> Would help to have TID in the logs, so we can drill down accordingly. >> >> >> On Thu, Nov 24, 2016 at 11:48 AM, Holden Karau <hol...@pigscanfly.ca> >> wrote: >> >>> Hi, >>> >>> The TaskContext isn't currently exposed in PySpark but I've been meaning >>> to look at exposing at least some of TaskContext for parity in PySpark. Is >>> there a particular use case which you want this for? Would help with >>> crafting the JIRA :) >>> >>> Cheers, >>> >>> Holden :) >>> >>> On Thu, Nov 24, 2016 at 1:39 AM, ofer <ofer.elias...@gmail.com> wrote: >>> >>>> Hi, >>>> Is there a way to get in PYSPARK something like TaskContext from a code >>>> running on executor like in scala spark? >>>> >>>> If not - how can i know my task id from inside the executors? >>>> >>>> Thanks! >>>> >>>> >>>> >>>> -- >>>> View this message in context: http://apache-spark-user-list. >>>> 1001560.n3.nabble.com/PySpark-TaskContext-tp28125.html >>>> Sent from the Apache Spark User List mailing list archive at Nabble.com. >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe e-mail: user-unsubscr...@spark.apache.org >>>> >>>> >>> >>> >>> -- >>> Cell : 425-233-8271 >>> Twitter: https://twitter.com/holdenkarau >>> >> >> >> >> -- >> Regards, >> Ofer Eliassaf >> > > > > -- > Cell : 425-233-8271 > Twitter: https://twitter.com/holdenkarau > -- Cell : 425-233-8271 Twitter: https://twitter.com/holdenkarau
