ah, great, thanks! sorry I missed that, I'll watch that jira. On Mon, Aug 27, 2018 at 12:41 PM Ilan Filonenko <i...@cornell.edu> wrote:
> A JIRA has been opened up on this exact topic: SPARK-25236 > <https://issues.apache.org/jira/browse/SPARK-25236>, a few days ago, > after seeing another case of print(_, file=sys.stderr) in a most recent > review. I agree that we should include logging for PySpark workers. > > On Mon, Aug 27, 2018 at 1:29 PM, Imran Rashid < > iras...@cloudera.com.invalid> wrote: > >> Another question on pyspark code -- how come there is no logging at all? >> does python logging have an unreasonable overhead, or its impossible to >> configure or something? >> >> I'm really surprised nobody has ever wanted to me able to turn on some >> debug or trace logging in pyspark by just configuring a logging level. >> >> For me, I wanted this during debugging while developing -- I'd work on >> some part of the code and drop in a bunch of print statements. Then I'd >> rip those out when I think I'm ready to submit a patch. But then I realize >> I forgot some case, then more debugging -- oh gotta add those print >> statements in again ... >> >> does somebody jsut need to setup the configuration properly, or is there >> a bigger reason to avoid logging in python? >> >> thanks, >> Imran >> > >
