Re: no logging in pyspark code?

FYI, we do have a basic logging by warnings module. 2018년 8월 28일 (화) 오전 2:05, Imran Rashid 님이 작성: > ah, great, thanks! sorry I missed that, I'll watch that jira. > > On Mon, Aug 27, 2018 at 12:41 PM Ilan Filonenko wrote: > >> A JIRA has been opened up on this exact topic: SPARK-25236 >>

Re: no logging in pyspark code?

ah, great, thanks! sorry I missed that, I'll watch that jira. On Mon, Aug 27, 2018 at 12:41 PM Ilan Filonenko wrote: > A JIRA has been opened up on this exact topic: SPARK-25236 > , a few days ago, > after seeing another case of print(_,

Re: no logging in pyspark code?

A JIRA has been opened up on this exact topic: SPARK-25236 , a few days ago, after seeing another case of print(_, file=sys.stderr) in a most recent review. I agree that we should include logging for PySpark workers. On Mon, Aug 27, 2018 at 1:29

no logging in pyspark code?

Another question on pyspark code -- how come there is no logging at all? does python logging have an unreasonable overhead, or its impossible to configure or something? I'm really surprised nobody has ever wanted to me able to turn on some debug or trace logging in pyspark by just configuring a