Re: How to configure log4j in pyspark to get log level, file name, and line number

2022-01-21 Thread Andrew Davidson
Interesting. I noticed that my drive log messages with time stamp, function name but no line number. However log message in other python files only contain the messages. All of my python code is a single zip file. The zip file is job submit argument 2022-01-21 19:45:02 WARN __main__:? -

How to configure log4j in pyspark to get log level, file name, and line number

2022-01-20 Thread Andrew Davidson
Hi When I use python logging for my unit test. I am able to control the output format. I get the log level, the file and line number, then the msg [INFO testEstimatedScalingFactors.py:166 - test_B_convertCountsToInts()] BEGIN In my spark driver I am able to get the log4j logger spark