We need a centralized spark logging solution. Ideally, it should: * Allow any Spark process to log at multiple levels (info, warn, debug) using a single line, similar to log4j * All logs should go to a central location - so, to read the logs, we don't need to check each worker by itself * Ideally, it should be configurable so that when the code is run standalone (not on spark) to use a different (local) log (this last point is point is optional, because we could add it with a wrapper)
Can you recommend something? How do you handle logging and debugging spark applications? Do you go through the logs on each machine?