Hi TD, I thought about that but was not sure whether this will have any impact in spark UI/ Executor runner as it redirects stream to stderr/stdout. But ideally it should not as it will fetch the log record from stderr file (which is latest)..
Is my understanding correct? Thanks, Sourav On Tue, Mar 25, 2014 at 3:26 AM, Tathagata Das <[email protected]>wrote: > You can use RollingFileAppenders in log4j.properties. > > http://logging.apache.org/log4j/extras/apidocs/org/apache/log4j/rolling/RollingFileAppender.html > > You can have other scripts delete old logs. > > TD > > > On Mon, Mar 24, 2014 at 12:20 AM, Sourav Chandra < > [email protected]> wrote: > >> Hi, >> >> I have few questions regarding log file management in spark: >> >> 1. Currently I did not find any way to modify the lof file name for >> executor/drivers). Its hardcoded as stdout and stderr. Also there is no log >> rotation. >> >> In case of streaming application this will grow forever and become >> unmanageable. Is there any way to overcome this? >> >> Thanks, >> -- >> >> Sourav Chandra >> >> Senior Software Engineer >> >> · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · >> >> [email protected] >> >> o: +91 80 4121 8723 >> >> m: +91 988 699 3746 >> >> skype: sourav.chandra >> >> Livestream >> >> "Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd >> Block, Koramangala Industrial Area, >> >> Bangalore 560034 >> >> www.livestream.com >> > > -- Sourav Chandra Senior Software Engineer · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · [email protected] o: +91 80 4121 8723 m: +91 988 699 3746 skype: sourav.chandra Livestream "Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd Block, Koramangala Industrial Area, Bangalore 560034 www.livestream.com
