Hello, thanks for the answer.
I have tried already to modify the log4j settings, but got some weird
errors. My /opt/PredictionIO-0.12.1/conf/log4j.properties file looks like
this:
log4j.rootLogger=WARN, file
# console appender
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.follow=true
log4j.appender.console.layout=org.apache.log4j.EnhancedPatternLayout
log4j.appender.console.layout.ConversionPattern=[%p] [%c{1}]
%m%n%throwable{0}
# file appender
log4j.appender.file=org.apache.log4j.DailyRollingFileAppender
log4j.appender.file.File=${pio.log.dir}/pio.log
log4j.appender.file.ImmediateFlush=true
log4j.appender.file.Append=true
log4j.appender.file.DatePattern='.'yyyy-MM-dd-a
log4j.appender.file.layout=org.apache.log4j.EnhancedPatternLayout
log4j.appender.file.layout.ConversionPattern=%d %-5p %c [%t] - %m%n
# quiet some packages that are too verbose
log4j.logger.org.elasticsearch=WARN
log4j.logger.org.apache.hadoop=WARN
log4j.logger.org.apache.hadoop.hbase.zookeeper=ERROR
log4j.logger.org.apache.hadoop.util.NativeCodeLoader=ERROR
log4j.logger.org.apache.spark=WARN
log4j.logger.org.apache.zookeeper=ERROR
log4j.logger.org.eclipse.jetty=WARN
log4j.logger.org.spark-project.jetty=WARN
log4j.logger.akka=WARN
I'm using a cronjob to gzip the generated log files, like this:
20 0,12 * * * find /var/log/predictionio/ -type f -iname "*.log.2*" !
-name "*.gz" -exec gzip {} \;
After every cronjob run i receive an error message, that the file size
changed while zipping. And indeed, some of the pio deploys are using the
original, now deleted log files.
java 3489 pio 21w REG 259,1 8223485244 267366
/var/log/predictionio/pio.log.2018-09-10-AM (deleted)
java 3565 pio 229w REG 259,1 6805803 267367
/var/log/predictionio/pio.log
java 3648 pio 21w REG 259,1 8223485244 267366
/var/log/predictionio/pio.log.2018-09-10-AM (deleted)
java 3773 pio 268w REG 259,1 14781293004 256008
/var/log/predictionio/pio.log.2018-09-12-AM (deleted)
pio 3487 0.0 0.0 11224 1736 ? Ss Sep10 0:00 /bin/bash
-c cd /home/pio/UR/job1/ && /opt/PredictionIO/bin/pio deploy --port 8001
pio 3489 0.0 0.2 7020024 41048 ? Sl Sep10 1:33 \_ java
-cp
/opt/PredictionIO-0.12.1/conf:/opt/PredictionIO-0.12.1/lib/spark/pio-data-elasticsea
pio 3565 0.3 1.9 4657440 317268 ? Sl Sep10 13:12 \_
/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -cp
/opt/PredictionIO-0.12.1/conf/:/opt/Pr
pio 3642 0.0 0.0 11224 1796 ? Ss Sep10 0:00 /bin/bash
-c cd /home/pio/UR/job2/ && /opt/PredictionIO/bin/pio deploy
pio 3648 0.0 0.2 7020024 44400 ? Sl Sep10 1:30 \_ java
-cp
/opt/PredictionIO-0.12.1/conf:/opt/PredictionIO-0.12.1/lib/spark/pio-data-elasticsea
pio 3773 7.9 6.2 4664636 997184 ? Sl Sep10 290:41 \_
/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -cp
/opt/PredictionIO-0.12.1/conf/:/opt/Pr
Any idea why some of the deploys are using the old logfile?
Thanks,
Joe
Juha Syrjälä <juha.syrj...@gmail.com> ezt írta (időpont: 2018. szept. 12.,
Sze, 19:45):
> Hi,
>
> PIO and UR seem to use log4j library for logging.
>
> https://en.wikipedia.org/wiki/Log4j
> https://logging.apache.org/log4j/2.x/
>
> So you can specify a custom log4j.properties file which will do to file
> rotation. Other option would be to use some external software that does log
> rotation, e.g https://linux.die.net/man/8/logrotate
>
> For example with deployed engines:
> nohup pio deploy -v my-engine.json --port 8001 -- --driver-java-options
> -Dlog4j.configuration=file:/home/aml/my-stuff/log4j.properties
>
> log4j.properties file:
> log4j.rootLogger=INFO, rollingfile
>
> # console appender
> log4j.appender.console=org.apache.log4j.ConsoleAppender
> log4j.appender.console.follow=true
> log4j.appender.console.layout=org.apache.log4j.EnhancedPatternLayout
> log4j.appender.console.layout.ConversionPattern=[%p] [%c{1}]
> %m%n%throwable{0}
>
> # rolling file appender
> log4j.appender.rollingfile=org.apache.log4j.RollingFileAppender
> log4j.appender.rollingfile.File=/var/log/pio/my-engine.log
> log4j.appender.rollingfile.MaxFileSize=100MB
> log4j.appender.rollingfile.MaxBackupIndex=20
> log4j.appender.rollingfile.layout=org.apache.log4j.EnhancedPatternLayout
> log4j.appender.rollingfile.layout.ConversionPattern=%d %-5p %c [%t] - %m%n
>
> # quiet some packages that are too verbose
> log4j.logger.org.elasticsearch=WARN
> log4j.logger.org.apache.hadoop=WARN
> log4j.logger.org.apache.hadoop.hbase.zookeeper=ERROR
> log4j.logger.org.apache.hadoop.util.NativeCodeLoader=ERROR
> log4j.logger.org.apache.spark=WARN
> log4j.logger.org.apache.zookeeper=ERROR
> log4j.logger.org.eclipse.jetty=WARN
> log4j.logger.org.spark-project.jetty=WARN
> log4j.logger.akka=WARN
>
>
>
> On 10 Sep 2018, at 16.22, József Hábit <jozsef.ha...@gmail.com> wrote:
>
> Hello,
>
> what is the proper way to rotate the log files generated by Pio/UR?
>
> Thanks in advance!
> Jozsef Habit
>
>
>
