Re: Ideas for a new RollingFileAppender
that should be writable. I don't think you can make it totally headless because the user still needs to be aware a file is written and he needs write permissions to it, so we might as well make the default implementation work without creating files and provide an implementation that persists the data locally. So applications that upgrade to the new version behave the same. ** ** ** ** On Fri, Aug 23, 2013 at 10:54 AM, Dominik Psenner dpsen...@gmail.com wrote: Good Morning, I understand your point. That would be exactly how the current RFA behaves. However, people are lazy and when something doesn’t work they are first frustrated and then they may throw the framework over board or ask silly questions. The latter happens about once every month. It is not nice to have the answer ready “this is not supported and it won’t ever be” because then discussions start about the why-not and may-not-be or could-we-not. As I see it, the time has come that we should provide a solution that works. There are times when documentation does solve an issue but this time it is not. Of course this is not an easy task, but easy tasks are annoying anyway. Hard tasks make the life tasty, don’t they? J Cheers *Von:* d_k [mailto:mail...@gmail.com] *Gesendet:* Dienstag, 20. August 2013 21:16 *An:* Log4NET Dev *Betreff:* Re: Ideas for a new RollingFileAppender Will it be reasonable to say that as long as the target folder path is static log4net will clean up old files and if the target folder path is dynamic it will clean them as long as the process is up? On Thu, Aug 15, 2013 at 4:44 PM, Stefan Bodewig bode...@apache.org wrote: On 2013-08-14, Dominik Psenner wrote: 2013/8/14 Stefan Bodewig bode...@apache.org The application runs for 3 days and thus log4net rolls the logfile 3 times. Well, I live in a world where all log files of an application are kept inside the same folder :-) For me too, but I've seen enough to know better. Might be a matter of educating the culprits. Yes, I know things like this happen. More seriously, to me it sounds like a bit of feature creep. I wouldn't consider my logging framework to be the one who's responsible for mailing non empty files on rolling. That's the job of logrotate/logwatch or similar specialiced tools IMHO. One could even argue disposal of logfiles should be out of scope. Maybe that's just me. Yes and no. The framework should at least be able to prevent filesystem pollution by itself when told to clean up after itself. I'm not sure I agree with that. But arguing against that is almost arguing against having a RollingFileAppender in the first place. So I'll give in. Stefan ** **
Re: Ideas for a new RollingFileAppender
Hard tasks are fun, but I wouldn't want the next version to be stuck forever because of it. What if we make it optional? If the user wishes to roll dynamic paths he should flip a switch and specify a target path that should be writable. I don't think you can make it totally headless because the user still needs to be aware a file is written and he needs write permissions to it, so we might as well make the default implementation work without creating files and provide an implementation that persists the data locally. So applications that upgrade to the new version behave the same. On Fri, Aug 23, 2013 at 10:54 AM, Dominik Psenner dpsen...@gmail.comwrote: Good Morning, ** ** I understand your point. That would be exactly how the current RFA behaves. However, people are lazy and when something doesn’t work they are first frustrated and then they may throw the framework over board or ask silly questions. The latter happens about once every month. It is not nice to have the answer ready “this is not supported and it won’t ever be” because then discussions start about the why-not and may-not-be or could-we-not. ** ** As I see it, the time has come that we should provide a solution that works. There are times when documentation does solve an issue but this time it is not. Of course this is not an easy task, but easy tasks are annoying anyway. Hard tasks make the life tasty, don’t they? J ** ** Cheers ** ** *Von:* d_k [mailto:mail...@gmail.com] *Gesendet:* Dienstag, 20. August 2013 21:16 *An:* Log4NET Dev *Betreff:* Re: Ideas for a new RollingFileAppender ** ** Will it be reasonable to say that as long as the target folder path is static log4net will clean up old files and if the target folder path is dynamic it will clean them as long as the process is up? ** ** On Thu, Aug 15, 2013 at 4:44 PM, Stefan Bodewig bode...@apache.org wrote: On 2013-08-14, Dominik Psenner wrote: 2013/8/14 Stefan Bodewig bode...@apache.org The application runs for 3 days and thus log4net rolls the logfile 3 times. Well, I live in a world where all log files of an application are kept inside the same folder :-) For me too, but I've seen enough to know better. Might be a matter of educating the culprits. Yes, I know things like this happen. More seriously, to me it sounds like a bit of feature creep. I wouldn't consider my logging framework to be the one who's responsible for mailing non empty files on rolling. That's the job of logrotate/logwatch or similar specialiced tools IMHO. One could even argue disposal of logfiles should be out of scope. Maybe that's just me. Yes and no. The framework should at least be able to prevent filesystem pollution by itself when told to clean up after itself. I'm not sure I agree with that. But arguing against that is almost arguing against having a RollingFileAppender in the first place. So I'll give in. Stefan ** **
Re: Ideas for a new RollingFileAppender
Will it be reasonable to say that as long as the target folder path is static log4net will clean up old files and if the target folder path is dynamic it will clean them as long as the process is up? On Thu, Aug 15, 2013 at 4:44 PM, Stefan Bodewig bode...@apache.org wrote: On 2013-08-14, Dominik Psenner wrote: 2013/8/14 Stefan Bodewig bode...@apache.org The application runs for 3 days and thus log4net rolls the logfile 3 times. Well, I live in a world where all log files of an application are kept inside the same folder :-) For me too, but I've seen enough to know better. Might be a matter of educating the culprits. Yes, I know things like this happen. More seriously, to me it sounds like a bit of feature creep. I wouldn't consider my logging framework to be the one who's responsible for mailing non empty files on rolling. That's the job of logrotate/logwatch or similar specialiced tools IMHO. One could even argue disposal of logfiles should be out of scope. Maybe that's just me. Yes and no. The framework should at least be able to prevent filesystem pollution by itself when told to clean up after itself. I'm not sure I agree with that. But arguing against that is almost arguing against having a RollingFileAppender in the first place. So I'll give in. Stefan
Re: Ideas for a new RollingFileAppender
On 2013-08-14, Dominik Psenner wrote: 2013/8/14 Stefan Bodewig bode...@apache.org The application runs for 3 days and thus log4net rolls the logfile 3 times. Well, I live in a world where all log files of an application are kept inside the same folder :-) For me too, but I've seen enough to know better. Might be a matter of educating the culprits. Yes, I know things like this happen. More seriously, to me it sounds like a bit of feature creep. I wouldn't consider my logging framework to be the one who's responsible for mailing non empty files on rolling. That's the job of logrotate/logwatch or similar specialiced tools IMHO. One could even argue disposal of logfiles should be out of scope. Maybe that's just me. Yes and no. The framework should at least be able to prevent filesystem pollution by itself when told to clean up after itself. I'm not sure I agree with that. But arguing against that is almost arguing against having a RollingFileAppender in the first place. So I'll give in. Stefan
Re: Ideas for a new RollingFileAppender
You've looked into this far more than I have but do your really feel the persistent storage is needed? To me scenarios like I have manually deleted some log files or I have reconfigured the rolling conditions don't really have to be supported by our rolling logic. If people interfere with rolling they better clean up themselves IMHO. On 2013-08-13, Dominik Psenner wrote: Let's imagine a rolling condition that rolls every day. The dispose strategy is to keep a maximum of 30 files and the rolled file format is: C:/myfancypath/%date/logfile.log The application runs for 3 days and thus log4net rolls the logfile 3 times. Well, I live in a world where all log files of an application are kept inside the same folder :-) More seriously, to me it sounds like a bit of feature creep. I wouldn't consider my logging framework to be the one who's responsible for mailing non empty files on rolling. That's the job of logrotate/logwatch or similar specialiced tools IMHO. One could even argue disposal of logfiles should be out of scope. Maybe that's just me. Are you really sure you want to put that much non-logging logic into an appender? Stefan
Re: Ideas for a new RollingFileAppender
2013/8/14 Stefan Bodewig bode...@apache.org The application runs for 3 days and thus log4net rolls the logfile 3 times. Well, I live in a world where all log files of an application are kept inside the same folder :-) For me too, but I've seen enough to know better. More seriously, to me it sounds like a bit of feature creep. I wouldn't consider my logging framework to be the one who's responsible for mailing non empty files on rolling. That's the job of logrotate/logwatch or similar specialiced tools IMHO. One could even argue disposal of logfiles should be out of scope. Maybe that's just me. Yes and no. The framework should at least be able to prevent filesystem pollution by itself when told to clean up after itself. At the moment log4net is not up to solve that task. Are you really sure you want to put that much non-logging logic into an appender? .. not at all. But the ideas sound great and make sense, don't they? After all we would at least clean up that configuration mess that the current RFA has to live with. It's completely irrational at the moment. Even I have to read up things in the FAQ to get the configuration of the RFA right after 2 unsuccessful tries.
Re: Ideas for a new RollingFileAppender
Inlined.
On Mon, Aug 12, 2013 at 11:06 AM, Stefan Bodewig bode...@apache.org wrote:
[just changing the subject :-)]
On 2013-08-12, Dominik Psenner wrote:
Another issue that I just remembered now is that currently there's no
way to
roll empty files on date/time boundaries. Rolling only happens when a log
event is generated. If there's no log event, files won't be rolled. Thus
we
should investigate if we should implement the appender as a buffering
async
appender that does the rolling in his async work cycles and whose work
cycles do not depend on the availability of log events.
Making the appender async is a major change and it has ramifications that
might prove problematic like unflushed buffers.
And if we make the buffer of size 1 then its really synchronous again,
isn't it?
Does a 'roll' create a new file? So if there are no log event the folder
will be polluted with empty files?
If a 'roll' does not create a file then we can make the decision to roll or
not as part of the logging process.
Whats wrong with rolling on logging events?
Futher, the rolling file appender should probably receive a persistent
storage where he can keep a set of rolled files he has rolled in the
past.
This way we eliminate the dark magic behind detecting rolled files. It
allows the user also to modify or mix the rolling strategies and/or
rolling
filename configuration from one instance to the next. The persistence
will
have to include a set of filenames. It would probably be nice to allow
people configuring a relative filepath that will be stripped from the
filenames. Otherwise we would break the usecase where a user moves an
application from, let's say, drive C: to drive D: along with all logfiles
because the rolled filenames would no longer be found and some files
would
become zombies that pollute the drive. Thus a persistable rolling history
class could look alike:
Public class RollingHistory {
Public Liststring Files { get; }
I'm up for a clean way to roll but I think that a logging framework that
creates files other than logs is odd. And adding a dependency on some
persistent storage provider is also odd.
I think we can assume that applications that need to perform rolling on
size or date are probably long running which means we can save this
information in memory and in case the application crashed or restarts then
yes, the old files will remain on disk and it should be solved by a local
policy because it the exception, not the rule.
To exchange information between the rolling strategies, rolling
conditions
and the rolling file appender itself we should probably use a data class.
The name of it could be RollingContext and basically could be
implemented
like:
public class RollingContext {
public RollingHistory RollingHistory { get; }
public string Logfile { get; }
public long CurrentLogfileSize { get; }
That means that a rolling strategy could be an interface like:
public interface RollingStrategy {
public void DoRolling(RollingContext context);
which either throws exceptions on failure or returns true|false to notify
the caller that something went wrong and log the errors into the log4net
internal debug log.
The next topic is the rolling condition. The rolling condition decides if
rolling should be done right now. Thus its interface should be something
like:
public interface IRollingCondition {
public bool IsRollingConditionMet(RollingContext context);
Concrete rolling conditions could be:
* Roll if filesize exceeds a specific limit
* Roll if a specific date/time condition is met
== this is already kind of supported with an implementation of a
cron-like syntax
Making the rolling condition pluggable would greatly benefit others to
invent their own rolling conditions. For example an implementation of
IRollingCondition like:
public class ManualRollingCondition implements IRollingCondition {
public static bool RollOnNextCycle = false;
public bool IsRollingConditionMet(RollingContext context) {
if(RollOnNextCycle) {
// reset roll on next cycle
RollOnNextCycle = false;
return true;
}
return false;
could let the application that uses log4net decide when a file should be
rolled by a simple invoke to:
ManualRollingCondition.RollOnNextCycle=true;
And now we come to the disposal of old files. Since we have a persistent
set
of old files we should pass it into the disposing logic and the
interface to
it could look like:
Public interface IDisposeStrategy {
Public bool DoDisposal(RollingContext context);
And a concrete implementation of it could be:
Public class LimitNumberOfFilesDisposeStrategy implements
IDisposeStrategy {
Public int MaxNumberOfFiles { get; set; }
Public bool DoDisposal(RollingContext context) {
//
Re: Ideas for a new RollingFileAppender
On 2013-08-13, d_k wrote: Whats wrong with rolling on logging events? For one we have JIRA issues that say something is wrong :-) OK, as long as your application logs frequently, there is no problem. But if you have a policy that rolls on date boundaries and your application tends to be idle at night you won't know when the next log event after midnight happens - so when do you schedule backing up your logfiles? Rolling on events is fine as long as your policy is to roll on size but if it is time based you might get too fuzzy for some use cases. I'm also not sure going async is the right approach for this particular issue (it might be worthwile to go full asnyc in all appenders but that's a totally different story) - but I agree with the goal that rolling on dates should be predicitve. Why is the disposal of log files a different strategy? Shouldn't it be part of the rolling process? Shouldn't the RollingStrategy.Roll method create and delete the needed files? You may want to roll and dispose at different times. Roll based on size and dispose based on time or something like that. It will be far easier to implement that if both issues are divided. Stefan
Re: Ideas for a new RollingFileAppender
On 2013-08-12, Dominik Psenner wrote:
The log filename should be determined dynamically to allow users including
information like %date, %time in the filename. It has often been requested
and up to now it is supported only marginally.
+1
Another issue that I just remembered now is that currently there's no
way to roll empty files on date/time boundaries. Rolling only happens
when a log event is generated. If there's no log event, files won't be
rolled. Thus we should investigate if we should implement the appender
as a buffering async appender that does the rolling in his async work
cycles and whose work cycles do not depend on the availability of log
events.
I already said I'm not sure mixing this into async is the way to go. It
may be easier to just start timers if the rolling condition is a time
based one. No strong opinion here, though, as I haven't thought through
either approach.
Futher, the rolling file appender should probably receive a persistent
storage where he can keep a set of rolled files he has rolled in the
past.
You've looked into this far more than I have but do your really feel the
persistent storage is needed? To me scenarios like I have manually
deleted some log files or I have reconfigured the rolling conditions
don't really have to be supported by our rolling logic. If people
interfere with rolling they better clean up themselves IMHO.
That means that a rolling strategy could be an interface like:
public interface RollingStrategy {
public void DoRolling(RollingContext context);
and DoRolling would do what? Perform the actual rolling our just tell
us whether log files should be rolled? In the first case all those
strategies would implement the same copy file X.i to file X.i+1 logic
somewhere. Feels wrong.
The next topic is the rolling condition. The rolling condition decides if
rolling should be done right now.
OK, this answers part of my question above. So what does
RollingStrategy do?
Making the rolling condition pluggable would greatly benefit others to
invent their own rolling conditions.
Absolutely.
And now we come to the disposal of old files.
I agree it should be separate from rolling and be as pluggable as the
rolling condition.
Stefan
Re: Ideas for a new RollingFileAppender
Inlines again and sorry for the previous empty message. I hit the send button before I actually wrote something. ;-) 2013/8/13 d_k mail...@gmail.com Inlined. On Mon, Aug 12, 2013 at 11:06 AM, Stefan Bodewig bode...@apache.orgwrote: [just changing the subject :-)] On 2013-08-12, Dominik Psenner wrote: Another issue that I just remembered now is that currently there's no way to roll empty files on date/time boundaries. Rolling only happens when a log event is generated. If there's no log event, files won't be rolled. Thus we should investigate if we should implement the appender as a buffering async appender that does the rolling in his async work cycles and whose work cycles do not depend on the availability of log events. Making the appender async is a major change and it has ramifications that might prove problematic like unflushed buffers. And if we make the buffer of size 1 then its really synchronous again, isn't it? Does a 'roll' create a new file? That would depend on what rolling strategy is used. One might want to send emails whenever a logfile is not empty. So if there are no log event the folder will be polluted with empty files? If a 'roll' does not create a file then we can make the decision to roll or not as part of the logging process. If it is configured to do so, yes, it pollutes the file system with empty files. I'm up for a clean way to roll but I think that a logging framework that creates files other than logs is odd. And adding a dependency on some persistent storage provider is also odd. .. you've gotta stick what you've got. I think we can assume that applications that need to perform rolling on size or date are probably long running which means we can save this information in memory and in case the application crashed or restarts then yes, the old files will remain on disk and it should be solved by a local policy because it the exception, not the rule. You're wrong. Rolling is mainly there to limit the possible quota logfiles can use. I think that an XML hierarchy will be easier to parse than strings containing conditions. Something like: rollOndate hours='3' //rollOn disposefiles filenumber='50' //dispose And we can provide some custom data to be passed back to a custom rolling strategy as a string and the you can parse it the way you want in the strategy you've written. We're not stupid enough to invent something unparseable .. ;-)
Re: Ideas for a new RollingFileAppender
Making the appender async is a major change and it has ramifications that might prove problematic like unflushed buffers. And if we make the buffer of size 1 then its really synchronous again, isn't it? Yes. What's the point? A file appender has to be async if it wants to be fast. See: http://logging.apache.org/log4j/2.x/manual/async.html#Performance It's left up to whoever uses the framework to decide if he wants performance or safety.
