Re: bash shell question

[Larry Ploetz]

On 1/11/09 8:44 AM, John Summerfield wrote:


Probably. How do you know how many blocks to skip?


I don't -- I calculate approximate bytes not-sparse from end-of-file
using stat -- see below. Coincidentally, it happens to be the number of
bytes in all and only the blocks at the end of the file that contain
non-sparse data. Or, to put it another way, I don't calculate how many
blocks to skip from the beginning of the file, I calculate how many
blocks to read from the end of the file.





...


It is easy? I'm missing something.



Again, this is only for log files -- only appended to, for when they get
sparse because of logrotate copytruncate or "> filename" or similar
truncation, sparse-ness at the beginning of the file.





See my post Tue, 6 Jan 2009 14:30:24 -0800 for a full example, but
here's the main part:

tail -c $(( $(stat -c %b\*%B daemon.log) )) daemon.log ...

Pretty easy, eh? Again, again, this only works for log files, although I
should have said for sparse or not-sparse log files, and if sparse,
where sparse-ness is at the beginning.

- Larry

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390

Re: bash shell question

[John Summerfield]

Erik N Johnson wrote:

The man page I have claims to be part of 4th Berkeley Distribution
Manual Section 8.

However the authors both have redhat e-mail addresses.

Its inclusion in section 8 does more or less qualify it as 'system'
software, which in modern systems one CERTAINLY expects to store
sparse files efficiently.  So I agree, this probably ought to be
brought to the attention of the package maintainers.  Anybody know who
that would be?


If you got it from redhat, bugzilla.redhat.com
If you got it from SUSE, report it there.
Likewise Debian, Slack.

If I rote some software and RH/SUSE/Debian etc distributed it to
gazillions of people, I would not want those gazillions of people
complaining to me that it doesn't work. That's what the distro-makers
should be handling.

fwiw if you want support for PHP4 now, that's exactly what you must do.
rhe folk at php.net have gone on to other things and effectively
orphaned php4. As all software writers must do.



--

Cheers
John

-- spambait
1...@coco.merseine.nu  z1...@coco.merseine.nu
-- Advice
http://webfoot.com/advice/email.top.php
http://www.catb.org/~esr/faqs/smart-questions.html
http://support.microsoft.com/kb/555375

You cannot reply off-list:-)

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390

Re: bash shell question

[John Summerfield]

Larry Ploetz wrote:

On 1/7/09 1:44 PM, John Summerfield wrote:


I see no reason logrotate should not handle sparse files well when
copying them.


Yeah, there's no reason why it shouldn't, but it doesn't (see my empiric
test in previous post).



06:42 [sum...@numbat ~]$ cp --help | grep sparse
  --sparse=WHENcontrol creation of sparse files

This is probably the best way of cleaning the NULs people are concerned
about.


I've always interpreted "crude heuristic" to mean that cp reads all the
virtual \x00's, but writes sparsely. Since what I want with sparse log
files, where the sparse-ness is always at the beginning (corresponding
to the previously archived portions of the log file),  this didn't seem
as efficient as skipping over the beginning blocks that were only sparse:


Probably. How do you know how many blocks to skip?




$ man cp | grep -iwC1 sparse
   --sparse=WHEN
  control creation of sparse files

--

   By default, sparse SOURCE files are detected by a crude
heuristic  and
   the  corresponding  DEST  file  is  made  sparse as well.  That
is the
   behavior selected by --sparse=auto.  Specify --sparse=always to
create
   a  sparse  DEST  file  whenever the SOURCE file contains a long
enough
   sequence of zero bytes.  Use --sparse=never  to  inhibit
creation  of
   sparse files.

And since it's so easy to determine which block first contains real
data, I figured, "why not start copying there?"


It is easy? I'm missing something.



Again, this is only for log files -- only appended to, for when they get
sparse because of logrotate copytruncate or "> filename" or similar
truncation, sparse-ness at the beginning of the file.




--

Cheers
John

-- spambait
1...@coco.merseine.nu  z1...@coco.merseine.nu
-- Advice
http://webfoot.com/advice/email.top.php
http://www.catb.org/~esr/faqs/smart-questions.html
http://support.microsoft.com/kb/555375

You cannot reply off-list:-)

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390

Re: bash shell question

[Patrick Spinler]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Larry Ploetz wrote:

> Or ripping cp's "crude heuristic" out and
> putting it here.

That's probably your best bet.  It doesn't look too complex:

  /* Even with --sparse=always, try to create holes only
 if the destination is a regular file.  */
  if (x->sparse_mode == SPARSE_ALWAYS && S_ISREG (sb.st_mode))
make_holes = true;

#if HAVE_STRUCT_STAT_ST_BLOCKS
  if (x->sparse_mode == SPARSE_AUTO && S_ISREG (sb.st_mode))
{
  /* Use a heuristic to determine whether SRC_NAME contains any
 sparse blocks. */

  if (fstat (source_desc, &sb))
{
  error (0, errno, _("cannot fstat %s"), quote (src_name));
  return_val = false;
  goto close_src_and_dst_desc;
}

  /* If the file has fewer blocks than would normally
 be needed for a file of its size, then
 at least one of the blocks in the file is a hole. */
  if (S_ISREG (sb.st_mode)
  && sb.st_size / ST_NBLOCKSIZE > ST_NBLOCKS (sb))
make_holes = true;
}
#endif

The loop where it does the actual file copying is a bit longer and more
complex (long enough to be annoying in an email), but can be found in
copy.c in the coreutils srpm.  Hopefully it should be possible to copy
the ideas to logrotate.

Let me know if you'd like my help.

- -- Pat

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAklpY6AACgkQNObCqA8uBswTwQCbBkQNMk7+ebbuQoTJp8D9VH7B
Fd4AnRWc/n2VoxvZV5PHz/ZPzGv6xKnl
=NRzg
-END PGP SIGNATURE-

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390

Re: bash shell question

[Larry Ploetz]

On 1/9/09 4:53 PM, Patrick Spinler wrote:

...

At this point I'd suggest reporting a bug through the distributions,
probably all three of Novell (via OpenSUSE), Redhat (via Fedora) and
Debian.  Create a bug entry in each distro's bug track.  If you provide
a reproducible problem report, and especially a code patch that fixes
it, it should be easy to get the fix in.



Looks like this needs to be changed, in logrotate.c 3.7.4:

 413 while ((cnt = read(fdcurr, buf, sizeof(buf))) > 0) {
 414 if (write(fdsave, buf, cnt) != cnt) {
 415 message(MESS_ERROR, "error writing to %s: %s\n",
 416 saveLog, strerror(errno));
 417 close(fdcurr);
 418 close(fdsave);
 419 return 1;
 420 }

However, I'm clueless -- in general, but specifically about how to
convert this into the equivalent of "cp --sparse=auto", as John
Summerfield pointed out. Short of making a system (3) call (probably not
portable) to invoke cp. Or ripping cp's "crude heuristic" out and
putting it here.

I recently read an article somewhere about seek_hole and seek_data, from
ZFS, being implemented soon in Linux, and/or an alternative someone
submitted, but can't find it now :-(

- Larry

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390

Re: bash shell question

[Patrick Spinler]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1


Well, according to this reference to the debian package information,
http://tinyurl.com/8enmjq, logrotate originated in Fedora/Redhat.  It
also has a list of original developers you could google.

FreeBSD's logrotate, here:
http://www.freebsdsoftware.org/sysutils/logrotate.html, has no notes on
it's origin, but is about from the same time (1999) that debian obtained
theirs.

The fedora package info, here,
https://admin.fedoraproject.org/pkgdb/packages/name/logrotate, appears
to have a link to the current fedora developer.

At this point I'd suggest reporting a bug through the distributions,
probably all three of Novell (via OpenSUSE), Redhat (via Fedora) and
Debian.  Create a bug entry in each distro's bug track.  If you provide
a reproducible problem report, and especially a code patch that fixes
it, it should be easy to get the fix in.

- -- Pat

Erik N Johnson wrote:
> The man page I have claims to be part of 4th Berkeley Distribution
> Manual Section 8.
>
> However the authors both have redhat e-mail addresses.
>
> Its inclusion in section 8 does more or less qualify it as 'system'
> software, which in modern systems one CERTAINLY expects to store
> sparse files efficiently.  So I agree, this probably ought to be
> brought to the attention of the package maintainers.  Anybody know who
> that would be?
>
> Erik Johnson
>
> On Thu, Jan 8, 2009 at 2:32 PM, Larry Ploetz  wrote:
>> On 1/8/09 12:21 PM, Erik N Johnson wrote:
>>> Does this failure to behave as one might hope perhaps constitute a bug
>>> in a widely used admin utility?
>>>
>>>
>> Hmmm...I think so, now that you mention it...at least, I know I'd prefer
>> logrotate to keep sparse files sparse.
>>
>> - Larry
>>
>> --
>> For LINUX-390 subscribe / signoff / archive access instructions,
>> send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or
>> visit
>> http://www.marist.edu/htbin/wlvindex?LINUX-390
>>
>
> --
> For LINUX-390 subscribe / signoff / archive access instructions,
> send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
> http://www.marist.edu/htbin/wlvindex?LINUX-390

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkln8aQACgkQNObCqA8uBsxjYgCfRAT0jKx+mU+0PuUGwwyFlI0M
wDIAnjfAvd9UTRq6qpUJ2rPhjlcvxkWa
=puM5
-END PGP SIGNATURE-

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390

Re: bash shell question

[Erik N Johnson]

The man page I have claims to be part of 4th Berkeley Distribution
Manual Section 8.

However the authors both have redhat e-mail addresses.

Its inclusion in section 8 does more or less qualify it as 'system'
software, which in modern systems one CERTAINLY expects to store
sparse files efficiently.  So I agree, this probably ought to be
brought to the attention of the package maintainers.  Anybody know who
that would be?

Erik Johnson

On Thu, Jan 8, 2009 at 2:32 PM, Larry Ploetz  wrote:
> On 1/8/09 12:21 PM, Erik N Johnson wrote:
>>
>> Does this failure to behave as one might hope perhaps constitute a bug
>> in a widely used admin utility?
>>
>>
>
> Hmmm...I think so, now that you mention it...at least, I know I'd prefer
> logrotate to keep sparse files sparse.
>
> - Larry
>
> --
> For LINUX-390 subscribe / signoff / archive access instructions,
> send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or
> visit
> http://www.marist.edu/htbin/wlvindex?LINUX-390
>

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390

Re: bash shell question

[Larry Ploetz]

On 1/8/09 12:21 PM, Erik N Johnson wrote:

Does this failure to behave as one might hope perhaps constitute a bug
in a widely used admin utility?




Hmmm...I think so, now that you mention it...at least, I know I'd prefer
logrotate to keep sparse files sparse.

- Larry

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390

Re: bash shell question

[Erik N Johnson]

Does this failure to behave as one might hope perhaps constitute a bug
in a widely used admin utility?

Erik Johnson

On Wed, Jan 7, 2009 at 4:00 PM, Larry Ploetz  wrote:
> On 1/7/09 1:44 PM, John Summerfield wrote:
>>
>> I see no reason logrotate should not handle sparse files well when
>> copying them.
>
> Yeah, there's no reason why it shouldn't, but it doesn't (see my empiric
> test in previous post).
>
>>
>> 06:42 [sum...@numbat ~]$ cp --help | grep sparse
>>  --sparse=WHENcontrol creation of sparse files
>>
>> This is probably the best way of cleaning the NULs people are concerned
>> about.
>
> I've always interpreted "crude heuristic" to mean that cp reads all the
> virtual \x00's, but writes sparsely. Since what I want with sparse log
> files, where the sparse-ness is always at the beginning (corresponding
> to the previously archived portions of the log file),  this didn't seem
> as efficient as skipping over the beginning blocks that were only sparse:
>
> $ man cp | grep -iwC1 sparse
>   --sparse=WHEN
>  control creation of sparse files
>
> --
>
>   By default, sparse SOURCE files are detected by a crude
> heuristic  and
>   the  corresponding  DEST  file  is  made  sparse as well.  That
> is the
>   behavior selected by --sparse=auto.  Specify --sparse=always to
> create
>   a  sparse  DEST  file  whenever the SOURCE file contains a long
> enough
>   sequence of zero bytes.  Use --sparse=never  to  inhibit
> creation  of
>   sparse files.
>
> And since it's so easy to determine which block first contains real
> data, I figured, "why not start copying there?"
>
> Again, this is only for log files -- only appended to, for when they get
> sparse because of logrotate copytruncate or "> filename" or similar
> truncation, sparse-ness at the beginning of the file.
>
> - Larry
>
> --
> For LINUX-390 subscribe / signoff / archive access instructions,
> send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or
> visit
> http://www.marist.edu/htbin/wlvindex?LINUX-390
>

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390

Re: bash shell question

[Larry Ploetz]

On 1/7/09 1:44 PM, John Summerfield wrote:


I see no reason logrotate should not handle sparse files well when
copying them.


Yeah, there's no reason why it shouldn't, but it doesn't (see my empiric
test in previous post).



06:42 [sum...@numbat ~]$ cp --help | grep sparse
  --sparse=WHENcontrol creation of sparse files

This is probably the best way of cleaning the NULs people are concerned
about.


I've always interpreted "crude heuristic" to mean that cp reads all the
virtual \x00's, but writes sparsely. Since what I want with sparse log
files, where the sparse-ness is always at the beginning (corresponding
to the previously archived portions of the log file),  this didn't seem
as efficient as skipping over the beginning blocks that were only sparse:

$ man cp | grep -iwC1 sparse
   --sparse=WHEN
  control creation of sparse files

--

   By default, sparse SOURCE files are detected by a crude
heuristic  and
   the  corresponding  DEST  file  is  made  sparse as well.  That
is the
   behavior selected by --sparse=auto.  Specify --sparse=always to
create
   a  sparse  DEST  file  whenever the SOURCE file contains a long
enough
   sequence of zero bytes.  Use --sparse=never  to  inhibit
creation  of
   sparse files.

And since it's so easy to determine which block first contains real
data, I figured, "why not start copying there?"

Again, this is only for log files -- only appended to, for when they get
sparse because of logrotate copytruncate or "> filename" or similar
truncation, sparse-ness at the beginning of the file.

- Larry

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390

Re: bash shell question

[John Summerfield]

Larry Ploetz wrote:

On 1/7/09 10:15 AM, Mark Post wrote:

On 1/7/2009 at  5:44 AM, John
Summerfield


wrote:
-snip-


logrotate expects you're logging sensibly. John is not.

"sensibly" means the program can close and reopen the logfile on demand.



That is not a requirement for logrotate.  If nothing else the
"copytruncate" parameter will do what is desired.  I had to wind up
using that for ntpd, since it never closes its log file, and any
signals ntpd gets result in it shutting down.




Right, but copytruncate doesn't create sparse files when it's rotating
old generations (i.e., it probably reads the file and creates real \x00
where there was sparse-ness):


I see no reason logrotate should not handle sparse files well when
copying them.

06:42 [sum...@numbat ~]$ cp --help | grep sparse
  --sparse=WHENcontrol creation of sparse files

This is probably the best way of cleaning the NULs people are concerned
about.





--

Cheers
John

-- spambait
1...@coco.merseine.nu  z1...@coco.merseine.nu
-- Advice
http://webfoot.com/advice/email.top.php
http://www.catb.org/~esr/faqs/smart-questions.html
http://support.microsoft.com/kb/555375

You cannot reply off-list:-)

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390

Re: bash shell question

[John McKown]

I am the OP. What I have done is use a neat little utility called
cronolog. Basically what we currently have going is a daemon like:

daemon >>output.log 2>&1 &

and output.log just gets bigger and bigger until we shut it down, "rm" the
output.log (or mv it somewhere), then start up again. Using cronolog, I
can do the following instead:

(daemon 2>&1) | cronolog output.log.%Y-%m-%d

The cronolog program reads stdin and creates output.log.-MM-DD. At
each local midnight, cronolog will close its output and create a new
output.log.-MM-DD based on the new date. This way, we get a daily log
and don't need to do anything with the actual daemon.

http://sourceforge.net/projects/cronolog

--
Q: What do theoretical physicists drink beer from?
A: Ein Stein.

Maranatha!
John McKown

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390

Re: bash shell question

[John Summerfield]

Mark Post wrote:

On 1/7/2009 at  5:44 AM, John Summerfield 

wrote:
-snip-

logrotate expects you're logging sensibly. John is not.

"sensibly" means the program can close and reopen the logfile on demand.


That is not a requirement for logrotate.  If nothing else the "copytruncate" 
parameter will do what is desired.  I had to wind up using that for ntpd, since it never 
closes its log file, and any signals ntpd gets result in it shutting down.




I just checked the man page, it seems "copytruncate" does what Larry
suggested, and with the same warning about loss of data.




--

Cheers
John

-- spambait
1...@coco.merseine.nu  z1...@coco.merseine.nu
-- Advice
http://webfoot.com/advice/email.top.php
http://www.catb.org/~esr/faqs/smart-questions.html
http://support.microsoft.com/kb/555375

You cannot reply off-list:-)

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390

Re: bash shell question

[Melissa Curry]

Melissa Curry
Sales Representative

Velocity Software, Inc.
196-D Castro Street
Mountain View, CA
94041-1202

Office: (650) 964-8867
Cell: (415) 994-4579
fax: (650) 964-9012

meli...@velocitysoftware.com
www.velocitysoftware.com
- Original Message -
From: "Larry Ploetz" 
To: 
Sent: Wednesday, January 07, 2009 11:39 AM
Subject: Re: bash shell question



On 1/7/09 10:15 AM, Mark Post wrote:

On 1/7/2009 at  5:44 AM, John
Summerfield


wrote:
-snip-


logrotate expects you're logging sensibly. John is not.

"sensibly" means the program can close and reopen the logfile on demand.



That is not a requirement for logrotate.  If nothing else the
"copytruncate" parameter will do what is desired.  I had to wind up using
that for ntpd, since it never closes its log file, and any signals ntpd
gets result in it shutting down.




Right, but copytruncate doesn't create sparse files when it's rotating
old generations (i.e., it probably reads the file and creates real \x00
where there was sparse-ness):

$ du -csxB1 /tmp/t3*
397312  /tmp/t3
1277952 /tmp/t3.1
135168  /tmp/t3.2
8192/tmp/t3.logrotate
4096/tmp/t3.sparse
8192/tmp/t3.state
1830912 total
$ du -csxB1 --apparent-size /tmp/t3*
1654180 /tmp/t3
1267728 /tmp/t3.1
126808  /tmp/t3.2
43  /tmp/t3.logrotate
104857600   /tmp/t3.sparse
48  /tmp/t3.state
1048579048807   total

t3 is the current (and, coincidentally, closed) log, and is sparse
(397312 <= 1654180), but t3.1 (1277952 >= 1267728) and t3.2 (135168 >=
126808) aren't. Real sparse files like t3.sparse (4096 <= 104857600)
are almost always smaller than their apparent size.

- Larry

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or
visit
http://www.marist.edu/htbin/wlvindex?LINUX-390



--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390

Re: bash shell question

[Larry Ploetz]

On 1/7/09 11:47 AM, Mark Post wrote:


There are only a very few sparse files ever written to /var/log/.  If space is 
a concern because they might get really big when being copied, then having 
logrotate compress them will solve that.  The few sparse files I am aware of 
aren't kept open all the time, so copytruncate isn't necessary for them in the 
first place; they just get renamed.




I'm not sure the original poster logged to /var/log -- I was just
suggesting a solution for an existing problem, not a general solution.
Even so, logrotate is the process that created the sparse file in the
first place, when using copytruncate, then subsequently un-sparsed it's
own sparse files. I really hate reading billions of virtual \x00 for
compression, much less the virtually-same virtual \x00 each time a new
generation is compressed. Hence my solution with sparse log files of
reading only non-sparse blocks (well, reading the first non-sparse block
and subsequent blocks) and feeding them, sans leading \x00's, to bzip2.

I think logrotate's copytruncate is a remedial solution at best, and if
you were going to resort to logrotate anyway, you may as well use my
method instead, until you implemented proper regular log closing and
rotation. Or just say to hell with it and let daemon run forever, using
my method always -- keeps disk space usage to a minimum, allows
compression of old logs, and splitting of logs by date/time/size.

- Larry

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390

Re: bash shell question

[Mark Post]

>>> On 1/7/2009 at  2:39 PM, Larry Ploetz  wrote: 
-snip- 
> Right, but copytruncate doesn't create sparse files when it's rotating
> old generations (i.e., it probably reads the file and creates real \x00
> where there was sparse-ness):

There are only a very few sparse files ever written to /var/log/.  If space is 
a concern because they might get really big when being copied, then having 
logrotate compress them will solve that.  The few sparse files I am aware of 
aren't kept open all the time, so copytruncate isn't necessary for them in the 
first place; they just get renamed.


Mark Post

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390

Re: bash shell question

[Larry Ploetz]

On 1/7/09 10:15 AM, Mark Post wrote:

On 1/7/2009 at  5:44 AM, John Summerfield


wrote:
-snip-


logrotate expects you're logging sensibly. John is not.

"sensibly" means the program can close and reopen the logfile on demand.



That is not a requirement for logrotate.  If nothing else the "copytruncate" 
parameter will do what is desired.  I had to wind up using that for ntpd, since it never 
closes its log file, and any signals ntpd gets result in it shutting down.




Right, but copytruncate doesn't create sparse files when it's rotating
old generations (i.e., it probably reads the file and creates real \x00
where there was sparse-ness):

$ du -csxB1 /tmp/t3*
397312  /tmp/t3
1277952 /tmp/t3.1
135168  /tmp/t3.2
8192/tmp/t3.logrotate
4096/tmp/t3.sparse
8192/tmp/t3.state
1830912 total
$ du -csxB1 --apparent-size /tmp/t3*
1654180 /tmp/t3
1267728 /tmp/t3.1
126808  /tmp/t3.2
43  /tmp/t3.logrotate
104857600   /tmp/t3.sparse
48  /tmp/t3.state
1048579048807   total

t3 is the current (and, coincidentally, closed) log, and is sparse
(397312 <= 1654180), but t3.1 (1277952 >= 1267728) and t3.2 (135168 >=
126808) aren't. Real sparse files like t3.sparse (4096 <= 104857600)
are almost always smaller than their apparent size.

- Larry

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390

Re: bash shell question

[David Boyes]

As someone else pointed out, use logger and let syslogd worry about it.
Syslogd is smart enough to be well-behaved wrt to log files, and you can
also copy messages to central logging servers and a Whole Lot More.

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390

Re: bash shell question

[Larry Ploetz]

On 1/7/09 2:51 AM, John Summerfield wrote:


I think there exist these problems:
1. Unwritten data held in a buffer may exceed your arbitrary one or two
lines. It would be safer to live with that, either ignoring it or
programming around it depending on the significance of the log.
2. I was concerned about loss of data, but atm I don't see how it would
happen.



Right, I forgot to point that out -- there is a small chance of losing
some data...I'm assuming the kernel will pass unwritten-to-disk/buffered
data to bzip2, as it is a file that the kernel knows about (duh), but
daemon may write lines just after bzip2 has received EOF, but before
bash truncates the file -- those would be lost (I guess throwing a
`sync' command between the bzip2 and bash file truncation may eliminate
loss of data in process I/O buffer but not yet in kernel file buffer?).
I made the additional assumption that log files can reasonably be
99.999% saved, since one doesn't inspect 99% of all logs' contents.
However, as John points out, one should be aware of potential loss.
--

 
*
Larry Ploetz
Systems Administrator
Carnegie Institution of Washington
Department of Plant Biology
The Arabidopsis Information Resource
650 325 1521 x 296 la...@tairgroup.org *

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390

Re: bash shell question

[Mark Post]

>>> On 1/7/2009 at  5:44 AM, John Summerfield 
wrote: 
-snip-
> logrotate expects you're logging sensibly. John is not.
> 
> "sensibly" means the program can close and reopen the logfile on demand.

That is not a requirement for logrotate.  If nothing else the "copytruncate" 
parameter will do what is desired.  I had to wind up using that for ntpd, since 
it never closes its log file, and any signals ntpd gets result in it shutting 
down.


Mark Post

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390

Re: bash shell question

[John McKown]

On Wed, 7 Jan 2009, Ronald van der Laan wrote:

> John,
>
> Apache comes with rotatelogs that appends a timestamp to the logfile and
> starts a new logfile after a certain interval.
> This works without the need to stop and rotate the logfile.
>
> daemon args >>daemon.log
> >
>   daemon args 2>&1 |  rotatelogs daemon.log 86400
>
> This will put all output (normal + error) into a file called
> daemon.log.xxx and start with a new file each day.
> Change 86400 into 3600 and you'll get your logs per hour.
> With SuSE the code is in /usr/sbin/rotatelogs and for Apache2 it is called
> /usr/sbin/rotatelogs2
> There is also a man page for more information.
>
> Ronald van der Laan

Thanks! I think something like this was another thing that I was trying to
remember, but just drawing a blank on.

--
Q: What do theoretical physicists drink beer from?
A: Ein Stein.

Maranatha!
John McKown

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390

Re: bash shell question

[John Summerfield]

Larry Ploetz wrote:

On 1/6/09 1:50 PM, John McKown wrote:

I've got a small problem. I have a daemon which I cannot easily restart
because it is production and people are using it. The daemon is started
with something like:

daemon args>>daemon.log

The file "daemon.log" is getting very huge. The correct way to fix
this is
to stop the daemon, "mv" or "rm" the "daemon.log", then restart the
daemon. But I have a vague memory that it is sometimes possible to
"reset"
a file to "empty" simply by doing a:



daemon.log



You're probably thinking of "> daemon.log" to truncate a file.


and that will, at times, work even if the daemon is not restarted. Is my
memory correct? Or is that some sort of "special case" which does not
apply when bash does a>>  redirect of stdin?



No, append isn't a special case.


We plan to fix this by "not doing that!", but instead piping the stdout
from the daemon to a process called "cronolog" which works by
automatically changing the output file at midnight by changing the "date
portion" of the log name.



I believe some people don't think this is kosher, but I like:

bzip2 -c daemon.log > daemon.log.$(date +%Y%m%d).bz2 && > daemon.log

but if daemon keeps daemon.log open, daemon.log will become a sparse
file the next time daemon writes to it. A sparse file doesn't take up
any disk space, in this case, except what's subsequently written to it
by daemon. You have to remember not to copy the first line of daemon.log
if you do this again (e.g., tail +2 daemon.log | ..., or if you want to
be more efficient:


I think there exist these problems:
1. Unwritten data held in a buffer may exceed your arbitrary one or two
lines. It would be safer to live with that, either ignoring it or
programming around it depending on the significance of the log.
2. I was concerned about loss of data, but atm I don't see how it would
happen.

In addition to Ronald's suggestion, one can also run the messages
through logger, part of the syslog package.. However, both ideas require
a restart.


-- spambait
1...@coco.merseine.nu  z1...@coco.merseine.nu
-- Advice
http://webfoot.com/advice/email.top.php
http://www.catb.org/~esr/faqs/smart-questions.html
http://support.microsoft.com/kb/555375

You cannot reply off-list:-)

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390

Re: bash shell question

[John Summerfield]

Kok Leong Chan wrote:

Perhaps "logrotate" can help you here.

From the manpage:

logrotate  is  designed to ease administration of systems that generate
large numbers of log files.  It allows automatic rotation, compression,
removal, and mailing of log files.  Each log file may be handled daily,
weekly, monthly, or when it grows too large.



No.
logrotate expects you're logging sensibly. John is not.

"sensibly" means the program can close and reopen the logfile on demand.



--

Cheers
John

-- spambait
1...@coco.merseine.nu  z1...@coco.merseine.nu
-- Advice
http://webfoot.com/advice/email.top.php
http://www.catb.org/~esr/faqs/smart-questions.html
http://support.microsoft.com/kb/555375

You cannot reply off-list:-)

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390

Re: bash shell question

[Ronald van der Laan]

John,

Apache comes with rotatelogs that appends a timestamp to the logfile and
starts a new logfile after a certain interval.
This works without the need to stop and rotate the logfile.

daemon args >>daemon.log
>
  daemon args 2>&1 |  rotatelogs daemon.log 86400

This will put all output (normal + error) into a file called
daemon.log.xxx and start with a new file each day.
Change 86400 into 3600 and you'll get your logs per hour.
With SuSE the code is in /usr/sbin/rotatelogs and for Apache2 it is called
/usr/sbin/rotatelogs2
There is also a man page for more information.

Ronald van der Laan

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390

Re: bash shell question

[Kok Leong Chan]

Perhaps "logrotate" can help you here.

>From the manpage:

logrotate  is  designed to ease administration of systems that generate
large numbers of log files.  It allows automatic rotation, compression,
removal, and mailing of log files.  Each log file may be handled daily,
weekly, monthly, or when it grows too large.



- Chan Kok Leong




John McKown 
Sent by: Linux on 390 Port 
01/07/2009 05:50 AM
Please respond to
Linux on 390 Port 


To
LINUX-390@VM.MARIST.EDU
cc

Subject
bash shell question






I've got a small problem. I have a daemon which I cannot easily restart
because it is production and people are using it. The daemon is started
with something like:

daemon args >>daemon.log

The file "daemon.log" is getting very huge. The correct way to fix this is
to stop the daemon, "mv" or "rm" the "daemon.log", then restart the
daemon. But I have a vague memory that it is sometimes possible to "reset"
a file to "empty" simply by doing a:

>daemon.log

and that will, at times, work even if the daemon is not restarted. Is my
memory correct? Or is that some sort of "special case" which does not
apply when bash does a >> redirect of stdin?

We plan to fix this by "not doing that!", but instead piping the stdout
from the daemon to a process called "cronolog" which works by
automatically changing the output file at midnight by changing the "date
portion" of the log name.

--
Q: What do theoretical physicists drink beer from?
A: Ein Stein.

Maranatha!
John McKown

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or
visit
http://www.marist.edu/htbin/wlvindex?LINUX-390


--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390

Re: bash shell question

[Larry Ploetz]

On 1/6/09 1:50 PM, John McKown wrote:

I've got a small problem. I have a daemon which I cannot easily restart
because it is production and people are using it. The daemon is started
with something like:

daemon args>>daemon.log

The file "daemon.log" is getting very huge. The correct way to fix this is
to stop the daemon, "mv" or "rm" the "daemon.log", then restart the
daemon. But I have a vague memory that it is sometimes possible to "reset"
a file to "empty" simply by doing a:



daemon.log



You're probably thinking of "> daemon.log" to truncate a file.


and that will, at times, work even if the daemon is not restarted. Is my
memory correct? Or is that some sort of "special case" which does not
apply when bash does a>>  redirect of stdin?



No, append isn't a special case.


We plan to fix this by "not doing that!", but instead piping the stdout
from the daemon to a process called "cronolog" which works by
automatically changing the output file at midnight by changing the "date
portion" of the log name.



I believe some people don't think this is kosher, but I like:

bzip2 -c daemon.log > daemon.log.$(date +%Y%m%d).bz2 && > daemon.log

but if daemon keeps daemon.log open, daemon.log will become a sparse
file the next time daemon writes to it. A sparse file doesn't take up
any disk space, in this case, except what's subsequently written to it
by daemon. You have to remember not to copy the first line of daemon.log
if you do this again (e.g., tail +2 daemon.log | ..., or if you want to
be more efficient:

tail -c $(( $(stat -c %b\*%B daemon.log) )) daemon.log | tr -d '\000'\| ...

(unless you're worried about embedded nulls in the real portion of
daemon.log, in which case replace "tr -d '\000'" with a slightly less
efficient "sed -e '1s/^\000//g'".)

If you do the above on a regular basis, you never have to bounce daemon,
and can still have, e.g. daily, log files. And bonus -- eventually you
can have a file that looks like it's larger than the file system it's on!

A good backup program will back up sparse files sparsely. I don't know
about TSM, but Legato Networker isn't a good backup program :-(

- Larry

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390