Re: Feature request: gzipped LyX files
On Tue, 2002-12-31 at 10:21, Bruce Sass wrote:
On Fri, 28 Dec 2002, Darren Freeman wrote:
On Sat, 2002-12-28 at 07:23, Bruce Sass wrote:
On Thu, 27 Dec 2002, Darren Freeman wrote:
So I would say this: apart from obvious shortening of bloated symbols,
leave them readable (and compatible!). As long as gzipping becomes the
standard, that's a good thing since it's a tiny penalty for a large
gain.
Frankly, I consider it a hack.
ya, also, tiny and large are relative
I think the gain would depend on the situation; negative in the case
of nothing but small live files, huge for archives of large files.
Is one group more important than the other?
No. But how many small LyX files have you seen lately? =)
That is all I usually see. Lately I've been importing programs
(scripts of a few K, those small and poorly documented system level
bits of code I've generated) by wrapping them with LyX or LaTeX and
NoWeb. I'll get around to generating the templates for the wrappers
directly from .lyx files sooner or later. So, I'm doing both small
files and editor manipulations.
Even those files sound like they benefit from compression. Try it on
them and see, I'd be amazed if a script wrapped in LyX wouldn't compress
nicely.
But when I made the comment I was referring to how large a seemingly
empty LyX-generated document is nowadays =)
Making compressed the default is raising the bar wrt systems LyX is
usable with, for no clear benefit. An old slow box can probably get a
new HD, cheap, but it is a lot tougher to cope with running out of CPU
cycles, memory, or patience...
I never said make it the *only* choice, just the default. Those that
have slow boxes will most probably be clever enough to deselect a
checkbox (or change the extension).
Making compressed the default means that in addition to uncompressing
that tarball I just downloaded, I'll have to wait for LyX to
uncompress the .lyx files in it. If I manually uncompress packaged
stuff it will be left behind as cruft when a new version of the
package is installed (the files don't appear in the package manager's
DB)... but I guess that wouldn't matter 'cause I'd be going in to
uncompress the LyX User's Guide anyways.
Whoa - how *slow* is your box??? Decompressing small text files, like
100k, isn't supposed to be obviously slower than just reading them off
the disc. My old Amiga did that pretty quickly and it ran at 28 MHz.
Why exactly would you be uncompressing the users guide manually? The
whole *point* is for it to be automatic. And if you remain with an old
version of LyX then I would expect that you have an old version of the
UG, so you don't need to decompress it anyway.
I would much rather see LyX default to using the fastest method
possible with respect to file handling, even if that means a less
human readable .lyx... but compressing everything is the kind of
Well as of right now we have a certain LyX format. Doing nothing to it
is the most compatible thing we can do. Compressing it with gzip is not
a lot worse as everyone with a half decent OS has gunzip handy.
The chances are that people reading a new LyX file on an old version of
LyX will get a segfault anyway so they probably don't care about
backwards compatibility =)
But in the interests of not gutting the core of LyX the text version of
the file shouldn't be mangled to bits to save space. That's what
compression is for.
thing best handled by a filesystem (imo). I mean, if I want to
There's nothing wrong with compressing files!
If I email a file to a friend I appreciate it being a small file, even
if we both use compressed filesystems. I shouldn't have to manually
compress it, it should just be small already. Same if I burn a CD or do
a million other file related things that don't use my host filesystem.
encrypt everything I install CFS, do cattach secret-lyx-files and
work as normal, if my system can handle or I'm willing to put up with
the overhead.
I would love to see LyX transparently handle *.lyx.{gz,bz2,zip,?} or
*.ly{g,b,z,?} (respectively), but it should be up to the user to
determine what the default behaviour will be.
Of course.
Maybe system and user preference options like `default compression
method', `enable automatic (re)compression', `compressed file suffix
style (force it or not)', ..., along with a `save as compressed' menu
item or [none, gzip, bzip, zip, ?], etc. radio buttons in a `save
file' dialog, eh.
Too many options bamboozle newbies. I know ;) Most people won't have an
issue with compressing new files, I expect. Many won't even know the
difference.
Ok, so put them in an advanced options dialog/submenu, with
defaults set to the status quo. ;)
I honestly don't see where your resistance to moving everyone over to
compression is coming from. Even coputers from 10 years ago can handle
compressing a text file. People saving
Re: Feature request: gzipped LyX files
On Sat, 2002-12-28 at 06:20, John Levon wrote: On Fri, Dec 27, 2002 at 07:35:07AM +1030, Darren Freeman wrote: What is the goal? The representation of the user's data is the goal, as I see it. There is more than one goal. And adding ugly compression code is a stupid thing to do before considering sensible limitation of the data itself. Doesn't have to be more than calls to zlib, as I understand it. Or at it's laziest, calls to gzip =) Fortunately this thing called source coding allows us to do both: create an easy to read text file, then store it with an efficiency close to 100 %. It's not a hack at all, it's called information theory. I see no reason not to use it here. It's just like creating a water-tight file format which can be converted to human-readable form, only done the other way around. I hate binary document files, yes, even when they're just compressed. Wha? So you like a document.txt file, but hate it when it's document.txt.gz ??? I don't think it's useful Why not? It's smaller isn't it? In the case of some LyX files, by a factor of 10 to 20. At any rate it's interesting to me. And I'm sure it's useful to lots of people. I would appreciate the option, even if it's not the default and is stealthily hidden so as to be as hard to find as possible. john Darren
Re: Feature request: gzipped LyX files
On Tue, 2002-12-31 at 07:44, Dr. Richard E. Hawkins wrote: On Fri, Dec 20, 2002 at 11:26:59AM +0100, Jean-Marc Lasgouttes wrote: Andre == Andre Poenitz [EMAIL PROTECTED] writes: Andre I'd rather make .lyx gzipped by default. Doesn't Staroffice do Andre something similar? We could have both the plain .lyx format and a compressed .lyz format. Or use the plain old .lyx.gz. We could also do what emacs does: if the file is already compressed, uncompress it and compress it back on saving. If it is not compressed, leve it alone. This seems to make the most sense. Plain text storage that can survive user error and disk failure is a strentgth of lyx. Here here. Aye, aye! Also, bz2 compression can handle losing bits out of the middle, unlike gzip. Interesting. It's a shame that bz2 isn't really widespread in its use, although it's probably installed on most systems. Andre This is an extra burden for people like me who regularly do Andre searchreplace in the .lyx file itself, but I'd guess that's Andre the minority. I would not be so sure about that. I assumed we were the overwhelming majority :) There's too many things you just can't do with lyx that need vi on the lyx file. VI can transparently un-gzip files. It transparently re-gzips them when you're done! At least mine does =) Less can also read gzips. So really, when you think about it, we're implementing less clever stuff than less, since we don't transparently edit .lyx.gz files =) hawk Darren
Re: Feature request: gzipped LyX files
On Fri, Jan 03, 2003 at 02:50:52AM +1030, Darren Freeman wrote: On Tue, 2002-12-31 at 07:44, Dr. Richard E. Hawkins wrote: On Fri, Dec 20, 2002 at 11:26:59AM +0100, Jean-Marc Lasgouttes wrote: Also, bz2 compression can handle losing bits out of the middle, unlike gzip. Interesting. It's a shame that bz2 isn't really widespread in its use, although it's probably installed on most systems. It's been part of the base install of FreeBSD for a year or so, and I assume similarly for the other *bsds. VI can transparently un-gzip files. It transparently re-gzips them when you're done! At least mine does =) Less can also read gzips. I think these are installtion-dependent options--it depends upon how they were compiled. hawk -- Richard E. Hawkins, Asst. Prof. of Economics/\ ASCII ribbon campaign [EMAIL PROTECTED] Smeal 178 (814) 375-4700 \ / against HTML mail These opinions will not be those of Xand postings. Penn State until it pays my retainer. / \
Re: Feature request: gzipped LyX files
On Fri, 2003-01-03 at 08:37, Dr. Richard E. Hawkins wrote: On Fri, Jan 03, 2003 at 02:50:52AM +1030, Darren Freeman wrote: On Tue, 2002-12-31 at 07:44, Dr. Richard E. Hawkins wrote: On Fri, Dec 20, 2002 at 11:26:59AM +0100, Jean-Marc Lasgouttes wrote: Also, bz2 compression can handle losing bits out of the middle, unlike gzip. Interesting. It's a shame that bz2 isn't really widespread in its use, although it's probably installed on most systems. It's been part of the base install of FreeBSD for a year or so, and I assume similarly for the other *bsds. Still gzip is more of a standard. Is there a comparable replacement for zlib, such as maybe a bz2lib? VI can transparently un-gzip files. It transparently re-gzips them when you're done! At least mine does =) Less can also read gzips. I think these are installtion-dependent options--it depends upon how they were compiled. Oh well - users can always pipe it through gzip manually if they don't have clever vi or less installations. hawk Darren
Re: Feature request: gzipped LyX files
On Tue, 2002-12-31 at 10:21, Bruce Sass wrote:
> On Fri, 28 Dec 2002, Darren Freeman wrote:
> > On Sat, 2002-12-28 at 07:23, Bruce Sass wrote:
> > > On Thu, 27 Dec 2002, Darren Freeman wrote:
> > > > > > So I would say this: apart from obvious shortening of bloated symbols,
> > > > > > leave them readable (and compatible!). As long as gzipping becomes the
> > > > > > standard, that's a good thing since it's a tiny penalty for a large
> > > > > > gain.
> > > > >
> > > > > Frankly, I consider it a hack.
> > >
> > > ya, also, "tiny" and "large" are relative
> > >
> > > I think the gain would depend on the situation; negative in the case
> > > of nothing but small live files, huge for archives of large files.
> > > Is one group more important than the other?
> >
> > No. But how many "small" LyX files have you seen lately? =)
>
> That is all I usually see. Lately I've been importing programs
> (scripts of a few K, those small and poorly documented system level
> bits of code I've generated) by wrapping them with LyX or LaTeX and
> NoWeb. I'll get around to generating the templates for the wrappers
> directly from .lyx files sooner or later. So, I'm doing both small
> files and editor manipulations.
Even those files sound like they benefit from compression. Try it on
them and see, I'd be amazed if a script wrapped in LyX wouldn't compress
nicely.
But when I made the comment I was referring to how large a seemingly
empty LyX-generated document is nowadays =)
> > > Making compressed the default is raising the bar wrt systems LyX is
> > > usable with, for no clear benefit. An old slow box can probably get a
> > > new HD, cheap, but it is a lot tougher to cope with running out of CPU
> > > cycles, memory, or patience...
> >
> > I never said make it the *only* choice, just the default. Those that
> > have slow boxes will most probably be clever enough to deselect a
> > checkbox (or change the extension).
>
> Making compressed the default means that in addition to uncompressing
> that tarball I just downloaded, I'll have to wait for LyX to
> uncompress the .lyx files in it. If I manually uncompress packaged
> stuff it will be left behind as cruft when a new version of the
> package is installed (the files don't appear in the package manager's
> DB)... but I guess that wouldn't matter 'cause I'd be going in to
> uncompress the LyX User's Guide anyways.
Whoa - how *slow* is your box??? Decompressing small text files, like
100k, isn't supposed to be obviously slower than just reading them off
the disc. My old Amiga did that pretty quickly and it ran at 28 MHz.
Why exactly would you be uncompressing the users guide manually? The
whole *point* is for it to be automatic. And if you remain with an old
version of LyX then I would expect that you have an old version of the
UG, so you don't need to decompress it anyway.
> I would much rather see LyX default to using the fastest method
> possible with respect to file handling, even if that means a less
> human readable .lyx... but compressing everything is the kind of
Well as of right now we have a certain LyX format. Doing nothing to it
is the most compatible thing we can do. Compressing it with gzip is not
a lot worse as everyone with a half decent OS has gunzip handy.
The chances are that people reading a new LyX file on an old version of
LyX will get a segfault anyway so they probably don't care about
backwards compatibility =)
But in the interests of not gutting the core of LyX the text version of
the file shouldn't be mangled to bits to save space. That's what
compression is for.
> thing best handled by a filesystem (imo). I mean, if I want to
There's nothing wrong with compressing files!
If I email a file to a friend I appreciate it being a small file, even
if we both use compressed filesystems. I shouldn't have to manually
compress it, it should just be small already. Same if I burn a CD or do
a million other file related things that don't use my host filesystem.
> encrypt everything I install CFS, do "cattach secret-lyx-files" and
> work as normal, if my system can handle or I'm willing to put up with
> the overhead.
>
>
> > > I would love to see LyX transparently handle *.lyx.{gz,bz2,zip,?} or
> > > *.ly{g,b,z,?} (respectively), but it should be up to the user to
> > > determine what the default behaviour will be.
> >
> > Of course.
> >
> > > Maybe system and user preference options like `default compression
> > > method', `enable automatic (re)compression', `compressed file suffix
> > > style (force it or not)', ..., along with a `save as compressed' menu
> > > item or [none, gzip, bzip, zip, ?], etc. radio buttons in a `save
> > > file' dialog, eh.
> >
> > Too many options bamboozle newbies. I know ;) Most people won't have an
> > issue with compressing new files, I expect. Many won't even know the
> > difference.
>
> Ok, so put them in an "advanced options" dialog/submenu, with
> defaults set to the status quo. ;)
I honestly don't see
Re: Feature request: gzipped LyX files
On Sat, 2002-12-28 at 06:20, John Levon wrote: > On Fri, Dec 27, 2002 at 07:35:07AM +1030, Darren Freeman wrote: > > > What is the goal? > > > > The representation of the user's data is the goal, as I see it. > > There is more than one goal. And adding ugly compression code is a > stupid thing to do before considering sensible limitation of the data > itself. Doesn't have to be more than calls to zlib, as I understand it. Or at it's laziest, calls to gzip =) > > Fortunately this thing called source coding allows us to do both: create > > an easy to read text file, then store it with an efficiency close to 100 > > %. It's not a hack at all, it's called information theory. I see no > > reason not to use it here. It's just like creating a water-tight file > > format which can be converted to human-readable form, only done the > > other way around. > > I hate binary document files, yes, even when they're just compressed. Wha? So you like a document.txt file, but hate it when it's document.txt.gz ??? > I don't think it's useful Why not? It's smaller isn't it? In the case of some LyX files, by a factor of 10 to 20. At any rate it's interesting to me. And I'm sure it's useful to lots of people. I would appreciate the option, even if it's not the default and is stealthily hidden so as to be as hard to find as possible. > john Darren
Re: Feature request: gzipped LyX files
On Tue, 2002-12-31 at 07:44, Dr. Richard E. Hawkins wrote: > On Fri, Dec 20, 2002 at 11:26:59AM +0100, Jean-Marc Lasgouttes wrote: > > > "Andre" == Andre Poenitz <[EMAIL PROTECTED]> writes: > > > > Andre> I'd rather make .lyx gzipped by default. Doesn't Staroffice do > > Andre> something similar? > > > We could have both the plain .lyx format and a compressed .lyz format. > > > Or use the plain old .lyx.gz. > > > We could also do what emacs does: if the file is already compressed, > > uncompress it and compress it back on saving. If it is not compressed, > > leve it alone. > > This seems to make the most sense. Plain text storage that can survive > user error and disk failure is a strentgth of lyx. Here here. Aye, aye! > Also, bz2 compression can handle losing bits out of the middle, unlike > gzip. Interesting. It's a shame that bz2 isn't really widespread in its use, although it's probably installed on most systems. > > Andre> This is an extra burden for people like me who regularly do > > Andre> search in the .lyx file itself, but I'd guess that's > > Andre> the minority. > > > I would not be so sure about that. > > I assumed we were the overwhelming majority :) > > There's too many things you just can't do with lyx that need vi on the > lyx file. VI can transparently un-gzip files. It transparently re-gzips them when you're done! At least mine does =) Less can also read gzips. So really, when you think about it, we're implementing less clever stuff than "less", since we don't transparently edit .lyx.gz files =) > hawk Darren
Re: Feature request: gzipped LyX files
On Fri, Jan 03, 2003 at 02:50:52AM +1030, Darren Freeman wrote: > On Tue, 2002-12-31 at 07:44, Dr. Richard E. Hawkins wrote: > > On Fri, Dec 20, 2002 at 11:26:59AM +0100, Jean-Marc Lasgouttes wrote: > > Also, bz2 compression can handle losing bits out of the middle, unlike > > gzip. > Interesting. It's a shame that bz2 isn't really widespread in its use, > although it's probably installed on most systems. It's been part of the base install of FreeBSD for a year or so, and I assume similarly for the other *bsds. > VI can transparently un-gzip files. It transparently re-gzips them when > you're done! At least mine does =) > Less can also read gzips. I think these are installtion-dependent options--it depends upon how they were compiled. hawk -- Richard E. Hawkins, Asst. Prof. of Economics/"\ ASCII ribbon campaign [EMAIL PROTECTED] Smeal 178 (814) 375-4700 \ / against HTML mail These opinions will not be those of Xand postings. Penn State until it pays my retainer. / \
Re: Feature request: gzipped LyX files
On Fri, 2003-01-03 at 08:37, Dr. Richard E. Hawkins wrote: > On Fri, Jan 03, 2003 at 02:50:52AM +1030, Darren Freeman wrote: > > On Tue, 2002-12-31 at 07:44, Dr. Richard E. Hawkins wrote: > > > On Fri, Dec 20, 2002 at 11:26:59AM +0100, Jean-Marc Lasgouttes wrote: > > > > Also, bz2 compression can handle losing bits out of the middle, unlike > > > gzip. > > > Interesting. It's a shame that bz2 isn't really widespread in its use, > > although it's probably installed on most systems. > > It's been part of the base install of FreeBSD for a year or so, and I > assume similarly for the other *bsds. Still gzip is more of a standard. Is there a comparable replacement for zlib, such as maybe a "bz2lib"? > > VI can transparently un-gzip files. It transparently re-gzips them when > > you're done! At least mine does =) > > > Less can also read gzips. > > I think these are installtion-dependent options--it depends upon how > they were compiled. Oh well - users can always pipe it through gzip manually if they don't have clever vi or less installations. > hawk Darren
Re: Feature request: gzipped LyX files
On Fri, Dec 20, 2002 at 11:26:59AM +0100, Jean-Marc Lasgouttes wrote: Andre == Andre Poenitz [EMAIL PROTECTED] writes: Andre I'd rather make .lyx gzipped by default. Doesn't Staroffice do Andre something similar? We could have both the plain .lyx format and a compressed .lyz format. Or use the plain old .lyx.gz. We could also do what emacs does: if the file is already compressed, uncompress it and compress it back on saving. If it is not compressed, leve it alone. This seems to make the most sense. Plain text storage that can survive user error and disk failure is a strentgth of lyx. Also, bz2 compression can handle losing bits out of the middle, unlike gzip. Andre This is an extra burden for people like me who regularly do Andre searchreplace in the .lyx file itself, but I'd guess that's Andre the minority. I would not be so sure about that. I assumed we were the overwhelming majority :) There's too many things you just can't do with lyx that need vi on the lyx file. hawk -- Richard E. Hawkins, Asst. Prof. of Economics/\ ASCII ribbon campaign [EMAIL PROTECTED] Smeal 178 (814) 375-4700 \ / against HTML mail These opinions will not be those of Xand postings. Penn State until it pays my retainer. / \
Re: Feature request: gzipped LyX files
On Fri, 28 Dec 2002, Darren Freeman wrote:
On Sat, 2002-12-28 at 07:23, Bruce Sass wrote:
On Thu, 27 Dec 2002, Darren Freeman wrote:
So I would say this: apart from obvious shortening of bloated symbols,
leave them readable (and compatible!). As long as gzipping becomes the
standard, that's a good thing since it's a tiny penalty for a large
gain.
Frankly, I consider it a hack.
ya, also, tiny and large are relative
I think the gain would depend on the situation; negative in the case
of nothing but small live files, huge for archives of large files.
Is one group more important than the other?
No. But how many small LyX files have you seen lately? =)
That is all I usually see. Lately I've been importing programs
(scripts of a few K, those small and poorly documented system level
bits of code I've generated) by wrapping them with LyX or LaTeX and
NoWeb. I'll get around to generating the templates for the wrappers
directly from .lyx files sooner or later. So, I'm doing both small
files and editor manipulations.
Making compressed the default is raising the bar wrt systems LyX is
usable with, for no clear benefit. An old slow box can probably get a
new HD, cheap, but it is a lot tougher to cope with running out of CPU
cycles, memory, or patience...
I never said make it the *only* choice, just the default. Those that
have slow boxes will most probably be clever enough to deselect a
checkbox (or change the extension).
Making compressed the default means that in addition to uncompressing
that tarball I just downloaded, I'll have to wait for LyX to
uncompress the .lyx files in it. If I manually uncompress packaged
stuff it will be left behind as cruft when a new version of the
package is installed (the files don't appear in the package manager's
DB)... but I guess that wouldn't matter 'cause I'd be going in to
uncompress the LyX User's Guide anyways.
I would much rather see LyX default to using the fastest method
possible with respect to file handling, even if that means a less
human readable .lyx... but compressing everything is the kind of
thing best handled by a filesystem (imo). I mean, if I want to
encrypt everything I install CFS, do cattach secret-lyx-files and
work as normal, if my system can handle or I'm willing to put up with
the overhead.
I would love to see LyX transparently handle *.lyx.{gz,bz2,zip,?} or
*.ly{g,b,z,?} (respectively), but it should be up to the user to
determine what the default behaviour will be.
Of course.
Maybe system and user preference options like `default compression
method', `enable automatic (re)compression', `compressed file suffix
style (force it or not)', ..., along with a `save as compressed' menu
item or [none, gzip, bzip, zip, ?], etc. radio buttons in a `save
file' dialog, eh.
Too many options bamboozle newbies. I know ;) Most people won't have an
issue with compressing new files, I expect. Many won't even know the
difference.
Ok, so put them in an advanced options dialog/submenu, with
defaults set to the status quo. ;)
- Bruce
Re: Feature request: gzipped LyX files
On Fri, Dec 20, 2002 at 11:26:59AM +0100, Jean-Marc Lasgouttes wrote: > > "Andre" == Andre Poenitz <[EMAIL PROTECTED]> writes: > Andre> I'd rather make .lyx gzipped by default. Doesn't Staroffice do > Andre> something similar? > We could have both the plain .lyx format and a compressed .lyz format. > Or use the plain old .lyx.gz. > We could also do what emacs does: if the file is already compressed, > uncompress it and compress it back on saving. If it is not compressed, > leve it alone. This seems to make the most sense. Plain text storage that can survive user error and disk failure is a strentgth of lyx. Also, bz2 compression can handle losing bits out of the middle, unlike gzip. > Andre> This is an extra burden for people like me who regularly do > Andre> search in the .lyx file itself, but I'd guess that's > Andre> the minority. > I would not be so sure about that. I assumed we were the overwhelming majority :) There's too many things you just can't do with lyx that need vi on the lyx file. hawk -- Richard E. Hawkins, Asst. Prof. of Economics/"\ ASCII ribbon campaign [EMAIL PROTECTED] Smeal 178 (814) 375-4700 \ / against HTML mail These opinions will not be those of Xand postings. Penn State until it pays my retainer. / \
Re: Feature request: gzipped LyX files
On Fri, 28 Dec 2002, Darren Freeman wrote:
> On Sat, 2002-12-28 at 07:23, Bruce Sass wrote:
> > On Thu, 27 Dec 2002, Darren Freeman wrote:
> > > > > So I would say this: apart from obvious shortening of bloated symbols,
> > > > > leave them readable (and compatible!). As long as gzipping becomes the
> > > > > standard, that's a good thing since it's a tiny penalty for a large
> > > > > gain.
> > > >
> > > > Frankly, I consider it a hack.
> >
> > ya, also, "tiny" and "large" are relative
> >
> > I think the gain would depend on the situation; negative in the case
> > of nothing but small live files, huge for archives of large files.
> > Is one group more important than the other?
>
> No. But how many "small" LyX files have you seen lately? =)
That is all I usually see. Lately I've been importing programs
(scripts of a few K, those small and poorly documented system level
bits of code I've generated) by wrapping them with LyX or LaTeX and
NoWeb. I'll get around to generating the templates for the wrappers
directly from .lyx files sooner or later. So, I'm doing both small
files and editor manipulations.
> > Making compressed the default is raising the bar wrt systems LyX is
> > usable with, for no clear benefit. An old slow box can probably get a
> > new HD, cheap, but it is a lot tougher to cope with running out of CPU
> > cycles, memory, or patience...
>
> I never said make it the *only* choice, just the default. Those that
> have slow boxes will most probably be clever enough to deselect a
> checkbox (or change the extension).
Making compressed the default means that in addition to uncompressing
that tarball I just downloaded, I'll have to wait for LyX to
uncompress the .lyx files in it. If I manually uncompress packaged
stuff it will be left behind as cruft when a new version of the
package is installed (the files don't appear in the package manager's
DB)... but I guess that wouldn't matter 'cause I'd be going in to
uncompress the LyX User's Guide anyways.
I would much rather see LyX default to using the fastest method
possible with respect to file handling, even if that means a less
human readable .lyx... but compressing everything is the kind of
thing best handled by a filesystem (imo). I mean, if I want to
encrypt everything I install CFS, do "cattach secret-lyx-files" and
work as normal, if my system can handle or I'm willing to put up with
the overhead.
> > I would love to see LyX transparently handle *.lyx.{gz,bz2,zip,?} or
> > *.ly{g,b,z,?} (respectively), but it should be up to the user to
> > determine what the default behaviour will be.
>
> Of course.
>
> > Maybe system and user preference options like `default compression
> > method', `enable automatic (re)compression', `compressed file suffix
> > style (force it or not)', ..., along with a `save as compressed' menu
> > item or [none, gzip, bzip, zip, ?], etc. radio buttons in a `save
> > file' dialog, eh.
>
> Too many options bamboozle newbies. I know ;) Most people won't have an
> issue with compressing new files, I expect. Many won't even know the
> difference.
Ok, so put them in an "advanced options" dialog/submenu, with
defaults set to the status quo. ;)
- Bruce
Re: Feature request: gzipped LyX files
On Sat, Dec 28, 2002 at 05:15:38PM +1030, Darren Freeman spake thusly: On Sat, 2002-12-28 at 01:23, Martin Vermeer wrote: On Fri, Dec 27, 2002 at 07:35:07AM +1030, Darren Freeman spake thusly: I don't quite buy this class of arguments that Moore's Law will obsolesce the art of efficient coding... Hey I had to give up my assembler in favour of a C compiler. Then I realised that not every cycle counts when a billion of them happen every second. I bet that on typical bodies of code, your C compiler actually produces more efficient code overall than you could by hand assembly! I'm not talking about dirty tricks, just about good sense. Like moving things outside a loop if they are constant for the loop variable (That's precisely what using column/row attributes as defaults is, in fact!). ... Have fun, and merry xmas, Darren A Happy Newyear! Agreed. A happy new year. ;) Martin Darren Martin msg50375/pgp0.pgp Description: PGP signature
Re: Feature request: gzipped LyX files
On Fri, Dec 27, 2002 at 07:35:07AM +1030, Darren Freeman spake thusly: On Sun, 2002-12-22 at 00:27, John Levon wrote: On Sat, Dec 21, 2002 at 10:51:42PM +1030, Darren Freeman wrote: ... So I would say this: apart from obvious shortening of bloated symbols, leave them readable (and compatible!). As long as gzipping becomes the standard, that's a good thing since it's a tiny penalty for a large gain. Frankly, I consider it a hack. Why? Are we concerned with the size of a gzipped file or the original?? What is the goal? The representation of the user's data is the goal, as I see it. Please consider that even if we store the tables in gzipped form, they will be unzipped before use and handling, even if only in memory. This makes the unzipped file size relevant from a performance viewpoint. I could think of other situations where we want the unzipped version actually on disk. Having the option of something that is both legible and compact -- as we have had so far with LyX -- is a strong point that I am loath to give up without a fight :-) We can come up with some ultra-optimised code that only LyX can read, in which the information rate is nearly optimal. Or we can make it more readable like a normal text file, but let the information rate go much lower, like 5 %. Fortunately this thing called source coding allows us to do both: create an easy to read text file, then store it with an efficiency close to 100 %. It's not a hack at all, it's called information theory. I see no reason not to use it here. It's just like creating a water-tight file format which can be converted to human-readable form, only done the other way around. Create a human readable form and then compress it into a water tight format. This format can then be manipulated using standard tools, like gunzip and a text editor. What could be more elegant? It's fine... but not as a substitute for doing a reasonably effective job in the first place (yes, even if that would slightly inflate the .gz file!). Remember also that a needlessly bloated text file format is a pain to work with, or even just read, in an editor too. I don't quite buy this class of arguments that Moore's Law will obsolesce the art of efficient coding... john Have fun, and merry xmas, Darren A Happy Newyear! Martin msg50376/pgp0.pgp Description: PGP signature
Re: Feature request: gzipped LyX files
On Sat, 2002-12-28 at 01:23, Martin Vermeer wrote: On Fri, Dec 27, 2002 at 07:35:07AM +1030, Darren Freeman spake thusly: On Sun, 2002-12-22 at 00:27, John Levon wrote: On Sat, Dec 21, 2002 at 10:51:42PM +1030, Darren Freeman wrote: ... So I would say this: apart from obvious shortening of bloated symbols, leave them readable (and compatible!). As long as gzipping becomes the standard, that's a good thing since it's a tiny penalty for a large gain. Frankly, I consider it a hack. Why? Are we concerned with the size of a gzipped file or the original?? What is the goal? The representation of the user's data is the goal, as I see it. Please consider that even if we store the tables in gzipped form, they will be unzipped before use and handling, even if only in memory. This makes the unzipped file size relevant from a performance viewpoint. I could think of other situations where we want the unzipped version actually on disk. Having the option of something that is both legible and compact -- as we have had so far with LyX -- is a strong point that I am loath to give up without a fight :-) I didn't say make it *more* bloated. I just meant that if we have an hour a day to code improvements to the file format (preferably in time for 1.3.0 which would only happen if the change is small enough to slip through the feature freeze), then we need to look at where the biggest gain is. And I say it's adding the ability to compress the file which shaves off most of the bytes. Create a human readable form and then compress it into a water tight format. This format can then be manipulated using standard tools, like gunzip and a text editor. What could be more elegant? It's fine... but not as a substitute for doing a reasonably effective job in the first place (yes, even if that would slightly inflate the .gz file!). Remember also that a needlessly bloated text file format is a pain to work with, or even just read, in an editor too. The format we have right at this very moment in time is already working. It's not about putting in more effort to make it more bloat^H^H^H^H^Hreadable, it's about leaving it alone until compression works, which doesn't sound that hard to be honest. not an offer to code I don't quite buy this class of arguments that Moore's Law will obsolesce the art of efficient coding... Hey I had to give up my assembler in favour of a C compiler. Then I realised that not every cycle counts when a billion of them happen every second. john Have fun, and merry xmas, Darren A Happy Newyear! Agreed. A happy new year. ;) Martin Darren
Re: Feature request: gzipped LyX files
On Sat, 2002-12-28 at 07:23, Bruce Sass wrote:
On Thu, 27 Dec 2002, Darren Freeman wrote:
So I would say this: apart from obvious shortening of bloated symbols,
leave them readable (and compatible!). As long as gzipping becomes the
standard, that's a good thing since it's a tiny penalty for a large
gain.
Frankly, I consider it a hack.
ya, also, tiny and large are relative
I think the gain would depend on the situation; negative in the case
of nothing but small live files, huge for archives of large files.
Is one group more important than the other?
No. But how many small LyX files have you seen lately? =)
Why? Are we concerned with the size of a gzipped file or the original??
What is the goal?
The representation of the user's data is the goal, as I see it.
Making compressed the default is raising the bar wrt systems LyX is
usable with, for no clear benefit. An old slow box can probably get a
new HD, cheap, but it is a lot tougher to cope with running out of CPU
cycles, memory, or patience...
I never said make it the *only* choice, just the default. Those that
have slow boxes will most probably be clever enough to deselect a
checkbox (or change the extension).
I would love to see LyX transparently handle *.lyx.{gz,bz2,zip,?} or
*.ly{g,b,z,?} (respectively), but it should be up to the user to
determine what the default behaviour will be.
Of course.
Maybe system and user preference options like `default compression
method', `enable automatic (re)compression', `compressed file suffix
style (force it or not)', ..., along with a `save as compressed' menu
item or [none, gzip, bzip, zip, ?], etc. radio buttons in a `save
file' dialog, eh.
Too many options bamboozle newbies. I know ;) Most people won't have an
issue with compressing new files, I expect. Many won't even know the
difference.
- Bruce
Have fun,
Darren
Re: Feature request: gzipped LyX files
On Thu, 27 Dec 2002, Darren Freeman wrote:
So I would say this: apart from obvious shortening of bloated symbols,
leave them readable (and compatible!). As long as gzipping becomes the
standard, that's a good thing since it's a tiny penalty for a large
gain.
Frankly, I consider it a hack.
ya, also, tiny and large are relative
I think the gain would depend on the situation; negative in the case
of nothing but small live files, huge for archives of large files.
Is one group more important than the other?
Why? Are we concerned with the size of a gzipped file or the original??
What is the goal?
The representation of the user's data is the goal, as I see it.
We can come up with some ultra-optimised code that only LyX can read, in
which the information rate is nearly optimal. Or we can make it more
readable like a normal text file, but let the information rate go much
lower, like 5 %.
Fortunately this thing called source coding allows us to do both: create
an easy to read text file, then store it with an efficiency close to 100
%. It's not a hack at all, it's called information theory. I see no
reason not to use it here. It's just like creating a water-tight file
format which can be converted to human-readable form, only done the
other way around.
Create a human readable form and then compress it into a water tight
format. This format can then be manipulated using standard tools, like
gunzip and a text editor. What could be more elegant?
Making compressed the default is raising the bar wrt systems LyX is
usable with, for no clear benefit. An old slow box can probably get a
new HD, cheap, but it is a lot tougher to cope with running out of CPU
cycles, memory, or patience...
I would love to see LyX transparently handle *.lyx.{gz,bz2,zip,?} or
*.ly{g,b,z,?} (respectively), but it should be up to the user to
determine what the default behaviour will be.
Maybe system and user preference options like `default compression
method', `enable automatic (re)compression', `compressed file suffix
style (force it or not)', ..., along with a `save as compressed' menu
item or [none, gzip, bzip, zip, ?], etc. radio buttons in a `save
file' dialog, eh.
- Bruce
Re: Feature request: gzipped LyX files
On Fri, Dec 27, 2002 at 07:35:07AM +1030, Darren Freeman wrote: What is the goal? The representation of the user's data is the goal, as I see it. There is more than one goal. And adding ugly compression code is a stupid thing to do before considering sensible limitation of the data itself. Fortunately this thing called source coding allows us to do both: create an easy to read text file, then store it with an efficiency close to 100 %. It's not a hack at all, it's called information theory. I see no reason not to use it here. It's just like creating a water-tight file format which can be converted to human-readable form, only done the other way around. I hate binary document files, yes, even when they're just compressed. I don't think it's useful john -- I will eat a rubber tire to the music of The Flight of the Bumblebee
Re: Feature request: gzipped LyX files
On Sat, Dec 28, 2002 at 05:15:38PM +1030, Darren Freeman spake thusly: > On Sat, 2002-12-28 at 01:23, Martin Vermeer wrote: > > On Fri, Dec 27, 2002 at 07:35:07AM +1030, Darren Freeman spake thusly: > > I don't quite buy this class of arguments that Moore's Law will > > obsolesce the art of efficient coding... > > Hey I had to give up my assembler in favour of a C compiler. Then I > realised that not every cycle counts when a billion of them happen every > second. I bet that on typical bodies of code, your C compiler actually produces more efficient code overall than you could by hand assembly! I'm not talking about dirty tricks, just about good sense. Like moving things outside a loop if they are constant for the loop variable (That's precisely what using column/row attributes as defaults is, in fact!). ... > > > Have fun, and merry xmas, > > > Darren > > > > A Happy Newyear! > > Agreed. A happy new year. ;) > > > Martin > > Darren > Martin msg50375/pgp0.pgp Description: PGP signature
Re: Feature request: gzipped LyX files
On Fri, Dec 27, 2002 at 07:35:07AM +1030, Darren Freeman spake thusly: > On Sun, 2002-12-22 at 00:27, John Levon wrote: > > On Sat, Dec 21, 2002 at 10:51:42PM +1030, Darren Freeman wrote: ... > > > So I would say this: apart from obvious shortening of bloated symbols, > > > leave them readable (and compatible!). As long as gzipping becomes the > > > standard, that's a good thing since it's a tiny penalty for a large > > > gain. > > > > Frankly, I consider it a hack. > > Why? Are we concerned with the size of a gzipped file or the original?? > > What is the goal? > > The representation of the user's data is the goal, as I see it. Please consider that even if we store the tables in gzipped form, they will be unzipped before use and handling, even if only in memory. This makes the unzipped file size relevant from a performance viewpoint. I could think of other situations where we want the unzipped version actually on disk. Having the option of something that is both legible and compact -- as we have had so far with LyX -- is a strong point that I am loath to give up without a fight :-) > We can come up with some ultra-optimised code that only LyX can read, in > which the information rate is nearly optimal. Or we can make it more > readable like a normal text file, but let the information rate go much > lower, like 5 %. > > Fortunately this thing called source coding allows us to do both: create > an easy to read text file, then store it with an efficiency close to 100 > %. It's not a hack at all, it's called information theory. I see no > reason not to use it here. It's just like creating a water-tight file > format which can be converted to human-readable form, only done the > other way around. > > Create a human readable form and then compress it into a water tight > format. This format can then be manipulated using standard tools, like > gunzip and a text editor. What could be more elegant? It's fine... but not as a substitute for doing a reasonably effective job in the first place (yes, even if that would slightly inflate the .gz file!). Remember also that a needlessly bloated text file format is a pain to work with, or even just read, in an editor too. I don't quite buy this class of arguments that Moore's Law will obsolesce the art of efficient coding... > > john > > Have fun, and merry xmas, > Darren A Happy Newyear! Martin msg50376/pgp0.pgp Description: PGP signature
Re: Feature request: gzipped LyX files
On Sat, 2002-12-28 at 01:23, Martin Vermeer wrote: > On Fri, Dec 27, 2002 at 07:35:07AM +1030, Darren Freeman spake thusly: > > > On Sun, 2002-12-22 at 00:27, John Levon wrote: > > > On Sat, Dec 21, 2002 at 10:51:42PM +1030, Darren Freeman wrote: > > ... > > > > > So I would say this: apart from obvious shortening of bloated symbols, > > > > leave them readable (and compatible!). As long as gzipping becomes the > > > > standard, that's a good thing since it's a tiny penalty for a large > > > > gain. > > > > > > Frankly, I consider it a hack. > > > > Why? Are we concerned with the size of a gzipped file or the original?? > > > > What is the goal? > > > > The representation of the user's data is the goal, as I see it. > > Please consider that even if we store the tables in gzipped form, they > will be unzipped before use and handling, even if only in memory. This > makes the unzipped file size relevant from a performance viewpoint. I > could think of other situations where we want the unzipped version > actually on disk. Having the option of something that is both legible > and compact -- as we have had so far with LyX -- is a strong point > that I am loath to give up without a fight :-) I didn't say make it *more* bloated. I just meant that if we have an hour a day to code improvements to the file format (preferably in time for 1.3.0 which would only happen if the change is small enough to slip through the feature freeze), then we need to look at where the biggest gain is. And I say it's adding the ability to compress the file which shaves off most of the bytes. > > Create a human readable form and then compress it into a water tight > > format. This format can then be manipulated using standard tools, like > > gunzip and a text editor. What could be more elegant? > > It's fine... but not as a substitute for doing a reasonably effective > job in the first place (yes, even if that would slightly inflate the .gz > file!). Remember also that a needlessly bloated text file format is a > pain to work with, or even just read, in an editor too. The format we have right at this very moment in time is already working. It's not about putting in more effort to make it more bloat^H^H^H^H^Hreadable, it's about leaving it alone until compression works, which doesn't sound that hard to be honest. > I don't quite buy this class of arguments that Moore's Law will > obsolesce the art of efficient coding... Hey I had to give up my assembler in favour of a C compiler. Then I realised that not every cycle counts when a billion of them happen every second. > > > john > > > > Have fun, and merry xmas, > > Darren > > A Happy Newyear! Agreed. A happy new year. ;) > Martin Darren
Re: Feature request: gzipped LyX files
On Sat, 2002-12-28 at 07:23, Bruce Sass wrote:
> On Thu, 27 Dec 2002, Darren Freeman wrote:
> > > > So I would say this: apart from obvious shortening of bloated symbols,
> > > > leave them readable (and compatible!). As long as gzipping becomes the
> > > > standard, that's a good thing since it's a tiny penalty for a large
> > > > gain.
> > >
> > > Frankly, I consider it a hack.
>
> ya, also, "tiny" and "large" are relative
>
> I think the gain would depend on the situation; negative in the case
> of nothing but small live files, huge for archives of large files.
> Is one group more important than the other?
No. But how many "small" LyX files have you seen lately? =)
> > Why? Are we concerned with the size of a gzipped file or the original??
> >
> > What is the goal?
> >
> > The representation of the user's data is the goal, as I see it.
> Making compressed the default is raising the bar wrt systems LyX is
> usable with, for no clear benefit. An old slow box can probably get a
> new HD, cheap, but it is a lot tougher to cope with running out of CPU
> cycles, memory, or patience...
I never said make it the *only* choice, just the default. Those that
have slow boxes will most probably be clever enough to deselect a
checkbox (or change the extension).
> I would love to see LyX transparently handle *.lyx.{gz,bz2,zip,?} or
> *.ly{g,b,z,?} (respectively), but it should be up to the user to
> determine what the default behaviour will be.
Of course.
> Maybe system and user preference options like `default compression
> method', `enable automatic (re)compression', `compressed file suffix
> style (force it or not)', ..., along with a `save as compressed' menu
> item or [none, gzip, bzip, zip, ?], etc. radio buttons in a `save
> file' dialog, eh.
Too many options bamboozle newbies. I know ;) Most people won't have an
issue with compressing new files, I expect. Many won't even know the
difference.
> - Bruce
Have fun,
Darren
Re: Feature request: gzipped LyX files
On Thu, 27 Dec 2002, Darren Freeman wrote:
> > > So I would say this: apart from obvious shortening of bloated symbols,
> > > leave them readable (and compatible!). As long as gzipping becomes the
> > > standard, that's a good thing since it's a tiny penalty for a large
> > > gain.
> >
> > Frankly, I consider it a hack.
ya, also, "tiny" and "large" are relative
I think the gain would depend on the situation; negative in the case
of nothing but small live files, huge for archives of large files.
Is one group more important than the other?
> Why? Are we concerned with the size of a gzipped file or the original??
>
> What is the goal?
>
> The representation of the user's data is the goal, as I see it.
>
> We can come up with some ultra-optimised code that only LyX can read, in
> which the information rate is nearly optimal. Or we can make it more
> readable like a normal text file, but let the information rate go much
> lower, like 5 %.
>
> Fortunately this thing called source coding allows us to do both: create
> an easy to read text file, then store it with an efficiency close to 100
> %. It's not a hack at all, it's called information theory. I see no
> reason not to use it here. It's just like creating a water-tight file
> format which can be converted to human-readable form, only done the
> other way around.
>
> Create a human readable form and then compress it into a water tight
> format. This format can then be manipulated using standard tools, like
> gunzip and a text editor. What could be more elegant?
Making compressed the default is raising the bar wrt systems LyX is
usable with, for no clear benefit. An old slow box can probably get a
new HD, cheap, but it is a lot tougher to cope with running out of CPU
cycles, memory, or patience...
I would love to see LyX transparently handle *.lyx.{gz,bz2,zip,?} or
*.ly{g,b,z,?} (respectively), but it should be up to the user to
determine what the default behaviour will be.
Maybe system and user preference options like `default compression
method', `enable automatic (re)compression', `compressed file suffix
style (force it or not)', ..., along with a `save as compressed' menu
item or [none, gzip, bzip, zip, ?], etc. radio buttons in a `save
file' dialog, eh.
- Bruce
Re: Feature request: gzipped LyX files
On Fri, Dec 27, 2002 at 07:35:07AM +1030, Darren Freeman wrote: > What is the goal? > > The representation of the user's data is the goal, as I see it. There is more than one goal. And adding ugly compression code is a stupid thing to do before considering sensible limitation of the data itself. > Fortunately this thing called source coding allows us to do both: create > an easy to read text file, then store it with an efficiency close to 100 > %. It's not a hack at all, it's called information theory. I see no > reason not to use it here. It's just like creating a water-tight file > format which can be converted to human-readable form, only done the > other way around. I hate binary document files, yes, even when they're just compressed. I don't think it's useful john -- "I will eat a rubber tire to the music of The Flight of the Bumblebee"
Re: Feature request: gzipped LyX files
On Sun, 2002-12-22 at 00:27, John Levon wrote: On Sat, Dec 21, 2002 at 10:51:42PM +1030, Darren Freeman wrote: I agree in principle, but would like to see tests to show which activity is actually more productive. Remember that shrinking the symbol names Why would we do that ? Because I did very basic tests to show that implementing compression as a standard will have a much, much larger impact on file size than stuffing around with the symbols in the XML source. In fact, removing some symbols increased the file size slightly, for reasons unknown. Since it was way to simple to prove that this is the case on a real document, somebody could jump in and do it for that case. Well either that or just argue until we all get sick of the idea, and achieve nothing =) shouldn't have much of an effect on the gzip file size.. You'd have to actually start trimming away symbols to get a real benefit. Exactly. And with the advent of lyx2lyx the final argument against not outputting default values for tabulars is gone. Like I said before, if it's really easy then go right ahead. But there is an even bigger benefit to using zlib. And since that sounds pretty easy to somebody with the familiarity to do it, I would like to see compression used first. Just removing the default params from a large table can easily halve the file size. Did you try it *with* compression? Because I did, but on a small dummy document. And it didn't help. Then consider that there's no reason : \end_inset /cell cell \begin_inset Text can't be : /cellcell Sure. That would be a clever thing to do, right after enabling compression =) So I would say this: apart from obvious shortening of bloated symbols, leave them readable (and compatible!). As long as gzipping becomes the standard, that's a good thing since it's a tiny penalty for a large gain. Frankly, I consider it a hack. Why? Are we concerned with the size of a gzipped file or the original?? What is the goal? The representation of the user's data is the goal, as I see it. We can come up with some ultra-optimised code that only LyX can read, in which the information rate is nearly optimal. Or we can make it more readable like a normal text file, but let the information rate go much lower, like 5 %. Fortunately this thing called source coding allows us to do both: create an easy to read text file, then store it with an efficiency close to 100 %. It's not a hack at all, it's called information theory. I see no reason not to use it here. It's just like creating a water-tight file format which can be converted to human-readable form, only done the other way around. Create a human readable form and then compress it into a water tight format. This format can then be manipulated using standard tools, like gunzip and a text editor. What could be more elegant? john Have fun, and merry xmas, Darren
Re: Feature request: gzipped LyX files
On Sun, 2002-12-22 at 00:27, John Levon wrote: > On Sat, Dec 21, 2002 at 10:51:42PM +1030, Darren Freeman wrote: > > > I agree in principle, but would like to see tests to show which activity > > is actually more productive. Remember that shrinking the symbol names > > Why would we do that ? Because I did very basic tests to show that implementing compression as a standard will have a much, much larger impact on file size than stuffing around with the symbols in the XML source. In fact, removing some symbols increased the file size slightly, for reasons unknown. Since it was way to simple to prove that this is the case on a real document, somebody could jump in and do it for that case. Well either that or just argue until we all get sick of the idea, and achieve nothing =) > > shouldn't have much of an effect on the gzip file size.. You'd have to > > actually start trimming away symbols to get a real benefit. > > Exactly. And with the advent of lyx2lyx the final argument against not > outputting default values for tabulars is gone. Like I said before, if it's really easy then go right ahead. But there is an even bigger benefit to using zlib. And since that sounds pretty easy to somebody with the familiarity to do it, I would like to see compression used first. > Just removing the default params from a large table can easily halve the > file size. Did you try it *with* compression? Because I did, but on a small dummy document. And it didn't help. > Then consider that there's no reason : > > \end_inset > > > \begin_inset Text > > can't be : > > Sure. That would be a clever thing to do, right after enabling compression =) > > So I would say this: apart from obvious shortening of bloated symbols, > > leave them readable (and compatible!). As long as gzipping becomes the > > standard, that's a good thing since it's a tiny penalty for a large > > gain. > > Frankly, I consider it a hack. Why? Are we concerned with the size of a gzipped file or the original?? What is the goal? The representation of the user's data is the goal, as I see it. We can come up with some ultra-optimised code that only LyX can read, in which the information rate is nearly optimal. Or we can make it more readable like a normal text file, but let the information rate go much lower, like 5 %. Fortunately this thing called source coding allows us to do both: create an easy to read text file, then store it with an efficiency close to 100 %. It's not a hack at all, it's called information theory. I see no reason not to use it here. It's just like creating a water-tight file format which can be converted to human-readable form, only done the other way around. Create a human readable form and then compress it into a water tight format. This format can then be manipulated using standard tools, like gunzip and a text editor. What could be more elegant? > john Have fun, and merry xmas, Darren
Re: Feature request: gzipped LyX files
On Sun, 2002-12-22 at 04:13, Martin Vermeer wrote: On Sat, Dec 21, 2002 at 10:51:42PM +1030, Darren Freeman spake thusly: ... But as for eliminating symbols altogether by more intelligent use of defaults, I'd say: go for it! That will shave off some data. But before making the statement definitive, I had another go. I removed what I thought looked like redundant info, from the already smaller file. I only touched the tables. I shaved off a further 2.3 kB. But upon compressing it, I actually added 26 bytes. I have no definitive explanation for why, but gzip obviously couldn't take advantage of patterns that were present before. I find this a bit surprising. Especially if it would be typical, which I have difficulty believing. Well somebody needs to try it again on a real document. And it needs to be a real document dominated by tables and other XML-intensive stuff. So in this very simple example, there was hardly any gain in the compressed file despite putting in work to shave off up to 5 kB from the original. It would be very handy if the person coding file I/O could come up with some real tests on publically available documents to show how big a saving is to be had by reducing the bloat, if it's going to be compressed anyway. Have fun, Darren As I see it, the default values should be the column values for align, leftline and rightline, and the row values for valign, topline and bottomline (the latter is apparently not done, or at least no advantage taken of it IIUC). It would require some coding I suppose, but that alone would eliminate most of these attributes from the cells. Yeah I would expect that clever defaults would be a good thing, at least for readability if not the compressed file. For width and usebox, the defaults should clearly be 0pt and none. And I agree that abbreviating the attribute names and values is not a good idea after all. And the metric we should use for success is what it does to the *uncompressed* file. IMHO. I say that the metric should be a combination of readability of the uncompressed file, and size of the compressed file. I don't have a suggestion of how to quantify that =) But clearly the compressed file is pretty insensitive to the sorts of changes we are talking about so that leaves readability as the key point. Unless people want to keep uncompressed copies lying around =( Martin Have fun, Darren
Re: Feature request: gzipped LyX files
On Sun, 2002-12-22 at 04:13, Martin Vermeer wrote: > On Sat, Dec 21, 2002 at 10:51:42PM +1030, Darren Freeman spake thusly: > > ... > > > But as for eliminating symbols altogether by more intelligent use of > > defaults, I'd say: go for it! That will shave off some data. > > > > But before making the statement definitive, I had another go. I removed > > what I thought looked like redundant info, from the already smaller > > file. I only touched the tables. I shaved off a further 2.3 kB. But upon > > compressing it, I actually added 26 bytes. I have no definitive > > explanation for why, but gzip obviously couldn't take advantage of > > patterns that were present before. > > I find this a bit surprising. Especially if it would be typical, which > I have difficulty believing. Well somebody needs to try it again on a real document. And it needs to be a real document dominated by tables and other XML-intensive stuff. > > So in this very simple example, there was hardly any gain in the > > compressed file despite putting in work to shave off up to 5 kB from the > > original. It would be very handy if the person coding file I/O could > > come up with some real tests on publically available documents to show > > how big a saving is to be had by reducing the bloat, if it's going to be > > compressed anyway. > > > Have fun, > > Darren > > As I see it, the default values should be the column values for align, > leftline and rightline, and the row values for valign, topline and > bottomline (the latter is apparently not done, or at least no > advantage taken of it IIUC). It would require some coding I suppose, > but that alone would eliminate most of these attributes from the cells. Yeah I would expect that clever defaults would be a good thing, at least for readability if not the compressed file. > For width and usebox, the defaults should clearly be 0pt and none. And > I agree that abbreviating the attribute names and values is not a good > idea after all. > > And the metric we should use for success is what it does to the > *uncompressed* file. IMHO. I say that the metric should be a combination of readability of the uncompressed file, and size of the compressed file. I don't have a suggestion of how to quantify that =) But clearly the compressed file is pretty insensitive to the sorts of changes we are talking about so that leaves readability as the key point. Unless people want to keep uncompressed copies lying around =( > Martin Have fun, Darren
Re: Feature request: gzipped LyX files
Whoa, Kuba, that was a pretty interesting reply. Thanks!!! Yeah we never said it was compression, it was a sarcastic joke against M$ to say that we compress our files by saving them over and over =) On Sat, 2002-12-21 at 03:43, Kuba Ober wrote: size. So if your file won't fit on floppy, do that until it does =) This is not due to compression at all. M$ office doesn't really compress its data. In fact, it inflates it! Here's how: This is somewhat simplified, but it's sad and true. Very sad, very ture =) Cheers, Kuba Ober Darren
Re: Feature request: gzipped LyX files
On Sat, 2002-12-21 at 00:01, John Levon wrote: On Fri, Dec 20, 2002 at 09:49:56AM +0100, Andre Poenitz wrote: I'd rather make .lyx gzipped by default. Doesn't Staroffice do something similar? Lets look at the bloat first. You don't fix kernel bugs first by running 3 redundant machines ... I agree in principle, but would like to see tests to show which activity is actually more productive. Remember that shrinking the symbol names shouldn't have much of an effect on the gzip file size.. You'd have to actually start trimming away symbols to get a real benefit. As an example of what I mean, I prepared an example lyx file with two tables. There was hardly anything in this document, but its size is 11 kB. After piping through gzip it's just over 1 kB. (all files mentioned are attached in lyz form) Then I took the original and began doing find/replace to swap symbol names for abbreviations. I shaved off about 2.5 kB, with no loss of information. But gzipping it gave only a 52 byte saving. This is because gzip stores only one copy of each symbol and refers to it everywhere it is needed in the document. So shortening the symbols only shortens the single copy used by gzip. 52 bytes isn't worth the readability impact, since we want people to be able to do what they like to the file, including use it in 50 years, or grep it now. So I would say this: apart from obvious shortening of bloated symbols, leave them readable (and compatible!). As long as gzipping becomes the standard, that's a good thing since it's a tiny penalty for a large gain. But as for eliminating symbols altogether by more intelligent use of defaults, I'd say: go for it! That will shave off some data. But before making the statement definitive, I had another go. I removed what I thought looked like redundant info, from the already smaller file. I only touched the tables. I shaved off a further 2.3 kB. But upon compressing it, I actually added 26 bytes. I have no definitive explanation for why, but gzip obviously couldn't take advantage of patterns that were present before. So in this very simple example, there was hardly any gain in the compressed file despite putting in work to shave off up to 5 kB from the original. It would be very handy if the person coding file I/O could come up with some real tests on publically available documents to show how big a saving is to be had by reducing the bloat, if it's going to be compressed anyway. And maybe a comparison between gzip and bzip2? But I think bzip2 is less platform independent. I would note though that from a command line, gzipping is different from piping through gzip. Running gzip on the file has the effect of storing the original name in the file, piping doesn't. In reality people should be able to rename x.lyz into y.lyz. If they extract y.lyz they shouldn't get x.lyx. So let's *not* look at the bloat first, let's look at gzipping first =) regards john Have fun, Darren table-test-small.lyz Description: GNU Zip compressed data table-test.lyz Description: GNU Zip compressed data table-test-cropped.lyz Description: GNU Zip compressed data
Re: Feature request: gzipped LyX files
On Sat, Dec 21, 2002 at 10:51:42PM +1030, Darren Freeman wrote: I agree in principle, but would like to see tests to show which activity is actually more productive. Remember that shrinking the symbol names Why would we do that ? shouldn't have much of an effect on the gzip file size.. You'd have to actually start trimming away symbols to get a real benefit. Exactly. And with the advent of lyx2lyx the final argument against not outputting default values for tabulars is gone. Just removing the default params from a large table can easily halve the file size. Then consider that there's no reason : \end_inset /cell cell \begin_inset Text can't be : /cellcell So I would say this: apart from obvious shortening of bloated symbols, leave them readable (and compatible!). As long as gzipping becomes the standard, that's a good thing since it's a tiny penalty for a large gain. Frankly, I consider it a hack. john -- ALL television is children's television. - Richard Adler
Re: Feature request: gzipped LyX files
On Sat, Dec 21, 2002 at 10:51:42PM +1030, Darren Freeman spake thusly: ... But as for eliminating symbols altogether by more intelligent use of defaults, I'd say: go for it! That will shave off some data. But before making the statement definitive, I had another go. I removed what I thought looked like redundant info, from the already smaller file. I only touched the tables. I shaved off a further 2.3 kB. But upon compressing it, I actually added 26 bytes. I have no definitive explanation for why, but gzip obviously couldn't take advantage of patterns that were present before. I find this a bit surprising. Especially if it would be typical, which I have difficulty believing. So in this very simple example, there was hardly any gain in the compressed file despite putting in work to shave off up to 5 kB from the original. It would be very handy if the person coding file I/O could come up with some real tests on publically available documents to show how big a saving is to be had by reducing the bloat, if it's going to be compressed anyway. Have fun, Darren As I see it, the default values should be the column values for align, leftline and rightline, and the row values for valign, topline and bottomline (the latter is apparently not done, or at least no advantage taken of it IIUC). It would require some coding I suppose, but that alone would eliminate most of these attributes from the cells. For width and usebox, the defaults should clearly be 0pt and none. And I agree that abbreviating the attribute names and values is not a good idea after all. And the metric we should use for success is what it does to the *uncompressed* file. IMHO. ... Martin msg50319/pgp0.pgp Description: PGP signature
Re: Feature request: gzipped LyX files
Whoa, Kuba, that was a pretty interesting reply. Thanks!!! Yeah we never said it was compression, it was a sarcastic joke against M$ to say that we compress our files by saving them over and over =) On Sat, 2002-12-21 at 03:43, Kuba Ober wrote: > > size. So if your file won't fit on floppy, do that until it does =) > > This is not due to compression at all. M$ office doesn't really compress its > data. In fact, it inflates it! Here's how: > > This is somewhat simplified, but it's sad and true. Very sad, very ture =) > Cheers, Kuba Ober Darren
Re: Feature request: gzipped LyX files
On Sat, 2002-12-21 at 00:01, John Levon wrote: > On Fri, Dec 20, 2002 at 09:49:56AM +0100, Andre Poenitz wrote: > > > I'd rather make .lyx gzipped by default. Doesn't Staroffice do something > > similar? > > Lets look at the bloat first. You don't fix kernel bugs first by running > 3 redundant machines ... I agree in principle, but would like to see tests to show which activity is actually more productive. Remember that shrinking the symbol names shouldn't have much of an effect on the gzip file size.. You'd have to actually start trimming away symbols to get a real benefit. As an example of what I mean, I prepared an example lyx file with two tables. There was hardly anything in this document, but its size is 11 kB. After piping through gzip it's just over 1 kB. (all files mentioned are attached in lyz form) Then I took the original and began doing find/replace to swap symbol names for abbreviations. I shaved off about 2.5 kB, with no loss of information. But gzipping it gave only a 52 byte saving. This is because gzip stores only one copy of each symbol and refers to it everywhere it is needed in the document. So shortening the symbols only shortens the single copy used by gzip. 52 bytes isn't worth the readability impact, since we want people to be able to do what they like to the file, including use it in 50 years, or grep it now. So I would say this: apart from obvious shortening of bloated symbols, leave them readable (and compatible!). As long as gzipping becomes the standard, that's a good thing since it's a tiny penalty for a large gain. But as for eliminating symbols altogether by more intelligent use of defaults, I'd say: go for it! That will shave off some data. But before making the statement definitive, I had another go. I removed what I thought looked like redundant info, from the already smaller file. I only touched the tables. I shaved off a further 2.3 kB. But upon compressing it, I actually added 26 bytes. I have no definitive explanation for why, but gzip obviously couldn't take advantage of patterns that were present before. So in this very simple example, there was hardly any gain in the compressed file despite putting in work to shave off up to 5 kB from the original. It would be very handy if the person coding file I/O could come up with some real tests on publically available documents to show how big a saving is to be had by reducing the bloat, if it's going to be compressed anyway. And maybe a comparison between gzip and bzip2? But I think bzip2 is less platform independent. I would note though that from a command line, gzipping is different from piping through gzip. Running gzip on the file has the effect of storing the original name in the file, piping doesn't. In reality people should be able to rename x.lyz into y.lyz. If they extract y.lyz they shouldn't get x.lyx. So let's *not* look at the bloat first, let's look at gzipping first =) > regards > john Have fun, Darren table-test-small.lyz Description: GNU Zip compressed data table-test.lyz Description: GNU Zip compressed data table-test-cropped.lyz Description: GNU Zip compressed data
Re: Feature request: gzipped LyX files
On Sat, Dec 21, 2002 at 10:51:42PM +1030, Darren Freeman wrote: > I agree in principle, but would like to see tests to show which activity > is actually more productive. Remember that shrinking the symbol names Why would we do that ? > shouldn't have much of an effect on the gzip file size.. You'd have to > actually start trimming away symbols to get a real benefit. Exactly. And with the advent of lyx2lyx the final argument against not outputting default values for tabulars is gone. Just removing the default params from a large table can easily halve the file size. Then consider that there's no reason : \end_inset \begin_inset Text can't be : > So I would say this: apart from obvious shortening of bloated symbols, > leave them readable (and compatible!). As long as gzipping becomes the > standard, that's a good thing since it's a tiny penalty for a large > gain. Frankly, I consider it a hack. john -- "ALL television is children's television." - Richard Adler
Re: Feature request: gzipped LyX files
On Sat, Dec 21, 2002 at 10:51:42PM +1030, Darren Freeman spake thusly: ... > But as for eliminating symbols altogether by more intelligent use of > defaults, I'd say: go for it! That will shave off some data. > > But before making the statement definitive, I had another go. I removed > what I thought looked like redundant info, from the already smaller > file. I only touched the tables. I shaved off a further 2.3 kB. But upon > compressing it, I actually added 26 bytes. I have no definitive > explanation for why, but gzip obviously couldn't take advantage of > patterns that were present before. I find this a bit surprising. Especially if it would be typical, which I have difficulty believing. > So in this very simple example, there was hardly any gain in the > compressed file despite putting in work to shave off up to 5 kB from the > original. It would be very handy if the person coding file I/O could > come up with some real tests on publically available documents to show > how big a saving is to be had by reducing the bloat, if it's going to be > compressed anyway. > Have fun, > Darren As I see it, the default values should be the column values for align, leftline and rightline, and the row values for valign, topline and bottomline (the latter is apparently not done, or at least no advantage taken of it IIUC). It would require some coding I suppose, but that alone would eliminate most of these attributes from the cells. For width and usebox, the defaults should clearly be 0pt and none. And I agree that abbreviating the attribute names and values is not a good idea after all. And the metric we should use for success is what it does to the *uncompressed* file. IMHO. ... Martin msg50319/pgp0.pgp Description: PGP signature
Re: Feature request: gzipped LyX files
On Fri, Dec 20, 2002 at 06:06:29PM +1030, Darren Freeman wrote: Any objections? Comments? I'd rather make .lyx gzipped by default. Doesn't Staroffice do something similar? This is an extra burden for people like me who regularly do searchreplace in the .lyx file itself, but I'd guess that's the minority. Andre' -- Those who desire to give up Freedom in order to gain Security, will not have, nor do they deserve, either one. (T. Jefferson)
Re: Feature request: gzipped LyX files
Andre Poenitz wrote: On Fri, Dec 20, 2002 at 06:06:29PM +1030, Darren Freeman wrote: Any objections? Comments? I'd rather make .lyx gzipped by default. Doesn't Staroffice do something similar? They use the zip compression, because you can append to it unlike tar.gz. koffice uses the same approach. If you compress anyways, why not put any referred to pictures, bib files and so on into the archive? -- Moritz Moeller-Herrmann [EMAIL PROTECTED] wiss. Mitarbeiter, IMGB La loi, dans un grand souci d'égalité, interdit aux riches comme aux pauvres de coucher sous les ponts, de mendier dans les rues et de voler du pain. (ANATOLE FRANCE)
Re: Feature request: gzipped LyX files
On Fri, Dec 20, 2002 at 11:01:53AM +0100, Moritz Moeller-Herrmann wrote: I'd rather make .lyx gzipped by default. Doesn't Staroffice do something similar? They use the zip compression, because you can append to it unlike tar.gz. koffice uses the same approach. If you compress anyways, why not put any referred to pictures, bib files and so on into the archive? That would be a natural extension... Andre' -- Those who desire to give up Freedom in order to gain Security, will not have, nor do they deserve, either one. (T. Jefferson)
Re: Feature request: gzipped LyX files
Andre == Andre Poenitz [EMAIL PROTECTED] writes: Andre On Fri, Dec 20, 2002 at 06:06:29PM +1030, Darren Freeman wrote: Any objections? Comments? Andre I'd rather make .lyx gzipped by default. Doesn't Staroffice do Andre something similar? We could have both the plain .lyx format and a compressed .lyz format. Or use the plain old .lyx.gz. We could also do what emacs does: if the file is already compressed, uncompress it and compress it back on saving. If it is not compressed, leve it alone. Andre This is an extra burden for people like me who regularly do Andre searchreplace in the .lyx file itself, but I'd guess that's Andre the minority. I would not be so sure about that. JMarc
Re: Feature request: gzipped LyX files
On Fri, 2002-12-20 at 20:19, Andre Poenitz wrote: On Fri, Dec 20, 2002 at 07:30:46PM +1030, Darren Freeman wrote: I'm not sure how it works, but on my Mandrake 9.0 system I can just do something like less x.gz and it will transparently unpack it. I guess there's some kind of shell extension. Maybe bash does it automatically? Nah. 'man less'. ($LESSOPEN I think) Well it's not just less that does it. I've noticed a few command-line tools doing it. I just tried less and vi, they both do it. Maybe it's just them. But calling it .lyx.gz makes it easy for people to gunzip, edit, gzip. And making it on by default but deselectable, makes it easy for people preserving compatibility to switch it off. Too many switches are confusing. gzipping is almost uniformly better than not, so I see not much reason to bloat the UI. Fair enough, I would prefer only gzipping so if nobody yells about wanting non-gzip then go for gzip-only. For example, some retarded OSes like Windows don't know anything about gzip, so what would the Win client do? Disable the feature? zlib would be linked to the LyX executable. It just gets better and better =) I noticed that PDF uses some kind of compression, LZW I think. In fact, a group of my friends coined the term Microsoft Compression to refer to the amazing phenomenon of saving the same Powerpoint document over and over again until the file size is as small as possible. We have observed that it incrementally grows with each save, until about 3 times the smallest size, then jumping back to the smallest size. So if your file won't fit on floppy, do that until it does =) What shits me about MC is that when you exceed your disc quota at uni, Powerpoint silently fux up your document with no hints. So you could lose your work (happened to me just before a presentation once). Why does the file format need to change size when saving the same thing??? To stop competitors reversing the format? Andre' Darren
Re: Feature request: gzipped LyX files
On Fri, 2002-12-20 at 20:56, Jean-Marc Lasgouttes wrote: Andre == Andre Poenitz [EMAIL PROTECTED] writes: Andre On Fri, Dec 20, 2002 at 06:06:29PM +1030, Darren Freeman wrote: Any objections? Comments? Andre I'd rather make .lyx gzipped by default. Doesn't Staroffice do Andre something similar? We could have both the plain .lyx format and a compressed .lyz format. Yeah I like the two format idea. Make .lyz default and maybe detect the extension when saving to decide what to do. Minimum UI impact, maybe add a tooltip to remind old timers about the new feature. Or use the plain old .lyx.gz. Makes it obvious what the file is, at least. With a gzipped file, you don't even get the name of LyX in plaintext in the file. Somebody receiving a .lyz might not think to pipe it through gunzip to figure out what it is. We could also do what emacs does: if the file is already compressed, uncompress it and compress it back on saving. If it is not compressed, leve it alone. Until LyX crashes =) Leaving you with the uncompressed format which you might not remember to convert back. Also theres the issue of emergency saving. I wouldn't want emergency saves to use zlib in case they don't succede. Hence you automatically need to be able to handle uncompressed files. One other question, is are we supporting LyX on platforms too slow to be expected to use compression? On a huge document, like a book, it might take a while. But then do we say that if you write a book, use a better computer? =) (hint: yes yes yes) Andre This is an extra burden for people like me who regularly do Andre searchreplace in the .lyx file itself, but I'd guess that's Andre the minority. I would not be so sure about that. But it's not hard to decompress on the command line: gunzip x.lyx.gz ; kwrite x.lyz ; gzip x.lyx JMarc Darren
Re: Feature request: gzipped LyX files
Andre Poenitz wrote: On Fri, Dec 20, 2002 at 11:01:53AM +0100, Moritz Moeller-Herrmann wrote: I'd rather make .lyx gzipped by default. Doesn't Staroffice do something similar? They use the zip compression, because you can append to it unlike tar.gz. koffice uses the same approach. If you compress anyways, why not put any referred to pictures, bib files and so on into the archive? That would be a natural extension... I'd vote against that, at least as a default setting: There are many things (especially *huge* bibtex references) that I'm using in most of my writings. I really don't want to bloat my text files by appending that over and over again... And of course there's also the problem of file system, as currently your input files can be spread all over your disk, even on network drives. I guess the approach of storing everything with the text would imply having sort of a subdir for all the files of one doc? Pit
Re: Feature request: gzipped LyX files
On Fri, Dec 20, 2002 at 12:27:53PM +0100, Peter Suetterlin wrote: I'd vote against that, at least as a default setting: There are many things (especially *huge* bibtex references) that I'm using in most of my writings. I really don't want to bloat my text files by appending that over and over again... And of course there's also the problem of file system, as currently your input files can be spread all over your disk, even on network drives. I guess the approach of storing everything with the text would imply having sort of a subdir for all the files of one doc? Would a checkbox 'save this in the .lyx' for graphics insets be ok? Andre' -- Those who desire to give up Freedom in order to gain Security, will not have, nor do they deserve, either one. (T. Jefferson)
Re: Feature request: gzipped LyX files
Andre Poenitz wrote: On Fri, Dec 20, 2002 at 12:27:53PM +0100, Peter Suetterlin wrote: I'd vote against that, at least as a default setting: There are many things (especially *huge* bibtex references) that I'm using in most of my writings. I really don't want to bloat my text files by appending that over and over again... And of course there's also the problem of file system, as currently your input files can be spread all over your disk, even on network drives. I guess the approach of storing everything with the text would imply having sort of a subdir for all the files of one doc? Would a checkbox 'save this in the .lyx' for graphics insets be ok? Just some quick thoughts: Hmm, maybe not on an image-by-image basis. I'd rather like a special save-option 'save as complete document', where everything - including all external parts - is saved in one tar/zip/whatever archive. This would/should then also change all the reference paths in the document so that everything is located within one (sub)directory. A 'standard' save should not touch the paths within the document. Pit
Re: Feature request: gzipped LyX files
On Fri, 2002-12-20 at 21:58, Andre Poenitz wrote: On Fri, Dec 20, 2002 at 12:27:53PM +0100, Peter Suetterlin wrote: I'd vote against that, at least as a default setting: There are many things (especially *huge* bibtex references) that I'm using in most of my writings. I really don't want to bloat my text files by appending that over and over again... And of course there's also the problem of file system, as currently your input files can be spread all over your disk, even on network drives. I guess the approach of storing everything with the text would imply having sort of a subdir for all the files of one doc? Would a checkbox 'save this in the .lyx' for graphics insets be ok? Andre' Clever ;) Darren
Re: Feature request: gzipped LyX files
On Fri, 2002-12-20 at 22:41, Peter Suetterlin wrote: Andre Poenitz wrote: Hmm, maybe not on an image-by-image basis. I'd rather like a special save-option 'save as complete document', where everything - including all external parts - is saved in one tar/zip/whatever archive. This would/should then also change all the reference paths in the document so that everything is located within one (sub)directory. A 'standard' save should not touch the paths within the document. I rather like this idea since I keep a set of images that changes dynamically, but gets used all over the place. When I publish, I would like to save a snapshot of the images I used so I can recreate what I had then. Yes, it's a job for CVS. But creating a tarball of my document would be cool too. Then I could pass it on as is. We could then include a couple of scripts in the root directory of the archive that automatically exports to various file formats. Recipients of the tarball need not have any idea what LyX is to run the scripts, although perhaps requiring a working install of LyX is something that can't be done without. Maybe as an option of some kind of a self-converting archive, the document is exported as human-readable TeX and included, so that LyX is no longer required to export. E-print archives could then pick and choose from the contents of the archive to automatically create the format required. Just some thoughts, no implication of practicality =) Pit Darren
Re: Feature request: gzipped LyX files
On Fri, Dec 20, 2002 at 09:49:56AM +0100, Andre Poenitz wrote: I'd rather make .lyx gzipped by default. Doesn't Staroffice do something similar? Lets look at the bloat first. You don't fix kernel bugs first by running 3 redundant machines ... regards john -- ALL television is children's television. - Richard Adler
Re: Feature request: gzipped LyX files
In fact, a group of my friends coined the term Microsoft Compression to refer to the amazing phenomenon of saving the same Powerpoint document over and over again until the file size is as small as possible. We have observed that it incrementally grows with each save, until about 3 times the smallest size, then jumping back to the smallest size. So if your file won't fit on floppy, do that until it does =) This is not due to compression at all. M$ office doesn't really compress its data. In fact, it inflates it! Here's how: It is due to the fact that well... all non-XML m$ office documents are saved in ole compound files. OLE compound files are just another name for a supposedly-lightweight but quite s***ty filesystem which is maintained inside of a file (on linux a.k.a. mounting a filesystem image in a file through the loop). Office documents are small filesystems with all their drawbacks (and benefits where applicable ;-): filesystems usually have significant amount of slack -- unused free space. What m$ office does is that it defragments and downsizes that filesystem when the amount of slack exceeds some threshold (either fixed or as a multiple of used space). What shits me about MC is that when you exceed your disc quota at uni, Powerpoint silently fux up your document with no hints. So you could lose your work (happened to me just before a presentation once). That's becuase, like with everyday filesystems, OLE compound documents (which are treated by M$ office as filesystems) may be kept mounted. When they are kept mounted, they can also dynamically grow (when you add data to them). That's the reason why office apps: 1. lock the files when using them (an otherwise completely unnecessary thing) 2. may grow the files up to a certain limit -- the free space reclamation algorithm in the m$ OLE compound file implementation is pretty trivial and doesn't do a good job at all 3. will fux the files if they cannot grow them -- because they keep the filesystem mounted, and they just treat it as a real filesystem of unlimited size -- they don't try to verify whether it can be grown or not, and they add/change the data in the file even when you're not saving it. This is somewhat simplified, but it's sad and true. Cheers, Kuba Ober
Re: Feature request: gzipped LyX files
On Fri, Dec 20, 2002 at 12:13:44PM -0500, Kuba Ober wrote: In fact, a group of my friends coined the term Microsoft Compression to refer to the amazing phenomenon of saving the same Powerpoint document over and over again until the file size is as small as possible. We have observed that it incrementally grows with each save, until about 3 times the smallest size, then jumping back to the smallest size. So if your file won't fit on floppy, do that until it does =) What I heard is another story. When you change a M$ word/powerpoint file (a so-called compound file system which have structure inside a file), it appends the changes to the end of the file (or to a internal folder, say, changed). Only when you 'save as' the file to a new one will these changes be applied and the new file will be in its optimal size. It is said that in this way, it is easier to implement undo, faster to save etc., with only a *slight* cost of of disk space. -- Bo Peng
Re: Feature request: gzipped LyX files
On pitek 20 grudzie 2002 04:09 pm, Bo Peng wrote: On Fri, Dec 20, 2002 at 12:13:44PM -0500, Kuba Ober wrote: In fact, a group of my friends coined the term Microsoft Compression to refer to the amazing phenomenon of saving the same Powerpoint document over and over again until the file size is as small as possible. We have observed that it incrementally grows with each save, until about 3 times the smallest size, then jumping back to the smallest size. So if your file won't fit on floppy, do that until it does =) What I heard is another story. When you change a M$ word/powerpoint file (a so-called compound file system which have structure inside a file), it appends the changes to the end of the file (or to a internal folder, say, changed). Only when you 'save as' the file to a new one will these changes be applied and the new file will be in its optimal size. It is said that in this way, it is easier to implement undo, faster to save etc., with only a *slight* cost of of disk space. Exactly that happens. The changes are appended to a stream in the OLE compound file, and OLE compound files are notorious for their space wastage. Cheers, Kuba Ober
Re: Feature request: gzipped LyX files
On Fri, Dec 20, 2002 at 06:06:29PM +1030, Darren Freeman wrote: > Any objections? Comments? I'd rather make .lyx gzipped by default. Doesn't Staroffice do something similar? This is an extra burden for people like me who regularly do search in the .lyx file itself, but I'd guess that's the minority. Andre' -- Those who desire to give up Freedom in order to gain Security, will not have, nor do they deserve, either one. (T. Jefferson)
Re: Feature request: gzipped LyX files
Andre Poenitz wrote: > On Fri, Dec 20, 2002 at 06:06:29PM +1030, Darren Freeman wrote: >> Any objections? Comments? > > I'd rather make .lyx gzipped by default. Doesn't Staroffice do something > similar? They use the zip compression, because you can append to it unlike tar.gz. koffice uses the same approach. If you compress anyways, why not put any referred to pictures, bib files and so on into the archive? -- Moritz Moeller-Herrmann [EMAIL PROTECTED] wiss. Mitarbeiter, IMGB La loi, dans un grand souci d'égalité, interdit aux riches comme aux pauvres de coucher sous les ponts, de mendier dans les rues et de voler du pain. (ANATOLE FRANCE)
Re: Feature request: gzipped LyX files
On Fri, Dec 20, 2002 at 11:01:53AM +0100, Moritz Moeller-Herrmann wrote: > > I'd rather make .lyx gzipped by default. Doesn't Staroffice do something > > similar? > > They use the zip compression, because you can append to it unlike tar.gz. > koffice uses the same approach. > > If you compress anyways, why not put any referred to pictures, bib files and > so on into the archive? That would be a natural extension... Andre' -- Those who desire to give up Freedom in order to gain Security, will not have, nor do they deserve, either one. (T. Jefferson)
Re: Feature request: gzipped LyX files
> "Andre" == Andre Poenitz <[EMAIL PROTECTED]> writes: Andre> On Fri, Dec 20, 2002 at 06:06:29PM +1030, Darren Freeman wrote: >> Any objections? Comments? Andre> I'd rather make .lyx gzipped by default. Doesn't Staroffice do Andre> something similar? We could have both the plain .lyx format and a compressed .lyz format. Or use the plain old .lyx.gz. We could also do what emacs does: if the file is already compressed, uncompress it and compress it back on saving. If it is not compressed, leve it alone. Andre> This is an extra burden for people like me who regularly do Andre> search in the .lyx file itself, but I'd guess that's Andre> the minority. I would not be so sure about that. JMarc
Re: Feature request: gzipped LyX files
On Fri, 2002-12-20 at 20:19, Andre Poenitz wrote: > On Fri, Dec 20, 2002 at 07:30:46PM +1030, Darren Freeman wrote: > > I'm not sure how it works, but on my Mandrake 9.0 system I can just do > > something like less x.gz and it will transparently unpack it. I guess > > there's some kind of shell extension. Maybe bash does it automatically? > > Nah. 'man less'. ($LESSOPEN I think) Well it's not just less that does it. I've noticed a few command-line tools doing it. I just tried less and vi, they both do it. Maybe it's just them. > > But calling it .lyx.gz makes it easy for people to gunzip, edit, gzip. > > > > And making it on by default but deselectable, makes it easy for people > > preserving compatibility to switch it off. > > Too many switches are confusing. gzipping is almost uniformly better than > not, so I see not much reason to bloat the UI. Fair enough, I would prefer only gzipping so if nobody yells about wanting non-gzip then go for gzip-only. > > For example, some retarded > > OSes like Windows don't know anything about gzip, so what would the Win > > client do? Disable the feature? > > zlib would be linked to the LyX executable. It just gets better and better =) I noticed that PDF uses some kind of compression, LZW I think. In fact, a group of my friends coined the term "Microsoft Compression" to refer to the amazing phenomenon of saving the same Powerpoint document over and over again until the file size is as small as possible. We have observed that it incrementally grows with each save, until about 3 times the smallest size, then jumping back to the smallest size. So if your file won't fit on floppy, do that until it does =) What shits me about MC is that when you exceed your disc quota at uni, Powerpoint silently fux up your document with no hints. So you could lose your work (happened to me just before a presentation once). Why does the file format need to change size when saving the same thing??? To stop competitors reversing the format? > Andre' Darren
Re: Feature request: gzipped LyX files
On Fri, 2002-12-20 at 20:56, Jean-Marc Lasgouttes wrote: > > "Andre" == Andre Poenitz <[EMAIL PROTECTED]> writes: > > Andre> On Fri, Dec 20, 2002 at 06:06:29PM +1030, Darren Freeman wrote: > >> Any objections? Comments? > > Andre> I'd rather make .lyx gzipped by default. Doesn't Staroffice do > Andre> something similar? > > We could have both the plain .lyx format and a compressed .lyz format. Yeah I like the two format idea. Make .lyz default and maybe detect the extension when saving to decide what to do. Minimum UI impact, maybe add a tooltip to remind old timers about the new feature. > Or use the plain old .lyx.gz. Makes it obvious what the file is, at least. With a gzipped file, you don't even get the name of LyX in plaintext in the file. Somebody receiving a .lyz might not think to pipe it through gunzip to figure out what it is. > We could also do what emacs does: if the file is already compressed, > uncompress it and compress it back on saving. If it is not compressed, > leve it alone. Until LyX crashes =) Leaving you with the uncompressed format which you might not remember to convert back. Also theres the issue of emergency saving. I wouldn't want emergency saves to use zlib in case they don't succede. Hence you automatically need to be able to handle uncompressed files. One other question, is are we supporting LyX on platforms too slow to be expected to use compression? On a huge document, like a book, it might take a while. But then do we say that if you write a book, use a better computer? =) (hint: yes yes yes) > Andre> This is an extra burden for people like me who regularly do > Andre> search in the .lyx file itself, but I'd guess that's > Andre> the minority. > > I would not be so sure about that. But it's not hard to decompress on the command line: gunzip x.lyx.gz ; kwrite x.lyz ; gzip x.lyx > JMarc Darren
Re: Feature request: gzipped LyX files
Andre Poenitz wrote: > On Fri, Dec 20, 2002 at 11:01:53AM +0100, Moritz Moeller-Herrmann wrote: > > > I'd rather make .lyx gzipped by default. Doesn't Staroffice do something > > > similar? > > > > They use the zip compression, because you can append to it unlike tar.gz. > > koffice uses the same approach. > > > > If you compress anyways, why not put any referred to pictures, bib files and > > so on into the archive? > > That would be a natural extension... I'd vote against that, at least as a default setting: There are many things (especially *huge* bibtex references) that I'm using in most of my writings. I really don't want to bloat my text files by appending that over and over again... And of course there's also the problem of file system, as currently your input files can be spread all over your disk, even on network drives. I guess the approach of storing everything with the text would imply having sort of a subdir for all the files of one doc? Pit
Re: Feature request: gzipped LyX files
On Fri, Dec 20, 2002 at 12:27:53PM +0100, Peter Suetterlin wrote: > I'd vote against that, at least as a default setting: > There are many things (especially *huge* bibtex references) that I'm > using in most of my writings. I really don't want to bloat my text > files by appending that over and over again... > > And of course there's also the problem of file system, as currently your > input files can be spread all over your disk, even on network drives. > I guess the approach of storing everything with the text would imply > having sort of a subdir for all the files of one doc? Would a checkbox 'save this in the .lyx' for graphics insets be ok? Andre' -- Those who desire to give up Freedom in order to gain Security, will not have, nor do they deserve, either one. (T. Jefferson)
Re: Feature request: gzipped LyX files
Andre Poenitz wrote: > On Fri, Dec 20, 2002 at 12:27:53PM +0100, Peter Suetterlin wrote: > > I'd vote against that, at least as a default setting: > > There are many things (especially *huge* bibtex references) that I'm > > using in most of my writings. I really don't want to bloat my text > > files by appending that over and over again... > > > > And of course there's also the problem of file system, as currently your > > input files can be spread all over your disk, even on network drives. > > I guess the approach of storing everything with the text would imply > > having sort of a subdir for all the files of one doc? > > Would a checkbox 'save this in the .lyx' for graphics insets be ok? Just some quick thoughts: Hmm, maybe not on an image-by-image basis. I'd rather like a special save-option 'save as complete document', where everything - including all external parts - is saved in one tar/zip/whatever archive. This would/should then also change all the reference paths in the document so that everything is located within one (sub)directory. A 'standard' save should not touch the paths within the document. Pit
Re: Feature request: gzipped LyX files
On Fri, 2002-12-20 at 21:58, Andre Poenitz wrote: > On Fri, Dec 20, 2002 at 12:27:53PM +0100, Peter Suetterlin wrote: > > I'd vote against that, at least as a default setting: > > There are many things (especially *huge* bibtex references) that I'm > > using in most of my writings. I really don't want to bloat my text > > files by appending that over and over again... > > > > And of course there's also the problem of file system, as currently your > > input files can be spread all over your disk, even on network drives. > > I guess the approach of storing everything with the text would imply > > having sort of a subdir for all the files of one doc? > > Would a checkbox 'save this in the .lyx' for graphics insets be ok? > > Andre' Clever ;) Darren
Re: Feature request: gzipped LyX files
On Fri, 2002-12-20 at 22:41, Peter Suetterlin wrote: > Andre Poenitz wrote: > Hmm, maybe not on an image-by-image basis. I'd rather like a special > save-option 'save as complete document', where everything - including > all external parts - is saved in one tar/zip/whatever archive. > > This would/should then also change all the reference paths in the > document so that everything is located within one (sub)directory. > > A 'standard' save should not touch the paths within the document. I rather like this idea since I keep a set of images that changes dynamically, but gets used all over the place. When I publish, I would like to save a snapshot of the images I used so I can recreate what I had then. Yes, it's a job for CVS. But creating a tarball of my document would be cool too. Then I could pass it on as is. We could then include a couple of scripts in the root directory of the archive that automatically exports to various file formats. Recipients of the tarball need not have any idea what LyX is to run the scripts, although perhaps requiring a working install of LyX is something that can't be done without. Maybe as an option of some kind of a self-converting archive, the document is exported as human-readable TeX and included, so that LyX is no longer required to export. E-print archives could then pick and choose from the contents of the archive to automatically create the format required. Just some thoughts, no implication of practicality =) >Pit Darren
Re: Feature request: gzipped LyX files
On Fri, Dec 20, 2002 at 09:49:56AM +0100, Andre Poenitz wrote: > I'd rather make .lyx gzipped by default. Doesn't Staroffice do something > similar? Lets look at the bloat first. You don't fix kernel bugs first by running 3 redundant machines ... regards john -- "ALL television is children's television." - Richard Adler
Re: Feature request: gzipped LyX files
> In fact, a group of my friends coined the term "Microsoft Compression" > to refer to the amazing phenomenon of saving the same Powerpoint > document over and over again until the file size is as small as > possible. We have observed that it incrementally grows with each save, > until about 3 times the smallest size, then jumping back to the smallest > size. So if your file won't fit on floppy, do that until it does =) This is not due to compression at all. M$ office doesn't really compress its data. In fact, it inflates it! Here's how: It is due to the fact that well... all non-XML m$ office documents are saved in ole compound files. OLE compound files are just another name for a supposedly-lightweight but quite s***ty filesystem which is maintained inside of a file (on linux a.k.a. mounting a filesystem image in a file through the loop). Office documents are small filesystems with all their drawbacks (and benefits where applicable ;-): filesystems usually have significant amount of "slack" -- unused free space. What m$ office does is that it defragments and downsizes that "filesystem" when the amount of slack exceeds some threshold (either fixed or as a multiple of used space). > What shits me about MC is that when you exceed your disc quota at uni, > Powerpoint silently fux up your document with no hints. So you could > lose your work (happened to me just before a presentation once). That's becuase, like with everyday filesystems, OLE compound documents (which are treated by M$ office as filesystems) may be kept mounted. When they are kept mounted, they can also dynamically grow (when you add data to them). That's the reason why office apps: 1. lock the files when using them (an otherwise completely unnecessary thing) 2. may grow the files up to a certain limit -- the free space reclamation algorithm in the m$ OLE compound file implementation is pretty trivial and doesn't do a good job at all 3. will fux the files if they cannot grow them -- because they keep the "filesystem" mounted, and they just treat it as a real filesystem of unlimited size -- they don't try to verify whether it can be grown or not, and they add/change the data in the file even when you're not saving it. This is somewhat simplified, but it's sad and true. Cheers, Kuba Ober
Re: Feature request: gzipped LyX files
On Fri, Dec 20, 2002 at 12:13:44PM -0500, Kuba Ober wrote: > > In fact, a group of my friends coined the term "Microsoft Compression" > > to refer to the amazing phenomenon of saving the same Powerpoint > > document over and over again until the file size is as small as > > possible. We have observed that it incrementally grows with each save, > > until about 3 times the smallest size, then jumping back to the smallest > > size. So if your file won't fit on floppy, do that until it does =) What I heard is another story. When you change a M$ word/powerpoint file (a so-called compound file system which have structure inside a file), it appends the changes to the end of the file (or to a internal folder, say, changed). Only when you 'save as' the file to a new one will these changes be applied and the new file will be in its optimal size. It is said that in this way, it is easier to implement undo, faster to save etc., with only a *slight* cost of of disk space. -- Bo Peng
Re: Feature request: gzipped LyX files
On pitek 20 grudzie 2002 04:09 pm, Bo Peng wrote: > On Fri, Dec 20, 2002 at 12:13:44PM -0500, Kuba Ober wrote: > > > In fact, a group of my friends coined the term "Microsoft Compression" > > > to refer to the amazing phenomenon of saving the same Powerpoint > > > document over and over again until the file size is as small as > > > possible. We have observed that it incrementally grows with each save, > > > until about 3 times the smallest size, then jumping back to the > > > smallest size. So if your file won't fit on floppy, do that until it > > > does =) > > What I heard is another story. When you change a M$ word/powerpoint file > (a so-called compound file system which have structure inside a file), > it appends the changes to the end of the file (or to a internal folder, > say, changed). Only when you 'save as' the file to a new one will these > changes be applied and the new file will be in its optimal size. > > It is said that in this way, it is easier to implement undo, faster to > save etc., with only a *slight* cost of of disk space. Exactly that happens. The changes are appended to a stream in the OLE compound file, and OLE compound files are notorious for their space wastage. Cheers, Kuba Ober
