Re: GZip{In,Out}putStream in GIO?
> Does libgsf use the GIO APIs? It can take a GFile* and treat that as a zipfile. The way you would read from the zipfile member files would not be GFile* based. For the output side, turn all that around. Does that answer your question? M. ___ gtk-devel-list mailing list gtk-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
On Sat, 2009-08-15 at 10:07 -0400, Morten Welinder wrote: > > FWIW, Sugar uses zip quite extensively to bundle content and software > > and we would love to move from using python's zipfile to something > > glib-based. > > Why all this reinvent-the-wheel effort? libgsf gives you access > to zipfiles and is glib based right now. Does libgsf use the GIO APIs? Ideally, people should just be able to write their programs to expect a GInputStream. And then if some other library happens to be providing a FooRot13InputStream, it doesn't matter. I understand that libgsf predates GIO considerably. For my purposes, if I needed to deal with complicated structured files, I'd certainly look to libgsf. But all I need to do is decompress a file and read it. Right now, it took all of 231 lines of C to do it in Yelp, and most of that is boilerplate. It's just not worth an extra dependency for me. -- Shaun ___ gtk-devel-list mailing list gtk-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
On Sat, Aug 15, 2009 at 16:07, Morten Welinder wrote: >> FWIW, Sugar uses zip quite extensively to bundle content and software >> and we would love to move from using python's zipfile to something >> glib-based. > > Why all this reinvent-the-wheel effort? libgsf gives you access > to zipfiles and is glib based right now. I would prefer if this functionality was provided in glib itself because of support costs. Regards, Tomeu ___ gtk-devel-list mailing list gtk-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
> FWIW, Sugar uses zip quite extensively to bundle content and software > and we would love to move from using python's zipfile to something > glib-based. Why all this reinvent-the-wheel effort? libgsf gives you access to zipfiles and is glib based right now. Morten ___ gtk-devel-list mailing list gtk-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
On Fri, Aug 14, 2009 at 22:38, Shaun McCance wrote: >> > 2009/7/31 Brian J. Tarricone: >> >> On 07/31/2009 05:48 AM, Mikkel Kamstrup Erlandsen wrote: >> >> >> >>> From the looks of it, it should be straight forward to write >> >>> GZip{In,Out}putStream classes based on zlib >> >> I'd say call it GCompressed{In,Out}putStream and have it either >> >> auto-detect >> >> the compression type, or have a param in the API to specify from an enum >> >> (or >> >> both). People can add support for other types of compression as time goes >> >> on. > > It's nice to see I'm not the only person that's after this. > I'm not sure what other people need this for, but I thought > I'd give a data point for why it's useful to me. > > Yelp currently has its own subclass of GIOChannel that does > decompression on the fly for gzip, bzip2, and lzma. > > http://git.gnome.org/cgit/yelp/tree/src/yelp-io-channel.c > > It's not a particularly elegant solution. It has everything > in one file, with bzip2 and lzma functionality in #ifdefs, > and determines what to do by file extension. But it does the > job for Yelp, which needs to read man and info pages that may > be compressed. > > (It's also a fairly trivial and not-complete implementation. > For instance, it doesn't do seeking, because Yelp doesn't > need it.) > > I've been looking at converting Yelp over to GIO, so having > such an InputStream would be very useful to me. If it's not > in GLib, I'll have to do my own. > > I'd really need all three of the above compression schemes to > work. If a gzip-only InputStream is done, I can always ship > my own bzip2 and lzma implementations. > > A flexible system using runtime-loadable backends is fine for me, > as long as I know bzip2 and lzma implementations are available. > I don't particularly care about having a guarantee that they're > there. Distributions that ship lzma-compressed man pages can > just make sure the backend is installed. Not my problem. > > And yes, this is me volunteering to do the work, but not until > I get around to gutting Yelp, which will be in maybe two months. > I'll have to do the work anyway for Yelp. It would be nice to > get input from the maintainers about what approach they think > is best. FWIW, Sugar uses zip quite extensively to bundle content and software and we would love to move from using python's zipfile to something glib-based. Have no resources to offer, but I will tell people interested on working on bundles to consider helping out with this. Regards, Tomeu -- «Sugar Labs is anyone who participates in improving and using Sugar. What Sugar Labs does is determined by the participants.» - David Farning ___ gtk-devel-list mailing list gtk-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
> > 2009/7/31 Brian J. Tarricone: > >> On 07/31/2009 05:48 AM, Mikkel Kamstrup Erlandsen wrote: > >> > >>> From the looks of it, it should be straight forward to write > >>> GZip{In,Out}putStream classes based on zlib > >> I'd say call it GCompressed{In,Out}putStream and have it either auto-detect > >> the compression type, or have a param in the API to specify from an enum > >> (or > >> both). People can add support for other types of compression as time goes > >> on. It's nice to see I'm not the only person that's after this. I'm not sure what other people need this for, but I thought I'd give a data point for why it's useful to me. Yelp currently has its own subclass of GIOChannel that does decompression on the fly for gzip, bzip2, and lzma. http://git.gnome.org/cgit/yelp/tree/src/yelp-io-channel.c It's not a particularly elegant solution. It has everything in one file, with bzip2 and lzma functionality in #ifdefs, and determines what to do by file extension. But it does the job for Yelp, which needs to read man and info pages that may be compressed. (It's also a fairly trivial and not-complete implementation. For instance, it doesn't do seeking, because Yelp doesn't need it.) I've been looking at converting Yelp over to GIO, so having such an InputStream would be very useful to me. If it's not in GLib, I'll have to do my own. I'd really need all three of the above compression schemes to work. If a gzip-only InputStream is done, I can always ship my own bzip2 and lzma implementations. A flexible system using runtime-loadable backends is fine for me, as long as I know bzip2 and lzma implementations are available. I don't particularly care about having a guarantee that they're there. Distributions that ship lzma-compressed man pages can just make sure the backend is installed. Not my problem. And yes, this is me volunteering to do the work, but not until I get around to gutting Yelp, which will be in maybe two months. I'll have to do the work anyway for Yelp. It would be nice to get input from the maintainers about what approach they think is best. -- Shaun ___ gtk-devel-list mailing list gtk-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
From: "Mikkel Kamstrup Erlandsen", Date: 04/08/2009 00:22, Wrote: > On Mon, 2009-08-03 at 07:58 -0400, Freddie Unpenstein wrote: >> Archiving formats would be better supported by GVFS, wouldn't they...? >> Treating an archive as a virtual directory. > I'm pretty confident that in 98%[1] of the use cases the developer > needs to know that this is really about an on-disk file and not a > directory. Bear in mind that I would need to do a > g_file_enumerate_children() and open streams on nested URIs I get from > the GFileInfos returned by the GFileEnumerator. I understand this... I don't know the details, but I had a suspicion it would be a pain. However, it may be possible to abstract some of that away to return a simple list of names, or the stream corresponding to a given name...? >> However, zip has a few compression formats, rar compression can be >> applied both before and after bundling of the files (a rar "solid >> archive" is similar to a tar.gz archive). > Remember we can't build anything around RAR format due to incompatible > licensing, except of the file-roller way (spawning external commands). It was more an example than anything... I have heard there are licencing issues with RAR... It's the fact that the compression can be on either side of the archive structure that I thought worth bringing up... Same as for tar/(g|b)zip... You can quite conceivably have a tar archive of gzipped files. > libarchive can be easily used as a "filter" between two streams, > transparently decoding incoming data. This should be really implemented > as an extension, we don't want to have another dependency in glib. That sounds fair enough... A quick glance over at libarchive, and it seems to do pretty much everything required (can't tell if it handles concatenated gzipped files). But embracing it looks like duplicating a fair chunk of functionality. I wonder if it would be better to do it over in a more GLib/GIO-esque style than trying to embrace it. GIO streams can completely replace the I/O layers at both ends of the libarchive engine, while compression and at least some of the archive format handling is done by external libraries. It really doesn't look like there's much left for libarchive to do, apart from format detection, at least as far as I could tell from my quick look over the documentation. Anyhow... Wish I'd known about libarchive a while back. I really have to practice my Google technique... :( I do, though, wish there was a compression library that was able to snapshot an in-progress compression, allowing you to "rewind" to a saved state and continue from there. Fredderic Toupee Find toupees to help you look your best! Click now! http://tagline.excite.com/fc/FgElN1g05cL012R3SRh8CWFbYT1xuTXgkfM2tibgFh5Lo8J77cLU8SCTbBO/___ gtk-devel-list mailing list gtk-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
2009/8/3 Freddie Unpenstein : > From: Steve Fr-23;cinaux, Date: 03/08/2009 19:38, Wrote: > >> Bastien Nocera wrote: >> > I could think of at least 5 types of compressions that would be useful >> > to have without having to use a command-line tool to decompress: >> > - gzip for anything and everything that can come from a web server (in >> > my case, iTunes Music Store playlist parsing, or more widely, GOffice >> > file formats) >> > - zip for OpenOffice.org documents and Comic Books (evince) >> > - 7zip/LZMA/Xz formats for Comic Books >> > - Rar for the same as above >> Aren't there two classes of file types there ? Compression vs Archiving > > Archiving formats would be better supported by GVFS, wouldn't they...? > Treating an archive as a virtual directory. >From an end user perspective I can see the value in this abstraction - being able to browse into archieves, however from a developer perspective I think I would mostly find it as an annoying abstraction. I'm pretty confident that in 98%[1] of the use cases the developer needs to know that this is really about an on-disk file and not a directory. Bear in mind that I would need to do a g_file_enumerate_children() and open streams on nested URIs I get from the GFileInfos returned by the GFileEnumerator. >From a programmatic perspective I like the Java approach much better. Abstracting the ZipInputStream to general archives should be trivial (and making it more GIO-stylish is also a task for the reader): InputStream rawStream = new FileInputStream("/tmp/foo.zip"); ZipInputStream zip = new ZipInputStream(rawStream); ZipEntry entry = zip.nextEntry(); System.out.println("Got zip entry: " + entry.getPath()); InputStream nestedStream = entry.getStream(); System.out.println("Entry contents: " + nestedStream.readTheWholeBloodyStream()); // Do the zip.nextEntry() entry dance until we have read all entries -- Cheers, Mikkel [1]: And because I include arbitrary made up statistics I automatically loose this argument ;-) ___ gtk-devel-list mailing list gtk-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
> I could think of at least 5 types of compressions that would be useful > to have without having to use a command-line tool to decompress: > - gzip for anything and everything that can come from a web server (in > my case, iTunes Music Store playlist parsing, or more widely, GOffice > file formats) > - zip for OpenOffice.org documents and Comic Books (evince) > - 7zip/LZMA/Xz formats for Comic Books > - Rar for the same as above libgsf already handles zip, tar, and ole2 archives. (And gzip/bzip2 compressed streams, but reimplementing those doee not sound like much work.) Morten ___ gtk-devel-list mailing list gtk-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
On Mon, 2009-08-03 at 07:58 -0400, Freddie Unpenstein wrote: > Archiving formats would be better supported by GVFS, wouldn't they...? > Treating an archive as a virtual directory. We already have gvfsd-archive for some time, based on libarchive, able to handle TAR and ZIP archiving formats with GZIP, BZ2 compression, easily extendable to LZMA and XZ (available from libarchive, not yet enabled by default in gvfs). > However, zip has a few compression formats, rar compression can be > applied both before and after bundling of the files (a rar "solid > archive" is similar to a tar.gz archive). Remember we can't build anything around RAR format due to incompatible licensing, except of the file-roller way (spawning external commands). > I guess, though, you still need to understand the archive format, for > at least some of those formats. So I suppose some kind of archive > handler would be needed to pick out the individual streams for > concatenation. Given that, reading a zip file as a compressed stream > could instantiate a concatenation object that sets up the appropriate > archive handler and just iterates over the available files, joining > the uncompressed data of each individual decompression stream together > as a single continuous stream. Which would give you the option to set > up the archive handler yourself and extract the one(s) you want > individually. This would be a boon even for gzipped files, giving you > the option to extract concatenated compressed streams individually > rather than as a solid indistinguishable stream. libarchive can be easily used as a "filter" between two streams, transparently decoding incoming data. This should be really implemented as an extension, we don't want to have another dependency in glib. P.S.: sorry for the previous empty e-mail, key shortcuts are sometimes dangerous in Evolution... -- Tomas Bzatek ___ gtk-devel-list mailing list gtk-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
On Mon, 2009-08-03 at 07:58 -0400, Freddie Unpenstein wrote: > From: Steve Fr-23;cinaux, Date: 03/08/2009 19:38, Wrote: > > > Bastien Nocera wrote: > > > I could think of at least 5 types of compressions that would be > useful > > > to have without having to use a command-line tool to decompress: > > > - gzip for anything and everything that can come from a web server > (in > > > my case, iTunes Music Store playlist parsing, or more widely, > GOffice > > > file formats) > > > - zip for OpenOffice.org documents and Comic Books (evince) > > > - 7zip/LZMA/Xz formats for Comic Books > > > - Rar for the same as above > > Aren't there two classes of file types there ? Compression vs > Archiving > > Archiving formats would be better supported by GVFS, wouldn't they...? > Treating an archive as a virtual directory. > > However, zip has a few compression formats, rar compression can be > applied both before and after bundling of the files (a rar "solid > archive" is similar to a tar.gz archive). There may be some value to > supporting those compression formats even without supporting the > entire archive format. The compression is pretty much the hard part of > handling these archives. > > > Going the extra step further, though, the next obvious idea would be > to do the same thing that gunzip does when you concatenate together a > series of gzipped files - it simply returns the uncompressed content > of each file sequentially. > > I guess, though, you still need to understand the archive format, for > at least some of those formats. So I suppose some kind of archive > handler would be needed to pick out the individual streams for > concatenation. Given that, reading a zip file as a compressed stream > could instantiate a concatenation object that sets up the appropriate > archive handler and just iterates over the available files, joining > the uncompressed data of each individual decompression stream together > as a single continuous stream. Which would give you the option to set > up the archive handler yourself and extract the one(s) you want > individually. This would be a boon even for gzipped files, giving you > the option to extract concatenated compressed streams individually > rather than as a solid indistinguishable stream. > > The ability to manipulate archives as well as simple compression would > be fantastic. I did at one stage need to apply a "filter" to the files > matching a certain mask within a tgz archive. With support like that, > I could uncompress the tgz, split the tar, pass through files I'm not > interested in, apply the filtering to the ones I am, and then > repackage and re-compress the lot on the way out, without having to > mess around with spawning external commands and correctly handling the > myriad of things that can go wrong. > ___ gtk-devel-list mailing list gtk-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
From: Steve Fr-23;cinaux, Date: 03/08/2009 19:38, Wrote: > Bastien Nocera wrote: > > I could think of at least 5 types of compressions that would be useful > > to have without having to use a command-line tool to decompress: > > - gzip for anything and everything that can come from a web server (in > > my case, iTunes Music Store playlist parsing, or more widely, GOffice > > file formats) > > - zip for OpenOffice.org documents and Comic Books (evince) > > - 7zip/LZMA/Xz formats for Comic Books > > - Rar for the same as above > Aren't there two classes of file types there ? Compression vs Archiving Archiving formats would be better supported by GVFS, wouldn't they...? Treating an archive as a virtual directory. However, zip has a few compression formats, rar compression can be applied both before and after bundling of the files (a rar "solid archive" is similar to a tar.gz archive). There may be some value to supporting those compression formats even without supporting the entire archive format. The compression is pretty much the hard part of handling these archives. Going the extra step further, though, the next obvious idea would be to do the same thing that gunzip does when you concatenate together a series of gzipped files - it simply returns the uncompressed content of each file sequentially. I guess, though, you still need to understand the archive format, for at least some of those formats. So I suppose some kind of archive handler would be needed to pick out the individual streams for concatenation. Given that, reading a zip file as a compressed stream could instantiate a concatenation object that sets up the appropriate archive handler and just iterates over the available files, joining the uncompressed data of each individual decompression stream together as a single continuous stream. Which would give you the option to set up the archive handler yourself and extract the one(s) you want individually. This would be a boon even for gzipped files, giving you the option to extract concatenated compressed streams individually rather than as a solid indistinguishable stream. The ability to manipulate archives as well as simple compression would be fantastic. I did at one stage need to apply a "filter" to the files matching a certain mask within a tgz archive. With support like that, I could uncompress the tgz, split the tar, pass through files I'm not interested in, apply the filtering to the ones I am, and then repackage and re-compress the lot on the way out, without having to mess around with spawning external commands and correctly handling the myriad of things that can go wrong. Fredderic Worried about your move? Relocation Services can do the work for you. Click Here! Relocation Services http://tagline.excite.com/fc/FgElN1gwiSlWpkKFMLvDoO0n7NxKyoB5FLHN0tUNYVebcYZiatk3W9RdaIg/___ gtk-devel-list mailing list gtk-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
On Mon, 03 Aug 2009 11:37:53 +0200, Steve Frécinaux wrote: > I mean, zip, 7z, and rar are archiving format who store files in a > compressed fashion (kind of like a tar of gzipped files) so rather than > just having a "stream" you need to have some support for archives there, > and not just the compression part like gzip or bzip2... 7zip's LZMA algorithm is implemented in xz-utils (previously called lzma-utils) in the same way as gzip and bzip2 work. Also, 7-zip itself can be used (with some limitations) in streamed mode on *nix. -- < Jernej Simončič ><><><><>< http://eternallybored.org/ > ___ gtk-devel-list mailing list gtk-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
2009/8/3 Steve Frécinaux : > Bastien Nocera wrote: >> >> I could think of at least 5 types of compressions that would be useful >> to have without having to use a command-line tool to decompress: >> - gzip for anything and everything that can come from a web server (in >> my case, iTunes Music Store playlist parsing, or more widely, GOffice >> file formats) >> - zip for OpenOffice.org documents and Comic Books (evince) >> - 7zip/LZMA/Xz formats for Comic Books >> - Rar for the same as above > > Aren't there two classes of file types there ? Compression vs Archiving > > I mean, zip, 7z, and rar are archiving format who store files in a > compressed fashion (kind of like a tar of gzipped files) so rather than just > having a "stream" you need to have some support for archives there, and not > just the compression part like gzip or bzip2... I wonder if we should split this in two approaches, having gvfs modules for archives (compressed or not) and gio-loaders for compression only formats (gzip and bzip2). -- Un saludo, Alberto Ruiz ___ gtk-devel-list mailing list gtk-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
Bastien Nocera wrote: I could think of at least 5 types of compressions that would be useful to have without having to use a command-line tool to decompress: - gzip for anything and everything that can come from a web server (in my case, iTunes Music Store playlist parsing, or more widely, GOffice file formats) - zip for OpenOffice.org documents and Comic Books (evince) - 7zip/LZMA/Xz formats for Comic Books - Rar for the same as above Aren't there two classes of file types there ? Compression vs Archiving I mean, zip, 7z, and rar are archiving format who store files in a compressed fashion (kind of like a tar of gzipped files) so rather than just having a "stream" you need to have some support for archives there, and not just the compression part like gzip or bzip2... ___ gtk-devel-list mailing list gtk-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
Am Fri, 31 Jul 2009 15:07:10 -0700 schrieb "Brian J. Tarricone" : > On 07/31/2009 01:59 PM, Mikkel Kamstrup Erlandsen wrote: > > 2009/7/31 Brian J. Tarricone: > >> On 07/31/2009 05:48 AM, Mikkel Kamstrup Erlandsen wrote: > >> > >>> From the looks of it, it should be straight forward to write > >>> GZip{In,Out}putStream classes based on zlib > >> I'd say call it GCompressed{In,Out}putStream and have it either > >> auto-detect the compression type, or have a param in the API to > >> specify from an enum (or both). People can add support for other > >> types of compression as time goes on. > > > > The prime benefit of having dedicated classes over your approach is > > having syntactically ensured that you are indeed working with > > GZipped buffers. Personally I like that, but this is of course 100% > > subjective. > > Well, sure, but otherwise you can end up with classes for gzip, > bzip2, zip, 7zip, rar, etc. That alone is 10 extra classes, and I'm > sure there are other compression schemes people might want classes > for. That sounds messy to me. > > Having a constructor for a generic class that takes a parameter for > the compression type would give you what you want, assuming it's set > up so it returns an error if the content you pass it doesn't appear > to be of the selected type. > > I guess it doesn't really matter either way. You could even > implement GCompressedInputStream as a sort of class cluster that > returns a (private) subclass depending on the compression type, or > have all (public) classes be a subclass of a > GCompressed{In,Out}putStream class, and for all operations (except > for construction) you'd call methods on the superclass. > > And I think there's a benefit to support format auto-detection if the > developer doesn't care what format the input is in. That's possibly > a little more difficult to do with explicit subclasses, though the > class-cluster method would still work and yet maintain separate > classes for each compression format. I like the idea of a generic class and only supplying a MIME type, so that different compression implementations can go in a separate package just like GVfs backends today. It makes me think of gzip.GzipFile in Python which I like because it lets you use a compressed file just like a plain file, albeit I miss the autodetection there, so I second having that feature in GCompressedInputStream. Just my 2 cents, Christian ___ gtk-devel-list mailing list gtk-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
2009/8/1 A. Walton : > On Sat, Aug 1, 2009 at 1:54 PM, Mikkel Kamstrup > Erlandsen wrote: >> 2009/8/1 Bastien Nocera : >>> On Fri, 2009-07-31 at 21:17 +0100, Alberto Ruiz wrote: 2009/7/31 Brian J. Tarricone : > On 07/31/2009 05:48 AM, Mikkel Kamstrup Erlandsen wrote: > >> From the looks of it, it should be straight forward to write >> GZip{In,Out}putStream classes based on zlib > > I'd say call it GCompressed{In,Out}putStream and have it either > auto-detect > the compression type, or have a param in the API to specify from an enum > (or > both). People can add support for other types of compression as time > goes > on. And expand the list of glib dependencies to the infinite? I don't think I like such an idea at this level of the API. >>> >>> You could claim the same about the gdk-pixbuf loaders, or the gvfs >>> backends. The point is that you'd want to make your original class >>> flexible enough that people could extend upon it (even if you don't >>> start off with a plugin API). >>> >>> I could think of at least 5 types of compressions that would be useful >>> to have without having to use a command-line tool to decompress: >>> - gzip for anything and everything that can come from a web server (in >>> my case, iTunes Music Store playlist parsing, or more widely, GOffice >>> file formats) >>> - zip for OpenOffice.org documents and Comic Books (evince) >>> - 7zip/LZMA/Xz formats for Comic Books >>> - Rar for the same as above >>> >>> I'd be happy seeing at least the first one, very happy to see the first >>> 3, that all have open source implementations, and would provide more >>> robust support for those file types in applications. >> >> Ok, taking Brian's ideas into consideration, how about this: >> GCompressedInputStream (extrapolate to output streams yourself) is an >> abstract base class. It has static factories: >> >> GCompressedInputStream* >> g_compressed_inputstream_new_for_type (const gchar* type_name); >> >> GCompressedInputStream* >> g_compressed_inputstream_new_for_stream (GInputStream* in); > > More like > > GInputStream * > g_compressed_input_stream_new (GInputStream *in); > > g_compressed_input_stream_new_for_type(GInputStream *in, const gchar > *type) Right. I was a bit fast to hit send there :-) And the Compression*Stream would extend the Filter*Stream base classes. > SNIP > ... interface which G(G,B)Zip(I,O)Stream implements with the actual > implementations hidden like GLocalFile; what public methods would > those have anyways? (Not rhetorical, curious). Compression level for GZip comes to mind. There are probably other parameters. I think that the static factory approach hiding the actual impl. is good, but I would also like to expose a GZip*Stream directly. > Also, Zip is probably not a good idea since you'd have to deal with > virtual files and crypto. Makes better sense as a GVFS backend I > think. Yeah, archive files require another approach. When we have finished the pure compression formats we might also have a better idea whether or not it makes sense to create a similar framework for the archive formats. -- Cheers, Mikkel ___ gtk-devel-list mailing list gtk-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
On Sat, Aug 1, 2009 at 1:54 PM, Mikkel Kamstrup Erlandsen wrote: > 2009/8/1 Bastien Nocera : >> On Fri, 2009-07-31 at 21:17 +0100, Alberto Ruiz wrote: >>> 2009/7/31 Brian J. Tarricone : >>> > On 07/31/2009 05:48 AM, Mikkel Kamstrup Erlandsen wrote: >>> > >>> >> From the looks of it, it should be straight forward to write >>> >> GZip{In,Out}putStream classes based on zlib >>> > >>> > I'd say call it GCompressed{In,Out}putStream and have it either >>> > auto-detect >>> > the compression type, or have a param in the API to specify from an enum >>> > (or >>> > both). People can add support for other types of compression as time goes >>> > on. >>> >>> And expand the list of glib dependencies to the infinite? >>> I don't think I like such an idea at this level of the API. >> >> You could claim the same about the gdk-pixbuf loaders, or the gvfs >> backends. The point is that you'd want to make your original class >> flexible enough that people could extend upon it (even if you don't >> start off with a plugin API). >> >> I could think of at least 5 types of compressions that would be useful >> to have without having to use a command-line tool to decompress: >> - gzip for anything and everything that can come from a web server (in >> my case, iTunes Music Store playlist parsing, or more widely, GOffice >> file formats) >> - zip for OpenOffice.org documents and Comic Books (evince) >> - 7zip/LZMA/Xz formats for Comic Books >> - Rar for the same as above >> >> I'd be happy seeing at least the first one, very happy to see the first >> 3, that all have open source implementations, and would provide more >> robust support for those file types in applications. > > Ok, taking Brian's ideas into consideration, how about this: > GCompressedInputStream (extrapolate to output streams yourself) is an > abstract base class. It has static factories: > > GCompressedInputStream* > g_compressed_inputstream_new_for_type (const gchar* type_name); > > GCompressedInputStream* > g_compressed_inputstream_new_for_stream (GInputStream* in); More like GInputStream * g_compressed_input_stream_new (GInputStream *in); g_compressed_input_stream_new_for_type(GInputStream *in, const gchar *type) isn't a bad idea either, since you could specify the type at construction rather than force mime auto-detection which could be a pain (thinking network streams). Probably should do better than "_new" if we go the interface route. This is how all of the other filter streams are specified in GIO, IMO it's pointless to diverge much here. Also wonder if it makes sense as a base class or whether it should be an interface which G(G,B)Zip(I,O)Stream implements with the actual implementations hidden like GLocalFile; what public methods would those have anyways? (Not rhetorical, curious). Also, Zip is probably not a good idea since you'd have to deal with virtual files and crypto. Makes better sense as a GVFS backend I think. -A. Walton > > We have concrete classes for the most used compressions publicly > available, eg. GZip, Zip[1] (and BZip2?). Other compressions can be > registered via GIO extension points (one for compressors and one for > decompressors). > > -- > Cheers, > Mikkel > > [1]: Modulo the fact that Zip files can't be parsed as simple streams. > Maybe a GArchive*putStream base class is needed. > ___ > gtk-devel-list mailing list > gtk-devel-list@gnome.org > http://mail.gnome.org/mailman/listinfo/gtk-devel-list ___ gtk-devel-list mailing list gtk-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
2009/8/1 Bastien Nocera : > On Fri, 2009-07-31 at 21:17 +0100, Alberto Ruiz wrote: >> 2009/7/31 Brian J. Tarricone : >> > On 07/31/2009 05:48 AM, Mikkel Kamstrup Erlandsen wrote: >> > >> >> From the looks of it, it should be straight forward to write >> >> GZip{In,Out}putStream classes based on zlib >> > >> > I'd say call it GCompressed{In,Out}putStream and have it either auto-detect >> > the compression type, or have a param in the API to specify from an enum >> > (or >> > both). People can add support for other types of compression as time goes >> > on. >> >> And expand the list of glib dependencies to the infinite? >> I don't think I like such an idea at this level of the API. > > You could claim the same about the gdk-pixbuf loaders, or the gvfs > backends. The point is that you'd want to make your original class > flexible enough that people could extend upon it (even if you don't > start off with a plugin API). > > I could think of at least 5 types of compressions that would be useful > to have without having to use a command-line tool to decompress: > - gzip for anything and everything that can come from a web server (in > my case, iTunes Music Store playlist parsing, or more widely, GOffice > file formats) > - zip for OpenOffice.org documents and Comic Books (evince) > - 7zip/LZMA/Xz formats for Comic Books > - Rar for the same as above > > I'd be happy seeing at least the first one, very happy to see the first > 3, that all have open source implementations, and would provide more > robust support for those file types in applications. Ok, taking Brian's ideas into consideration, how about this: GCompressedInputStream (extrapolate to output streams yourself) is an abstract base class. It has static factories: GCompressedInputStream* g_compressed_inputstream_new_for_type (const gchar* type_name); GCompressedInputStream* g_compressed_inputstream_new_for_stream (GInputStream* in); We have concrete classes for the most used compressions publicly available, eg. GZip, Zip[1] (and BZip2?). Other compressions can be registered via GIO extension points (one for compressors and one for decompressors). -- Cheers, Mikkel [1]: Modulo the fact that Zip files can't be parsed as simple streams. Maybe a GArchive*putStream base class is needed. ___ gtk-devel-list mailing list gtk-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
On 07/31/2009 01:17 PM, Alberto Ruiz wrote: 2009/7/31 Brian J. Tarricone: On 07/31/2009 05:48 AM, Mikkel Kamstrup Erlandsen wrote: From the looks of it, it should be straight forward to write GZip{In,Out}putStream classes based on zlib I'd say call it GCompressed{In,Out}putStream and have it either auto-detect the compression type, or have a param in the API to specify from an enum (or both). People can add support for other types of compression as time goes on. And expand the list of glib dependencies to the infinite? I don't think I like such an idea at this level of the API. Eh, they can be optional, or dlopen()ed at runtime. The API could be flexible enough (using the existing 'extension points' architecture) to allow people to add their own implementations for their own use, or some interested party could bundle a set of them in another package... though in that case the entire compressed stream class could be in a separate library. Instead of using an enum to select the compression type, a string identifier could be used instead (perhaps the mime type). IMO optional deps aren't really a big deal. If people thought it'd be useful to have one or two mandatory deps (say zlib and bzip2, for example), that might be ok too, since they're pretty ubiquitous. -brian ___ gtk-devel-list mailing list gtk-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
On Fri, 2009-07-31 at 21:17 +0100, Alberto Ruiz wrote: > 2009/7/31 Brian J. Tarricone : > > On 07/31/2009 05:48 AM, Mikkel Kamstrup Erlandsen wrote: > > > >> From the looks of it, it should be straight forward to write > >> GZip{In,Out}putStream classes based on zlib > > > > I'd say call it GCompressed{In,Out}putStream and have it either auto-detect > > the compression type, or have a param in the API to specify from an enum (or > > both). People can add support for other types of compression as time goes > > on. > > And expand the list of glib dependencies to the infinite? > I don't think I like such an idea at this level of the API. You could claim the same about the gdk-pixbuf loaders, or the gvfs backends. The point is that you'd want to make your original class flexible enough that people could extend upon it (even if you don't start off with a plugin API). I could think of at least 5 types of compressions that would be useful to have without having to use a command-line tool to decompress: - gzip for anything and everything that can come from a web server (in my case, iTunes Music Store playlist parsing, or more widely, GOffice file formats) - zip for OpenOffice.org documents and Comic Books (evince) - 7zip/LZMA/Xz formats for Comic Books - Rar for the same as above I'd be happy seeing at least the first one, very happy to see the first 3, that all have open source implementations, and would provide more robust support for those file types in applications. ___ gtk-devel-list mailing list gtk-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
On 07/31/2009 01:59 PM, Mikkel Kamstrup Erlandsen wrote: 2009/7/31 Brian J. Tarricone: On 07/31/2009 05:48 AM, Mikkel Kamstrup Erlandsen wrote: From the looks of it, it should be straight forward to write GZip{In,Out}putStream classes based on zlib I'd say call it GCompressed{In,Out}putStream and have it either auto-detect the compression type, or have a param in the API to specify from an enum (or both). People can add support for other types of compression as time goes on. The prime benefit of having dedicated classes over your approach is having syntactically ensured that you are indeed working with GZipped buffers. Personally I like that, but this is of course 100% subjective. Well, sure, but otherwise you can end up with classes for gzip, bzip2, zip, 7zip, rar, etc. That alone is 10 extra classes, and I'm sure there are other compression schemes people might want classes for. That sounds messy to me. Having a constructor for a generic class that takes a parameter for the compression type would give you what you want, assuming it's set up so it returns an error if the content you pass it doesn't appear to be of the selected type. I guess it doesn't really matter either way. You could even implement GCompressedInputStream as a sort of class cluster that returns a (private) subclass depending on the compression type, or have all (public) classes be a subclass of a GCompressed{In,Out}putStream class, and for all operations (except for construction) you'd call methods on the superclass. And I think there's a benefit to support format auto-detection if the developer doesn't care what format the input is in. That's possibly a little more difficult to do with explicit subclasses, though the class-cluster method would still work and yet maintain separate classes for each compression format. -brian ___ gtk-devel-list mailing list gtk-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
2009/7/31 Brian J. Tarricone : > On 07/31/2009 05:48 AM, Mikkel Kamstrup Erlandsen wrote: > >> From the looks of it, it should be straight forward to write >> GZip{In,Out}putStream classes based on zlib > > I'd say call it GCompressed{In,Out}putStream and have it either auto-detect > the compression type, or have a param in the API to specify from an enum (or > both). People can add support for other types of compression as time goes > on. The prime benefit of having dedicated classes over your approach is having syntactically ensured that you are indeed working with GZipped buffers. Personally I like that, but this is of course 100% subjective. -- Cheers, Mikkel ___ gtk-devel-list mailing list gtk-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
2009/7/31 Brian J. Tarricone : > On 07/31/2009 05:48 AM, Mikkel Kamstrup Erlandsen wrote: > >> From the looks of it, it should be straight forward to write >> GZip{In,Out}putStream classes based on zlib > > I'd say call it GCompressed{In,Out}putStream and have it either auto-detect > the compression type, or have a param in the API to specify from an enum (or > both). People can add support for other types of compression as time goes > on. And expand the list of glib dependencies to the infinite? I don't think I like such an idea at this level of the API. > -brian > ___ > gtk-devel-list mailing list > gtk-devel-list@gnome.org > http://mail.gnome.org/mailman/listinfo/gtk-devel-list > -- Un saludo, Alberto Ruiz ___ gtk-devel-list mailing list gtk-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
On 07/31/2009 05:48 AM, Mikkel Kamstrup Erlandsen wrote: From the looks of it, it should be straight forward to write GZip{In,Out}putStream classes based on zlib I'd say call it GCompressed{In,Out}putStream and have it either auto-detect the compression type, or have a param in the API to specify from an enum (or both). People can add support for other types of compression as time goes on. -brian ___ gtk-devel-list mailing list gtk-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
2009/7/31 Mikkel Kamstrup Erlandsen : > 2009/7/31 Jody Goldberg : >> On Fri, Jul 31, 2009 at 02:48:47PM +0200, Mikkel Kamstrup Erlandsen wrote: >>> Hi, >>> >>> I've been eye balling the GIO docs for a while without finding >>> in-/output for gzip compression... So if I missed it stop me now :-) >>> >>> From the looks of it, it should be straight forward to write >>> GZip{In,Out}putStream classes based on zlib[1]. If I write these >>> classes would a zlib dep. be OK for GIO? It is pretty ubiquitous as >>> far as I can tell (and it is already a depency of Gtk+)... And I am >>> not really all that keen on writing my own gzip/DEFLATE implementation >>> (to say the least). >>> >>> Another thing is the naming. Since this is not about supporting the >>> Zip archive format (but that would be nice too) the name GZip*Stream >>> might be bad. On the other hand i abhor the ALLCAPS GZIPInputStream >>> used in fx. Java. What about GGZip*Stream - eeek!? >>> >> >> libgsf has all these, along with a gio wrapper. > > I assume you mean: > http://projects.gnome.org/libgsf/gsf-Compression.html? I could not > find the GIO wrapper in the documentation (assuming that it is > unreleased yet?). Anyways - my question is really if this > functionality should not be in GLib/GIO? I sort of like the proposal. Cairo and libpng depends on zlib and there's no Win32 API to replace it so the implementation would be fully cross platform it's very basic functionality IMHO and having it in a GIO wrapper would be awesome. I'd love to see BZip2/7zip/Tar/rar/... wrappers somewhere else in the platform though (and kill most of the file-roller codebase for that matter). Only downside from my point of view is that in Windows this would mean yet another .dll to distribute. > Adding a native GIO impl. would of course be looking heavily towards > what already exists in Gsf. > > -- > Cheers, > Mikkel > ___ > gtk-devel-list mailing list > gtk-devel-list@gnome.org > http://mail.gnome.org/mailman/listinfo/gtk-devel-list > -- Un saludo, Alberto Ruiz ___ gtk-devel-list mailing list gtk-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
2009/7/31 Jody Goldberg : > On Fri, Jul 31, 2009 at 02:48:47PM +0200, Mikkel Kamstrup Erlandsen wrote: >> Hi, >> >> I've been eye balling the GIO docs for a while without finding >> in-/output for gzip compression... So if I missed it stop me now :-) >> >> From the looks of it, it should be straight forward to write >> GZip{In,Out}putStream classes based on zlib[1]. If I write these >> classes would a zlib dep. be OK for GIO? It is pretty ubiquitous as >> far as I can tell (and it is already a depency of Gtk+)... And I am >> not really all that keen on writing my own gzip/DEFLATE implementation >> (to say the least). >> >> Another thing is the naming. Since this is not about supporting the >> Zip archive format (but that would be nice too) the name GZip*Stream >> might be bad. On the other hand i abhor the ALLCAPS GZIPInputStream >> used in fx. Java. What about GGZip*Stream - eeek!? >> > > libgsf has all these, along with a gio wrapper. I assume you mean: http://projects.gnome.org/libgsf/gsf-Compression.html? I could not find the GIO wrapper in the documentation (assuming that it is unreleased yet?). Anyways - my question is really if this functionality should not be in GLib/GIO? Adding a native GIO impl. would of course be looking heavily towards what already exists in Gsf. -- Cheers, Mikkel ___ gtk-devel-list mailing list gtk-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
On Fri, Jul 31, 2009 at 02:48:47PM +0200, Mikkel Kamstrup Erlandsen wrote: > Hi, > > I've been eye balling the GIO docs for a while without finding > in-/output for gzip compression... So if I missed it stop me now :-) > > From the looks of it, it should be straight forward to write > GZip{In,Out}putStream classes based on zlib[1]. If I write these > classes would a zlib dep. be OK for GIO? It is pretty ubiquitous as > far as I can tell (and it is already a depency of Gtk+)... And I am > not really all that keen on writing my own gzip/DEFLATE implementation > (to say the least). > > Another thing is the naming. Since this is not about supporting the > Zip archive format (but that would be nice too) the name GZip*Stream > might be bad. On the other hand i abhor the ALLCAPS GZIPInputStream > used in fx. Java. What about GGZip*Stream - eeek!? > libgsf has all these, along with a gio wrapper. ___ gtk-devel-list mailing list gtk-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/gtk-devel-list