Re: Request for Features/Ideas: A std.archive package
On Monday, 25 May 2015 at 03:33:37 UTC, Rikki Cattermole wrote: Anyway, take a look at e.g. IWritableCapsuleArchive. There is a LOT more whitespace needed to be added in that file. Could you elaborate? Functions/etc in the interfaces are grouped by purpose (opIndexAssign/opSliceAssign are grouped because they're related operators; fileName and lastModifed because they're both bits of metadata; put and save because they're both range interface functions).
Re: Request for Features/Ideas: A std.archive package
On 25/05/2015 10:48 p.m., Liam McSherry wrote:
On Monday, 25 May 2015 at 03:33:37 UTC, Rikki Cattermole wrote:
Anyway, take a look at e.g. IWritableCapsuleArchive.
There is a LOT more whitespace needed to be added in that file.
Could you elaborate?
Functions/etc in the interfaces are grouped by purpose
(opIndexAssign/opSliceAssign are grouped because they're related
operators; fileName and lastModifed because they're both bits of
metadata; put and save because they're both range interface functions).
Basically there is no lines between one function declaration and another.
E.g.
/// description here
/// and here
/// one more
void func() { ... }
/// description here
/// and here
/// and yes one more
void func2() { ... }
Re: Request for Features/Ideas: A std.archive package
On Monday, 25 May 2015 at 11:50:25 UTC, Rikki Cattermole wrote: Basically there is no lines between one function declaration and another. I've added some extra white-space.
Re: Request for Features/Ideas: A std.archive package
On 26/05/2015 1:39 a.m., Liam McSherry wrote: On Monday, 25 May 2015 at 11:50:25 UTC, Rikki Cattermole wrote: Basically there is no lines between one function declaration and another. I've added some extra white-space. Awesome thanks, it'll make it a little easier to get it through review. There is still a few things left to fix, but no issue for now.
Re: Request for Features/Ideas: A std.archive package
A first draft of the interfaces is available here: https://github.com/McSherry/phobos/blob/std.archive/std/archive/interfaces.d Please feel free to tear to pieces, make suggestions, etc.
Re: Request for Features/Ideas: A std.archive package
On 25/05/2015 9:32 a.m., Liam McSherry wrote: A first draft of the interfaces is available here: https://github.com/McSherry/phobos/blob/std.archive/std/archive/interfaces.d Please feel free to tear to pieces, make suggestions, etc. I'm impressed an interface has been started! Anyway, take a look at e.g. IWritableCapsuleArchive. There is a LOT more whitespace needed to be added in that file.
Re: Request for Features/Ideas: A std.archive package
On Thursday, 21 May 2015 at 09:16:47 UTC, John Colvin wrote: On Thursday, 21 May 2015 at 08:55:05 UTC, Jacob Carlborg wrote: On 2015-05-21 00:46, Walter Bright wrote: Clearly, we need a Deimos entry for libarchive. https://github.com/d-programming-deimos No, it needs to be a Dub package. Can we please kill Deimos. Most (all?) deimos repositories are dub packages. Actually only few deimos bindings have dub packages. Seems like It's supposed to just copy the code to your project, which is not very nice. Also, out of topic, is there a way to propose binding to deimos? Did not find anything on that.
Re: Request for Features/Ideas: A std.archive package
On 22/05/2015 7:57 p.m., FreeSlave wrote: On Thursday, 21 May 2015 at 09:16:47 UTC, John Colvin wrote: On Thursday, 21 May 2015 at 08:55:05 UTC, Jacob Carlborg wrote: On 2015-05-21 00:46, Walter Bright wrote: Clearly, we need a Deimos entry for libarchive. https://github.com/d-programming-deimos No, it needs to be a Dub package. Can we please kill Deimos. Most (all?) deimos repositories are dub packages. Actually only few deimos bindings have dub packages. Seems like It's supposed to just copy the code to your project, which is not very nice. Deimos is pre dub. Really deimos as a concept can be stopped now. Also, out of topic, is there a way to propose binding to deimos? Did not find anything on that.
Re: Request for Features/Ideas: A std.archive package
On Wednesday, 20 May 2015 at 21:01:15 UTC, weaselcat wrote: who still uses rar? rar and zip are quite popular in the wild, 7zip is relatively rare (suffers a little publicity issue). And major advancement of rar over zip is unicode support.
Re: Request for Features/Ideas: A std.archive package
On 2015-05-21 00:46, Walter Bright wrote: Clearly, we need a Deimos entry for libarchive. https://github.com/d-programming-deimos No, it needs to be a Dub package. Can we please kill Deimos. -- /Jacob Carlborg
Re: Request for Features/Ideas: A std.archive package
On 2015-05-20 22:00, ketmar wrote: `spawnProcess(unrar, x, uselessarchive.rar);` That is the ugly workaround. Which means you need to include a tool for unraring along with your own application. -- /Jacob Carlborg
Re: Request for Features/Ideas: A std.archive package
On Thursday, 21 May 2015 at 08:55:05 UTC, Jacob Carlborg wrote: On 2015-05-21 00:46, Walter Bright wrote: Clearly, we need a Deimos entry for libarchive. https://github.com/d-programming-deimos No, it needs to be a Dub package. Can we please kill Deimos. Most (all?) deimos repositories are dub packages.
Re: Request for Features/Ideas: A std.archive package
On Thu, 21 May 2015 10:56:14 +0200, Jacob Carlborg wrote: On 2015-05-20 22:00, ketmar wrote: `spawnProcess(unrar, x, uselessarchive.rar);` That is the ugly workaround. Which means you need to include a tool for unraring along with your own application. or simply don't support rar archives. keep supporting that crap motivates people to create more rar archives, and the plague will not stop. signature.asc Description: PGP signature
Re: Request for Features/Ideas: A std.archive package
On Thu, 21 May 2015 10:55:04 +0200, Jacob Carlborg wrote: On 2015-05-21 00:46, Walter Bright wrote: Clearly, we need a Deimos entry for libarchive. https://github.com/d-programming-deimos No, it needs to be a Dub package. Can we please kill Deimos. can we please kill dub instead? signature.asc Description: PGP signature
Re: Request for Features/Ideas: A std.archive package
On Sunday, 17 May 2015 at 16:32:31 UTC, Liam McSherry wrote: Please have a look at lz4 - it's just awesome. Like ketmar mentioned, lzma is by far more relevant these days than rar.
Re: Request for Features/Ideas: A std.archive package
On Tuesday, 19 May 2015 at 19:28:03 UTC, Liam McSherry wrote: On Tuesday, 19 May 2015 at 18:46:34 UTC, Jacob Carlborg wrote: Support for RAR archives would be nice. It would be nice, but there isn't a great deal of documentation (publicly, at least). The RAR developers provide code for extracting it, but the licence forbids reverse engineering. http://www.rarlab.com/rar/unrarsrc-5.2.7.tar.gz why do you need reverse engineering?
Re: Request for Features/Ideas: A std.archive package
On Wednesday, 20 May 2015 at 08:26:11 UTC, Kagamin wrote: http://www.rarlab.com/rar/unrarsrc-5.2.7.tar.gz why do you need reverse engineering? UnRAR only extracts RAR archives, there's no facility to create them. If you wanted to create them, you would need to do some form of reverse engineering. UnRAR's licence is also fairly widely considered to be bad. *** ** UnRAR - free utility for RAR archives ** ** ** ** ** ** ~ ** *** **License for use and distribution of ** ** ** ** ** ** ~~~ ** ** ** ** ** ** FREE portable version ~ The source code of UnRAR utility is freeware. This means: 1. All copyrights to RAR and the utility UnRAR are exclusively owned by the author - Alexander Roshal. 2. The UnRAR sources may be used in any software to handle RAR archives without limitations free of charge, but cannot be used to re-create the RAR compression algorithm, which is proprietary. Distribution of modified UnRAR sources in separate form or as a part of other software is permitted, provided that it is clearly stated in the documentation and source comments that the code may not be used to develop a RAR (WinRAR) compatible archiver. 3. The UnRAR utility may be freely distributed. It is allowed to distribute UnRAR inside of other software packages. 4. THE RAR ARCHIVER AND THE UnRAR UTILITY ARE DISTRIBUTED AS IS. NO WARRANTY OF ANY KIND IS EXPRESSED OR IMPLIED. YOU USE AT YOUR OWN RISK. THE AUTHOR WILL NOT BE LIABLE FOR DATA LOSS, DAMAGES, LOSS OF PROFITS OR ANY OTHER KIND OF LOSS WHILE USING OR MISUSING THIS SOFTWARE. 5. Installing and using the UnRAR utility signifies acceptance of these terms and conditions of the license. 6. If you don't agree with terms of the license you must remove UnRAR files from your storage devices and cease to use the utility. Thank you for your interest in RAR and UnRAR. Alexander L. Roshal If UnRAR-derived code was in Phobos, everyone that used Phobos would be required to accept the above licence, or remove the RAR archiving module from their system.
Re: Request for Features/Ideas: A std.archive package
On Tue, 19 May 2015 20:46:33 +0200, Jacob Carlborg wrote: Support for RAR archives would be nice. considering unrar's license... why, 7zip is better in all ways, and it's opensource. i will really like to see a module to work with 7z archives in Phobos. rar archives will die, almost like ace archives are effectively dead now (and ACE was pretty good for it's time!). signature.asc Description: PGP signature
Re: Request for Features/Ideas: A std.archive package
On Wednesday, 20 May 2015 at 14:43:17 UTC, Kagamin wrote: UnRAR only extracts RAR archives That's what all 3rd party archivers do. Only rarlab software can create rar archives. Which is one reason for not including RAR support, even if it would be nice. I just checked the RAR 5.0 technote, and it only appears to cover the file format. No algorithms are described. So, there doesn't seem to be a way to support RAR without using UnRAR, and using UnRAR doesn't seem too likely due to its licence.
Re: Request for Features/Ideas: A std.archive package
On Wed, 20 May 2015 20:57:32 +0200, Jacob Carlborg wrote: On 2015-05-20 15:46, ketmar wrote: considering unrar's license... why, 7zip is better in all ways, and it's opensource. Doesn't matter, it's still used. BTW, I'm only interested in extracting. `spawnProcess(unrar, x, uselessarchive.rar);` signature.asc Description: PGP signature
Re: Request for Features/Ideas: A std.archive package
On Wednesday, 20 May 2015 at 18:57:32 UTC, Jacob Carlborg wrote: On 2015-05-20 15:46, ketmar wrote: considering unrar's license... why, 7zip is better in all ways, and it's opensource. Doesn't matter, it's still used. BTW, I'm only interested in extracting. who still uses rar?
Re: Request for Features/Ideas: A std.archive package
On Wednesday, 20 May 2015 at 18:55:11 UTC, Jacob Carlborg wrote: libarchive supports it, which has a BSD license. Yeah, I know that we most likely cannot use it. If RAR support is to be had, interfacing with libarchive is probably going to be the best/easiest way to do it, although it would probably be better to focus on adding support for those archives that don't need an extra library first.
Re: Request for Features/Ideas: A std.archive package
On 2015-05-20 14:19, Liam McSherry wrote: On Wednesday, 20 May 2015 at 08:26:11 UTC, Kagamin wrote: http://www.rarlab.com/rar/unrarsrc-5.2.7.tar.gz why do you need reverse engineering? UnRAR only extracts RAR archives, there's no facility to create them. If you wanted to create them, you would need to do some form of reverse engineering. UnRAR's licence is also fairly widely considered to be bad. I'm only interested in extracting RAR archives. There are many tools that can extract a lot of different archives but only create a couple of them. -- /Jacob Carlborg
Re: Request for Features/Ideas: A std.archive package
On 5/20/2015 11:56 AM, Jacob Carlborg wrote: interested in extracting RAR archives. There are many tools that can extract a lot of different archives but only create a couple of them. That's fine, as long as the licensing issue can be worked out, i.e. we can do a Boost implementation. Otherwise, Phobos won't support it, although RAR tools can be provided separately.
Re: Request for Features/Ideas: A std.archive package
On 5/20/2015 12:40 PM, Liam McSherry wrote: On Wednesday, 20 May 2015 at 18:55:11 UTC, Jacob Carlborg wrote: libarchive supports it, which has a BSD license. Yeah, I know that we most likely cannot use it. If RAR support is to be had, interfacing with libarchive is probably going to be the best/easiest way to do it, although it would probably be better to focus on adding support for those archives that don't need an extra library first. Clearly, we need a Deimos entry for libarchive. https://github.com/d-programming-deimos
Re: Request for Features/Ideas: A std.archive package
On Wednesday, 20 May 2015 at 12:19:07 UTC, Liam McSherry wrote: On Wednesday, 20 May 2015 at 08:26:11 UTC, Kagamin wrote: http://www.rarlab.com/rar/unrarsrc-5.2.7.tar.gz why do you need reverse engineering? UnRAR only extracts RAR archives That's what all 3rd party archivers do. Only rarlab software can create rar archives. If UnRAR-derived code was in Phobos, everyone that used Phobos would be required to accept the above licence, or remove the RAR archiving module from their system. I guess, the code was requested to be written. How to distribute it is a different question.
Re: Request for Features/Ideas: A std.archive package
On Tuesday, 19 May 2015 at 12:31:15 UTC, Dejan Lekic wrote: We need two things actually: 1) compress package with set of commonly used compression algorithms. std.alg.compress comes to mind as package name. A compression package would certainly be useful as the number of supported archives grows. 2) VFS module/package with set of interfaces and reference implementations for as many formats as possible. 3) It would be extremely useful to have something like Java SPI (http://en.wikipedia.org/wiki/Service_provider_interface) in D. All the VFS implementations would then follow this standard and give us truly runtime-replaceable components. More about this: http://resources.sei.cmu.edu/asset_files/TechnicalNote/2002_004_001_13958.pdf I imagine a VFS interface and an interface for file-based archives like Zip/Tar/7z would be fairly similar. Support for virtual filesystems could probably be implemented as part of the archive package. WRT runtime replaceability, the defining of a single set of exceptions (without making them too specific) for all std.archive submodules should help. The fact that there aren't any standard exceptions seems to come up a fair bit (DIP33, for example).
Re: Request for Features/Ideas: A std.archive package
On Sunday, 17 May 2015 at 16:32:31 UTC, Liam McSherry wrote: Phobos currently has packages for working with various archives (Zlib/Gzip, Zip), and it's probably reasonable to expect that support for more archive formats will be added in future. Before any more are added, it would probably be beneficial to define an interface that any modules adding support for archives should obey, and having archive related modules under std.archive for organisational purposes probably wouldn't hurt. I saw a suggestion for improved archive support on the D wiki's wish list, and the code link 'std.archive' module in the review queue now 404s, so it looks to be the case that there isn't currently an effort to implement an archiving package for Phobos. What I would propose for such a package is: - Split archives in to two categories: stream-based (Gzip, Zlib, Bzip, etc) and file-based (Zip, Tar). - Make support for creating archives optional (e.g. by having IStreamArchive and IWritableStreamArchive). - Make compression-related functionality separate (e.g. through an ICompressableArchive(T) interface, where T is an enum specifying algorithms). I'm not sure on this one, so ideas are especially welcome for it. - A hierarchy of exceptions for implementations under std.archive rather than the current Phobos norm of everything having its own exceptions. I'd like to know whether there would be demand for such a package, and what the community would want in terms of the API and features. I haven't yet written any code for the package as the API is going to be one of the most important parts, and I thought it would be best to have agreement on what is wanted before starting on it. We need two things actually: 1) compress package with set of commonly used compression algorithms. std.alg.compress comes to mind as package name. 2) VFS module/package with set of interfaces and reference implementations for as many formats as possible. 3) It would be extremely useful to have something like Java SPI (http://en.wikipedia.org/wiki/Service_provider_interface) in D. All the VFS implementations would then follow this standard and give us truly runtime-replaceable components. More about this: http://resources.sei.cmu.edu/asset_files/TechnicalNote/2002_004_001_13958.pdf
Re: Request for Features/Ideas: A std.archive package
On 2015-05-17 18:32, Liam McSherry wrote: Phobos currently has packages for working with various archives (Zlib/Gzip, Zip), and it's probably reasonable to expect that support for more archive formats will be added in future. Before any more are added, it would probably be beneficial to define an interface that any modules adding support for archives should obey, and having archive related modules under std.archive for organisational purposes probably wouldn't hurt. I saw a suggestion for improved archive support on the D wiki's wish list, and the code link 'std.archive' module in the review queue now 404s, so it looks to be the case that there isn't currently an effort to implement an archiving package for Phobos. What I would propose for such a package is: - Split archives in to two categories: stream-based (Gzip, Zlib, Bzip, etc) and file-based (Zip, Tar). - Make support for creating archives optional (e.g. by having IStreamArchive and IWritableStreamArchive). - Make compression-related functionality separate (e.g. through an ICompressableArchive(T) interface, where T is an enum specifying algorithms). I'm not sure on this one, so ideas are especially welcome for it. - A hierarchy of exceptions for implementations under std.archive rather than the current Phobos norm of everything having its own exceptions. I'd like to know whether there would be demand for such a package, and what the community would want in terms of the API and features. I haven't yet written any code for the package as the API is going to be one of the most important parts, and I thought it would be best to have agreement on what is wanted before starting on it. Support for RAR archives would be nice. -- /Jacob Carlborg
Re: Request for Features/Ideas: A std.archive package
On Tuesday, 19 May 2015 at 18:46:34 UTC, Jacob Carlborg wrote: Support for RAR archives would be nice. It would be nice, but there isn't a great deal of documentation (publicly, at least). The RAR developers provide code for extracting it, but the licence forbids reverse engineering. At that point, it's murky water and down to law. For example, the UK's Copyright, Designs, and Patents Act allows reverse-engineering (and explicitly supersedes any licences), but only under certain conditons. The act states that decompilation/reverse-engineering is not allowed if you [supply] the information obtained by the decompiling to any person to whom it is not necessary to supply it in order to achieve the permitted objective. Whether releasing D source code derived from a decompiled binary counts as [supplying] to any person to whom it is not necessary... is not something I'm qualified to determine. RARLAB have published technotes on the format, but I haven't read through them and it is possible that they don't give enough information. The technote on the RARLAB website also only seems to be for RAR 5.0, so supporting older versions could be an issue.
Re: Request for Features/Ideas: A std.archive package
On Sunday, 17 May 2015 at 17:32:10 UTC, Liam McSherry wrote: On Sunday, 17 May 2015 at 17:02:13 UTC, Andrei Alexandrescu wrote: On 5/17/15 9:32 AM, Liam McSherry wrote: I'd like to know whether there would be demand for such a package, and what the community would want in terms of the API and features. I haven't yet written any code for the package as the API is going to be one of the most important parts, and I thought it would be best to have agreement on what is wanted before starting on it. That'd be great. A few thoughts: (a) should use ranges through and through; (b) might be interesting to allow libarchive http://www.libarchive.org as engine. That's BSD licensed, but we can interface with it. Or at least use it as reference for low-level design. -- Andrei Ranges definitely. Rather than add libarchive as a dependency, it might be something to port the libarchive code to D. That would require licensing sections of std.archive under BSD rather than Boost, though. It doesn't looks like a good idea to me (to port to D but more simply to use this lib), just look to this: https://github.com/libarchive/libarchive/issues?q=is%3Aopen+is%3Aissue over 100 open bug reports ! The maintainers of the D package would have to follow this from near and it couldn't be well kept in sync. with phobos.
Re: Request for Features/Ideas: A std.archive package
On 5/17/2015 10:02 AM, Andrei Alexandrescu wrote: On 5/17/15 9:32 AM, Liam McSherry wrote: I'd like to know whether there would be demand for such a package, and what the community would want in terms of the API and features. I haven't yet written any code for the package as the API is going to be one of the most important parts, and I thought it would be best to have agreement on what is wanted before starting on it. That'd be great. A few thoughts: (a) should use ranges through and through; (b) might be interesting to allow libarchive http://www.libarchive.org as engine. That's BSD licensed, but we can interface with it. Or at least use it as reference for low-level design. -- Andrei I've thought about making such a package many times. A couple archives nobody ever thinks about are .lib and .a archives. Interestingly, they work pretty much just like .zip files, except without compression. In fact, if I was still working on linkers, I'd use .zip as the library format! Of course, like idiots, programmers keep inventing (badly) new archive formats. Oh, and lest I forget, it should not have a streaming interface. It should have a range interface.
Re: Request for Features/Ideas: A std.archive package
On Sunday, 17 May 2015 at 17:02:13 UTC, Andrei Alexandrescu wrote: On 5/17/15 9:32 AM, Liam McSherry wrote: I'd like to know whether there would be demand for such a package, and what the community would want in terms of the API and features. I haven't yet written any code for the package as the API is going to be one of the most important parts, and I thought it would be best to have agreement on what is wanted before starting on it. That'd be great. A few thoughts: (a) should use ranges through and through; (b) might be interesting to allow libarchive http://www.libarchive.org as engine. That's BSD licensed, but we can interface with it. Or at least use it as reference for low-level design. -- Andrei Ranges definitely. Rather than add libarchive as a dependency, it might be something to port the libarchive code to D. That would require licensing sections of std.archive under BSD rather than Boost, though.
Re: Request for Features/Ideas: A std.archive package
On 5/17/15 9:32 AM, Liam McSherry wrote: I'd like to know whether there would be demand for such a package, and what the community would want in terms of the API and features. I haven't yet written any code for the package as the API is going to be one of the most important parts, and I thought it would be best to have agreement on what is wanted before starting on it. That'd be great. A few thoughts: (a) should use ranges through and through; (b) might be interesting to allow libarchive http://www.libarchive.org as engine. That's BSD licensed, but we can interface with it. Or at least use it as reference for low-level design. -- Andrei
Re: Request for Features/Ideas: A std.archive package
On Sunday, 17 May 2015 at 18:01:42 UTC, wabi wrote: It doesn't looks like a good idea to me (to port to D but more simply to use this lib), just look to this: https://github.com/libarchive/libarchive/issues?q=is%3Aopen+is%3Aissue over 100 open bug reports ! The maintainers of the D package would have to follow this from near and it couldn't be well kept in sync. with phobos. I would still think that a native D implementation would be preferable to a dependency, although having some implementations in D doesn't preclude using libarchive for others.
Re: Request for Features/Ideas: A std.archive package
On Sunday, 17 May 2015 at 20:03:02 UTC, Walter Bright wrote: I've thought about making such a package many times. A couple archives nobody ever thinks about are .lib and .a archives. Interestingly, they work pretty much just like .zip files, except without compression. In fact, if I was still working on linkers, I'd use .zip as the library format! Of course, like idiots, programmers keep inventing (badly) new archive formats. I think .a just fell in to relative obscurity after .tar came about, especially since .tar can store much longer file names (100 bytes + 155 bytes of prefix/path, vs .a's 16 bytes). Having support for it in Phobos out-of-the-box would probably be useful for anyone writing a compiler/linker, though. Oh, and lest I forget, it should not have a streaming interface. It should have a range interface. If this is in response to stream-based, apologies, I mustn't have made it clear what I meant. Stream was used for want of a better word, as Zlib/Gzip/etc store data as one big stream of data, while Zip/Tar/etc structure their data in to files and the like.
