> files come in all kinds, and zipping them may not be the best option > depending on the use-case (for already compressed ones for example). > Maybe you are referring to text files? In that case, zipping them would > save quite a bit of disk space indeed. >
I am only interested in text, yes. In my case this approach results in a mean compression ratio of ~16. I would second Lhassan here, on using a subclassed FilesPipeline, with a > custom STORE_SCHEME referencing your store. > Have you tried this? > You are right this is indeed possible! This had not occurred to me. Or are you having trouble getting this to work? Maybe the documentation is > a bit daunting? in which case we can improve it. > An example on how to write a custom store would have been really helpful. FYI, and all you googlers out there, the zip-backed file store is available at https://github.com/kaspermarstal/fszipfilesstore. Thank you for your help! -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
