Hello Lhassan, i am afraid this still produces uncompressed files:
output.csv: UTF-8 Unicode English text, with very long lines, with CRLF, LF line terminators Just to clarify: I'd like to get scrapyd to write its output to its `items_dir = /path/to/items/` and compress it to save disk space. Quite the same as you'd do with `scrapy`(not `scrapyd`) using "-" for `--output` and piping the output through gzip for examle (or bzip2 or...) -- Best regards, Stefan Am Dienstag, 15. Juli 2014 12:06:21 UTC+2 schrieb Lhassan Baazzi: > > Hi, > > It's already supported, `scrapy crawl my_spider -o output.csv -t csv` > > > Regards. > --------- > Lhassan Baazzi | Web Developer PHP - Symfony - JS - Scrapy > Email/Gtalk: [email protected] <javascript:> - Skype: baazzilhassan - > Twitter: @baazzilhassan <http://twitter.com/baazzilhassan> > Blog: http://blog.jbinfo.io/ > Donate - PayPal - > <https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=BR744DG33RAGN> > > > 2014-07-15 9:28 GMT+00:00 Suchkultur <[email protected] <javascript:>>: > >> I would like if it is good (or maybe bad) idea to give scrapy a feature >> which allows writing items in compressed json, xml, csv and other >> supported >> plaintext output-formats to the local disk. >> >> I know that this could be solved by using "-" to write to stdout and >> compress >> it on the fly on unix-like systems. But i don't know if this is feasible >> using scrapyd >> on a remote system - or is it? >> >> That's why i'd suppose it may be a good extension which would belong to >> the >> settings.py file of the scrapy project. Probably as an optional >> middleware(?) >> >> The background is: We use scrapy to write on disk without any further >> processing >> and batch-process the output afterwards using an ETL toolchain which >> cleans up the >> raw crawling results after processing. >> >> We'd like to reduce the disk space which is used inbetween these two >> tasks. >> >> >> -- >> Best regards, >> Stefan Antoni >> >> -- >> You received this message because you are subscribed to the Google Groups >> "scrapy-users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To post to this group, send email to [email protected] >> <javascript:>. >> Visit this group at http://groups.google.com/group/scrapy-users. >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
