open a JIRA issue. we will use a gzip on both ends of the pipe . On the slave side you can say <str name="zip">true<str> as an extra option to compress and send data from server --Noble
On Wed, Oct 29, 2008 at 3:06 PM, Simon Collins <[EMAIL PROTECTED]> wrote: > I have now optimized the index - down to 325mb, it compresses down to 20mb. > > I think the new replication thing in solr is great, but if it could compress > the files it's sending, it would be an awful lot more useful when > replicating, as we are, between sites. > > > > -------------------------------------------------------- > > Simon Collins > Systems Analyst > > Telephone: 01904 606 867 > Fax Number: 01904 528 791 > > shoe-shop.com ltd > Catherine House > Northminster Business Park > Upper Poppleton, YORK > YO26 6QU > www.shoe-shop.com > -------------------------------------------------------- > > This message (and any associated files) is intended only for the use of the > individual or entity to which it is addressed and may contain information > that is confidential, subject to copyright or constitutes a trade secret. If > you are not the intended recipient you are hereby notified that any > dissemination, copying or distribution of this message, or files associated > with this message, is strictly prohibited. If you have received this message > in error, please notify us immediately by replying to the message and > deleting it from your computer. Messages sent to and from us may be monitored. > > Internet communications cannot be guaranteed to be secure or error-free as > information could be intercepted, corrupted, lost, destroyed, arrive late or > incomplete, or contain viruses. Therefore, we do not accept responsibility > for any errors or omissions that are present in this message, or any > attachment, that have arisen as a result of e-mail transmission. If > verification is required, please request a hard-copy version. Any views or > opinions presented are solely those of the author and do not necessarily > represent those of the company. (PAVD001) > Shoe-shop.com Limited is a company registered in England and Wales with > company number 03817232. Vat Registration GB 734 256 241. Registered Office > Catherine House, Northminster Business Park, Upper Poppleton, YORK, YO26 6QU. > > > -----Original Message----- > > From: Noble Paul നോബിള് नोब्ळ् [mailto:[EMAIL PROTECTED] > Sent: 29 October 2008 03:29 > To: solr-user@lucene.apache.org > Subject: Re: replication handler - compression > > The new replication feature does not use any unix commands , it is > pure java. On the fly compression is hard but possible. > I wish to repeat the question. Did you optimize the index? Because a > 10:1 compression is not usually observed in an optimized index. Our > own experiments showed compression of around 10:6 for optimized > indexes. > > --Noble > > On Wed, Oct 29, 2008 at 3:41 AM, Lance Norskog <[EMAIL PROTECTED]> wrote: >> Aha! The hint to the actual problem: "When compressed with winzip". You are >> running Solr on Windows. >> >> Snapshots don't work on Windows: they depend on a Unix file system feature. >> You may be copying the entire index. Not just that, it could be inconsistent. >> This is a fine topic for a "best practices for Windows" wiki page. >> >> The 'scp' program what you want. It has an option to compress on the fly >> without saving anything to disk. 'Rcopy' in particular has features to only >> copy what is not already at the target. The Putty suite 'pscp' program also >> has the compression feature. >> >> Lance >> >> -----Original Message----- >> From: Noble Paul നോബിള് नोब्ळ् [mailto:[EMAIL PROTECTED] >> Sent: Monday, October 27, 2008 9:36 PM >> To: solr-user@lucene.apache.org >> Subject: Re: replication handler - compression >> >>> It is useful only if your bandwidth is very low. >>> Otherwise the cost of copying/comprressing/decompressing can take up >>> more time than we save. >> >> I mean compressing and transferring. If the optimized index itself has a >> very high compression ratio then it is worth exploring the option of >> compresssing and transferring. And do not assume that all the files in the >> index directory is transferred during replication. It only transfers the >> files which are used by the current commit point and the ones which are >> absent in the slave >> >> >>> >>> >>> >>> On Tue, Oct 28, 2008 at 2:49 AM, Simon Collins >>> <[EMAIL PROTECTED]> wrote: >>>> Is there an option on the replication handler to compress the files? >>>> >>>> >>>> >>>> I'm trying to replicate off site, and seem to have accumulated about >>>> 1.4gb. When compressed with winzip of all things i can get this down >>>> to about 10% of the size. >>>> >>>> >>>> >>>> Is compression in the pipeline / can it be if not! >>>> >>>> >>>> >>>> simon >>>> >>>> >>>> >>>> This message has been scanned for malware by SurfControl plc. >>>> www.surfcontrol.com >>>> >>> >>> >>> >>> -- >>> --Noble Paul >>> >> >> >> >> -- >> --Noble Paul >> >> > > > > -- > --Noble Paul > -- --Noble Paul
