open a JIRA issue. we will use a gzip on both ends of the pipe . On
the slave side you can say
<str name="zip">true<str>
as an extra option to compress and send data from server
--Noble




On Wed, Oct 29, 2008 at 3:06 PM, Simon Collins
<[EMAIL PROTECTED]> wrote:
> I have now optimized the index - down to 325mb, it compresses down to 20mb.
>
> I think the new replication thing in solr is great, but if it could compress 
> the files it's sending, it would be an awful lot more useful when 
> replicating, as we are, between sites.
>
>
>
> --------------------------------------------------------
>
> Simon Collins
> Systems Analyst
>
> Telephone: 01904 606 867
> Fax Number: 01904 528 791
>
> shoe-shop.com ltd
> Catherine House
> Northminster Business Park
> Upper Poppleton, YORK
> YO26 6QU
> www.shoe-shop.com
> --------------------------------------------------------
>
> This message (and any associated files) is intended only for the use of the 
> individual or entity to which it is addressed and may contain information 
> that is confidential, subject to copyright or constitutes a trade secret. If 
> you are not the intended recipient you are hereby notified that any 
> dissemination, copying or distribution of this message, or files associated 
> with this message, is strictly prohibited. If you have received this message 
> in error, please notify us immediately by replying to the message and 
> deleting it from your computer. Messages sent to and from us may be monitored.
>
> Internet communications cannot be guaranteed to be secure or error-free as 
> information could be intercepted, corrupted, lost, destroyed, arrive late or 
> incomplete, or contain viruses. Therefore, we do not accept responsibility 
> for any errors or omissions that are present in this message, or any 
> attachment, that have arisen as a result of e-mail transmission. If 
> verification is required, please request a hard-copy version. Any views or 
> opinions presented are solely those of the author and do not necessarily 
> represent those of the company. (PAVD001)
> Shoe-shop.com Limited is a company registered in England and Wales with 
> company number 03817232. Vat Registration GB 734 256 241. Registered Office 
> Catherine House, Northminster Business Park, Upper Poppleton, YORK, YO26 6QU.
>
>
> -----Original Message-----
>
> From: Noble Paul നോബിള്‍ नोब्ळ् [mailto:[EMAIL PROTECTED]
> Sent: 29 October 2008 03:29
> To: solr-user@lucene.apache.org
> Subject: Re: replication handler - compression
>
> The new replication feature does not use any unix commands , it is
> pure java.  On the fly compression is hard but possible.
> I wish to repeat the question. Did you optimize the index? Because a
> 10:1 compression is not usually observed in an optimized index. Our
> own experiments showed compression of around 10:6 for optimized
> indexes.
>
> --Noble
>
> On Wed, Oct 29, 2008 at 3:41 AM, Lance Norskog <[EMAIL PROTECTED]> wrote:
>> Aha! The hint to the actual problem: "When compressed with winzip". You are 
>> running Solr on Windows.
>>
>> Snapshots don't work on Windows: they depend on a Unix file system feature. 
>> You may be copying the entire index. Not just that, it could be inconsistent.
>> This is a fine topic for a "best practices for Windows" wiki page.
>>
>> The 'scp' program what you want. It has an option to compress on the fly 
>> without saving anything to disk. 'Rcopy' in particular has features to only 
>> copy what is not already at the target.  The Putty suite 'pscp' program also 
>> has the compression feature.
>>
>> Lance
>>
>> -----Original Message-----
>> From: Noble Paul നോബിള്‍ नोब्ळ् [mailto:[EMAIL PROTECTED]
>> Sent: Monday, October 27, 2008 9:36 PM
>> To: solr-user@lucene.apache.org
>> Subject: Re: replication handler - compression
>>
>>> It is useful only if your bandwidth is very low.
>>> Otherwise the cost of copying/comprressing/decompressing can take up
>>> more time than we save.
>>
>> I mean compressing and transferring. If the optimized index itself has a 
>> very high compression ratio  then it is worth exploring the option of 
>> compresssing and transferring. And do not assume that all the files in the 
>> index directory is transferred during replication. It only transfers the 
>> files which are used by the current commit point and the ones which are 
>> absent in the slave
>>
>>
>>>
>>>
>>>
>>> On Tue, Oct 28, 2008 at 2:49 AM, Simon Collins
>>> <[EMAIL PROTECTED]> wrote:
>>>> Is there an option on the replication handler to compress the files?
>>>>
>>>>
>>>>
>>>> I'm trying to replicate off site, and seem to have accumulated about
>>>> 1.4gb. When compressed with winzip of all things i can get this down
>>>> to about 10% of the size.
>>>>
>>>>
>>>>
>>>> Is compression in the pipeline / can it be if not!
>>>>
>>>>
>>>>
>>>> simon
>>>>
>>>>
>>>>
>>>> This message has been scanned for malware by SurfControl plc.
>>>> www.surfcontrol.com
>>>>
>>>
>>>
>>>
>>> --
>>> --Noble Paul
>>>
>>
>>
>>
>> --
>> --Noble Paul
>>
>>
>
>
>
> --
> --Noble Paul
>



-- 
--Noble Paul
  • replication handler - ... Simon Collins
    • Re: replication h... Noble Paul നോബിള്‍ नोब्ळ्
      • Re: replicati... Noble Paul നോബിള്‍ नोब्ळ्
        • RE: repli... Lance Norskog
          • Query... Nguyen, Joe
            • ... Nguyen, Joe
          • Re: r... Noble Paul നോബിള്‍ नोब्ळ्
            • ... Simon Collins
              • ... christophe
              • ... Noble Paul നോബിള്‍ नोब्ळ्
                • ... Bill Au
                • ... Walter Underwood
                • ... Noble Paul നോബിള്‍ नोब्ळ्
                • ... Walter Underwood
                • ... Chris Hostetter
                • ... Noble Paul നോബിള്‍ नोब्ळ्
                • ... Chris Hostetter
                • ... Erik Hatcher
                • ... Otis Gospodnetic
                • ... Walter Underwood

Reply via email to