[jira] Updated: (LUCENE-1960) Remove deprecated Field.Store.COMPRESS
[ https://issues.apache.org/jira/browse/LUCENE-1960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uwe Schindler updated LUCENE-1960: -- Attachment: index.29.nocfs.zip index.29.cfs.zip lucene-1960-1.patch Modified patch with testcase for backwards compatibility. It was a little bit trick to check if a field was actually compressed in the source index, but it worked with FieldSelectorResult.SIZE and a test-compression/chars*2. The test was done before/after optimize and it is verified that before the field was still compressed (even it is larger in reality, harhar) and uncompressed after optimize. Should I commit the index creation to the current 2.9 branch or not? I can commit this, if everybody is happy (because I have the zip files already in my checkout added). Or will you do it, Michael? Remove deprecated Field.Store.COMPRESS -- Key: LUCENE-1960 URL: https://issues.apache.org/jira/browse/LUCENE-1960 Project: Lucene - Java Issue Type: Task Reporter: Michael Busch Assignee: Michael Busch Priority: Minor Fix For: 3.0 Attachments: index.29.cfs.zip, index.29.nocfs.zip, lucene-1960-1.patch, lucene-1960-1.patch, lucene-1960-1.patch, lucene-1960.patch, optimize-time.txt Also remove FieldForMerge and related code. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Updated: (LUCENE-1960) Remove deprecated Field.Store.COMPRESS
[ https://issues.apache.org/jira/browse/LUCENE-1960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uwe Schindler updated LUCENE-1960: -- Attachment: lucene-1960-1-branch29.patch lucene-1960-1.patch Remove deprecated Field.Store.COMPRESS -- Key: LUCENE-1960 URL: https://issues.apache.org/jira/browse/LUCENE-1960 Project: Lucene - Java Issue Type: Task Reporter: Michael Busch Assignee: Michael Busch Priority: Minor Fix For: 3.0 Attachments: index.29.cfs.zip, index.29.nocfs.zip, lucene-1960-1-branch29.patch, lucene-1960-1.patch, lucene-1960-1.patch, lucene-1960-1.patch, lucene-1960-1.patch, lucene-1960.patch, optimize-time.txt Also remove FieldForMerge and related code. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Updated: (LUCENE-1960) Remove deprecated Field.Store.COMPRESS
[ https://issues.apache.org/jira/browse/LUCENE-1960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uwe Schindler updated LUCENE-1960: -- Attachment: index.29.nocfs.zip index.29.cfs.zip Here updated patch and ZIP index files. Now also binary fields are tested in the same way. Even document ids get a string compressed field, odd document ids a binary one. Also attached is the patch for the 2.9 branch (not BW branch!!!). In trunk, the index creation is commented out, it's just there for reference. Remove deprecated Field.Store.COMPRESS -- Key: LUCENE-1960 URL: https://issues.apache.org/jira/browse/LUCENE-1960 Project: Lucene - Java Issue Type: Task Reporter: Michael Busch Assignee: Michael Busch Priority: Minor Fix For: 3.0 Attachments: index.29.cfs.zip, index.29.cfs.zip, index.29.nocfs.zip, index.29.nocfs.zip, lucene-1960-1-branch29.patch, lucene-1960-1.patch, lucene-1960-1.patch, lucene-1960-1.patch, lucene-1960-1.patch, lucene-1960.patch, optimize-time.txt Also remove FieldForMerge and related code. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Updated: (LUCENE-1960) Remove deprecated Field.Store.COMPRESS
[ https://issues.apache.org/jira/browse/LUCENE-1960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uwe Schindler updated LUCENE-1960: -- Attachment: lucene-1960-1.patch Small optimization. Remove deprecated Field.Store.COMPRESS -- Key: LUCENE-1960 URL: https://issues.apache.org/jira/browse/LUCENE-1960 Project: Lucene - Java Issue Type: Task Reporter: Michael Busch Assignee: Michael Busch Priority: Minor Fix For: 3.0 Attachments: index.29.cfs.zip, index.29.cfs.zip, index.29.nocfs.zip, index.29.nocfs.zip, lucene-1960-1-branch29.patch, lucene-1960-1.patch, lucene-1960-1.patch, lucene-1960-1.patch, lucene-1960-1.patch, lucene-1960.patch, optimize-time.txt Also remove FieldForMerge and related code. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Updated: (LUCENE-1960) Remove deprecated Field.Store.COMPRESS
[ https://issues.apache.org/jira/browse/LUCENE-1960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uwe Schindler updated LUCENE-1960: -- Attachment: (was: lucene-1960-1.patch) Remove deprecated Field.Store.COMPRESS -- Key: LUCENE-1960 URL: https://issues.apache.org/jira/browse/LUCENE-1960 Project: Lucene - Java Issue Type: Task Reporter: Michael Busch Assignee: Michael Busch Priority: Minor Fix For: 3.0 Attachments: index.29.cfs.zip, index.29.cfs.zip, index.29.nocfs.zip, index.29.nocfs.zip, lucene-1960-1-branch29.patch, lucene-1960-1.patch, lucene-1960-1.patch, lucene-1960-1.patch, lucene-1960-1.patch, lucene-1960.patch, optimize-time.txt Also remove FieldForMerge and related code. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Updated: (LUCENE-1960) Remove deprecated Field.Store.COMPRESS
[ https://issues.apache.org/jira/browse/LUCENE-1960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uwe Schindler updated LUCENE-1960: -- Attachment: lucene-1960-1.patch An additional check, that the raw merge mode is disabled for all segments in 2.9 index and enabled after optimizing with 3.0. Remove deprecated Field.Store.COMPRESS -- Key: LUCENE-1960 URL: https://issues.apache.org/jira/browse/LUCENE-1960 Project: Lucene - Java Issue Type: Task Reporter: Michael Busch Assignee: Michael Busch Priority: Minor Fix For: 3.0 Attachments: index.29.cfs.zip, index.29.cfs.zip, index.29.nocfs.zip, index.29.nocfs.zip, lucene-1960-1-branch29.patch, lucene-1960-1.patch, lucene-1960-1.patch, lucene-1960-1.patch, lucene-1960-1.patch, lucene-1960-1.patch, lucene-1960.patch, optimize-time.txt Also remove FieldForMerge and related code. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Updated: (LUCENE-1960) Remove deprecated Field.Store.COMPRESS
[ https://issues.apache.org/jira/browse/LUCENE-1960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uwe Schindler updated LUCENE-1960: -- Attachment: index.29.nocfs.zip index.29.cfs.zip Remove deprecated Field.Store.COMPRESS -- Key: LUCENE-1960 URL: https://issues.apache.org/jira/browse/LUCENE-1960 Project: Lucene - Java Issue Type: Task Reporter: Michael Busch Assignee: Michael Busch Priority: Minor Fix For: 3.0 Attachments: index.29.cfs.zip, index.29.cfs.zip, index.29.cfs.zip, index.29.nocfs.zip, index.29.nocfs.zip, index.29.nocfs.zip, lucene-1960-1-branch29.patch, lucene-1960-1.patch, lucene-1960-1.patch, lucene-1960-1.patch, lucene-1960-1.patch, lucene-1960-1.patch, lucene-1960.patch, optimize-time.txt Also remove FieldForMerge and related code. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Updated: (LUCENE-1960) Remove deprecated Field.Store.COMPRESS
[ https://issues.apache.org/jira/browse/LUCENE-1960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uwe Schindler updated LUCENE-1960: -- Attachment: lucene-1960-1-branch29.patch lucene-1960-1.patch Attached the new indexes and patch that stores the compressed size together with the compressed value as stored field in the 2.9 index. This is now secure and invariant on different in-JDK compression implementations. Remove deprecated Field.Store.COMPRESS -- Key: LUCENE-1960 URL: https://issues.apache.org/jira/browse/LUCENE-1960 Project: Lucene - Java Issue Type: Task Reporter: Michael Busch Assignee: Michael Busch Priority: Minor Fix For: 3.0 Attachments: index.29.cfs.zip, index.29.cfs.zip, index.29.cfs.zip, index.29.nocfs.zip, index.29.nocfs.zip, index.29.nocfs.zip, lucene-1960-1-branch29.patch, lucene-1960-1-branch29.patch, lucene-1960-1.patch, lucene-1960-1.patch, lucene-1960-1.patch, lucene-1960-1.patch, lucene-1960-1.patch, lucene-1960-1.patch, lucene-1960.patch, optimize-time.txt Also remove FieldForMerge and related code. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Updated: (LUCENE-1960) Remove deprecated Field.Store.COMPRESS
[ https://issues.apache.org/jira/browse/LUCENE-1960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uwe Schindler updated LUCENE-1960: -- Attachment: lucene-1960-1.patch Add an assertion in FieldsReader that checks, that 3.0 indexes have no compressed fields. Also a small test cleanup. Its ready to commit now! Remove deprecated Field.Store.COMPRESS -- Key: LUCENE-1960 URL: https://issues.apache.org/jira/browse/LUCENE-1960 Project: Lucene - Java Issue Type: Task Reporter: Michael Busch Assignee: Michael Busch Priority: Minor Fix For: 3.0 Attachments: index.29.cfs.zip, index.29.cfs.zip, index.29.cfs.zip, index.29.nocfs.zip, index.29.nocfs.zip, index.29.nocfs.zip, lucene-1960-1-branch29.patch, lucene-1960-1-branch29.patch, lucene-1960-1.patch, lucene-1960-1.patch, lucene-1960-1.patch, lucene-1960-1.patch, lucene-1960-1.patch, lucene-1960-1.patch, lucene-1960-1.patch, lucene-1960.patch, optimize-time.txt Also remove FieldForMerge and related code. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Updated: (LUCENE-1960) Remove deprecated Field.Store.COMPRESS
[ https://issues.apache.org/jira/browse/LUCENE-1960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uwe Schindler updated LUCENE-1960: -- Attachment: optimize-time.txt Attached is my comparison of an unoptimzed Lucene 2.9 index optimized with 2.9 and optimized with 3.0 with the latest patch. The index was about 5.7 GB big and contained 4 compressed stored fields per document (in addition to ther fields uncompressed) containing XML documents. After optimization with 3.0, the size doubled (which is because of the very good compression of XML documents). The optimization took about double time with 3.0, because the fields were decompressed and no addRawDocuments was used. To confirm, that everything worked normal after this initial optimization, I updated 38 documents in both indexes and optimized again. The optimization time of 2.9 was identical, 3.0 took a little bit longer, which is because the uncompressed field created more copy i/o, which you see in the time output as system time. User time during the initial 2.9 - 3.0 optimization was much larger. The attached document contains all numbers together with the checkindex outputs before and after each step. Remove deprecated Field.Store.COMPRESS -- Key: LUCENE-1960 URL: https://issues.apache.org/jira/browse/LUCENE-1960 Project: Lucene - Java Issue Type: Task Reporter: Michael Busch Assignee: Michael Busch Priority: Minor Fix For: 3.0 Attachments: lucene-1960-1.patch, lucene-1960-1.patch, lucene-1960.patch, optimize-time.txt Also remove FieldForMerge and related code. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Updated: (LUCENE-1960) Remove deprecated Field.Store.COMPRESS
[
https://issues.apache.org/jira/browse/LUCENE-1960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Busch updated LUCENE-1960:
--
Attachment: lucene-1960-1.patch
It was as easy as changing this method in FieldsReader:
{code:java}
boolean canReadRawDocs() {
// Disable reading raw docs in 2.x format, because of the removal of
compressed
// fields in 3.0. We don't want rawDocs() to decode field bits to figure out
// if a field was compressed, hence we enforce ordinary (non-raw) stored
field merges
// for 3.0 indexes.
return format = FieldsWriter.FORMAT_LUCENE_3_0_NO_COMPRESSED_FIELDS;
}
{code}
Uwe, I made some quick tests and it looks good. But I don't have any indexes
with compressed fields (we don't use them), so I'll wait for you to test it out
with your indexes that you mentioned.
Remove deprecated Field.Store.COMPRESS
--
Key: LUCENE-1960
URL: https://issues.apache.org/jira/browse/LUCENE-1960
Project: Lucene - Java
Issue Type: Task
Reporter: Michael Busch
Assignee: Michael Busch
Priority: Minor
Fix For: 3.0
Attachments: lucene-1960-1.patch, lucene-1960-1.patch,
lucene-1960.patch
Also remove FieldForMerge and related code.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Updated: (LUCENE-1960) Remove deprecated Field.Store.COMPRESS
[ https://issues.apache.org/jira/browse/LUCENE-1960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Busch updated LUCENE-1960: -- Attachment: lucene-1960-1.patch Adds the compress bit back to FieldsWriter and the uncompress code to FieldsReader. Uwe, could you review the patch? Remove deprecated Field.Store.COMPRESS -- Key: LUCENE-1960 URL: https://issues.apache.org/jira/browse/LUCENE-1960 Project: Lucene - Java Issue Type: Task Reporter: Michael Busch Assignee: Michael Busch Priority: Minor Fix For: 3.0 Attachments: lucene-1960-1.patch, lucene-1960.patch Also remove FieldForMerge and related code. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Updated: (LUCENE-1960) Remove deprecated Field.Store.COMPRESS
[ https://issues.apache.org/jira/browse/LUCENE-1960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Busch updated LUCENE-1960: -- Attachment: lucene-1960.patch All core contrib tests pass. Remove deprecated Field.Store.COMPRESS -- Key: LUCENE-1960 URL: https://issues.apache.org/jira/browse/LUCENE-1960 Project: Lucene - Java Issue Type: Task Reporter: Michael Busch Assignee: Michael Busch Fix For: 3.0 Attachments: lucene-1960.patch Also remove FieldForMerge and related code. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Updated: (LUCENE-1960) Remove deprecated Field.Store.COMPRESS
[ https://issues.apache.org/jira/browse/LUCENE-1960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Busch updated LUCENE-1960: -- Priority: Minor (was: Major) Remove deprecated Field.Store.COMPRESS -- Key: LUCENE-1960 URL: https://issues.apache.org/jira/browse/LUCENE-1960 Project: Lucene - Java Issue Type: Task Reporter: Michael Busch Assignee: Michael Busch Priority: Minor Fix For: 3.0 Attachments: lucene-1960.patch Also remove FieldForMerge and related code. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org
