[jira] [Updated] (LUCENE-4583) StraightBytesDocValuesField fails if bytes 32k
[
https://issues.apache.org/jira/browse/LUCENE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael McCandless updated LUCENE-4583:
---
Attachment: LUCENE-4583.patch
bq. I'm confused by the following comment:
I fixed the comment; it's because those DVFormats use PagedBytes.fillSlice,
which cannot handle more than 2 pages.
New patch w/ that fix ...
StraightBytesDocValuesField fails if bytes 32k
Key: LUCENE-4583
URL: https://issues.apache.org/jira/browse/LUCENE-4583
Project: Lucene - Core
Issue Type: Bug
Components: core/index
Affects Versions: 4.0, 4.1, 5.0
Reporter: David Smiley
Assignee: Michael McCandless
Priority: Critical
Fix For: 5.0, 4.5
Attachments: LUCENE-4583.patch, LUCENE-4583.patch, LUCENE-4583.patch,
LUCENE-4583.patch, LUCENE-4583.patch, LUCENE-4583.patch, LUCENE-4583.patch,
LUCENE-4583.patch
I didn't observe any limitations on the size of a bytes based DocValues field
value in the docs. It appears that the limit is 32k, although I didn't get
any friendly error telling me that was the limit. 32k is kind of small IMO;
I suspect this limit is unintended and as such is a bug.The following
test fails:
{code:java}
public void testBigDocValue() throws IOException {
Directory dir = newDirectory();
IndexWriter writer = new IndexWriter(dir, writerConfig(false));
Document doc = new Document();
BytesRef bytes = new BytesRef((4+4)*4097);//4096 works
bytes.length = bytes.bytes.length;//byte data doesn't matter
doc.add(new StraightBytesDocValuesField(dvField, bytes));
writer.addDocument(doc);
writer.commit();
writer.close();
DirectoryReader reader = DirectoryReader.open(dir);
DocValues docValues = MultiDocValues.getDocValues(reader, dvField);
//FAILS IF BYTES IS BIG!
docValues.getSource().getBytes(0, bytes);
reader.close();
dir.close();
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-4583) StraightBytesDocValuesField fails if bytes 32k
[
https://issues.apache.org/jira/browse/LUCENE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael McCandless updated LUCENE-4583:
---
Attachment: LUCENE-4583.patch
New patch.
Thanks for the review Shai; I removed that leftover print and sync'd
patch to trunk. I think it's ready ... I'll wait a few days.
StraightBytesDocValuesField fails if bytes 32k
Key: LUCENE-4583
URL: https://issues.apache.org/jira/browse/LUCENE-4583
Project: Lucene - Core
Issue Type: Bug
Components: core/index
Affects Versions: 4.0, 4.1, 5.0
Reporter: David Smiley
Assignee: Michael McCandless
Priority: Critical
Fix For: 5.0, 4.5
Attachments: LUCENE-4583.patch, LUCENE-4583.patch, LUCENE-4583.patch,
LUCENE-4583.patch, LUCENE-4583.patch, LUCENE-4583.patch, LUCENE-4583.patch
I didn't observe any limitations on the size of a bytes based DocValues field
value in the docs. It appears that the limit is 32k, although I didn't get
any friendly error telling me that was the limit. 32k is kind of small IMO;
I suspect this limit is unintended and as such is a bug.The following
test fails:
{code:java}
public void testBigDocValue() throws IOException {
Directory dir = newDirectory();
IndexWriter writer = new IndexWriter(dir, writerConfig(false));
Document doc = new Document();
BytesRef bytes = new BytesRef((4+4)*4097);//4096 works
bytes.length = bytes.bytes.length;//byte data doesn't matter
doc.add(new StraightBytesDocValuesField(dvField, bytes));
writer.addDocument(doc);
writer.commit();
writer.close();
DirectoryReader reader = DirectoryReader.open(dir);
DocValues docValues = MultiDocValues.getDocValues(reader, dvField);
//FAILS IF BYTES IS BIG!
docValues.getSource().getBytes(0, bytes);
reader.close();
dir.close();
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-4583) StraightBytesDocValuesField fails if bytes 32k
[
https://issues.apache.org/jira/browse/LUCENE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael McCandless updated LUCENE-4583:
---
Attachment: LUCENE-4583.patch
Another iteration on the patch:
* I added constants MAX_BINARY_FIELD_LENGTH to
Lucene4{0,2}DocValuesFormat, and then reference that in the
Writer/Consumer to catch too-big values.
* I added another test to BaseDocValuesFormatTestCase, to test the
exact maximum length value.
* I fixed that test failure, by passing the String field to the
codecAcceptsHugeBinaryValues method, and adding a _TestUtil helper
method to check this.
An alternative to the protected method would be to have two separate
tests in the base class, one test verifying a clean IllegalArgumentExc
is thrown when the value is too big, and another verifying huge binary
values can be indexed successfully. And then I'd fix each DVFormat's
test to subclass and @Ignore whichever base test is not appropriate.
But I don't think this would simplify things much? Ie,
TestDocValuesFormat would still need logic to check depending on the
default codec.
StraightBytesDocValuesField fails if bytes 32k
Key: LUCENE-4583
URL: https://issues.apache.org/jira/browse/LUCENE-4583
Project: Lucene - Core
Issue Type: Bug
Components: core/index
Affects Versions: 4.0, 4.1, 5.0
Reporter: David Smiley
Priority: Critical
Fix For: 4.4
Attachments: LUCENE-4583.patch, LUCENE-4583.patch, LUCENE-4583.patch,
LUCENE-4583.patch, LUCENE-4583.patch, LUCENE-4583.patch
I didn't observe any limitations on the size of a bytes based DocValues field
value in the docs. It appears that the limit is 32k, although I didn't get
any friendly error telling me that was the limit. 32k is kind of small IMO;
I suspect this limit is unintended and as such is a bug.The following
test fails:
{code:java}
public void testBigDocValue() throws IOException {
Directory dir = newDirectory();
IndexWriter writer = new IndexWriter(dir, writerConfig(false));
Document doc = new Document();
BytesRef bytes = new BytesRef((4+4)*4097);//4096 works
bytes.length = bytes.bytes.length;//byte data doesn't matter
doc.add(new StraightBytesDocValuesField(dvField, bytes));
writer.addDocument(doc);
writer.commit();
writer.close();
DirectoryReader reader = DirectoryReader.open(dir);
DocValues docValues = MultiDocValues.getDocValues(reader, dvField);
//FAILS IF BYTES IS BIG!
docValues.getSource().getBytes(0, bytes);
reader.close();
dir.close();
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-4583) StraightBytesDocValuesField fails if bytes 32k
[
https://issues.apache.org/jira/browse/LUCENE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael McCandless updated LUCENE-4583:
---
Attachment: LUCENE-4583.patch
New patch, cutting over to PagedBytes. I also revived .getDataInput/Output
(from 4.x).
I also added javadocs about these limits in DocValuesType.
There are still nocommits to resolve ... the biggest one is whether we should
fix default DV impl to accept values 32 KB ... right now it throws IAE.
StraightBytesDocValuesField fails if bytes 32k
Key: LUCENE-4583
URL: https://issues.apache.org/jira/browse/LUCENE-4583
Project: Lucene - Core
Issue Type: Bug
Components: core/index
Affects Versions: 4.0, 4.1, 5.0
Reporter: David Smiley
Priority: Critical
Fix For: 4.4
Attachments: LUCENE-4583.patch, LUCENE-4583.patch, LUCENE-4583.patch,
LUCENE-4583.patch
I didn't observe any limitations on the size of a bytes based DocValues field
value in the docs. It appears that the limit is 32k, although I didn't get
any friendly error telling me that was the limit. 32k is kind of small IMO;
I suspect this limit is unintended and as such is a bug.The following
test fails:
{code:java}
public void testBigDocValue() throws IOException {
Directory dir = newDirectory();
IndexWriter writer = new IndexWriter(dir, writerConfig(false));
Document doc = new Document();
BytesRef bytes = new BytesRef((4+4)*4097);//4096 works
bytes.length = bytes.bytes.length;//byte data doesn't matter
doc.add(new StraightBytesDocValuesField(dvField, bytes));
writer.addDocument(doc);
writer.commit();
writer.close();
DirectoryReader reader = DirectoryReader.open(dir);
DocValues docValues = MultiDocValues.getDocValues(reader, dvField);
//FAILS IF BYTES IS BIG!
docValues.getSource().getBytes(0, bytes);
reader.close();
dir.close();
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-4583) StraightBytesDocValuesField fails if bytes 32k
[
https://issues.apache.org/jira/browse/LUCENE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael McCandless updated LUCENE-4583:
---
Attachment: LUCENE-4583.patch
bq. I do not think this should change.
OK I just removed that nocommit.
bq. I just had a quick discussion about this with Robert, and since
AppendingLongBuffer stores deltas from the minimum value of the block (and not
0), AppendingLongBuffer is better (ie. faster and more compact) than
MonotonicAppendingLongBuffer to store lengths. This means that if all lengths
are 7, 8 or 9 in a block, it will only require 2 bits per value instead of 4.
Ahh, OK: I switched back to AppendingLongBuffer.
New patch. I factored the testHugeBinaryValues up into the
BaseDocValuesFormatTestCase base class, and added protected method so
the codecs that don't accept huge binary values can say so. I also
added a test case for Facet42DVFormat, and cut back to
AppendingLongBuffer.
I downgraded the nocommit about the spooky unused PagedBytes.blockEnds
to a TODO ... this class is somewhat dangerous because e.g. you can
use copyUsingLengthPrefix method and then get a .getDataOutput and get
corrumpted bytes out.
StraightBytesDocValuesField fails if bytes 32k
Key: LUCENE-4583
URL: https://issues.apache.org/jira/browse/LUCENE-4583
Project: Lucene - Core
Issue Type: Bug
Components: core/index
Affects Versions: 4.0, 4.1, 5.0
Reporter: David Smiley
Priority: Critical
Fix For: 4.4
Attachments: LUCENE-4583.patch, LUCENE-4583.patch, LUCENE-4583.patch,
LUCENE-4583.patch, LUCENE-4583.patch
I didn't observe any limitations on the size of a bytes based DocValues field
value in the docs. It appears that the limit is 32k, although I didn't get
any friendly error telling me that was the limit. 32k is kind of small IMO;
I suspect this limit is unintended and as such is a bug.The following
test fails:
{code:java}
public void testBigDocValue() throws IOException {
Directory dir = newDirectory();
IndexWriter writer = new IndexWriter(dir, writerConfig(false));
Document doc = new Document();
BytesRef bytes = new BytesRef((4+4)*4097);//4096 works
bytes.length = bytes.bytes.length;//byte data doesn't matter
doc.add(new StraightBytesDocValuesField(dvField, bytes));
writer.addDocument(doc);
writer.commit();
writer.close();
DirectoryReader reader = DirectoryReader.open(dir);
DocValues docValues = MultiDocValues.getDocValues(reader, dvField);
//FAILS IF BYTES IS BIG!
docValues.getSource().getBytes(0, bytes);
reader.close();
dir.close();
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-4583) StraightBytesDocValuesField fails if bytes 32k
[
https://issues.apache.org/jira/browse/LUCENE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael McCandless updated LUCENE-4583:
---
Attachment: LUCENE-4583.patch
I iterated from Barakat's patch: improved the test, and added
enforcing of the limit in the appropriate DocValuesFormats impls.
Disk, SimpleText, CheapBastard and (I think ... still need a test
here) Facet42DVFormat don't have a limit, but Lucene40/42 still do.
bq. I'm -1 to bumping the limit
Are you also against just fixing the limit in the core code
(IndexWriter/BinaryDocValuesWriter) and leaving the limit enforced in
the existing DVFormats (my patch)?
I thought that was a good compromise ...
This way at least users can still build their own / use DVFormats that
don't have the limit.
StraightBytesDocValuesField fails if bytes 32k
Key: LUCENE-4583
URL: https://issues.apache.org/jira/browse/LUCENE-4583
Project: Lucene - Core
Issue Type: Bug
Components: core/index
Affects Versions: 4.0, 4.1, 5.0
Reporter: David Smiley
Priority: Critical
Fix For: 4.3
Attachments: LUCENE-4583.patch, LUCENE-4583.patch, LUCENE-4583.patch
I didn't observe any limitations on the size of a bytes based DocValues field
value in the docs. It appears that the limit is 32k, although I didn't get
any friendly error telling me that was the limit. 32k is kind of small IMO;
I suspect this limit is unintended and as such is a bug.The following
test fails:
{code:java}
public void testBigDocValue() throws IOException {
Directory dir = newDirectory();
IndexWriter writer = new IndexWriter(dir, writerConfig(false));
Document doc = new Document();
BytesRef bytes = new BytesRef((4+4)*4097);//4096 works
bytes.length = bytes.bytes.length;//byte data doesn't matter
doc.add(new StraightBytesDocValuesField(dvField, bytes));
writer.addDocument(doc);
writer.commit();
writer.close();
DirectoryReader reader = DirectoryReader.open(dir);
DocValues docValues = MultiDocValues.getDocValues(reader, dvField);
//FAILS IF BYTES IS BIG!
docValues.getSource().getBytes(0, bytes);
reader.close();
dir.close();
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-4583) StraightBytesDocValuesField fails if bytes 32k
[
https://issues.apache.org/jira/browse/LUCENE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uwe Schindler updated LUCENE-4583:
--
Fix Version/s: (was: 4.3)
4.4
StraightBytesDocValuesField fails if bytes 32k
Key: LUCENE-4583
URL: https://issues.apache.org/jira/browse/LUCENE-4583
Project: Lucene - Core
Issue Type: Bug
Components: core/index
Affects Versions: 4.0, 4.1, 5.0
Reporter: David Smiley
Priority: Critical
Fix For: 4.4
Attachments: LUCENE-4583.patch, LUCENE-4583.patch, LUCENE-4583.patch
I didn't observe any limitations on the size of a bytes based DocValues field
value in the docs. It appears that the limit is 32k, although I didn't get
any friendly error telling me that was the limit. 32k is kind of small IMO;
I suspect this limit is unintended and as such is a bug.The following
test fails:
{code:java}
public void testBigDocValue() throws IOException {
Directory dir = newDirectory();
IndexWriter writer = new IndexWriter(dir, writerConfig(false));
Document doc = new Document();
BytesRef bytes = new BytesRef((4+4)*4097);//4096 works
bytes.length = bytes.bytes.length;//byte data doesn't matter
doc.add(new StraightBytesDocValuesField(dvField, bytes));
writer.addDocument(doc);
writer.commit();
writer.close();
DirectoryReader reader = DirectoryReader.open(dir);
DocValues docValues = MultiDocValues.getDocValues(reader, dvField);
//FAILS IF BYTES IS BIG!
docValues.getSource().getBytes(0, bytes);
reader.close();
dir.close();
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-4583) StraightBytesDocValuesField fails if bytes 32k
[
https://issues.apache.org/jira/browse/LUCENE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Barakat Barakat updated LUCENE-4583:
Attachment: LUCENE-4583.patch
I recently switched from 4.1 to 4.3, and my patch needed to be updated because
of the changes to DocValues. The problem was almost fixed for BinaryDocValues,
but it just needed one little change. I've attached a patch that removes the
BinaryDocValues exception when the length is over BYTE_BLOCK_SIZE (32k), fixes
ByteBlockPool#readBytes:348, and changes the
TestDocValuesIndexing#testTooLargeBytes test to check for accuracy.
StraightBytesDocValuesField fails if bytes 32k
Key: LUCENE-4583
URL: https://issues.apache.org/jira/browse/LUCENE-4583
Project: Lucene - Core
Issue Type: Bug
Components: core/index
Affects Versions: 4.0, 4.1, 5.0
Reporter: David Smiley
Priority: Critical
Fix For: 4.3
Attachments: LUCENE-4583.patch, LUCENE-4583.patch
I didn't observe any limitations on the size of a bytes based DocValues field
value in the docs. It appears that the limit is 32k, although I didn't get
any friendly error telling me that was the limit. 32k is kind of small IMO;
I suspect this limit is unintended and as such is a bug.The following
test fails:
{code:java}
public void testBigDocValue() throws IOException {
Directory dir = newDirectory();
IndexWriter writer = new IndexWriter(dir, writerConfig(false));
Document doc = new Document();
BytesRef bytes = new BytesRef((4+4)*4097);//4096 works
bytes.length = bytes.bytes.length;//byte data doesn't matter
doc.add(new StraightBytesDocValuesField(dvField, bytes));
writer.addDocument(doc);
writer.commit();
writer.close();
DirectoryReader reader = DirectoryReader.open(dir);
DocValues docValues = MultiDocValues.getDocValues(reader, dvField);
//FAILS IF BYTES IS BIG!
docValues.getSource().getBytes(0, bytes);
reader.close();
dir.close();
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-4583) StraightBytesDocValuesField fails if bytes 32k
[
https://issues.apache.org/jira/browse/LUCENE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Barakat Barakat updated LUCENE-4583:
Attachment: LUCENE-4583.patch
I updated the code to work when start isn't zero. The code can still crash if
you ask for a length that goes beyond the total size of the paged bytes, but
I'm not sure how you guys like to prevent things like that. The code seems to
be working fine with our Solr core so far. I am new to posting patches and
writing test units in Java so please let me know if there is anything wrong
with the code.
StraightBytesDocValuesField fails if bytes 32k
Key: LUCENE-4583
URL: https://issues.apache.org/jira/browse/LUCENE-4583
Project: Lucene - Core
Issue Type: Bug
Components: core/index
Affects Versions: 4.0, 4.1, 5.0
Reporter: David Smiley
Priority: Critical
Fix For: 4.1
Attachments: LUCENE-4583.patch
I didn't observe any limitations on the size of a bytes based DocValues field
value in the docs. It appears that the limit is 32k, although I didn't get
any friendly error telling me that was the limit. 32k is kind of small IMO;
I suspect this limit is unintended and as such is a bug.The following
test fails:
{code:java}
public void testBigDocValue() throws IOException {
Directory dir = newDirectory();
IndexWriter writer = new IndexWriter(dir, writerConfig(false));
Document doc = new Document();
BytesRef bytes = new BytesRef((4+4)*4097);//4096 works
bytes.length = bytes.bytes.length;//byte data doesn't matter
doc.add(new StraightBytesDocValuesField(dvField, bytes));
writer.addDocument(doc);
writer.commit();
writer.close();
DirectoryReader reader = DirectoryReader.open(dir);
DocValues docValues = MultiDocValues.getDocValues(reader, dvField);
//FAILS IF BYTES IS BIG!
docValues.getSource().getBytes(0, bytes);
reader.close();
dir.close();
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-4583) StraightBytesDocValuesField fails if bytes 32k
[
https://issues.apache.org/jira/browse/LUCENE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yonik Seeley updated LUCENE-4583:
-
Fix Version/s: 4.1
StraightBytesDocValuesField fails if bytes 32k
Key: LUCENE-4583
URL: https://issues.apache.org/jira/browse/LUCENE-4583
Project: Lucene - Core
Issue Type: Bug
Components: core/index
Affects Versions: 4.0, 4.1, 5.0
Reporter: David Smiley
Priority: Critical
Fix For: 4.1
I didn't observe any limitations on the size of a bytes based DocValues field
value in the docs. It appears that the limit is 32k, although I didn't get
any friendly error telling me that was the limit. 32k is kind of small IMO;
I suspect this limit is unintended and as such is a bug.The following
test fails:
{code:java}
public void testBigDocValue() throws IOException {
Directory dir = newDirectory();
IndexWriter writer = new IndexWriter(dir, writerConfig(false));
Document doc = new Document();
BytesRef bytes = new BytesRef((4+4)*4097);//4096 works
bytes.length = bytes.bytes.length;//byte data doesn't matter
doc.add(new StraightBytesDocValuesField(dvField, bytes));
writer.addDocument(doc);
writer.commit();
writer.close();
DirectoryReader reader = DirectoryReader.open(dir);
DocValues docValues = MultiDocValues.getDocValues(reader, dvField);
//FAILS IF BYTES IS BIG!
docValues.getSource().getBytes(0, bytes);
reader.close();
dir.close();
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
