[jira] [Commented] (LUCENE-4239) Provide access to PackedInts' low-level blocks <-> values conversion methods
[
https://issues.apache.org/jira/browse/LUCENE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419917#comment-13419917
]
Adrien Grand commented on LUCENE-4239:
--
I just committed. Let's see how we can improve it!
> Provide access to PackedInts' low-level blocks <-> values conversion methods
>
>
> Key: LUCENE-4239
> URL: https://issues.apache.org/jira/browse/LUCENE-4239
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/other
>Reporter: Adrien Grand
>Assignee: Adrien Grand
>Priority: Minor
> Fix For: 4.0
>
> Attachments: LUCENE-4239.patch
>
>
> In LUCENE-4161 we started to make the {{PackedInts}} API more flexible so
> that codecs could use it whenever they need to (un)pack integers. There are
> two posting formats in progress (For and PFor, LUCENE-3892) that perform a
> lot of integer (un)packing but the current API still has limits :
> - it only works with long[] arrays, whereas these codecs need to manipulate
> int[] arrays,
> - the packed reader iterators work great for unpacking long sequences of
> integers, but they would probably cause a lot of overhead to decode lots of
> short integer sequences such as the ones that can be generated by For and
> PFor.
> I've been looking at the For/PFor branch and it has a
> {{PackedIntsDecompress}} class
> (http://svn.apache.org/repos/asf/lucene/dev/branches/pforcodec_3892/lucene/core/src/java/org/apache/lucene/codecs/pfor/PackedIntsDecompress.java)
> which is very similar to {{oal.util.packed.BulkOperation}}
> (package-private), so maybe we should find a way to expose this class so that
> the For/PFor branch can directly use it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-4239) Provide access to PackedInts' low-level blocks <-> values conversion methods
[
https://issues.apache.org/jira/browse/LUCENE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419902#comment-13419902
]
Han Jiang commented on LUCENE-4239:
---
Hi Adrien, you're right! There is something wrong with my subversion. I see the
codes in your patch, but cannot get that method fully patched. I'll wait for
your commit!
> Provide access to PackedInts' low-level blocks <-> values conversion methods
>
>
> Key: LUCENE-4239
> URL: https://issues.apache.org/jira/browse/LUCENE-4239
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/other
>Reporter: Adrien Grand
>Assignee: Adrien Grand
>Priority: Minor
> Fix For: 4.0
>
> Attachments: LUCENE-4239.patch
>
>
> In LUCENE-4161 we started to make the {{PackedInts}} API more flexible so
> that codecs could use it whenever they need to (un)pack integers. There are
> two posting formats in progress (For and PFor, LUCENE-3892) that perform a
> lot of integer (un)packing but the current API still has limits :
> - it only works with long[] arrays, whereas these codecs need to manipulate
> int[] arrays,
> - the packed reader iterators work great for unpacking long sequences of
> integers, but they would probably cause a lot of overhead to decode lots of
> short integer sequences such as the ones that can be generated by For and
> PFor.
> I've been looking at the For/PFor branch and it has a
> {{PackedIntsDecompress}} class
> (http://svn.apache.org/repos/asf/lucene/dev/branches/pforcodec_3892/lucene/core/src/java/org/apache/lucene/codecs/pfor/PackedIntsDecompress.java)
> which is very similar to {{oal.util.packed.BulkOperation}}
> (package-private), so maybe we should find a way to expose this class so that
> the For/PFor branch can directly use it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-4239) Provide access to PackedInts' low-level blocks <-> values conversion methods
[
https://issues.apache.org/jira/browse/LUCENE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419893#comment-13419893
]
Adrien Grand commented on LUCENE-4239:
--
Han, perhaps the patch didn't apply well on your computer. I just tried to
re-apply it and I've had no problem with Packed64SingleBlockBulkOperation32.
I'll commit shortly on the pfor branch...
> Provide access to PackedInts' low-level blocks <-> values conversion methods
>
>
> Key: LUCENE-4239
> URL: https://issues.apache.org/jira/browse/LUCENE-4239
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/other
>Reporter: Adrien Grand
>Assignee: Adrien Grand
>Priority: Minor
> Fix For: 4.0
>
> Attachments: LUCENE-4239.patch
>
>
> In LUCENE-4161 we started to make the {{PackedInts}} API more flexible so
> that codecs could use it whenever they need to (un)pack integers. There are
> two posting formats in progress (For and PFor, LUCENE-3892) that perform a
> lot of integer (un)packing but the current API still has limits :
> - it only works with long[] arrays, whereas these codecs need to manipulate
> int[] arrays,
> - the packed reader iterators work great for unpacking long sequences of
> integers, but they would probably cause a lot of overhead to decode lots of
> short integer sequences such as the ones that can be generated by For and
> PFor.
> I've been looking at the For/PFor branch and it has a
> {{PackedIntsDecompress}} class
> (http://svn.apache.org/repos/asf/lucene/dev/branches/pforcodec_3892/lucene/core/src/java/org/apache/lucene/codecs/pfor/PackedIntsDecompress.java)
> which is very similar to {{oal.util.packed.BulkOperation}}
> (package-private), so maybe we should find a way to expose this class so that
> the For/PFor branch can directly use it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-4239) Provide access to PackedInts' low-level blocks <-> values conversion methods
[
https://issues.apache.org/jira/browse/LUCENE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419430#comment-13419430
]
Han Jiang commented on LUCENE-4239:
---
bq. I think we should just commit this current patch onto the block PF branch
...
+1, but shall we wait Adrien to fix the missing methods first?
> Provide access to PackedInts' low-level blocks <-> values conversion methods
>
>
> Key: LUCENE-4239
> URL: https://issues.apache.org/jira/browse/LUCENE-4239
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/other
>Reporter: Adrien Grand
>Assignee: Adrien Grand
>Priority: Minor
> Fix For: 4.0
>
> Attachments: LUCENE-4239.patch
>
>
> In LUCENE-4161 we started to make the {{PackedInts}} API more flexible so
> that codecs could use it whenever they need to (un)pack integers. There are
> two posting formats in progress (For and PFor, LUCENE-3892) that perform a
> lot of integer (un)packing but the current API still has limits :
> - it only works with long[] arrays, whereas these codecs need to manipulate
> int[] arrays,
> - the packed reader iterators work great for unpacking long sequences of
> integers, but they would probably cause a lot of overhead to decode lots of
> short integer sequences such as the ones that can be generated by For and
> PFor.
> I've been looking at the For/PFor branch and it has a
> {{PackedIntsDecompress}} class
> (http://svn.apache.org/repos/asf/lucene/dev/branches/pforcodec_3892/lucene/core/src/java/org/apache/lucene/codecs/pfor/PackedIntsDecompress.java)
> which is very similar to {{oal.util.packed.BulkOperation}}
> (package-private), so maybe we should find a way to expose this class so that
> the For/PFor branch can directly use it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-4239) Provide access to PackedInts' low-level blocks <-> values conversion methods
[
https://issues.apache.org/jira/browse/LUCENE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419415#comment-13419415
]
Michael McCandless commented on LUCENE-4239:
I think we should just commit this current patch onto the block PF branch
(https://svn.apache.org/repos/asf/lucene/dev/branches/pforcodec_3892 )? Then
we can iterate on it, from both ends...
> Provide access to PackedInts' low-level blocks <-> values conversion methods
>
>
> Key: LUCENE-4239
> URL: https://issues.apache.org/jira/browse/LUCENE-4239
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/other
>Reporter: Adrien Grand
>Assignee: Adrien Grand
>Priority: Minor
> Fix For: 4.0
>
> Attachments: LUCENE-4239.patch
>
>
> In LUCENE-4161 we started to make the {{PackedInts}} API more flexible so
> that codecs could use it whenever they need to (un)pack integers. There are
> two posting formats in progress (For and PFor, LUCENE-3892) that perform a
> lot of integer (un)packing but the current API still has limits :
> - it only works with long[] arrays, whereas these codecs need to manipulate
> int[] arrays,
> - the packed reader iterators work great for unpacking long sequences of
> integers, but they would probably cause a lot of overhead to decode lots of
> short integer sequences such as the ones that can be generated by For and
> PFor.
> I've been looking at the For/PFor branch and it has a
> {{PackedIntsDecompress}} class
> (http://svn.apache.org/repos/asf/lucene/dev/branches/pforcodec_3892/lucene/core/src/java/org/apache/lucene/codecs/pfor/PackedIntsDecompress.java)
> which is very similar to {{oal.util.packed.BulkOperation}}
> (package-private), so maybe we should find a way to expose this class so that
> the For/PFor branch can directly use it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-4239) Provide access to PackedInts' low-level blocks <-> values conversion methods
[
https://issues.apache.org/jira/browse/LUCENE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419283#comment-13419283
]
Han Jiang commented on LUCENE-4239:
---
Thank you Adrien! We'll work easier with this Decoder/Encoder interface.
However, This patch isn't passing ant-compile under latest trunk, seems that
encoder/decoder methods for Packed64SingleBlockBulkOperation32 are missing?
Anyway, we're not using docId up to 32 bits currently, I'll test the
performance later.
Since we have to handle IndexInput/Output at upper level, we prefer to use
direct int[] rather than IntBuffer. Actually, we had a patch making
PackedIntsDecompress handle int array instead:
https://issues.apache.org/jira/secure/attachment/12532888/LUCENE-3892_for_int%5B%5D.patch
(the file name was ForDecompressImpl.java). Performance test shows little
difference between these two versions, but as int[] is clear and simple, I
think that should be what we hope to use.
So... maybe you can provide us methods like: encode(int[] values, long[]
blocks, int iterations), decode(long[] blocks, int[] values, int iterations)?
> Provide access to PackedInts' low-level blocks <-> values conversion methods
>
>
> Key: LUCENE-4239
> URL: https://issues.apache.org/jira/browse/LUCENE-4239
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/other
>Reporter: Adrien Grand
>Assignee: Adrien Grand
>Priority: Minor
> Fix For: 4.0
>
> Attachments: LUCENE-4239.patch
>
>
> In LUCENE-4161 we started to make the {{PackedInts}} API more flexible so
> that codecs could use it whenever they need to (un)pack integers. There are
> two posting formats in progress (For and PFor, LUCENE-3892) that perform a
> lot of integer (un)packing but the current API still has limits :
> - it only works with long[] arrays, whereas these codecs need to manipulate
> int[] arrays,
> - the packed reader iterators work great for unpacking long sequences of
> integers, but they would probably cause a lot of overhead to decode lots of
> short integer sequences such as the ones that can be generated by For and
> PFor.
> I've been looking at the For/PFor branch and it has a
> {{PackedIntsDecompress}} class
> (http://svn.apache.org/repos/asf/lucene/dev/branches/pforcodec_3892/lucene/core/src/java/org/apache/lucene/codecs/pfor/PackedIntsDecompress.java)
> which is very similar to {{oal.util.packed.BulkOperation}}
> (package-private), so maybe we should find a way to expose this class so that
> the For/PFor branch can directly use it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
