[jira] [Commented] (ACCUMULO-956) Iterator to transform key parts
[ https://issues.apache.org/jira/browse/ACCUMULO-956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13557781#comment-13557781 ] Hudson commented on ACCUMULO-956: - Integrated in Accumulo-Trunk #649 (See [https://builds.apache.org/job/Accumulo-Trunk/649/]) ACCUMULO-956 applied patch from Brian Loss (Revision 1435383) Result = SUCCESS kturner : Files : * /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/iterators/user/TransformingIterator.java Iterator to transform key parts --- Key: ACCUMULO-956 URL: https://issues.apache.org/jira/browse/ACCUMULO-956 Project: Accumulo Issue Type: Improvement Reporter: Brian Loss Assignee: Brian Loss Fix For: 1.5.0 Attachments: key_transforming_iterator-1.patch, key_transforming_iterator-2.patch, key_transforming_iterator-3.patch, key_transforming_iterator.patch, transforming_iterator-1.patch Iterators that transform parts of the key can be tricky if any transformation affects sort ordering. Implement an iterator that takes care of the tricky details that come with modifying sort order (e.g., handling scan-time iterator reconstruction and the associated seek). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ACCUMULO-956) Iterator to transform key parts
[ https://issues.apache.org/jira/browse/ACCUMULO-956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13556383#comment-13556383 ] Hudson commented on ACCUMULO-956: - Integrated in Accumulo-Trunk #643 (See [https://builds.apache.org/job/Accumulo-Trunk/643/]) ACCUMULO-956 formatted source code (Revision 1434768) ACCUMULO-956 checkin of patch from Brain Loss (Revision 1434762) Result = SUCCESS kturner : Files : * /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/iterators/user/KeyTransformingIterator.java * /accumulo/trunk/core/src/test/java/org/apache/accumulo/core/iterators/user/KeyTransformingIteratorTest.java kturner : Files : * /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/iterators/user/KeyTransformingIterator.java * /accumulo/trunk/core/src/test/java/org/apache/accumulo/core/iterators/user/KeyTransformingIteratorTest.java Iterator to transform key parts --- Key: ACCUMULO-956 URL: https://issues.apache.org/jira/browse/ACCUMULO-956 Project: Accumulo Issue Type: Improvement Reporter: Brian Loss Fix For: 1.5.0 Attachments: key_transforming_iterator-1.patch, key_transforming_iterator-2.patch, key_transforming_iterator-3.patch, key_transforming_iterator.patch Iterators that transform parts of the key can be tricky if any transformation affects sort ordering. Implement an iterator that takes care of the tricky details that come with modifying sort order (e.g., handling scan-time iterator reconstruction and the associated seek). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ACCUMULO-956) Iterator to transform key parts
[ https://issues.apache.org/jira/browse/ACCUMULO-956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13556605#comment-13556605 ] Hudson commented on ACCUMULO-956: - Integrated in Accumulo-Trunk #644 (See [https://builds.apache.org/job/Accumulo-Trunk/644/]) ACCUMULO-956 fixed some issues and added unit test for transforming iterator (Revision 1434881) Result = SUCCESS kturner : Files : * /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/iterators/user/KeyTransformingIterator.java * /accumulo/trunk/core/src/test/java/org/apache/accumulo/core/iterators/user/KeyTransformingIteratorTest.java Iterator to transform key parts --- Key: ACCUMULO-956 URL: https://issues.apache.org/jira/browse/ACCUMULO-956 Project: Accumulo Issue Type: Improvement Reporter: Brian Loss Fix For: 1.5.0 Attachments: key_transforming_iterator-1.patch, key_transforming_iterator-2.patch, key_transforming_iterator-3.patch, key_transforming_iterator.patch Iterators that transform parts of the key can be tricky if any transformation affects sort ordering. Implement an iterator that takes care of the tricky details that come with modifying sort order (e.g., handling scan-time iterator reconstruction and the associated seek). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ACCUMULO-956) Iterator to transform key parts
[ https://issues.apache.org/jira/browse/ACCUMULO-956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13556806#comment-13556806 ] Hudson commented on ACCUMULO-956: - Integrated in Accumulo-Trunk-Hadoop-2.0 #4 (See [https://builds.apache.org/job/Accumulo-Trunk-Hadoop-2.0/4/]) ACCUMULO-956 generlaized transforming iterator, added some sanity checks to it, added some more unit test, added some static config methods (Revision 1434955) Result = FAILURE kturner : Files : * /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/iterators/user/KeyTransformingIterator.java * /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/iterators/user/TransformingIterator.java * /accumulo/trunk/core/src/test/java/org/apache/accumulo/core/iterators/user/KeyTransformingIteratorTest.java * /accumulo/trunk/core/src/test/java/org/apache/accumulo/core/iterators/user/TransformingIteratorTest.java Iterator to transform key parts --- Key: ACCUMULO-956 URL: https://issues.apache.org/jira/browse/ACCUMULO-956 Project: Accumulo Issue Type: Improvement Reporter: Brian Loss Fix For: 1.5.0 Attachments: key_transforming_iterator-1.patch, key_transforming_iterator-2.patch, key_transforming_iterator-3.patch, key_transforming_iterator.patch Iterators that transform parts of the key can be tricky if any transformation affects sort ordering. Implement an iterator that takes care of the tricky details that come with modifying sort order (e.g., handling scan-time iterator reconstruction and the associated seek). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ACCUMULO-956) Iterator to transform key parts
[ https://issues.apache.org/jira/browse/ACCUMULO-956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13556826#comment-13556826 ] Hudson commented on ACCUMULO-956: - Integrated in Accumulo-Trunk #646 (See [https://builds.apache.org/job/Accumulo-Trunk/646/]) ACCUMULO-956 generlaized transforming iterator, added some sanity checks to it, added some more unit test, added some static config methods (Revision 1434955) Result = SUCCESS kturner : Files : * /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/iterators/user/KeyTransformingIterator.java * /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/iterators/user/TransformingIterator.java * /accumulo/trunk/core/src/test/java/org/apache/accumulo/core/iterators/user/KeyTransformingIteratorTest.java * /accumulo/trunk/core/src/test/java/org/apache/accumulo/core/iterators/user/TransformingIteratorTest.java Iterator to transform key parts --- Key: ACCUMULO-956 URL: https://issues.apache.org/jira/browse/ACCUMULO-956 Project: Accumulo Issue Type: Improvement Reporter: Brian Loss Fix For: 1.5.0 Attachments: key_transforming_iterator-1.patch, key_transforming_iterator-2.patch, key_transforming_iterator-3.patch, key_transforming_iterator.patch Iterators that transform parts of the key can be tricky if any transformation affects sort ordering. Implement an iterator that takes care of the tricky details that come with modifying sort order (e.g., handling scan-time iterator reconstruction and the associated seek). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ACCUMULO-956) Iterator to transform key parts
[ https://issues.apache.org/jira/browse/ACCUMULO-956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13552868#comment-13552868 ] Brian Loss commented on ACCUMULO-956: - I went ahead and changed it. The code didn't look as bloated as I thought it might :), so the new patch is up. Iterator to transform key parts --- Key: ACCUMULO-956 URL: https://issues.apache.org/jira/browse/ACCUMULO-956 Project: Accumulo Issue Type: Improvement Reporter: Brian Loss Fix For: 1.5.0 Attachments: key_transforming_iterator-1.patch, key_transforming_iterator-2.patch, key_transforming_iterator.patch Iterators that transform parts of the key can be tricky if any transformation affects sort ordering. Implement an iterator that takes care of the tricky details that come with modifying sort order (e.g., handling scan-time iterator reconstruction and the associated seek). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ACCUMULO-956) Iterator to transform key parts
[
https://issues.apache.org/jira/browse/ACCUMULO-956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13552981#comment-13552981
]
Keith Turner commented on ACCUMULO-956:
---
Code review notes for patch key-transforming-iterator_2.patch.
Notes for KeyTransformingIterator :
* Javadoc should outline why TransformingIter does vis filtering. Someone new
to this may wonder why this iterator is doing that, and why it can't be done by
a follow on iterator. Should also document that you need to configure the
untransformed auths on the scanner and the transformed auths on this iterator.
* javadoc for canSee() is wrong
* transformKeys() is only called when super.hasTop() is true, but it handles
the case when super.hasTop() is false. I was looking to see if this case could
cause a NPE, but then realized it would never occur. Not that anything needs to
be done about this, just something I observed
* For scans if the transformation creates an invalid colvis, this would be
caught. For compactions seems like it would not be caught. May consider
checking that colvis is valid for compactions, do not need to evaluate it, just
ensure it parses. Could use a seperate cache for this. If no auths are passed
for a scan, maybe the transformation is not checked?
* May need to transform column families for seek. Could provide a hook like
the following for the user. Seek would call this hook. There is a bug related
to column qualifiers fetched by the user that are not passed through seek, but
filtered at a lower level. Trying to find the ticket for this. Column
families can still be handled though.
{code:java}
//this function is called by seek to transform column families for seek to
parent iterator
protected CollectionByteSequence columnFamilies
transformColumnFamilies(CollectionByteSequence colFams){
return colFams;
}
{code}
Notes for unit test :
* Need a unit test that transforms to vis a user can see. I think it only
transforms to vis a user can not see.
* Need to test deep copy
* need to test compaction/not scanning
* Is there a test for a range thats before all data and a range thats after
all data?
* Need to test all combinations of range for the following conditions.
Start/end key exist inclusive/exclusive, start/end key not exist (between
existing keys) exclusive/inclusive. I think some of the case are tested, but
not all.
* Need to test seek that passes column families that are transformed.
All of the code looks really nice. For some reason isSetAfterPart() made me
smile.
Iterator to transform key parts
---
Key: ACCUMULO-956
URL: https://issues.apache.org/jira/browse/ACCUMULO-956
Project: Accumulo
Issue Type: Improvement
Reporter: Brian Loss
Fix For: 1.5.0
Attachments: key_transforming_iterator-1.patch,
key_transforming_iterator-2.patch, key_transforming_iterator.patch
Iterators that transform parts of the key can be tricky if any transformation
affects sort ordering. Implement an iterator that takes care of the tricky
details that come with modifying sort order (e.g., handling scan-time
iterator reconstruction and the associated seek).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ACCUMULO-956) Iterator to transform key parts
[ https://issues.apache.org/jira/browse/ACCUMULO-956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13552991#comment-13552991 ] Keith Turner commented on ACCUMULO-956: --- It may be necessary to filter based on column family. For example, if colfam C can be transformed to col fam A or B (depending on other parts of the key). If a seek came in for col fam A, then the soruce would be seeked for colfam C. However this could result in transformations to col fam B, which we would want to suppress. Iterator to transform key parts --- Key: ACCUMULO-956 URL: https://issues.apache.org/jira/browse/ACCUMULO-956 Project: Accumulo Issue Type: Improvement Reporter: Brian Loss Fix For: 1.5.0 Attachments: key_transforming_iterator-1.patch, key_transforming_iterator-2.patch, key_transforming_iterator.patch Iterators that transform parts of the key can be tricky if any transformation affects sort ordering. Implement an iterator that takes care of the tricky details that come with modifying sort order (e.g., handling scan-time iterator reconstruction and the associated seek). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ACCUMULO-956) Iterator to transform key parts
[
https://issues.apache.org/jira/browse/ACCUMULO-956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13551289#comment-13551289
]
William Slacum commented on ACCUMULO-956:
-
You may want to think about being able to make an inversion/reversal function
part of your interface, otherwise I don't believe you can have a function that
transforms arbitrary subsets of keys associated with a given prefix.
Say I have a sequence of keys for a row {{r}}:
{{r cf: 1 [v1]}}
{{r cf: 2 [v2]}}
{{r cf: 3 [v3]}}
{{r cf: 4 [v4]}}
{{r cf: 5 [v5]}}
Now let's say I want a transforming iterator that does aggregates the values
with odd and even qual
ifiers into a list, such that the sequence above generates two keys:
{{r cf: even [v2, v4]}}
{{r cf: odd [v1, v3, v5]}}
After I generate each entry and return them to the client, it's possible the
iterator stack gets torn down and rebuilt. If it happens after {{r cf: even
[v2, v4]}}, the new stack will be seeked with a range that has a non-inclusive
start key of {{r cf: even}}. That is after all of the data used to generate the
transformed set of keys, so a client would never see the key with the 'odd'
column qualifier.
It's also possible that this was never an intended use of the transforming
iterator, but it's something to keep in mind.
Iterator to transform key parts
---
Key: ACCUMULO-956
URL: https://issues.apache.org/jira/browse/ACCUMULO-956
Project: Accumulo
Issue Type: Improvement
Reporter: Brian Loss
Fix For: 1.5.0
Attachments: key_transforming_iterator.patch
Iterators that transform parts of the key can be tricky if any transformation
affects sort ordering. Implement an iterator that takes care of the tricky
details that come with modifying sort order (e.g., handling scan-time
iterator reconstruction and the associated seek).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ACCUMULO-956) Iterator to transform key parts
[ https://issues.apache.org/jira/browse/ACCUMULO-956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13551295#comment-13551295 ] Brian Loss commented on ACCUMULO-956: - I not sure there is a problem. The KeyTransformingIterator transforms only single keys, so you can't do any aggregation. If you had some kind of aggregating iterator on top of it on the stack, it would need to read enough keys from its source to do the aggregation. To produce your example, I think you'd have to scan all the cf: keys from the source to know it has what it needs for r cf: even [v2, v4]. Since that iterator would be scanning its source, there is no problem with the stack getting torn down and re-seeked. Or am I missing something? Iterator to transform key parts --- Key: ACCUMULO-956 URL: https://issues.apache.org/jira/browse/ACCUMULO-956 Project: Accumulo Issue Type: Improvement Reporter: Brian Loss Fix For: 1.5.0 Attachments: key_transforming_iterator.patch Iterators that transform parts of the key can be tricky if any transformation affects sort ordering. Implement an iterator that takes care of the tricky details that come with modifying sort order (e.g., handling scan-time iterator reconstruction and the associated seek). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
