Could this be an opportunity to abstract away the bucket/key relation? This is a rather cumbersome mechanism. For the user, having a primary and secondary key in this instance seems like it makes it more difficult to use the system easily.
Sent with Good (www.good.com) ________________________________ From: Timothy Farkas <[email protected]> Sent: Monday, April 4, 2016 8:40:00 PM To: [email protected] Subject: Spooled Data Structures Hi All, I'll be working on implementing Spooled Data structures. The overall proposal is the following: Overall design proposal: Spooled Data structures will be designed to work ontop of a key value store. Spooled data structures would be used as a component inside of an operator. The way the component would function is as followed: StateFactory: The StateFactory is an interface. The interface will have the setup, beginWindow, endWindow and teardown callbacks which an operator would have to call. The StateFactory is used to create handles to spooled DataStructures. The three types of spooled data structures it can create are Lists, Maps, and MultivaluedMaps Like this. StateFactory myFactory = new ManagedStateStateFactoryImpl(); List mySpooledList = myFactory.createNewList(byte[] keyPrefix, long bucket); Map mySpooledMap = myFactory.createNewMap(byte[] keyPrefix, long bucket); Map mySpooledMultiValuesMap = myFactory.createNewMultiValuedMap(byte[] keyPrefix, long bucket); The StateFactory can have specific implementations for different backing stores, and can be set as a property on operators. More complex spooled datastructures can simply be created from spooled lists and maps. More details are on the ticket https://issues.apache.org/jira/browse/APEXMALHAR-2026 Thanks, Tim ________________________________________________________ The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
