Hi Ilya, After the Map or List is created it will work like a normal in memory java List or Map. We can simplify instantiating the object further though by automatically generating the Key Prefixes in the StateFactory object and only requiring the bucket to be specified once on the StateFactory like this:
StateFactory myFactory = new ManagedStateStateFactoryImpl(long bucket); List mySpooledList = myFactory.createNewList(); Map mySpooledMap = myFactory.createNewMap(); Map mySpooledMultiValuedMap = myFactory.createNewMultiValuedMap(); I don't think we can completely eliminate the bucket since it's required for partitioning. Thanks, Tim On Mon, Apr 4, 2016 at 9:11 PM, Ganelin, Ilya <[email protected]> wrote: > Could this be an opportunity to abstract away the bucket/key relation? > This is a rather cumbersome mechanism. For the user, having a primary and > secondary key in this instance seems like it makes it more difficult to use > the system easily. > > > > Sent with Good (www.good.com) > ________________________________ > From: Timothy Farkas <[email protected]> > Sent: Monday, April 4, 2016 8:40:00 PM > To: [email protected] > Subject: Spooled Data Structures > > Hi All, > > I'll be working on implementing Spooled Data structures. The overall > proposal is the following: > > Overall design proposal: > > Spooled Data structures will be designed to work ontop of a key value > store. Spooled data structures would be used as a component inside of an > operator. The way the component would function is as followed: > > StateFactory: > > The StateFactory is an interface. The interface will have the setup, > beginWindow, endWindow and teardown callbacks which an operator would have > to call. The StateFactory is used to create handles to spooled > DataStructures. The three types of spooled data structures it can create > are Lists, Maps, and MultivaluedMaps Like this. > > StateFactory myFactory = new ManagedStateStateFactoryImpl(); > List mySpooledList = myFactory.createNewList(byte[] keyPrefix, long > bucket); > Map mySpooledMap = myFactory.createNewMap(byte[] keyPrefix, long bucket); > Map mySpooledMultiValuesMap = myFactory.createNewMultiValuedMap(byte[] > keyPrefix, long bucket); > > The StateFactory can have specific implementations for different backing > stores, and can be set as a property on operators. More complex spooled > datastructures can simply be created from spooled lists and maps. > > > More details are on the ticket > https://issues.apache.org/jira/browse/APEXMALHAR-2026 > > > Thanks, > Tim > ________________________________________________________ > > The information contained in this e-mail is confidential and/or > proprietary to Capital One and/or its affiliates and may only be used > solely in performance of work or services for Capital One. The information > transmitted herewith is intended only for use by the individual or entity > to which it is addressed. If the reader of this message is not the intended > recipient, you are hereby notified that any review, retransmission, > dissemination, distribution, copying or other use of, or taking of any > action in reliance upon this information is strictly prohibited. If you > have received this communication in error, please contact the sender and > delete the material from your computer. >
