> On May 7, 2015, 2:35 p.m., Milinda Pathirage wrote: > > samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateTimeWindowAutoOp.java, > > line 30 > > <https://reviews.apache.org/r/33749/diff/2/?file=948519#file948519line30> > > > > Hi Yi, What is automated operator in this context?
It is evolving now and will be removed. The original thought is trying to make two overlayed APIs on top of core set of window APIs: a) used by OperatorRouter to automatically execute the connected operators (i.e. the DAG for the query); b) a more low-level API that allows programmer to control when to add message and when to retrieve/flush results. After an internal team discussion, we are going to remove b). Instead, we will add support for user callback functions in the Operator to be invoked right before the process() is called and right before the result is collected and sent. > On May 7, 2015, 2:35 p.m., Milinda Pathirage wrote: > > samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateTimeWindowOp.java, > > line 76 > > <https://reviews.apache.org/r/33749/diff/2/?file=948520#file948520line76> > > > > When will this input disabling happens for a window operator? Refer to the design doc, the input to a window operator may be disabled if the downstream operator can not accept the output of the window operator (e.g. if the joiner needs to wait for the output from another stream to be available before accepting this window's output). It should not happen in normal case, but may occur if we optimize the message store recovery with lazy recovery or the two streams are skewed w/ large offsets that is larger than the retention size. - Yi ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33749/#review82824 ----------------------------------------------------------- On May 4, 2015, 6:58 a.m., Yi Pan (Data Infrastructure) wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/33749/ > ----------------------------------------------------------- > > (Updated May 4, 2015, 6:58 a.m.) > > > Review request for samza, Chris Riccomini, Guozhang Wang, Milinda Pathirage, > Navina Ramesh, and Naveen Somasundaram. > > > Bugs: SAMZA-650 > https://issues.apache.org/jira/browse/SAMZA-650 > > > Repository: samza > > > Description > ------- > > WIP: SAMZA-650 window store implementation > > First patch to implemement window store and message store. > > - Added window store initial implementation > - Added MessageStore initial implementation as well > - Completed skeleton APIs in window operators to illustrate the use cases of > window store and message store > - Added a StreamTask test case w/ a single window operator usage > - Unit tests are still WIP and will be updated later > > > Diffs > ----- > > samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Relation.java > PRE-CREATION > samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Stream.java > PRE-CREATION > samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Table.java > PRE-CREATION > samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Tuple.java > PRE-CREATION > > samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/Operator.java > PRE-CREATION > > samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/SqlOperatorFactory.java > PRE-CREATION > > samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/spec/OperatorSpec.java > PRE-CREATION > > samza-sql-core/src/main/java/org/apache/samza/sql/data/IncomingMessageTuple.java > PRE-CREATION > > samza-sql-core/src/main/java/org/apache/samza/sql/exception/OperatorException.java > PRE-CREATION > > samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperator.java > PRE-CREATION > > samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperatorFactoryImpl.java > PRE-CREATION > > samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperatorSpec.java > PRE-CREATION > > samza-sql-core/src/main/java/org/apache/samza/sql/operators/join/StreamStreamJoinSpec.java > PRE-CREATION > > samza-sql-core/src/main/java/org/apache/samza/sql/operators/join/StreamStreamJoiner.java > PRE-CREATION > > samza-sql-core/src/main/java/org/apache/samza/sql/operators/partition/PartitionSpec.java > PRE-CREATION > > samza-sql-core/src/main/java/org/apache/samza/sql/operators/relation/JoinSpec.java > PRE-CREATION > > samza-sql-core/src/main/java/org/apache/samza/sql/operators/stream/InsertStreamSpec.java > PRE-CREATION > > samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/BoundedTimeWindow.java > PRE-CREATION > > samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateTimeWindow.java > PRE-CREATION > > samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateTimeWindowAutoOp.java > PRE-CREATION > > samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateTimeWindowOp.java > PRE-CREATION > > samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateWindowOp.java > PRE-CREATION > > samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowOp.java > PRE-CREATION > > samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowOpSpec.java > PRE-CREATION > > samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowSpec.java > PRE-CREATION > > samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowState.java > PRE-CREATION > > samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/FilteredMessageIterator.java > PRE-CREATION > > samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/HashPrefixedMessageStore.java > PRE-CREATION > > samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/MessageStore.java > PRE-CREATION > > samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/MessageStoreSpec.java > PRE-CREATION > > samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/OffsetKey.java > PRE-CREATION > > samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/OrderedStoreKey.java > PRE-CREATION > > samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/PrefixedKey.java > PRE-CREATION > samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/Range.java > PRE-CREATION > > samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/TimeAndOffsetKey.java > PRE-CREATION > > samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/TimeKey.java > PRE-CREATION > > samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/WindowOutputStream.java > PRE-CREATION > > samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/WindowState.java > PRE-CREATION > > samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/WindowStore.java > PRE-CREATION > samza-sql-core/src/main/java/org/apache/samza/system/sql/LongOffset.java > PRE-CREATION > samza-sql-core/src/main/java/org/apache/samza/system/sql/Offset.java > PRE-CREATION > > samza-sql-core/src/main/java/org/apache/samza/task/sql/OperatorMessageCollector.java > PRE-CREATION > > samza-sql-core/src/main/java/org/apache/samza/task/sql/StoreMessageCollector.java > PRE-CREATION > > samza-sql-core/src/test/java/org/apache/samza/task/sql/RandomOperatorTask.java > PRE-CREATION > > samza-sql-core/src/test/java/org/apache/samza/task/sql/RandomWindowOperatorTask.java > PRE-CREATION > samza-sql-core/src/test/java/org/apache/samza/task/sql/StreamSqlTask.java > PRE-CREATION > > Diff: https://reviews.apache.org/r/33749/diff/ > > > Testing > ------- > > ./gradlew clean :samza-sql-core:build > > > Thanks, > > Yi Pan (Data Infrastructure) > >
