----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/54020/ -----------------------------------------------------------
(Updated Nov. 28, 2016, 2:11 p.m.) Review request for samza, Jagadish Venkatraman and Yi Pan (Data Infrastructure). Changes ------- Updated with Jagadish's feedback. Inlined OperatorImpls#getOrCreateOperatorImpl() to remove dependency on k-v store Entry class. Summary (updated) ----------------- Operator API refactoring Repository: samza Description (updated) ------- Some suggestions for an Operator API refactor and misc. cleanup. It does contain some implementation changes, mostly due to deleted, extracted or merged classes. (e.g. OperatorFactory + ChainedOperators == OperatorImpls). Since git marked several moved classes as (delete + new) instead, it's probably best to apply the diff locally and browse the code in an IDE. Some of the changes, in no particular order: * Extracted XFunction interfaces into a .functions package in -api. * -api's internal.Operators is now the -operators's spec.* package. Extracted interfaces and classes. Factory methods are now in OperatorSpecs. * -api's MessageStreams is now -api's MessageStream interface and -operators's MessageStreamImpl. * -api's internal.Windows classes are now in -api's .window package. Extracted interfaces and classes, but no implementation changes. * OperatorFactory + ChainedOperators is now OperatorImpls, which is used from StreamOperatorAdaptorTask. * Added a NoOpOperatorImpl, which acts as the root node for the OperatorImpl DAG returned by OperatorImpls. * Removed usages of reactivestreams APIs since current code looks simpler without them. We can add them back when we need features like backpressure etc. * Removed the InputSystemMessage interface. * Made field names consistent (e.g Fn suffix for functions everywhere etc.). * Some method/class visibility changes due to moved classes. Haven't done this for all the classes yet. * General documentation changes, mostly to make public APIs clearer. Haven't done this for all the classes yet. There are additional questions/tasks that we can address in future RBs: Updating Window and Trigger APIs. Merging samza-operator into samza-core. Questions about Message timestamp and Offset comparison semantics. Questions about OperatorSpec serialization (e.g. ID generation). Questions about StateStoreImpl and StoreFunctions. Diffs (updated) ----- samza-api/src/main/java/org/apache/samza/operators/MessageStream.java dede631e072c2803839242dbf08b0149bfb76adb samza-api/src/main/java/org/apache/samza/operators/MessageStreams.java 51bf482d6ccc6720828bd13a128a4cce9d6204e1 samza-api/src/main/java/org/apache/samza/operators/StreamOperatorTask.java PRE-CREATION samza-api/src/main/java/org/apache/samza/operators/TriggerBuilder.java 5b3f4d05cd1aa5ceecb2dd285966f3366d69cf92 samza-api/src/main/java/org/apache/samza/operators/WindowState.java 759f2d8766804b2b5e47edf6759b3e6e2923c466 samza-api/src/main/java/org/apache/samza/operators/Windows.java 6619f41596548056d08fc71822eb94262ca2f120 samza-api/src/main/java/org/apache/samza/operators/data/IncomingSystemMessage.java 3c9874dcffeaa9300e5dbc984301cb495483b0c8 samza-api/src/main/java/org/apache/samza/operators/data/InputSystemMessage.java 5c23e746456a5e3058600a41ad7a91857839d61c samza-api/src/main/java/org/apache/samza/operators/data/Message.java 8441682a6446e9a10a173d1c5c8ae307a2fdd111 samza-api/src/main/java/org/apache/samza/operators/functions/FilterFunction.java PRE-CREATION samza-api/src/main/java/org/apache/samza/operators/functions/JoinFunction.java PRE-CREATION samza-api/src/main/java/org/apache/samza/operators/functions/SinkFunction.java PRE-CREATION samza-api/src/main/java/org/apache/samza/operators/internal/Operators.java f06387c9f49592a901ce01d996aa7ae7c95e89e2 samza-api/src/main/java/org/apache/samza/operators/internal/Trigger.java 3b50e2b8f640851a974ad01bc92425e50319692f samza-api/src/main/java/org/apache/samza/operators/internal/WindowFn.java 489e5b855b5de44b7ebf2c93b956ec50b7398ffb samza-api/src/main/java/org/apache/samza/operators/internal/WindowOutput.java 643b703648146f0bbae3f111099778b8c228d7b4 samza-api/src/main/java/org/apache/samza/operators/task/StreamOperatorTask.java 42c8f74544b5fd66af843ef7be2b38dd8e494253 samza-api/src/main/java/org/apache/samza/operators/windows/SessionWindow.java PRE-CREATION samza-api/src/main/java/org/apache/samza/operators/windows/StoreFunctions.java PRE-CREATION samza-api/src/main/java/org/apache/samza/operators/windows/Window.java PRE-CREATION samza-api/src/main/java/org/apache/samza/operators/windows/Windows.java PRE-CREATION samza-api/src/test/java/org/apache/samza/operators/TestMessage.java 8c5628757883168ce3f5a324884b967401b615fa samza-api/src/test/java/org/apache/samza/operators/TestMessageStream.java 4dbe233b5e3499c600efe67ba2ec6128afcda905 samza-api/src/test/java/org/apache/samza/operators/TestMessageStreams.java c5fcceba22afb816b2ea4d799d59683de0e58b2e samza-api/src/test/java/org/apache/samza/operators/TestOutputMessage.java 14e6562b66a26a88ce17b171f71eec600a4bd387 samza-api/src/test/java/org/apache/samza/operators/TestTriggerBuilder.java 927b14b56f8cd0cc2cae8f9801c026a84cce3c22 samza-api/src/test/java/org/apache/samza/operators/TestWindows.java 8a25a9655349aff83664bccd20704ea2bd03e662 samza-api/src/test/java/org/apache/samza/operators/data/TestIncomingSystemMessage.java b734e87ab0ae7cb8b1db2339ef7cdb18b27e2e01 samza-api/src/test/java/org/apache/samza/operators/data/TestLongOffset.java 943c47f58c6013f213c1761f5c08a3fae0570968 samza-api/src/test/java/org/apache/samza/operators/internal/TestOperators.java d9944860426b3e93de43e32744686c29775802c9 samza-api/src/test/java/org/apache/samza/operators/internal/TestTrigger.java 0f35a7cb4a451d6d2cd50dc804e4a8c162855d3c samza-api/src/test/java/org/apache/samza/operators/internal/TestWindowOutput.java 268c9fceea795b1681d27c53b2966ab8b08ba7f7 samza-operator/src/main/java/org/apache/samza/operators/MessageStreamImpl.java PRE-CREATION samza-operator/src/main/java/org/apache/samza/operators/StreamOperatorAdaptorTask.java PRE-CREATION samza-operator/src/main/java/org/apache/samza/operators/impl/ChainedOperators.java 1eee2dcb8b2b4b0ffab9981f4bfb5ec3e11ecc15 samza-operator/src/main/java/org/apache/samza/operators/impl/OperatorFactory.java ea90878173e81830743b47f2e6e12b61b0dc4dcb samza-operator/src/main/java/org/apache/samza/operators/impl/OperatorImpl.java efa6a9679fa7a15b3b5136e15a6c049793edc5d3 samza-operator/src/main/java/org/apache/samza/operators/impl/OperatorImpls.java PRE-CREATION samza-operator/src/main/java/org/apache/samza/operators/impl/ProcessorContext.java cc7ef2b97d9dbd50a0bf90d0e8af96c9bd1a12f7 samza-operator/src/main/java/org/apache/samza/operators/impl/SimpleOperatorImpl.java b0f4f27288d1798dbec05337ff2d04fafa20b2d5 samza-operator/src/main/java/org/apache/samza/operators/impl/SinkOperatorImpl.java a8a639e06dbb5e8319e55a83ecac4347e80cf3fc samza-operator/src/main/java/org/apache/samza/operators/impl/StateStoreImpl.java 7840b5bf11c8e8be58ec59fc59b8e70aa6ad909a samza-operator/src/main/java/org/apache/samza/operators/impl/join/PartialJoinOpImpl.java 4238d45df1f35dc04f4d7bff665be6eb2abd2537 samza-operator/src/main/java/org/apache/samza/operators/impl/window/SessionWindowImpl.java 0d6141e7acc7224cb7545b9a027729b09fc48489 samza-operator/src/main/java/org/apache/samza/operators/spec/OperatorSpecs.java PRE-CREATION samza-operator/src/main/java/org/apache/samza/operators/spec/PartialJoinOperatorSpec.java PRE-CREATION samza-operator/src/main/java/org/apache/samza/operators/spec/SinkOperatorSpec.java PRE-CREATION samza-operator/src/main/java/org/apache/samza/operators/spec/StreamOperatorSpec.java PRE-CREATION samza-operator/src/main/java/org/apache/samza/operators/spec/WindowOperatorSpec.java PRE-CREATION samza-operator/src/main/java/org/apache/samza/task/StreamOperatorAdaptorTask.java c2f780db3b8b2b21f8aa23bd849d5d347bda1265 samza-operator/src/test/java/org/apache/samza/operators/TestMessageStreamImpl.java PRE-CREATION samza-operator/src/test/java/org/apache/samza/operators/impl/TestChainedOperators.java e3a70e809a5022cd7e9f1622c20d8b05cede73c4 samza-operator/src/test/java/org/apache/samza/operators/impl/TestOperatorFactory.java cb4576cb888bb97803169dbb04c943e0acaf43df samza-operator/src/test/java/org/apache/samza/operators/impl/TestOperatorImpl.java 4bd467d54ecc1ceb6174bb46bd48c3d60a4cefa7 samza-operator/src/test/java/org/apache/samza/operators/impl/TestProcessorContext.java 224245e4ae0b83ac9c09e6cc6f9abf7cc4f43b0d samza-operator/src/test/java/org/apache/samza/operators/impl/TestSimpleOperatorImpl.java de029ea20b656d18df1d2c2b089fa3a91289d212 samza-operator/src/test/java/org/apache/samza/operators/impl/TestSinkOperatorImpl.java cdac3fc3ee47830c15449f9232c59ca1fc7f2bff samza-operator/src/test/java/org/apache/samza/operators/impl/TestStateStoreImpl.java 5ede75701ef4f77803dcbaa7f108626dbafed115 samza-operator/src/test/java/org/apache/samza/operators/impl/window/TestSessionWindowImpl.java 719ab99bfabe0d17c35ce6aefb3b3639e15cf956 samza-operator/src/test/java/org/apache/samza/task/BroadcastOperatorTask.java d1b0a8864e53f1371a7d45e1deef443412abd6e7 samza-operator/src/test/java/org/apache/samza/task/InputJsonSystemMessage.java 88aa159c6930fb76a0795944169d29a80c4747da samza-operator/src/test/java/org/apache/samza/task/JoinOperatorTask.java f6b3ff8ff2a5ead0995c8e2cd5d97d26d02a8ab0 samza-operator/src/test/java/org/apache/samza/task/TestStreamOperatorAdaptorTask.java 47d6b3a859172174b52a06ba3fcbe2f12c4fdfeb samza-operator/src/test/java/org/apache/samza/task/TestStreamOperatorTasks.java 44efa6dc8dd828c5c111dd1979e74a13702ffd91 samza-operator/src/test/java/org/apache/samza/task/WindowOperatorTask.java de7bba5a1745760ef80f060c8131a60dcc069871 Diff: https://reviews.apache.org/r/54020/diff/ Testing ------- ./gradlew clean build works. Thanks, Prateek Maheshwari
