GitHub user steveblackmon opened a pull request:
https://github.com/apache/incubator-streams/pull/131
Streams 68,218
This is a big PR, because STREAMS-147/STREAMS-218 is tackling a big
problem: the proliferation of classes and code snippets that perform simple or
advanced conversion between typical document types String, ObjectNode,
Activity, and pojos generated from schemas matching provider data schemas.
The goal is to have platform-level utilities and processors that perform
these functions with a 'plug-in' architecture where individual modules can aid
those processors in conversion. Thus the responsibility of the module with
regard to conversion would be only to define schemas, classify documents that
align with those schemas, and provide classes the generic processors can use to
convert specific classified documents.
A major benefit of this approach are that the code footprint of each module
can be smaller - because generic type converters are available and easy to use,
components can set preconditions about what type of documents they will accept.
Stream pipelines will be able to use a less diverse set of processors to
accomplish the same work, so we can focus on making these utility components
solid and all modules and streams will benefit.
Additionally use cases involving in-bound queues of many types of documents
from multiple providers are now simple to implement, whereas without generic
converters those use cases would have been hard and ugly.
Look forward to your comments and seeing this approach or some variation
become the standard. I'm open to changes to the new interfaces or to the
converter resolution/instantiation lifecycle. Also please note that these
generic classes currently do not support a parallelism hint > 1, due to object
serialization errors. Still looking into this and will add tests to ensure
they can be parallelized.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/apache/incubator-streams STREAMS-68,218
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-streams/pull/131.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #131
----
commit aad0e88742b9156e6e11d1bf06d23133ec6ec961
Author: sblackmon <[email protected]>
Date: 2014-11-13T23:08:21Z
omni-bus update
primarily addresses STREAMS-218 (Generic Activity Serializer Processor)
addresses STREAMS-68 (ActivitySerializer should be ActivityConverter) - was
touching related classes anyway
some additional refactoring of class and method names is advisable but this
is a big chunk
resolves STREAMS-219 (.gitignore files generated by maven plugins)
needed to knock this out to avoid committing these files along with other
changes
commit 7a067ce133288fb79e41aed2fcc7ccd389ff1284
Author: sblackmon <[email protected]>
Date: 2014-11-13T23:35:41Z
fixed pig runtime test
commit 8737bdadfe595f06dd47b6652d538eec3e3d7f36
Author: sblackmon <[email protected]>
Date: 2014-11-14T00:15:44Z
assessed additional provider modules
either refactored for compatibility with streams-converters
and/or added comments/todos regarding deprecation etc...
commit 93026a7c2110df4ff441d2732f68085ebf933d53
Author: Steve Blackmon <[email protected]>
Date: 2014-11-15T18:51:11Z
improved error handling
serialize twitter type support classes
commit a24b3be723dbc7ccca7f34cc0f58ae8bb8838470
Author: Steve Blackmon <[email protected]>
Date: 2014-11-16T00:34:32Z
improved configuration scheme
commit df648a80d17c36220cba95a77331c1bbdcfd3532
Author: Steve Blackmon <[email protected]>
Date: 2014-11-16T00:55:54Z
Merge branch 'master' into STREAMS-68,218
Conflicts:
streams-contrib/streams-provider-facebook/src/main/java/org/apache/streams/facebook/api/FacebookPostActivityConverter.java
commit ce2a794dc9c79a28d3650377c58942fb95d82eb2
Author: Steve Blackmon <[email protected]>
Date: 2014-11-16T00:56:05Z
improved configuration scheme
commit 3b4135574735359ad819a1486fdd750387662f15
Author: Steve Blackmon <[email protected]>
Date: 2014-11-16T01:11:41Z
fixed mergeMerge branch 'master' into STREAMS-68,218
Conflicts:
streams-contrib/streams-provider-facebook/src/main/java/org/apache/streams/facebook/api/FacebookPostActivityConverter.java
commit b5cb7eb693be2fd3621c1b62c08f5548dee30dee
Author: Steve Blackmon <[email protected]>
Date: 2014-11-16T21:04:49Z
Added code comments
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
