[
https://issues.apache.org/jira/browse/SAMOA-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16040708#comment-16040708
]
ASF GitHub Bot commented on SAMOA-65:
-------------------------------------
GitHub user pwawrzyniak opened a pull request:
https://github.com/apache/incubator-samoa/pull/64
SAMOA-65 JSON serializer/deserializer dedicated for Kafka components
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/pwawrzyniak/incubator-samoa SAMOA-65-JSON
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-samoa/pull/64.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #64
----
commit 0e90fe8c107cd90a7baf16b5d4f334d1650a0afb
Author: pwawrzyniak <[email protected]>
Date: 2017-03-14T16:43:25Z
Initial structure of Kafka components.
Initital code for Kafka Consumer
commit 2168400f86189605b06fb531511235733bf3b6ca
Author: pwawrzyniak <[email protected]>
Date: 2017-03-17T10:05:14Z
Code for KafkaEntranceProcessor (consuming messages from Kafka)
commit cf2ff369989fa3560f1570dfaf32bcb746d51c25
Author: pwawrzyniak <[email protected]>
Date: 2017-03-17T10:54:47Z
Updated comments
commit 26d86da6ebdd53e440e8b1e65f5ba5ed9f5f609e
Author: pwawrzyniak <[email protected]>
Date: 2017-03-17T11:09:52Z
Sample serializer/deserializer for JSON and InstanceContentEvent
Updates in comments
commit 45b3546f9c525af385f646447a09d3683b70937d
Author: pwawrzyniak <[email protected]>
Date: 2017-03-17T14:40:25Z
KafkaDestinationProcessor implementation (sending msg to Kafka
commit 2e6d502a2909731cb9859ba3403407e6a33a2fc2
Author: pwawrzyniak <[email protected]>
Date: 2017-03-24T13:34:49Z
Tests for kafkaUtils and KafkaEntranceProcessor, minor changes in classes
commit 208febc0524ddc328ae1a785f0642650173f7101
Author: pwawrzyniak <[email protected]>
Date: 2017-04-11T14:44:30Z
Tests for KafkaDestinationProcessor, minor changes in classes
commit ff6b84dc5403c102388646a31f120c11ee416063
Author: Jakub Jankowski <[email protected]>
Date: 2017-04-28T15:08:29Z
Added Kafka Avro serializer with unit tests. Added Kafka task with unit test
commit 7997fd538c258e8844cbd1a81d4c6c43d8788aa4
Author: Jakub Jankowski <[email protected]>
Date: 2017-05-01T09:23:00Z
Merge branch 'SAMOA-65' of https://github.com/pwawrzyniak/incubator-samoa
into SAMOA-65
commit ae7889fce1f52afdaa6d7e6080cff54dd1b01b9b
Author: Jakub Jankowski <[email protected]>
Date: 2017-05-10T12:37:25Z
Fixing avro deserialization
commit 107c4c5f72fcacc38d02bfc7cd1552a1593cdd5a
Author: Jakub Jankowski <[email protected]>
Date: 2017-05-10T12:45:16Z
Fixing avro deserialization
commit cde0642ccc768f7492e8c5811b2f1232e58acec4
Author: pwawrzyniak <[email protected]>
Date: 2017-05-12T15:17:27Z
Rebuild of Consumer-related classes (added separate thread to read from
Kafka, blocking until sth received)
Major cleanup in tests
commit f20301a4c64c9487fdbcff17829b6a8018e9a00b
Author: pwawrzyniak <[email protected]>
Date: 2017-05-12T15:22:54Z
Minor change in tests
commit b4ee754c54ac5be51f9846704947e7c19c186e73
Author: Jakub Jankowski <[email protected]>
Date: 2017-05-16T12:11:36Z
Fixing avro deserializer (using ReflectDatumReader) to be able to read
arrays in InstanceData objects
commit d636ba90851d8424a6d3e651d80c87d0a8d7394e
Author: pwawrzyniak <[email protected]>
Date: 2017-05-16T13:00:33Z
Changes in JSON mapper
commit 57f80a5bf7d83c31d517a7d98741dd7457a4c84c
Author: pwawrzyniak <[email protected]>
Date: 2017-06-07T10:21:08Z
repo cleanup, split code into 3 branches
commit 04a28bb5ecbc039f07c17a91d57859c4651f56d9
Author: pwawrzyniak <[email protected]>
Date: 2017-06-07T10:50:15Z
JSON mapper with test
----
> Apache Kafka integration components for SAMOA
> ---------------------------------------------
>
> Key: SAMOA-65
> URL: https://issues.apache.org/jira/browse/SAMOA-65
> Project: SAMOA
> Issue Type: New Feature
> Components: SAMOA-API, SAMOA-Instances
> Reporter: Piotr Wawrzyniak
> Labels: kafka, sink, source, streaming
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> As of now Apache SAMOA includes no integration components for Apache Kafka,
> meaning in particular no possibility to read data coming from Kafka and write
> data with prediction results back to Kafka.
> The key assumptions for the development of Kafka-related components are as
> follows:
> 1) develop support for input data stream arriving to Apache Samoa via
> Apache Kafka
> 2) develop support for output data stream produced by Apache Samoa,
> including the results of stream mining and forwarded to Apache Kafka to be
> provided in this way to other modules consuming the stream.
> This makes the goal of this issue is to create the following components:
> 1) KafkaEntranceProcessor in samoa-api. This entrance processor will be
> able to accept incoming Kafka stream. It will require KafkaDeserializer
> interface implementation to be delivered. The role of Deserializer would be
> to translate incoming Apache Kafka messages into implementation of Instance
> interface of SAMOA.
> 2) KafkaDestinationProcessor in samoa-api. Similarly to the
> KafkaEntranceProcessor, this processor would require KafkaSerializer
> interface implementation to be delivered. The role of Serializer would be to
> create a Kafka message from the underlying Instance class.
> 3) KafkaStream, as the extension to existing streams (e.g.
> InstanceStream), would take similar role to other streams, and will provide
> the control over Instances flows in the entire topology.
> Moreover, the following assumptions are considered:
> 1) Components would be implemented with the use of most up-to-date version
> of Apache Kafka, i.e. 0.10
> 2) Samples of aforementioned Serializer and Deserializer would be
> delivered, both supporting AVRO and JSON serialization of Instance objects.
> 3) Sample testing classes providing reference use of Kafka source and
> destination would be included in the project as well.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
