[
https://issues.apache.org/jira/browse/APEXCORE-119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timothy Farkas reassigned APEXCORE-119:
---------------------------------------
Assignee: (was: Timothy Farkas)
> Add Support For A New Type Of (Distributed) Operator
> ----------------------------------------------------
>
> Key: APEXCORE-119
> URL: https://issues.apache.org/jira/browse/APEXCORE-119
> Project: Apache Apex Core
> Issue Type: Improvement
> Reporter: Timothy Farkas
>
> This JIRA Proposes support for a new type of distributed operator. Currently
> when an operator is partitioned there is no platform supported mechanism
> through which partitions can talk to each other. A Distributed operator would
> have an easy to use platform supported mechanism through which operators in a
> partitioning can exchange information with each other. Eventually Distributed
> operators would support running plain old single threaded java code
> transparently across partitions.
> In summary the goals would be to do the following:
> 1 - provide a platform supported fault tolerant mechanism through which
> operators in a partitioning can talk to each other.
> 2 - provide a platform supported way to run plain old single threaded java
> code accross all the partitions of a Distributed operator
> The benefits of implementing this would be huge:
> 1 - Using distributed operators we could support large in memory fault
> tolerant data structures (graphs, maps, arrays) in a fault tolerant way. Like
> Spark's RDD's but better.
> 2 - Plain old java code could be used to access and manipulate the data
> structures, without the user having the learn complex API's like with Spark.
> An implementation proposal and presentation are coming soon.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
