[jira] [Commented] (METRON-2157) To better abstract the streaming infrastructure, use Apache Beam.
[ https://issues.apache.org/jira/browse/METRON-2157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16864124#comment-16864124 ] Simon Elliston Ball commented on METRON-2157: - My one worry about it would be that it's a potentially more complex abstraction to tune, given the metron focus on high throughput. The beam programming model might fit our current use cases, but is a least common denominator. That said there has been a lot of work recently to abstract storm out, so beam could be a viable engine to run our business logic on. The challenge would then be to figure out the optimal runtime underneath that. I suspect this would benefit from a discuss thread on the mail lists. > To better abstract the streaming infrastructure, use Apache Beam. > - > > Key: METRON-2157 > URL: https://issues.apache.org/jira/browse/METRON-2157 > Project: Metron > Issue Type: Improvement >Reporter: Phillip Henry >Priority: Major > > Apache Beam is "[a]n advanced unified programming model" that can "Implement > batch and streaming data processing jobs that run on any execution engine" > (according to their [docs|[https://beam.apache.org/]). > If Metron were to use such an abstraction, there could be greater portability > between different offerings of streaming infrastructure (Spark, Flink etc). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-2157) To better abstract the streaming infrastructure, use Apache Beam.
[ https://issues.apache.org/jira/browse/METRON-2157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16864110#comment-16864110 ] Otto Fowler commented on METRON-2157: - So, the idea to investigate here, and likely have a [DISCUSS] thread about, is that instead of just abstracting the storm support away from the metron components, to allow for implementations in parallel to storm ( like spark structured streaming ), metron moves to beam as a single model that allows execution on these supported modules? > To better abstract the streaming infrastructure, use Apache Beam. > - > > Key: METRON-2157 > URL: https://issues.apache.org/jira/browse/METRON-2157 > Project: Metron > Issue Type: Improvement >Reporter: Phillip Henry >Priority: Major > > Apache Beam is "[a]n advanced unified programming model" that can "Implement > batch and streaming data processing jobs that run on any execution engine" > (according to their [docs|[https://beam.apache.org/]). > If Metron were to use such an abstraction, there could be greater portability > between different offerings of streaming infrastructure (Spark, Flink etc). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (METRON-2157) To better abstract the streaming infrastructure, use Apache Beam.
Phillip Henry created METRON-2157: - Summary: To better abstract the streaming infrastructure, use Apache Beam. Key: METRON-2157 URL: https://issues.apache.org/jira/browse/METRON-2157 Project: Metron Issue Type: Improvement Reporter: Phillip Henry Apache Beam is "[a]n advanced unified programming model" that can "Implement batch and streaming data processing jobs that run on any execution engine" (according to their [docs|[https://beam.apache.org/]). If Metron were to use such an abstraction, there could be greater portability between different offerings of streaming infrastructure (Spark, Flink etc). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
