Sounds like a good idea. I think the same can be done for Flink; Flink's and Spark's APIs are similar to a large degree.
Here also a link to the transforms: https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/stream/operators/ -Max On 04.06.19 03:20, Ahmet Altay wrote:
Thank you for the feedback so far. It seems like this will be generally helpful :) I guess next step would be, would anyone be interested in working in this area? We can potentially break this down into starter tasks. On Sat, Jun 1, 2019 at 7:00 PM Ankur Goenka <goe...@google.com <mailto:goe...@google.com>> wrote: +1 for the proposal. Compatibility Matrix <https://beam.apache.org/documentation/runners/capability-matrix/> can be a good place to show case parity between different runners. +1 Do you think we should write 2 way examples [Spark, Flink, ..]<=>Beam? Both ways, would be most useful I believe. On Sat, Jun 1, 2019 at 4:31 PM Reza Rokni <r...@google.com <mailto:r...@google.com>> wrote: For layer 1, what about working through this link as a starting point : https://spark.apache.org/docs/latest/rdd-programming-guide.html#transformations? +1 On Sat, 1 Jun 2019 at 09:21, Ahmet Altay <al...@google.com <mailto:al...@google.com>> wrote: Thank you Reza. That separation makes sense to me. On Wed, May 29, 2019 at 6:26 PM Reza Rokni <r...@google.com <mailto:r...@google.com>> wrote: +1 I think there will be at least two layers of this; Layer 1 - Using primitives : I do join, GBK, Aggregation... with system x this way, what is the canonical equivalent in Beam. Layer 2 - Patterns : I read and join Unbounded and Bounded Data in system x this way, what is the canonical equivalent in Beam. I suspect as a first pass Layer 1 is reasonably well bounded work, there would need to be agreement on "canonical" version of how to do something in Beam as this could be seen to be opinionated. As there are often a multitude of ways of doing x.... Once we identify a set of layer 1 items, we could crowd source the canonical implementations. I believe we can use our usual code review process to settle on a version that is agreeable. (Examples have the same issue, they are probably opinionated today based on the author but it works out.) On Thu, 30 May 2019 at 08:56, Ahmet Altay <al...@google.com <mailto:al...@google.com>> wrote: Hi all, Inspired by the user asking about a Spark feature in Beam [1] in the release thread, I searched the user@ list and noticed a few instances of people asking for question like "I can do X in Spark, how can I do that in Beam?" Would it make sense to add documentation to explain how certain tasks that can be accomplished in Beam with side by side examples of doing the same task in Beam/Spark etc. It could help with on-boarding because it will be easier for people to leverage their existing knowledge. It could also help other frameworks as well, because it will serve as a Rosetta stone with two translations. Questions I have are: - Would such a thing be a helpful? - Is it feasible? Would a few pages worth of examples can cover enough use cases? Thank you! Ahmet [1] https://lists.apache.org/thread.html/b73a54aa1e6e9933628f177b04a8f907c26cac854745fa081c478eff@%3Cdev.beam.apache.org%3E -- This email may be confidential and privileged. If you received this communication by mistake, please don't forward it to anyone else, please erase all copies and attachments, and please let me know that it has gone to the wrong person. The above terms reflect a potential business arrangement, are provided solely as a basis for further discussion, and are not intended to be and do not constitute a legally binding obligation. No legally binding obligations will be created, implied, or inferred until an agreement in final form is executed in writing by all parties involved. -- This email may be confidential and privileged. If you received this communication by mistake, please don't forward it to anyone else, please erase all copies and attachments, and please let me know that it has gone to the wrong person. The above terms reflect a potential business arrangement, are provided solely as a basis for further discussion, and are not intended to be and do not constitute a legally binding obligation. No legally binding obligations will be created, implied, or inferred until an agreement in final form is executed in writing by all parties involved.
