+1 for the proposal. Compatibility Matrix <https://beam.apache.org/documentation/runners/capability-matrix/> can be a good place to show case parity between different runners. Do you think we should write 2 way examples [Spark, Flink, ..]<=>Beam?
On Sat, Jun 1, 2019 at 4:31 PM Reza Rokni <r...@google.com> wrote: > For layer 1, what about working through this link as a starting point : > https://spark.apache.org/docs/latest/rdd-programming-guide.html#transformations > ? > > On Sat, 1 Jun 2019 at 09:21, Ahmet Altay <al...@google.com> wrote: > >> Thank you Reza. That separation makes sense to me. >> >> On Wed, May 29, 2019 at 6:26 PM Reza Rokni <r...@google.com> wrote: >> >>> +1 >>> >>> I think there will be at least two layers of this; >>> >>> Layer 1 - Using primitives : I do join, GBK, Aggregation... with system >>> x this way, what is the canonical equivalent in Beam. >>> Layer 2 - Patterns : I read and join Unbounded and Bounded Data in >>> system x this way, what is the canonical equivalent in Beam. >>> >>> I suspect as a first pass Layer 1 is reasonably well bounded work, there >>> would need to be agreement on "canonical" version of how to do something in >>> Beam as this could be seen to be opinionated. As there are often a >>> multitude of ways of doing x.... >>> >> >> Once we identify a set of layer 1 items, we could crowd source the >> canonical implementations. I believe we can use our usual code review >> process to settle on a version that is agreeable. (Examples have the same >> issue, they are probably opinionated today based on the author but it works >> out.) >> >> >>> >>> >>> On Thu, 30 May 2019 at 08:56, Ahmet Altay <al...@google.com> wrote: >>> >>>> Hi all, >>>> >>>> Inspired by the user asking about a Spark feature in Beam [1] in the >>>> release thread, I searched the user@ list and noticed a few instances >>>> of people asking for question like "I can do X in Spark, how can I do that >>>> in Beam?" Would it make sense to add documentation to explain how certain >>>> tasks that can be accomplished in Beam with side by side examples of doing >>>> the same task in Beam/Spark etc. It could help with on-boarding because it >>>> will be easier for people to leverage their existing knowledge. It could >>>> also help other frameworks as well, because it will serve as a Rosetta >>>> stone with two translations. >>>> >>>> Questions I have are: >>>> - Would such a thing be a helpful? >>>> - Is it feasible? Would a few pages worth of examples can cover enough >>>> use cases? >>>> >>>> Thank you! >>>> Ahmet >>>> >>>> [1] >>>> https://lists.apache.org/thread.html/b73a54aa1e6e9933628f177b04a8f907c26cac854745fa081c478eff@%3Cdev.beam.apache.org%3E >>>> >>> >>> >>> -- >>> >>> This email may be confidential and privileged. If you received this >>> communication by mistake, please don't forward it to anyone else, please >>> erase all copies and attachments, and please let me know that it has gone >>> to the wrong person. >>> >>> The above terms reflect a potential business arrangement, are provided >>> solely as a basis for further discussion, and are not intended to be and do >>> not constitute a legally binding obligation. No legally binding obligations >>> will be created, implied, or inferred until an agreement in final form is >>> executed in writing by all parties involved. >>> >> > > -- > > This email may be confidential and privileged. If you received this > communication by mistake, please don't forward it to anyone else, please > erase all copies and attachments, and please let me know that it has gone > to the wrong person. > > The above terms reflect a potential business arrangement, are provided > solely as a basis for further discussion, and are not intended to be and do > not constitute a legally binding obligation. No legally binding obligations > will be created, implied, or inferred until an agreement in final form is > executed in writing by all parties involved. >
