Re: [PROPOSAL] Design Doc template for PTransforms

[Robert Bradshaw via dev]

+1

I'd love for this information to be accessible programmatically as well
(both directions: extracting parameters from a transform and constructing a
transform from parameters). Making this pattern easy could encourage
compliance.

On Thu, Aug 24, 2023 at 8:54 AM Svetak Sundhar via dev 
wrote:

> Thanks Kenn! I put up a PR  to
> add this reference to the Beam website.
>
> I'm actively using the template now, and one piece of feedback I have is
> to include a section for prototyped work. Maybe a table that allows the
> user to post links to Github prototype PRs for the options they are
> proposing?
>
> Thanks,
>
>
> Svetak Sundhar
>
>   Data Engineer
> s vetaksund...@google.com
>
>
>
> On Thu, Aug 24, 2023 at 11:15 AM Kerry Donny-Clark via dev <
> dev@beam.apache.org> wrote:
>
>> Thanks Kenn! I think this would be a great community resource. While we
>> don't want to enforce usage, perhaps we could introduce tooling to check
>> basic compliance and raise a warning. Similar to a linter or test suite.
>> Kerry
>>
>> On Thu, Aug 24, 2023 at 10:25 AM Kenneth Knowles  wrote:
>>
>>> Hi all,
>>>
>>> Based on some work I've been doing internally, I put together a public
>>> version of a design doc template for PTransforms.
>>>
>>> https://s.apache.org/ptransform-design-doc
>>>
>>> A major goal is to be explicit about important questions that make a
>>> transform robust:
>>>
>>>  - what are "all" the parameters to a transform?
>>>  - how could a transform fail?
>>>  - how could we monitor or measure the transform?
>>>  - how could we use a transform in a new context like YAML or a new SDK?
>>>
>>> All of these together add up to a PTransform being a more self-contained
>>> piece of software that can be understood and used in novel ways, instead of
>>> just defined by the code and behavior that may accrete over time tightly
>>> coupled to the SDK it was written with.
>>>
>>> LMK what you think. Of course, I can't force anyone to use it or not use
>>> it, except for my team internal to my employer :-)
>>>
>>> Kenn
>>>
>>

Re: [PROPOSAL] Design Doc template for PTransforms

[Svetak Sundhar via dev]

Thanks Kenn! I put up a PR  to
add this reference to the Beam website.

I'm actively using the template now, and one piece of feedback I have is to
include a section for prototyped work. Maybe a table that allows the user
to post links to Github prototype PRs for the options they are proposing?

Thanks,


Svetak Sundhar

  Data Engineer
s vetaksund...@google.com



On Thu, Aug 24, 2023 at 11:15 AM Kerry Donny-Clark via dev <
dev@beam.apache.org> wrote:

> Thanks Kenn! I think this would be a great community resource. While we
> don't want to enforce usage, perhaps we could introduce tooling to check
> basic compliance and raise a warning. Similar to a linter or test suite.
> Kerry
>
> On Thu, Aug 24, 2023 at 10:25 AM Kenneth Knowles  wrote:
>
>> Hi all,
>>
>> Based on some work I've been doing internally, I put together a public
>> version of a design doc template for PTransforms.
>>
>> https://s.apache.org/ptransform-design-doc
>>
>> A major goal is to be explicit about important questions that make a
>> transform robust:
>>
>>  - what are "all" the parameters to a transform?
>>  - how could a transform fail?
>>  - how could we monitor or measure the transform?
>>  - how could we use a transform in a new context like YAML or a new SDK?
>>
>> All of these together add up to a PTransform being a more self-contained
>> piece of software that can be understood and used in novel ways, instead of
>> just defined by the code and behavior that may accrete over time tightly
>> coupled to the SDK it was written with.
>>
>> LMK what you think. Of course, I can't force anyone to use it or not use
>> it, except for my team internal to my employer :-)
>>
>> Kenn
>>
>

Re: [PROPOSAL] Design Doc template for PTransforms

[Kerry Donny-Clark via dev]

Thanks Kenn! I think this would be a great community resource. While we
don't want to enforce usage, perhaps we could introduce tooling to check
basic compliance and raise a warning. Similar to a linter or test suite.
Kerry

On Thu, Aug 24, 2023 at 10:25 AM Kenneth Knowles  wrote:

> Hi all,
>
> Based on some work I've been doing internally, I put together a public
> version of a design doc template for PTransforms.
>
> https://s.apache.org/ptransform-design-doc
>
> A major goal is to be explicit about important questions that make a
> transform robust:
>
>  - what are "all" the parameters to a transform?
>  - how could a transform fail?
>  - how could we monitor or measure the transform?
>  - how could we use a transform in a new context like YAML or a new SDK?
>
> All of these together add up to a PTransform being a more self-contained
> piece of software that can be understood and used in novel ways, instead of
> just defined by the code and behavior that may accrete over time tightly
> coupled to the SDK it was written with.
>
> LMK what you think. Of course, I can't force anyone to use it or not use
> it, except for my team internal to my employer :-)
>
> Kenn
>

[PROPOSAL] Design Doc template for PTransforms

[Kenneth Knowles]

Hi all,

Based on some work I've been doing internally, I put together a public
version of a design doc template for PTransforms.

https://s.apache.org/ptransform-design-doc

A major goal is to be explicit about important questions that make a
transform robust:

 - what are "all" the parameters to a transform?
 - how could a transform fail?
 - how could we monitor or measure the transform?
 - how could we use a transform in a new context like YAML or a new SDK?

All of these together add up to a PTransform being a more self-contained
piece of software that can be understood and used in novel ways, instead of
just defined by the code and behavior that may accrete over time tightly
coupled to the SDK it was written with.

LMK what you think. Of course, I can't force anyone to use it or not use
it, except for my team internal to my employer :-)

Kenn