[
https://issues.apache.org/jira/browse/BEAM-10475?focusedWorklogId=499689&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-499689
]
ASF GitHub Bot logged work on BEAM-10475:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 12/Oct/20 22:29
Start Date: 12/Oct/20 22:29
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #13069:
URL: https://github.com/apache/beam/pull/13069#issuecomment-707376964
> > Regarding the yaml issues, I would like to see what the Python failures
were. If it's just that this coder is not implemented, perhaps simply
implementing it in Python as well would be the simplest.
>
> Yeah the failure is that the coder is not implemented. I have a separate
workspace for python. If it's okay to put python and java changes together I
can merge those changes to this PR.
Note that Go might also fail since it has some coder tests that rely on
standard_coder.yaml. One PR can be fine.
>
> > As for using the empty byte string vs. an explicit marker, I don't think
the extra byte in savings is significant. I think the bigger question is
whether logically we would want to allow the empty byte string as a shard key
(and if it would complicate code/the contract to explicitly avoid it). If it's
OK, it should be included as an example and test.
>
> Empty is different from null. We could 1) allow null value and add a
constraint on non-null shard id to be non-empty, or 2) not allow null value and
assume empty bytes to be non-existing/default shard id. Adding an indicator
would avoid the the constraint or assumption on the user side I think.
I think always being non-null will be easier for users and runner authors.
Having the default be "" is fine as it is expected that runners with their own
implementation may choose other values. Runners can also choose "" if they so
choose if they can differentiate internally as to whether something is re-using
an existing shard id or defining a new one. All the runner / user cares about
is that they are distinct.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 499689)
Time Spent: 6.5h (was: 6h 20m)
> GroupIntoBatches with Runner-determined Sharding
> ------------------------------------------------
>
> Key: BEAM-10475
> URL: https://issues.apache.org/jira/browse/BEAM-10475
> Project: Beam
> Issue Type: Improvement
> Components: runner-dataflow
> Reporter: Siyuan Chen
> Assignee: Siyuan Chen
> Priority: P2
> Labels: GCP, performance
> Time Spent: 6.5h
> Remaining Estimate: 0h
>
> [https://s.apache.org/sharded-group-into-batches|https://s.apache.org/sharded-group-into-batches__]
> Improve the existing Beam transform, GroupIntoBatches, to allow runners to
> choose different sharding strategies depending on how the data needs to be
> grouped. The goal is to help with the situation where the elements to process
> need to be co-located to reduce the overhead that would otherwise be incurred
> per element, while not losing the ability to scale the parallelism. The
> essential idea is to build a stateful DoFn with shardable states.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
