[
https://issues.apache.org/jira/browse/BEAM-10475?focusedWorklogId=499846&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-499846
]
ASF GitHub Bot logged work on BEAM-10475:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 13/Oct/20 06:17
Start Date: 13/Oct/20 06:17
Worklog Time Spent: 10m
Work Description: nehsyc commented on pull request #13069:
URL: https://github.com/apache/beam/pull/13069#issuecomment-707515750
> > > > Regarding the yaml issues, I would like to see what the Python
failures were. If it's just that this coder is not implemented, perhaps simply
implementing it in Python as well would be the simplest.
> > >
> > >
> > > Yeah the failure is that the coder is not implemented. I have a
separate workspace for python. If it's okay to put python and java changes
together I can merge those changes to this PR.
> >
> >
> > Note that Go might also fail since it has some coder tests that rely on
standard_coder.yaml. One PR can be fine.
Thanks for pointing it out. The standard coder test in Go now skips
ShardedKey coder test. Also added Python coder in this PR. (Sorry for the mess
of commits and force pushes. The last commit was for formatting.)
> > > > As for using the empty byte string vs. an explicit marker, I don't
think the extra byte in savings is significant. I think the bigger question is
whether logically we would want to allow the empty byte string as a shard key
(and if it would complicate code/the contract to explicitly avoid it). If it's
OK, it should be included as an example and test.
> > >
> > >
> > > Empty is different from null. We could 1) allow null value and add a
constraint on non-null shard id to be non-empty, or 2) not allow null value and
assume empty bytes to be non-existing/default shard id. Adding an indicator
would avoid the the constraint or assumption on the user side I think.
> >
> >
> > I think always being non-null will be easier for users and runner
authors. Having the default be "" is fine as it is expected that runners with
their own implementation may choose other values. Runners can also choose "" if
they so choose if they can differentiate internally as to whether something is
re-using an existing shard id or defining a new one. All the runner / user
cares about is that they are distinct.
>
> Sounds reasonable. It wouldn't add complexity in code to avoid shard id
being empty explicitly, unless the runner/user really wants to use empty bytes
as a meaningful shard id in which case they need to be responsible for
differentiating set/unset shard ids.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 499846)
Time Spent: 7h 50m (was: 7h 40m)
> GroupIntoBatches with Runner-determined Sharding
> ------------------------------------------------
>
> Key: BEAM-10475
> URL: https://issues.apache.org/jira/browse/BEAM-10475
> Project: Beam
> Issue Type: Improvement
> Components: runner-dataflow
> Reporter: Siyuan Chen
> Assignee: Siyuan Chen
> Priority: P2
> Labels: GCP, performance
> Time Spent: 7h 50m
> Remaining Estimate: 0h
>
> [https://s.apache.org/sharded-group-into-batches|https://s.apache.org/sharded-group-into-batches__]
> Improve the existing Beam transform, GroupIntoBatches, to allow runners to
> choose different sharding strategies depending on how the data needs to be
> grouped. The goal is to help with the situation where the elements to process
> need to be co-located to reduce the overhead that would otherwise be incurred
> per element, while not losing the ability to scale the parallelism. The
> essential idea is to build a stateful DoFn with shardable states.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
