It's fair. if we change the default value, we can perhaps add an error
handling logic so that (pcoll) | beam.Flatten() fails with an error that
recommends (pcoll) | beam.FlatMap(), instead of saying that input is not
an iterable.
On Thu, Mar 21, 2024 at 3:41 PM Joey Tran wrote:
> +1
>
> On Thu,
+1
On Thu, Mar 21, 2024 at 6:30 PM Robert Bradshaw via dev
wrote:
> I would be more comfortable with a default for FlatMap than overloading
> Flatten in this way. Distinguishing between
>
> (pcoll,) | beam.Flatten()
>
> and
>
> (pcoll) | beam.Flatten()
>
> seems a bit error prone.
>
>
>
I would be more comfortable with a default for FlatMap than overloading
Flatten in this way. Distinguishing between
(pcoll,) | beam.Flatten()
and
(pcoll) | beam.Flatten()
seems a bit error prone.
On Thu, Mar 21, 2024 at 2:23 PM Joey Tran wrote:
> Ah, I misunderstood your original
Ah, I misunderstood your original suggestion then. That makes sense then. I
have already seen someone get a little confused about the names and
surprised that Flatten doesn't do what FlatMap does.
On Thu, Mar 21, 2024 at 5:20 PM Valentyn Tymofieiev
wrote:
> Beam throws an error at submission
Beam throws an error at submission time in Python if you pass a single
PCollection to Flatten. The scenario you describe concerns a one-element
list.
On Thu, Mar 21, 2024, 13:43 Joey Tran wrote:
> I think it'd be quite surprising if beam.Flatten would become equivalent
> to FlatMap if passed
I think it'd be quite surprising if beam.Flatten would become equivalent to
FlatMap if passed only a single pcollection. One use case that would be
broken from that is cases where someone might be flattening a variable
number of pcollections, including possibly only one pcollection. In that
case,
One possible alternative is to define beam.Flatten for a single collection
to be functionally equivalent to beam.FlatMap(lambda x: x), but that would
be a larger change and such behavior might need to be consistent across
SDKs and documented. Adding a default value is a simpler change.
I can also
IIRC, Java has Flatten.iterables() and Flatten.collections(), the first of
which does what you want.
Giving FlatMap a default arg of lambda x: x is an interesting idea. The
only downside I see is a less clear error if one forgets to provide this
(now mandatory) parameter, but maybe that's low
That's not really the same thing, is it? `beam.Flatten` combines two or
more pcollections into a single pcollection while beam.FlatMap unpacks
iterables of elements (i.e. PCollection> -> PCollection)
On Thu, Mar 21, 2024 at 2:57 PM Valentyn Tymofieiev via dev <
dev@beam.apache.org> wrote:
> Hi,
Actually, disregard that, Flatten is used in a different context to flatten
multiple collections.
On Thu, Mar 21, 2024 at 11:55 AM Valentyn Tymofieiev
wrote:
> Hi, you can use beam.Flatten() instead.
>
> On Thu, Mar 21, 2024 at 10:55 AM Joey Tran
> wrote:
>
>> Hey all,
>>
>> Using an identity
Hi, you can use beam.Flatten() instead.
On Thu, Mar 21, 2024 at 10:55 AM Joey Tran
wrote:
> Hey all,
>
> Using an identity function for FlatMap comes up more often than using
> FlatMap without an identity function. Would it make sense to use the
> identity function as a default?
>
>
>
>
Hey all,
Using an identity function for FlatMap comes up more often than using
FlatMap without an identity function. Would it make sense to use the
identity function as a default?
+1 - validated some ML examples with the interactive runner
Thanks,
Danny
On Thu, Mar 21, 2024 at 9:21 AM Jan Lukavský wrote:
> +1 (binding)
>
> Tested Java SDK with FlinkRunner.
>
> Jan
> On 3/20/24 22:40, Chamikara Jayalath via dev wrote:
>
> +1 (binding)
>
> Tested multi-lang Java/Python
+1 (binding)
Tested Java SDK with FlinkRunner.
Jan
On 3/20/24 22:40, Chamikara Jayalath via dev wrote:
+1 (binding)
Tested multi-lang Java/Python pipelines and upgrading BQ/Kafka
transforms from 2.53.0 to 2.55.0 using the Transform Service.
Thanks,
Cham
On Tue, Mar 19, 2024 at 2:10 PM
This is your daily summary of Beam's current high priority issues that may need
attention.
See https://beam.apache.org/contribute/issue-priorities for the meaning and
expectations around issue priorities.
Unassigned P1 Issues:
https://github.com/apache/beam/issues/30683 The PreCommit Java
15 matches
Mail list logo
