That's great news, thanks Reuven! I will try this out soon.
On Sat, Jul 29, 2017 at 2:33 AM, Reuven Lax
wrote:
> The AvroIO PR is now merged, so you can write to different destinations
> based on the value. It's available in head, and will be in Beam 2.2.0.
>
> On Wed, Jul 26, 2017 at 10:00 AM,
The AvroIO PR is now merged, so you can write to different destinations
based on the value. It's available in head, and will be in Beam 2.2.0.
On Wed, Jul 26, 2017 at 10:00 AM, Reuven Lax wrote:
> Yes, there was! TextIO support is already merged into Beam (it missed the
> 2.1 cutoff, so it will
Yes, there was! TextIO support is already merged into Beam (it missed the
2.1 cutoff, so it will be in Beam 2.2.0). AvroIO support is in
https://github.com/apache/beam/pull/3541. This is almost ready to merge -
still waiting for final review from kennknowles on the Beam translation
changes.
Nobody
Hi all,
Was there any progress on this recently? I am particularly interested in
using value-dependent destinations in BigtableIO (writing to a specific
table depending on the value) and AvroIO (writing to specific GCS buckets
depending on the value).
Thanks,
Josh
On Fri, Jun 9, 2017 at 5:35 PM,
I'm putting together a proof-of-concept PR for option 1 to see how it
looks.
On Thu, Jun 8, 2017 at 4:07 PM, Reuven Lax wrote:
> After looking at everyone's comments, I think option 1 is the better
> approach - map destinations to a FilenamePolicy. It is a good parallel to
> what we do in BigQue
After looking at everyone's comments, I think option 1 is the better
approach - map destinations to a FilenamePolicy. It is a good parallel to
what we do in BigQueryIO (the main difference is that we're mapping to a
sharded filename, instead of a single destination like in BigQueryIO).
The main li
Hmm, on one hand this looks syntactically very appealing, on the other
hand, it's icky to have a function return a PTransform at runtime, only to
have some information be immediately extracted from that transform.
Moreover, not all TextIO.Write transforms will be legal to return - e.g.
most likely
Did you see that I modified the second proposal so that users can map
DestinationT to the actual PTransform (i.e. DestinationT->TextIO or
DestinationT->AvroIO). This means that users do not have to deal with
FileBasedSink or even know it exists.
I prefer the second approach for two reason:
1. It
I commented a little in the doc I want to reply on list because this is a
really great feature.
The two alternatives, as I understand them, both include mapping your
elements to an intermediate DestinationT that you can group by before
writing. Then the big picture decision is whether to map each
While Beam now supports file-based sinks that can depend on the current
window, I've seen interest in value-dependent sinks as well (and there's a
long-standing JIRA for this). I wrote up a short API proposal for this for
discussion on the list.
https://docs.google.com/document/d/1Bd9mJO1YC8vOoFOb
10 matches
Mail list logo