Re: Hello Beam Community!

[Reza Rokni]

Welcome!

On Sat, 14 Mar 2020, 01:27 Tomo Suzuki,  wrote:

> Welcome.
>
> On Fri, Mar 13, 2020 at 1:20 PM Udi Meiri  wrote:
>
>> Welcome!
>>
>>
>> On Fri, Mar 13, 2020 at 9:47 AM Yichi Zhang  wrote:
>>
>>> Welcome!
>>>
>>> On Fri, Mar 13, 2020 at 9:40 AM Ahmet Altay  wrote:
>>>
 Welcome Brittany!

 On Thu, Mar 12, 2020 at 6:32 PM Brittany Hermann 
 wrote:

> Hello Beam Community!
>
> My name is Brittany Hermann and I recently joined the Open Source team
> in Data Analytics at Google. As a Program Manager, I will be focusing on
> community engagement while getting to work on Apache Beam and Airflow
> projects! I have always thrived on creating healthy, diverse, and overall
> happy communities and am excited to bring that to the team. For a fun 
> fact,
> I am a big Wisconsin Badgers Football fan and have a goldendoodle puppy
> named Ollie!
>
> I look forward to collaborating with you all!
>
> Kind regards,
>
> Brittany Hermann
>
>
>
>
> --
> Regards,
> Tomo
>

Re: Letting workers fetch AWS keys for Kinesis

[Luke Cwik]

Streaming pipelines on Dataflow have a limitation where all initial
splitting[1] of the UnboundedSource is done upfront at pipeline creation
time. This has been a 5+ year issue.

The recent development in this space has been to use Splittable DoFns that
wrap UnboundedSources[2] and this would move the initial splitting to
happen at pipeline execution time.

You could try it out but be warned that this was checked in a week ago and
there are no major users of it to my knowledge. The progress in this space
has been steady for the past 9 months and might complete in 3-4 months for
Dataflow. If you have the time to help out, that estimate could be brought
in.

1:
https://github.com/apache/beam/blob/873f689f107ee2a5d10edb7e8cb87d7996d40eea/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/internal/CustomSources.java#L62
2:
https://github.com/apache/beam/blob/eb59dde36f74dba58d7fbd992cb741a811ba9a58/sdks/java/core/src/main/java/org/apache/beam/sdk/io/Read.java#L436

On Fri, Mar 13, 2020 at 1:30 PM Julien Phalip  wrote:

> Hi,
>
> We're running a Beam job in Dataflow that reads inputs from KinesisIO.
> Right now it works if we pass the AWS keys to the job like so:
>
> p.apply(KinesisIO.read()
> // ...
>.withAWSClientsProvider("AWS_KEY",  "AWS_SECRET", STREAM_REGION)
> .apply( ... ));
>
> Instead of passing the AWS keys upfront, we would like to let the workers
> directly fetch the keys at run time from an external secret management
> service (e.g. Vault). That way, we could give Vault access permissions to
> the Dataflow cluster's service account, and avoid having to pass the keys
> upfront in clear.
>
> We've tried implementing a custom AWSClientsProvider
> 
>  and
> then passing it like so:
>
> p.apply(KinesisIO.read()
>//...
>.withAWSClientsProvider(new CustomAWSClientsProvider()));
>
> However, from our testing it looks like the
> CustomAWSClientsProvider.getKinesisClient() method stills gets called at
> job submission time, before the job actually starts running.
>
> Is there a way to let worker nodes retrieve those keys themselves at run
> time to configure Kinesis access?
>
> Thanks,
>
> Julien
>

Contributor permission for Beam Jira tickets

[Anup Ratnakar Raikar]

Hi Everyone,

This is Anup from GE. I would like to contribute to this amazing project.
Can someone add me as a contributor for Beam's Jira issue tracker? I would
like to assign/create tickets to work on.

Thanks,
Anup

Re: Java SplittableDoFn Watermark API

[Luke Cwik]

Using the interfaces defined in pr/10992[1], I started migrating from
ProcessContext#updateWatermark to the WatermarkEstimators with pr/11126[2].

The PR is very WIP but it does include the necessary changes to the Watch
transform and also the UnboundedSource SDF wrapper to be able to report
watermarks via the WatermarkEstimator. Another interesting point was that
we expect the setWatermark/observeTimestamp methods to be high volume while
curentWatermark/getState to be low volume in comparison so I restructured
the current set of WatermarkEstimators around that and also intent to move
the validation of the watermark values to more appropriate places.

1: https://github.com/apache/beam/pull/10992
2: https://github.com/apache/beam/pull/11126

On Mon, Mar 9, 2020 at 10:18 AM Luke Cwik  wrote:

> The current set of watermark estimators in Apache Beam for UnboundedSource
> are:
> SQS - tracks the timestamp of the last unacked message (does not report
> monotonically increasing watermarks and assumes that the system will make
> sure to lower bound what is being reported)
> AMP - tracks timestamp of last message
> GCP Pubsub
> :
> uses last minute of data to compute watermark but also handles cases where
> there is no data for long periods of time
> Kafka: Supports a watermark policy function
> 
>  that
> tracks one of (processing time, ingestion time, or custom user based
> version that gets to see the individual records)
> Kinesis: Supports a watermark policy function
> 
>  that
> tracks one of (processing time, arrival time, or custom user based version
> that gets to see the individual records)
>
> Has anyone written or knows of custom user based watermark policy
> functions that have been implemented for Kafka or Kinesis?
>
> On Mon, Mar 9, 2020 at 9:54 AM Ismaël Mejía  wrote:
>
>> Thanks for the explanation on Watch + FileIO it is really clear. Extra
>> question
>> related to WatermarkEstimator, is it supposed to be called in pipelines
>> at the
>> same exact moments that getWatermark is today for Unbounded sources?
>>
>
> Yes. DoFn authors will invoke it either manually after they tell us that
> the watermark has advanced or "automatically" via the timestamp observing
> watermark estimators.
>
>
>> (slightly unrelated) There is an open JIRA for an issue related to watch
>> and
>> watermark progress that we should probably investigate. I will add
>> details there
>> and ping you once I have the time to go back to the subject.
>> https://issues.apache.org/jira/browse/BEAM-9134
>
>
> The Watch stuff is built on top of a very experimental version of SDF for
> classic runners. I know that some people have used it effectively though
> with watermark tracking working for them.
>
>
>> Thanks also for the pointers on the UnboudedSource SDF wrapper.
>>
>> > Why do you want to evolve a bounded SDF into an unbounded SDF (is the
>> > restriction truly unbounded)?
>>
>> To prove that the idea that we can use the same API/code for both Bounded
>> and
>> Unbounded sources holds true, but the more I think about it the harder it
>> is to
>> me to get an example that cannot be built with a combination of Unbounded
>> SDF a
>> la Watch + a Bounded SDF. Can you think of any?
>>
>
> No but I'm sure someone will think of something eventually.
>
> > ... Java has been less of a focus since it is the most mature
>> non-portable
>> > implementation but hopefully will move in that direction quickly soon.
>>
>> We MUST move into this direction if we want to sell the idea of removing
>> non
>> portable translations to runner authors. There are few numbers on
>> performance of
>> pipelines using portability + the Java SDK Harness but if we take the
>> ValidatesRunner tests as the only single case where we have numbers
>> (notice that
>> VR uses the EMBEDDED environment, the lowest overhead one for Java runs)
>> the
>> numbers are far from good from looking at the latest execution times in
>> Jenkins:
>>
>> - Flink Runner VR #6900: 6 mins 22s vs Portable VR #4332 + #4343: 32 mins
>>   (5 times slower)
>>
>> - Spark Runner VR #6854: 8 mins 45s vs Portable VR #2322: 27 mins
>>   (3 times slower)
>>
>> Of course we can argue that this use case of short lived pipelines with
>> not big
>> data is not the average Beam use case, but still the numbers don’t look
>> good to
>> sell.
>
>
> I agree.
>
>
>> On Wed, Mar 4, 2020 at 10:52 PM Luke Cwik  wrote:
>> >
>> >
>> >
>> > On Wed, Mar 4, 2020 at 7:36 AM Ismaël Mejía  wrote:
>> >>
>> 

Re: Autolink references for Apache Beam

[Ismaël Mejía]

Thanks Kyle for catching that, I pinged INFRA again hopefully it will
be working soon.

On Fri, Mar 13, 2020 at 8:49 PM Kyle Weaver  wrote:
>
> Sounds like a really cool feature! However, it doesn't seem to work: links 
> are missing the BEAM- prefix, e.g. https://issues.apache.org/jira/browse/9475
>
> On Fri, Mar 13, 2020 at 12:42 PM Ismaël Mejía  wrote:
>>
>> Github released some months ago a new feature to configure Autolink 
>> references
>> for Apache Beam a new github feature to have automatic links to issues in
>> external systems e.g.  JIRA for our case.
>>
>> https://github.blog/2019-10-14-introducing-autolink-references/
>>
>> For info I filled INFRA-19967 to request this for Beam and Infra quickly 
>> enabled
>> it, the links should be started to populate in the new PRs soon.

Letting workers fetch AWS keys for Kinesis

[Julien Phalip]

Hi,

We're running a Beam job in Dataflow that reads inputs from KinesisIO.
Right now it works if we pass the AWS keys to the job like so:

p.apply(KinesisIO.read()
// ...
   .withAWSClientsProvider("AWS_KEY",  "AWS_SECRET", STREAM_REGION)
.apply( ... ));

Instead of passing the AWS keys upfront, we would like to let the workers
directly fetch the keys at run time from an external secret management
service (e.g. Vault). That way, we could give Vault access permissions to
the Dataflow cluster's service account, and avoid having to pass the keys
upfront in clear.

We've tried implementing a custom AWSClientsProvider

and
then passing it like so:

p.apply(KinesisIO.read()
   //...
   .withAWSClientsProvider(new CustomAWSClientsProvider()));

However, from our testing it looks like the
CustomAWSClientsProvider.getKinesisClient() method stills gets called at
job submission time, before the job actually starts running.

Is there a way to let worker nodes retrieve those keys themselves at run
time to configure Kinesis access?

Thanks,

Julien

Re: [VOTE + INPUT] Beam Mascot Designs, 1st iteration - Deadline Wed March 18

[Kenneth Knowles]

Nice!

1. Personally, I especially want it to be something easy for people to draw
and improvise in new poses / situations. Here is the sheet about the Go
Gopher that is about what I am thinking about, and amusing:
https://golang.org/doc/gopher/modelsheet.jpg. I think your slides do a
really good job of outlining similar things.

In order to encourage less talented artists to improvise using the
character, I favor simple & welcoming as the main depiction. I prefer the
side 3/4 view for the main depiction.

Color style - A - flat
Line style - A1 - heavy (I like heavy lines in all cases)
Color palette - works for me (I'm not sensitive to this)

Comments:

 - I really like the final slide and the one labeled "final outcome" which
have glowing wings. I wanted to note that A1 does not have the glowing
wings and it feels a bit flat with just one orange
 - I notice that in B1 the antennae are fully enclosed by lines, whereas A1
and C1 the end is left open to imply a connection to the head. I think B1
offers more playful possibilities for the antennae to be depicted in
different positions. There's the issue of being internally consistent with
how the arms are connected... looks good either way

Kenn

On Fri, Mar 13, 2020 at 12:33 PM Aizhamal Nurmamat kyzy 
wrote:

> Thank you Julian!
> Here is the link to the slides
> https://docs.google.com/presentation/d/1cEydTxQcsTJES_JCRvC_xgK7HMCIfnxmuawd7x2Bk_o/edit?usp=sharing
>
> On Fri, Mar 13, 2020 at 12:25 PM Julian Bruno 
> wrote:
>
>> Hello Apache Beam Community,
>>
>>
>> Together with Aizhamal and her team, we have been working on the design
>> of the Apache Beam mascot.
>>
>> We now need input from the community to continue moving forward with the
>> design. Please share your input no later than Wednesday, March 18, at noon
>> Pacific Time. Below you will find a link to the presentation of the work
>> process and we are eager to know what you think of the current design [1].
>>
>> Our questions to you:
>>
>> 1. Does the final outcome of the design process meet the community’s
>> expectations and represent Apache Beam? If not, what things should change?
>>
>> For printing, do you prefer sideways or front view for the character?
>>
>> 2. Which color style do you like? A. Flat Color  B. Gradient Colors C.
>> Shadows + Flat Colors (slide 13)
>>
>> 3. If you chose A, what line style do you like on the flat color style?
>> A1. Heavy Lines A2. Thin Lines A3.  No Lines (slide 15)
>>
>> If you chose B, what line style do you like on the gradient color style?
>> B1. Heavy Lines B2. Thin Lines B3.  No Lines (slide 17)
>>
>> If you chose C, what line style do you like on the shadows + flat color
>> style? C1. Heavy Lines C2. Thin Lines C3.  No Lines (slide 19)
>>
>> 4. What do you think about the color palette?
>>
>> Please reply inline, so it is clear what exactly you are referring to. The
>> vote and input phase will be open until Wednesday, March 18, at 12 pm
>> Pacific Time. And we will incorporate the feedback to the next design
>> iteration of the mascot.
>>
>> Thank you,
>>
>> Julian
>>
>>
>> --
>> Julian Bruno // Visual Artist & Graphic Designer
>>  (510) 367-0551 / SF Bay Area, CA
>> www.instagram.com/julbro.art
>>
>> ᐧ
>>
>

Re: Autolink references for Apache Beam

[Kyle Weaver]

Sounds like a really cool feature! However, it doesn't seem to work: links
are missing the BEAM- prefix, e.g.
https://issues.apache.org/jira/browse/9475

On Fri, Mar 13, 2020 at 12:42 PM Ismaël Mejía  wrote:

> Github released some months ago a new feature to configure Autolink
> references
> for Apache Beam a new github feature to have automatic links to issues in
> external systems e.g.  JIRA for our case.
>
> https://github.blog/2019-10-14-introducing-autolink-references/
>
> For info I filled INFRA-19967 to request this for Beam and Infra quickly
> enabled
> it, the links should be started to populate in the new PRs soon.
>

Autolink references for Apache Beam

[Ismaël Mejía]

Github released some months ago a new feature to configure Autolink references
for Apache Beam a new github feature to have automatic links to issues in
external systems e.g.  JIRA for our case.

https://github.blog/2019-10-14-introducing-autolink-references/

For info I filled INFRA-19967 to request this for Beam and Infra quickly enabled
it, the links should be started to populate in the new PRs soon.

Re: [VOTE + INPUT] Beam Mascot Designs, 1st iteration - Deadline Wed March 18

[Aizhamal Nurmamat kyzy]

Thank you Julian!
Here is the link to the slides
https://docs.google.com/presentation/d/1cEydTxQcsTJES_JCRvC_xgK7HMCIfnxmuawd7x2Bk_o/edit?usp=sharing

On Fri, Mar 13, 2020 at 12:25 PM Julian Bruno 
wrote:

> Hello Apache Beam Community,
>
>
> Together with Aizhamal and her team, we have been working on the design of
> the Apache Beam mascot.
>
> We now need input from the community to continue moving forward with the
> design. Please share your input no later than Wednesday, March 18, at noon
> Pacific Time. Below you will find a link to the presentation of the work
> process and we are eager to know what you think of the current design [1].
>
> Our questions to you:
>
> 1. Does the final outcome of the design process meet the community’s
> expectations and represent Apache Beam? If not, what things should change?
>
> For printing, do you prefer sideways or front view for the character?
>
> 2. Which color style do you like? A. Flat Color  B. Gradient Colors C.
> Shadows + Flat Colors (slide 13)
>
> 3. If you chose A, what line style do you like on the flat color style?
> A1. Heavy Lines A2. Thin Lines A3.  No Lines (slide 15)
>
> If you chose B, what line style do you like on the gradient color style?
> B1. Heavy Lines B2. Thin Lines B3.  No Lines (slide 17)
>
> If you chose C, what line style do you like on the shadows + flat color
> style? C1. Heavy Lines C2. Thin Lines C3.  No Lines (slide 19)
>
> 4. What do you think about the color palette?
>
> Please reply inline, so it is clear what exactly you are referring to. The
> vote and input phase will be open until Wednesday, March 18, at 12 pm
> Pacific Time. And we will incorporate the feedback to the next design
> iteration of the mascot.
>
> Thank you,
>
> Julian
>
>
> --
> Julian Bruno // Visual Artist & Graphic Designer
>  (510) 367-0551 / SF Bay Area, CA
> www.instagram.com/julbro.art
>
> ᐧ
>

[VOTE + INPUT] Beam Mascot Designs, 1st iteration - Deadline Wed March 18

[Julian Bruno]

Hello Apache Beam Community,


Together with Aizhamal and her team, we have been working on the design of
the Apache Beam mascot.

We now need input from the community to continue moving forward with the
design. Please share your input no later than Wednesday, March 18, at noon
Pacific Time. Below you will find a link to the presentation of the work
process and we are eager to know what you think of the current design [1].

Our questions to you:

1. Does the final outcome of the design process meet the community’s
expectations and represent Apache Beam? If not, what things should change?

For printing, do you prefer sideways or front view for the character?

2. Which color style do you like? A. Flat Color  B. Gradient Colors C.
Shadows + Flat Colors (slide 13)

3. If you chose A, what line style do you like on the flat color style? A1.
Heavy Lines A2. Thin Lines A3.  No Lines (slide 15)

If you chose B, what line style do you like on the gradient color style?
B1. Heavy Lines B2. Thin Lines B3.  No Lines (slide 17)

If you chose C, what line style do you like on the shadows + flat color
style? C1. Heavy Lines C2. Thin Lines C3.  No Lines (slide 19)

4. What do you think about the color palette?

Please reply inline, so it is clear what exactly you are referring to. The
vote and input phase will be open until Wednesday, March 18, at 12 pm
Pacific Time. And we will incorporate the feedback to the next design
iteration of the mascot.

Thank you,

Julian


-- 
Julian Bruno // Visual Artist & Graphic Designer
 (510) 367-0551 / SF Bay Area, CA
www.instagram.com/julbro.art

ᐧ

Re: Hello Beam Community!

[Tomo Suzuki]

Welcome.

On Fri, Mar 13, 2020 at 1:20 PM Udi Meiri  wrote:

> Welcome!
>
>
> On Fri, Mar 13, 2020 at 9:47 AM Yichi Zhang  wrote:
>
>> Welcome!
>>
>> On Fri, Mar 13, 2020 at 9:40 AM Ahmet Altay  wrote:
>>
>>> Welcome Brittany!
>>>
>>> On Thu, Mar 12, 2020 at 6:32 PM Brittany Hermann 
>>> wrote:
>>>
 Hello Beam Community!

 My name is Brittany Hermann and I recently joined the Open Source team
 in Data Analytics at Google. As a Program Manager, I will be focusing on
 community engagement while getting to work on Apache Beam and Airflow
 projects! I have always thrived on creating healthy, diverse, and overall
 happy communities and am excited to bring that to the team. For a fun fact,
 I am a big Wisconsin Badgers Football fan and have a goldendoodle puppy
 named Ollie!

 I look forward to collaborating with you all!

 Kind regards,

 Brittany Hermann




-- 
Regards,
Tomo

Re: Hello Beam Community!

[Udi Meiri]

Welcome!


On Fri, Mar 13, 2020 at 9:47 AM Yichi Zhang  wrote:

> Welcome!
>
> On Fri, Mar 13, 2020 at 9:40 AM Ahmet Altay  wrote:
>
>> Welcome Brittany!
>>
>> On Thu, Mar 12, 2020 at 6:32 PM Brittany Hermann 
>> wrote:
>>
>>> Hello Beam Community!
>>>
>>> My name is Brittany Hermann and I recently joined the Open Source team
>>> in Data Analytics at Google. As a Program Manager, I will be focusing on
>>> community engagement while getting to work on Apache Beam and Airflow
>>> projects! I have always thrived on creating healthy, diverse, and overall
>>> happy communities and am excited to bring that to the team. For a fun fact,
>>> I am a big Wisconsin Badgers Football fan and have a goldendoodle puppy
>>> named Ollie!
>>>
>>> I look forward to collaborating with you all!
>>>
>>> Kind regards,
>>>
>>> Brittany Hermann
>>>
>>>
>>>


smime.p7s
Description: S/MIME Cryptographic Signature

Re: Hello Beam Community!

[Yichi Zhang]

Welcome!

On Fri, Mar 13, 2020 at 9:40 AM Ahmet Altay  wrote:

> Welcome Brittany!
>
> On Thu, Mar 12, 2020 at 6:32 PM Brittany Hermann 
> wrote:
>
>> Hello Beam Community!
>>
>> My name is Brittany Hermann and I recently joined the Open Source team in
>> Data Analytics at Google. As a Program Manager, I will be focusing on
>> community engagement while getting to work on Apache Beam and Airflow
>> projects! I have always thrived on creating healthy, diverse, and overall
>> happy communities and am excited to bring that to the team. For a fun fact,
>> I am a big Wisconsin Badgers Football fan and have a goldendoodle puppy
>> named Ollie!
>>
>> I look forward to collaborating with you all!
>>
>> Kind regards,
>>
>> Brittany Hermann
>>
>>
>>

Re: Hello Beam Community!

[Ahmet Altay]

Welcome Brittany!

On Thu, Mar 12, 2020 at 6:32 PM Brittany Hermann 
wrote:

> Hello Beam Community!
>
> My name is Brittany Hermann and I recently joined the Open Source team in
> Data Analytics at Google. As a Program Manager, I will be focusing on
> community engagement while getting to work on Apache Beam and Airflow
> projects! I have always thrived on creating healthy, diverse, and overall
> happy communities and am excited to bring that to the team. For a fun fact,
> I am a big Wisconsin Badgers Football fan and have a goldendoodle puppy
> named Ollie!
>
> I look forward to collaborating with you all!
>
> Kind regards,
>
> Brittany Hermann
>
>
>

Buggy/Slow FileIO.write()/sink implementation

[Anand Singh Kunwar]

Hi beam devs,

Context:
I have been experimenting with generating columnar data from prometheus
metric data to write to Google Cloud Storage. My pipeline takes input of
Prometheus Remote Write HTTP payload from kafka(this is compressed in
snappy and protobuf encoded), my first 2 steps of the pipeline do the
uncompression and decoding and make a metric object. I window this input to
fixed windows of 1 minute and write the window to GCS in ORC format. I have
been seeing huge lag in my pipeline.

Problem/Bug:
The custom FileIO.write().sink implementation for ORCIO writes to GCS using
the ORC library. In my sink implementation I even implemented all
operations as no-ops, even then I saw a huge lag in my pipeline. When I
comment out the FileIO transformation(that is acutally a no-op), my
pipeline keeps up with the input load.
Looking up online my problem seems to relate to this
https://stackoverflow.com/questions/54094102/beam-pipeline-kafka-to-hdfs-by-time-buckets
.

This is what my code looks like:

p.apply("ReadLines", KafkaIO.read().withBootstrapServers(
"mykafka:9092")
.withTopic(options.getInputTopic())

.withConsumerConfigUpdates(ImmutableMap.of(ConsumerConfig.GROUP_ID_CONFIG,
"custom-id",
ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "true"))
.withKeyDeserializer(LongDeserializer.class)

.withValueDeserializer(ByteArrayDeserializer.class).withoutMetadata())
.apply("UncompressSnappy", ParDo.of(new UncompressSnappy()))
.apply("DecodeProto", ParDo.of(new DecodePromProto()))
.apply("MapTSSample", ParDo.of(new MapTSSample()))

.apply(Window.into(FixedWindows.of(Duration.standardMinutes(1)))
.withTimestampCombiner(TimestampCombiner.END_OF_WINDOW))
.apply(new WriteOneFilePerWindow(options.getOutput(), 1, ".orc"));

This is what WriteOneFilePerWindow.java's expand looks like for me:

public PDone expand(PCollection input) {
input.apply(FileIO.write().to(filenamePrefix).withNaming(new
MyFileNaming(filenameSuffix))
.withNumShards(numShards).via(ORCIO.sink()));
return PDone.in(input.getPipeline());
}


Best
Anand Singh Kunwar

Re: Request to be added as Jira contributor

[Piotr Szuberski]

Thanks! :)

On 2020/03/12 19:14:43, Ahmet Altay  wrote: 
> Done. Welcome!
> 
> On Wed, Mar 11, 2020 at 1:51 PM Piotr Szuberski 
> wrote:
> 
> > *Hi, my name is Piotr. I am interested in contributing java sdk to the
> > Apache Beam SDK. I'd like to be added **as a Jira contributor so that I
> > can assign issues to myself. My ASF Jira Username is piotr-szuberski .*
> >
>