Re: Question about unbounded in-memory PCollection

[Rui Wang]

Does TestStream.java
[1]
satisfy your need?



-Rui

[1]
https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestStream.java

On Tue, May 7, 2019 at 2:47 PM Chengzhi Zhao 
wrote:

> Hi Beam Team,
>
> I am new to here and recently study the programming guide, I have a
> question about the in-memory data,
> https://beam.apache.org/documentation/programming-guide/#creating-a-pcollection
>
> Is there a way to create unbounded PCollection from the in-memory
> collection? I want to test the unbounded PCollection locally and don't know
> what's the easiest way to get unbounded PCollection. Please let me know if
> I am doing something wrong or I should use a file system to do it.
>
> Thanks in advance!
>
> Best,
> Chengzhi
>
> On Tue, May 7, 2019 at 5:46 PM Chengzhi Zhao 
> wrote:
>
>> Hi Beam Team,
>>
>> I am new to here and recently study the programming guide, I have a
>> question about the in-memory data,
>> https://beam.apache.org/documentation/programming-guide/#creating-a-pcollection
>>
>> Is there a way to create unbounded PCollection from the in-memory
>> collection? I want to test the unbounded PCollection locally and don't know
>> what's the easiest way to get unbounded PCollection. Please let me know if
>> I am doing something wrong or I should use a file system to do it.
>>
>> Thanks in advance!
>>
>> Best,
>> Chengzhi
>>
>

Re: Question about unbounded in-memory PCollection

[Chengzhi Zhao]

Hi Beam Team,

I am new to here and recently study the programming guide, I have a
question about the in-memory data,
https://beam.apache.org/documentation/programming-guide/#creating-a-pcollection

Is there a way to create unbounded PCollection from the in-memory
collection? I want to test the unbounded PCollection locally and don't know
what's the easiest way to get unbounded PCollection. Please let me know if
I am doing something wrong or I should use a file system to do it.

Thanks in advance!

Best,
Chengzhi

On Tue, May 7, 2019 at 5:46 PM Chengzhi Zhao 
wrote:

> Hi Beam Team,
>
> I am new to here and recently study the programming guide, I have a
> question about the in-memory data,
> https://beam.apache.org/documentation/programming-guide/#creating-a-pcollection
>
> Is there a way to create unbounded PCollection from the in-memory
> collection? I want to test the unbounded PCollection locally and don't know
> what's the easiest way to get unbounded PCollection. Please let me know if
> I am doing something wrong or I should use a file system to do it.
>
> Thanks in advance!
>
> Best,
> Chengzhi
>

Re: Streaming inserts BQ with Java SDK Beam

[Andres Angel]

Pablo thanks so much I will explore this method then for   STREAMING_INSERTS
,
this answer worth much for us :)

thanks.

On Tue, May 7, 2019 at 1:05 PM Pablo Estrada  wrote:

> Hi Andres!
> You can definitely do streaming inserts using the Java SDK. This is
> available with BigQueryIO.write(). Specifically, you can use the
> `withMethod`[1] call to specify whether you want batch loads or streaming
> inserts. If you specify streaming inserts, Beam should insert rows as they
> come in bundles.
> Hope that helps
> -P.
>
> [1]
> https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.Write.html#withMethod-org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.Write.Method-
>
> On Tue, May 7, 2019 at 9:58 AM Andres Angel <
> ingenieroandresan...@gmail.com> wrote:
>
>> Hello everyone,
>>
>> I need to use BigQuery inserts within my beam pipeline, hence I know well
>> the built-in IO options offer `BigQueryIO`, however this will insert in a
>> batch fashion to BQ creating underneath a BQ load job. I instead need to
>> trigger a streaming insert into BQ, and I was reviewing the Java SDK
>> documentation but seems like this is not possible.
>>
>> In the other hand, I have the python SDK and I found this GitHub
>> documentation
>> 
>> code where they are using a method *InsertAll
>> *
>>  which
>> is apparently what I need. If this is official I would like to know if
>> there is a naive fashion to trigger stream inserts in BQ using the Java SDK.
>>
>> thanks so much for your feedback
>> AU
>>
>

Re: Streaming inserts BQ with Java SDK Beam

[Alex Van Boxel]

I think you really need a peculiar reason to force streamingInsert in a
batch job. In batch mode you. Note that you will quickly hit the quota
limit in batch mode: "
Maximum rows per second: 100,000 rows per second, per project", as in batch
load you can process a lot more information in a shorter time.

I know you can force a batch mode in streaming mode, I don't know for the
other way around.

_/
_/ Alex Van Boxel


On Tue, May 7, 2019 at 6:58 PM Andres Angel 
wrote:

> Hello everyone,
>
> I need to use BigQuery inserts within my beam pipeline, hence I know well
> the built-in IO options offer `BigQueryIO`, however this will insert in a
> batch fashion to BQ creating underneath a BQ load job. I instead need to
> trigger a streaming insert into BQ, and I was reviewing the Java SDK
> documentation but seems like this is not possible.
>
> In the other hand, I have the python SDK and I found this GitHub
> documentation
> 
> code where they are using a method *InsertAll
> *
>  which
> is apparently what I need. If this is official I would like to know if
> there is a naive fashion to trigger stream inserts in BQ using the Java SDK.
>
> thanks so much for your feedback
> AU
>

Re: Is it safe to cache the value of a singleton view (with a global window) in a DoFn?

[Lukasz Cwik]

Keep your code simple and rely on the runner caching the value locally so
it should be very cheap to access. If you have a performance issue due to a
runner lacking caching, it would be good to hear about it so we could file
a JIRA about it.

On Mon, May 6, 2019 at 4:24 PM Kenneth Knowles  wrote:

> A singleton view in the global window and no triggering does have just a
> single immutable value. (It really ought to have an updated value in the
> presence of triggers, but I believe instead you will receive a crash. I
> haven't tested.)
>
> In general, a side input yields one value per window. Dataflow in batch
> will already do what you describe, but works with all windows. Dataflow in
> streaming has some caching but if you see a problem that is interesting
> information.
>
> Kenn
>
> On Sat, May 4, 2019 at 9:19 AM Steve Niemitz  wrote:
>
>> I have a singleton view in a global window that is read from a DoFn.  I'm
>> curious if its "correct" to cache that value from the view, or if I need to
>> read it every time.
>>
>> As a (simplified) example, if I were to generate the view as such:
>>
>> input.getPipeline
>>   .apply(Create.of(Collections.singleton[Void](null)))
>>   .apply(MapElements.via(new SimpleFunction[Void, JLong]() {
>> override def apply(input: Void): JLong = {
>>   Instant.now().getMillis
>> }
>>   })).apply(View.asSingleton[JLong]())
>>
>> and then read it from a DoFn (using context.sideInput), is it guaranteed
>> that:
>> - every instance of the DoFn will read the same value?
>> - The value will never change?
>>
>> If so it seems like it'd be safe to cache the value inside the DoFn.  It
>> seems like this would be the case, but I've also seen cases in dataflow
>> where the UI indicates that the MapElements step above produced more than
>> one element, so I'm curious what people have to say.
>>
>> Thanks!
>>
>

Re: Streaming inserts BQ with Java SDK Beam

[Pablo Estrada]

Hi Andres!
You can definitely do streaming inserts using the Java SDK. This is
available with BigQueryIO.write(). Specifically, you can use the
`withMethod`[1] call to specify whether you want batch loads or streaming
inserts. If you specify streaming inserts, Beam should insert rows as they
come in bundles.
Hope that helps
-P.

[1]
https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.Write.html#withMethod-org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.Write.Method-

On Tue, May 7, 2019 at 9:58 AM Andres Angel 
wrote:

> Hello everyone,
>
> I need to use BigQuery inserts within my beam pipeline, hence I know well
> the built-in IO options offer `BigQueryIO`, however this will insert in a
> batch fashion to BQ creating underneath a BQ load job. I instead need to
> trigger a streaming insert into BQ, and I was reviewing the Java SDK
> documentation but seems like this is not possible.
>
> In the other hand, I have the python SDK and I found this GitHub
> documentation
> 
> code where they are using a method *InsertAll
> *
>  which
> is apparently what I need. If this is official I would like to know if
> there is a naive fashion to trigger stream inserts in BQ using the Java SDK.
>
> thanks so much for your feedback
> AU
>

Streaming inserts BQ with Java SDK Beam

[Andres Angel]

Hello everyone,

I need to use BigQuery inserts within my beam pipeline, hence I know well
the built-in IO options offer `BigQueryIO`, however this will insert in a
batch fashion to BQ creating underneath a BQ load job. I instead need to
trigger a streaming insert into BQ, and I was reviewing the Java SDK
documentation but seems like this is not possible.

In the other hand, I have the python SDK and I found this GitHub
documentation

code where they are using a method *InsertAll
*
which
is apparently what I need. If this is official I would like to know if
there is a naive fashion to trigger stream inserts in BQ using the Java SDK.

thanks so much for your feedback
AU

Re: Apache BEAM on Flink in production

[Austin Bennett]

On the Beam YouTube channel:
https://www.youtube.com/channel/UChNnb_YO_7B0HlW6FhAXZZQ you can see two
talks from people at Lyft; they use Beam on Flink.

Other users can also chime in as to how they are running.

Would also suggest coming to BeamSummit.org in Berlin in June and/or
sharing experiences or coming to ApacheCon in September, where we are to
have 2 tracks in each of 2 days focused on Beam
https://www.apachecon.com/acna19/index.html




On Tue, May 7, 2019 at 6:52 AM  wrote:

> Hi all,
>
>
>
> We currently run Apache Flink based data load processes (fairly simple
> streaming ETL jobs) and are looking at converting to Apache BEAM to give
> more flexibility on the runner.
>
>
>
> Is anyone aware of any organisations running Apache BEAM on Flink in
> production. Does anyone have any case studies they would be able to share?
>
>
>
> Many thanks,
>
>
>
> Steve
>
> This communication and any attachments are confidential and intended
> solely for the addressee. If you are not the intended recipient please
> advise us immediately and delete it. Unless specifically stated in the
> message or otherwise indicated, you may not duplicate, redistribute or
> forward this message and any attachments are not intended for distribution
> to, or use by any person or entity in any jurisdiction or country where
> such distribution or use would be contrary to local law or regulation.
> NatWest Markets Plc  or any affiliated entity ("NatWest Markets") accepts
> no responsibility for any changes made to this message after it was sent.
> Unless otherwise specifically indicated, the contents of this
> communication and its attachments are for information purposes only and
> should not be regarded as an offer or solicitation to buy or sell a product
> or service, confirmation of any transaction, a valuation, indicative price
> or an official statement. Trading desks may have a position or interest
> that is inconsistent with any views expressed in this message. In
> evaluating the information contained in this message, you should know that
> it could have been previously provided to other clients and/or internal
> NatWest Markets personnel, who could have already acted on it.
> NatWest Markets cannot provide absolute assurances that all electronic
> communications (sent or received) are secure, error free, not corrupted,
> incomplete or virus free and/or that they will not be lost, mis-delivered,
> destroyed, delayed or intercepted/decrypted by others. Therefore NatWest
> Markets disclaims all liability with regards to electronic communications
> (and the contents therein) if they are corrupted, lost destroyed, delayed,
> incomplete, mis-delivered, intercepted, decrypted or otherwise
> misappropriated by others.
> Any electronic communication that is conducted within or through NatWest
> Markets systems will be subject to being archived, monitored and produced
> to regulators and in litigation in accordance with NatWest Markets’ policy
> and local laws, rules and regulations. Unless expressly prohibited by local
> law, electronic communications may be archived in countries other than the
> country in which you are located, and may be treated in accordance with the
> laws and regulations of the country of each individual included in the
> entire chain.
> Copyright NatWest Markets plc. All rights reserved. See
> http://www.natwestmarkets.com/legal/s-t-discl.html for further risk
> disclosure.
>

Apache BEAM on Flink in production

[Stephen.Hesketh]

Hi all,

We currently run Apache Flink based data load processes (fairly simple 
streaming ETL jobs) and are looking at converting to Apache BEAM to give more 
flexibility on the runner.

Is anyone aware of any organisations running Apache BEAM on Flink in 
production. Does anyone have any case studies they would be able to share?

Many thanks,

Steve


This communication and any attachments are confidential and intended solely for 
the addressee. If you are not the intended recipient please advise us 
immediately and delete it. Unless specifically stated in the message or 
otherwise indicated, you may not duplicate, redistribute or forward this 
message and any attachments are not intended for distribution to, or use by any 
person or entity in any jurisdiction or country where such distribution or use 
would be contrary to local law or regulation. NatWest Markets Plc  or any 
affiliated entity ("NatWest Markets") accepts no responsibility for any changes 
made to this message after it was sent.

Unless otherwise specifically indicated, the contents of this communication and 
its attachments are for information purposes only and should not be regarded as 
an offer or solicitation to buy or sell a product or service, confirmation of 
any transaction, a valuation, indicative price or an official statement. 
Trading desks may have a position or interest that is inconsistent with any 
views expressed in this message. In evaluating the information contained in 
this message, you should know that it could have been previously provided to 
other clients and/or internal NatWest Markets personnel, who could have already 
acted on it.

NatWest Markets cannot provide absolute assurances that all electronic 
communications (sent or received) are secure, error free, not corrupted, 
incomplete or virus free and/or that they will not be lost, mis-delivered, 
destroyed, delayed or intercepted/decrypted by others. Therefore NatWest 
Markets disclaims all liability with regards to electronic communications (and 
the contents therein) if they are corrupted, lost destroyed, delayed, 
incomplete, mis-delivered, intercepted, decrypted or otherwise misappropriated 
by others.

Any electronic communication that is conducted within or through NatWest 
Markets systems will be subject to being archived, monitored and produced to 
regulators and in litigation in accordance with NatWest Markets’ policy and 
local laws, rules and regulations. Unless expressly prohibited by local law, 
electronic communications may be archived in countries other than the country 
in which you are located, and may be treated in accordance with the laws and 
regulations of the country of each individual included in the entire chain.

Copyright NatWest Markets plc. All rights reserved. See 
http://www.natwestmarkets.com/legal/s-t-discl.html for further risk disclosure.