Re: Build blocking on

[Robert Burke]

It's concerning to me that 1) the Go dependency resolution via gogradle is
flaky, and 2) that it can block other languages.

I suppose 2) makes sense since it's part of the container bootstrapping
code, but that makes 1) a serious problem, of which I wasn't aware.
I should have time to investigate this in the next two weeks.

On Mon, 25 Mar 2019 at 18:08, Michael Luckey  wrote:

> Just for the record,
>
> using a vm here, because did not yet get all task running on my mac, and
> did not want to mess with my setup.
>
> So installed vanilla ubuntu-18.04 LTS on virtual box, 26GB ram, 6 cores
> and further
>
> sudo apt update
>
> sudo apt install gcc
>
> sudo apt install make
>
> sudo apt install perl
>
> sudo apt install curl
>
> sudo apt install openjdk-8-jdk
>
> sudo apt install python
>
> sudo apt install -y software-properties-common
>
> sudo add-apt-repository ppa:deadsnakes/ppa
>
> sudo apt update
>
> sudo apt install python3.5
>
> sudo apt-get install apt-transport-https ca-certificates curl gnupg-agent
> software-properties-common
>
> curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key
> add -
>
> sudo apt-key fingerprint 0EBFCD88
>
> sudo add-apt-repository "deb [arch=amd64]
> https://download.docker.com/linux/ubuntu \
>
> $(lsb_release -cs) \
>
> stable"
>
> sudo apt-get update
>
> sudo apt-get install docker-ce docker-ce-cli containerd.io
>
> sudo groupadd docker
>
> sudo usermod -aG docker $USER
>
> git config --global user.email "d...@spam.me"
>
> git config --global user.name "Some Guy"
>
> curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
>
> sudo python get-pip.py
>
> rm get-pip.py
>
> sudo pip install --upgrade virtualenv
>
> sudo pip install cython
>
> sudo apt-get install python-dev
>
> sudo apt-get install python3-distutils
>
> sudo apt-get install python3-dev # for python3.x installs
>
>
> git clone https://github.com/apache/beam.git cd beam/ ./gradlew build
>
> Nothing else changed/added. (hopefully, need to reassure myself here)
>
> Unfortunately, this is failing. Need to exclude those python tests (and of
> course website, which usually fails on lira links)
>
> So I might be missing some env settings for gap, dunno. Probably missed
> some docs.
>
>
>
> On Tue, Mar 26, 2019 at 1:46 AM Michael Luckey 
> wrote:
>
>> Thanks Udi for trying that!
>>
>> In fact, the go dependency resolution is flaky. Did not look into that,
>> but just rerunning usually works. Of course, less than optimal, but,
>> well...
>>
>> Running build target is of course just an aggregation of task to run. And
>> unfortunately just running that
>>
>> ./gradlew  :beam-sdks-python:testPy2Gcp
>>
>> stalls on my (virtual) machine.
>>
>> On Tue, Mar 26, 2019 at 1:35 AM Udi Meiri  wrote:
>>
>>> Okay, `./gradlew build` failed pretty quickly for me:
>>>
>>> > Task :beam-sdks-go:resolveBuildDependencies FAILED
>>> cloud.google.com/go: commit='4f6c921ec566a33844f4e7879b31cd8575a6982d',
>>> urls=[https://code.googlesource.com/gocloud] does not exist in
>>> /usr/local/google/home/ehudm/.gradle/go/repo/
>>> cloud.google.com/go/625660c387d9403fde4d73cacaf2d2ac, updating will be
>>> performed.
>>>
>>> https://gradle.com/s/x5zqbc5zwd3bg
>>>
>>> (Now I remember why I stopped using `build` :/)
>>>
>>> On Mon, Mar 25, 2019 at 5:30 PM Udi Meiri  wrote:
>>>
 It shouldn't stall. That's a bug.
 OTOH, I never use the `build` target.
 I'll try running that myself.

 On Mon, Mar 25, 2019, 07:24 Michael Luckey  wrote:

> Hi,
>
> trying to run './gradlew build' on vanilla setup, my build
> consistently stalls during execution of python gcp tests, e.g. on both of
> - > :beam-sdks-python:testPy2Gcp
> - > :beam-sdks-python-test-suites-tox-py35:testPy35Gcp
>
> Console output:
>  snip 
> test_big_query_standard_sql
> (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT)
> ... SKIP: IT is skipped because --test-pipeline-options is not specified
> test_big_query_standard_sql_kms_key
> (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT)
> ... SKIP: This test requires BQ Dataflow native source support for KMS,
> which is not available yet.
> test_multiple_destinations_transform
> (apache_beam.io.gcp.bigquery_file_loads_test.BigQueryFileLoadsIT) ... 
> SKIP:
> IT is skipped because --test-pipeline-options is not specified
> test_one_job_fails_all_jobs_fail
> (apache_beam.io.gcp.bigquery_file_loads_test.BigQueryFileLoadsIT) ... 
> SKIP:
> IT is skipped because --test-pipeline-options is not specified
> test_records_traverse_transform_with_mocks
> (apache_beam.io.gcp.bigquery_file_loads_test.TestBigQueryFileLoads) ...
>
> output ends here, would expect a failed or ok here.
>
>
> Afterwards no progress - even waiting for hours. Any idea, what might
> be causing this? Do I need to add some GCP properties for this task ?
>
>

Re: Writing bytes to BigQuery with beam

[Valentyn Tymofieiev]

Thanks everyone for input on this thread. I think there is a confusion
between not specifying the schema, and asking BigQuery to do schema
autodetection. This is not the same thing, however in recent changes to BQ
IO that happened after 2.11 release, we are forcing schema autodetection,
when schema is not specified, see: [1].

I think we need to revise this ahead of 2.12. It may be better if users
explicitly opt-in to schema autodetection if they wish. Autodetection is an
approximation, and in particular, as we figured out in this thread, it does
not work correctly for BYTES data.

I suspect that if we disable schema autodetection, and/or make previous
implementation of BQ sink a default option, we will be able to write BYTES
data to a previously created BQ table without specifying the schema, and
making a call to BQ to fetch the schema won't be necessary. We'd need to
verify that.

Another interesting note, as per Juta's analysis
,
google-cloud-bigquery client does not require additional base64 encoding
for bytes, so once we migrate to use this client, base64 encoding/decoding
of Bytes data won't be necessary in Beam.

[1]
https://github.com/apache/beam/blob/0b71f541e93f3bd69af87ad8a6db46ccb4a01ddc/sdks/python/apache_beam/io/gcp/bigquery_tools.py#L321
.
[2]
https://docs.google.com/document/d/19zvDycWzF82MmtCmxrhqqyXKaRq8slRIjdxE6E8MObA/edit#bookmark=id.7pfrsz1c8hcj

On Mon, Mar 25, 2019 at 2:26 PM Chamikara Jayalath 
wrote:

>
>
> On Mon, Mar 25, 2019 at 2:16 PM Pablo Estrada  wrote:
>
>> +Chamikara Jayalath  with the new BigQuery sink,
>> schema autodetection is supported (it's a very simple thing to have). Do
>> you think we should not have it?
>> Best
>> -P.
>>
>
> Ah good to know. But IMO users should be able to write to existing tables
> without specifying a schema (when CEATE_DISPOSITION is CREATE_NEVER for
> example). How do users enable schema auto-detection ? Probably this should
> not be enabled by default and we should clearly advertise that bytes type
> is not supported (or support it with extra information). Just my 2 cents.
>
> Thanks,
> Cham
>
>
>>
>> On Mon, Mar 25, 2019 at 11:01 AM Chamikara Jayalath 
>> wrote:
>>
>>>
>>>
>>> On Mon, Mar 25, 2019 at 2:03 AM Juta Staes  wrote:
>>>

 On Mon, 25 Mar 2019 at 06:15, Valentyn Tymofieiev 
 wrote:

> We received feedback on
> https://issuetracker.google.com/issues/129006689 - BQ developers say
> that schema identification is done and they discourage to use schema
> autodetection in tables using BYTES. In light of this, I think may be fair
> to recommend Beam users to specify BQ schemas as well when they interact
> with BQ, and call out that writing binary data to BQ will likely fail
> unless schema is specified. Does that make sense?
>

 Given that schema autodetect does not work for bytes I think it is
 indeed a good solution to require users to specify BQ schemas as well when
 they write to BQ

 So new summary:
 1. Beam will base64-encode raw bytes, before passing them to BQ over
 rest API. This will be a change in behavior for Python 2 (for good 
 reasons).
 2. When reading data from BQ, all fields of type BYTES will be
 base64-decoded.
 3. Beam will send an API call to BigQuery to get table schema,
 whenever schema is not supplied, to work around
 https://issuetracker.google.com/issues/129006689. Beam will require
 users to specify the schema when writing bytes to BQ.

>>>
>>> I'm not sure why we reached this conclusion. We (Beam) does not use BQ
>>> schema auto detection feature currently.  So why not just send an API
>>> signal to get the schema when users are writing to existing tables ? Also,
>>> even if we decide to support schema auto detection in the future we will
>>> not be able to support this for BYTEs type (due to the restriction by BQ).
>>>
>>>
 Thanks all for your input on this!
 Juta

Re: Build blocking on

[Michael Luckey]

Just for the record,

using a vm here, because did not yet get all task running on my mac, and
did not want to mess with my setup.

So installed vanilla ubuntu-18.04 LTS on virtual box, 26GB ram, 6 cores and
further

sudo apt update

sudo apt install gcc

sudo apt install make

sudo apt install perl

sudo apt install curl

sudo apt install openjdk-8-jdk

sudo apt install python

sudo apt install -y software-properties-common

sudo add-apt-repository ppa:deadsnakes/ppa

sudo apt update

sudo apt install python3.5

sudo apt-get install apt-transport-https ca-certificates curl gnupg-agent
software-properties-common

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

sudo apt-key fingerprint 0EBFCD88

sudo add-apt-repository "deb [arch=amd64]
https://download.docker.com/linux/ubuntu \

$(lsb_release -cs) \

stable"

sudo apt-get update

sudo apt-get install docker-ce docker-ce-cli containerd.io

sudo groupadd docker

sudo usermod -aG docker $USER

git config --global user.email "d...@spam.me"

git config --global user.name "Some Guy"

curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py

sudo python get-pip.py

rm get-pip.py

sudo pip install --upgrade virtualenv

sudo pip install cython

sudo apt-get install python-dev

sudo apt-get install python3-distutils

sudo apt-get install python3-dev # for python3.x installs


git clone https://github.com/apache/beam.git cd beam/ ./gradlew build

Nothing else changed/added. (hopefully, need to reassure myself here)

Unfortunately, this is failing. Need to exclude those python tests (and of
course website, which usually fails on lira links)

So I might be missing some env settings for gap, dunno. Probably missed
some docs.



On Tue, Mar 26, 2019 at 1:46 AM Michael Luckey  wrote:

> Thanks Udi for trying that!
>
> In fact, the go dependency resolution is flaky. Did not look into that,
> but just rerunning usually works. Of course, less than optimal, but,
> well...
>
> Running build target is of course just an aggregation of task to run. And
> unfortunately just running that
>
> ./gradlew  :beam-sdks-python:testPy2Gcp
>
> stalls on my (virtual) machine.
>
> On Tue, Mar 26, 2019 at 1:35 AM Udi Meiri  wrote:
>
>> Okay, `./gradlew build` failed pretty quickly for me:
>>
>> > Task :beam-sdks-go:resolveBuildDependencies FAILED
>> cloud.google.com/go: commit='4f6c921ec566a33844f4e7879b31cd8575a6982d',
>> urls=[https://code.googlesource.com/gocloud] does not exist in
>> /usr/local/google/home/ehudm/.gradle/go/repo/
>> cloud.google.com/go/625660c387d9403fde4d73cacaf2d2ac, updating will be
>> performed.
>>
>> https://gradle.com/s/x5zqbc5zwd3bg
>>
>> (Now I remember why I stopped using `build` :/)
>>
>> On Mon, Mar 25, 2019 at 5:30 PM Udi Meiri  wrote:
>>
>>> It shouldn't stall. That's a bug.
>>> OTOH, I never use the `build` target.
>>> I'll try running that myself.
>>>
>>> On Mon, Mar 25, 2019, 07:24 Michael Luckey  wrote:
>>>
 Hi,

 trying to run './gradlew build' on vanilla setup, my build consistently
 stalls during execution of python gcp tests, e.g. on both of
 - > :beam-sdks-python:testPy2Gcp
 - > :beam-sdks-python-test-suites-tox-py35:testPy35Gcp

 Console output:
  snip 
 test_big_query_standard_sql
 (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT)
 ... SKIP: IT is skipped because --test-pipeline-options is not specified
 test_big_query_standard_sql_kms_key
 (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT)
 ... SKIP: This test requires BQ Dataflow native source support for KMS,
 which is not available yet.
 test_multiple_destinations_transform
 (apache_beam.io.gcp.bigquery_file_loads_test.BigQueryFileLoadsIT) ... SKIP:
 IT is skipped because --test-pipeline-options is not specified
 test_one_job_fails_all_jobs_fail
 (apache_beam.io.gcp.bigquery_file_loads_test.BigQueryFileLoadsIT) ... SKIP:
 IT is skipped because --test-pipeline-options is not specified
 test_records_traverse_transform_with_mocks
 (apache_beam.io.gcp.bigquery_file_loads_test.TestBigQueryFileLoads) ...

 output ends here, would expect a failed or ok here.


 Afterwards no progress - even waiting for hours. Any idea, what might
 be causing this? Do I need to add some GCP properties for this task ?

 Any ideas, what I am doing wrong?

 best,

 michel

Re: Frequent failures on beam8

[Yifan Zou]

the beam8 is disabled by now.

On Mon, Mar 25, 2019 at 2:06 PM Mikhail Gryzykhin  wrote:

> Yifan is looking into this.
>
> On Mon, Mar 25, 2019 at 1:55 PM Boyuan Zhang  wrote:
>
>> Hey all,
>>
>> Could anyone help take a look at beam8
>> ? Seems like there are
>> many tests failed on beam8 owing to infra problems.
>>
>> Thanks!
>>
>

Re: Build blocking on

[Michael Luckey]

Thanks Udi for trying that!

In fact, the go dependency resolution is flaky. Did not look into that, but
just rerunning usually works. Of course, less than optimal, but, well...

Running build target is of course just an aggregation of task to run. And
unfortunately just running that

./gradlew  :beam-sdks-python:testPy2Gcp

stalls on my (virtual) machine.

On Tue, Mar 26, 2019 at 1:35 AM Udi Meiri  wrote:

> Okay, `./gradlew build` failed pretty quickly for me:
>
> > Task :beam-sdks-go:resolveBuildDependencies FAILED
> cloud.google.com/go: commit='4f6c921ec566a33844f4e7879b31cd8575a6982d',
> urls=[https://code.googlesource.com/gocloud] does not exist in
> /usr/local/google/home/ehudm/.gradle/go/repo/
> cloud.google.com/go/625660c387d9403fde4d73cacaf2d2ac, updating will be
> performed.
>
> https://gradle.com/s/x5zqbc5zwd3bg
>
> (Now I remember why I stopped using `build` :/)
>
> On Mon, Mar 25, 2019 at 5:30 PM Udi Meiri  wrote:
>
>> It shouldn't stall. That's a bug.
>> OTOH, I never use the `build` target.
>> I'll try running that myself.
>>
>> On Mon, Mar 25, 2019, 07:24 Michael Luckey  wrote:
>>
>>> Hi,
>>>
>>> trying to run './gradlew build' on vanilla setup, my build consistently
>>> stalls during execution of python gcp tests, e.g. on both of
>>> - > :beam-sdks-python:testPy2Gcp
>>> - > :beam-sdks-python-test-suites-tox-py35:testPy35Gcp
>>>
>>> Console output:
>>>  snip 
>>> test_big_query_standard_sql
>>> (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT)
>>> ... SKIP: IT is skipped because --test-pipeline-options is not specified
>>> test_big_query_standard_sql_kms_key
>>> (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT)
>>> ... SKIP: This test requires BQ Dataflow native source support for KMS,
>>> which is not available yet.
>>> test_multiple_destinations_transform
>>> (apache_beam.io.gcp.bigquery_file_loads_test.BigQueryFileLoadsIT) ... SKIP:
>>> IT is skipped because --test-pipeline-options is not specified
>>> test_one_job_fails_all_jobs_fail
>>> (apache_beam.io.gcp.bigquery_file_loads_test.BigQueryFileLoadsIT) ... SKIP:
>>> IT is skipped because --test-pipeline-options is not specified
>>> test_records_traverse_transform_with_mocks
>>> (apache_beam.io.gcp.bigquery_file_loads_test.TestBigQueryFileLoads) ...
>>>
>>> output ends here, would expect a failed or ok here.
>>>
>>>
>>> Afterwards no progress - even waiting for hours. Any idea, what might be
>>> causing this? Do I need to add some GCP properties for this task ?
>>>
>>> Any ideas, what I am doing wrong?
>>>
>>> best,
>>>
>>> michel
>>>
>>>

Re: Build blocking on

[Udi Meiri]

It shouldn't stall. That's a bug.
OTOH, I never use the `build` target.
I'll try running that myself.

On Mon, Mar 25, 2019, 07:24 Michael Luckey  wrote:

> Hi,
>
> trying to run './gradlew build' on vanilla setup, my build consistently
> stalls during execution of python gcp tests, e.g. on both of
> - > :beam-sdks-python:testPy2Gcp
> - > :beam-sdks-python-test-suites-tox-py35:testPy35Gcp
>
> Console output:
>  snip 
> test_big_query_standard_sql
> (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT)
> ... SKIP: IT is skipped because --test-pipeline-options is not specified
> test_big_query_standard_sql_kms_key
> (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT)
> ... SKIP: This test requires BQ Dataflow native source support for KMS,
> which is not available yet.
> test_multiple_destinations_transform
> (apache_beam.io.gcp.bigquery_file_loads_test.BigQueryFileLoadsIT) ... SKIP:
> IT is skipped because --test-pipeline-options is not specified
> test_one_job_fails_all_jobs_fail
> (apache_beam.io.gcp.bigquery_file_loads_test.BigQueryFileLoadsIT) ... SKIP:
> IT is skipped because --test-pipeline-options is not specified
> test_records_traverse_transform_with_mocks
> (apache_beam.io.gcp.bigquery_file_loads_test.TestBigQueryFileLoads) ...
>
> output ends here, would expect a failed or ok here.
>
>
> Afterwards no progress - even waiting for hours. Any idea, what might be
> causing this? Do I need to add some GCP properties for this task ?
>
> Any ideas, what I am doing wrong?
>
> best,
>
> michel
>
>


smime.p7s
Description: S/MIME Cryptographic Signature

Re: Build blocking on

[Udi Meiri]

Okay, `./gradlew build` failed pretty quickly for me:

> Task :beam-sdks-go:resolveBuildDependencies FAILED
cloud.google.com/go: commit='4f6c921ec566a33844f4e7879b31cd8575a6982d',
urls=[https://code.googlesource.com/gocloud] does not exist in
/usr/local/google/home/ehudm/.gradle/go/repo/
cloud.google.com/go/625660c387d9403fde4d73cacaf2d2ac, updating will be
performed.

https://gradle.com/s/x5zqbc5zwd3bg

(Now I remember why I stopped using `build` :/)

On Mon, Mar 25, 2019 at 5:30 PM Udi Meiri  wrote:

> It shouldn't stall. That's a bug.
> OTOH, I never use the `build` target.
> I'll try running that myself.
>
> On Mon, Mar 25, 2019, 07:24 Michael Luckey  wrote:
>
>> Hi,
>>
>> trying to run './gradlew build' on vanilla setup, my build consistently
>> stalls during execution of python gcp tests, e.g. on both of
>> - > :beam-sdks-python:testPy2Gcp
>> - > :beam-sdks-python-test-suites-tox-py35:testPy35Gcp
>>
>> Console output:
>>  snip 
>> test_big_query_standard_sql
>> (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT)
>> ... SKIP: IT is skipped because --test-pipeline-options is not specified
>> test_big_query_standard_sql_kms_key
>> (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT)
>> ... SKIP: This test requires BQ Dataflow native source support for KMS,
>> which is not available yet.
>> test_multiple_destinations_transform
>> (apache_beam.io.gcp.bigquery_file_loads_test.BigQueryFileLoadsIT) ... SKIP:
>> IT is skipped because --test-pipeline-options is not specified
>> test_one_job_fails_all_jobs_fail
>> (apache_beam.io.gcp.bigquery_file_loads_test.BigQueryFileLoadsIT) ... SKIP:
>> IT is skipped because --test-pipeline-options is not specified
>> test_records_traverse_transform_with_mocks
>> (apache_beam.io.gcp.bigquery_file_loads_test.TestBigQueryFileLoads) ...
>>
>> output ends here, would expect a failed or ok here.
>>
>>
>> Afterwards no progress - even waiting for hours. Any idea, what might be
>> causing this? Do I need to add some GCP properties for this task ?
>>
>> Any ideas, what I am doing wrong?
>>
>> best,
>>
>> michel
>>
>>


smime.p7s
Description: S/MIME Cryptographic Signature

Re: Beam Dependency Check Report (2019-03-23)

[Chamikara Jayalath]

Thanks for fixing this Yifan.

Also, a kind reminder to everybody to pay attention to dependency reports
and claim any JIRAs for dependencies that you would like to see upgraded.

Thanks,
Cham

On Sun, Mar 24, 2019 at 10:46 PM Yifan Zou  wrote:

> The job failed because of PyYAML load(file) is deprecated. I have PR-8123
> to fix that. The job passed with the PR and the report was sent to myself
> just for verification purpose. You can see the latest report in here:
>
>
> https://builds.apache.org/view/A-D/view/Beam/view/All/job/beam_Dependency_Check/189/artifact/src/build/dependencyUpdates/beam-dependency-check-report.html
>
> On Fri, Mar 22, 2019 at 6:09 PM Apache Jenkins Server <
> jenk...@builds.apache.org> wrote:
>
>> ERROR: File
>> 'src/build/dependencyUpdates/beam-dependency-check-report.html' does not
>> exist
>
>

Re: Writing bytes to BigQuery with beam

[Chamikara Jayalath]

On Mon, Mar 25, 2019 at 2:16 PM Pablo Estrada  wrote:

> +Chamikara Jayalath  with the new BigQuery sink,
> schema autodetection is supported (it's a very simple thing to have). Do
> you think we should not have it?
> Best
> -P.
>

Ah good to know. But IMO users should be able to write to existing tables
without specifying a schema (when CEATE_DISPOSITION is CREATE_NEVER for
example). How do users enable schema auto-detection ? Probably this should
not be enabled by default and we should clearly advertise that bytes type
is not supported (or support it with extra information). Just my 2 cents.

Thanks,
Cham


>
> On Mon, Mar 25, 2019 at 11:01 AM Chamikara Jayalath 
> wrote:
>
>>
>>
>> On Mon, Mar 25, 2019 at 2:03 AM Juta Staes  wrote:
>>
>>>
>>> On Mon, 25 Mar 2019 at 06:15, Valentyn Tymofieiev 
>>> wrote:
>>>
 We received feedback on
 https://issuetracker.google.com/issues/129006689 - BQ developers say
 that schema identification is done and they discourage to use schema
 autodetection in tables using BYTES. In light of this, I think may be fair
 to recommend Beam users to specify BQ schemas as well when they interact
 with BQ, and call out that writing binary data to BQ will likely fail
 unless schema is specified. Does that make sense?

>>>
>>> Given that schema autodetect does not work for bytes I think it is
>>> indeed a good solution to require users to specify BQ schemas as well when
>>> they write to BQ
>>>
>>> So new summary:
>>> 1. Beam will base64-encode raw bytes, before passing them to BQ over
>>> rest API. This will be a change in behavior for Python 2 (for good reasons).
>>> 2. When reading data from BQ, all fields of type BYTES will be
>>> base64-decoded.
>>> 3. Beam will send an API call to BigQuery to get table schema, whenever
>>> schema is not supplied, to work around
>>> https://issuetracker.google.com/issues/129006689. Beam will require
>>> users to specify the schema when writing bytes to BQ.
>>>
>>
>> I'm not sure why we reached this conclusion. We (Beam) does not use BQ
>> schema auto detection feature currently.  So why not just send an API
>> signal to get the schema when users are writing to existing tables ? Also,
>> even if we decide to support schema auto detection in the future we will
>> not be able to support this for BYTEs type (due to the restriction by BQ).
>>
>>
>>> Thanks all for your input on this!
>>> Juta
>>>
>>>

Re: Writing bytes to BigQuery with beam

[Pablo Estrada]

+Chamikara Jayalath  with the new BigQuery sink,
schema autodetection is supported (it's a very simple thing to have). Do
you think we should not have it?
Best
-P.

On Mon, Mar 25, 2019 at 11:01 AM Chamikara Jayalath 
wrote:

>
>
> On Mon, Mar 25, 2019 at 2:03 AM Juta Staes  wrote:
>
>>
>> On Mon, 25 Mar 2019 at 06:15, Valentyn Tymofieiev 
>> wrote:
>>
>>> We received feedback on https://issuetracker.google.com/issues/129006689 -
>>> BQ developers say that schema identification is done and they discourage to
>>> use schema autodetection in tables using BYTES. In light of this, I think
>>> may be fair to recommend Beam users to specify BQ schemas as well when they
>>> interact with BQ, and call out that writing binary data to BQ will likely
>>> fail unless schema is specified. Does that make sense?
>>>
>>
>> Given that schema autodetect does not work for bytes I think it is indeed
>> a good solution to require users to specify BQ schemas as well when they
>> write to BQ
>>
>> So new summary:
>> 1. Beam will base64-encode raw bytes, before passing them to BQ over rest
>> API. This will be a change in behavior for Python 2 (for good reasons).
>> 2. When reading data from BQ, all fields of type BYTES will be
>> base64-decoded.
>> 3. Beam will send an API call to BigQuery to get table schema, whenever
>> schema is not supplied, to work around
>> https://issuetracker.google.com/issues/129006689. Beam will require
>> users to specify the schema when writing bytes to BQ.
>>
>
> I'm not sure why we reached this conclusion. We (Beam) does not use BQ
> schema auto detection feature currently.  So why not just send an API
> signal to get the schema when users are writing to existing tables ? Also,
> even if we decide to support schema auto detection in the future we will
> not be able to support this for BYTEs type (due to the restriction by BQ).
>
>
>> Thanks all for your input on this!
>> Juta
>>
>>

Re: Frequent failures on beam8

[Mikhail Gryzykhin]

Yifan is looking into this.

On Mon, Mar 25, 2019 at 1:55 PM Boyuan Zhang  wrote:

> Hey all,
>
> Could anyone help take a look at beam8
> ? Seems like there are
> many tests failed on beam8 owing to infra problems.
>
> Thanks!
>

Frequent failures on beam8

[Boyuan Zhang]

Hey all,

Could anyone help take a look at beam8
? Seems like there are
many tests failed on beam8 owing to infra problems.

Thanks!

Re: [ANNOUNCE] New committer announcement: Mark Liu

[Melissa Pashniak]

Congrats Mark!


On Mon, Mar 25, 2019 at 12:14 PM Ankur Goenka  wrote:

> Congratulations Mark!
>
> On Mon, Mar 25, 2019 at 12:04 PM Jason Kuster 
> wrote:
>
>> Wonderful, congrats Mark!
>>
>> On Mon, Mar 25, 2019 at 11:30 AM Alan Myrvold 
>> wrote:
>>
>>> congratulations, Mark!!!
>>>
>>> On Mon, Mar 25, 2019 at 10:05 AM Ruoyun Huang  wrote:
>>>
 Congratulations Mark!

 On Mon, Mar 25, 2019 at 9:31 AM Udi Meiri  wrote:

> Congrats Mark!
>
> On Mon, Mar 25, 2019 at 9:24 AM Ahmet Altay  wrote:
>
>> Congratulations, Mark! 🎉
>>
>> On Mon, Mar 25, 2019 at 7:24 AM Tim Robertson <
>> timrobertson...@gmail.com> wrote:
>>
>>> Congratulations Mark!
>>>
>>>
>>> On Mon, Mar 25, 2019 at 3:18 PM Michael Luckey 
>>> wrote:
>>>
 Nice! Congratulations, Mark.

 On Mon, Mar 25, 2019 at 2:42 PM Katarzyna Kucharczyk <
 ka.kucharc...@gmail.com> wrote:

> Congratulations, Mark! 🎉
>
> On Mon, Mar 25, 2019 at 11:24 AM Gleb Kanterov 
> wrote:
>
>> Congratulations!
>>
>> On Mon, Mar 25, 2019 at 10:23 AM Łukasz Gajowy <
>> lgaj...@apache.org> wrote:
>>
>>> Congrats! :)
>>>
>>>
>>>
>>> pon., 25 mar 2019 o 08:11 Aizhamal Nurmamat kyzy <
>>> aizha...@google.com> napisał(a):
>>>
 Congratulations, Mark!

 On Sun, Mar 24, 2019 at 23:18 Pablo Estrada 
 wrote:

> Yeaah  Mark! : ) Congrats : D
>
> On Sun, Mar 24, 2019 at 10:32 PM Yifan Zou <
> yifan...@google.com> wrote:
>
>> Congratulations Mark!
>>
>> On Sun, Mar 24, 2019 at 10:25 PM Connell O'Callaghan <
>> conne...@google.com> wrote:
>>
>>> Well done congratulations Mark!!!
>>>
>>> On Sun, Mar 24, 2019 at 10:17 PM Robert Burke <
>>> rob...@frantil.com> wrote:
>>>
 Congratulations Mark! 🎉

 On Sun, Mar 24, 2019, 10:08 PM Valentyn Tymofieiev <
 valen...@google.com> wrote:

> Congratulations, Mark!
>
> Thanks for your contributions, in particular for your
> efforts to parallelize test execution for Python SDK and 
> increase the speed
> of Python precommit checks.
>
> On Sun, Mar 24, 2019 at 9:40 PM Kenneth Knowles <
> k...@apache.org> wrote:
>
>> Hi all,
>>
>> Please join me and the rest of the Beam PMC in welcoming
>> a new committer: Mark Liu.
>>
>> Mark has been contributing to Beam since late 2016! He
>> has proposed 100+ pull requests. Mark was instrumental in 
>> expanding test
>> and infrastructure coverage, especially for Python. In
>> consideration of Mark's contributions, the Beam PMC trusts 
>> Mark with the
>> responsibilities of a Beam committer [1].
>>
>> Thank you, Mark, for your contributions.
>>
>> Kenn
>>
>> [1] https://beam.apache.org/contribute/become-a-committer
>> /#an-apache-beam-committer
>>
> --

 *Aizhamal Nurmamat kyzy*

 Open Source Program Manager

 646-355-9740 Mobile

 601 North 34th Street, Seattle, WA 98103



>>
>> --
>> Cheers,
>> Gleb
>>
>

 --
 
 Ruoyun  Huang


>>
>> --
>> ---
>> Jason Kuster
>> Apache Beam / Google Cloud Dataflow
>>
>> See something? Say something. go/jasonkuster-feedback
>> 
>>
>

Re: [ANNOUNCE] New committer announcement: Mark Liu

[Ankur Goenka]

Congratulations Mark!

On Mon, Mar 25, 2019 at 12:04 PM Jason Kuster 
wrote:

> Wonderful, congrats Mark!
>
> On Mon, Mar 25, 2019 at 11:30 AM Alan Myrvold  wrote:
>
>> congratulations, Mark!!!
>>
>> On Mon, Mar 25, 2019 at 10:05 AM Ruoyun Huang  wrote:
>>
>>> Congratulations Mark!
>>>
>>> On Mon, Mar 25, 2019 at 9:31 AM Udi Meiri  wrote:
>>>
 Congrats Mark!

 On Mon, Mar 25, 2019 at 9:24 AM Ahmet Altay  wrote:

> Congratulations, Mark! 🎉
>
> On Mon, Mar 25, 2019 at 7:24 AM Tim Robertson <
> timrobertson...@gmail.com> wrote:
>
>> Congratulations Mark!
>>
>>
>> On Mon, Mar 25, 2019 at 3:18 PM Michael Luckey 
>> wrote:
>>
>>> Nice! Congratulations, Mark.
>>>
>>> On Mon, Mar 25, 2019 at 2:42 PM Katarzyna Kucharczyk <
>>> ka.kucharc...@gmail.com> wrote:
>>>
 Congratulations, Mark! 🎉

 On Mon, Mar 25, 2019 at 11:24 AM Gleb Kanterov 
 wrote:

> Congratulations!
>
> On Mon, Mar 25, 2019 at 10:23 AM Łukasz Gajowy 
> wrote:
>
>> Congrats! :)
>>
>>
>>
>> pon., 25 mar 2019 o 08:11 Aizhamal Nurmamat kyzy <
>> aizha...@google.com> napisał(a):
>>
>>> Congratulations, Mark!
>>>
>>> On Sun, Mar 24, 2019 at 23:18 Pablo Estrada 
>>> wrote:
>>>
 Yeaah  Mark! : ) Congrats : D

 On Sun, Mar 24, 2019 at 10:32 PM Yifan Zou 
 wrote:

> Congratulations Mark!
>
> On Sun, Mar 24, 2019 at 10:25 PM Connell O'Callaghan <
> conne...@google.com> wrote:
>
>> Well done congratulations Mark!!!
>>
>> On Sun, Mar 24, 2019 at 10:17 PM Robert Burke <
>> rob...@frantil.com> wrote:
>>
>>> Congratulations Mark! 🎉
>>>
>>> On Sun, Mar 24, 2019, 10:08 PM Valentyn Tymofieiev <
>>> valen...@google.com> wrote:
>>>
 Congratulations, Mark!

 Thanks for your contributions, in particular for your
 efforts to parallelize test execution for Python SDK and 
 increase the speed
 of Python precommit checks.

 On Sun, Mar 24, 2019 at 9:40 PM Kenneth Knowles <
 k...@apache.org> wrote:

> Hi all,
>
> Please join me and the rest of the Beam PMC in welcoming
> a new committer: Mark Liu.
>
> Mark has been contributing to Beam since late 2016! He has
> proposed 100+ pull requests. Mark was instrumental in 
> expanding test and
> infrastructure coverage, especially for Python. In
> consideration of Mark's contributions, the Beam PMC trusts 
> Mark with the
> responsibilities of a Beam committer [1].
>
> Thank you, Mark, for your contributions.
>
> Kenn
>
> [1] https://beam.apache.org/contribute/become-a-committer/
> #an-apache-beam-committer
>
 --
>>>
>>> *Aizhamal Nurmamat kyzy*
>>>
>>> Open Source Program Manager
>>>
>>> 646-355-9740 Mobile
>>>
>>> 601 North 34th Street, Seattle, WA 98103
>>>
>>>
>>>
>
> --
> Cheers,
> Gleb
>

>>>
>>> --
>>> 
>>> Ruoyun  Huang
>>>
>>>
>
> --
> ---
> Jason Kuster
> Apache Beam / Google Cloud Dataflow
>
> See something? Say something. go/jasonkuster-feedback
> 
>

Re: [ANNOUNCE] New committer announcement: Mark Liu

[Jason Kuster]

Wonderful, congrats Mark!

On Mon, Mar 25, 2019 at 11:30 AM Alan Myrvold  wrote:

> congratulations, Mark!!!
>
> On Mon, Mar 25, 2019 at 10:05 AM Ruoyun Huang  wrote:
>
>> Congratulations Mark!
>>
>> On Mon, Mar 25, 2019 at 9:31 AM Udi Meiri  wrote:
>>
>>> Congrats Mark!
>>>
>>> On Mon, Mar 25, 2019 at 9:24 AM Ahmet Altay  wrote:
>>>
 Congratulations, Mark! 🎉

 On Mon, Mar 25, 2019 at 7:24 AM Tim Robertson <
 timrobertson...@gmail.com> wrote:

> Congratulations Mark!
>
>
> On Mon, Mar 25, 2019 at 3:18 PM Michael Luckey 
> wrote:
>
>> Nice! Congratulations, Mark.
>>
>> On Mon, Mar 25, 2019 at 2:42 PM Katarzyna Kucharczyk <
>> ka.kucharc...@gmail.com> wrote:
>>
>>> Congratulations, Mark! 🎉
>>>
>>> On Mon, Mar 25, 2019 at 11:24 AM Gleb Kanterov 
>>> wrote:
>>>
 Congratulations!

 On Mon, Mar 25, 2019 at 10:23 AM Łukasz Gajowy 
 wrote:

> Congrats! :)
>
>
>
> pon., 25 mar 2019 o 08:11 Aizhamal Nurmamat kyzy <
> aizha...@google.com> napisał(a):
>
>> Congratulations, Mark!
>>
>> On Sun, Mar 24, 2019 at 23:18 Pablo Estrada 
>> wrote:
>>
>>> Yeaah  Mark! : ) Congrats : D
>>>
>>> On Sun, Mar 24, 2019 at 10:32 PM Yifan Zou 
>>> wrote:
>>>
 Congratulations Mark!

 On Sun, Mar 24, 2019 at 10:25 PM Connell O'Callaghan <
 conne...@google.com> wrote:

> Well done congratulations Mark!!!
>
> On Sun, Mar 24, 2019 at 10:17 PM Robert Burke <
> rob...@frantil.com> wrote:
>
>> Congratulations Mark! 🎉
>>
>> On Sun, Mar 24, 2019, 10:08 PM Valentyn Tymofieiev <
>> valen...@google.com> wrote:
>>
>>> Congratulations, Mark!
>>>
>>> Thanks for your contributions, in particular for your
>>> efforts to parallelize test execution for Python SDK and 
>>> increase the speed
>>> of Python precommit checks.
>>>
>>> On Sun, Mar 24, 2019 at 9:40 PM Kenneth Knowles <
>>> k...@apache.org> wrote:
>>>
 Hi all,

 Please join me and the rest of the Beam PMC in welcoming a
 new committer: Mark Liu.

 Mark has been contributing to Beam since late 2016! He has
 proposed 100+ pull requests. Mark was instrumental in 
 expanding test and
 infrastructure coverage, especially for Python. In
 consideration of Mark's contributions, the Beam PMC trusts 
 Mark with the
 responsibilities of a Beam committer [1].

 Thank you, Mark, for your contributions.

 Kenn

 [1] https://beam.apache.org/contribute/become-a-committer/
 #an-apache-beam-committer

>>> --
>>
>> *Aizhamal Nurmamat kyzy*
>>
>> Open Source Program Manager
>>
>> 646-355-9740 Mobile
>>
>> 601 North 34th Street, Seattle, WA 98103
>>
>>
>>

 --
 Cheers,
 Gleb

>>>
>>
>> --
>> 
>> Ruoyun  Huang
>>
>>

-- 
---
Jason Kuster
Apache Beam / Google Cloud Dataflow

See something? Say something. go/jasonkuster-feedback

Re: [ANNOUNCE] New committer announcement: Mark Liu

[Alan Myrvold]

congratulations, Mark!!!

On Mon, Mar 25, 2019 at 10:05 AM Ruoyun Huang  wrote:

> Congratulations Mark!
>
> On Mon, Mar 25, 2019 at 9:31 AM Udi Meiri  wrote:
>
>> Congrats Mark!
>>
>> On Mon, Mar 25, 2019 at 9:24 AM Ahmet Altay  wrote:
>>
>>> Congratulations, Mark! 🎉
>>>
>>> On Mon, Mar 25, 2019 at 7:24 AM Tim Robertson 
>>> wrote:
>>>
 Congratulations Mark!


 On Mon, Mar 25, 2019 at 3:18 PM Michael Luckey 
 wrote:

> Nice! Congratulations, Mark.
>
> On Mon, Mar 25, 2019 at 2:42 PM Katarzyna Kucharczyk <
> ka.kucharc...@gmail.com> wrote:
>
>> Congratulations, Mark! 🎉
>>
>> On Mon, Mar 25, 2019 at 11:24 AM Gleb Kanterov 
>> wrote:
>>
>>> Congratulations!
>>>
>>> On Mon, Mar 25, 2019 at 10:23 AM Łukasz Gajowy 
>>> wrote:
>>>
 Congrats! :)



 pon., 25 mar 2019 o 08:11 Aizhamal Nurmamat kyzy <
 aizha...@google.com> napisał(a):

> Congratulations, Mark!
>
> On Sun, Mar 24, 2019 at 23:18 Pablo Estrada 
> wrote:
>
>> Yeaah  Mark! : ) Congrats : D
>>
>> On Sun, Mar 24, 2019 at 10:32 PM Yifan Zou 
>> wrote:
>>
>>> Congratulations Mark!
>>>
>>> On Sun, Mar 24, 2019 at 10:25 PM Connell O'Callaghan <
>>> conne...@google.com> wrote:
>>>
 Well done congratulations Mark!!!

 On Sun, Mar 24, 2019 at 10:17 PM Robert Burke <
 rob...@frantil.com> wrote:

> Congratulations Mark! 🎉
>
> On Sun, Mar 24, 2019, 10:08 PM Valentyn Tymofieiev <
> valen...@google.com> wrote:
>
>> Congratulations, Mark!
>>
>> Thanks for your contributions, in particular for your efforts
>> to parallelize test execution for Python SDK and increase the 
>> speed of
>> Python precommit checks.
>>
>> On Sun, Mar 24, 2019 at 9:40 PM Kenneth Knowles <
>> k...@apache.org> wrote:
>>
>>> Hi all,
>>>
>>> Please join me and the rest of the Beam PMC in welcoming a
>>> new committer: Mark Liu.
>>>
>>> Mark has been contributing to Beam since late 2016! He has
>>> proposed 100+ pull requests. Mark was instrumental in expanding 
>>> test and
>>> infrastructure coverage, especially for Python. In
>>> consideration of Mark's contributions, the Beam PMC trusts Mark 
>>> with the
>>> responsibilities of a Beam committer [1].
>>>
>>> Thank you, Mark, for your contributions.
>>>
>>> Kenn
>>>
>>> [1] https://beam.apache.org/contribute/become-a-committer/
>>> #an-apache-beam-committer
>>>
>> --
>
> *Aizhamal Nurmamat kyzy*
>
> Open Source Program Manager
>
> 646-355-9740 Mobile
>
> 601 North 34th Street, Seattle, WA 98103
>
>
>
>>>
>>> --
>>> Cheers,
>>> Gleb
>>>
>>
>
> --
> 
> Ruoyun  Huang
>
>

Re: Writing bytes to BigQuery with beam

[Chamikara Jayalath]

On Mon, Mar 25, 2019 at 2:03 AM Juta Staes  wrote:

>
> On Mon, 25 Mar 2019 at 06:15, Valentyn Tymofieiev 
> wrote:
>
>> We received feedback on https://issuetracker.google.com/issues/129006689 -
>> BQ developers say that schema identification is done and they discourage to
>> use schema autodetection in tables using BYTES. In light of this, I think
>> may be fair to recommend Beam users to specify BQ schemas as well when they
>> interact with BQ, and call out that writing binary data to BQ will likely
>> fail unless schema is specified. Does that make sense?
>>
>
> Given that schema autodetect does not work for bytes I think it is indeed
> a good solution to require users to specify BQ schemas as well when they
> write to BQ
>
> So new summary:
> 1. Beam will base64-encode raw bytes, before passing them to BQ over rest
> API. This will be a change in behavior for Python 2 (for good reasons).
> 2. When reading data from BQ, all fields of type BYTES will be
> base64-decoded.
> 3. Beam will send an API call to BigQuery to get table schema, whenever
> schema is not supplied, to work around
> https://issuetracker.google.com/issues/129006689. Beam will require users
> to specify the schema when writing bytes to BQ.
>

I'm not sure why we reached this conclusion. We (Beam) does not use BQ
schema auto detection feature currently.  So why not just send an API
signal to get the schema when users are writing to existing tables ? Also,
even if we decide to support schema auto detection in the future we will
not be able to support this for BYTEs type (due to the restriction by BQ).


> Thanks all for your input on this!
> Juta
>
>

Re: [ANNOUNCE] New committer announcement: Mark Liu

[Chamikara Jayalath]

Congrats Mark!

On Mon, Mar 25, 2019 at 10:50 AM Alexey Romanenko 
wrote:

> Congratulations, Mark!
>
> On 25 Mar 2019, at 18:36, Mark Liu  wrote:
>
> Thank you all! It's a great pleasure to work on Beam!
>
> Mark
>
> On Mon, Mar 25, 2019 at 10:18 AM Robin Qiu  wrote:
>
>> Congratulations, Mark!
>>
>> On Mon, Mar 25, 2019 at 9:31 AM Udi Meiri  wrote:
>>
>>> Congrats Mark!
>>>
>>> On Mon, Mar 25, 2019 at 9:24 AM Ahmet Altay  wrote:
>>>
 Congratulations, Mark! 🎉

 On Mon, Mar 25, 2019 at 7:24 AM Tim Robertson <
 timrobertson...@gmail.com> wrote:

> Congratulations Mark!
>
>
> On Mon, Mar 25, 2019 at 3:18 PM Michael Luckey 
> wrote:
>
>> Nice! Congratulations, Mark.
>>
>> On Mon, Mar 25, 2019 at 2:42 PM Katarzyna Kucharczyk <
>> ka.kucharc...@gmail.com> wrote:
>>
>>> Congratulations, Mark! 🎉
>>>
>>> On Mon, Mar 25, 2019 at 11:24 AM Gleb Kanterov 
>>> wrote:
>>>
 Congratulations!

 On Mon, Mar 25, 2019 at 10:23 AM Łukasz Gajowy 
 wrote:

> Congrats! :)
>
>
>
> pon., 25 mar 2019 o 08:11 Aizhamal Nurmamat kyzy <
> aizha...@google.com> napisał(a):
>
>> Congratulations, Mark!
>>
>> On Sun, Mar 24, 2019 at 23:18 Pablo Estrada 
>> wrote:
>>
>>> Yeaah  Mark! : ) Congrats : D
>>>
>>> On Sun, Mar 24, 2019 at 10:32 PM Yifan Zou 
>>> wrote:
>>>
 Congratulations Mark!

 On Sun, Mar 24, 2019 at 10:25 PM Connell O'Callaghan <
 conne...@google.com> wrote:

> Well done congratulations Mark!!!
>
> On Sun, Mar 24, 2019 at 10:17 PM Robert Burke <
> rob...@frantil.com> wrote:
>
>> Congratulations Mark! 🎉
>>
>> On Sun, Mar 24, 2019, 10:08 PM Valentyn Tymofieiev <
>> valen...@google.com> wrote:
>>
>>> Congratulations, Mark!
>>>
>>> Thanks for your contributions, in particular for your
>>> efforts to parallelize test execution for Python SDK and 
>>> increase the speed
>>> of Python precommit checks.
>>>
>>> On Sun, Mar 24, 2019 at 9:40 PM Kenneth Knowles <
>>> k...@apache.org> wrote:
>>>
 Hi all,

 Please join me and the rest of the Beam PMC in welcoming a
 new committer: Mark Liu.

 Mark has been contributing to Beam since late 2016! He has
 proposed 100+ pull requests. Mark was instrumental in 
 expanding test and
 infrastructure coverage, especially for Python. In
 consideration of Mark's contributions, the Beam PMC trusts 
 Mark with the
 responsibilities of a Beam committer [1].

 Thank you, Mark, for your contributions.

 Kenn

 [1] https://beam.apache.org/contribute/become-a-committer/
 #an-apache-beam-committer

>>> --
>>
>> *Aizhamal Nurmamat kyzy*
>> Open Source Program Manager
>> 646-355-9740 Mobile
>> 601 North 34th Street, Seattle, WA 98103
>>
>>

 --
 Cheers,
 Gleb

>>>
>

Re: [ANNOUNCE] New committer announcement: Mark Liu

[Alexey Romanenko]

Congratulations, Mark!

> On 25 Mar 2019, at 18:36, Mark Liu  wrote:
> 
> Thank you all! It's a great pleasure to work on Beam!
> 
> Mark
> 
> On Mon, Mar 25, 2019 at 10:18 AM Robin Qiu  > wrote:
> Congratulations, Mark!
> 
> On Mon, Mar 25, 2019 at 9:31 AM Udi Meiri  > wrote:
> Congrats Mark!
> 
> On Mon, Mar 25, 2019 at 9:24 AM Ahmet Altay  > wrote:
> Congratulations, Mark! 🎉
> 
> On Mon, Mar 25, 2019 at 7:24 AM Tim Robertson  > wrote:
> Congratulations Mark!
> 
> 
> On Mon, Mar 25, 2019 at 3:18 PM Michael Luckey  > wrote:
> Nice! Congratulations, Mark.
> 
> On Mon, Mar 25, 2019 at 2:42 PM Katarzyna Kucharczyk  > wrote:
> Congratulations, Mark! 🎉
> 
> On Mon, Mar 25, 2019 at 11:24 AM Gleb Kanterov  > wrote:
> Congratulations!
> 
> On Mon, Mar 25, 2019 at 10:23 AM Łukasz Gajowy  > wrote:
> Congrats! :)
> 
> 
> 
> pon., 25 mar 2019 o 08:11 Aizhamal Nurmamat kyzy  > napisał(a):
> Congratulations, Mark!
> 
> On Sun, Mar 24, 2019 at 23:18 Pablo Estrada  > wrote:
> Yeaah  Mark! : ) Congrats : D
> 
> On Sun, Mar 24, 2019 at 10:32 PM Yifan Zou  > wrote:
> Congratulations Mark!
> 
> On Sun, Mar 24, 2019 at 10:25 PM Connell O'Callaghan  > wrote:
> Well done congratulations Mark!!! 
> 
> On Sun, Mar 24, 2019 at 10:17 PM Robert Burke  > wrote:
> Congratulations Mark! 🎉
> 
> On Sun, Mar 24, 2019, 10:08 PM Valentyn Tymofieiev  > wrote:
> Congratulations, Mark!
> 
> Thanks for your contributions, in particular for your efforts to parallelize 
> test execution for Python SDK and increase the speed of Python precommit 
> checks. 
> 
> On Sun, Mar 24, 2019 at 9:40 PM Kenneth Knowles  > wrote:
> Hi all,
> 
> Please join me and the rest of the Beam PMC in welcoming a new committer: 
> Mark Liu.
> 
> Mark has been contributing to Beam since late 2016! He has proposed 100+ pull 
> requests. Mark was instrumental in expanding test and infrastructure 
> coverage, especially for Python. In consideration of Mark's contributions, 
> the Beam PMC trusts Mark with the responsibilities of a Beam committer [1].
> 
> Thank you, Mark, for your contributions.
> 
> Kenn
> 
> [1] 
> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>  
> --
>  
> 
> Aizhamal Nurmamat kyzy
> Open Source Program Manager
> 646-355-9740 Mobile
> 601 North 34th Street, Seattle, WA 98103
> 
> 
> 
> 
> -- 
> Cheers,
> Gleb

Re: [ANNOUNCE] New committer announcement: Mark Liu

[Mark Liu]

Thank you all! It's a great pleasure to work on Beam!

Mark

On Mon, Mar 25, 2019 at 10:18 AM Robin Qiu  wrote:

> Congratulations, Mark!
>
> On Mon, Mar 25, 2019 at 9:31 AM Udi Meiri  wrote:
>
>> Congrats Mark!
>>
>> On Mon, Mar 25, 2019 at 9:24 AM Ahmet Altay  wrote:
>>
>>> Congratulations, Mark! 🎉
>>>
>>> On Mon, Mar 25, 2019 at 7:24 AM Tim Robertson 
>>> wrote:
>>>
 Congratulations Mark!


 On Mon, Mar 25, 2019 at 3:18 PM Michael Luckey 
 wrote:

> Nice! Congratulations, Mark.
>
> On Mon, Mar 25, 2019 at 2:42 PM Katarzyna Kucharczyk <
> ka.kucharc...@gmail.com> wrote:
>
>> Congratulations, Mark! 🎉
>>
>> On Mon, Mar 25, 2019 at 11:24 AM Gleb Kanterov 
>> wrote:
>>
>>> Congratulations!
>>>
>>> On Mon, Mar 25, 2019 at 10:23 AM Łukasz Gajowy 
>>> wrote:
>>>
 Congrats! :)



 pon., 25 mar 2019 o 08:11 Aizhamal Nurmamat kyzy <
 aizha...@google.com> napisał(a):

> Congratulations, Mark!
>
> On Sun, Mar 24, 2019 at 23:18 Pablo Estrada 
> wrote:
>
>> Yeaah  Mark! : ) Congrats : D
>>
>> On Sun, Mar 24, 2019 at 10:32 PM Yifan Zou 
>> wrote:
>>
>>> Congratulations Mark!
>>>
>>> On Sun, Mar 24, 2019 at 10:25 PM Connell O'Callaghan <
>>> conne...@google.com> wrote:
>>>
 Well done congratulations Mark!!!

 On Sun, Mar 24, 2019 at 10:17 PM Robert Burke <
 rob...@frantil.com> wrote:

> Congratulations Mark! 🎉
>
> On Sun, Mar 24, 2019, 10:08 PM Valentyn Tymofieiev <
> valen...@google.com> wrote:
>
>> Congratulations, Mark!
>>
>> Thanks for your contributions, in particular for your efforts
>> to parallelize test execution for Python SDK and increase the 
>> speed of
>> Python precommit checks.
>>
>> On Sun, Mar 24, 2019 at 9:40 PM Kenneth Knowles <
>> k...@apache.org> wrote:
>>
>>> Hi all,
>>>
>>> Please join me and the rest of the Beam PMC in welcoming a
>>> new committer: Mark Liu.
>>>
>>> Mark has been contributing to Beam since late 2016! He has
>>> proposed 100+ pull requests. Mark was instrumental in expanding 
>>> test and
>>> infrastructure coverage, especially for Python. In
>>> consideration of Mark's contributions, the Beam PMC trusts Mark 
>>> with the
>>> responsibilities of a Beam committer [1].
>>>
>>> Thank you, Mark, for your contributions.
>>>
>>> Kenn
>>>
>>> [1] https://beam.apache.org/contribute/become-a-committer/
>>> #an-apache-beam-committer
>>>
>> --
>
> *Aizhamal Nurmamat kyzy*
>
> Open Source Program Manager
>
> 646-355-9740 Mobile
>
> 601 North 34th Street, Seattle, WA 98103
>
>
>
>>>
>>> --
>>> Cheers,
>>> Gleb
>>>
>>

Re: docs: java-dependencies

[Melissa Pashniak]

I was looking at this recently to try and figure out how to generate this
in a reliable way for Java, but hadn't come up with a clean solution yet.
For the related Python dependencies page [1], I recently wrote a script
that scrapes the dependencies and generates an HTML table that can be
copied into the markdown file, and I updated the page with deps info up to
2.11.0. I need to do some cleanup on the script, and then I will add it to
the repo. With that, it could be made part of the release process for the
Python side of things.

[1] https://beam.apache.org/documentation/sdks/python-dependencies/


On Sun, Mar 24, 2019 at 9:21 PM Kenneth Knowles  wrote:

> I had forgotten about that page. I think it is a good idea to include it
> in the release process. I would rephrase the page a little bit to make it
> clear that dependency conflicts are normal and expected in Java so this is
> an FYI page about the versions we test with. Users may have to pin to other
> versions, and that is normal too.
>
> We can scrape some of this from the gradle script source code. It would be
> better to use the output of some Gradle command to list the deps. But since
> Beam is a collection of many, many modules that could be a lot of trouble.
>
> Kenn
>
> On Sun, Mar 24, 2019 at 10:37 AM David Morávek  wrote:
>
>> Hello,
>>
>> I've run into java-dependencies page
>> , while
>> upgrading joda-time. This page is supposed to track dependencies across
>> releases, but the last entry is for 2.9 release.
>>
>> Do we still want to maintain this? If so, could we somehow include it in
>> the release process?
>>
>> Also we should add missing entries for 2.10 and 2.11, is there any script
>> I can use to generate this?
>>
>> D.
>>
>

Re: [ANNOUNCE] New committer announcement: Mark Liu

[Robin Qiu]

Congratulations, Mark!

On Mon, Mar 25, 2019 at 9:31 AM Udi Meiri  wrote:

> Congrats Mark!
>
> On Mon, Mar 25, 2019 at 9:24 AM Ahmet Altay  wrote:
>
>> Congratulations, Mark! 🎉
>>
>> On Mon, Mar 25, 2019 at 7:24 AM Tim Robertson 
>> wrote:
>>
>>> Congratulations Mark!
>>>
>>>
>>> On Mon, Mar 25, 2019 at 3:18 PM Michael Luckey 
>>> wrote:
>>>
 Nice! Congratulations, Mark.

 On Mon, Mar 25, 2019 at 2:42 PM Katarzyna Kucharczyk <
 ka.kucharc...@gmail.com> wrote:

> Congratulations, Mark! 🎉
>
> On Mon, Mar 25, 2019 at 11:24 AM Gleb Kanterov 
> wrote:
>
>> Congratulations!
>>
>> On Mon, Mar 25, 2019 at 10:23 AM Łukasz Gajowy 
>> wrote:
>>
>>> Congrats! :)
>>>
>>>
>>>
>>> pon., 25 mar 2019 o 08:11 Aizhamal Nurmamat kyzy <
>>> aizha...@google.com> napisał(a):
>>>
 Congratulations, Mark!

 On Sun, Mar 24, 2019 at 23:18 Pablo Estrada 
 wrote:

> Yeaah  Mark! : ) Congrats : D
>
> On Sun, Mar 24, 2019 at 10:32 PM Yifan Zou 
> wrote:
>
>> Congratulations Mark!
>>
>> On Sun, Mar 24, 2019 at 10:25 PM Connell O'Callaghan <
>> conne...@google.com> wrote:
>>
>>> Well done congratulations Mark!!!
>>>
>>> On Sun, Mar 24, 2019 at 10:17 PM Robert Burke <
>>> rob...@frantil.com> wrote:
>>>
 Congratulations Mark! 🎉

 On Sun, Mar 24, 2019, 10:08 PM Valentyn Tymofieiev <
 valen...@google.com> wrote:

> Congratulations, Mark!
>
> Thanks for your contributions, in particular for your efforts
> to parallelize test execution for Python SDK and increase the 
> speed of
> Python precommit checks.
>
> On Sun, Mar 24, 2019 at 9:40 PM Kenneth Knowles <
> k...@apache.org> wrote:
>
>> Hi all,
>>
>> Please join me and the rest of the Beam PMC in welcoming a
>> new committer: Mark Liu.
>>
>> Mark has been contributing to Beam since late 2016! He has
>> proposed 100+ pull requests. Mark was instrumental in expanding 
>> test and
>> infrastructure coverage, especially for Python. In
>> consideration of Mark's contributions, the Beam PMC trusts Mark 
>> with the
>> responsibilities of a Beam committer [1].
>>
>> Thank you, Mark, for your contributions.
>>
>> Kenn
>>
>> [1] https://beam.apache.org/contribute/become-a-committer/
>> #an-apache-beam-committer
>>
> --

 *Aizhamal Nurmamat kyzy*

 Open Source Program Manager

 646-355-9740 Mobile

 601 North 34th Street, Seattle, WA 98103



>>
>> --
>> Cheers,
>> Gleb
>>
>

Re: [ANNOUNCE] New committer announcement: Mark Liu

[Ruoyun Huang]

Congratulations Mark!

On Mon, Mar 25, 2019 at 9:31 AM Udi Meiri  wrote:

> Congrats Mark!
>
> On Mon, Mar 25, 2019 at 9:24 AM Ahmet Altay  wrote:
>
>> Congratulations, Mark! 🎉
>>
>> On Mon, Mar 25, 2019 at 7:24 AM Tim Robertson 
>> wrote:
>>
>>> Congratulations Mark!
>>>
>>>
>>> On Mon, Mar 25, 2019 at 3:18 PM Michael Luckey 
>>> wrote:
>>>
 Nice! Congratulations, Mark.

 On Mon, Mar 25, 2019 at 2:42 PM Katarzyna Kucharczyk <
 ka.kucharc...@gmail.com> wrote:

> Congratulations, Mark! 🎉
>
> On Mon, Mar 25, 2019 at 11:24 AM Gleb Kanterov 
> wrote:
>
>> Congratulations!
>>
>> On Mon, Mar 25, 2019 at 10:23 AM Łukasz Gajowy 
>> wrote:
>>
>>> Congrats! :)
>>>
>>>
>>>
>>> pon., 25 mar 2019 o 08:11 Aizhamal Nurmamat kyzy <
>>> aizha...@google.com> napisał(a):
>>>
 Congratulations, Mark!

 On Sun, Mar 24, 2019 at 23:18 Pablo Estrada 
 wrote:

> Yeaah  Mark! : ) Congrats : D
>
> On Sun, Mar 24, 2019 at 10:32 PM Yifan Zou 
> wrote:
>
>> Congratulations Mark!
>>
>> On Sun, Mar 24, 2019 at 10:25 PM Connell O'Callaghan <
>> conne...@google.com> wrote:
>>
>>> Well done congratulations Mark!!!
>>>
>>> On Sun, Mar 24, 2019 at 10:17 PM Robert Burke <
>>> rob...@frantil.com> wrote:
>>>
 Congratulations Mark! 🎉

 On Sun, Mar 24, 2019, 10:08 PM Valentyn Tymofieiev <
 valen...@google.com> wrote:

> Congratulations, Mark!
>
> Thanks for your contributions, in particular for your efforts
> to parallelize test execution for Python SDK and increase the 
> speed of
> Python precommit checks.
>
> On Sun, Mar 24, 2019 at 9:40 PM Kenneth Knowles <
> k...@apache.org> wrote:
>
>> Hi all,
>>
>> Please join me and the rest of the Beam PMC in welcoming a
>> new committer: Mark Liu.
>>
>> Mark has been contributing to Beam since late 2016! He has
>> proposed 100+ pull requests. Mark was instrumental in expanding 
>> test and
>> infrastructure coverage, especially for Python. In
>> consideration of Mark's contributions, the Beam PMC trusts Mark 
>> with the
>> responsibilities of a Beam committer [1].
>>
>> Thank you, Mark, for your contributions.
>>
>> Kenn
>>
>> [1] https://beam.apache.org/contribute/become-a-committer/
>> #an-apache-beam-committer
>>
> --

 *Aizhamal Nurmamat kyzy*

 Open Source Program Manager

 646-355-9740 Mobile

 601 North 34th Street, Seattle, WA 98103



>>
>> --
>> Cheers,
>> Gleb
>>
>

-- 

Ruoyun  Huang

Re: [ANNOUNCE] New committer announcement: Mark Liu

[Udi Meiri]

Congrats Mark!

On Mon, Mar 25, 2019 at 9:24 AM Ahmet Altay  wrote:

> Congratulations, Mark! 🎉
>
> On Mon, Mar 25, 2019 at 7:24 AM Tim Robertson 
> wrote:
>
>> Congratulations Mark!
>>
>>
>> On Mon, Mar 25, 2019 at 3:18 PM Michael Luckey 
>> wrote:
>>
>>> Nice! Congratulations, Mark.
>>>
>>> On Mon, Mar 25, 2019 at 2:42 PM Katarzyna Kucharczyk <
>>> ka.kucharc...@gmail.com> wrote:
>>>
 Congratulations, Mark! 🎉

 On Mon, Mar 25, 2019 at 11:24 AM Gleb Kanterov 
 wrote:

> Congratulations!
>
> On Mon, Mar 25, 2019 at 10:23 AM Łukasz Gajowy 
> wrote:
>
>> Congrats! :)
>>
>>
>>
>> pon., 25 mar 2019 o 08:11 Aizhamal Nurmamat kyzy 
>> napisał(a):
>>
>>> Congratulations, Mark!
>>>
>>> On Sun, Mar 24, 2019 at 23:18 Pablo Estrada 
>>> wrote:
>>>
 Yeaah  Mark! : ) Congrats : D

 On Sun, Mar 24, 2019 at 10:32 PM Yifan Zou 
 wrote:

> Congratulations Mark!
>
> On Sun, Mar 24, 2019 at 10:25 PM Connell O'Callaghan <
> conne...@google.com> wrote:
>
>> Well done congratulations Mark!!!
>>
>> On Sun, Mar 24, 2019 at 10:17 PM Robert Burke 
>> wrote:
>>
>>> Congratulations Mark! 🎉
>>>
>>> On Sun, Mar 24, 2019, 10:08 PM Valentyn Tymofieiev <
>>> valen...@google.com> wrote:
>>>
 Congratulations, Mark!

 Thanks for your contributions, in particular for your efforts
 to parallelize test execution for Python SDK and increase the 
 speed of
 Python precommit checks.

 On Sun, Mar 24, 2019 at 9:40 PM Kenneth Knowles <
 k...@apache.org> wrote:

> Hi all,
>
> Please join me and the rest of the Beam PMC in welcoming a new
>  committer: Mark Liu.
>
> Mark has been contributing to Beam since late 2016! He has
> proposed 100+ pull requests. Mark was instrumental in expanding 
> test and
> infrastructure coverage, especially for Python. In
> consideration of Mark's contributions, the Beam PMC trusts Mark 
> with the
> responsibilities of a Beam committer [1].
>
> Thank you, Mark, for your contributions.
>
> Kenn
>
> [1] https://beam.apache.org/contribute/become-a-committer/
> #an-apache-beam-committer
>
 --
>>>
>>> *Aizhamal Nurmamat kyzy*
>>>
>>> Open Source Program Manager
>>>
>>> 646-355-9740 Mobile
>>>
>>> 601 North 34th Street, Seattle, WA 98103
>>>
>>>
>>>
>
> --
> Cheers,
> Gleb
>



smime.p7s
Description: S/MIME Cryptographic Signature

Re: [ANNOUNCE] New committer announcement: Mark Liu

[Ahmet Altay]

Congratulations, Mark! 🎉

On Mon, Mar 25, 2019 at 7:24 AM Tim Robertson 
wrote:

> Congratulations Mark!
>
>
> On Mon, Mar 25, 2019 at 3:18 PM Michael Luckey 
> wrote:
>
>> Nice! Congratulations, Mark.
>>
>> On Mon, Mar 25, 2019 at 2:42 PM Katarzyna Kucharczyk <
>> ka.kucharc...@gmail.com> wrote:
>>
>>> Congratulations, Mark! 🎉
>>>
>>> On Mon, Mar 25, 2019 at 11:24 AM Gleb Kanterov  wrote:
>>>
 Congratulations!

 On Mon, Mar 25, 2019 at 10:23 AM Łukasz Gajowy 
 wrote:

> Congrats! :)
>
>
>
> pon., 25 mar 2019 o 08:11 Aizhamal Nurmamat kyzy 
> napisał(a):
>
>> Congratulations, Mark!
>>
>> On Sun, Mar 24, 2019 at 23:18 Pablo Estrada 
>> wrote:
>>
>>> Yeaah  Mark! : ) Congrats : D
>>>
>>> On Sun, Mar 24, 2019 at 10:32 PM Yifan Zou 
>>> wrote:
>>>
 Congratulations Mark!

 On Sun, Mar 24, 2019 at 10:25 PM Connell O'Callaghan <
 conne...@google.com> wrote:

> Well done congratulations Mark!!!
>
> On Sun, Mar 24, 2019 at 10:17 PM Robert Burke 
> wrote:
>
>> Congratulations Mark! 🎉
>>
>> On Sun, Mar 24, 2019, 10:08 PM Valentyn Tymofieiev <
>> valen...@google.com> wrote:
>>
>>> Congratulations, Mark!
>>>
>>> Thanks for your contributions, in particular for your efforts to
>>> parallelize test execution for Python SDK and increase the speed of 
>>> Python
>>> precommit checks.
>>>
>>> On Sun, Mar 24, 2019 at 9:40 PM Kenneth Knowles 
>>> wrote:
>>>
 Hi all,

 Please join me and the rest of the Beam PMC in welcoming a new
 committer: Mark Liu.

 Mark has been contributing to Beam since late 2016! He has
 proposed 100+ pull requests. Mark was instrumental in expanding 
 test and
 infrastructure coverage, especially for Python. In
 consideration of Mark's contributions, the Beam PMC trusts Mark 
 with the
 responsibilities of a Beam committer [1].

 Thank you, Mark, for your contributions.

 Kenn

 [1] https://beam.apache.org/contribute/become-a-committer/
 #an-apache-beam-committer

>>> --
>>
>> *Aizhamal Nurmamat kyzy*
>>
>> Open Source Program Manager
>>
>> 646-355-9740 Mobile
>>
>> 601 North 34th Street, Seattle, WA 98103
>>
>>
>>

 --
 Cheers,
 Gleb

>>>

Build blocking on

[Michael Luckey]

Hi,

trying to run './gradlew build' on vanilla setup, my build consistently
stalls during execution of python gcp tests, e.g. on both of
- > :beam-sdks-python:testPy2Gcp
- > :beam-sdks-python-test-suites-tox-py35:testPy35Gcp

Console output:
 snip 
test_big_query_standard_sql
(apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT)
... SKIP: IT is skipped because --test-pipeline-options is not specified
test_big_query_standard_sql_kms_key
(apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT)
... SKIP: This test requires BQ Dataflow native source support for KMS,
which is not available yet.
test_multiple_destinations_transform
(apache_beam.io.gcp.bigquery_file_loads_test.BigQueryFileLoadsIT) ... SKIP:
IT is skipped because --test-pipeline-options is not specified
test_one_job_fails_all_jobs_fail
(apache_beam.io.gcp.bigquery_file_loads_test.BigQueryFileLoadsIT) ... SKIP:
IT is skipped because --test-pipeline-options is not specified
test_records_traverse_transform_with_mocks
(apache_beam.io.gcp.bigquery_file_loads_test.TestBigQueryFileLoads) ...

output ends here, would expect a failed or ok here.


Afterwards no progress - even waiting for hours. Any idea, what might be
causing this? Do I need to add some GCP properties for this task ?

Any ideas, what I am doing wrong?

best,

michel

Re: [ANNOUNCE] New committer announcement: Mark Liu

[Tim Robertson]

Congratulations Mark!


On Mon, Mar 25, 2019 at 3:18 PM Michael Luckey  wrote:

> Nice! Congratulations, Mark.
>
> On Mon, Mar 25, 2019 at 2:42 PM Katarzyna Kucharczyk <
> ka.kucharc...@gmail.com> wrote:
>
>> Congratulations, Mark! 🎉
>>
>> On Mon, Mar 25, 2019 at 11:24 AM Gleb Kanterov  wrote:
>>
>>> Congratulations!
>>>
>>> On Mon, Mar 25, 2019 at 10:23 AM Łukasz Gajowy 
>>> wrote:
>>>
 Congrats! :)



 pon., 25 mar 2019 o 08:11 Aizhamal Nurmamat kyzy 
 napisał(a):

> Congratulations, Mark!
>
> On Sun, Mar 24, 2019 at 23:18 Pablo Estrada 
> wrote:
>
>> Yeaah  Mark! : ) Congrats : D
>>
>> On Sun, Mar 24, 2019 at 10:32 PM Yifan Zou 
>> wrote:
>>
>>> Congratulations Mark!
>>>
>>> On Sun, Mar 24, 2019 at 10:25 PM Connell O'Callaghan <
>>> conne...@google.com> wrote:
>>>
 Well done congratulations Mark!!!

 On Sun, Mar 24, 2019 at 10:17 PM Robert Burke 
 wrote:

> Congratulations Mark! 🎉
>
> On Sun, Mar 24, 2019, 10:08 PM Valentyn Tymofieiev <
> valen...@google.com> wrote:
>
>> Congratulations, Mark!
>>
>> Thanks for your contributions, in particular for your efforts to
>> parallelize test execution for Python SDK and increase the speed of 
>> Python
>> precommit checks.
>>
>> On Sun, Mar 24, 2019 at 9:40 PM Kenneth Knowles 
>> wrote:
>>
>>> Hi all,
>>>
>>> Please join me and the rest of the Beam PMC in welcoming a new
>>> committer: Mark Liu.
>>>
>>> Mark has been contributing to Beam since late 2016! He has
>>> proposed 100+ pull requests. Mark was instrumental in expanding 
>>> test and
>>> infrastructure coverage, especially for Python. In
>>> consideration of Mark's contributions, the Beam PMC trusts Mark 
>>> with the
>>> responsibilities of a Beam committer [1].
>>>
>>> Thank you, Mark, for your contributions.
>>>
>>> Kenn
>>>
>>> [1] https://beam.apache.org/contribute/become-a-committer/
>>> #an-apache-beam-committer
>>>
>> --
>
> *Aizhamal Nurmamat kyzy*
>
> Open Source Program Manager
>
> 646-355-9740 Mobile
>
> 601 North 34th Street, Seattle, WA 98103
>
>
>
>>>
>>> --
>>> Cheers,
>>> Gleb
>>>
>>

Re: [ANNOUNCE] New committer announcement: Mark Liu

[Michael Luckey]

Nice! Congratulations, Mark.

On Mon, Mar 25, 2019 at 2:42 PM Katarzyna Kucharczyk <
ka.kucharc...@gmail.com> wrote:

> Congratulations, Mark! 🎉
>
> On Mon, Mar 25, 2019 at 11:24 AM Gleb Kanterov  wrote:
>
>> Congratulations!
>>
>> On Mon, Mar 25, 2019 at 10:23 AM Łukasz Gajowy 
>> wrote:
>>
>>> Congrats! :)
>>>
>>>
>>>
>>> pon., 25 mar 2019 o 08:11 Aizhamal Nurmamat kyzy 
>>> napisał(a):
>>>
 Congratulations, Mark!

 On Sun, Mar 24, 2019 at 23:18 Pablo Estrada  wrote:

> Yeaah  Mark! : ) Congrats : D
>
> On Sun, Mar 24, 2019 at 10:32 PM Yifan Zou 
> wrote:
>
>> Congratulations Mark!
>>
>> On Sun, Mar 24, 2019 at 10:25 PM Connell O'Callaghan <
>> conne...@google.com> wrote:
>>
>>> Well done congratulations Mark!!!
>>>
>>> On Sun, Mar 24, 2019 at 10:17 PM Robert Burke 
>>> wrote:
>>>
 Congratulations Mark! 🎉

 On Sun, Mar 24, 2019, 10:08 PM Valentyn Tymofieiev <
 valen...@google.com> wrote:

> Congratulations, Mark!
>
> Thanks for your contributions, in particular for your efforts to
> parallelize test execution for Python SDK and increase the speed of 
> Python
> precommit checks.
>
> On Sun, Mar 24, 2019 at 9:40 PM Kenneth Knowles 
> wrote:
>
>> Hi all,
>>
>> Please join me and the rest of the Beam PMC in welcoming a new
>> committer: Mark Liu.
>>
>> Mark has been contributing to Beam since late 2016! He has
>> proposed 100+ pull requests. Mark was instrumental in expanding test 
>> and
>> infrastructure coverage, especially for Python. In consideration
>> of Mark's contributions, the Beam PMC trusts Mark with the 
>> responsibilities
>> of a Beam committer [1].
>>
>> Thank you, Mark, for your contributions.
>>
>> Kenn
>>
>> [1] https://beam.apache.org/contribute/become-a-committer/
>> #an-apache-beam-committer
>>
> --

 *Aizhamal Nurmamat kyzy*

 Open Source Program Manager

 646-355-9740 Mobile

 601 North 34th Street, Seattle, WA 98103



>>
>> --
>> Cheers,
>> Gleb
>>
>

Re: Apache Beam, Spark, Hadoop and AWS cross accounts

[Jeff Klukas]

I don't think I'm fully understanding the input part of your pipeline since
it looks like it's using some custom IO, but I am fairly confident that
FileIO.write() will never produce empty files, so the behavior you describe
sounds expected if you're reading in an empty file.

FileIO doesn't copy files, but rather is a container for read transforms
that read file contents into a PCollection and write transforms that batch
together records from a PCollection and write serialized contents to files.

I generally would not expect the FileIO machinery to be able to preserve
file structure or naming between input and output. If that's what you want,
then you may want to read file names and write your own ParDo to run S3
copy operations.

On Sun, Mar 24, 2019 at 12:07 PM Carlos Baeza 
wrote:

> Hi Guys,
>
> I’m new in Apache Beam. In my current project we have follow scenario:
>
> - We run transformations via Apache Beam Pipeline into Amazon AWS (using
> Cluster by Spark, Hadoop). We can in future produce big Files.
> - The generated a AVRO file that should be stored from AWS Account A to
> AWS Account B in "S3://some_store". The process is started in Account A
> because here is the required data access layer.
> - The first experimentation shown:
> - A empty file is created (0 bytes -> MyTransformation.avro) in
> Account
> A
> - After process is finished, no file appear in Account B
> "S3://sone_store”. File is missing.
>
> The process defined in Account A look like:
>
> ---
> final SimplePipelineOptions options =
> PipelineOptionsFactory.fromArgs(args).as(SimplePipelineOptions.class);
> final Pipeline pipeline = Pipeline.create(options);
> pipeline.apply(
>  SomeDataIO.read()
>  .withZookeeperQuorum(options.getZookeeperQuorum())
>  .withDataVersion(UUID.fromString(options.getVersionId()),
> options.getDataVersion())
>  .withView(DictionaryModelView
>  .create(MODEL, new ProcessSomeData(;
>
> Join.innerJoin(someData, someData)
>  .apply(Values.create())
>  .apply(ParDo.of(new ExtractSomeData()))
>  .apply(FileIO.>writeDynamic()
>  .by(KV::getKey)
>  .via(fn, AvroIO.sink(AVFeature.class))
>  .withNaming("MyTransformation.avro")
>  .to("S3://sone_store")
>  .withNumShards(1)
>  .withDestinationCoder(StringUtf8Coder.of()));
>
> pipeline.run().waitUntilFinish();
>
> ---
>
> The class ProcessSomeData is responsible to extract some data from our
> persistence layer and process it.
>
> In test, running from Account B, all work fine, we can produce the AVRO
> File (34 KB) and store the file into Account B S3 store ->
> S3://some_store
> But running in the cloud starting the process from Account A, then we
> lost the file (MyTransformation.avro, 0 bytes). -> File has not been
> copied.
> AWS S3 configuration from Account B give full access to Account A.
>
> 1. Some idea what goes wrong?
> 2. Maybe FileIO.Write.to(...) is not able to store data between AWS
> cross accounts?
> 3. Should I create my self a java client to store in Account B?
> 4. Can FileIO copy 0 bytes file?
>
> Any help is appreciate.
>
> Many thanks in advance !
>
> Carlos
>
>
>
>

Re: [ANNOUNCE] New committer announcement: Mark Liu

[Katarzyna Kucharczyk]

Congratulations, Mark! 🎉

On Mon, Mar 25, 2019 at 11:24 AM Gleb Kanterov  wrote:

> Congratulations!
>
> On Mon, Mar 25, 2019 at 10:23 AM Łukasz Gajowy  wrote:
>
>> Congrats! :)
>>
>>
>>
>> pon., 25 mar 2019 o 08:11 Aizhamal Nurmamat kyzy 
>> napisał(a):
>>
>>> Congratulations, Mark!
>>>
>>> On Sun, Mar 24, 2019 at 23:18 Pablo Estrada  wrote:
>>>
 Yeaah  Mark! : ) Congrats : D

 On Sun, Mar 24, 2019 at 10:32 PM Yifan Zou  wrote:

> Congratulations Mark!
>
> On Sun, Mar 24, 2019 at 10:25 PM Connell O'Callaghan <
> conne...@google.com> wrote:
>
>> Well done congratulations Mark!!!
>>
>> On Sun, Mar 24, 2019 at 10:17 PM Robert Burke 
>> wrote:
>>
>>> Congratulations Mark! 🎉
>>>
>>> On Sun, Mar 24, 2019, 10:08 PM Valentyn Tymofieiev <
>>> valen...@google.com> wrote:
>>>
 Congratulations, Mark!

 Thanks for your contributions, in particular for your efforts to
 parallelize test execution for Python SDK and increase the speed of 
 Python
 precommit checks.

 On Sun, Mar 24, 2019 at 9:40 PM Kenneth Knowles 
 wrote:

> Hi all,
>
> Please join me and the rest of the Beam PMC in welcoming a new
> committer: Mark Liu.
>
> Mark has been contributing to Beam since late 2016! He has
> proposed 100+ pull requests. Mark was instrumental in expanding test 
> and
> infrastructure coverage, especially for Python. In consideration
> of Mark's contributions, the Beam PMC trusts Mark with the 
> responsibilities
> of a Beam committer [1].
>
> Thank you, Mark, for your contributions.
>
> Kenn
>
> [1] https://beam.apache.org/contribute/become-a-committer/
> #an-apache-beam-committer
>
 --
>>>
>>> *Aizhamal Nurmamat kyzy*
>>>
>>> Open Source Program Manager
>>>
>>> 646-355-9740 Mobile
>>>
>>> 601 North 34th Street, Seattle, WA 98103
>>>
>>>
>>>
>
> --
> Cheers,
> Gleb
>

Beam Dependency Check Report (2019-03-25)

[Apache Jenkins Server]

ERROR: File 'src/build/dependencyUpdates/beam-dependency-check-report.html' does not exist

Re: [ANNOUNCE] New committer announcement: Mark Liu

[Gleb Kanterov]

Congratulations!

On Mon, Mar 25, 2019 at 10:23 AM Łukasz Gajowy  wrote:

> Congrats! :)
>
>
>
> pon., 25 mar 2019 o 08:11 Aizhamal Nurmamat kyzy 
> napisał(a):
>
>> Congratulations, Mark!
>>
>> On Sun, Mar 24, 2019 at 23:18 Pablo Estrada  wrote:
>>
>>> Yeaah  Mark! : ) Congrats : D
>>>
>>> On Sun, Mar 24, 2019 at 10:32 PM Yifan Zou  wrote:
>>>
 Congratulations Mark!

 On Sun, Mar 24, 2019 at 10:25 PM Connell O'Callaghan <
 conne...@google.com> wrote:

> Well done congratulations Mark!!!
>
> On Sun, Mar 24, 2019 at 10:17 PM Robert Burke 
> wrote:
>
>> Congratulations Mark! 🎉
>>
>> On Sun, Mar 24, 2019, 10:08 PM Valentyn Tymofieiev <
>> valen...@google.com> wrote:
>>
>>> Congratulations, Mark!
>>>
>>> Thanks for your contributions, in particular for your efforts to
>>> parallelize test execution for Python SDK and increase the speed of 
>>> Python
>>> precommit checks.
>>>
>>> On Sun, Mar 24, 2019 at 9:40 PM Kenneth Knowles 
>>> wrote:
>>>
 Hi all,

 Please join me and the rest of the Beam PMC in welcoming a new
 committer: Mark Liu.

 Mark has been contributing to Beam since late 2016! He has proposed
 100+ pull requests. Mark was instrumental in expanding test and
 infrastructure coverage, especially for Python. In consideration
 of Mark's contributions, the Beam PMC trusts Mark with the 
 responsibilities
 of a Beam committer [1].

 Thank you, Mark, for your contributions.

 Kenn

 [1] https://beam.apache.org/contribute/become-a-committer/
 #an-apache-beam-committer

>>> --
>>
>> *Aizhamal Nurmamat kyzy*
>>
>> Open Source Program Manager
>>
>> 646-355-9740 Mobile
>>
>> 601 North 34th Street, Seattle, WA 98103
>>
>>
>>

-- 
Cheers,
Gleb

Re: [ANNOUNCE] New committer announcement: Mark Liu

[Łukasz Gajowy]

Congrats! :)



pon., 25 mar 2019 o 08:11 Aizhamal Nurmamat kyzy 
napisał(a):

> Congratulations, Mark!
>
> On Sun, Mar 24, 2019 at 23:18 Pablo Estrada  wrote:
>
>> Yeaah  Mark! : ) Congrats : D
>>
>> On Sun, Mar 24, 2019 at 10:32 PM Yifan Zou  wrote:
>>
>>> Congratulations Mark!
>>>
>>> On Sun, Mar 24, 2019 at 10:25 PM Connell O'Callaghan <
>>> conne...@google.com> wrote:
>>>
 Well done congratulations Mark!!!

 On Sun, Mar 24, 2019 at 10:17 PM Robert Burke 
 wrote:

> Congratulations Mark! 🎉
>
> On Sun, Mar 24, 2019, 10:08 PM Valentyn Tymofieiev <
> valen...@google.com> wrote:
>
>> Congratulations, Mark!
>>
>> Thanks for your contributions, in particular for your efforts to
>> parallelize test execution for Python SDK and increase the speed of 
>> Python
>> precommit checks.
>>
>> On Sun, Mar 24, 2019 at 9:40 PM Kenneth Knowles 
>> wrote:
>>
>>> Hi all,
>>>
>>> Please join me and the rest of the Beam PMC in welcoming a new
>>> committer: Mark Liu.
>>>
>>> Mark has been contributing to Beam since late 2016! He has proposed
>>> 100+ pull requests. Mark was instrumental in expanding test and
>>> infrastructure coverage, especially for Python. In consideration of
>>> Mark's contributions, the Beam PMC trusts Mark with the 
>>> responsibilities of
>>> a Beam committer [1].
>>>
>>> Thank you, Mark, for your contributions.
>>>
>>> Kenn
>>>
>>> [1] https://beam.apache.org/contribute/become-a-committer/
>>> #an-apache-beam-committer
>>>
>> --
>
> *Aizhamal Nurmamat kyzy*
>
> Open Source Program Manager
>
> 646-355-9740 Mobile
>
> 601 North 34th Street, Seattle, WA 98103
>
>
>

Re: Writing bytes to BigQuery with beam

[Juta Staes]

On Mon, 25 Mar 2019 at 06:15, Valentyn Tymofieiev 
wrote:

> We received feedback on https://issuetracker.google.com/issues/129006689 -
> BQ developers say that schema identification is done and they discourage to
> use schema autodetection in tables using BYTES. In light of this, I think
> may be fair to recommend Beam users to specify BQ schemas as well when they
> interact with BQ, and call out that writing binary data to BQ will likely
> fail unless schema is specified. Does that make sense?
>

Given that schema autodetect does not work for bytes I think it is indeed a
good solution to require users to specify BQ schemas as well when they
write to BQ

So new summary:
1. Beam will base64-encode raw bytes, before passing them to BQ over rest
API. This will be a change in behavior for Python 2 (for good reasons).
2. When reading data from BQ, all fields of type BYTES will be
base64-decoded.
3. Beam will send an API call to BigQuery to get table schema, whenever
schema is not supplied, to work around
https://issuetracker.google.com/issues/129006689. Beam will require users
to specify the schema when writing bytes to BQ.

Thanks all for your input on this!
Juta

Re: [ANNOUNCE] New committer announcement: Mark Liu

[Aizhamal Nurmamat kyzy]

Congratulations, Mark!

On Sun, Mar 24, 2019 at 23:18 Pablo Estrada  wrote:

> Yeaah  Mark! : ) Congrats : D
>
> On Sun, Mar 24, 2019 at 10:32 PM Yifan Zou  wrote:
>
>> Congratulations Mark!
>>
>> On Sun, Mar 24, 2019 at 10:25 PM Connell O'Callaghan 
>> wrote:
>>
>>> Well done congratulations Mark!!!
>>>
>>> On Sun, Mar 24, 2019 at 10:17 PM Robert Burke 
>>> wrote:
>>>
 Congratulations Mark! 🎉

 On Sun, Mar 24, 2019, 10:08 PM Valentyn Tymofieiev 
 wrote:

> Congratulations, Mark!
>
> Thanks for your contributions, in particular for your efforts to
> parallelize test execution for Python SDK and increase the speed of Python
> precommit checks.
>
> On Sun, Mar 24, 2019 at 9:40 PM Kenneth Knowles 
> wrote:
>
>> Hi all,
>>
>> Please join me and the rest of the Beam PMC in welcoming a new
>> committer: Mark Liu.
>>
>> Mark has been contributing to Beam since late 2016! He has proposed
>> 100+ pull requests. Mark was instrumental in expanding test and
>> infrastructure coverage, especially for Python. In consideration of
>> Mark's contributions, the Beam PMC trusts Mark with the responsibilities 
>> of
>> a Beam committer [1].
>>
>> Thank you, Mark, for your contributions.
>>
>> Kenn
>>
>> [1] https://beam.apache.org/contribute/become-a-committer/#an-apache-
>> beam-committer
>>
> --

*Aizhamal Nurmamat kyzy*

Open Source Program Manager

646-355-9740 Mobile

601 North 34th Street, Seattle, WA 98103