Hello!

[Paweł Pasterz]

My name is Paweł and I am a developer from Warsaw/Poland. I'd like to
contribute to beam project therefore I am kindly asking for JIRA
permissions.

Regards
Paweł

Re: [DISCUSS] BIP reloaded

[David Morávek]

Hi Jan,

I think this is more pretty much what we currently do, just a little bit
more transparent for the community. If the process is standardized, it can
open doors for bigger contributions from people not familiar with the
process. Also it's way easier to track progress of BIPs, than documents
linked from the mailing list.

Big +1 ;)

D.

On Sun, Dec 8, 2019 at 12:42 PM Jan Lukavský  wrote:

> Hi,
>
> I'd like to revive a discussion that was taken some year and a half ago
> [1], which included a concept of "BIP" (Beam Improvement Proposal) - an
> equivalent of "FLIP" (flink), "KIP" (kafka), "SPIP" (spark), and so on.
>
> The discussion then ended without any (public) conclusion, so I'd like
> to pick up from there. There were questions related to:
>
>   a) how does the concept of BIP differ from simple plain JIRA?
>
>   b) what does it bring to the community?
>
> I'd like to outline my point of view on both of these aspects (they are
> related).
>
> BIP differs from JIRA by definition of a process:
>
> BIP -> vote -> consensus -> JIRA -> implementation
>
> This process (although it might seem a little unnecessary formal) brings
> the following benefits:
>
>   i) improves community's overall awareness of planned and in-progress
> features
>
>   ii) makes it possible to prioritize long-term goals (create "roadmap"
> that was mentioned in the referred thread)
>
>   iii) by casting explicit vote on each improvement proposal diminishes
> the probability of wasted work - as opposed to our current state, where
> it is hard to tell when there is a consensus and what actions need to be
> done in order to reach one if there isn't
>
>   iv) BIPs that eventually pass a vote can be regarded as "to be
> included in some short term" and so new BIPs can build upon them,
> without the risk of having to be redefined if their dependency for
> whatever reason don't make it to the implementation
>
> Although this "process" might look rigid and corporate, it actually
> brings better transparency and overall community health. This is
> especially important as the community grows and becomes more and more
> distributed. There are many, many open questions in this proposal that
> need to be clarified, my current intent is to grab a grasp about how the
> community feels about this.
>
> Looking forward to any comments,
>
>   Jan
>
> [1]
>
> https://lists.apache.org/thread.html/4e1fffa2fde8e750c6d769bf4335853ad05b360b8bd248ad119cc185%40%3Cdev.beam.apache.org%3E
>
>

Re: Python: pytest migration update

[Kenneth Knowles]

On Mon, Dec 9, 2019 at 6:34 PM Udi Meiri  wrote:

> Valentyn, the speedup is due to parallelization.
>
> On Mon, Dec 9, 2019 at 6:12 PM Chad Dombrova  wrote:
>
>>
>> On Mon, Dec 9, 2019 at 5:36 PM Udi Meiri  wrote:
>>
>>> I have given this some thought honestly don't know if splitting into
>>> separate jobs will help.
>>> - I have seen race conditions with running setuptools in parallel, so
>>> more isolation is better.
>>>
>>
>> What race conditions have you seen?  I think if we're doing things right,
>> this should not be happening, but I don't think we're doing things right.
>> One thing that I've noticed is that we're building into the source
>> directory, but I also think we're also doing weird things like trying to
>> copy the source directory beforehand.  I really think this system is
>> tripping over many non-standard choices that have been made along the way.
>> I have never these sorts of problems with in unittests that use tox, even
>> when many are running in parallel.  I got pulled away from it, but I'm
>> really hoping to address these issues here:
>> https://github.com/apache/beam/pull/10038.
>>
>
> This comment
> 
> summarizes what I believe may be the issue (setuptools races).
>
> I believe copying the source directory was done in an effort to isolate
> the parallel builds (setuptools, cythonize).
>

Peanut gallery: containerized Jenkins builds seem like they would help, and
they are the current recommended best practice, but we are not there yet.
Agree/disagree?

What benefits do you see from splitting up the jobs?
>>>
>>
>> The biggest problem is that the jobs are doing too much and take too
>> long.  This simple fact compounds all of the other problems.  It seems
>> pretty obvious that we need to do less in each job, as long as the sum of
>> all of these smaller jobs is not substantially longer than the one
>> monolithic job.
>>
>
For some reason I keep forgetting the answer to this question: are we
caching pypi immutable artifacts on every Jenkins worker?


>
>> Benefits:
>>
>> - failures specific to a particular python version will be easier to spot
>> in the jenkins error summary, and cheaper to re-queue.  right now the
>> jenkins report mushes all of the failures together in a way that makes it
>> nearly impossible to tell which python version they correspond to.  only
>> the gradle scan gives you this insight, but it doesn't break the errors by
>> test.
>>
>
> I agree Jenkins handles duplicate test names pretty badly (reloading will
> periodically give you a different result).
>

Saw this in Java too w/ ValidatesRunner suites when they ran in one Jenkins
job. Worthwhile to avoid.

Kenn


> With pytest I've been able to set the suite name so that should help with
> identification. (I need to add pytest*.xml collection to the Jenkins job
> first)
>
>
>> - failures common to all python versions will be reported to the user
>> earlier, at which point they can cancel the other jobs if desired.  *this
>> is by far the biggest benefit. * why wait for 2 hours to see the same
>> failure reported for 5 versions of python?  if that had run on one version
>> of python I could maybe see that error in 30 minutes (while potentially
>> other python versions waited in the queue).  Repeat for each change pushed.
>> - flaky jobs will be cheaper to requeue (since it will affect a
>> smaller/shorter job)
>> - if xdist is giving us the parallel boost we're hoping for we should get
>> under the 2 hour mark every time
>>
>> Basically we're talking about getting feedback to users faster.
>>
>
> +1
>
>
>>
>> I really don't mind pasting a few more phrases if it means faster
>> feedback.
>>
>> -chad
>>
>>
>>
>>
>>>
>>> On Mon, Dec 9, 2019 at 4:17 PM Chad Dombrova  wrote:
>>>
 After this PR goes in should we revisit breaking up the python tests
 into separate jenkins jobs by python version?  One of the problems with
 that plan originally was that we lost the parallelism that gradle provides
 because we were left with only one tox task per jenkins job, and so the
 total time to complete all python jenkins jobs went up a lot.  With
 pytest + xdist we should hopefully be able to keep the parallelism even
 with just one tox task.  This could be a big win.  I feel like I'm spending
 more time monitoring and re-queuing timed-out jenkins jobs lately than I am
 writing code.

 On Mon, Dec 9, 2019 at 10:32 AM Udi Meiri  wrote:

> This PR  (in review)
> migrates py27-gcp to using pytest.
> It reduces the testPy2Gcp task down to ~13m
> 
> (from ~45m). This speedup will probably be lower once all 8 tasks are 
> using
> pytest.
> It also adds 5 previously uncollected te

Re: Python: pytest migration update

[Udi Meiri]

Valentyn, the speedup is due to parallelization.

On Mon, Dec 9, 2019 at 6:12 PM Chad Dombrova  wrote:

>
> On Mon, Dec 9, 2019 at 5:36 PM Udi Meiri  wrote:
>
>> I have given this some thought honestly don't know if splitting into
>> separate jobs will help.
>> - I have seen race conditions with running setuptools in parallel, so
>> more isolation is better.
>>
>
> What race conditions have you seen?  I think if we're doing things right,
> this should not be happening, but I don't think we're doing things right.
> One thing that I've noticed is that we're building into the source
> directory, but I also think we're also doing weird things like trying to
> copy the source directory beforehand.  I really think this system is
> tripping over many non-standard choices that have been made along the way.
> I have never these sorts of problems with in unittests that use tox, even
> when many are running in parallel.  I got pulled away from it, but I'm
> really hoping to address these issues here:
> https://github.com/apache/beam/pull/10038.
>

This comment

summarizes what I believe may be the issue (setuptools races).

I believe copying the source directory was done in an effort to isolate the
parallel builds (setuptools, cythonize).


>
>> What benefits do you see from splitting up the jobs?
>>
>
> The biggest problem is that the jobs are doing too much and take too
> long.  This simple fact compounds all of the other problems.  It seems
> pretty obvious that we need to do less in each job, as long as the sum of
> all of these smaller jobs is not substantially longer than the one
> monolithic job.
>
> Benefits:
>
> - failures specific to a particular python version will be easier to spot
> in the jenkins error summary, and cheaper to re-queue.  right now the
> jenkins report mushes all of the failures together in a way that makes it
> nearly impossible to tell which python version they correspond to.  only
> the gradle scan gives you this insight, but it doesn't break the errors by
> test.
>

I agree Jenkins handles duplicate test names pretty badly (reloading will
periodically give you a different result).
With pytest I've been able to set the suite name so that should help with
identification. (I need to add pytest*.xml collection to the Jenkins job
first)


> - failures common to all python versions will be reported to the user
> earlier, at which point they can cancel the other jobs if desired.  *this
> is by far the biggest benefit. * why wait for 2 hours to see the same
> failure reported for 5 versions of python?  if that had run on one version
> of python I could maybe see that error in 30 minutes (while potentially
> other python versions waited in the queue).  Repeat for each change pushed.
> - flaky jobs will be cheaper to requeue (since it will affect a
> smaller/shorter job)
> - if xdist is giving us the parallel boost we're hoping for we should get
> under the 2 hour mark every time
>
> Basically we're talking about getting feedback to users faster.
>

+1


>
> I really don't mind pasting a few more phrases if it means faster feedback.
>
> -chad
>
>
>
>
>>
>> On Mon, Dec 9, 2019 at 4:17 PM Chad Dombrova  wrote:
>>
>>> After this PR goes in should we revisit breaking up the python tests
>>> into separate jenkins jobs by python version?  One of the problems with
>>> that plan originally was that we lost the parallelism that gradle provides
>>> because we were left with only one tox task per jenkins job, and so the
>>> total time to complete all python jenkins jobs went up a lot.  With
>>> pytest + xdist we should hopefully be able to keep the parallelism even
>>> with just one tox task.  This could be a big win.  I feel like I'm spending
>>> more time monitoring and re-queuing timed-out jenkins jobs lately than I am
>>> writing code.
>>>
>>> On Mon, Dec 9, 2019 at 10:32 AM Udi Meiri  wrote:
>>>
 This PR  (in review)
 migrates py27-gcp to using pytest.
 It reduces the testPy2Gcp task down to ~13m
 
 (from ~45m). This speedup will probably be lower once all 8 tasks are using
 pytest.
 It also adds 5 previously uncollected tests.

>>>


smime.p7s
Description: S/MIME Cryptographic Signature

Re: Python: pytest migration update

[Chad Dombrova]

On Mon, Dec 9, 2019 at 5:36 PM Udi Meiri  wrote:

> I have given this some thought honestly don't know if splitting into
> separate jobs will help.
> - I have seen race conditions with running setuptools in parallel, so more
> isolation is better.
>

What race conditions have you seen?  I think if we're doing things right,
this should not be happening, but I don't think we're doing things right.
One thing that I've noticed is that we're building into the source
directory, but I also think we're also doing weird things like trying to
copy the source directory beforehand.  I really think this system is
tripping over many non-standard choices that have been made along the way.
I have never these sorts of problems with in unittests that use tox, even
when many are running in parallel.  I got pulled away from it, but I'm
really hoping to address these issues here:
https://github.com/apache/beam/pull/10038.

>
> What benefits do you see from splitting up the jobs?
>

The biggest problem is that the jobs are doing too much and take too long.
This simple fact compounds all of the other problems.  It seems pretty
obvious that we need to do less in each job, as long as the sum of all of
these smaller jobs is not substantially longer than the one monolithic job.

Benefits:

- failures specific to a particular python version will be easier to spot
in the jenkins error summary, and cheaper to re-queue.  right now the
jenkins report mushes all of the failures together in a way that makes it
nearly impossible to tell which python version they correspond to.  only
the gradle scan gives you this insight, but it doesn't break the errors by
test.
- failures common to all python versions will be reported to the user
earlier, at which point they can cancel the other jobs if desired.  *this
is by far the biggest benefit. * why wait for 2 hours to see the same
failure reported for 5 versions of python?  if that had run on one version
of python I could maybe see that error in 30 minutes (while potentially
other python versions waited in the queue).  Repeat for each change pushed.
- flaky jobs will be cheaper to requeue (since it will affect a
smaller/shorter job)
- if xdist is giving us the parallel boost we're hoping for we should get
under the 2 hour mark every time

Basically we're talking about getting feedback to users faster.

I really don't mind pasting a few more phrases if it means faster feedback.

-chad




>
> On Mon, Dec 9, 2019 at 4:17 PM Chad Dombrova  wrote:
>
>> After this PR goes in should we revisit breaking up the python tests into
>> separate jenkins jobs by python version?  One of the problems with that
>> plan originally was that we lost the parallelism that gradle provides
>> because we were left with only one tox task per jenkins job, and so the
>> total time to complete all python jenkins jobs went up a lot.  With
>> pytest + xdist we should hopefully be able to keep the parallelism even
>> with just one tox task.  This could be a big win.  I feel like I'm spending
>> more time monitoring and re-queuing timed-out jenkins jobs lately than I am
>> writing code.
>>
>> On Mon, Dec 9, 2019 at 10:32 AM Udi Meiri  wrote:
>>
>>> This PR  (in review)
>>> migrates py27-gcp to using pytest.
>>> It reduces the testPy2Gcp task down to ~13m
>>> 
>>> (from ~45m). This speedup will probably be lower once all 8 tasks are using
>>> pytest.
>>> It also adds 5 previously uncollected tests.
>>>
>>

Re: Python: pytest migration update

[Valentyn Tymofieiev]

Thanks a lot for your work on pytest, Udi! Is the speedup due to running
more tests in parallel or pytest spends less time orchestrating the run?

On Mon, Dec 9, 2019 at 5:59 PM Valentyn Tymofieiev 
wrote:

>
>
> On Mon, Dec 9, 2019 at 5:36 PM Udi Meiri  wrote:
>
>> I have given this some thought honestly don't know if splitting into
>> separate jobs will help.
>> - I have seen race conditions with running setuptools in parallel, so
>> more isolation is better. OTOH, if 2 setuptools scripts run at the same
>> time on the same machine they might still race (same homedir, same /tmp
>> dir).
>> - Retrying due to flakes will be faster, but if something is broken
>> you'll need to write 4x the number of "run python precommit" phrases.
>>
>
> there is also "retest this please" which will rerun all precommits.
>
>
>> - Increased parallelism may also run into quota issues with Dataflow.
>>
>> What benefits do you see from splitting up the jobs?
>>
>> On Mon, Dec 9, 2019 at 4:17 PM Chad Dombrova  wrote:
>>
>>> After this PR goes in should we revisit breaking up the python tests
>>> into separate jenkins jobs by python version?  One of the problems with
>>> that plan originally was that we lost the parallelism that gradle provides
>>> because we were left with only one tox task per jenkins job, and so the
>>> total time to complete all python jenkins jobs went up a lot.  With
>>> pytest + xdist we should hopefully be able to keep the parallelism even
>>> with just one tox task.  This could be a big win.  I feel like I'm spending
>>> more time monitoring and re-queuing timed-out jenkins jobs lately than I am
>>> writing code.
>>>
>>> On Mon, Dec 9, 2019 at 10:32 AM Udi Meiri  wrote:
>>>
 This PR  (in review)
 migrates py27-gcp to using pytest.
 It reduces the testPy2Gcp task down to ~13m
 
 (from ~45m). This speedup will probably be lower once all 8 tasks are using
 pytest.
 It also adds 5 previously uncollected tests.

>>>

Re: Python: pytest migration update

[Valentyn Tymofieiev]

On Mon, Dec 9, 2019 at 5:36 PM Udi Meiri  wrote:

> I have given this some thought honestly don't know if splitting into
> separate jobs will help.
> - I have seen race conditions with running setuptools in parallel, so more
> isolation is better. OTOH, if 2 setuptools scripts run at the same time on
> the same machine they might still race (same homedir, same /tmp dir).
> - Retrying due to flakes will be faster, but if something is broken you'll
> need to write 4x the number of "run python precommit" phrases.
>

there is also "retest this please" which will rerun all precommits.


> - Increased parallelism may also run into quota issues with Dataflow.
>
> What benefits do you see from splitting up the jobs?
>
> On Mon, Dec 9, 2019 at 4:17 PM Chad Dombrova  wrote:
>
>> After this PR goes in should we revisit breaking up the python tests into
>> separate jenkins jobs by python version?  One of the problems with that
>> plan originally was that we lost the parallelism that gradle provides
>> because we were left with only one tox task per jenkins job, and so the
>> total time to complete all python jenkins jobs went up a lot.  With
>> pytest + xdist we should hopefully be able to keep the parallelism even
>> with just one tox task.  This could be a big win.  I feel like I'm spending
>> more time monitoring and re-queuing timed-out jenkins jobs lately than I am
>> writing code.
>>
>> On Mon, Dec 9, 2019 at 10:32 AM Udi Meiri  wrote:
>>
>>> This PR  (in review)
>>> migrates py27-gcp to using pytest.
>>> It reduces the testPy2Gcp task down to ~13m
>>> 
>>> (from ~45m). This speedup will probably be lower once all 8 tasks are using
>>> pytest.
>>> It also adds 5 previously uncollected tests.
>>>
>>

Re: Python: pytest migration update

[Udi Meiri]

I have given this some thought honestly don't know if splitting into
separate jobs will help.
- I have seen race conditions with running setuptools in parallel, so more
isolation is better. OTOH, if 2 setuptools scripts run at the same time on
the same machine they might still race (same homedir, same /tmp dir).
- Retrying due to flakes will be faster, but if something is broken you'll
need to write 4x the number of "run python precommit" phrases.
- Increased parallelism may also run into quota issues with Dataflow.

What benefits do you see from splitting up the jobs?

On Mon, Dec 9, 2019 at 4:17 PM Chad Dombrova  wrote:

> After this PR goes in should we revisit breaking up the python tests into
> separate jenkins jobs by python version?  One of the problems with that
> plan originally was that we lost the parallelism that gradle provides
> because we were left with only one tox task per jenkins job, and so the
> total time to complete all python jenkins jobs went up a lot.  With
> pytest + xdist we should hopefully be able to keep the parallelism even
> with just one tox task.  This could be a big win.  I feel like I'm spending
> more time monitoring and re-queuing timed-out jenkins jobs lately than I am
> writing code.
>
> On Mon, Dec 9, 2019 at 10:32 AM Udi Meiri  wrote:
>
>> This PR  (in review) migrates
>> py27-gcp to using pytest.
>> It reduces the testPy2Gcp task down to ~13m
>> 
>> (from ~45m). This speedup will probably be lower once all 8 tasks are using
>> pytest.
>> It also adds 5 previously uncollected tests.
>>
>


smime.p7s
Description: S/MIME Cryptographic Signature

request for access: pypi and dockerhub

[Udi Meiri]

Hi,

I'm following the release guide
, and it says I need
access to a couple of repos:
- apache_beam pypi project, need maintainer or owner grant, user: udim
- DockerHub Push Permission, need to be part of "maintainer team", user:
udim

Thanks!


smime.p7s
Description: S/MIME Cryptographic Signature

Re: Python: pytest migration update

[Chad Dombrova]

After this PR goes in should we revisit breaking up the python tests into
separate jenkins jobs by python version?  One of the problems with that
plan originally was that we lost the parallelism that gradle provides
because we were left with only one tox task per jenkins job, and so the
total time to complete all python jenkins jobs went up a lot.  With
pytest + xdist we should hopefully be able to keep the parallelism even
with just one tox task.  This could be a big win.  I feel like I'm spending
more time monitoring and re-queuing timed-out jenkins jobs lately than I am
writing code.

On Mon, Dec 9, 2019 at 10:32 AM Udi Meiri  wrote:

> This PR  (in review) migrates
> py27-gcp to using pytest.
> It reduces the testPy2Gcp task down to ~13m
> 
> (from ~45m). This speedup will probably be lower once all 8 tasks are using
> pytest.
> It also adds 5 previously uncollected tests.
>

Re: Precommits fire for wrong PRs

[Kenneth Knowles]

One reason it would trigger is that it modified
buildSrc/.../BeamModulePlugin.groovy

Edits there are rare. But if it is a problem, each of the "applyXYZNature"
methods could and probably should be moved to a separate plugin, maybe with
a little work.

Kenn

On Thu, Dec 5, 2019 at 2:38 AM Michał Walenia 
wrote:

> Hi all,
> I noticed that sometimes the precommit jobs are launched for unrelated
> PRs, eg. Python or website precommit runs for some Java and gradle changes.
> AFAIK, the 'responsibility regions' for the jobs are defined in Job DSL
> scripts and are regexes that file changed paths are checked against.
> for Python PreCommit the paths are:
>   '^model/.*$',
>   '^sdks/python/.*$',
>   '^release/.*$',
>   '^build.gradle$',
>   '^buildSrc/.*$',
>   '^gradle/.*$',
>   '^gradle.properties$',
>   '^gradlew$',
>   '^gradle.bat$',
>   '^settings.gradle$'
>
> and for Website:
>   '^website/.*$'
>   '^build.gradle$',
>   '^buildSrc/.*$',
>   '^gradle/.*$',
>   '^gradle.properties$',
>   '^gradlew$',
>   '^gradle.bat$',
>   '^settings.gradle$'
>
> What I don't understand is why they both triggered for Łukasz's PR
> , which touches some Java
> files, Jenkins scripts and two non top-level gradle files.
> Can anyone shed some light on this? I'd like to understand it thoroughly -
> by fixing misfires of precommits it would be possible to alleviate the
> strain on our Jenkins workers.
> Thanks!
> Michal
> --
>
> Michał Walenia
> Polidea  | Software Engineer
>
> M: +48 791 432 002 <+48791432002>
> E: michal.wale...@polidea.com
>
> Unique Tech
> Check out our projects! 
>

Quota limitation for Java tests

[Kirill Kozlov]

Hello everyone!

It looks like JavaPostCommit Jenkins tests [1] are failing due to CPU quota
limitations.
Could someone please look into this?

[1]
https://builds.apache.org/job/beam_PostCommit_Java/4838/testReport/junit/org.apache.beam.examples.complete/TrafficMaxLaneFlowIT/testE2ETrafficMaxLaneFlow/

--
Kirill

Hello Beam Developers!

[Daniel Collins]

My name is Daniel, I'm a developer on the Cloud Pub/Sub Team at Google in New 
York.

I'm looking to make some contributions to the Cloud Pub/Sub integration with 
beam. I was hoping that I could be added to the JIRA as a contributor.

Looking forward to working with everyone!

Re: Poor Python 3.x performance on Dataflow?

[Valentyn Tymofieiev]

For now we should run Py3 and Py2 tests alongside each other to get a
side-by-side comparison. I suggest we open a Jira ticket to investigate the
difference in performance . We have limited performance test coverage on
Python 3 in Beam, so more Py3 tests would help a lot here, thanks for
adding them.

On Fri, Dec 6, 2019 at 9:43 AM Robert Bradshaw  wrote:

> This is very surprising--I would expect the times to quite similar. Do
> you have profiles for where the (difference in) time is spent? With
> differences like these, I wonder if there are issues with container
> setup (e.g. some things not being installed or cached) for Python 3.
>
> On Fri, Dec 6, 2019 at 9:06 AM Kamil Wasilewski
>  wrote:
> >
> > Hi all,
> >
> > Python 2.7 won't be maintained past 2020 and that's why we want to
> migrate all Python performance tests in Beam from Python 2.7 to Python 3.7.
> However, I was surprised by seeing that after switching Dataflow tests to
> Python 3.x they are a few times slower. For example, the same ParDo test
> that takes approx. 8 minutes to run on Python 2.7 needs approx. 21 minutes
> on Python 3.x. You can find all the results I gathered and the setup here.
> >
> > Do you know any possible reason for this? This issue makes it impossible
> to do the migration, because of the limited resources on Jenkins (almost
> every job would be aborted).
> >
> > Thanks,
> > Kamil
>

Python: pytest migration update

[Udi Meiri]

This PR  (in review) migrates
py27-gcp to using pytest.
It reduces the testPy2Gcp task down to ~13m

(from ~45m). This speedup will probably be lower once all 8 tasks are using
pytest.
It also adds 5 previously uncollected tests.


smime.p7s
Description: S/MIME Cryptographic Signature

Re: Support alias in ElasticsearchIO

[Pablo Estrada]

+j...@nanthrax.net  Do you have thoughts on this JB?

On Sun, Dec 1, 2019 at 1:12 AM Jing Chen  wrote:

> Hi folks,
> afaik, the existing ElasticsearchIO doesn't support alias, all operations
> are based on index name at the moment.
>
> While alias is a powerful feature to manage indices (at least from ES5, an
> alias could be associated with more than one alias), there are multiple use
> cases which might need alias, e.g logging management.
>
> I would love to hear opinion from the community on whether or not we
> should support alias in ElasticsearchIO, and blockers or suggestion on
> implementation.
>
> Thanks
> Jing
>

Beam Dependency Check Report (2019-12-09)

[Apache Jenkins Server]

High Priority Dependency Updates Of Beam Python SDK:


  Dependency Name
  Current Version
  Latest Version
  Release Date Of the Current Used Version
  Release Date Of The Latest Release
  JIRA Issue
  
google-cloud-datastore
1.7.4
1.10.0
2019-05-27
2019-10-21BEAM-8443
mock
2.0.0
3.0.5
2019-05-20
2019-05-20BEAM-7369
oauth2client
3.0.0
4.1.3
2018-12-10
2018-12-10BEAM-6089
pytest
4.6.7
5.3.1
None
2019-12-02BEAM-8606
Sphinx
1.8.5
2.2.2
2019-05-20
2019-12-09BEAM-7370
tenacity
5.1.5
6.0.0
2019-11-11
2019-11-11BEAM-8607
High Priority Dependency Updates Of Beam Java SDK:


  Dependency Name
  Current Version
  Latest Version
  Release Date Of the Current Used Version
  Release Date Of The Latest Release
  JIRA Issue
  
com.alibaba:fastjson
1.2.49
1.2.62
2018-08-04
2019-10-07BEAM-8632
com.datastax.cassandra:cassandra-driver-core
3.6.0
4.0.0
2018-08-29
2019-03-18BEAM-8674
com.datastax.cassandra:cassandra-driver-mapping
3.6.0
3.8.0
2018-08-29
2019-10-29BEAM-8749
com.esotericsoftware:kryo
4.0.2
5.0.0-RC4
2018-03-20
2019-04-14BEAM-5809
com.esotericsoftware.kryo:kryo
2.21
2.24.0
2013-02-27
2014-05-04BEAM-5574
com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin
0.20.0
0.27.0
2019-02-11
2019-10-21BEAM-6645
com.github.spotbugs:spotbugs
3.1.12
4.0.0-beta4
2019-03-01
2019-09-18BEAM-7792
com.github.spotbugs:spotbugs-annotations
3.1.12
4.0.0-beta4
2019-03-01
2019-09-18BEAM-6951
com.google.api:gax-grpc
1.38.0
1.51.0
2019-02-05
2019-12-04BEAM-8676
com.google.api.grpc:grpc-google-cloud-datacatalog-v1beta1
0.27.0-alpha
0.30.0-alpha
2019-10-03
2019-11-20BEAM-8853
com.google.api.grpc:grpc-google-cloud-pubsub-v1
1.43.0
1.84.0
2019-01-23
2019-12-04BEAM-8677
com.google.api.grpc:grpc-google-common-protos
1.12.0
1.17.0
2018-06-29
2019-10-04BEAM-8633
com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta1
0.44.0
0.85.0
2019-01-23
2019-12-05BEAM-8678
com.google.api.grpc:proto-google-cloud-bigtable-v2
0.44.0
1.7.1
2019-01-23
2019-11-13BEAM-8679
com.google.api.grpc:proto-google-cloud-datacatalog-v1beta1
0.27.0-alpha
0.30.0-alpha
2019-10-03
2019-11-20BEAM-8854
com.google.api.grpc:proto-google-cloud-datastore-v1
0.44.0
0.85.0
2019-01-23
2019-12-05BEAM-8680
com.google.api.grpc:proto-google-cloud-pubsub-v1
1.43.0
1.84.0
2019-01-23
2019-12-04BEAM-8681
com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1
1.6.0
1.47.0
2019-01-23
2019-12-05BEAM-8682
com.google.api.grpc:proto-google-common-protos
1.12.0
1.17.0
2018-06-29
2019-10-04BEAM-6899
com.google.apis:google-api-services-bigquery
v2-rev20181221-1.28.0
v2-rev20190917-1.30.3
2019-01-17
2019-10-09BEAM-8684
com.google.apis:google-api-services-clouddebugger
v2-rev20181114-1.28.0
v2-rev20191003-1.30.3
2019-01-17
2019-10-19BEAM-8750
com.google.apis:google-api-services-cloudresourcemanager
v1-rev20181015-1.28.0
v2-rev20191115-1.30.3
2019-01-17
2019-11-25BEAM-8751
com.google.apis:google-api-services-dataflow
v1b3-rev20190927-1.28.0
v1beta3-rev12-1.20.0
2019-10-11
2015-04-29BEAM-8752
com.google.apis:google-api-services-pubsub
v1-rev20181213-1.28.0
v1-rev2019-1.30.3
2019-01-18
2019-11-26BEAM-8753
com.google.apis:google-api-services-storage
v1-rev20181109-1.28.0
v1-rev20191011-1.30.3
2019-01-18
2019-10-30BEAM-8754
com.google.auth:google-auth-library-credentials
0.13.0
0.18.0
2019-01-17
2019-10-09BEAM-6478
com.google.auth:google-auth-library-oauth2-http
0.12.0
0.18.0
2018-11-14
2019-10-09BEAM-8685
com.google.cloud:google-cloud-bigquery
1.28.0
1.102.0
2018-04-27
2019-12-05BEAM-8687
com.google.cloud:google-cloud-bigquerystorage
0.79.0-alpha
0.120.0-be