Congratulations!
On Fri, Sep 11, 2020, 5:54 PM Andrew Psaltis
wrote:
> Congrats!
>
> On Sat, Sep 12, 2020 at 7:43 AM Reza Rokni wrote:
>
>> Thanx everyone! Looking forward to being able to contribute more :-)
>>
>> On Sat, Sep 12, 2020 at 4:33 AM Valentyn Tymofieiev
>> wrote:
>>
>>> Congrats!
+1 (binding)
Rebased fork and run internal performance tests.
While doing so, I run into the unit test issue below with the fn_runner
(Python direct runner), which did not occur with 2.21 [1]. That processing
time timers are not supported wasn't an issue previously, because the
timer, though decl
2.23 wasn't marked released in JIRA (which I just did). Please add the step
to the release guide if not already present.
https://issues.apache.org/jira/projects/BEAM/versions/12347145
On Thu, Jul 30, 2020 at 3:05 PM Kyle Weaver wrote:
> Hi Eleanore, there have been no changes to Beam's supporte
It appears that there is coverage missing in the Grafana dashboards (it
could also be that I just don't find it).
For example:
https://apache-beam-testing.appspot.com/explore?dashboard=5751884853805056
The GBK and ParDo tests have a selection for {batch, streaming} and SDK. No
coverage for stream
Congrats!
On Tue, Jun 30, 2020 at 5:05 PM Aizhamal Nurmamat kyzy
wrote:
> Thanks everyone! It's been really nice to work with all of you :)
>
> On Tue, Jun 30, 2020, 1:45 PM Ismaël Mejía wrote:
>
>> Congratulations Aizhamal!
>> And thanks for all the dedication to the project.
>>
>> On Tue, Ju
Congratulations!
On Tue, Jun 16, 2020 at 1:27 PM Valentyn Tymofieiev
wrote:
> Congratulations!
>
> On Tue, Jun 16, 2020 at 11:41 AM Ahmet Altay wrote:
>
>> Congratulations!
>>
>> On Tue, Jun 16, 2020 at 10:05 AM Pablo Estrada
>> wrote:
>>
>>> Yooohooo! Thanks for all your contributions and ha
+1
On Tue, Jun 9, 2020 at 9:41 AM Robert Bradshaw wrote:
> Makes sense to me.
>
> On Tue, Jun 9, 2020 at 8:45 AM Maximilian Michels wrote:
>
>> Thanks of the heads-up, Tyson! It's a sensible decision to remove
>> unsupported runners.
>>
>> -Max
>>
>> On 09.06.20 16:51, Tyson Hamilton wrote:
>>
If all Python dependencies are pre-installed on the yarn container hosts,
then you can use the process environment to spawn processes, like so:
https://github.com/lyft/flinkk8soperator/blob/bb8834d69e8621d636ef2085fdc167a9d2c2bfa3/examples/beam-python/src/beam_example/pipeline.py#L16-L17
Thomas
>
> I think the "set_version.sh" script could be called in the release
> scripts to remove the -SNAPSHOT suffix on the release branch.
>
>
I would expect the release branch to have the next -SNAPSHOT version (not
the case currently):
https://github.com/apache/beam/blob/release-2.20.0/gradle.proper
Hi Jacek,
Most of the developer documentation can be found on the cwiki, in this case
https://cwiki.apache.org/confluence/display/BEAM/Gradle+Tips
To build just a runner, for example, Flink:
./gradlew :runners:flink:1.10:job-server:runShadow
Thomas
On Sat, May 23, 2020 at 10:29 AM Jacek Lasko
Please note https://github.com/apache/beam/pull/11777 - another bug we
found while trying to upgrade to 2.21
Other than that things look good.
On Thu, May 21, 2020 at 4:14 PM Ahmet Altay wrote:
> +1 (again, validated with new whl files.)
>
> What about https://issues.apache.org/jira/browse/BEAM
On Thu, May 21, 2020 at 4:08 PM Robert Bradshaw wrote:
> On Thu, May 21, 2020 at 3:55 PM Luke Cwik wrote:
>
>> The 2.22 release is also being worked on. Rather allow that release pick
>> up anything that isn't critical.
>>
>
> +1. The question is whether this will result in broken installs (e.g.
; https://pip.pypa.io/en/stable/reference/pip/#pep-517-and-518-support
> It would help when using tools like pip ("pip wheel"), but I'm not sure
> what the alternative for "python setup.py sdist" is.
>
>
> On Thu, May 7, 2020 at 10:40 PM Thomas Weise wrote:
>
May 7, 2020 at 2:25 PM Udi Meiri wrote:
> It's hard to say without more details what's going on. Ahmet you're right
> that it installs build-requirements.txt and retries calling
> generate_proto_files().
>
> Thomas, were there additional stacktraces? (after a "Dur
bably not the issue, but double checking: are you running "pip install
> -r sdks/python/build-requirements.txt" first?
>
> On Wed, May 6, 2020 at 7:22 PM Thomas Weise wrote:
>
>> I'm working on rebasing our fork to 2.21.0 and run into a problem
>> installing grpcio
I'm working on rebasing our fork to 2.21.0 and run into a problem
installing grpcio-tools that leads to *ModuleNotFoundError: No module named
'grpc_tools' *(see details below)
I cannot reproduce this locally.
Any =suggestions on what to look for?
Thanks,
Thomas
[?25hBuilding wheels for collect
e URLs is fine with me as long as the old urls will work
>>>>> too.
>>>>>
>>>>> But do we need to change the filenames for the blog posts to
>>>>> accomplish that? It's nice that the blog post markdown files start with a
>>>>>
+1 for removing the default runner. It has always been the Beam user
expectation that a runner needs to be selected.
"PortableRunner" isn't a runner (despite its name) - it's a proxy to a
runner that the user specifies via job_endpoint.
Thanks for cleaning this up!
On Thu, Apr 30, 2020 at 10:11
For changed URLs, will previous URLs be mapped to avoid broken external
links?
On Thu, Apr 30, 2020 at 9:34 AM Aizhamal Nurmamat kyzy
wrote:
> Hi,
>
> To give a little more context regarding the URLs, the date should still
> appear on the blog post, but not on the URL.
> For example, we'd have:
er it after the current work is merged to 2.21.0.
>>>>
>>>> Thank you all for providing feedback and ideas.
>>>>
>>>>
>>>> On Thu, Apr 16, 2020 at 10:24 AM Kyle Weaver
>>>> wrote:
>>>>
>>>>> > I tri
Here is the release tag: https://github.com/apache/beam/releases/tag/v2.20.0
On Fri, Apr 24, 2020 at 9:28 AM Kyle Weaver wrote:
> > Is is possible we are missing git tag for this release? I cannot find it.
>
> You mean https://github.com/apache/beam/tree/release-2.20.0?
>
> On Fri, Apr 24, 2020
t;>>
>>> Please let me know what you think and if there are other things can be
>>> improved.
>>>
>>> Hannah
>>>
>>>
>>>
>>> On Wed, Apr 15, 2020 at 2:30 PM Kyle Weaver wrote:
>>>
>>>> Looks like the
The new feature to assemble licenses is very useful but appears to add
several minutes (7-8?) build time to jobs that need to build a container.
Does it also seem to cause occasional build failures?
https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Phrase/131/
Would it be possible
Thanks!
On Wed, Mar 4, 2020 at 1:56 PM Kyle Weaver wrote:
> (Link to previous thread on this issue:
> https://lists.apache.org/thread.html/r1e2640f6d06d0a09ed63cd7134423a390a6cd1332569869cba404df6%40%3Cdev.beam.apache.org%3E
> )
>
> On Wed, Mar 4, 2020 at 1:44 PM Thomas Weise
I run into this problem today and found that removing
https://oss.sonatype.org/content/repositories/staging/ from
buildSrc/src/main/groovy/org/apache/beam/gradle/Repositories.groovy
also resolves the issue.
Is it possible that a flaky repository can poison the gradle cache? Do we
need this reposit
Hi,
I run into the error shown below with local build.
The dependencies actually exist in the release repo, and the build is
successful when removing
https://oss.sonatype.org/content/repositories/staging
from
buildSrc/src/main/groovy/org/apache/beam/gradle/Repositories.groovy
That raises the
Congratulations!
On Mon, Feb 24, 2020, 3:38 PM Ankur Goenka wrote:
> Congratulations Chad!
>
> On Mon, Feb 24, 2020 at 3:34 PM Ahmet Altay wrote:
>
>> Congratulations!
>>
>> On Mon, Feb 24, 2020 at 3:25 PM Sam Bourne wrote:
>>
>>> Nice one Chad. Your typing efforts are very welcomed.
>>>
>>>
Congratulations!
On Mon, Feb 24, 2020 at 6:45 AM Ismaël Mejía wrote:
> Congrats Jincheng!
>
> On Mon, Feb 24, 2020 at 1:39 PM Gleb Kanterov wrote:
>
>> Congratulations!
>>
>> On Mon, Feb 24, 2020 at 1:18 PM Hequn Cheng wrote:
>>
>>> Congratulations Jincheng, well deserved!
>>>
>>> Best,
>>> H
Congratulations!
On Tue, Feb 18, 2020 at 8:33 AM Ismaël Mejía wrote:
> Congrats Alex! Well done!
>
> On Tue, Feb 18, 2020 at 5:10 PM Gleb Kanterov wrote:
>
>> Congratulations!
>>
>> On Tue, Feb 18, 2020 at 5:02 PM Brian Hulette
>> wrote:
>>
>>> Congratulations Alex! Well deserved!
>>>
>>> On
+1 for dropping Flink 1.7 support
I agree that we should avoid the SNAPSHOT dependency.
Generally I think that the version support should remain based on user
feedback and independent of the Flink release policy. If there are Beam
users that require an older Flink version and it is difficult to u
Impressive, probably the fastest/smoothest Beam release so far.
On Mon, Feb 3, 2020 at 10:45 AM Boyuan Zhang wrote:
> I'm happy to announce that we have unanimously approved this release.
>
> There are 5 approving votes, 4 of which are binging:
> * Ahmet Altay
> * Ismaël Mejía
> * Jean-Baptiste
I don't have anything conclusive yet; it could also be related to our
infra. I would not block the release.
Thomas
On Wed, Jan 22, 2020 at 1:01 PM Udi Meiri wrote:
> Thomas, please let us know if you learn more about possible root causes to
> the regression you're seeing.
> Also, if you believe
When trying to upgrade our fork from 2.16 to 2.18, we see a significant
performance degradation. This applies to Flink portable streaming with the
Python SDK. I don't know what the cause is yet.
If anyone else has done validation with a similar setup and this RC, it
would be good to know the resul
+1 for the namespace proposal.
It is similar to github repos. Top-level is the org, then single level for
repo (beam-abc, beam-xzy, ..)
On Wed, Jan 15, 2020 at 1:45 PM Robert Bradshaw wrote:
> Various tags of the same image should at least logically be the same
> thing, so I agree that we sho
Congratulations!
On Mon, Dec 23, 2019 at 1:39 PM Udi Meiri wrote:
> Congrats Kasia!
>
> On Mon, Dec 23, 2019 at 1:23 PM Kyle Weaver wrote:
>
>> Congrats Kasia! And thanks for sharing, Pablo.
>>
>> On Mon, Dec 23, 2019 at 4:16 PM Pablo Estrada wrote:
>>
>>> Hi everyone,
>>>
>>> Please join me
ing, if we divided the key range across the workers.
>
State caching and load balancing are mutually exclusive and I added a check
to enforce that. For the use case that I'm looking at, the cost of state
access compared to that of running the expensive / high latency operations
is tiny to
e();
>
>
> On Mon, Nov 25, 2019 at 10:05 AM Robert Bradshaw
> wrote:
>
>> boot.go could be updated to recognize NO_ARTIFACTS_STAGED_TOKEN as
>> well. (Should this constant be put in a common location?)
>>
>> On Sat, Nov 23, 2019 at 9:16 AM Thomas Weis
JIRA: https://issues.apache.org/jira/browse/BEAM-8816
On Thu, Nov 21, 2019 at 10:44 AM Thomas Weise wrote:
> Hi Luke,
>
> Thanks for the background and it is exciting to see the progress on the
> SDF side. It will help with this use case and many other challenges. I
> imagine
JIRA: https://issues.apache.org/jira/browse/BEAM-8815
On Fri, Nov 22, 2019 at 5:31 PM Thomas Weise wrote:
> I'm running into the issue Kyle points out when I try to run a pipeline
> that does not use artifact staging:
>
> 2019-11-23
I'm running into the issue Kyle points out when I try to run a pipeline
that does not use artifact staging:
2019-11-23 01:09:18,442 WARN
org.apache.beam.runners.fnexecution.artifact.AbstractArtifactRetrievalService
- GetManifest for
/tmp/beam-artifact-staging/job_53cad419-a8c0-472c-8486-f795cc88
We have not seen the issue with Python 3.6 on 2.16+ after applying this
patch. 🎉
Thanks!
On Thu, Nov 21, 2019 at 4:41 PM Thomas Weise wrote:
> We are currently verifying the patch. Will report back tomorrow.
>
> On Thu, Nov 21, 2019 at 8:40 AM Valentyn Tymofieiev
> wrote:
>
&
ython.org/issue34572, it was very helpful.
>
> On Thu, Nov 21, 2019 at 8:25 AM Thomas Weise wrote:
>
>> Valentyn, thanks a lot for following up on this.
>>
>> If the change can be cherry picked in isolation, we should be able to
>> verify this soon (with 2.16).
>>
hat would add a basic form of
> dynamic work rebalancing for all runners that decide to use it
>
> 1: https://github.com/apache/beam/pull/10045
> 2: https://github.com/apache/beam/pull/10065
> 3: https://github.com/apache/beam/pull/10074
>
> On Wed, Nov 20, 2019 at 10:49 AM Thomas
Valentyn, thanks a lot for following up on this.
If the change can be cherry picked in isolation, we should be able to
verify this soon (with 2.16).
On Thu, Nov 21, 2019 at 8:12 AM Valentyn Tymofieiev
wrote:
> To close the loop here: To my knowledge this issue affects all Python 3
> users of P
Congratulations!
On Wed, Nov 20, 2019, 7:56 PM Chamikara Jayalath
wrote:
> Congrats!!
>
> On Wed, Nov 20, 2019 at 5:21 PM Daniel Oliveira
> wrote:
>
>> Thank you everyone! I won't let you down. o7
>>
>> On Wed, Nov 20, 2019 at 2:12 PM Ruoyun Huang wrote:
>>
>>> Congrats Daniel!
>>>
>>> On Wed
We found a problem with uneven utilization of SDK workers causing excessive
latency with Streaming/Python/Flink. Remember that with Python, we need to
execute multiple worker processes on a machine instead of relying on
threads in a single worker, which requires the runner to make a decision to
whi
It would be great to have an index for those materials.
Maybe as cwiki page, which is easy to edit and watch. Similar to:
https://cwiki.apache.org/confluence/display/BEAM/Design+Documents
Thomas
On Fri, Nov 15, 2019 at 10:52 AM Kenneth Knowles wrote:
> We have a section for this:
> https://b
Any update regarding the release?
The list still shows 10 open issues:
https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20fixVersion%20%3D%202.17.0%20and%20resolution%20is%20EMPTY
Is the RC blocked on those?
On Mon, Oct 28, 2019 at 12:46 PM Ahmet Altay wrote:
>
>
> On
Congratulations!
On Fri, Nov 15, 2019 at 6:34 AM Connell O'Callaghan
wrote:
> Well done Brian!!!
>
> Kenn thank you for sharing
>
> On Fri, Nov 15, 2019 at 6:31 AM Cyrus Maden wrote:
>
>> Congrats Brian!
>>
>> On Fri, Nov 15, 2019 at 5:25 AM Ismaël Mejía wrote:
>>
>>> Congratulations Bria
Done, you should be all set.
On Thu, Nov 14, 2019 at 9:57 AM Elliotte Rusty Harold
wrote:
> Hello,
>
> May I please have access to edit the Wiki? username is elharo
>
> Thanks.
>
> --
> Elliotte Rusty Harold
> elh...@ibiblio.org
>
Awesome, thanks Chad!
On Wed, Nov 13, 2019 at 10:26 PM Chad Dombrova wrote:
> Hi Thomas,
>
>
>> Will this include the ability for users to configure logging via pipeline
>> options?
>>
>
> We're working on a proposal to allow pluggable logging handlers that can
> be configured via pipeline optio
+1 for using a logger hierarchy.
Will this include the ability for users to configure logging via pipeline
options?
On Wed, Nov 13, 2019 at 11:04 AM Chad Dombrova wrote:
>
> On Wed, Nov 13, 2019 at 10:52 AM Robert Bradshaw
> wrote:
>
>> I would be in favor of using module-level loggers as wel
FWIW there are currently at least 2 instances of capability matrix [1] [2].
[1] has been in need of a refresh for a while.
[2] is more useful but only covers portable runners and is hard to find.
Thomas
[1] https://beam.apache.org/documentation/runners/capability-matrix/
[2] https://s.apache.or
t;>> Using timestamp as a tag is an option as well, as long as runners know
>>>> which timestamp they should use.
>>>>
>>>> Hannah
>>>>
>>>> On Wed, Oct 2, 2019 at 10:13 AM Alan Myrvold
>>>> wrote:
>>>>
The expansion service can be provided by the job server, as done in the
Flink runner. It needs to be available at pipeline construction time, but
there is no need to run a separate service.
Thomas
On Mon, Nov 4, 2019 at 12:03 PM Robert Bradshaw wrote:
> On Mon, Nov 4, 2019 at 11:54 AM Chamikara
This thread was very helpful to find more detail in
https://jira.apache.org/jira/browse/BEAM-7870
It would be great to have cross-language current state mentioned as top
level entry on https://beam.apache.org/roadmap/
On Mon, Sep 16, 2019 at 6:07 PM Chamikara Jayalath
wrote:
> Thanks for the n
More here:
https://lists.apache.org/thread.html/07131e314e229ec60100eaa2c0cf6dfc206bf2b0f78c3cee9ebb0bda@%3Cdev.beam.apache.org%3E
On Fri, Nov 1, 2019 at 10:56 AM Chamikara Jayalath
wrote:
> I think it makes sense to override published docker images with locally
> built versions when testing HE
monstrating best practices for
> building an image that contains both Flink and Beam SDK dependencies? That
> would be useful.
>
> -chad
>
>
> On Fri, Nov 1, 2019 at 10:18 AM Thomas Weise wrote:
>
>> For folks looking to run Beam on Flink on k8s, see update in [1]
>
; 1.9+, which has implications for creation of task managers in kubernetes.
>>
>> [1]
>> https://docs.google.com/document/d/1h68XOG-EyOFfcomd2N7usHK1X429pJSMiwZwAXCwx1k/edit#heading=h.72szmms7ufza
>> [2] https://issues.apache.org/jira/browse/FLINK-12761
>>
>> -
OPBACK in this case seems fair.
>
> On 31.10.19 16:04, Thomas Weise wrote:
> >
> >
> > On Thu, Oct 31, 2019 at 3:55 AM Maximilian Michels > <mailto:m...@apache.org>> wrote:
> >
> > > Thanks for clarifying. So when I run "./flink my_pipe
On Thu, Oct 31, 2019 at 3:55 AM Maximilian Michels wrote:
> > Thanks for clarifying. So when I run "./flink my_pipeline.jar" or
> > upload the jar via the REST API (and its main method invoked on the
> > master) then [auto] reads the config and does the right thing, but if
> > I do java my_pipel
The current semantics of flink_master are tied to the Flink Java API. The
Flink client / Java API isn't a "REST API". It now uses the REST API
somewhere deep in RemoteEnvironment when the flink_master value is
host:port, but it does a lot of other things as well, such are parsing
config files and r
s
>> proposal is simply isomorphic to defining new primitive transforms
>> which some (all?) runners are just expected to understand.
>>
>> On Tue, Aug 20, 2019 at 10:11 AM Thomas Weise wrote:
>> >
>> >
>> >
>> > On Tue, Aug 20, 2019 at 8
Hi Chad,
Thanks for bringing up this discussion. I think for most of it the Flink
list would be the better suited place, but given that there are more and
more Beam users interested to deploy on Beam on Flink on k8s, I will leave
the placement to you ;-)
As for the differences between FlinkK8sOpe
Thanks Mikhail!
JIRA issues pointed to a resolved ticket.
This should list the open items:
https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20fixVersion%20%3D%202.17.0%20and%20resolution%20is%20EMPTY
On Thu, Oct 24, 2019 at 11:16 AM Mikhail Gryzykhin
wrote:
> Hello everyo
cessful.
>
> Many thanks,
> Matthias
>
> On Tue, 13 Aug 2019 at 01:49, Thomas Weise wrote:
>
>> Yes, everyone should have comment access for this to make sense. Sorry
>> for the confusion.
>>
>>
>> On Mon, Aug 12, 2019 at 5:30 PM Kenneth Knowles wro
Thanks for spotting this.
Access granted (for user "jincheng").
On Thu, Oct 10, 2019 at 12:01 AM jincheng sun
wrote:
> Hi all,
>
> The docker config has been removed with the latest python3 docker related
> commit [1], So the command:
> > ./gradlew :sdks:python:container:docker
> in `Python
Depends on JIRA volume, also. There are projects where the dev@ is heavily
populated with create JIRA notifications (and other traffic goes under a
bit).
I'm generally in favor of making the creation of JIRAs more visible. But if
there isn't broad enough interest these notifications can still be s
Also
>> portrayal of coding as shooting lasers.
>>
>> Kenn
>>
>> On Wed, Oct 9, 2019 at 8:56 PM Thomas Weise wrote:
>>
>>> Cute, could be used as screensaver!
>>>
>>>
>>> On Wed, Oct 9, 2019 at 5:16 PM Pablo Estrada wrote:
Cute, could be used as screensaver!
On Wed, Oct 9, 2019 at 5:16 PM Pablo Estrada wrote:
> I saw this on the internets. It's pretty cute, there's all of us at some
> point - I think:
> https://www.youtube.com/watch?v=lRToJBDi7C0
>
> Best
> -P.
>
It probably makes sense to separate official project channels from external
ones like Beam Summit and meetups. Beam Summit is about Beam, but it is
"third party" and not under the project umbrella. Operation of the youtube
channel might also need clarification.
On Wed, Oct 9, 2019 at 4:35 PM Robe
Welcome, Diksha!
On Fri, Oct 4, 2019 at 8:47 AM diksha gupta
wrote:
> Hi, I am Diksha Gupta, outreachy intern.
> I will work with your host on beamSQL.
>
Welcome, Manuela!
For getting familiar with the Beam development environment in general, I
would recommend to take a look at:
https://beam.apache.org/get-started/quickstart-py/
https://cwiki.apache.org/confluence/display/BEAM/Nexmark
For contributing and collaborating in general, please take a
I think there is a different reason why the release manager should probably
merge/approve all PRs that go into the release branch while the release is
in progress:
If/when the need arises for another RC, then only those changes should be
included that are deemed blockers or explicitly agreed. Othe
Tue, Sep 24, 2019 at 7:08 PM Thomas Weise wrote:
> Hi Hannah,
>
> I believe this is unexpected from the developer perspective. When building
> something locally, we do expect that to be used. We may need to change to
> not pull when the image is available locally, at least when it
Congratulations, Alan!
On Mon, Sep 30, 2019 at 4:47 AM Ismaël Mejía wrote:
> Congrats Alan!
>
> On Mon, Sep 30, 2019, 11:20 AM Tanay Tummalapalli
> wrote:
>
>> Congratulations, Alan!
>>
>>
>> On Mon, Sep 30, 2019 at 1:03 PM Gleb Kanterov wrote:
>>
>>> Congratulations!
>>>
>>> On Sat, Sep 28,
+1 for adding the coder
Please also add a test here:
https://github.com/apache/beam/blob/master/model/fn-execution/src/main/resources/org/apache/beam/model/fnexecution/v1/standard_coders.yaml
On Fri, Sep 27, 2019 at 5:17 PM Chad Dombrova wrote:
> Are there any dissenting votes to making a Bool
perts here who may be able to advise.
>
>
> On Wed, Sep 25, 2019 at 6:58 AM Thomas Weise wrote:
>
>> After running through the entire bisect based on the 2.16 release branch
>> I found that the regression was caused by our own Cython setup. So green
>> light for the 2.
After running through the entire bisect based on the 2.16 release branch I
found that the regression was caused by our own Cython setup. So green
light for the 2.16.0 release.
Thomas
On Tue, Sep 17, 2019 at 1:21 PM Thomas Weise wrote:
> Hi Valentyn,
>
> Thanks for the reminder. The
Hi Hannah,
I believe this is unexpected from the developer perspective. When building
something locally, we do expect that to be used. We may need to change to
not pull when the image is available locally, at least when it is a
snapshot/master branch. Release images should be immutable anyways.
T
+1 for making --experiments=beam_fn_api default.
Can the Dataflow runner driver just remove the setting if it is not
compatible?
On Tue, Sep 17, 2019 at 11:33 AM Maximilian Michels wrote:
> +dev
>
> The beam_fn_api flag and the way it is automatically set is error-prone.
> Is there anything tha
file JIRAs once you have
>>>> additional information.
>>>>
>>>> Valentyn, I think the performance regression could be investigated now,
>>>> by running whatever benchmarks that is available against 2.14, 2.15 and
>>>> head and see if the same regr
Engineer | github.com/ibzib | kcwea...@google.com
>
>
> On Thu, Sep 12, 2019 at 5:48 PM Thomas Weise wrote:
>
>> This should become much better with 2.16 when we have the Docker images
>> prebuilt.
>>
>> Docker is probably still the best option for Python on a
This should become much better with 2.16 when we have the Docker images
prebuilt.
Docker is probably still the best option for Python on a JVM based runner
in a local environment that does not have a development setup.
On Thu, Sep 12, 2019 at 1:09 PM Kyle Weaver wrote:
> +dev I think we shoul
David, thanks for working on the Flink 1.9 support, this is very exciting!
Previous discussion on list/PRs (predating even the JIRA referenced by
Max), already pointed to the removal of support for Flink 1.5/1.6 support.
Although we generally need to consider what the Beam users need (not just
wha
On Fri, Sep 6, 2019 at 6:23 PM Ahmet Altay wrote:
>
>
> On Fri, Sep 6, 2019 at 6:17 PM Thomas Weise wrote:
>
>>
>>
>> On Fri, Sep 6, 2019 at 2:24 PM Valentyn Tymofieiev
>> wrote:
>>
>>> +Mark Liu has added some benchmarks running across
>
>
>
> On Fri, Sep 6, 2019 at 8:38 AM Ahmet Altay wrote:
>
>> +Valentyn Tymofieiev do we have benchmarks in
>> different python versions? Was there a recent change that is specific to
>> python 3.x ?
>>
>> On Fri, Sep 6, 2019 at 8:36 AM Thomas Weise wrote:
&g
The issue is only visible with Python 3.6, not 2.7.
If there is a framework in place to add a streaming test, that would be
great. We would use what we have internally as starting point.
On Thu, Sep 5, 2019 at 5:00 PM Ahmet Altay wrote:
>
>
> On Thu, Sep 5, 2019 at 4:15 PM Thomas Wei
not there yet (it's in progress).
> Best
> -P.
>
> On Thu, Sep 5, 2019 at 3:35 PM Thomas Weise wrote:
>
>> It probably won't be practical to do a bisect due to the high cost of
>> each iteration with our fork/deploy setup.
>>
>> Perhaps it is time t
n on flink
> benchmarks for chicago example?
> +Alan Myrvold +Yifan Zou It
> would be good to have alerts on benchmarks. Do we have such an ability
> today?
>
> [1] https://apache-beam-testing.appspot.com/dashboard-admin
>
> On Thu, Sep 5, 2019 at 3:15 PM Thomas Weise
Hi,
Are there any performance tests run for the Python SDK as part of release
verification (or otherwise as well)?
I see what appears to be a regression in master (compared to 2.14) with our
in-house application (~ 25% jump in cpu utilization and corresponds drop in
throughput).
I wanted to see
35 PM Thomas Weise wrote:
> Awesome, thank you!
>
>
> On Wed, Sep 4, 2019 at 3:22 PM Hannah Jiang
> wrote:
>
>> Hi Thomas
>>
>> I created snapshot images from head as of around 2PM today.
>> You can pull images from gcr.io/apache-beam-testing/beam/sdks/snapsh
Awesome, thank you!
On Wed, Sep 4, 2019 at 3:22 PM Hannah Jiang wrote:
> Hi Thomas
>
> I created snapshot images from head as of around 2PM today.
> You can pull images from gcr.io/apache-beam-testing/beam/sdks/snapshot.
>
> Thanks,
> Hannah
>
> On Wed, Sep 4, 2
om
> head.
>
> Please let me know if you have any questions.
> Hannah
>
> On Wed, Sep 4, 2019 at 10:57 AM Thomas Weise wrote:
>
>> I actually found something in [1], but it is 2.15 unfortunately.
>>
>> [1]
>> https://console.cloud.google.com/gcr/image
I actually found something in [1], but it is 2.15 unfortunately.
[1]
https://console.cloud.google.com/gcr/images/apache-beam-testing/GLOBAL/beam/sdks/release/python2.7?gcrImageListsize=30
On Wed, Sep 4, 2019 at 10:35 AM Thomas Weise wrote:
> Thanks for working on this. Do you happen to h
Thanks for working on this. Do you happen to have publicly accessible
snapshots published for your testing currently (even when the final
location isn't sorted out)?
I would like to use a 2.16 based Python SDK image for working on my
downstream project, but could not find anything in
gcr.io/apache
Yifan, thanks for managing this release. It went smoothly!
On Fri, Aug 23, 2019 at 2:32 PM Kenneth Knowles wrote:
> Nice work!
>
> On Fri, Aug 23, 2019 at 11:26 AM Charles Chen wrote:
>
>> Thank you Yifan!
>>
>> On Fri, Aug 23, 2019 at 11:12 AM Hannah Jiang
>> wrote:
>>
>>> Thank you Yifan!
>
Congrats!
On Mon, Aug 26, 2019 at 3:22 PM Heejong Lee wrote:
> Congratulations! :)
>
> On Mon, Aug 26, 2019 at 2:44 PM Rui Wang wrote:
>
>> Congratulations!
>>
>>
>> -Rui
>>
>> On Mon, Aug 26, 2019 at 2:36 PM Hannah Jiang
>> wrote:
>>
>>> Congratulations Valentyn, well deserved!
>>>
>>> On Mo
cument/d/1h68XOG-EyOFfcomd2N7usHK1X429pJSMiwZwAXCwx1k/edit#heading=h.72szmms7ufza
> [2] https://issues.apache.org/jira/browse/FLINK-12761
>
> -chad
>
>
> On Tue, Aug 13, 2019 at 9:14 AM Thomas Weise wrote:
>
>> There have been a few comments in the doc and I'm goin
> dynamically repartitioning the key ranges. Even in case of fine-grained
> recovery of workers and their key ranges, we would simply generate new
> cache tokens for a particular worker.
>
>
> Thanks,
> Max
>
> On 21.08.19 09:33, Reuven Lax wrote:
> > Dataflow doe
1 - 100 of 444 matches
Mail list logo