Re:Any plan to add support for graph computation

[Ray]

I open an issue on Jira for your review
https://issues.apache.org/jira/projects/BEAM/issues/BEAM-3096?filter=allopenissues






在 2017-10-25 14:43:04，"Ray"  写道：

Hi, Dear Apache Beam Developer,


I am new to Apache Beam and excited to find such wonderful tool to handle 
big-data across different popular runners.
I am just wondering is there any plan to add support for graph computation like 
GraphX on Spark?


* graph representation in PCollection 
* basic statistics like vertex/edge count
* base function like vertex/edge-wise mapreduce task(i.e., count the outgoing 
degree of a vertex)
* base function like subgraph combine/join
* ..


Thanks again for this great Apache Beam you brought to us,
Looking forward to your reply.



【网易自营】好吃到爆！鲜香弹滑加热即食，经典13香/麻辣小龙虾仅75元3斤>>  

JB offline for 5 days

[Jean-Baptiste Onofré]

Hi guys,

Just to let you know that I will be offline for the next 5 days (vacation).

See you end of next week !

Regards
JB
--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Is mergebot down?

[Jean-Baptiste Onofré]

Hi,

I have some issue with mergebot as well.

By the way, I switched Apache Karaf to gitbox and we have a "merge pull request" 
button. Pretty convenient ;)


Regards
JB

On 10/26/2017 12:14 AM, Eugene Kirpichov wrote:

Hello,

On this PR
https://github.com/apache/beam-site/pull/334#issuecomment-338551253 the
mergebot seems to not activate by "@asfgit merge". Jason, could you take a
look?



--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Java pre/post commit test suite breakage

[Robert Bradshaw]

On Oct 25, 2017 1:04 PM, "Kenneth Knowles"  wrote:

Hi all,

I wanted to circle back on this. We've had continual quasi-outages, where
Jenkins occasionally schedules jobs but not reliably. Meanwhile, many of
our most important jobs continued to hit their timeouts when they did get
scheduled. On top of that, I've run the wrong command once before merging a
PR, resulting in a couple broken precommit runs.

Just so everyone knows the actions that have been taken and what we still
need to do:

 - Java postcommit and precommit timeouts bumped to 240 minutes (!!)
 - Seed job made independent of other jobs, so we can experiment safely
 - RAT config fixed

The latency of precommit is still prohibitive for effective use. Not only
are we waiting too long, but the number of workers needed to avoid backlog
is excessive. Here are some ideas for things we can do:

 - Proceed further on the pipeline job, which is what we want to do long
term


+1

 - Precommit: run only on demand, so workers are not congested by automatic
builds; we could also pick and choose what we want to run


I find automatically running is generally helpful, but at the very least we
could cancel redundant runs (e.g. when more commits are pushed).

 - Consider non-Maven build orchestrators that can do dependency-driven
builds


Huge +1. We don't need to waste 240 minutes on Java tests when nothing but
Python changed, and vice-versa. Likewise many subcomponents could be tested
without retesting their bases and siblings.

 - Precommit: run fewer tests; I think this would mean leaving out some
modules that we are OK finding issues in the postcommit, or making them on
demand only


-1. In my experience postcommit failures are much harder to debug and often
mask other breakage (plus impacting everyone's development). This is a
choice of last resort to me; seems we have a lot of other options that
don't reduce coverage.

 - ValidatesRunner tests: Instead of running a single mavenJob, run
multiple, once which just installs while skipping everything else, then
follow it with just running the tests we care about


+1.

 - Examples integration tests: separate from the maven precommit; also run
as sequenced invocations

These are just some ideas; I honestly don't know. I think my upper limit
for default precommit feedback is probably 30 minutes, and even there I am
not very happy.


We should be able to get a good signal in less than 30 minutes. Lint, smoke
tests, etc. I think there is a more extensive set of tests that should pass
before merging.

Also it might not hurt to audit our tests and see if there are a small
number that take a disproportionate amount of time.

Any other suggestions?

Kenn

On Mon, Oct 23, 2017 at 3:29 PM, Kenneth Knowles  wrote:

> I want to wait and get some green from Jenkins running against the HEAD
> groovy scripts to confirm. I haven't sat at my desk long enough to see a
> full `mvn -P release clean verify` yet.
>
> On Mon, Oct 23, 2017 at 3:27 PM, Kenneth Knowles  wrote:
>
>> I was easily able to reproduce all the sorts of failures (and some more!)
>>
>> Here are some things that now work that didn't work, or didn't work
>> correctly, before
>>
>>  - mvn apache-rat:check
>>  - mvn -P release apache-rat:check
>>  - mvn -P release -f somewhere/else/pom.xml apache-rat:check
>>
>> It turns out we had these issues:
>>
>>  - items in .gitignore that our RAT config did not ignore
>>  - sub-modules actually *did* inherit the RAT config, depending on the
>> command you run
>>  - paths in the RAT exclude were relative to current dir
>>  - paths in our RAT exclude that aren't even part of our codebase or
>> generated targets, but just things that Jenkins dropped in the workspace,
>> and we were cloning directly into the workspace
>>
>> I think with the new Jenkins config and the new pom.xml things should
>> work well.
>>
>> Incidentally, Valentyn, changes to Jenkins job DSL groovy scripts do not
>> take effect from being merged but only when the seed job next runs.
(TODO:
>> fix this). I am still trying to get the relevant Jenkins UI pages to load
>> to get your change incorporated into the live scripts, which were
regressed
>> by being run from an old PR (TODO: fix the fact that this can happen).
>>
>> Kenn
>>
>> On Mon, Oct 23, 2017 at 2:21 PM, Valentyn Tymofieiev <
>> valen...@google.com.invalid> wrote:
>>
>>> Thanks a lot!
>>>
>>> Kenn, Were you able to reproduce RAT failures and test the fix locally?
I
>>> think "mvn clean verify -Prelease" still passes for me.
>>>
>>> Timeout for the test suite has been increased in
>>> https://github.com/apache/
>>> beam/pull/4028 .
>>>
>>> On Mon, Oct 23, 2017 at 2:10 PM, Kenneth Knowles >> >
>>> wrote:
>>>
>>> > Wrong link - https://github.com/apache/beam/pull/4027
>>> >
>>> > On Mon, Oct 23, 2017 at 2:10 PM, Kenneth Knowles 
>>> wrote:
>>> >
>>> > > Yea, root cause is the config bug I described. Proposed fix at
>>> > > https://github.com/apache/beam/pull

Re: Java pre/post commit test suite breakage

[Kenneth Knowles]

On Wed, Oct 25, 2017 at 3:59 PM, Lukasz Cwik 
wrote:

> Another suggestion would be to break apart the project into multiple maven
> projects which are released on independent schedules.
>

I think at some point this makes a lot of sense. Libraries of transforms
built against a stable SDK like Java should be OK, and this is a big chunk
of the build time. Fine to move to postcommit / selective precommit too.

Kenn



> Does anyone have any data points for what has worked for other open source
> communities in the past?
> I only have anecdotal evidence that Apache Ant+Ivy has worked well as it
> was much faster then Apache Maven.
>
> On Wed, Oct 25, 2017 at 1:03 PM, Kenneth Knowles 
> wrote:
>
> > Hi all,
> >
> > I wanted to circle back on this. We've had continual quasi-outages, where
> > Jenkins occasionally schedules jobs but not reliably. Meanwhile, many of
> > our most important jobs continued to hit their timeouts when they did get
> > scheduled. On top of that, I've run the wrong command once before
> merging a
> > PR, resulting in a couple broken precommit runs.
> >
> > Just so everyone knows the actions that have been taken and what we still
> > need to do:
> >
> >  - Java postcommit and precommit timeouts bumped to 240 minutes (!!)
> >  - Seed job made independent of other jobs, so we can experiment safely
> >  - RAT config fixed
> >
> > The latency of precommit is still prohibitive for effective use. Not only
> > are we waiting too long, but the number of workers needed to avoid
> backlog
> > is excessive. Here are some ideas for things we can do:
> >
> >  - Proceed further on the pipeline job, which is what we want to do long
> > term
> >  - Precommit: run only on demand, so workers are not congested by
> automatic
> > builds; we could also pick and choose what we want to run
> >  - Consider non-Maven build orchestrators that can do dependency-driven
> > builds
> >  - Precommit: run fewer tests; I think this would mean leaving out some
> > modules that we are OK finding issues in the postcommit, or making them
> on
> > demand only
> >  - ValidatesRunner tests: Instead of running a single mavenJob, run
> > multiple, once which just installs while skipping everything else, then
> > follow it with just running the tests we care about
> >  - Examples integration tests: separate from the maven precommit; also
> run
> > as sequenced invocations
> >
> > These are just some ideas; I honestly don't know. I think my upper limit
> > for default precommit feedback is probably 30 minutes, and even there I
> am
> > not very happy.
> >
> > Any other suggestions?
> >
> > Kenn
> >
> > On Mon, Oct 23, 2017 at 3:29 PM, Kenneth Knowles  wrote:
> >
> > > I want to wait and get some green from Jenkins running against the HEAD
> > > groovy scripts to confirm. I haven't sat at my desk long enough to see
> a
> > > full `mvn -P release clean verify` yet.
> > >
> > > On Mon, Oct 23, 2017 at 3:27 PM, Kenneth Knowles 
> wrote:
> > >
> > >> I was easily able to reproduce all the sorts of failures (and some
> > more!)
> > >>
> > >> Here are some things that now work that didn't work, or didn't work
> > >> correctly, before
> > >>
> > >>  - mvn apache-rat:check
> > >>  - mvn -P release apache-rat:check
> > >>  - mvn -P release -f somewhere/else/pom.xml apache-rat:check
> > >>
> > >> It turns out we had these issues:
> > >>
> > >>  - items in .gitignore that our RAT config did not ignore
> > >>  - sub-modules actually *did* inherit the RAT config, depending on the
> > >> command you run
> > >>  - paths in the RAT exclude were relative to current dir
> > >>  - paths in our RAT exclude that aren't even part of our codebase or
> > >> generated targets, but just things that Jenkins dropped in the
> > workspace,
> > >> and we were cloning directly into the workspace
> > >>
> > >> I think with the new Jenkins config and the new pom.xml things should
> > >> work well.
> > >>
> > >> Incidentally, Valentyn, changes to Jenkins job DSL groovy scripts do
> not
> > >> take effect from being merged but only when the seed job next runs.
> > (TODO:
> > >> fix this). I am still trying to get the relevant Jenkins UI pages to
> > load
> > >> to get your change incorporated into the live scripts, which were
> > regressed
> > >> by being run from an old PR (TODO: fix the fact that this can happen).
> > >>
> > >> Kenn
> > >>
> > >> On Mon, Oct 23, 2017 at 2:21 PM, Valentyn Tymofieiev <
> > >> valen...@google.com.invalid> wrote:
> > >>
> > >>> Thanks a lot!
> > >>>
> > >>> Kenn, Were you able to reproduce RAT failures and test the fix
> > locally? I
> > >>> think "mvn clean verify -Prelease" still passes for me.
> > >>>
> > >>> Timeout for the test suite has been increased in
> > >>> https://github.com/apache/
> > >>> beam/pull/4028 .
> > >>>
> > >>> On Mon, Oct 23, 2017 at 2:10 PM, Kenneth Knowles
> >  > >>> >
> > >>> wrote:
> > >>>
> > >>> > Wrong link - https://github.com/apache/beam/pull/4027
> > >>> >
> >

Re: Update on 2.2.0 release

[Reuven Lax]

I'm still trying to figure the problem out. I think there's a conflict
between an explicit dependency and an implicit one.

On Tue, Oct 24, 2017 at 1:29 PM, Reuven Lax  wrote:

> Update: there is a dependency error in the release branch, so am fixing
> that and retrying.
>
> On Tue, Oct 24, 2017 at 12:43 PM, Reuven Lax  wrote:
>
>> We've been blocked for a couple of weeks on BEAM-3011, however that is
>> now in and I just merged the PR into the release branch. We've been having
>> a lot of problems with the Jenkins test infrastructure for Beam the last
>> few weeks, which is why it took so long to merge.
>>
>> I am now manually verifying that everything passes on the release branch.
>> Once verification completes, I will cut an RC1.
>>
>> Reuven
>>
>> On Mon, Oct 23, 2017 at 9:33 PM, Jean-Baptiste Onofré 
>> wrote:
>>
>>> Hi guys,
>>>
>>> can we move forward on 2.2.0 release ?
>>>
>>> Only two Jira are still open for 2.2.0:
>>>
>>> BEAM-3011 - Pin Runner harness container image in Python SDK
>>> BEAM-2271 - Release guide or pom.xml needs update to avoid releasing
>>> Python binary artifacts
>>>
>>> Both are related to Python SDK.
>>> Can we have an update about this ?
>>> Are they blocker for the 2.2.0 release ?
>>>
>>> @Reuven: I think we are good to do a 2.2.0.RC1. Thoughts ?
>>>
>>> Thanks !
>>> Regards
>>> JB
>>>
>>> On 10/10/2017 12:53 AM, Reuven Lax wrote:
>>>
 I've now cut a release branch for 2.2.0.

 On Mon, Oct 9, 2017 at 2:01 AM, Jean-Baptiste Onofré 
 wrote:

 It sounds good to me.
>
> Regards
> JB
>
>
> On 10/09/2017 12:38 AM, Reuven Lax wrote:
>
> At this point it's taken over three weeks to resolve release-blocking
>> issues, and there are still some remaining. Monday morning I will go
>> ahead
>> and cut an initial release candidate, so we get a snapshot to base
>> on. The
>> remaining blocking issues will either be cherry picked into an
>> additional
>> release candidate or punted to 2.3.0, depending on severity.
>>
>> Reuven
>>
>> On Wed, Sep 20, 2017 at 5:34 AM, Jean-Baptiste Onofré <
>> j...@nanthrax.net>
>> wrote:
>>
>> Hi Reuven,
>>
>>>
>>> thanks for the update. I will pick up some Jira in my bucket (to
>>> verify/double check or implement).
>>>
>>> Regards
>>> JB
>>>
>>>
>>> On 09/20/2017 04:23 AM, Reuven Lax wrote:
>>>
>>> There are 7 issues left on
>>>
 https://issues.apache.org/jira/projects/BEAM/versions/12341044
 after I
 moved some items to 2.3.0 if they appeared to not need to be in
 2.2.0
 (some
 were originally slated for 2.1.0, and there was no apparent
 progress on
 them). Please object if you feel that it's important that any of
 these
 be
 in 2.2.0

 Of the remaining issues, three (BEAM-2956, BEAM-2298, BEAM-2271)
 appear
 to
 already be fixed; please verify this and close the issue if you are
 the
 reporter. The remaining issues (BEAM-2834, BEAM-2858, BEAM-2870,
 BEAM-29540) all have outstanding pull requests. As soon as all of
 these
 pull requests are merged, I will start the release cut.

 Reuven


 --

>>> Jean-Baptiste Onofré
>>> jbono...@apache.org
>>> http://blog.nanthrax.net
>>> Talend - http://www.talend.com
>>>
>>>
>>>
>> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
>

>>> --
>>> Jean-Baptiste Onofré
>>> jbono...@apache.org
>>> http://blog.nanthrax.net
>>> Talend - http://www.talend.com
>>>
>>
>>
>

Re: Java pre/post commit test suite breakage

[Lukasz Cwik]

Another suggestion would be to break apart the project into multiple maven
projects which are released on independent schedules.

Does anyone have any data points for what has worked for other open source
communities in the past?
I only have anecdotal evidence that Apache Ant+Ivy has worked well as it
was much faster then Apache Maven.

On Wed, Oct 25, 2017 at 1:03 PM, Kenneth Knowles 
wrote:

> Hi all,
>
> I wanted to circle back on this. We've had continual quasi-outages, where
> Jenkins occasionally schedules jobs but not reliably. Meanwhile, many of
> our most important jobs continued to hit their timeouts when they did get
> scheduled. On top of that, I've run the wrong command once before merging a
> PR, resulting in a couple broken precommit runs.
>
> Just so everyone knows the actions that have been taken and what we still
> need to do:
>
>  - Java postcommit and precommit timeouts bumped to 240 minutes (!!)
>  - Seed job made independent of other jobs, so we can experiment safely
>  - RAT config fixed
>
> The latency of precommit is still prohibitive for effective use. Not only
> are we waiting too long, but the number of workers needed to avoid backlog
> is excessive. Here are some ideas for things we can do:
>
>  - Proceed further on the pipeline job, which is what we want to do long
> term
>  - Precommit: run only on demand, so workers are not congested by automatic
> builds; we could also pick and choose what we want to run
>  - Consider non-Maven build orchestrators that can do dependency-driven
> builds
>  - Precommit: run fewer tests; I think this would mean leaving out some
> modules that we are OK finding issues in the postcommit, or making them on
> demand only
>  - ValidatesRunner tests: Instead of running a single mavenJob, run
> multiple, once which just installs while skipping everything else, then
> follow it with just running the tests we care about
>  - Examples integration tests: separate from the maven precommit; also run
> as sequenced invocations
>
> These are just some ideas; I honestly don't know. I think my upper limit
> for default precommit feedback is probably 30 minutes, and even there I am
> not very happy.
>
> Any other suggestions?
>
> Kenn
>
> On Mon, Oct 23, 2017 at 3:29 PM, Kenneth Knowles  wrote:
>
> > I want to wait and get some green from Jenkins running against the HEAD
> > groovy scripts to confirm. I haven't sat at my desk long enough to see a
> > full `mvn -P release clean verify` yet.
> >
> > On Mon, Oct 23, 2017 at 3:27 PM, Kenneth Knowles  wrote:
> >
> >> I was easily able to reproduce all the sorts of failures (and some
> more!)
> >>
> >> Here are some things that now work that didn't work, or didn't work
> >> correctly, before
> >>
> >>  - mvn apache-rat:check
> >>  - mvn -P release apache-rat:check
> >>  - mvn -P release -f somewhere/else/pom.xml apache-rat:check
> >>
> >> It turns out we had these issues:
> >>
> >>  - items in .gitignore that our RAT config did not ignore
> >>  - sub-modules actually *did* inherit the RAT config, depending on the
> >> command you run
> >>  - paths in the RAT exclude were relative to current dir
> >>  - paths in our RAT exclude that aren't even part of our codebase or
> >> generated targets, but just things that Jenkins dropped in the
> workspace,
> >> and we were cloning directly into the workspace
> >>
> >> I think with the new Jenkins config and the new pom.xml things should
> >> work well.
> >>
> >> Incidentally, Valentyn, changes to Jenkins job DSL groovy scripts do not
> >> take effect from being merged but only when the seed job next runs.
> (TODO:
> >> fix this). I am still trying to get the relevant Jenkins UI pages to
> load
> >> to get your change incorporated into the live scripts, which were
> regressed
> >> by being run from an old PR (TODO: fix the fact that this can happen).
> >>
> >> Kenn
> >>
> >> On Mon, Oct 23, 2017 at 2:21 PM, Valentyn Tymofieiev <
> >> valen...@google.com.invalid> wrote:
> >>
> >>> Thanks a lot!
> >>>
> >>> Kenn, Were you able to reproduce RAT failures and test the fix
> locally? I
> >>> think "mvn clean verify -Prelease" still passes for me.
> >>>
> >>> Timeout for the test suite has been increased in
> >>> https://github.com/apache/
> >>> beam/pull/4028 .
> >>>
> >>> On Mon, Oct 23, 2017 at 2:10 PM, Kenneth Knowles
>  >>> >
> >>> wrote:
> >>>
> >>> > Wrong link - https://github.com/apache/beam/pull/4027
> >>> >
> >>> > On Mon, Oct 23, 2017 at 2:10 PM, Kenneth Knowles 
> >>> wrote:
> >>> >
> >>> > > Yea, root cause is the config bug I described. Proposed fix at
> >>> > > https://github.com/apache/beam/pull/4019/files. I'm working with
> >>> infra
> >>> > to
> >>> > > sort out other build issues that are probably not related.
> >>> > >
> >>> > > On Mon, Oct 23, 2017 at 1:53 PM, Lukasz Cwik
> >>> 
> >>> > > wrote:
> >>> > >
> >>> > >> The build breakage I outlined is being tracked in
> >>> > >> https://issues.apache.org/jira/browse/BEAM-30

Is mergebot down?

[Eugene Kirpichov]

Hello,

On this PR
https://github.com/apache/beam-site/pull/334#issuecomment-338551253 the
mergebot seems to not activate by "@asfgit merge". Jason, could you take a
look?

Re: Any plan to add support for graph computation

[Reuven Lax]

It's something that has been discussed, but I don't know that anyone is
currently working on it. One challenge is figuring out how an
iterative-processing API works with Beam's windowing model.

On Tue, Oct 24, 2017 at 11:43 PM, Ray  wrote:

> Hi, Dear Apache Beam Developer,
>
>
> I am new to Apache Beam and excited to find such wonderful tool to handle
> big-data across different popular runners.
> I am just wondering is there any plan to add support for graph computation
> like GraphX on Spark?
>
>
> * graph representation in PCollection
> * basic statistics like vertex/edge count
> * base function like vertex/edge-wise mapreduce task(i.e., count the
> outgoing degree of a vertex)
> * base function like subgraph combine/join
> * ..
>
>
> Thanks again for this great Apache Beam you brought to us,
> Looking forward to your reply.

Re: Java pre/post commit test suite breakage

[Kenneth Knowles]

Hi all,

I wanted to circle back on this. We've had continual quasi-outages, where
Jenkins occasionally schedules jobs but not reliably. Meanwhile, many of
our most important jobs continued to hit their timeouts when they did get
scheduled. On top of that, I've run the wrong command once before merging a
PR, resulting in a couple broken precommit runs.

Just so everyone knows the actions that have been taken and what we still
need to do:

 - Java postcommit and precommit timeouts bumped to 240 minutes (!!)
 - Seed job made independent of other jobs, so we can experiment safely
 - RAT config fixed

The latency of precommit is still prohibitive for effective use. Not only
are we waiting too long, but the number of workers needed to avoid backlog
is excessive. Here are some ideas for things we can do:

 - Proceed further on the pipeline job, which is what we want to do long
term
 - Precommit: run only on demand, so workers are not congested by automatic
builds; we could also pick and choose what we want to run
 - Consider non-Maven build orchestrators that can do dependency-driven
builds
 - Precommit: run fewer tests; I think this would mean leaving out some
modules that we are OK finding issues in the postcommit, or making them on
demand only
 - ValidatesRunner tests: Instead of running a single mavenJob, run
multiple, once which just installs while skipping everything else, then
follow it with just running the tests we care about
 - Examples integration tests: separate from the maven precommit; also run
as sequenced invocations

These are just some ideas; I honestly don't know. I think my upper limit
for default precommit feedback is probably 30 minutes, and even there I am
not very happy.

Any other suggestions?

Kenn

On Mon, Oct 23, 2017 at 3:29 PM, Kenneth Knowles  wrote:

> I want to wait and get some green from Jenkins running against the HEAD
> groovy scripts to confirm. I haven't sat at my desk long enough to see a
> full `mvn -P release clean verify` yet.
>
> On Mon, Oct 23, 2017 at 3:27 PM, Kenneth Knowles  wrote:
>
>> I was easily able to reproduce all the sorts of failures (and some more!)
>>
>> Here are some things that now work that didn't work, or didn't work
>> correctly, before
>>
>>  - mvn apache-rat:check
>>  - mvn -P release apache-rat:check
>>  - mvn -P release -f somewhere/else/pom.xml apache-rat:check
>>
>> It turns out we had these issues:
>>
>>  - items in .gitignore that our RAT config did not ignore
>>  - sub-modules actually *did* inherit the RAT config, depending on the
>> command you run
>>  - paths in the RAT exclude were relative to current dir
>>  - paths in our RAT exclude that aren't even part of our codebase or
>> generated targets, but just things that Jenkins dropped in the workspace,
>> and we were cloning directly into the workspace
>>
>> I think with the new Jenkins config and the new pom.xml things should
>> work well.
>>
>> Incidentally, Valentyn, changes to Jenkins job DSL groovy scripts do not
>> take effect from being merged but only when the seed job next runs. (TODO:
>> fix this). I am still trying to get the relevant Jenkins UI pages to load
>> to get your change incorporated into the live scripts, which were regressed
>> by being run from an old PR (TODO: fix the fact that this can happen).
>>
>> Kenn
>>
>> On Mon, Oct 23, 2017 at 2:21 PM, Valentyn Tymofieiev <
>> valen...@google.com.invalid> wrote:
>>
>>> Thanks a lot!
>>>
>>> Kenn, Were you able to reproduce RAT failures and test the fix locally? I
>>> think "mvn clean verify -Prelease" still passes for me.
>>>
>>> Timeout for the test suite has been increased in
>>> https://github.com/apache/
>>> beam/pull/4028 .
>>>
>>> On Mon, Oct 23, 2017 at 2:10 PM, Kenneth Knowles >> >
>>> wrote:
>>>
>>> > Wrong link - https://github.com/apache/beam/pull/4027
>>> >
>>> > On Mon, Oct 23, 2017 at 2:10 PM, Kenneth Knowles 
>>> wrote:
>>> >
>>> > > Yea, root cause is the config bug I described. Proposed fix at
>>> > > https://github.com/apache/beam/pull/4019/files. I'm working with
>>> infra
>>> > to
>>> > > sort out other build issues that are probably not related.
>>> > >
>>> > > On Mon, Oct 23, 2017 at 1:53 PM, Lukasz Cwik
>>> 
>>> > > wrote:
>>> > >
>>> > >> The build breakage I outlined is being tracked in
>>> > >> https://issues.apache.org/jira/browse/BEAM-3092
>>> > >>
>>> > >> On Mon, Oct 23, 2017 at 11:54 AM, Lukasz Cwik 
>>> wrote:
>>> > >>
>>> > >> > Another breaking change was caused by https://github.com/apache/
>>> > >> > beam/commit/241d3cedd5a8fd3f360b8ec2f3a8ef5001cbca98 because it
>>> > changed
>>> > >> > the build layout on the Jenkins server and our RAT rules expected
>>> to
>>> > >> apply
>>> > >> > from a root directory. I pinged Kenneth Knowles about it and he
>>> said
>>> > he
>>> > >> was
>>> > >> > taking a look.
>>> > >> >
>>> > >> > On Mon, Oct 23, 2017 at 11:53 AM, Raghu Angadi
>>> > >> >> > >> > > wrote:
>>> > >> >
>>> > >> >> Regd (1) :
>>> > >> >>

Re: UnconsumedReads

[Griselda Cuevas]

Hi Chaim - could you elaborate a little more on how you're using or want to
use the UnconsumedReads so we can help?



Gris Cuevas Zambrano

g...@google.com

Open Source Strategy

345 Spear Street, San Francisco, 94105



On 17 October 2017 at 06:03, Chaim Turkel  wrote:

> what is the purpose of this phase?
>
> chaim
>

Re: [Proposal] Add performance tests for commonly used file-based I/O PTransforms

[Chamikara Jayalath]

Thanks Łukasz and the team for the proposal. I think fixing this JIRA will
allow us to keep track of the performance of widely used
source/sink/runner/file-system combinations of Beam SDK. As Łukasz
mentioned, all comments are welcome.

Thanks,
Cham

On Wed, Oct 25, 2017 at 8:08 AM Łukasz Gajowy 
wrote:

> Hello Beam Community!
>
>
> During the last year many of Beam developers has put much effort in
> developing and discussing means of testing beam transforms. We would like
> to benefit from that and implement performance tests for file-based I/O
> Transforms.
>
>
> This proposal is strictly related to BEAM-3060 issue. Here’s the link to
> the doc:
>
> https://docs.google.com/document/d/1dA-5s6OHiP_cz-NRAbwapoKF5MEC1wKps4A5tFbIPKE/edit
>
>
> All comments are deeply appreciated.
>
>
> Thanks!
>
> ŁG
>

Re: [Proposal] Apache Beam Swag Store

[Griselda Cuevas]

Ok, I just discovered I can't attach a file :s

Here is the link:
https://drive.google.com/file/d/0B-QOa4cFPVznZWlsVFgxMXVwNnM/
view?usp=sharing



On 25 October 2017 at 08:57, Lukasz Cwik  wrote:

> I don't see any attachments to the e-mail.
>
> On Wed, Oct 25, 2017 at 8:51 AM, Griselda Cuevas 
> wrote:
>
> > Hi Everyone,
> >
> > I'd like to propose the creation of an online swag store for Apache Beam
> > where anyone could order swag from a wide selection and get it deliver to
> > their home or office. I got in touch with a provider who could include
> this
> > service as part of an order I'm placing to send swag for the Meetups
> we're
> > organizing this year. What do you think?
> >
> > I'd also like to get your feedback on the swag I'm requesting (you can
> see
> > it in the pdf I attached), what do you think of the colors, design, etc.?
> >
> > Lastly, I'll be ordering swag for Meetup organized this year so if you're
> > hosting one or speaking at one get in touch with me to send some!
> >
> > Cheers,
> > G
> >
>

Re: [Proposal] Apache Beam Swag Store

[Lukasz Cwik]

I don't see any attachments to the e-mail.

On Wed, Oct 25, 2017 at 8:51 AM, Griselda Cuevas 
wrote:

> Hi Everyone,
>
> I'd like to propose the creation of an online swag store for Apache Beam
> where anyone could order swag from a wide selection and get it deliver to
> their home or office. I got in touch with a provider who could include this
> service as part of an order I'm placing to send swag for the Meetups we're
> organizing this year. What do you think?
>
> I'd also like to get your feedback on the swag I'm requesting (you can see
> it in the pdf I attached), what do you think of the colors, design, etc.?
>
> Lastly, I'll be ordering swag for Meetup organized this year so if you're
> hosting one or speaking at one get in touch with me to send some!
>
> Cheers,
> G
>

[Proposal] Apache Beam Swag Store

[Griselda Cuevas]

Hi Everyone,

I'd like to propose the creation of an online swag store for Apache Beam
where anyone could order swag from a wide selection and get it deliver to
their home or office. I got in touch with a provider who could include this
service as part of an order I'm placing to send swag for the Meetups we're
organizing this year. What do you think?

I'd also like to get your feedback on the swag I'm requesting (you can see
it in the pdf I attached), what do you think of the colors, design, etc.?

Lastly, I'll be ordering swag for Meetup organized this year so if you're
hosting one or speaking at one get in touch with me to send some!

Cheers,
G

[Proposal] Add performance tests for commonly used file-based I/O PTransforms

[Łukasz Gajowy]

Hello Beam Community!


During the last year many of Beam developers has put much effort in
developing and discussing means of testing beam transforms. We would like
to benefit from that and implement performance tests for file-based I/O
Transforms.


This proposal is strictly related to BEAM-3060 issue. Here’s the link to
the doc:
https://docs.google.com/document/d/1dA-5s6OHiP_cz-NRAbwapoKF5MEC1wKps4A5tFbIPKE/edit


All comments are deeply appreciated.


Thanks!

ŁG