Towards a spec for robust streaming SQL, Part 2

2017-07-24 Thread Tyler Akidau 
Hello Flink, Calcite, and Beam dev lists!

Linked below is the second document I promised way back in April regarding
a collaborative spec for streaming SQL in Beam/Calcite/Flink (& apologies
for the delay; I thought I was nearly done a while back and then temporal
joins expanded to something much larger than expected).

To repeat what it says in the doc, my hope is that it can serve various
purposes over it's lifetime:

   -
   - A discussion ground for ironing out any remaining features necessary
   for supporting robust streaming semantics in Calcite SQL.

   - A rough, high-level source of truth for tracking efforts underway in
   support of this, currently spanning the Calcite, Flink, and Beam projects.

   - A written specification of the changes that were made, for the sake of
   understanding the delta after the fact.

The first and third points are, IMO, the most important. AFAIK, there are a
few features missing still that need to be defined (e.g., triggers
equivalents via EMIT, robust temporal join support). I'm also proposing a
clear distinction of streams and tables, which I think is important, but
which I believe is not the approach most folks have been taking in this
area. Sorting out these open issues and then having a concise record of the
solutions adopted will be important for providing a solid streaming
experience and teaching folks how to use it.

At any rate, I would much appreciate it if anyone with an interest in this
stuff could please take a look and add comments/suggestions/references to
related work in flight/etc as appropriate. For now please use
comments/suggestions, but if you really want to dive in with edit access,
let me know.

The doc: http://s.apache.org/streaming-sql-spec

-Tyler

Re: [S]FTP support as Pipeline I/O

2017-07-24 Thread Jean-Baptiste Onofré 
In Camel, we have different mode: with local file caching or using streaming 
when possible (it depends of the body in the Exchange).


So, I think we can do the same in Beam.

Regards
JB

On 07/24/2017 09:38 PM, Eugene Kirpichov wrote:

I think Camille may have referred to python standard library class StringIO
which means collecting the output into a string - and then I suppose
uploading the string to FTP. That could work (similar stuff exists in Java
library) but would limit us to files whose content fits in memory.

On Mon, Jul 24, 2017, 12:31 PM Jean-Baptiste Onofré  wrote:


I guess TextIO ? ;)

Regards
JB

On Jul 24, 2017, 21:27, at 21:27, Eugene Kirpichov
 wrote:

What is StringIO?

On Mon, Jul 24, 2017 at 1:47 AM Tolsa, Camille

wrote:


Not necessary with StringIO

On 24 July 2017 at 09:47, Reuven Lax 

wrote:



This would require writing data to local files in order to upload

it to

the

remote FTP, right?

On Mon, Jul 24, 2017 at 12:31 AM, Jean-Baptiste Onofré



wrote:


Hi Lucas,

IMHO, it's not a IO, it's a filesystem that TextIO and others can

support

(like GFS or HDFS).

It's what we did in Camel: the ftp component is just an extend of

file

component.

It means that we would be able to do:

pipeline.apply(TextIO.from("ftp://...;)).

Thoughts ?

If agree, I would be happy to work on this (with any help ;)).

Regards
JB


On 07/23/2017 07:39 AM, Lucas Arruda wrote:


Hi Beam folks,

I would like to suggest the creation of a Pipeline I/O to

support

FTP/SFTP

as both source and sink locations for data processing. I've done

some

research and it looks like there isn't any kind of development

ongoing

for

this (at least not on Jira).

I'd like to know your thoughts and if someone would like to

help/support

this initiative. In case someone has started a thing already

please

let

me

know ;)

Thank you,



--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com





--








This e-mail transmission (message and any attached files) may contain
information that is proprietary, privileged and/or confidential to

Veolia

Environnement and/or its affiliates and is intended exclusively for

the

person(s) to whom it is addressed. If you are not the intended

recipient,

please notify the sender by return e-mail and delete all copies of

this

e-mail, including all attachments. Unless expressly authorized, any

use,

disclosure, publication, retransmission or dissemination of this

e-mail

and/or of its attachments is strictly prohibited.

Ce message electronique et ses fichiers attaches sont strictement
confidentiels et peuvent contenir des elements dont Veolia

Environnement

et/ou l'une de ses entites affiliees sont proprietaires. Ils sont

donc

destines a l'usage de leurs seuls destinataires. Si vous avez recu ce
message par erreur, merci de le retourner a son emetteur et de le

detruire

ainsi que toutes les pieces attachees. L'utilisation, la divulgation,

la

publication, la distribution, ou la reproduction non expressement
autorisees de ce message et de ses pieces attachees sont interdites.














--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: [S]FTP support as Pipeline I/O

2017-07-24 Thread Eugene Kirpichov 
I think Camille may have referred to python standard library class StringIO
which means collecting the output into a string - and then I suppose
uploading the string to FTP. That could work (similar stuff exists in Java
library) but would limit us to files whose content fits in memory.

On Mon, Jul 24, 2017, 12:31 PM Jean-Baptiste Onofré  wrote:

> I guess TextIO ? ;)
>
> Regards
> JB
>
> On Jul 24, 2017, 21:27, at 21:27, Eugene Kirpichov
>  wrote:
> >What is StringIO?
> >
> >On Mon, Jul 24, 2017 at 1:47 AM Tolsa, Camille
> >
> >wrote:
> >
> >> Not necessary with StringIO
> >>
> >> On 24 July 2017 at 09:47, Reuven Lax 
> >wrote:
> >>
> >> > This would require writing data to local files in order to upload
> >it to
> >> the
> >> > remote FTP, right?
> >> >
> >> > On Mon, Jul 24, 2017 at 12:31 AM, Jean-Baptiste Onofré
> >
> >> > wrote:
> >> >
> >> > > Hi Lucas,
> >> > >
> >> > > IMHO, it's not a IO, it's a filesystem that TextIO and others can
> >> support
> >> > > (like GFS or HDFS).
> >> > >
> >> > > It's what we did in Camel: the ftp component is just an extend of
> >file
> >> > > component.
> >> > >
> >> > > It means that we would be able to do:
> >> > >
> >> > > pipeline.apply(TextIO.from("ftp://...;)).
> >> > >
> >> > > Thoughts ?
> >> > >
> >> > > If agree, I would be happy to work on this (with any help ;)).
> >> > >
> >> > > Regards
> >> > > JB
> >> > >
> >> > >
> >> > > On 07/23/2017 07:39 AM, Lucas Arruda wrote:
> >> > >
> >> > >> Hi Beam folks,
> >> > >>
> >> > >> I would like to suggest the creation of a Pipeline I/O to
> >support
> >> > FTP/SFTP
> >> > >> as both source and sink locations for data processing. I've done
> >some
> >> > >> research and it looks like there isn't any kind of development
> >ongoing
> >> > for
> >> > >> this (at least not on Jira).
> >> > >>
> >> > >> I'd like to know your thoughts and if someone would like to
> >> help/support
> >> > >> this initiative. In case someone has started a thing already
> >please
> >> let
> >> > me
> >> > >> know ;)
> >> > >>
> >> > >> Thank you,
> >> > >>
> >> > >>
> >> > > --
> >> > > Jean-Baptiste Onofré
> >> > > jbono...@apache.org
> >> > > http://blog.nanthrax.net
> >> > > Talend - http://www.talend.com
> >> > >
> >> >
> >>
> >> --
> >>
> >>
> >>
>
> >
> >> This e-mail transmission (message and any attached files) may contain
> >> information that is proprietary, privileged and/or confidential to
> >Veolia
> >> Environnement and/or its affiliates and is intended exclusively for
> >the
> >> person(s) to whom it is addressed. If you are not the intended
> >recipient,
> >> please notify the sender by return e-mail and delete all copies of
> >this
> >> e-mail, including all attachments. Unless expressly authorized, any
> >use,
> >> disclosure, publication, retransmission or dissemination of this
> >e-mail
> >> and/or of its attachments is strictly prohibited.
> >>
> >> Ce message electronique et ses fichiers attaches sont strictement
> >> confidentiels et peuvent contenir des elements dont Veolia
> >Environnement
> >> et/ou l'une de ses entites affiliees sont proprietaires. Ils sont
> >donc
> >> destines a l'usage de leurs seuls destinataires. Si vous avez recu ce
> >> message par erreur, merci de le retourner a son emetteur et de le
> >detruire
> >> ainsi que toutes les pieces attachees. L'utilisation, la divulgation,
> >la
> >> publication, la distribution, ou la reproduction non expressement
> >> autorisees de ce message et de ses pieces attachees sont interdites.
> >>
> >>
>
> >
> >>
>

Re: [S]FTP support as Pipeline I/O

2017-07-24 Thread Jean-Baptiste Onofré 
I guess TextIO ? ;)

Regards
JB

On Jul 24, 2017, 21:27, at 21:27, Eugene Kirpichov 
 wrote:
>What is StringIO?
>
>On Mon, Jul 24, 2017 at 1:47 AM Tolsa, Camille
>
>wrote:
>
>> Not necessary with StringIO
>>
>> On 24 July 2017 at 09:47, Reuven Lax 
>wrote:
>>
>> > This would require writing data to local files in order to upload
>it to
>> the
>> > remote FTP, right?
>> >
>> > On Mon, Jul 24, 2017 at 12:31 AM, Jean-Baptiste Onofré
>
>> > wrote:
>> >
>> > > Hi Lucas,
>> > >
>> > > IMHO, it's not a IO, it's a filesystem that TextIO and others can
>> support
>> > > (like GFS or HDFS).
>> > >
>> > > It's what we did in Camel: the ftp component is just an extend of
>file
>> > > component.
>> > >
>> > > It means that we would be able to do:
>> > >
>> > > pipeline.apply(TextIO.from("ftp://...;)).
>> > >
>> > > Thoughts ?
>> > >
>> > > If agree, I would be happy to work on this (with any help ;)).
>> > >
>> > > Regards
>> > > JB
>> > >
>> > >
>> > > On 07/23/2017 07:39 AM, Lucas Arruda wrote:
>> > >
>> > >> Hi Beam folks,
>> > >>
>> > >> I would like to suggest the creation of a Pipeline I/O to
>support
>> > FTP/SFTP
>> > >> as both source and sink locations for data processing. I've done
>some
>> > >> research and it looks like there isn't any kind of development
>ongoing
>> > for
>> > >> this (at least not on Jira).
>> > >>
>> > >> I'd like to know your thoughts and if someone would like to
>> help/support
>> > >> this initiative. In case someone has started a thing already
>please
>> let
>> > me
>> > >> know ;)
>> > >>
>> > >> Thank you,
>> > >>
>> > >>
>> > > --
>> > > Jean-Baptiste Onofré
>> > > jbono...@apache.org
>> > > http://blog.nanthrax.net
>> > > Talend - http://www.talend.com
>> > >
>> >
>>
>> --
>>
>>
>>
>
>> This e-mail transmission (message and any attached files) may contain
>> information that is proprietary, privileged and/or confidential to
>Veolia
>> Environnement and/or its affiliates and is intended exclusively for
>the
>> person(s) to whom it is addressed. If you are not the intended
>recipient,
>> please notify the sender by return e-mail and delete all copies of
>this
>> e-mail, including all attachments. Unless expressly authorized, any
>use,
>> disclosure, publication, retransmission or dissemination of this
>e-mail
>> and/or of its attachments is strictly prohibited.
>>
>> Ce message electronique et ses fichiers attaches sont strictement
>> confidentiels et peuvent contenir des elements dont Veolia
>Environnement
>> et/ou l'une de ses entites affiliees sont proprietaires. Ils sont
>donc
>> destines a l'usage de leurs seuls destinataires. Si vous avez recu ce
>> message par erreur, merci de le retourner a son emetteur et de le
>detruire
>> ainsi que toutes les pieces attachees. L'utilisation, la divulgation,
>la
>> publication, la distribution, ou la reproduction non expressement
>> autorisees de ce message et de ses pieces attachees sont interdites.
>>
>>
>
>>

Re: [S]FTP support as Pipeline I/O

2017-07-24 Thread Eugene Kirpichov 
What is StringIO?

On Mon, Jul 24, 2017 at 1:47 AM Tolsa, Camille 
wrote:

> Not necessary with StringIO
>
> On 24 July 2017 at 09:47, Reuven Lax  wrote:
>
> > This would require writing data to local files in order to upload it to
> the
> > remote FTP, right?
> >
> > On Mon, Jul 24, 2017 at 12:31 AM, Jean-Baptiste Onofré 
> > wrote:
> >
> > > Hi Lucas,
> > >
> > > IMHO, it's not a IO, it's a filesystem that TextIO and others can
> support
> > > (like GFS or HDFS).
> > >
> > > It's what we did in Camel: the ftp component is just an extend of file
> > > component.
> > >
> > > It means that we would be able to do:
> > >
> > > pipeline.apply(TextIO.from("ftp://...;)).
> > >
> > > Thoughts ?
> > >
> > > If agree, I would be happy to work on this (with any help ;)).
> > >
> > > Regards
> > > JB
> > >
> > >
> > > On 07/23/2017 07:39 AM, Lucas Arruda wrote:
> > >
> > >> Hi Beam folks,
> > >>
> > >> I would like to suggest the creation of a Pipeline I/O to support
> > FTP/SFTP
> > >> as both source and sink locations for data processing. I've done some
> > >> research and it looks like there isn't any kind of development ongoing
> > for
> > >> this (at least not on Jira).
> > >>
> > >> I'd like to know your thoughts and if someone would like to
> help/support
> > >> this initiative. In case someone has started a thing already please
> let
> > me
> > >> know ;)
> > >>
> > >> Thank you,
> > >>
> > >>
> > > --
> > > Jean-Baptiste Onofré
> > > jbono...@apache.org
> > > http://blog.nanthrax.net
> > > Talend - http://www.talend.com
> > >
> >
>
> --
>
>
> 
> This e-mail transmission (message and any attached files) may contain
> information that is proprietary, privileged and/or confidential to Veolia
> Environnement and/or its affiliates and is intended exclusively for the
> person(s) to whom it is addressed. If you are not the intended recipient,
> please notify the sender by return e-mail and delete all copies of this
> e-mail, including all attachments. Unless expressly authorized, any use,
> disclosure, publication, retransmission or dissemination of this e-mail
> and/or of its attachments is strictly prohibited.
>
> Ce message electronique et ses fichiers attaches sont strictement
> confidentiels et peuvent contenir des elements dont Veolia Environnement
> et/ou l'une de ses entites affiliees sont proprietaires. Ils sont donc
> destines a l'usage de leurs seuls destinataires. Si vous avez recu ce
> message par erreur, merci de le retourner a son emetteur et de le detruire
> ainsi que toutes les pieces attachees. L'utilisation, la divulgation, la
> publication, la distribution, ou la reproduction non expressement
> autorisees de ce message et de ses pieces attachees sont interdites.
>
> 
>

Custom window merging

2017-07-24 Thread Etienne Chauchot 

Hi all,

There is now 2 new ValidatesRunner tests: 
WindowTest.testMergingCustomWindows and 
WindowTest.testMergingCustomWindowsKeyedCollection. The aim of these 
tests is to verify that the runners can handle custom windowFn 
(extensions of windowFn that, for example, could rely on elements in 
addition to timestamps).


As new runners are coming, I wanted to let you know that there is also a 
new category tag UsesCustomWindowMerging that you can use to skip these 
tests while running ValidatesRunner tests on runners that do not support 
custom window merging yet.


Besides, there is also an ongoing related PR 
(https://github.com/apache/beam/pull/3592) to enhance the test utils 
methods of WindowFnTestUtils.


Etienne

Re: [CANCEL][VOTE] Release 2.1.0, release candidate #2

2017-07-24 Thread Sourabh Bajaj 
I created PR/3627 for cherry picking a fix for BEAM-2636.

On Mon, Jul 24, 2017 at 8:20 AM Ismaël Mejía  wrote:

> Not a blocker but maybe it is worth considering the fix for
> https://issues.apache.org/jira/browse/BEAM-2587 too.
>
> I also was bitten by this issue and I could only get it to work by
> doing a 'pip install --user grpcio-tools' (not sure if this is a
> proper solution but it works for me), however when I validated the
> python only source code it worked out of the box without issue.
>
> On Mon, Jul 24, 2017 at 2:37 PM, Jean-Baptiste Onofré 
> wrote:
> > Awesome !
> >
> > Thanks Aljoscha
> >
> > Regards
> > JB
> >
> >
> > On 07/24/2017 02:32 PM, Aljoscha Krettek wrote:
> >>
> >> I opened a PR against the release-2.1.0 branch:
> >> https://github.com/apache/beam/pull/3625
> >> 
> >>
> >> This should not fail any tests since it was recently reviewed and merged
> >> for the master.
> >>
> >> Best,
> >> Aljoscha
> >>
> >>> On 24. Jul 2017, at 14:09, Jean-Baptiste Onofré 
> wrote:
> >>>
> >>> +1
> >>>
> >>> Definitely good to have it for RC3.
> >>>
> >>> Regards
> >>> JB
> >>>
> >>> On 07/24/2017 02:05 PM, Aljoscha Krettek wrote:
> 
>  When we're cutting a new RC anyways we could also include the fixes
> for
>  https://issues.apache.org/jira/browse/BEAM-2571
>  . It's an actual
> bug in the
>  Flink Runner and the fix for that is a set of three fixes that should
> be
>  easy to cherry-pick on top of the release branch.
>  If we agree I could open a PR for that.
>  Best,
>  Aljoscha
> >
> > On 24. Jul 2017, at 13:47, Aviem Zur  wrote:
> >
> > We also have two tests failing in Spark runner as detailed by the
> > following
> > two tickets:
> > https://issues.apache.org/jira/browse/BEAM-2670
> > https://issues.apache.org/jira/browse/BEAM-2671
> >
> > On Mon, Jul 24, 2017 at 11:44 AM Jean-Baptiste Onofré <
> j...@nanthrax.net>
> > wrote:
> >
> >> Hi all,
> >>
> >> due to https://issues.apache.org/jira/browse/BEAM-2662, I cancel
> this
> >> vote.
> >>
> >> We also have a build issue with the Spark runner that I would like
> to
> >> fix
> >> for RC3:
> >>
> >>
> >>
> >>
> https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_ValidatesRunner_Spark/2446/
> >>
> >> So, we are going to work on the Spark runner test fix for RC3
> >> (BEAM-2662 is
> >> already fixed on release-2.1.0 branch).
> >>
> >> I will submit RC3 to vote as soon as Spark runner tests are fully
> OK.
> >>
> >> Regards
> >> JB
> >>
> >> On 07/18/2017 06:30 PM, Jean-Baptiste Onofré wrote:
> >>>
> >>> Hi everyone,
> >>>
> >>> Please review and vote on the release candidate #2 for the version
> >>
> >> 2.1.0, as
> >>>
> >>> follows:
> >>>
> >>> [ ] +1, Approve the release
> >>> [ ] -1, Do not approve the release (please provide specific
> comments)
> >>>
> >>>
> >>> The complete staging area is available for your review, which
> >>> includes:
> >>> * JIRA release notes [1],
> >>> * the official Apache source release to be deployed to
> >>> dist.apache.org
> >>
> >> [2],
> >>>
> >>> which is signed with the key with fingerprint C8282E76 [3],
> >>> * all artifacts to be deployed to the Maven Central Repository [4],
> >>> * source code tag "v2.1.0-RC2" [5],
> >>> * website pull request listing the release and publishing the API
> >>
> >> reference
> >>>
> >>> manual [6].
> >>> * Python artifacts are deployed along with the source release to
> the
> >>> dist.apache.org [2].
> >>>
> >>> The vote will be open for at least 72 hours. It is adopted by
> >>> majority
> >>
> >> approval,
> >>>
> >>> with at least 3 PMC affirmative votes.
> >>>
> >>> Thanks,
> >>> JB
> >>>
> >>> [1]
> >>>
> >>
> >>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12340528
> >>>
> >>>
> >>> [2] https://dist.apache.org/repos/dist/dev/beam/2.1.0/
> >>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> >>> [4]
> >>
> >>
> https://repository.apache.org/content/repositories/orgapachebeam-1019/
> >>>
> >>> [5] https://github.com/apache/beam/tree/v2.1.0-RC2
> >>> [6] https://github.com/apache/beam-site/pull/270
> >>
> >>
> >> --
> >> Jean-Baptiste Onofré
> >> jbono...@apache.org
> >> http://blog.nanthrax.net
> >> Talend - http://www.talend.com
> >>
> >>>
> >>> --
> >>> Jean-Baptiste Onofré
> >>> jbono...@apache.org
> >>> http://blog.nanthrax.net
> >>> Talend - http://www.talend.com
> >>
> >>
> >>
> >
> > --
> > Jean-Baptiste Onofré
> >

Re: [CANCEL][VOTE] Release 2.1.0, release candidate #2

2017-07-24 Thread Ismaël Mejía 
Not a blocker but maybe it is worth considering the fix for
https://issues.apache.org/jira/browse/BEAM-2587 too.

I also was bitten by this issue and I could only get it to work by
doing a 'pip install --user grpcio-tools' (not sure if this is a
proper solution but it works for me), however when I validated the
python only source code it worked out of the box without issue.

On Mon, Jul 24, 2017 at 2:37 PM, Jean-Baptiste Onofré  wrote:
> Awesome !
>
> Thanks Aljoscha
>
> Regards
> JB
>
>
> On 07/24/2017 02:32 PM, Aljoscha Krettek wrote:
>>
>> I opened a PR against the release-2.1.0 branch:
>> https://github.com/apache/beam/pull/3625
>> 
>>
>> This should not fail any tests since it was recently reviewed and merged
>> for the master.
>>
>> Best,
>> Aljoscha
>>
>>> On 24. Jul 2017, at 14:09, Jean-Baptiste Onofré  wrote:
>>>
>>> +1
>>>
>>> Definitely good to have it for RC3.
>>>
>>> Regards
>>> JB
>>>
>>> On 07/24/2017 02:05 PM, Aljoscha Krettek wrote:

 When we're cutting a new RC anyways we could also include the fixes for
 https://issues.apache.org/jira/browse/BEAM-2571
 . It's an actual bug in 
 the
 Flink Runner and the fix for that is a set of three fixes that should be
 easy to cherry-pick on top of the release branch.
 If we agree I could open a PR for that.
 Best,
 Aljoscha
>
> On 24. Jul 2017, at 13:47, Aviem Zur  wrote:
>
> We also have two tests failing in Spark runner as detailed by the
> following
> two tickets:
> https://issues.apache.org/jira/browse/BEAM-2670
> https://issues.apache.org/jira/browse/BEAM-2671
>
> On Mon, Jul 24, 2017 at 11:44 AM Jean-Baptiste Onofré 
> wrote:
>
>> Hi all,
>>
>> due to https://issues.apache.org/jira/browse/BEAM-2662, I cancel this
>> vote.
>>
>> We also have a build issue with the Spark runner that I would like to
>> fix
>> for RC3:
>>
>>
>>
>> https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_ValidatesRunner_Spark/2446/
>>
>> So, we are going to work on the Spark runner test fix for RC3
>> (BEAM-2662 is
>> already fixed on release-2.1.0 branch).
>>
>> I will submit RC3 to vote as soon as Spark runner tests are fully OK.
>>
>> Regards
>> JB
>>
>> On 07/18/2017 06:30 PM, Jean-Baptiste Onofré wrote:
>>>
>>> Hi everyone,
>>>
>>> Please review and vote on the release candidate #2 for the version
>>
>> 2.1.0, as
>>>
>>> follows:
>>>
>>> [ ] +1, Approve the release
>>> [ ] -1, Do not approve the release (please provide specific comments)
>>>
>>>
>>> The complete staging area is available for your review, which
>>> includes:
>>> * JIRA release notes [1],
>>> * the official Apache source release to be deployed to
>>> dist.apache.org
>>
>> [2],
>>>
>>> which is signed with the key with fingerprint C8282E76 [3],
>>> * all artifacts to be deployed to the Maven Central Repository [4],
>>> * source code tag "v2.1.0-RC2" [5],
>>> * website pull request listing the release and publishing the API
>>
>> reference
>>>
>>> manual [6].
>>> * Python artifacts are deployed along with the source release to the
>>> dist.apache.org [2].
>>>
>>> The vote will be open for at least 72 hours. It is adopted by
>>> majority
>>
>> approval,
>>>
>>> with at least 3 PMC affirmative votes.
>>>
>>> Thanks,
>>> JB
>>>
>>> [1]
>>>
>>
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12340528
>>>
>>>
>>> [2] https://dist.apache.org/repos/dist/dev/beam/2.1.0/
>>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>>> [4]
>>
>> https://repository.apache.org/content/repositories/orgapachebeam-1019/
>>>
>>> [5] https://github.com/apache/beam/tree/v2.1.0-RC2
>>> [6] https://github.com/apache/beam-site/pull/270
>>
>>
>> --
>> Jean-Baptiste Onofré
>> jbono...@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>>>
>>> --
>>> Jean-Baptiste Onofré
>>> jbono...@apache.org
>>> http://blog.nanthrax.net
>>> Talend - http://www.talend.com
>>
>>
>>
>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com

Re: [CANCEL][VOTE] Release 2.1.0, release candidate #2

2017-07-24 Thread Aljoscha Krettek 
I opened a PR against the release-2.1.0 branch: 
https://github.com/apache/beam/pull/3625 


This should not fail any tests since it was recently reviewed and merged for 
the master.

Best,
Aljoscha

> On 24. Jul 2017, at 14:09, Jean-Baptiste Onofré  wrote:
> 
> +1
> 
> Definitely good to have it for RC3.
> 
> Regards
> JB
> 
> On 07/24/2017 02:05 PM, Aljoscha Krettek wrote:
>> When we're cutting a new RC anyways we could also include the fixes for 
>> https://issues.apache.org/jira/browse/BEAM-2571 
>> . It's an actual bug in the 
>> Flink Runner and the fix for that is a set of three fixes that should be 
>> easy to cherry-pick on top of the release branch.
>> If we agree I could open a PR for that.
>> Best,
>> Aljoscha
>>> On 24. Jul 2017, at 13:47, Aviem Zur  wrote:
>>> 
>>> We also have two tests failing in Spark runner as detailed by the following
>>> two tickets:
>>> https://issues.apache.org/jira/browse/BEAM-2670
>>> https://issues.apache.org/jira/browse/BEAM-2671
>>> 
>>> On Mon, Jul 24, 2017 at 11:44 AM Jean-Baptiste Onofré 
>>> wrote:
>>> 
 Hi all,
 
 due to https://issues.apache.org/jira/browse/BEAM-2662, I cancel this
 vote.
 
 We also have a build issue with the Spark runner that I would like to fix
 for RC3:
 
 
 https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_ValidatesRunner_Spark/2446/
 
 So, we are going to work on the Spark runner test fix for RC3 (BEAM-2662 is
 already fixed on release-2.1.0 branch).
 
 I will submit RC3 to vote as soon as Spark runner tests are fully OK.
 
 Regards
 JB
 
 On 07/18/2017 06:30 PM, Jean-Baptiste Onofré wrote:
> Hi everyone,
> 
> Please review and vote on the release candidate #2 for the version
 2.1.0, as
> follows:
> 
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
> 
> 
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release to be deployed to dist.apache.org
 [2],
> which is signed with the key with fingerprint C8282E76 [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag "v2.1.0-RC2" [5],
> * website pull request listing the release and publishing the API
 reference
> manual [6].
> * Python artifacts are deployed along with the source release to the
> dist.apache.org [2].
> 
> The vote will be open for at least 72 hours. It is adopted by majority
 approval,
> with at least 3 PMC affirmative votes.
> 
> Thanks,
> JB
> 
> [1]
> 
 https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12340528
> 
> [2] https://dist.apache.org/repos/dist/dev/beam/2.1.0/
> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> [4]
 https://repository.apache.org/content/repositories/orgapachebeam-1019/
> [5] https://github.com/apache/beam/tree/v2.1.0-RC2
> [6] https://github.com/apache/beam-site/pull/270
 
 --
 Jean-Baptiste Onofré
 jbono...@apache.org
 http://blog.nanthrax.net
 Talend - http://www.talend.com
 
> 
> -- 
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com

Re: [CANCEL][VOTE] Release 2.1.0, release candidate #2

2017-07-24 Thread Jean-Baptiste Onofré 

+1

Definitely good to have it for RC3.

Regards
JB

On 07/24/2017 02:05 PM, Aljoscha Krettek wrote:

When we're cutting a new RC anyways we could also include the fixes for 
https://issues.apache.org/jira/browse/BEAM-2571 
. It's an actual bug in the 
Flink Runner and the fix for that is a set of three fixes that should be easy to 
cherry-pick on top of the release branch.

If we agree I could open a PR for that.

Best,
Aljoscha


On 24. Jul 2017, at 13:47, Aviem Zur  wrote:

We also have two tests failing in Spark runner as detailed by the following
two tickets:
https://issues.apache.org/jira/browse/BEAM-2670
https://issues.apache.org/jira/browse/BEAM-2671

On Mon, Jul 24, 2017 at 11:44 AM Jean-Baptiste Onofré 
wrote:


Hi all,

due to https://issues.apache.org/jira/browse/BEAM-2662, I cancel this
vote.

We also have a build issue with the Spark runner that I would like to fix
for RC3:


https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_ValidatesRunner_Spark/2446/

So, we are going to work on the Spark runner test fix for RC3 (BEAM-2662 is
already fixed on release-2.1.0 branch).

I will submit RC3 to vote as soon as Spark runner tests are fully OK.

Regards
JB

On 07/18/2017 06:30 PM, Jean-Baptiste Onofré wrote:

Hi everyone,

Please review and vote on the release candidate #2 for the version

2.1.0, as

follows:

[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)


The complete staging area is available for your review, which includes:
* JIRA release notes [1],
* the official Apache source release to be deployed to dist.apache.org

[2],

which is signed with the key with fingerprint C8282E76 [3],
* all artifacts to be deployed to the Maven Central Repository [4],
* source code tag "v2.1.0-RC2" [5],
* website pull request listing the release and publishing the API

reference

manual [6].
* Python artifacts are deployed along with the source release to the
dist.apache.org [2].

The vote will be open for at least 72 hours. It is adopted by majority

approval,

with at least 3 PMC affirmative votes.

Thanks,
JB

[1]


https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12340528


[2] https://dist.apache.org/repos/dist/dev/beam/2.1.0/
[3] https://dist.apache.org/repos/dist/release/beam/KEYS
[4]

https://repository.apache.org/content/repositories/orgapachebeam-1019/

[5] https://github.com/apache/beam/tree/v2.1.0-RC2
[6] https://github.com/apache/beam-site/pull/270


--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com






--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: [CANCEL][VOTE] Release 2.1.0, release candidate #2

2017-07-24 Thread Aljoscha Krettek 
When we're cutting a new RC anyways we could also include the fixes for 
https://issues.apache.org/jira/browse/BEAM-2571 
. It's an actual bug in the 
Flink Runner and the fix for that is a set of three fixes that should be easy 
to cherry-pick on top of the release branch.

If we agree I could open a PR for that.

Best,
Aljoscha

> On 24. Jul 2017, at 13:47, Aviem Zur  wrote:
> 
> We also have two tests failing in Spark runner as detailed by the following
> two tickets:
> https://issues.apache.org/jira/browse/BEAM-2670
> https://issues.apache.org/jira/browse/BEAM-2671
> 
> On Mon, Jul 24, 2017 at 11:44 AM Jean-Baptiste Onofré 
> wrote:
> 
>> Hi all,
>> 
>> due to https://issues.apache.org/jira/browse/BEAM-2662, I cancel this
>> vote.
>> 
>> We also have a build issue with the Spark runner that I would like to fix
>> for RC3:
>> 
>> 
>> https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_ValidatesRunner_Spark/2446/
>> 
>> So, we are going to work on the Spark runner test fix for RC3 (BEAM-2662 is
>> already fixed on release-2.1.0 branch).
>> 
>> I will submit RC3 to vote as soon as Spark runner tests are fully OK.
>> 
>> Regards
>> JB
>> 
>> On 07/18/2017 06:30 PM, Jean-Baptiste Onofré wrote:
>>> Hi everyone,
>>> 
>>> Please review and vote on the release candidate #2 for the version
>> 2.1.0, as
>>> follows:
>>> 
>>> [ ] +1, Approve the release
>>> [ ] -1, Do not approve the release (please provide specific comments)
>>> 
>>> 
>>> The complete staging area is available for your review, which includes:
>>> * JIRA release notes [1],
>>> * the official Apache source release to be deployed to dist.apache.org
>> [2],
>>> which is signed with the key with fingerprint C8282E76 [3],
>>> * all artifacts to be deployed to the Maven Central Repository [4],
>>> * source code tag "v2.1.0-RC2" [5],
>>> * website pull request listing the release and publishing the API
>> reference
>>> manual [6].
>>> * Python artifacts are deployed along with the source release to the
>>> dist.apache.org [2].
>>> 
>>> The vote will be open for at least 72 hours. It is adopted by majority
>> approval,
>>> with at least 3 PMC affirmative votes.
>>> 
>>> Thanks,
>>> JB
>>> 
>>> [1]
>>> 
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12340528
>>> 
>>> [2] https://dist.apache.org/repos/dist/dev/beam/2.1.0/
>>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>>> [4]
>> https://repository.apache.org/content/repositories/orgapachebeam-1019/
>>> [5] https://github.com/apache/beam/tree/v2.1.0-RC2
>>> [6] https://github.com/apache/beam-site/pull/270
>> 
>> --
>> Jean-Baptiste Onofré
>> jbono...@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>

Re: [CANCEL][VOTE] Release 2.1.0, release candidate #2

2017-07-24 Thread Aviem Zur 
We also have two tests failing in Spark runner as detailed by the following
two tickets:
https://issues.apache.org/jira/browse/BEAM-2670
https://issues.apache.org/jira/browse/BEAM-2671

On Mon, Jul 24, 2017 at 11:44 AM Jean-Baptiste Onofré 
wrote:

> Hi all,
>
> due to https://issues.apache.org/jira/browse/BEAM-2662, I cancel this
> vote.
>
> We also have a build issue with the Spark runner that I would like to fix
> for RC3:
>
>
> https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_ValidatesRunner_Spark/2446/
>
> So, we are going to work on the Spark runner test fix for RC3 (BEAM-2662 is
> already fixed on release-2.1.0 branch).
>
> I will submit RC3 to vote as soon as Spark runner tests are fully OK.
>
> Regards
> JB
>
> On 07/18/2017 06:30 PM, Jean-Baptiste Onofré wrote:
> > Hi everyone,
> >
> > Please review and vote on the release candidate #2 for the version
> 2.1.0, as
> > follows:
> >
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> >
> > The complete staging area is available for your review, which includes:
> > * JIRA release notes [1],
> > * the official Apache source release to be deployed to dist.apache.org
> [2],
> > which is signed with the key with fingerprint C8282E76 [3],
> > * all artifacts to be deployed to the Maven Central Repository [4],
> > * source code tag "v2.1.0-RC2" [5],
> > * website pull request listing the release and publishing the API
> reference
> > manual [6].
> > * Python artifacts are deployed along with the source release to the
> > dist.apache.org [2].
> >
> > The vote will be open for at least 72 hours. It is adopted by majority
> approval,
> > with at least 3 PMC affirmative votes.
> >
> > Thanks,
> > JB
> >
> > [1]
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12340528
> >
> > [2] https://dist.apache.org/repos/dist/dev/beam/2.1.0/
> > [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> > [4]
> https://repository.apache.org/content/repositories/orgapachebeam-1019/
> > [5] https://github.com/apache/beam/tree/v2.1.0-RC2
> > [6] https://github.com/apache/beam-site/pull/270
>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>

Re: [S]FTP support as Pipeline I/O

2017-07-24 Thread Tolsa, Camille 
Not necessary with StringIO

On 24 July 2017 at 09:47, Reuven Lax  wrote:

> This would require writing data to local files in order to upload it to the
> remote FTP, right?
>
> On Mon, Jul 24, 2017 at 12:31 AM, Jean-Baptiste Onofré 
> wrote:
>
> > Hi Lucas,
> >
> > IMHO, it's not a IO, it's a filesystem that TextIO and others can support
> > (like GFS or HDFS).
> >
> > It's what we did in Camel: the ftp component is just an extend of file
> > component.
> >
> > It means that we would be able to do:
> >
> > pipeline.apply(TextIO.from("ftp://...;)).
> >
> > Thoughts ?
> >
> > If agree, I would be happy to work on this (with any help ;)).
> >
> > Regards
> > JB
> >
> >
> > On 07/23/2017 07:39 AM, Lucas Arruda wrote:
> >
> >> Hi Beam folks,
> >>
> >> I would like to suggest the creation of a Pipeline I/O to support
> FTP/SFTP
> >> as both source and sink locations for data processing. I've done some
> >> research and it looks like there isn't any kind of development ongoing
> for
> >> this (at least not on Jira).
> >>
> >> I'd like to know your thoughts and if someone would like to help/support
> >> this initiative. In case someone has started a thing already please let
> me
> >> know ;)
> >>
> >> Thank you,
> >>
> >>
> > --
> > Jean-Baptiste Onofré
> > jbono...@apache.org
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> >
>

-- 


This e-mail transmission (message and any attached files) may contain 
information that is proprietary, privileged and/or confidential to Veolia 
Environnement and/or its affiliates and is intended exclusively for the 
person(s) to whom it is addressed. If you are not the intended recipient, 
please notify the sender by return e-mail and delete all copies of this 
e-mail, including all attachments. Unless expressly authorized, any use, 
disclosure, publication, retransmission or dissemination of this e-mail 
and/or of its attachments is strictly prohibited. 

Ce message electronique et ses fichiers attaches sont strictement 
confidentiels et peuvent contenir des elements dont Veolia Environnement 
et/ou l'une de ses entites affiliees sont proprietaires. Ils sont donc 
destines a l'usage de leurs seuls destinataires. Si vous avez recu ce 
message par erreur, merci de le retourner a son emetteur et de le detruire 
ainsi que toutes les pieces attachees. L'utilisation, la divulgation, la 
publication, la distribution, ou la reproduction non expressement 
autorisees de ce message et de ses pieces attachees sont interdites.

[CANCEL][VOTE] Release 2.1.0, release candidate #2

2017-07-24 Thread Jean-Baptiste Onofré 

Hi all,

due to https://issues.apache.org/jira/browse/BEAM-2662, I cancel this vote.

We also have a build issue with the Spark runner that I would like to fix for 
RC3:

https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_ValidatesRunner_Spark/2446/

So, we are going to work on the Spark runner test fix for RC3 (BEAM-2662 is 
already fixed on release-2.1.0 branch).


I will submit RC3 to vote as soon as Spark runner tests are fully OK.

Regards
JB

On 07/18/2017 06:30 PM, Jean-Baptiste Onofré wrote:

Hi everyone,

Please review and vote on the release candidate #2 for the version 2.1.0, as 
follows:


[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)


The complete staging area is available for your review, which includes:
* JIRA release notes [1],
* the official Apache source release to be deployed to dist.apache.org [2], 
which is signed with the key with fingerprint C8282E76 [3],

* all artifacts to be deployed to the Maven Central Repository [4],
* source code tag "v2.1.0-RC2" [5],
* website pull request listing the release and publishing the API reference 
manual [6].
* Python artifacts are deployed along with the source release to the 
dist.apache.org [2].


The vote will be open for at least 72 hours. It is adopted by majority approval, 
with at least 3 PMC affirmative votes.


Thanks,
JB

[1] 
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12340528 


[2] https://dist.apache.org/repos/dist/dev/beam/2.1.0/
[3] https://dist.apache.org/repos/dist/release/beam/KEYS
[4] https://repository.apache.org/content/repositories/orgapachebeam-1019/
[5] https://github.com/apache/beam/tree/v2.1.0-RC2
[6] https://github.com/apache/beam-site/pull/270


--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: [S]FTP support as Pipeline I/O

2017-07-24 Thread Reuven Lax 
This would require writing data to local files in order to upload it to the
remote FTP, right?

On Mon, Jul 24, 2017 at 12:31 AM, Jean-Baptiste Onofré 
wrote:

> Hi Lucas,
>
> IMHO, it's not a IO, it's a filesystem that TextIO and others can support
> (like GFS or HDFS).
>
> It's what we did in Camel: the ftp component is just an extend of file
> component.
>
> It means that we would be able to do:
>
> pipeline.apply(TextIO.from("ftp://...;)).
>
> Thoughts ?
>
> If agree, I would be happy to work on this (with any help ;)).
>
> Regards
> JB
>
>
> On 07/23/2017 07:39 AM, Lucas Arruda wrote:
>
>> Hi Beam folks,
>>
>> I would like to suggest the creation of a Pipeline I/O to support FTP/SFTP
>> as both source and sink locations for data processing. I've done some
>> research and it looks like there isn't any kind of development ongoing for
>> this (at least not on Jira).
>>
>> I'd like to know your thoughts and if someone would like to help/support
>> this initiative. In case someone has started a thing already please let me
>> know ;)
>>
>> Thank you,
>>
>>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>

Re: [S]FTP support as Pipeline I/O

2017-07-24 Thread Tolsa, Camille 
Hello,

I would definitively appreciate this feature.
If i can help somehow tell me

Camille.

On 24 July 2017 at 09:31, Jean-Baptiste Onofré  wrote:

> Hi Lucas,
>
> IMHO, it's not a IO, it's a filesystem that TextIO and others can support
> (like GFS or HDFS).
>
> It's what we did in Camel: the ftp component is just an extend of file
> component.
>
> It means that we would be able to do:
>
> pipeline.apply(TextIO.from("ftp://...;)).
>
> Thoughts ?
>
> If agree, I would be happy to work on this (with any help ;)).
>
> Regards
> JB
>
>
> On 07/23/2017 07:39 AM, Lucas Arruda wrote:
>
>> Hi Beam folks,
>>
>> I would like to suggest the creation of a Pipeline I/O to support FTP/SFTP
>> as both source and sink locations for data processing. I've done some
>> research and it looks like there isn't any kind of development ongoing for
>> this (at least not on Jira).
>>
>> I'd like to know your thoughts and if someone would like to help/support
>> this initiative. In case someone has started a thing already please let me
>> know ;)
>>
>> Thank you,
>>
>>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>

-- 


This e-mail transmission (message and any attached files) may contain 
information that is proprietary, privileged and/or confidential to Veolia 
Environnement and/or its affiliates and is intended exclusively for the 
person(s) to whom it is addressed. If you are not the intended recipient, 
please notify the sender by return e-mail and delete all copies of this 
e-mail, including all attachments. Unless expressly authorized, any use, 
disclosure, publication, retransmission or dissemination of this e-mail 
and/or of its attachments is strictly prohibited. 

Ce message electronique et ses fichiers attaches sont strictement 
confidentiels et peuvent contenir des elements dont Veolia Environnement 
et/ou l'une de ses entites affiliees sont proprietaires. Ils sont donc 
destines a l'usage de leurs seuls destinataires. Si vous avez recu ce 
message par erreur, merci de le retourner a son emetteur et de le detruire 
ainsi que toutes les pieces attachees. L'utilisation, la divulgation, la 
publication, la distribution, ou la reproduction non expressement 
autorisees de ce message et de ses pieces attachees sont interdites.

Re: [S]FTP support as Pipeline I/O

2017-07-24 Thread Jean-Baptiste Onofré 

Hi Lucas,

IMHO, it's not a IO, it's a filesystem that TextIO and others can support (like 
GFS or HDFS).


It's what we did in Camel: the ftp component is just an extend of file 
component.

It means that we would be able to do:

pipeline.apply(TextIO.from("ftp://...;)).

Thoughts ?

If agree, I would be happy to work on this (with any help ;)).

Regards
JB

On 07/23/2017 07:39 AM, Lucas Arruda wrote:

Hi Beam folks,

I would like to suggest the creation of a Pipeline I/O to support FTP/SFTP
as both source and sink locations for data processing. I've done some
research and it looks like there isn't any kind of development ongoing for
this (at least not on Jira).

I'd like to know your thoughts and if someone would like to help/support
this initiative. In case someone has started a thing already please let me
know ;)

Thank you,



--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: [VOTE] Release 2.1.0, release candidate #2

2017-07-24 Thread Jean-Baptiste Onofré 

Great initiative Kenn !

I will take a look.

Regards
JB

On 07/24/2017 07:57 AM, Kenneth Knowles wrote:

Nice catch.

Per our discussion on RC2 and now this, I started a spreadsheet for release
criteria.

Template: https://s.apache.org/beam-release-validation
Copy for this release: https://s.apache.org/beam-2.1.0-release-validation

I just directly took the validation criteria for the 2.0.0 and put it into
a spreadsheet form. Committers/PMC please ask for edit access I will grant
it (I'm just doing to this to avoid the work of scraping the list of
people). Everyone else should still suggest criteria to add (or remove) and
sign up to validate them.

Changes to the overall format are also welcome.

Kenn


On Sun, Jul 23, 2017 at 9:46 PM, Jean-Baptiste Onofré 
wrote:


Hi Luke,

good catch. I tested with "my" beam-samples but not with quickstart.

I would consider as blocker.

Let's wait for feedback from others. Depending of this feedback, I will
cancel RC2 and prepare a RC3 with the PR cherry-picked.

Thanks,
Regards
JB


On 07/23/2017 10:27 PM, Lukasz Cwik wrote:


I was going through the release, and noticed that during validation of the
quickstart when using Java. The steps for using Spark were not working for
me. It seems as though the version of spark-streaming_2.10 is missing from
the generated archetype pom.xml. Filed
https://issues.apache.org/jira/browse/BEAM-2662.

I currently set this as a blocker on 2.1.0 because I would expect that our
quickstart should work. If the Spark runner owners don't think this is a
blocker then I give the release a +1, otherwise -1. The quickstart for
Apex, Direct, Flink local cluster mode and Dataflow worked for me.

On Sun, Jul 23, 2017 at 12:51 PM, Jean-Baptiste Onofré 
wrote:

Gently reminder, the vote is still open.


Here's my +1 (binding).

Regards
JB


On 07/18/2017 06:30 PM, Jean-Baptiste Onofré wrote:

Hi everyone,


Please review and vote on the release candidate #2 for the version
2.1.0,
as follows:

[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)


The complete staging area is available for your review, which includes:
* JIRA release notes [1],
* the official Apache source release to be deployed to dist.apache.org
[2], which is signed with the key with fingerprint C8282E76 [3],
* all artifacts to be deployed to the Maven Central Repository [4],
* source code tag "v2.1.0-RC2" [5],
* website pull request listing the release and publishing the API
reference manual [6].
* Python artifacts are deployed along with the source release to the
dist.apache.org [2].

The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PMC affirmative votes.

Thanks,
JB

[1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?proje
ctId=12319527=12340528
[2] https://dist.apache.org/repos/dist/dev/beam/2.1.0/
[3] https://dist.apache.org/repos/dist/release/beam/KEYS
[4] https://repository.apache.org/content/repositories/orgapache
beam-1019/
[5] https://github.com/apache/beam/tree/v2.1.0-RC2
[6] https://github.com/apache/beam-site/pull/270



--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com





--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com





--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

[S]FTP support as Pipeline I/O

2017-07-24 Thread Lucas Arruda 
Hi Beam folks,

I would like to suggest the creation of a Pipeline I/O to support FTP/SFTP
as both source and sink locations for data processing. I've done some
research and it looks like there isn't any kind of development ongoing for
this (at least not on Jira).

I'd like to know your thoughts and if someone would like to help/support
this initiative. In case someone has started a thing already please let me
know ;)

Thank you,

-- 
*Lucas Arruda*