Thanks for the clarification, Robert! I actually thought that Direct Runner was already supporting streaming (since the streaming_wordcount.py example was already in 2.0.0 release), but guess I was at most partially right.
How can I help the development or/and testing process of streaming? Also, is there any known timeline on getting Python SDK 2.0.0 supported on Dataflow? On Thu, Jun 1, 2017 at 9:47 AM, Robert Bradshaw <[email protected]> wrote: > Currently Beam Python is batch only, with support for the Dataflow and > Direct (single machine) runners only. However, there is active work > making Python run on other runners (the key term to follow is "Fn > API") and support streaming which I expect are not too far away. > > On Thu, Jun 1, 2017 at 6:11 AM, Jean-Baptiste Onofré <[email protected]> > wrote: > > Hi Sébastien, > > > > First, the Python SDK currently uses its own runner (kind of python > direct > > runner). It doesn't yet leverage the existing runner as the Java SDK > does. > > It's something we plan. > > > > 1. So, assuming you use the Python Direct runner, some preparation work > has > > been done, like support of bundles. However, the streaming support is not > > yet fully complete. Some pull request are in preparation around that: > > https://issues.apache.org/jira/browse/BEAM-1265 > > > > 2. Yes, PubSub support has been added: > > > > https://github.com/apache/beam/blob/master/sdks/python/ > apache_beam/io/gcp/pubsub.py > > > > as part of the GCP "IOs" (bigquery, bigtable, ...) > > > > However, I don't say how stable it is ;) > > > > I let the "Python guys" provide more details. > > > > Regards > > JB > > > > On 06/01/2017 03:05 PM, Morand, Sebastien wrote: > >> > >> Hi, > >> > >> The Python SDK has been released on may, 16th but with very little > >> information about new or breaking changes. > >> > >> Some important questions: > >> > >> 1. Is Streamline mode supported now? > >> > >> 2. Is PubSub source fully functional and stable? > >> > >> Thanks by advance, > >> Regards, > >> > >> *Sébastien MORAND* > >> Team Lead Solution Architect > >> Technology & Operations / Digital Factory > >> Veolia - Group Information Systems & Technology (IS&T) > >> Cell.:+33 7 52 66 20 81 / Direct: +33 1 85 57 71 08 > >> Bureau 0144C (Ouest) > >> 30, rue Madeleine-Vionnet - 93300 Aubervilliers, France > >> _www.veolia.com <http://www.veolia.com>_ > >> <http://www.veolia.com> > >> <https://www.facebook.com/veoliaenvironment/> > >> <https://www.youtube.com/user/veoliaenvironnement> > >> <https://www.linkedin.com/company/veolia-environnement> > >> <https://twitter.com/veolia> > >> > >> > >> > >> ------------------------------------------------------------ > -------------------------------- > >> This e-mail transmission (message and any attached files) may contain > >> information that is proprietary, privileged and/or confidential to > Veolia > >> Environnement and/or its affiliates and is intended exclusively for the > >> person(s) to whom it is addressed. If you are not the intended > recipient, > >> please notify the sender by return e-mail and delete all copies of this > >> e-mail, including all attachments. Unless expressly authorized, any use, > >> disclosure, publication, retransmission or dissemination of this e-mail > >> and/or of its attachments is strictly prohibited. > >> > >> Ce message electronique et ses fichiers attaches sont strictement > >> confidentiels et peuvent contenir des elements dont Veolia Environnement > >> et/ou l'une de ses entites affiliees sont proprietaires. Ils sont donc > >> destines a l'usage de leurs seuls destinataires. Si vous avez recu ce > >> message par erreur, merci de le retourner a son emetteur et de le > detruire > >> ainsi que toutes les pieces attachees. L'utilisation, la divulgation, la > >> publication, la distribution, ou la reproduction non expressement > autorisees > >> de ce message et de ses pieces attachees sont interdites. > >> > >> ------------------------------------------------------------ > -------------------------------- > > > > > > -- > > Jean-Baptiste Onofré > > [email protected] > > http://blog.nanthrax.net > > Talend - http://www.talend.com > -- Best regards, Dmitry Demeshchuk.
