Hi Sébastien, Currently there is no work in progress for including the write transforms for the locations you listed. You could develop your own version if interested. Please see WriteToBigquery transform [1] for reference.
Ahmet [1] https://github.com/apache/beam/blob/375bd3a6a53ba3ba7c965278dcb322875e1b4dca/sdks/python/apache_beam/io/gcp/bigquery.py#L1287 On Wed, Jun 27, 2018 at 2:48 AM, Sebastien Morand < [email protected]> wrote: > Hi, > > Thanks for your answer. > > Ok looking forward and ready to test alpha on this. Because we have > actually some use cases to send data to CloudSQL or Spanner or BigTable or > Firestore. As far as I read the documentation, there is no native support > for them and so we have already implemented a custom source support. > > So is there any work in progress for any of the 4 sinks above for native > support in the python SDK? > > Regards, > > *Sébastien MORAND* > Team Lead Solution Architect > Technology & Operations / Digital Factory > Veolia - Group Information Systems & Technology (IS&T) > Cell.: +33 6 12 03 41 15 / Direct: +33 1 85 57 71 08 > Bureau 0144C (Ouest) > 30, rue Madeleine-Vionnet > <https://maps.google.com/?q=30,+rue+Madeleine-Vionnet+%C2%A0-%C2%A0+93300+Aubervilliers,+France&entry=gmail&source=g> > - > <https://maps.google.com/?q=30,+rue+Madeleine-Vionnet+%C2%A0-%C2%A0+93300+Aubervilliers,+France&entry=gmail&source=g>93300 > Aubervilliers, France > <https://maps.google.com/?q=30,+rue+Madeleine-Vionnet+%C2%A0-%C2%A0+93300+Aubervilliers,+France&entry=gmail&source=g> > *www.veolia.com <http://www.veolia.com>* > <http://www.veolia.com> > <https://www.facebook.com/veoliaenvironment/> > <https://www.youtube.com/user/veoliaenvironnement> > <https://www.linkedin.com/company/veolia-environnement> > <https://twitter.com/veolia> > > > On Thu, 21 Jun 2018 at 16:53, Lukasz Cwik <[email protected]> wrote: > >> [email protected] >> >> Python streaming custom source support will be available via >> SplittableDoFn. It is actively being worked on by a few contributors but to >> my knowledge there is no roadmap yet for having support for this for >> Dataflow. >> >> >> On Thu, Jun 21, 2018 at 1:19 AM Sebastien Morand < >> [email protected]> wrote: >> >>> Hi, >>> >>> We need to setup streaming dataflow in python using developed source (in >>> the opposite of native source) for Cloud SQL integration and Firestore >>> integration. >>> >>> This is currently not supported as far as I understood the >>> documentation, can you confirm? Any roadmap on the topic? >>> >>> Thanks by advance, >>> Regards, >>> >>> *Sébastien MORAND* >>> Team Lead Solution Architect >>> Technology & Operations / Digital Factory >>> Veolia - Group Information Systems & Technology (IS&T) >>> Cell.: +33 6 12 03 41 15 / Direct: +33 1 85 57 71 08 >>> Bureau 0144C (Ouest) >>> 30, rue Madeleine-Vionnet >>> <https://maps.google.com/?q=30,+rue+Madeleine-Vionnet+%C2%A0-%C2%A0+93300+Aubervilliers,+France&entry=gmail&source=g> >>> - >>> <https://maps.google.com/?q=30,+rue+Madeleine-Vionnet+%C2%A0-%C2%A0+93300+Aubervilliers,+France&entry=gmail&source=g>93300 >>> Aubervilliers, France >>> <https://maps.google.com/?q=30,+rue+Madeleine-Vionnet+%C2%A0-%C2%A0+93300+Aubervilliers,+France&entry=gmail&source=g> >>> *www.veolia.com <http://www.veolia.com>* >>> <http://www.veolia.com> >>> <https://www.facebook.com/veoliaenvironment/> >>> <https://www.youtube.com/user/veoliaenvironnement> >>> <https://www.linkedin.com/company/veolia-environnement> >>> <https://twitter.com/veolia> >>> >>> >>> ------------------------------------------------------------ >>> -------------------------------- >>> This e-mail transmission (message and any attached files) may contain >>> information that is proprietary, privileged and/or confidential to Veolia >>> Environnement and/or its affiliates and is intended exclusively for the >>> person(s) to whom it is addressed. If you are not the intended recipient, >>> please notify the sender by return e-mail and delete all copies of this >>> e-mail, including all attachments. Unless expressly authorized, any use, >>> disclosure, publication, retransmission or dissemination of this e-mail >>> and/or of its attachments is strictly prohibited. >>> >>> Ce message electronique et ses fichiers attaches sont strictement >>> confidentiels et peuvent contenir des elements dont Veolia Environnement >>> et/ou l'une de ses entites affiliees sont proprietaires. Ils sont donc >>> destines a l'usage de leurs seuls destinataires. Si vous avez recu ce >>> message par erreur, merci de le retourner a son emetteur et de le detruire >>> ainsi que toutes les pieces attachees. L'utilisation, la divulgation, la >>> publication, la distribution, ou la reproduction non expressement >>> autorisees de ce message et de ses pieces attachees sont interdites. >>> ------------------------------------------------------------ >>> -------------------------------- >>> >> > > ------------------------------------------------------------ > -------------------------------- > This e-mail transmission (message and any attached files) may contain > information that is proprietary, privileged and/or confidential to Veolia > Environnement and/or its affiliates and is intended exclusively for the > person(s) to whom it is addressed. If you are not the intended recipient, > please notify the sender by return e-mail and delete all copies of this > e-mail, including all attachments. Unless expressly authorized, any use, > disclosure, publication, retransmission or dissemination of this e-mail > and/or of its attachments is strictly prohibited. > > Ce message electronique et ses fichiers attaches sont strictement > confidentiels et peuvent contenir des elements dont Veolia Environnement > et/ou l'une de ses entites affiliees sont proprietaires. Ils sont donc > destines a l'usage de leurs seuls destinataires. Si vous avez recu ce > message par erreur, merci de le retourner a son emetteur et de le detruire > ainsi que toutes les pieces attachees. L'utilisation, la divulgation, la > publication, la distribution, ou la reproduction non expressement > autorisees de ce message et de ses pieces attachees sont interdites. > ------------------------------------------------------------ > -------------------------------- >
