Hi Ahmet, Thanks a lot for pointing out that doc, I somehow missed it from the official Python SDK page!
One thing that comes to my mind is that generally one should probably use the 'install' command in setuptools, not 'build', like it's done in https://github.com/apache/beam/blob/master/sdks/python/ap ache_beam/examples/complete/juliaset/setup.py#L113. Reason being, the 'build' step seems to be executed on the original machine, not inside the runner's containers, while 'install' will be triggered inside of them. If I run a pipeline that uses setup.py with a "build" step, it fails due to being unable to "apt-get install libpq-dev" on a mac. I'm still trying to make it work with either build or install steps, talking to the Dataflow folks in parallel to get more understanding of what I'm doing wrong (Dataflow doesn't send out installation failure logs to Stackdriver, only runtime logs, so it seems). On Tue, Jun 6, 2017 at 9:21 AM, Ahmet Altay <[email protected]> wrote: > Hi, > > Please see Managing Python Pipeline Dependencies [1] for various ways on > installing additional dependencies. The section on non-python dependencies > is relevant to your question. > > Thank you, > Ahmet > > [1] https://beam.apache.org/documentation/sdks/python- > pipeline-dependencies/ > > On Mon, Jun 5, 2017 at 11:52 PM, Morand, Sebastien < > [email protected]> wrote: > >> Hi, >> >> Interested too. Could be fine for instance to add sftp BoundedSource, but >> compilalation of paramiko with ssl library (and so installation of ssl-dev) >> >> Regards, >> >> *Sébastien MORAND* >> Team Lead Solution Architect >> Technology & Operations / Digital Factory >> Veolia - Group Information Systems & Technology (IS&T) >> Cell.: +33 7 52 66 20 81 / Direct: +33 1 85 57 71 08 >> <+33%201%2085%2057%2071%2008> >> Bureau 0144C (Ouest) >> 30, rue Madeleine-Vionnet - 93300 Aubervilliers, France >> *www.veolia.com <http://www.veolia.com>* >> <http://www.veolia.com> >> <https://www.facebook.com/veoliaenvironment/> >> <https://www.youtube.com/user/veoliaenvironnement> >> <https://www.linkedin.com/company/veolia-environnement> >> <https://twitter.com/veolia> >> >> On 6 June 2017 at 08:01, Dmitry Demeshchuk <[email protected]> wrote: >> >>> Hi again, folks, >>> >>> How should I go about installing Python packages that require to be >>> built and/or require native dependencies like shared libraries or such? >>> >>> I guess, I could potentially build the C-based modules using the same >>> version of kernel and glibc that Dataflow is running, but doesn't seem like >>> there's any way to install shared libraries at these boxes, right? >>> >>> Thanks! >>> >>> -- >>> Best regards, >>> Dmitry Demeshchuk. >>> >> >> >> >> ------------------------------------------------------------ >> -------------------------------- >> This e-mail transmission (message and any attached files) may contain >> information that is proprietary, privileged and/or confidential to Veolia >> Environnement and/or its affiliates and is intended exclusively for the >> person(s) to whom it is addressed. If you are not the intended recipient, >> please notify the sender by return e-mail and delete all copies of this >> e-mail, including all attachments. Unless expressly authorized, any use, >> disclosure, publication, retransmission or dissemination of this e-mail >> and/or of its attachments is strictly prohibited. >> >> Ce message electronique et ses fichiers attaches sont strictement >> confidentiels et peuvent contenir des elements dont Veolia Environnement >> et/ou l'une de ses entites affiliees sont proprietaires. Ils sont donc >> destines a l'usage de leurs seuls destinataires. Si vous avez recu ce >> message par erreur, merci de le retourner a son emetteur et de le detruire >> ainsi que toutes les pieces attachees. L'utilisation, la divulgation, la >> publication, la distribution, ou la reproduction non expressement >> autorisees de ce message et de ses pieces attachees sont interdites. >> ------------------------------------------------------------ >> -------------------------------- >> > > -- Best regards, Dmitry Demeshchuk.
