[ https://issues.apache.org/jira/browse/BEAM-3106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16313731#comment-16313731 ]
Ahmet Altay commented on BEAM-3106: ----------------------------------- [~m...@maxroos.com] if it helps you can upgrade Beam's bigquery version as a temporary relief until we have a permanent fix to this issue. > Consider not pinning all python dependencies, or moving them to > requirements.txt > -------------------------------------------------------------------------------- > > Key: BEAM-3106 > URL: https://issues.apache.org/jira/browse/BEAM-3106 > Project: Beam > Issue Type: Wish > Components: build-system > Affects Versions: 2.1.0 > Environment: python > Reporter: Maximilian Roos > > Currently all python dependencies are [pinned or > capped|https://github.com/apache/beam/blob/master/sdks/python/setup.py#L97] > While there's a good argument for supplying a `requirements.txt` with well > tested dependencies, having them specified in `setup.py` forces them to an > exact state on each install of Beam. This makes using Beam in any environment > with other libraries nigh on impossible. > This is particularly severe for the `gcp` dependencies, where we have > libraries that won't work with an older version (but Beam _does_ work with an > newer version). We have to do a bunch of gymnastics to get the correct > versions installed because of this. Unfortunately, airflow repeats this > practice and conflicts on a number of dependencies, adding further > complication (but, again there is no real conflict). > I haven't seen this practice outside of the Apache & Google ecosystem - for > example no libraries in numerical python do this. Here's a [discussion on > SO|https://stackoverflow.com/questions/28509481/should-i-pin-my-python-dependencies-versions] -- This message was sent by Atlassian JIRA (v6.4.14#64029)