potiuk commented on issue #4483: [AIRFLOW-3673] Add official dockerfile URL: https://github.com/apache/airflow/pull/4483#issuecomment-454691659 @ashb. I tracked down the root cause of the "Popen" differences I had in mind. Now I know why I thought it was related to Popen. It was in fact`json.loads()` behaviour difference that caused our often problems where we developed in 3.5 and run in 3.5. In python3 (both 3.5 and 3.6) `check_output()` always produces bytes (in 2.7 it was `str` of course). But json.loads() in python 2.7 and 3.5 can only consume `str`. Fortunately (or unfortunately) In python3.6 - `json.loads()` can also consume "utf-8" encoded bytes (https://docs.python.org/3/whatsnew/3.6.html#json). We often (for setting up/tearing down test environment) run gcloud command that produces json output that we then parse and use in python to do something (for example kill all running compute instances). So then the problem is that `json.loads(subprocess.check_output())` works in python 2.7 (with 'str') and in python 3.6 (with bytes) but does not work for 3.5 (json.loads() cannot consume bytes). It can be easily fixed with `json.loads(subprocess.check_output().decode("utf-8")` which works on all versions. But it is super-easy to forget about it when your main development environment is 3.6 :) Here is some easily reproducible example run in python 2.7, 3.5 and 3.6: ## Python 2.7 ``` [potiuk:~/code/airflow-breeze/workspaces/polidea/airflow] AIRFLOW-3710-fix-warnings-in-Google-Cloud-Base-Hook(+1/-1)+ 27m45s ± python2.7 Python 2.7.10 (default, Aug 17 2018, 19:45:58) [GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.0.42)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import json; import subprocess; x=json.loads(subprocess.check_output(['gcloud','compute','instances','list','--format=json']));x [{u'cpuPlatform': u'Unknown CPU Platform', u'deletionProtection': False, u'kind': u'compute#instance', u'canIpForward': False, u'description': u'', u'zone': u'https://www.googleapis.com/compute/v1/projects/polidea-airflow/zones/europe-west3-b', u'tags': {u'fingerprint': u'42WmSpB8rSM='}, u'labelFingerprint': u'42WmSpB8rSM=', u'disks': [{u'deviceName': u'pa-1', u'kind': u'compute#attachedDisk', u'autoDelete': True, u'index': 0, u'boot': True, u'guestOsFeatures': [{u'type': u'VIRTIO_SCSI_MULTIQUEUE'}], u'licenses': [u'https://www.googleapis.com/compute/v1/projects/debian-cloud/global/licenses/debian-9-stretch'], u'mode': u'READ_WRITE', u'interface': u'SCSI', u'type': u'PERSISTENT', u'source': u'https://www.googleapis.com/compute/v1/projects/polidea-airflow/zones/europe-west3-b/disks/pa-1'}], u'startRestricted': False, u'name': u'pa-1', u'status': u'TERMINATED', u'scheduling': {u'automaticRestart': True, u'preemptible': False, u'onHostMaintenance': u'MIGRATE'}, u'machineType': u'https://www.googleapis.com/compute/v1/projects/polidea-airflow/zones/europe-west3-b/machineTypes/n1-standard-1', u'serviceAccounts': [{u'scopes': [u'https://www.googleapis.com/auth/cloud-platform'], u'email': u'[email protected]'}], u'networkInterfaces': [{u'kind': u'compute#networkInterface', u'network': u'https://www.googleapis.com/compute/v1/projects/polidea-airflow/global/networks/default', u'accessConfigs': [{u'networkTier': u'PREMIUM', u'kind': u'compute#accessConfig', u'type': u'ONE_TO_ONE_NAT', u'name': u'External NAT'}], u'networkIP': u'10.156.0.2', u'fingerprint': u'ah2_E5U9DlI=', u'subnetwork': u'https://www.googleapis.com/compute/v1/projects/polidea-airflow/regions/europe-west3/subnetworks/default', u'name': u'nic0'}], u'creationTimestamp': u'2018-09-25T01:29:48.779-07:00', u'id': u'2480086944131075860', u'selfLink': u'https://www.googleapis.com/compute/v1/projects/polidea-airflow/zones/europe-west3-b/instances/pa-1', u'metadata': {u'kind': u'compute#metadata', u'fingerprint': u'GDPUYxlwHe4='}}] >>> exit() ``` ## Python 3.6: ``` [potiuk:~/code/airflow-breeze/workspaces/polidea/airflow] AIRFLOW-3710-fix-warnings-in-Google-Cloud-Base-Hook(+1/-1)+ 28m9s ± python3.6 Python 3.6.7 (v3.6.7:6ec5cf24b7, Oct 20 2018, 03:02:14) [GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import json; import subprocess; x=json.loads(subprocess.check_output(['gcloud','compute','instances','list','--format=json']));x [{'canIpForward': False, 'cpuPlatform': 'Unknown CPU Platform', 'creationTimestamp': '2018-09-25T01:29:48.779-07:00', 'deletionProtection': False, 'description': '', 'disks': [{'autoDelete': True, 'boot': True, 'deviceName': 'pa-1', 'guestOsFeatures': [{'type': 'VIRTIO_SCSI_MULTIQUEUE'}], 'index': 0, 'interface': 'SCSI', 'kind': 'compute#attachedDisk', 'licenses': ['https://www.googleapis.com/compute/v1/projects/debian-cloud/global/licenses/debian-9-stretch'], 'mode': 'READ_WRITE', 'source': 'https://www.googleapis.com/compute/v1/projects/polidea-airflow/zones/europe-west3-b/disks/pa-1', 'type': 'PERSISTENT'}], 'id': '2480086944131075860', 'kind': 'compute#instance', 'labelFingerprint': '42WmSpB8rSM=', 'machineType': 'https://www.googleapis.com/compute/v1/projects/polidea-airflow/zones/europe-west3-b/machineTypes/n1-standard-1', 'metadata': {'fingerprint': 'GDPUYxlwHe4=', 'kind': 'compute#metadata'}, 'name': 'pa-1', 'networkInterfaces': [{'accessConfigs': [{'kind': 'compute#accessConfig', 'name': 'External NAT', 'networkTier': 'PREMIUM', 'type': 'ONE_TO_ONE_NAT'}], 'fingerprint': 'ah2_E5U9DlI=', 'kind': 'compute#networkInterface', 'name': 'nic0', 'network': 'https://www.googleapis.com/compute/v1/projects/polidea-airflow/global/networks/default', 'networkIP': '10.156.0.2', 'subnetwork': 'https://www.googleapis.com/compute/v1/projects/polidea-airflow/regions/europe-west3/subnetworks/default'}], 'scheduling': {'automaticRestart': True, 'onHostMaintenance': 'MIGRATE', 'preemptible': False}, 'selfLink': 'https://www.googleapis.com/compute/v1/projects/polidea-airflow/zones/europe-west3-b/instances/pa-1', 'serviceAccounts': [{'email': '[email protected]', 'scopes': ['https://www.googleapis.com/auth/cloud-platform']}], 'startRestricted': False, 'status': 'TERMINATED', 'tags': {'fingerprint': '42WmSpB8rSM='}, 'zone': 'https://www.googleapis.com/compute/v1/projects/polidea-airflow/zones/europe-west3-b'}] >>> quit() ``` ## Python 3.5 ``` [potiuk:~/code/airflow-breeze/workspaces/polidea/airflow] AIRFLOW-3710-fix-warnings-in-Google-Cloud-Base-Hook(+1/-1)+ 28m26s ± python3.5 Python 3.5.4 (v3.5.4:3f56838976, Aug 7 2017, 12:56:33) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import json; import subprocess; x=json.loads(subprocess.check_output(['gcloud','compute','instances','list','--format=json']));x Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/json/__init__.py", line 312, in loads s.__class__.__name__)) TypeError: the JSON object must be str, not 'bytes' >>> ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
