ncg81 opened a new issue #3: URL: https://github.com/apache/kibble-scanners/issues/3
Hello, I've been able to set-up the **Kibble Server** and **Elastic Search** on 2 different Ubuntu Machines and a Windows 10 machine by closely following the [docs](https://apache-kibble.readthedocs.io/en/latest/setup.html), however I am experiencing a no-joy moment setting-up **kibble-scanners**. - **Test Machine A** : `Ubuntu 20.04 LTS` , `Python 3.8.2`, `pip 20.0.2`, `gcc 9.3.0` 1. First issue, the **bttf** build fails: ``` cos@ubuntu:~/temp/kibble-scanners$ pip3 install -r requirements.txt Collecting bttf Using cached bttf-0.1.tar.gz (11 kB) Requirement already satisfied: certifi in /usr/lib/python3/dist-packages (from -r requirements.txt (line 2)) (2019.11.28) Requirement already satisfied: elasticsearch in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 3)) (7.6.0) Requirement already satisfied: psutil in /home/cos/.local/lib/python3.8/site-packages (from -r requirements.txt (line 4)) (5.7.0) Requirement already satisfied: python-dateutil in /usr/lib/python3/dist-packages (from -r requirements.txt (line 5)) (2.7.3) Requirement already satisfied: requests in /usr/lib/python3/dist-packages (from -r requirements.txt (line 6)) (2.22.0) Requirement already satisfied: urllib3>=1.21.1 in /usr/lib/python3/dist-packages (from elasticsearch->-r requirements.txt (line 3)) (1.25.8) Building wheels for collected packages: bttf Building wheel for bttf (setup.py) ... error ERROR: Command errored out with exit status 1: command: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-z2997c34/bttf/setup.py'"'"'; __file__='"'"'/tmp/pip-install-z2997c34/bttf/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-pnzdwahd cwd: /tmp/pip-install-z2997c34/bttf/ Complete output (37 lines): /tmp/pip-install-z2997c34/bttf/setup.py:15: DistDeprecationWarning: Features are deprecated and will be removed in a future version. See https://github.com/pypa/setuptools/issues/65. speedups = Feature( /usr/lib/python3.8/distutils/core.py:108: DistDeprecationWarning: Features are deprecated and will be removed in a future version. See https://github.com/pypa/setuptools/issues/65. _setup_distribution = dist = klass(attrs) running bdist_wheel running build running build_py creating build creating build/lib.linux-x86_64-3.8 creating build/lib.linux-x86_64-3.8/bttf copying bttf/__init__.py -> build/lib.linux-x86_64-3.8/bttf running egg_info writing bttf.egg-info/PKG-INFO writing dependency_links to bttf.egg-info/dependency_links.txt writing top-level names to bttf.egg-info/top_level.txt reading manifest file 'bttf.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' writing manifest file 'bttf.egg-info/SOURCES.txt' copying bttf/_bttf.c -> build/lib.linux-x86_64-3.8/bttf copying bttf/bytesformat.h -> build/lib.linux-x86_64-3.8/bttf copying bttf/typehacks.h -> build/lib.linux-x86_64-3.8/bttf running build_ext building 'bttf._bttf' extension creating build/temp.linux-x86_64-3.8 creating build/temp.linux-x86_64-3.8/bttf x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.8 -c bttf/_bttf.c -o build/temp.linux-x86_64-3.8/bttf/_bttf.o In file included from bttf/_bttf.c:2: bttf/bytesformat.h:5:1: error: static declaration of ‘_PyUnicode_FormatLong’ follows non-static declaration 5 | _PyUnicode_FormatLong(PyObject *val, int alt, int prec, int type) | ^~~~~~~~~~~~~~~~~~~~~ In file included from /usr/include/python3.8/unicodeobject.h:1037, from /usr/include/python3.8/Python.h:97, from bttf/_bttf.c:1: /usr/include/python3.8/cpython/unicodeobject.h:1218:23: note: previous declaration of ‘_PyUnicode_FormatLong’ was here 1218 | PyAPI_FUNC(PyObject*) _PyUnicode_FormatLong(PyObject *, int, int, int); | ^~~~~~~~~~~~~~~~~~~~~ error: command 'x86_64-linux-gnu-gcc' failed with exit status 1 ---------------------------------------- ERROR: Failed building wheel for bttf Running setup.py clean for bttf Failed to build bttf Installing collected packages: bttf Running setup.py install for bttf ... error ERROR: Command errored out with exit status 1: command: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-z2997c34/bttf/setup.py'"'"'; __file__='"'"'/tmp/pip-install-z2997c34/bttf/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-9r9h__jy/install-record.txt --single-version-externally-managed --user --prefix= --compile --install-headers /home/cos/.local/include/python3.8/bttf cwd: /tmp/pip-install-z2997c34/bttf/ Complete output (37 lines): /tmp/pip-install-z2997c34/bttf/setup.py:15: DistDeprecationWarning: Features are deprecated and will be removed in a future version. See https://github.com/pypa/setuptools/issues/65. speedups = Feature( /usr/lib/python3.8/distutils/core.py:108: DistDeprecationWarning: Features are deprecated and will be removed in a future version. See https://github.com/pypa/setuptools/issues/65. _setup_distribution = dist = klass(attrs) running install running build running build_py creating build creating build/lib.linux-x86_64-3.8 creating build/lib.linux-x86_64-3.8/bttf copying bttf/__init__.py -> build/lib.linux-x86_64-3.8/bttf running egg_info writing bttf.egg-info/PKG-INFO writing dependency_links to bttf.egg-info/dependency_links.txt writing top-level names to bttf.egg-info/top_level.txt reading manifest file 'bttf.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' writing manifest file 'bttf.egg-info/SOURCES.txt' copying bttf/_bttf.c -> build/lib.linux-x86_64-3.8/bttf copying bttf/bytesformat.h -> build/lib.linux-x86_64-3.8/bttf copying bttf/typehacks.h -> build/lib.linux-x86_64-3.8/bttf running build_ext building 'bttf._bttf' extension creating build/temp.linux-x86_64-3.8 creating build/temp.linux-x86_64-3.8/bttf x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.8 -c bttf/_bttf.c -o build/temp.linux-x86_64-3.8/bttf/_bttf.o In file included from bttf/_bttf.c:2: bttf/bytesformat.h:5:1: error: static declaration of ‘_PyUnicode_FormatLong’ follows non-static declaration 5 | _PyUnicode_FormatLong(PyObject *val, int alt, int prec, int type) | ^~~~~~~~~~~~~~~~~~~~~ In file included from /usr/include/python3.8/unicodeobject.h:1037, from /usr/include/python3.8/Python.h:97, from bttf/_bttf.c:1: /usr/include/python3.8/cpython/unicodeobject.h:1218:23: note: previous declaration of ‘_PyUnicode_FormatLong’ was here 1218 | PyAPI_FUNC(PyObject*) _PyUnicode_FormatLong(PyObject *, int, int, int); | ^~~~~~~~~~~~~~~~~~~~~ error: command 'x86_64-linux-gnu-gcc' failed with exit status 1 ---------------------------------------- ERROR: Command errored out with exit status 1: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-z2997c34/bttf/setup.py'"'"'; __file__='"'"'/tmp/pip-install-z2997c34/bttf/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-9r9h__jy/install-record.txt --single-version-externally-managed --user --prefix= --compile --install-headers /home/cos/.local/include/python3.8/bttf Check the logs for full command output. ``` 2. And then: ``` cos@ubuntu:~/temp/kibble-scanners$ python3 src/kibble-scanner.py [core]: Loaded plugins/scanners/git-sync v/0.1.2 (Sync plugin for Git repositories) [core]: Loaded plugins/scanners/git-census v/0.1.0 (Census Scanner for Git) [core]: Loaded plugins/scanners/git-sloc v/0.1.0 (SloC Counter for Git) [core]: Loaded plugins/scanners/git-evolution v/0.1.0 (Git Evolution Scanner) [core]: Loaded plugins/scanners/jira v/0.1.0 (Scanner for Atlassian JIRA) [core]: Loaded plugins/scanners/ponymail v/0.1.0 (Scanner plugin for Apache Pony Mail) [core]: Loaded plugins/scanners/ponymail-tone v/0.1.0 (Tone/Mood Scanner plugin for Apache Pony Mail) [core]: Loaded plugins/scanners/ponymail-kpe v/0.1.0 (Key Phrase Extraction plugin for Apache Pony Mail) [core]: Loaded plugins/scanners/pipermail v/0.1.0 (Scanner for GNU Mailman Pipermail) [core]: Loaded plugins/scanners/github-issues v/0.1.0 (Scanner for GitHub Issues) [core]: Loaded plugins/scanners/bugzilla v/0.1.0 (Scanner for BugZilla) [core]: Loaded plugins/scanners/gerrit v/0.1.1 (Scanner for Gerrit Code Review) [core]: Loaded plugins/scanners/jenkins v/0.1.0 (Scanner for Jenkins CI) [core]: Loaded plugins/scanners/buildbot v/0.1.0 (Scanner for Buildbot) [core]: Loaded plugins/scanners/travis v/0.1.0 (Scanner for Travis CI) [core]: Loaded plugins/scanners/discourse v/0.1.0 (Scanner for Discourse Forums) [core]: Kibble Scanner v/0.1.0 starting src/kibble-scanner.py:122: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details. config = yaml.load(open(CONFIG_FILE)) [core]: Loaded YAML config from conf/config.yaml [core]: Using direct ElasticSearch broker model [core]: Connecting to ElasticSearch database at localhost:9200... [core]: Connected! [core]: This is a type-less DB, expanding database names instead. [core]: We're using ES >= 7.x, NO DOC_TYPE! Traceback (most recent call last): File "src/kibble-scanner.py", line 180, in <module> main() File "src/kibble-scanner.py", line 136, in main for org in broker.organisations(): File "/home/cos/temp/kibble-scanners/src/plugins/brokers/kibbleES.py", line 339, in organisations res = self.DB.search( File "/home/cos/temp/kibble-scanners/src/plugins/brokers/kibbleES.py", line 90, in search return self.ES.search( File "/usr/local/lib/python3.8/dist-packages/elasticsearch/client/utils.py", line 92, in _wrapped return func(*args, params=params, headers=headers, **kwargs) TypeError: search() got an unexpected keyword argument '_source_include' ``` - **Test Machine B:** `Ubuntu 18.04.4 LTS`, `Python 3.6.9`, `pip 9.0.1`, `gcc 7.5.0` 1. First issue, the **bttf** build fails: ``` cos@ubuntu:~/temp/kibble-scanners$ pip3 install -r requirements.txt Collecting bttf (from -r requirements.txt (line 1)) Using cached https://files.pythonhosted.org/packages/e0/33/adb62781dcbc9124f0a58ca15d3aaa3bdb5968863e354a62bbd52ea8eb88/bttf-0.1.tar.gz Collecting certifi (from -r requirements.txt (line 2)) Using cached https://files.pythonhosted.org/packages/57/2b/26e37a4b034800c960a00c4e1b3d9ca5d7014e983e6e729e33ea2f36426c/certifi-2020.4.5.1-py2.py3-none-any.whl Collecting elasticsearch (from -r requirements.txt (line 3)) Using cached https://files.pythonhosted.org/packages/cc/cf/7973ac58090b960857da04add0b345415bf1e1741beddf4cbe136b8ad174/elasticsearch-7.6.0-py2.py3-none-any.whl Collecting psutil (from -r requirements.txt (line 4)) Collecting python-dateutil (from -r requirements.txt (line 5)) Using cached https://files.pythonhosted.org/packages/d4/70/d60450c3dd48ef87586924207ae8907090de0b306af2bce5d134d78615cb/python_dateutil-2.8.1-py2.py3-none-any.whl Collecting requests (from -r requirements.txt (line 6)) Using cached https://files.pythonhosted.org/packages/1a/70/1935c770cb3be6e3a8b78ced23d7e0f3b187f5cbfab4749523ed65d7c9b1/requests-2.23.0-py2.py3-none-any.whl Collecting urllib3>=1.21.1 (from elasticsearch->-r requirements.txt (line 3)) Using cached https://files.pythonhosted.org/packages/e1/e5/df302e8017440f111c11cc41a6b432838672f5a70aa29227bf58149dc72f/urllib3-1.25.9-py2.py3-none-any.whl Collecting six>=1.5 (from python-dateutil->-r requirements.txt (line 5)) Using cached https://files.pythonhosted.org/packages/65/eb/1f97cb97bfc2390a276969c6fae16075da282f5058082d4cb10c6c5c1dba/six-1.14.0-py2.py3-none-any.whl Collecting chardet<4,>=3.0.2 (from requests->-r requirements.txt (line 6)) Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl Collecting idna<3,>=2.5 (from requests->-r requirements.txt (line 6)) Using cached https://files.pythonhosted.org/packages/89/e3/afebe61c546d18fb1709a61bee788254b40e736cff7271c7de5de2dc4128/idna-2.9-py2.py3-none-any.whl Building wheels for collected packages: bttf Running setup.py bdist_wheel for bttf ... error Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-mno2wh38/bttf/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmp3sczfwj7pip-wheel- --python-tag cp36: running bdist_wheel running build running build_py creating build creating build/lib.linux-x86_64-3.6 creating build/lib.linux-x86_64-3.6/bttf copying bttf/__init__.py -> build/lib.linux-x86_64-3.6/bttf running egg_info writing bttf.egg-info/PKG-INFO writing dependency_links to bttf.egg-info/dependency_links.txt writing top-level names to bttf.egg-info/top_level.txt reading manifest file 'bttf.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' writing manifest file 'bttf.egg-info/SOURCES.txt' copying bttf/_bttf.c -> build/lib.linux-x86_64-3.6/bttf copying bttf/bytesformat.h -> build/lib.linux-x86_64-3.6/bttf copying bttf/typehacks.h -> build/lib.linux-x86_64-3.6/bttf running build_ext building 'bttf._bttf' extension creating build/temp.linux-x86_64-3.6 creating build/temp.linux-x86_64-3.6/bttf x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.6m -c bttf/_bttf.c -o build/temp.linux-x86_64-3.6/bttf/_bttf.o In file included from bttf/_bttf.c:2:0: bttf/bytesformat.h:5:1: error: static declaration of ‘_PyUnicode_FormatLong’ follows non-static declaration _PyUnicode_FormatLong(PyObject *val, int alt, int prec, int type) ^~~~~~~~~~~~~~~~~~~~~ In file included from /usr/include/python3.6m/Python.h:89:0, from bttf/_bttf.c:1: /usr/include/python3.6m/unicodeobject.h:2307:23: note: previous declaration of ‘_PyUnicode_FormatLong’ was here PyAPI_FUNC(PyObject*) _PyUnicode_FormatLong(PyObject *, int, int, int); ^~~~~~~~~~~~~~~~~~~~~ error: command 'x86_64-linux-gnu-gcc' failed with exit status 1 ---------------------------------------- Failed building wheel for bttf Running setup.py clean for bttf Failed to build bttf Installing collected packages: bttf, certifi, urllib3, elasticsearch, psutil, six, python-dateutil, chardet, idna, requests Running setup.py install for bttf ... error Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-mno2wh38/bttf/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-gge6tlz0-record/install-record.txt --single-version-externally-managed --compile --user --prefix=: running install running build running build_py creating build creating build/lib.linux-x86_64-3.6 creating build/lib.linux-x86_64-3.6/bttf copying bttf/__init__.py -> build/lib.linux-x86_64-3.6/bttf running egg_info writing bttf.egg-info/PKG-INFO writing dependency_links to bttf.egg-info/dependency_links.txt writing top-level names to bttf.egg-info/top_level.txt reading manifest file 'bttf.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' writing manifest file 'bttf.egg-info/SOURCES.txt' copying bttf/_bttf.c -> build/lib.linux-x86_64-3.6/bttf copying bttf/bytesformat.h -> build/lib.linux-x86_64-3.6/bttf copying bttf/typehacks.h -> build/lib.linux-x86_64-3.6/bttf running build_ext building 'bttf._bttf' extension creating build/temp.linux-x86_64-3.6 creating build/temp.linux-x86_64-3.6/bttf x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.6m -c bttf/_bttf.c -o build/temp.linux-x86_64-3.6/bttf/_bttf.o In file included from bttf/_bttf.c:2:0: bttf/bytesformat.h:5:1: error: static declaration of ‘_PyUnicode_FormatLong’ follows non-static declaration _PyUnicode_FormatLong(PyObject *val, int alt, int prec, int type) ^~~~~~~~~~~~~~~~~~~~~ In file included from /usr/include/python3.6m/Python.h:89:0, from bttf/_bttf.c:1: /usr/include/python3.6m/unicodeobject.h:2307:23: note: previous declaration of ‘_PyUnicode_FormatLong’ was here PyAPI_FUNC(PyObject*) _PyUnicode_FormatLong(PyObject *, int, int, int); ^~~~~~~~~~~~~~~~~~~~~ error: command 'x86_64-linux-gnu-gcc' failed with exit status 1 ---------------------------------------- Command "/usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-mno2wh38/bttf/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-gge6tlz0-record/install-record.txt --single-version-externally-managed --compile --user --prefix=" failed with error code 1 in /tmp/pip-build-mno2wh38/bttf/ ``` 2. And then: ``` cos@ubuntu:~/temp/kibble-scanners$ python3 src/kibble-scanner.py [core]: Loaded plugins/scanners/git-sync v/0.1.2 (Sync plugin for Git repositories) [core]: Loaded plugins/scanners/git-census v/0.1.0 (Census Scanner for Git) [core]: Loaded plugins/scanners/git-sloc v/0.1.0 (SloC Counter for Git) [core]: Loaded plugins/scanners/git-evolution v/0.1.0 (Git Evolution Scanner) /usr/lib/python3/dist-packages/requests/__init__.py:80: RequestsDependencyWarning: urllib3 (1.25.9) or chardet (3.0.4) doesn't match a supported version! RequestsDependencyWarning) [core]: Loaded plugins/scanners/jira v/0.1.0 (Scanner for Atlassian JIRA) [core]: Loaded plugins/scanners/ponymail v/0.1.0 (Scanner plugin for Apache Pony Mail) [core]: Loaded plugins/scanners/ponymail-tone v/0.1.0 (Tone/Mood Scanner plugin for Apache Pony Mail) [core]: Loaded plugins/scanners/ponymail-kpe v/0.1.0 (Key Phrase Extraction plugin for Apache Pony Mail) [core]: Loaded plugins/scanners/pipermail v/0.1.0 (Scanner for GNU Mailman Pipermail) [core]: Loaded plugins/scanners/github-issues v/0.1.0 (Scanner for GitHub Issues) [core]: Loaded plugins/scanners/bugzilla v/0.1.0 (Scanner for BugZilla) [core]: Loaded plugins/scanners/gerrit v/0.1.1 (Scanner for Gerrit Code Review) [core]: Loaded plugins/scanners/jenkins v/0.1.0 (Scanner for Jenkins CI) [core]: Loaded plugins/scanners/buildbot v/0.1.0 (Scanner for Buildbot) [core]: Loaded plugins/scanners/travis v/0.1.0 (Scanner for Travis CI) [core]: Loaded plugins/scanners/discourse v/0.1.0 (Scanner for Discourse Forums) [core]: Kibble Scanner v/0.1.0 starting src/kibble-scanner.py:122: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details. config = yaml.load(open(CONFIG_FILE)) [core]: Loaded YAML config from conf/config.yaml [core]: Using direct ElasticSearch broker model [core]: Connecting to ElasticSearch database at localhost:9200... [core]: Connected! [core]: This is a type-less DB, expanding database names instead. [core]: We're using ES >= 7.x, NO DOC_TYPE! Traceback (most recent call last): File "src/kibble-scanner.py", line 180, in <module> main() File "src/kibble-scanner.py", line 136, in main for org in broker.organisations(): File "/home/cos/temp/kibble-scanners/src/plugins/brokers/kibbleES.py", line 345, in organisations 'match_all': {} File "/home/cos/temp/kibble-scanners/src/plugins/brokers/kibbleES.py", line 94, in search body = body File "/home/cos/.local/lib/python3.6/site-packages/elasticsearch/client/utils.py", line 92, in _wrapped return func(*args, params=params, headers=headers, **kwargs) TypeError: search() got an unexpected keyword argument '_source_include' ``` Any ideas? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
