This is an automated email from the ASF dual-hosted git repository.
laiyingchun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pegasus.git
The following commit(s) were added to refs/heads/master by this push:
new 1b44329a9 chore(python-client-ci): build the server instead of
downloading it from external site (#1815)
1b44329a9 is described below
commit 1b44329a9b4394cccbd0a65909ce2bafbd910677
Author: Yingchun Lai <[email protected]>
AuthorDate: Tue Dec 26 19:15:14 2023 +0800
chore(python-client-ci): build the server instead of downloading it from
external site (#1815)
https://github.com/apache/incubator-pegasus/issues/1779
- Build Pegasus server binaries in the CI procedure instead of downloading
it from any site
- Remove python-client/test.sh
---
.github/workflows/test_python-client.yml | 66 +++++++++++++++++++++++------
python-client/README.md | 71 +++++++++++++++++++++-----------
python-client/test.sh | 44 --------------------
3 files changed, 99 insertions(+), 82 deletions(-)
diff --git a/.github/workflows/test_python-client.yml
b/.github/workflows/test_python-client.yml
index bdef9d221..c1eaaa7b7 100644
--- a/.github/workflows/test_python-client.yml
+++ b/.github/workflows/test_python-client.yml
@@ -1,4 +1,3 @@
-#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
@@ -15,7 +14,6 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-#
name: Test - python client
on:
@@ -35,21 +33,63 @@ defaults:
run:
shell: bash
+env:
+ ARTIFACT_NAME: release_for_python_client
+
jobs:
+ build_server:
+ name: Build server
+ runs-on: ubuntu-latest
+ env:
+ USE_JEMALLOC: OFF
+ BUILD_OPTIONS: -t release
+ container:
+ image: apache/pegasus:thirdparties-bin-test-ubuntu2204-${{
github.base_ref }}
+ steps:
+ - uses: actions/checkout@v3
+ - uses: "./.github/actions/rebuild_thirdparty_if_needed"
+ - uses: "./.github/actions/build_pegasus"
+ - uses: "./.github/actions/upload_artifact"
+
test:
- name: test
- runs-on: ubuntu-20.04
+ name: Test Python client
+ needs: build_server
+ runs-on: ubuntu-latest
+ container:
+ image: apache/pegasus:thirdparties-bin-test-ubuntu2204-${{
github.base_ref }}
steps:
- - name: Install thrift
- # python-client imports thrift package of 0.13.0, so we must use
thrift-compiler 0.13.0
- # to generate code as well. The thrift-compiler version on
ubuntu-20.04 is 0.13.0
- run: sudo apt-get install -y thrift-compiler
- uses: actions/checkout@v3
- - uses: actions/setup-python@v4
+ - uses: actions/setup-python@v5
with:
- python-version: '3.8'
- - name: test
+ python-version: '3.11'
+ # python-client imports thrift package of 0.13.0, so we must use
thrift-compiler 0.13.0
+ # to generate code as well. The thrift-compiler version on ubuntu-22.04
is 0.16.0, so
+ # build and install thrift-compiler 0.13.0 manually.
+ - name: Install thrift
+ run: |
+ export THRIFT_VERSION=0.13.0
+ wget --progress=dot:giga
https://github.com/apache/thrift/archive/refs/tags/v${THRIFT_VERSION}.tar.gz
+ tar -xzf v${THRIFT_VERSION}.tar.gz
+ cd thrift-${THRIFT_VERSION}
+ ./bootstrap.sh
+ ./configure --enable-libs=no
+ make -j $(nproc)
+ make install
+ cd - && rm -rf thrift-${THRIFT_VERSION} v${THRIFT_VERSION}.tar.gz
+ - name: Download artifact
+ uses: "./.github/actions/download_artifact"
+ - name: Start Pegasus cluster
+ run: |
+ export
LD_LIBRARY_PATH=$(pwd)/thirdparty/output/lib:${JAVA_HOME}/jre/lib/amd64/server
+ ulimit -s unlimited
+ ./run.sh start_onebox
+ - name: Run Python client tests
working-directory: ./python-client
run: |
- pip install -r requirement.txt
- ./test.sh
+ python -m pip install -r requirement.txt
+ thrift -v --gen py -out . ../idl/dsn.layer2.thrift
+ thrift -v --gen py -out . ../idl/rrdb.thrift
+ python setup.py install
+ python sample.py
+ cd tests
+ python -m unittest -v test_basics.TestBasics
diff --git a/python-client/README.md b/python-client/README.md
index 826099ceb..ce978df85 100644
--- a/python-client/README.md
+++ b/python-client/README.md
@@ -17,13 +17,12 @@ specific language governing permissions and limitations
under the License.
-->
-pegasus-python-client
-=====================
+# pegasus-python-client
It uses [Twisted](http://twistedmatrix.com) for the asynchronous communication
with pegasus server.
-Installation
-------------
+## Installation
+
Python 3.8+
pypegasus can be installed via pip as follows:
@@ -36,29 +35,45 @@ thrift --gen py -out . ../idl/dsn.layer2.thrift
python3 setup.py install
```
-Usage
------
+## Usage
+
There are some basic guide in [`sample.py`](sample.py).
-Test
-----
-Before testing, you should firstly start an onebox cluster, referring to [`how
to start an onebox cluster`](https://pegasus.apache.org/overview/onebox/).
+## Test
+
+Note: Before testing, you should firstly start an
[onebox](https://pegasus.apache.org/overview/onebox/).
+
+### Run sample
-## Basic interfaces test:
+```shell
+python3 sample.py
+```
-`cd tests && python3 -m unittest test_basics.TestBasics`
+### Basic interfaces test
-## Integration test:
+```shell
+cd tests
+python3 -m unittest test_basics.TestBasics
+```
-`cd tests && python3 -m twisted.trial test_integration.py`
-<!-- markdown-link-check-disable -->
-ATTENTION: you should firstly set proper pegasus shell path in
[`test_integration.py`](test_integration.py#L10).
-<!-- markdown-link-check-enable-->
+### Integration test
+> Note: You should firstly set proper Pegasus shell path in
test_integration.py.
+> ```python
+> shell_path = '/your/pegasus-shell/dir'
+> ```
-## Benchmark test:
+```shell
+cd tests
+python3 -m twisted.trial test_integration.py
+```
-`cd tests && python3 -m unittest test_benchmark.TestBasics`
+## Benchmark test
+
+```shell
+cd tests
+python3 -m unittest test_benchmark.TestBasics
+```
The test result on my personal PC (CPU: Intel i7-7700 3.60GHz, mem: 8G) is:
```
@@ -66,10 +81,13 @@ The test result on my personal PC (CPU: Intel i7-7700
3.60GHz, mem: 8G) is:
10000 remove cost: 7.61887693405 s, 0.000761887693405 s per op
10000 set cost: 6.26366090775 s, 0.000626366090775 s per op
```
-FAQ
----
-### Q: When I run sample.py, error occured as following:
-```
+
+## FAQ
+
+### Q: Error occurred about `'module' object has no attribute 'OP_NO_TLSv1_1'`:
+
+Error details:
+```shell
Traceback (most recent call last):
File "./sample.py", line 3, in <module>
from pgclient import *
@@ -80,9 +98,12 @@ Traceback (most recent call last):
TLSVersion.TLSv1_1: SSL.OP_NO_TLSv1_1,
AttributeError: 'module' object has no attribute 'OP_NO_TLSv1_1'
```
-### A: pyOpenSSL version too low
-the twisted version we used is 17.9.0, which requires pyOpenSSL>=16.0.0, you
can
-`pip install --upgrade pyopenssl`
+#### A: pyOpenSSL version too low
+The twisted version we used is 17.9.0, which requires pyOpenSSL>=16.0.0, you
can
+
+```shell
+pip install --upgrade pyopenssl
+```
ref: https://github.com/scrapy/scrapy/issues/2473
diff --git a/python-client/test.sh b/python-client/test.sh
deleted file mode 100755
index f8c270841..000000000
--- a/python-client/test.sh
+++ /dev/null
@@ -1,44 +0,0 @@
-#!/usr/bin/env bash
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-set -e
-
-PY_CLIENT_DIR=`pwd`
-PEGASUS_PKG="pegasus-tools-2.0.0-5d969e8-glibc2.12-release"
-PEGASUS_PKG_URL="https://github.com/apache/incubator-pegasus/releases/download/v2.0.0/pegasus-tools-2.0.0-5d969e8-glibc2.12-release.tar.gz";
-
-thrift -v --gen py -out . ../idl/dsn.layer2.thrift
-thrift -v --gen py -out . ../idl/rrdb.thrift
-
-# TODO(yingchun): how to install pypegasus before generating thrift files?
-python setup.py install
-
-# start pegasus onebox environment
-if [ ! -f ${PEGASUS_PKG}.tar.gz ]; then
- wget --quiet ${PEGASUS_PKG_URL}
- tar xf ${PEGASUS_PKG}.tar.gz
-fi
-cd ${PEGASUS_PKG}
-./run.sh clear_onebox
-./run.sh start_onebox -m 3 -r 3 -w
-
-cd "${PY_CLIENT_DIR}"
-python3 sample.py
-
-cd "${PY_CLIENT_DIR}"/tests
-python3 -m unittest -v test_basics.TestBasics
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
