This is an automated email from the ASF dual-hosted git repository.
lzljs3620320 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/paimon.git
The following commit(s) were added to refs/heads/master by this push:
new 52cabeb497 [python] Add ci and build to the python project (#5914)
52cabeb497 is described below
commit 52cabeb4971689b510ea61463963564bef93f6a8
Author: HeavenZH <[email protected]>
AuthorDate: Thu Jul 17 13:25:32 2025 +0800
[python] Add ci and build to the python project (#5914)
---
.github/workflows/check-licensing.yml | 6 +-
.github/workflows/paimon-python-checks.yml | 51 ++++
.gitignore | 5 +-
docs/content/program-api/python-api.md | 8 +-
paimon-python/README.md | 14 ++
paimon-python/dev/.rat-excludes | 20 ++
paimon-python/dev/cfg.ini | 28 +++
paimon-python/dev/check-licensing.sh | 79 ++++++
paimon-python/dev/lint-python.sh | 275 +++++++++++++++++++++
{pypaimon => paimon-python/pypaimon}/__init__.py | 0
.../pypaimon}/api/__init__.py | 14 +-
.../pypaimon}/api/api_response.py | 10 +-
.../pypaimon}/api/api_resquest.py | 2 +-
{pypaimon => paimon-python/pypaimon}/api/auth.py | 5 +-
{pypaimon => paimon-python/pypaimon}/api/client.py | 11 +-
.../pypaimon}/api/data_types.py | 0
.../pypaimon}/api/rest_json.py | 2 +-
.../pypaimon}/api/typedef.py | 0
.../pypaimon}/tests/__init__.py | 0
.../pypaimon}/tests/api_test.py | 29 ++-
paimon-python/setup.py | 50 ++++
pypaimon/README.md | 3 -
22 files changed, 564 insertions(+), 48 deletions(-)
diff --git a/.github/workflows/check-licensing.yml
b/.github/workflows/check-licensing.yml
index 101d331e90..774e91ddbe 100644
--- a/.github/workflows/check-licensing.yml
+++ b/.github/workflows/check-licensing.yml
@@ -51,4 +51,8 @@ jobs:
-Dexec.args="${{ env.MVN_BUILD_OUTPUT_FILE }} $(pwd) ${{
env.MVN_VALIDATION_DIR }}" \
-Dlog4j.configurationFile=file://$(pwd)/tools/ci/log4j.properties
env:
- MAVEN_OPTS: -Xmx4096m
\ No newline at end of file
+ MAVEN_OPTS: -Xmx4096m
+ - name: Check python licensing
+ run: |
+ chmod +x paimon-python/dev/check-licensing.sh
+ ./paimon-python/dev/check-licensing.sh
\ No newline at end of file
diff --git a/.github/workflows/paimon-python-checks.yml
b/.github/workflows/paimon-python-checks.yml
new file mode 100644
index 0000000000..e07b93e65c
--- /dev/null
+++ b/.github/workflows/paimon-python-checks.yml
@@ -0,0 +1,51 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+name: Check Code Style by Flake8 and Mypy
+
+on:
+ push:
+ pull_request:
+ paths-ignore:
+ - '**/*.md'
+
+env:
+ PYTHON_VERSION: "3.10"
+
+concurrency:
+ group: ${{ github.workflow }}-${{ github.event_name }}-${{
github.event.number || github.run_id }}
+ cancel-in-progress: true
+
+jobs:
+ lint-python:
+ runs-on: ubuntu-latest
+
+ steps:
+ - name: Checkout code
+ uses: actions/checkout@v2
+ - name: Set up Python
+ uses: actions/setup-python@v4
+ with:
+ python-version: ${{ env.PYTHON_VERSION }}
+ - name: Install dependencies
+ run: |
+ python -m pip install -q flake8==4.0.1 pytest~=7.0 requests 2>&1
>/dev/null
+ - name: Run lint-python.sh
+ run: |
+ chmod +x paimon-python/dev/lint-python.sh
+ ./paimon-python/dev/lint-python.sh
diff --git a/.gitignore b/.gitignore
index 2a8e3fb02b..c4c2d54fe9 100644
--- a/.gitignore
+++ b/.gitignore
@@ -19,7 +19,10 @@ target
.java-version
dependency-reduced-pom.xml
metastore_db/
-pypaimon/.idea/
+paimon-python/.idea/
+paimon-python/dist/
+paimon-python/*.egg-info/
+paimon-python/dev/log
### VS Code ###
.vscode/
diff --git a/docs/content/program-api/python-api.md
b/docs/content/program-api/python-api.md
index 80be8d4efc..691bcd5cdf 100644
--- a/docs/content/program-api/python-api.md
+++ b/docs/content/program-api/python-api.md
@@ -67,7 +67,7 @@ classpath via one of the following ways:
```python
import os
-from pypaimon.py4j import constants
+from pypaimon import constants
os.environ[constants.PYPAIMON_JAVA_CLASSPATH] = '/path/to/jars/*'
```
@@ -81,7 +81,7 @@ You can set JVM args via one of the following ways:
```python
import os
-from pypaimon.py4j import constants
+from pypaimon import constants
os.environ[constants.PYPAIMON_JVM_ARGS] = 'arg1 arg2 ...'
```
@@ -98,7 +98,7 @@ Otherwise, you should set hadoop classpath via one of the
following ways:
```python
import os
-from pypaimon.py4j import constants
+from pypaimon import constants
os.environ[constants.PYPAIMON_HADOOP_CLASSPATH] = '/path/to/jars/*'
```
@@ -111,7 +111,7 @@ If you just want to test codes in local, we recommend to
use [Flink Pre-bundled
Before coming into contact with the Table, you need to create a Catalog.
```python
-from pypaimon.py4j import Catalog
+from pypaimon import Catalog
# Note that keys and values are all string
catalog_options = {
diff --git a/paimon-python/README.md b/paimon-python/README.md
new file mode 100644
index 0000000000..4c2a464b1b
--- /dev/null
+++ b/paimon-python/README.md
@@ -0,0 +1,14 @@
+
+
+[](https://www.apache.org/licenses/LICENSE-2.0.html)
+
+# PyPaimon
+
+This PyPi package contains the Python APIs for using Paimon.
+# Build
+
+You can build the source package by executing the following command
+```commandline
+python setup.py sdist
+```
+The package is under `dist/`.
diff --git a/paimon-python/dev/.rat-excludes b/paimon-python/dev/.rat-excludes
new file mode 100644
index 0000000000..576371afae
--- /dev/null
+++ b/paimon-python/dev/.rat-excludes
@@ -0,0 +1,20 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+.gitignore
+rat-results.txt
diff --git a/paimon-python/dev/cfg.ini b/paimon-python/dev/cfg.ini
new file mode 100644
index 0000000000..195ed1c6cf
--- /dev/null
+++ b/paimon-python/dev/cfg.ini
@@ -0,0 +1,28 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+[flake8]
+# We follow PEP 8 (https://www.python.org/dev/peps/pep-0008/) with one
exception: lines can be
+# up to 100 characters in length, not 79.
+ignore=E226,E241,E305,E402,E722,E731,E741,W503,W504,F821
+max-line-length=120
+exclude=.tox/*,dev/*,build/*,dist/*
+
+[mypy]
+ignore_missing_imports = True
+strict_optional=False
diff --git a/paimon-python/dev/check-licensing.sh
b/paimon-python/dev/check-licensing.sh
new file mode 100755
index 0000000000..883c43d5a8
--- /dev/null
+++ b/paimon-python/dev/check-licensing.sh
@@ -0,0 +1,79 @@
+#!/usr/bin/env bash
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+download_rat_jar () {
+
URL="https://repo.maven.apache.org/maven2/org/apache/rat/apache-rat/${RAT_VERSION}/apache-rat-${RAT_VERSION}.jar";
+ JAR="$rat_jar"
+
+ # Download rat launch jar
+ echo "Attempting to fetch rat"
+ wget --quiet ${URL} -O "$JAR"
+}
+
+SOURCE_PACKAGE=${SOURCE_PACKAGE}
+
+export RAT_VERSION=0.15
+export rat_jar=rat/apache-rat-${RAT_VERSION}.jar
+
+if [ -z "${SOURCE_PACKAGE}" ]; then
+ BASE_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
+ PROJECT_ROOT="${BASE_DIR}/../"
+ cd ${PROJECT_ROOT}
+
+ # Sanity check to ensure that resolved paths are valid; a LICENSE file
should always exist in project root
+ if [ ! -f ${PROJECT_ROOT}/setup.py ]; then
+ echo "Project root path ${PROJECT_ROOT} is not valid; script may be in
the wrong directory."
+ exit 1
+ fi
+
+ RUN_RAT="java -jar ${rat_jar} -E ${PROJECT_ROOT}/dev/.rat-excludes -d
${PROJECT_ROOT}"
+else
+ EXTENSION='.tar.gz'
+ # get unzipped directory
+ PACKAGE_DIR="${SOURCE_PACKAGE:0:$((${#SOURCE_PACKAGE} - ${#EXTENSION}))}"
+ tar -xf ${SOURCE_PACKAGE}
+
+ RUN_RAT="java -jar ${rat_jar} -e PKG-INFO -e setup.cfg -e
pypaimon.egg-info/* -d ${PACKAGE_DIR}"
+fi
+
+mkdir -p rat
+download_rat_jar
+
+$RUN_RAT > rat/rat-results.txt
+
+if [ $? -ne 0 ]; then
+ echo "RAT exited abnormally"
+ exit 1
+fi
+
+ERRORS="$(cat rat/rat-results.txt | grep -e "??")"
+
+# clean
+rm -rf rat
+if [ -d "$PACKAGE_DIR" ]; then
+ rm -rf $PACKAGE_DIR
+fi
+
+if [[ -n "${ERRORS}" ]]; then
+ echo "Could not find Apache license headers in the following files:"
+ echo ${ERRORS}
+ exit 1
+else
+ echo -e "RAT checks passed."
+fi
diff --git a/paimon-python/dev/lint-python.sh b/paimon-python/dev/lint-python.sh
new file mode 100755
index 0000000000..02ca321161
--- /dev/null
+++ b/paimon-python/dev/lint-python.sh
@@ -0,0 +1,275 @@
+#!/usr/bin/env bash
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+# lint-python.sh
+# This script will prepare a virtual environment for many kinds of checks,
such as tox check, flake8 check.
+#
+
+# Printing infos both in log and console
+function print_function() {
+ local STAGE_LENGTH=48
+ local left_edge_len=
+ local right_edge_len=
+ local str
+ case "$1" in
+ "STAGE")
+ left_edge_len=$(((STAGE_LENGTH-${#2})/2))
+ right_edge_len=$((STAGE_LENGTH-${#2}-left_edge_len))
+ str="$(seq -s "=" $left_edge_len | tr -d "[:digit:]")""$2""$(seq
-s "=" $right_edge_len | tr -d "[:digit:]")"
+ ;;
+ "STEP")
+ str="$2"
+ ;;
+ *)
+ str="seq -s "=" $STAGE_LENGTH | tr -d "[:digit:]""
+ ;;
+ esac
+ echo $str | tee -a $LOG_FILE
+}
+
+function regexp_match() {
+ if echo $1 | grep -e $2 &>/dev/null; then
+ echo true
+ else
+ echo false
+ fi
+}
+
+# decide whether a array contains a specified element.
+function contains_element() {
+ arr=($1)
+ if echo "${arr[@]}" | grep -w "$2" &>/dev/null; then
+ echo true
+ else
+ echo false
+ fi
+}
+
+# create dir if needed
+function create_dir() {
+ if [ ! -d $1 ]; then
+ mkdir -p $1
+ if [ $? -ne 0 ]; then
+ echo "mkdir -p $1 failed. you can mkdir manually or exec the
script with \
+ the command: sudo ./lint-python.sh"
+ exit 1
+ fi
+ fi
+}
+
+# Collect checks
+function collect_checks() {
+ if [ ! -z "$EXCLUDE_CHECKS" ] && [ ! -z "$INCLUDE_CHECKS" ]; then
+ echo "You can't use option -s and -e simultaneously."
+ exit 1
+ fi
+ if [ ! -z "$EXCLUDE_CHECKS" ]; then
+ for (( i = 0; i < ${#EXCLUDE_CHECKS[@]}; i++)); do
+ if [[ `contains_element "${SUPPORT_CHECKS[*]}"
"${EXCLUDE_CHECKS[i]}_check"` = true ]]; then
+
SUPPORT_CHECKS=("${SUPPORT_CHECKS[@]/${EXCLUDE_CHECKS[i]}_check}")
+ else
+ echo "the check ${EXCLUDE_CHECKS[i]} is invalid."
+ exit 1
+ fi
+ done
+ fi
+ if [ ! -z "$INCLUDE_CHECKS" ]; then
+ REAL_SUPPORT_CHECKS=()
+ for (( i = 0; i < ${#INCLUDE_CHECKS[@]}; i++)); do
+ if [[ `contains_element "${SUPPORT_CHECKS[*]}"
"${INCLUDE_CHECKS[i]}_check"` = true ]]; then
+ REAL_SUPPORT_CHECKS+=("${INCLUDE_CHECKS[i]}_check")
+ else
+ echo "the check ${INCLUDE_CHECKS[i]} is invalid."
+ exit 1
+ fi
+ done
+ SUPPORT_CHECKS=(${REAL_SUPPORT_CHECKS[@]})
+ fi
+}
+
+# get all supported checks functions
+function get_all_supported_checks() {
+ _OLD_IFS=$IFS
+ IFS=$'\n'
+ SUPPORT_CHECKS=()
+ for fun in $(declare -F); do
+ if [[ `regexp_match "$fun" "_check$"` = true ]]; then
+ SUPPORT_CHECKS+=("${fun:11}")
+ fi
+ done
+ IFS=$_OLD_IFS
+}
+
+# exec all selected check stages
+function check_stage() {
+ print_function "STAGE" "checks starting"
+ for fun in ${SUPPORT_CHECKS[@]}; do
+ $fun
+ done
+ echo "All the checks are finished, the detailed information can be found
in: $LOG_FILE"
+}
+
+
+###############################################################All Checks
Definitions###############################################################
+#########################
+# This part defines all check functions such as tox_check and flake8_check
+# We make a rule that all check functions are suffixed with _ check. e.g.
tox_check, flake8_check
+#########################
+
+# Flake8 check
+function flake8_check() {
+ local PYTHON_SOURCE="$(find . \( -path ./dev -o -path ./.tox \) -prune -o
-type f -name "*.py" -print )"
+
+ print_function "STAGE" "flake8 checks"
+ if [ ! -f "$FLAKE8_PATH" ]; then
+ echo "For some unknown reasons, the flake8 package is not complete."
+ fi
+
+ if [[ ! "$PYTHON_SOURCE" ]]; then
+ echo "No python files found! Something is wrong exiting."
+ exit 1;
+ fi
+
+ # the return value of a pipeline is the status of the last command to exit
+ # with a non-zero status or zero if no command exited with a non-zero
status
+ set -o pipefail
+ ($FLAKE8_PATH --config=./dev/cfg.ini $PYTHON_SOURCE) 2>&1 | tee -a
$LOG_FILE
+
+ PYCODESTYLE_STATUS=$?
+ if [ $PYCODESTYLE_STATUS -ne 0 ]; then
+ print_function "STAGE" "flake8 checks... [FAILED]"
+ # Stop the running script.
+ exit 1;
+ else
+ print_function "STAGE" "flake8 checks... [SUCCESS]"
+ fi
+}
+
+# Pytest check
+function pytest_check() {
+
+ print_function "STAGE" "pytest checks"
+ if [ ! -f "$PYTEST_PATH" ]; then
+ echo "For some unknown reasons, the pytest package is not complete,\
+ you should exec the script with the parameter: -f"
+ fi
+
+ # the return value of a pipeline is the status of the last command to exit
+ # with a non-zero status or zero if no command exited with a non-zero
status
+ set -o pipefail
+ ($PYTEST_PATH) 2>&1 | tee -a $LOG_FILE
+
+ PYCODESTYLE_STATUS=$?
+ if [ $PYCODESTYLE_STATUS -ne 0 ]; then
+ print_function "STAGE" "pytest checks... [FAILED]"
+ # Stop the running script.
+ exit 1;
+ else
+ print_function "STAGE" "pytest checks... [SUCCESS]"
+ fi
+}
+###############################################################All Checks
Definitions###############################################################
+
+# CURRENT_DIR is "paimon-python/"
+SCRIPT_PATH="$(readlink -f "$0")"
+cd "$(dirname "$SCRIPT_PATH")/.." || exit
+CURRENT_DIR="$(pwd)"
+echo ${CURRENT_DIR}
+
+# flake8 path
+FLAKE8_PATH="$(which flake8)"
+echo $FLAKE8_PATH
+# mypy path
+MYPY_PATH="$(which mypy)"
+echo $MYPY_PATH
+# pytest path
+PYTEST_PATH="$(which pytest)"
+echo $PYTEST_PATH
+
+LOG_DIR=$CURRENT_DIR/dev/log
+
+if [ "$PAIMON_IDENT_STRING" == "" ]; then
+ PAIMON_IDENT_STRING="$USER"
+fi
+if [ "$HOSTNAME" == "" ]; then
+ HOSTNAME="$HOST"
+fi
+
+# the log file stores the checking result.
+LOG_FILE=$LOG_DIR/paimon-$PAIMON_IDENT_STRING-python-$HOSTNAME.log
+create_dir $LOG_DIR
+
+# clean LOG_FILE content
+echo >$LOG_FILE
+
+SUPPORT_CHECKS=()
+
+# search all supported check functions and put them into SUPPORT_CHECKS array
+get_all_supported_checks
+
+EXCLUDE_CHECKS=""
+
+INCLUDE_CHECKS=""
+
+# parse_opts
+USAGE="
+usage: $0 [options]
+-h print this help message and exit
+-e [tox,flake8,sphinx,mypy]
+ exclude checks which split by comma(,)
+-i [tox,flake8,sphinx,mypy]
+ include checks which split by comma(,)
+-l list all checks supported.
+Examples:
+ ./lint-python.sh => exec all checks.
+ ./lint-python.sh -e tox,flake8 => exclude checks tox,flake8.
+ ./lint-python.sh -i flake8 => include checks flake8.
+ ./lint-python.sh -l => list all checks supported.
+"
+while getopts "hfs:i:e:lr" arg; do
+ case "$arg" in
+ h)
+ printf "%s\\n" "$USAGE"
+ exit 2
+ ;;
+ e)
+ EXCLUDE_CHECKS=($(echo $OPTARG | tr ',' ' ' ))
+ ;;
+ i)
+ INCLUDE_CHECKS=($(echo $OPTARG | tr ',' ' ' ))
+ ;;
+ l)
+ printf "current supported checks includes:\n"
+ for fun in ${SUPPORT_CHECKS[@]}; do
+ echo ${fun%%_check*}
+ done
+ exit 2
+ ;;
+ ?)
+ printf "ERROR: did not recognize option '%s', please try -h\\n"
"$1"
+ exit 1
+ ;;
+ esac
+done
+
+# collect checks according to the options
+collect_checks
+
+check_stage
+
diff --git a/pypaimon/__init__.py b/paimon-python/pypaimon/__init__.py
similarity index 100%
rename from pypaimon/__init__.py
rename to paimon-python/pypaimon/__init__.py
diff --git a/pypaimon/api/__init__.py b/paimon-python/pypaimon/api/__init__.py
similarity index 96%
rename from pypaimon/api/__init__.py
rename to paimon-python/pypaimon/api/__init__.py
index a82d9ebb10..ef17c2ad7b 100644
--- a/pypaimon/api/__init__.py
+++ b/paimon-python/pypaimon/api/__init__.py
@@ -18,14 +18,14 @@
import logging
from typing import Dict, List, Optional, Callable
from urllib.parse import unquote
-from api.auth import RESTAuthFunction
-from api.api_response import PagedList, GetTableResponse,
ListDatabasesResponse, ListTablesResponse, \
+from .auth import RESTAuthFunction
+from .api_response import PagedList, GetTableResponse, ListDatabasesResponse,
ListTablesResponse, \
GetDatabaseResponse, ConfigResponse, PagedResponse
-from api.api_resquest import CreateDatabaseRequest, AlterDatabaseRequest
-from api.typedef import Identifier
-from api.client import HttpClient
-from api.auth import DLFAuthProvider, DLFToken
-from api.typedef import T
+from .api_resquest import CreateDatabaseRequest, AlterDatabaseRequest
+from .typedef import Identifier
+from .client import HttpClient
+from .auth import DLFAuthProvider, DLFToken
+from .typedef import T
class RESTCatalogOptions:
diff --git a/pypaimon/api/api_response.py
b/paimon-python/pypaimon/api/api_response.py
similarity index 97%
rename from pypaimon/api/api_response.py
rename to paimon-python/pypaimon/api/api_response.py
index 6005e11810..e93af5ae05 100644
--- a/pypaimon/api/api_response.py
+++ b/paimon-python/pypaimon/api/api_response.py
@@ -17,12 +17,12 @@ limitations under the License.
"""
from abc import ABC, abstractmethod
-from typing import Dict, Optional, Any, Generic, List
-from dataclasses import dataclass, field
+from typing import Dict, Optional, Generic, List
+from dataclasses import dataclass
-from api.rest_json import json_field
-from api.typedef import T
-from api.data_types import DataField
+from .rest_json import json_field
+from .typedef import T
+from .data_types import DataField
@dataclass
diff --git a/pypaimon/api/api_resquest.py
b/paimon-python/pypaimon/api/api_resquest.py
similarity index 97%
rename from pypaimon/api/api_resquest.py
rename to paimon-python/pypaimon/api/api_resquest.py
index 1434176849..8e7d14e418 100644
--- a/pypaimon/api/api_resquest.py
+++ b/paimon-python/pypaimon/api/api_resquest.py
@@ -20,7 +20,7 @@ from abc import ABC
from dataclasses import dataclass
from typing import Dict, List
-from api.rest_json import json_field
+from .rest_json import json_field
class RESTRequest(ABC):
diff --git a/pypaimon/api/auth.py b/paimon-python/pypaimon/api/auth.py
similarity index 97%
rename from pypaimon/api/auth.py
rename to paimon-python/pypaimon/api/auth.py
index aabefe2d49..ddeca5e5e6 100644
--- a/pypaimon/api/auth.py
+++ b/paimon-python/pypaimon/api/auth.py
@@ -42,7 +42,7 @@ class DLFToken:
security_token: Optional[str] = None
def __init__(self, options: Dict[str, str]):
- from api import RESTCatalogOptions
+ from . import RESTCatalogOptions
self.access_key_id = options.get(RESTCatalogOptions.DLF_ACCESS_KEY_ID)
self.access_key_secret =
options.get(RESTCatalogOptions.DLF_ACCESS_KEY_SECRET)
self.security_token =
options.get(RESTCatalogOptions.DLF_ACCESS_SECURITY_TOKEN)
@@ -186,7 +186,8 @@ class DLFAuthSignature:
result = cls._hmac_sha256(signing_key, string_to_sign)
signature = cls._hex_encode(result)
- credential = f"{cls.SIGNATURE_ALGORITHM}
Credential={dlf_token.access_key_id}/{date}/{region}/{cls.PRODUCT}/{cls.REQUEST_TYPE}"
+ credential = (f"{cls.SIGNATURE_ALGORITHM} "
+
f"Credential={dlf_token.access_key_id}/{date}/{region}/{cls.PRODUCT}/{cls.REQUEST_TYPE}")
signature_part = f"{cls.SIGNATURE_KEY}={signature}"
return f"{credential},{signature_part}"
diff --git a/pypaimon/api/client.py b/paimon-python/pypaimon/api/client.py
similarity index 98%
rename from pypaimon/api/client.py
rename to paimon-python/pypaimon/api/client.py
index 8c4b5b85e0..a7c69cb31d 100644
--- a/pypaimon/api/client.py
+++ b/paimon-python/pypaimon/api/client.py
@@ -27,9 +27,9 @@ import requests
from requests.adapters import HTTPAdapter
from urllib3 import Retry
-from api.auth import RESTAuthParameter
-from api.api_response import ErrorResponse
-from api.rest_json import JSON
+from .auth import RESTAuthParameter
+from .api_response import ErrorResponse
+from .rest_json import JSON
T = TypeVar('T', bound='RESTResponse')
@@ -75,11 +75,6 @@ class BadRequestException(RESTException):
super().__init__(message, *args)
-class BadRequestException(RESTException):
- """Exception for 400 Bad Request"""
- pass
-
-
class NotAuthorizedException(RESTException):
"""Exception for not authorized (401)"""
diff --git a/pypaimon/api/data_types.py
b/paimon-python/pypaimon/api/data_types.py
similarity index 100%
rename from pypaimon/api/data_types.py
rename to paimon-python/pypaimon/api/data_types.py
diff --git a/pypaimon/api/rest_json.py b/paimon-python/pypaimon/api/rest_json.py
similarity index 99%
rename from pypaimon/api/rest_json.py
rename to paimon-python/pypaimon/api/rest_json.py
index c6a61449d2..c416c1c680 100644
--- a/pypaimon/api/rest_json.py
+++ b/paimon-python/pypaimon/api/rest_json.py
@@ -19,7 +19,7 @@ import json
from dataclasses import field, fields, is_dataclass
from typing import Any, Type, Dict
-from api.typedef import T
+from .typedef import T
def json_field(json_name: str, **kwargs):
diff --git a/pypaimon/api/typedef.py b/paimon-python/pypaimon/api/typedef.py
similarity index 100%
rename from pypaimon/api/typedef.py
rename to paimon-python/pypaimon/api/typedef.py
diff --git a/pypaimon/tests/__init__.py
b/paimon-python/pypaimon/tests/__init__.py
similarity index 100%
rename from pypaimon/tests/__init__.py
rename to paimon-python/pypaimon/tests/__init__.py
diff --git a/pypaimon/tests/api_test.py
b/paimon-python/pypaimon/tests/api_test.py
similarity index 97%
rename from pypaimon/tests/api_test.py
rename to paimon-python/pypaimon/tests/api_test.py
index 74a5c44436..0fcbcb420d 100644
--- a/pypaimon/tests/api_test.py
+++ b/paimon-python/pypaimon/tests/api_test.py
@@ -25,13 +25,13 @@ from http.server import HTTPServer, BaseHTTPRequestHandler
from urllib.parse import urlparse
import unittest
-import api
-from api.api_response import (ConfigResponse, ListDatabasesResponse,
GetDatabaseResponse, TableMetadata, Schema,
- GetTableResponse, ListTablesResponse,
TableSchema, RESTResponse, PagedList, DataField)
-from api import RESTApi
-from api.rest_json import JSON
-from api.typedef import Identifier
-from api.data_types import AtomicInteger, DataTypeParser, AtomicType,
ArrayType, MapType, RowType
+import pypaimon.api as api
+from ..api.api_response import (ConfigResponse, ListDatabasesResponse,
GetDatabaseResponse, TableMetadata, Schema,
+ GetTableResponse, ListTablesResponse,
TableSchema, RESTResponse, PagedList, DataField)
+from ..api import RESTApi
+from ..api.rest_json import JSON
+from ..api.typedef import Identifier
+from ..api.data_types import AtomicInteger, DataTypeParser, AtomicType,
ArrayType, MapType, RowType
@dataclass
@@ -67,7 +67,7 @@ class ResourcePaths:
self.prefix = prefix.rstrip('/')
def config(self) -> str:
- return f"/v1/config"
+ return "/v1/config"
def databases(self) -> str:
return f"/v1/{self.prefix}/databases"
@@ -464,8 +464,6 @@ class RESTCatalogServer:
# Basic table operations
return self._table_handle(method, data, identifier)
- elif len(path_parts) >= 4:
- operation = path_parts[3]
return self._mock_response(ErrorResponse(None, None, "Not Found",
404), 404)
def _databases_api_handler(self, method: str, data: str,
@@ -525,7 +523,7 @@ class RESTCatalogServer:
schema = table_metadata.schema.to_schema()
path = schema.options.pop(PATH, None)
- response = self.mock_table(identifier, table_metadata, path,
schema);
+ response = self.mock_table(identifier, table_metadata, path,
schema)
return self._mock_response(response, 200)
#
# elif method == "POST":
@@ -861,13 +859,14 @@ class ApiTestCase(unittest.TestCase):
"prod_db": server.mock_database("prod_db", {"env": "prod"})
}
data_fields = [
- DataField( 0, "name", AtomicType('INT'), 'desc name'),
- DataField( 1, "arr11", ArrayType(True, AtomicType('INT')),
'desc arr11'),
- DataField( 2, "map11", MapType(False, AtomicType('INT'),
MapType(False, AtomicType('INT'), AtomicType('INT'))), 'desc arr11'),
+ DataField(0, "name", AtomicType('INT'), 'desc name'),
+ DataField(1, "arr11", ArrayType(True, AtomicType('INT')),
'desc arr11'),
+ DataField(2, "map11", MapType(False, AtomicType('INT'),
+ MapType(False,
AtomicType('INT'), AtomicType('INT'))), 'desc arr11'),
]
schema = TableSchema(len(data_fields), data_fields,
len(data_fields), [], [], {}, "")
test_tables = {
- "default.user": TableMetadata(uuid=str(uuid.uuid4()),
is_external=True,schema=schema),
+ "default.user": TableMetadata(uuid=str(uuid.uuid4()),
is_external=True, schema=schema),
}
server.table_metadata_store.update(test_tables)
server.database_store.update(test_databases)
diff --git a/paimon-python/setup.py b/paimon-python/setup.py
new file mode 100644
index 0000000000..d1ec439905
--- /dev/null
+++ b/paimon-python/setup.py
@@ -0,0 +1,50 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+from setuptools import setup, find_packages
+
+VERSION = "0.3.dev" # noqa
+
+PACKAGES = find_packages(include=['pypaimon*'])
+
+install_requires = []
+
+long_description = 'See Apache Paimon Python API \
+[Doc](https://paimon.apache.org/docs/master/program-api/python-api/) for
usage.'
+
+setup(
+ name='pypaimon',
+ version=VERSION,
+ packages=PACKAGES,
+ include_package_data=True,
+ install_requires=install_requires,
+ extras_require={},
+ description='Apache Paimon Python API',
+ long_description=long_description,
+ long_description_content_type='text/markdown',
+ author='Apache Software Foundation',
+ author_email='[email protected]',
+ url='https://paimon.apache.org',
+ classifiers=[
+ 'Development Status :: 5 - Production/Stable',
+ 'License :: OSI Approved :: Apache Software License',
+ 'Programming Language :: Python :: 3.8',
+ 'Programming Language :: Python :: 3.9',
+ 'Programming Language :: Python :: 3.10',
+ 'Programming Language :: Python :: 3.11'],
+ python_requires='>=3.8'
+)
diff --git a/pypaimon/README.md b/pypaimon/README.md
deleted file mode 100644
index ee90028fa1..0000000000
--- a/pypaimon/README.md
+++ /dev/null
@@ -1,3 +0,0 @@
-# PyPaimon
-
-This PyPi package contains the Python APIs for using Paimon.
