FYI: Orhan and I worked together last week to identify the RCA and fixes for the Postgres issues. Using his dev branch (test-pg15), I am able to confirm I was able to get the install, install-check, & dev-check passing in the following environments:
Ubuntu 22.04 (Per Orhan's instructions, I used pyxb-x to support Python 3.10): - PG 15 - passed - PG 13 - passed Rocky 9: - PG 15 - passed - PG 13 - passed Rocky 8: - PG 15 - passed - PG 13 - passed - Greenplum 6.25.1 - passed - Greenplum 7 Beta5 - passed Note(s): - PG 14 cannot be installed. I wasn't able to verify the PG 14 installation issue is fixed. Orhan confirmed he knows the issue and will include it in RC2. (Orhan's note: "Fixed PG 14 install as well. There was a fix for PG 15 to handle the anyarray -> anycompatiblearray change but it ignored PG 14. Now it should handle all 3 PG versions.) - Although I was unsuccessful at testing RC1 with PG 12, Orhan confirmed PG 12 is not supported on MADlib 2.1.0 (RC1). - For reference, I have attached my personal notes for the RC1 validation process. On Thu, Aug 31, 2023 at 1:52 AM Orhan Kislal <okis...@vmware.com> wrote: > Dear MADlib Community, > > The vote for MADlib 2.1.0 RC1 has failed because of a -1 binding vote. We > will address the issues mentioned during the voting and begin another vote > for RC2 shortly. > > Thanks, > > Orhan Kislal > -- Ed Espino Apache MADlib
Edward Espino Table of Contents _________________ 1. Test Systems .. 1. Ubuntu 22.04 ..... 1. eespino-1-ubuntu22 ..... 2. eespino-2-ubuntu22 .. 2. Rocky 8 ..... 1. eespino-1-rocky8 ..... 2. eespino-2-rocky8 .. 3. Rocky 9 2. Details .. 1. Prereqs & Helpers ..... 1. Ubuntu 22.04 ..... 2. Rocky 8 ..... 3. Rocky 9 ..... 4. All platforms .. 2. Postgres ..... 1. PG 15.4 ..... 2. PG 14.9 ..... 3. PG 13.12 ..... 4. PG 12.16 ..... 5. PG Retrieve Source & Build .. 3. Greenplum - Install RPM and Environment config ..... 1. Install RPM - copied to test environment with gcloud scp command ..... 2. Setup cluster ..... 3. greenplum_path & MASTER_DATA_DIRECTORY .. 4. Downloads, Check PGP signatures & checksums .. 5. Ubuntu 22.04 .. 6. Rocky Linux 8.8 .. 7. Rocky Linux 9.2 .. 8. Reproduction ..... 1. Ubuntu 22.04 ..... 2. Rocky 8 1 Test Systems ============== 1.1 Ubuntu 22.04 ~~~~~~~~~~~~~~~~ 1.1.1 eespino-1-ubuntu22 ------------------------ ,---- | gcloud compute instances create eespino-1-ubuntu22 \ | --project=data-gpdb-server \ | --zone=us-west1-b \ | --machine-type=e2-standard-32 \ | --network-interface=network-tier=PREMIUM,stack-type=IPV4_ONLY,subnet=default \ | --maintenance-policy=MIGRATE \ | --provisioning-model=STANDARD \ | --service-account=612089704425-comp...@developer.gserviceaccount.com \ | --scopes=https://www.googleapis.com/auth/devstorage.read_only,https://www.googleapis.com/auth/logging.write,https://www.googleapis.com/auth/monitoring.write,https://www.googleapis.com/auth/servicecontrol,https://www.googleapis.com/auth/service.management.readonly,https://www.googleapis.com/auth/trace.append \ | --create-disk=auto-delete=yes,boot=yes,device-name=instance-1,image=projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727,mode=rw,size=100,type=projects/data-gpdb-server/zones/us-central1-a/diskTypes/pd-balanced \ | --no-shielded-secure-boot \ | --shielded-vtpm \ | --shielded-integrity-monitoring \ | --labels=goog-ec-src=vm_add-gcloud \ | --reservation-affinity=any `---- ,---- | gcloud compute ssh --zone "us-west1-b" "eespino-1-ubuntu22" --project "data-gpdb-server" | gcloud compute instances list --filter="name=('eespino-1-ubuntu22')" --project "data-gpdb-server" | gcloud compute instances suspend --zone "us-west1-b" "eespino-1-ubuntu22" --project "data-gpdb-server" --quiet | gcloud compute instances resume --zone "us-west1-b" "eespino-1-ubuntu22" --project "data-gpdb-server" --quiet | #gcloud compute instances delete --zone "us-west1-b" "eespino-1-ubuntu22" --project "data-gpdb-server" --quiet `---- 1.1.2 eespino-2-ubuntu22 ------------------------ ,---- | gcloud compute instances create eespino-2-ubuntu22 \ | --project=data-gpdb-server \ | --zone=us-west1-b \ | --machine-type=e2-standard-32 \ | --network-interface=network-tier=PREMIUM,stack-type=IPV4_ONLY,subnet=default \ | --maintenance-policy=MIGRATE \ | --provisioning-model=STANDARD \ | --service-account=612089704425-comp...@developer.gserviceaccount.com \ | --scopes=https://www.googleapis.com/auth/devstorage.read_only,https://www.googleapis.com/auth/logging.write,https://www.googleapis.com/auth/monitoring.write,https://www.googleapis.com/auth/servicecontrol,https://www.googleapis.com/auth/service.management.readonly,https://www.googleapis.com/auth/trace.append \ | --create-disk=auto-delete=yes,boot=yes,device-name=instance-1,image=projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727,mode=rw,size=100,type=projects/data-gpdb-server/zones/us-central1-a/diskTypes/pd-balanced \ | --no-shielded-secure-boot \ | --shielded-vtpm \ | --shielded-integrity-monitoring \ | --labels=goog-ec-src=vm_add-gcloud \ | --reservation-affinity=any `---- ,---- | gcloud compute ssh --zone "us-west1-b" "eespino-2-ubuntu22" --project "data-gpdb-server" | gcloud compute instances list --filter="name=('eespino-2-ubuntu22')" --project "data-gpdb-server" | gcloud compute instances suspend --zone "us-west1-b" "eespino-2-ubuntu22" --project "data-gpdb-server" --quiet | gcloud compute instances resume --zone "us-west1-b" "eespino-2-ubuntu22" --project "data-gpdb-server" --quiet | #gcloud compute instances delete --zone "us-west1-b" "eespino-2-ubuntu22" --project "data-gpdb-server" --quiet `---- 1.2 Rocky 8 ~~~~~~~~~~~ 1.2.1 eespino-1-rocky8 ---------------------- ,---- | gcloud compute instances create eespino-1-rocky8 \ | --project=data-gpdb-server \ | --zone=us-west1-b \ | --machine-type=e2-standard-32 \ | --network-interface=network-tier=PREMIUM,stack-type=IPV4_ONLY,subnet=default \ | --maintenance-policy=MIGRATE \ | --provisioning-model=STANDARD \ | --service-account=612089704425-comp...@developer.gserviceaccount.com \ | --scopes=https://www.googleapis.com/auth/devstorage.read_only,https://www.googleapis.com/auth/logging.write,https://www.googleapis.com/auth/monitoring.write,https://www.googleapis.com/auth/servicecontrol,https://www.googleapis.com/auth/service.management.readonly,https://www.googleapis.com/auth/trace.append \ | --create-disk=auto-delete=yes,boot=yes,device-name=instance-1,image=projects/rocky-linux-cloud/global/images/rocky-linux-8-optimized-gcp-v20230809,mode=rw,size=100,type=projects/data-gpdb-server/zones/us-central1-a/diskTypes/pd-balanced \ | --no-shielded-secure-boot \ | --shielded-vtpm \ | --shielded-integrity-monitoring \ | --labels=goog-ec-src=vm_add-gcloud \ | --reservation-affinity=any `---- ,---- | gcloud compute scp --zone "us-west1-b" --project "data-gpdb-server" ~/workspace/apache/greenplum-db-6.25.1-rhel8-x86_64.rpm "eespino-1-rocky8:" | gcloud compute scp --zone "us-west1-b" --project "data-gpdb-server" ~/workspace/apache/greenplum-db-7.0.0-beta.5-el8-x86_64.rpm "eespino-1-rocky8:" | | gcloud compute ssh --zone "us-west1-b" "eespino-1-rocky8" --project "data-gpdb-server" | gcloud compute instances list --filter="name=('eespino-1-rocky8')" --project "data-gpdb-server" | gcloud compute instances suspend --zone "us-west1-b" "eespino-1-rocky8" --project "data-gpdb-server" --quiet | gcloud compute instances resume --zone "us-west1-b" "eespino-1-rocky8" --project "data-gpdb-server" --quiet | #gcloud compute instances delete --zone "us-west1-b" "eespino-1-rocky8" --project "data-gpdb-server" --quiet `---- 1.2.2 eespino-2-rocky8 ---------------------- ,---- | gcloud compute instances create eespino-2-rocky8 \ | --project=data-gpdb-server \ | --zone=us-west1-b \ | --machine-type=e2-standard-32 \ | --network-interface=network-tier=PREMIUM,stack-type=IPV4_ONLY,subnet=default \ | --maintenance-policy=MIGRATE \ | --provisioning-model=STANDARD \ | --service-account=612089704425-comp...@developer.gserviceaccount.com \ | --scopes=https://www.googleapis.com/auth/devstorage.read_only,https://www.googleapis.com/auth/logging.write,https://www.googleapis.com/auth/monitoring.write,https://www.googleapis.com/auth/servicecontrol,https://www.googleapis.com/auth/service.management.readonly,https://www.googleapis.com/auth/trace.append \ | --create-disk=auto-delete=yes,boot=yes,device-name=instance-1,image=projects/rocky-linux-cloud/global/images/rocky-linux-8-optimized-gcp-v20230809,mode=rw,size=100,type=projects/data-gpdb-server/zones/us-central1-a/diskTypes/pd-balanced \ | --no-shielded-secure-boot \ | --shielded-vtpm \ | --shielded-integrity-monitoring \ | --labels=goog-ec-src=vm_add-gcloud \ | --reservation-affinity=any `---- ,---- | gcloud compute scp --zone "us-west1-b" --project "data-gpdb-server" ~/workspace/apache/greenplum-db-6.25.1-rhel8-x86_64.rpm "eespino-2-rocky8:" | gcloud compute scp --zone "us-west1-b" --project "data-gpdb-server" ~/workspace/apache/greenplum-db-7.0.0-beta.5-el8-x86_64.rpm "eespino-2-rocky8:" | | gcloud compute ssh --zone "us-west1-b" "eespino-2-rocky8" --project "data-gpdb-server" | gcloud compute instances list --filter="name=('eespino-2-rocky8')" --project "data-gpdb-server" | gcloud compute instances suspend --zone "us-west1-b" "eespino-2-rocky8" --project "data-gpdb-server" --quiet | gcloud compute instances resume --zone "us-west1-b" "eespino-2-rocky8" --project "data-gpdb-server" --quiet | #gcloud compute instances delete --zone "us-west1-b" "eespino-2-rocky8" --project "data-gpdb-server" --quiet `---- 1.3 Rocky 9 ~~~~~~~~~~~ ,---- | gcloud compute instances create eespino-1-rocky9 \ | --project=data-gpdb-server \ | --zone=us-west1-b \ | --machine-type=e2-standard-32 \ | --network-interface=network-tier=PREMIUM,stack-type=IPV4_ONLY,subnet=default \ | --maintenance-policy=MIGRATE \ | --provisioning-model=STANDARD \ | --service-account=612089704425-comp...@developer.gserviceaccount.com \ | --scopes=https://www.googleapis.com/auth/devstorage.read_only,https://www.googleapis.com/auth/logging.write,https://www.googleapis.com/auth/monitoring.write,https://www.googleapis.com/auth/servicecontrol,https://www.googleapis.com/auth/service.management.readonly,https://www.googleapis.com/auth/trace.append \ | --create-disk=auto-delete=yes,boot=yes,device-name=instance-1,image=projects/rocky-linux-cloud/global/images/rocky-linux-9-optimized-gcp-v20230809,mode=rw,size=100,type=projects/data-gpdb-server/zones/us-west1-b/diskTypes/pd-balanced \ | --no-shielded-secure-boot \ | --shielded-vtpm \ | --shielded-integrity-monitoring \ | --labels=goog-ec-src=vm_add-gcloud \ | --reservation-affinity=any `---- ,---- | gcloud compute ssh --zone "us-west1-b" "eespino-1-rocky9" --project "data-gpdb-server" | gcloud compute instances list --filter="name=('eespino-1-rocky9')" --project "data-gpdb-server" | gcloud compute instances suspend --zone "us-west1-b" "eespino-1-rocky9" --project "data-gpdb-server" --quiet | gcloud compute instances resume --zone "us-west1-b" "eespino-1-rocky9" --project "data-gpdb-server" --quiet | #gcloud compute instances delete --zone "us-west1-b" "eespino-1-rocky9" --project "data-gpdb-server" --quiet `---- 2 Details ========= 2.1 Prereqs & Helpers ~~~~~~~~~~~~~~~~~~~~~ 2.1.1 Ubuntu 22.04 ------------------ ,---- | sudo apt-get update | sudo NEEDRESTART_SUSPEND=1 apt-get install -qq -y wget cmake gcc g++ flex bison patch libreadline-dev zlib1g-dev libxml2-dev libpython3-dev m4 bat python3-pip `---- 2.1.2 Rocky 8 ------------- ,---- | sudo dnf -d0 -y install wget cmake gcc gcc-c++ m4 flex bison patch readline-devel zlib-devel platform-python-devel libxml2-devel tmux python39 python39-devel python39-pip git | sudo alternatives --config python3 | sudo dnf -d0 -y install yum-utils epel-release | sudo yum-config-manager --disable epel | yum repolist | sudo dnf -d0 -y install --enablerepo=epel the_silver_searcher | | wget -nv -P $HOME https://github.com/sharkdp/bat/releases/download/v0.23.0/bat-v0.23.0-x86_64-unknown-linux-gnu.tar.gz | tar xf $HOME/bat-v0.23.0-x86_64-unknown-linux-gnu.tar.gz -C $HOME | sudo cp $HOME/bat-v0.23.0-x86_64-unknown-linux-gnu/bat /usr/local/bin `---- 2.1.3 Rocky 9 ------------- ,---- | sudo dnf -d0 -y install wget cmake gcc g++ m4 flex bison patch readline-devel zlib-devel python-devel libxml2-devel tmux git | sudo dnf -d0 -y install yum-utils epel-release | sudo yum-config-manager --disable epel | yum repolist | sudo dnf -d0 -y install --enablerepo=epel the_silver_searcher bat `---- 2.1.4 All platforms ------------------- ,---- | wget -nv https://gist.githubusercontent.com/simonista/8703722/raw/d08f2b4dc10452b97d3ca15386e9eed457a53c61/.vimrc -O ~/.vimrc | wget -nv https://raw.githubusercontent.com/tony/tmux-config/master/.tmux.conf -O ~/.tmux.conf | curl -sfL https://direnv.net/install.sh | bash `---- 2.2 Postgres ~~~~~~~~~~~~ 2.2.1 PG 15.4 ------------- ,---- | export PGVER=15.4 | export PATH=$HOME/pg${PGVER}-bin/bin:$PATH | export PGDATA=$HOME/pg${PGVER}-data `---- 2.2.2 PG 14.9 ------------- ,---- | export PGVER=14.9 | export PATH=$HOME/pg${PGVER}-bin/bin:$PATH | export PGDATA=$HOME/pg${PGVER}-data `---- 2.2.3 PG 13.12 -------------- ,---- | export PGVER=13.12 | export PATH=$HOME/pg${PGVER}-bin/bin:$PATH | export PGDATA=$HOME/pg${PGVER}-data `---- 2.2.4 PG 12.16 -------------- ,---- | export PGVER=12.16 | export PATH=$HOME/pg${PGVER}-bin/bin:$PATH | export PGDATA=$HOME/pg${PGVER}-data `---- 2.2.5 PG Retrieve Source & Build -------------------------------- ,---- | for PGVER in 15.4 14.9 13.12 12.16; do | | export PGVER=$PGVER | export PATH=$HOME/pg${PGVER}-bin/bin:$PATH | export PGDATA=$HOME/pg${PGVER}-data | | FILES="https://ftp.postgresql.org/pub/source/v${PGVER}/postgresql-${PGVER}.tar.gz | https://ftp.postgresql.org/pub/source/v${PGVER}/postgresql-${PGVER}.tar.gz.md5 | https://ftp.postgresql.org/pub/source/v${PGVER}/postgresql-${PGVER}.tar.gz.sha256"; | | for FILE in $FILES; do | wget -nv -P $HOME $FILE | done | | for FILE in $FILES; do | if [[ $FILE == *.sha256 ]]; then | echo "Validating sha256 checksum: $HOME/${FILE}" | sha256sum --check $HOME/$(basename $FILE) | if [ $? != 0 ]; then | echo "FATAL: sha256 check failed" | fi | fi | if [[ $FILE == *.md5 ]]; then | echo "Validating md5 checksum: $HOME/${FILE}" | md5sum --check $HOME/$(basename $FILE) | if [ $? != 0 ]; then | echo "FATAL: md5 check failed" | fi | fi | done | rm -rf $HOME/postgresql-${PGVER} | tar xf ~/postgresql-${PGVER}.tar.gz -C $HOME | pushd $HOME/postgresql-${PGVER} | ./configure --prefix=$HOME/pg${PGVER}-bin --with-python --with-libxml | make -j$(nproc); make check | rm -rf $HOME/pg${PGVER}-bin; make install | rm -rf ${PGDATA} | initdb | popd | done `---- 2.3 Greenplum - Install RPM and Environment config ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2.3.1 Install RPM - copied to test environment with gcloud scp command ---------------------------------------------------------------------- ,---- | # wget -nv https://github.com/greenplum-db/gpdb/releases/download/6.22.3/open-source-greenplum-db-6.22.3-rhel8-x86_64.rpm | # sudo dnf -d0 -y install open-source-greenplum-db-6.22.3-rhel8-x86_64.rpm | | sudo dnf install -y greenplum-db-6.25.1-rhel8-x86_64.rpm | | sudo dnf install -y greenplum-db-7.0.0-beta.5-el8-x86_64.rpm | pip3 install psycopg2 psutil `---- 2.3.2 Setup cluster ------------------- ,---- | ssh-keygen -t rsa -b 4096 -C eespino -f $HOME/.ssh/id_rsa -P "" | cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys | ssh-keyscan -t rsa $(hostname) > $HOME/.ssh/known_hosts | chmod 600 $HOME/.ssh/* | ssh -oStrictHostKeyChecking=no $(hostname) uptime | | cp /usr/local/greenplum-db/docs/cli_help/gpconfigs/gpinitsystem_config $HOME | | # Update gpinitystem file. | sed -i -e "s|\(^declare -a DATA_DIRECTORY.*\)|declare -a DATA_DIRECTORY=(/data/primary /data/primary)|" \ | -e "s|\(^MASTER_HOSTNAME=mdw$\)|MASTER_HOSTNAME=$(hostname)|" \ | -e "s|\(^#DATABASE_NAME=name_of_database.*$\)|DATABASE_NAME=gpadmin|" \ | -e "s|\(^#MACHINE_LIST_FILE=.*\)|MACHINE_LIST_FILE=$HOME/hostfile_gpinitsystem|" $HOME/gpinitsystem_config | | # Update gpinitystem file. | sed -i -e "s|\(^declare -a DATA_DIRECTORY.*\)|declare -a DATA_DIRECTORY=(/data/primary /data/primary)|" \ | -e "s|\(^COORDINATOR_HOSTNAME=cdw$\)|COORDINATOR_HOSTNAME=$(hostname)|" \ | -e "s|\(^#DATABASE_NAME=name_of_database.*$\)|DATABASE_NAME=eespino|" \ | -e "s|\(^SEG_PREFIX=gpseg.*$\)|SEG_PREFIX=gp7seg|" \ | -e "s|\(^#MACHINE_LIST_FILE=.*\)|MACHINE_LIST_FILE=$HOME/hostfile_gpinitsystem|" $HOME/gpinitsystem_config | | sudo rm -rf /data/master /data/coordinator /data/primary | sudo mkdir -p /data/master /data/coordinator /data/primary | sudo chown -R eespino.eespino /data/master /data/coordinator /data/primary | echo "$(hostname)" > $HOME/hostfile_gpinitsystem | | source /usr/local/greenplum-db/greenplum_path.sh | | export MASTER_DATA_DIRECTORY=/data/master/gpseg-1 | export COORDINATOR_DATA_DIRECTORY=/data/coordinator/gp7seg-1 | | # Initialize a Greenplum cluster (single-node/two primaries/no mirrors) | gpinitsystem -c $HOME/gpinitsystem_config -a | psql postgres -c 'select version()' | psql postgres -c 'show optimizer' | psql postgres -c 'select * from gp_segment_configuration' | gpstate `---- 2.3.3 greenplum_path & MASTER_DATA_DIRECTORY -------------------------------------------- ,---- | source /usr/local/greenplum-db/greenplum_path.sh | export MASTER_DATA_DIRECTORY=/data/master/gpseg-1 | export MASTER_DATA_DIRECTORY=/data/coordinator/gp7seg-1 `---- 2.4 Downloads, Check PGP signatures & checksums ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ,---- | FILES="https://dist.apache.org/repos/dist/dev/madlib/2.1.0.RC1/apache-madlib-2.1.0-src.tar.gz | https://dist.apache.org/repos/dist/dev/madlib/2.1.0.RC1/apache-madlib-2.1.0-src.tar.gz.asc | https://dist.apache.org/repos/dist/dev/madlib/2.1.0.RC1/apache-madlib-2.1.0-src.tar.gz.sha512"; | | for FILE in $FILES; do | wget -nv -P $HOME $FILE | done | | wget -nv -P $HOME https://dist.apache.org/repos/dist/dev/madlib/KEYS | gpg --import ./KEYS | | for FILE in $FILES; do | if [[ ${FILE} == *.asc ]]; then | echo "Validating signature: ${FILE}" | gpg --verify $HOME/$(basename $FILE) | if [ $? != 0 ]; then | echo "FATAL: gpg signature verification failed" | fi | fi | if [[ $FILE == *.sha512 ]]; then | echo "Validating sha512 checksum: ${FILE}" | sha512sum --check $HOME/$(basename $FILE) | if [ $? != 0 ]; then | echo "FATAL: sha512 check failed" | fi | fi | done | | rm -rf $HOME/apache-madlib-2.1.0-src | tar xf apache-madlib-2.1.0-src.tar.gz -C $HOME `---- 2.5 Ubuntu 22.04 ~~~~~~~~~~~~~~~~ ,---- | lsb_release -d | Description: Ubuntu 22.04.2 LTS `---- ,---- | pip3 uninstall pyxb | pip3 install protobuf==3.19.4 tensorflow==2.10.0 xgboost pyyaml scipy dill pgsanity hyperopt==0.2.5 scikit-learn pandas numpy rtree mock pyyaml pyxb-x | | # Test Orhan's dev branch | git clone https://github.com/orhankislal/madlib.git ~/apache-madlib-orhankislal | cd ~/apache-madlib-orhankislal | git checkout test-pg15 | | # Apache MADlib release | cd ~/apache-madlib-2.1.0-src | | rm -rf build; mkdir build; cd build | | cmake -DCMAKE_INSTALL_PREFIX=$HOME/madlib-v2.1.0 \ | -DPOSTGRESQL_15_PG_CONFIG=$HOME/pg15.4-bin/bin/pg_config \ | -DPOSTGRESQL_14_PG_CONFIG=$HOME/pg14.9-bin/bin/pg_config \ | -DPOSTGRESQL_13_PG_CONFIG=$HOME/pg13.12-bin/bin/pg_config \ | -DPOSTGRESQL_12_PG_CONFIG=$HOME/pg12.16-bin/bin/pg_config \ | -DPYTHON_EXECUTABLE:FILEPATH=/usr/bin/python3 \ | -DCMAKE_BUILD_TYPE=Release \ | .. | | make -j$(nproc) | | rm -rf $HOME/madlib-v2.1.0 | make install | | export PGVER=15.4 | export PGVER=14.9 | export PGVER=13.12 | export PGVER=12.16 | export PATH=$HOME/pg${PGVER}-bin/bin:$PATH | export PGDATA=$HOME/pg${PGVER}-data | export PGPORT=5432 | | pg_ctl -D ${PGDATA} -l ${PGDATA}/logfile start | pg_ctl -D ${PGDATA} -l ${PGDATA}/logfile stop | pg_ctl -D ${PGDATA} -l ${PGDATA}/logfile status | | psql template1 -c 'DROP DATABASE IF EXISTS madlibtest' | psql template1 -c 'CREATE DATABASE madlibtest' | psql template1 -c 'select version()' | | $HOME/madlib-v2.1.0/bin/madpack -c localhost:${PGPORT}/madlibtest -s madlib -p postgres install | $HOME/madlib-v2.1.0/bin/madpack -c localhost:${PGPORT}/madlibtest -s madlib -p postgres install-check | $HOME/madlib-v2.1.0/bin/madpack -c localhost:${PGPORT}/madlibtest -s madlib -p postgres dev-check | | grep ERROR /tmp/madlib.jik9jvmf/*/*.log `---- 2.6 Rocky Linux 8.8 ~~~~~~~~~~~~~~~~~~~ ,---- | cat /etc/os-release | NAME="Rocky Linux" | VERSION="8.8 (Green Obsidian)" | ID="rocky" | ID_LIKE="rhel centos fedora" | VERSION_ID="8.8" | PLATFORM_ID="platform:el8" | PRETTY_NAME="Rocky Linux 8.8 (Green Obsidian)" | ANSI_COLOR="0;32" | LOGO="fedora-logo-icon" | CPE_NAME="cpe:/o:rocky:rocky:8:GA" | HOME_URL="https://rockylinux.org/"; | BUG_REPORT_URL="https://bugs.rockylinux.org/"; | SUPPORT_END="2029-05-31" | ROCKY_SUPPORT_PRODUCT="Rocky-Linux-8" | ROCKY_SUPPORT_PRODUCT_VERSION="8.8" | REDHAT_SUPPORT_PRODUCT="Rocky Linux" | REDHAT_SUPPORT_PRODUCT_VERSION="8.8" `---- ,---- | # Install Python 3.9 dependencies | pip3 install protobuf==3.19.4 tensorflow==2.10.0 xgboost pyyaml scipy dill pgsanity hyperopt==0.2.5 scikit-learn pandas numpy rtree mock pyyaml pyxb | | Installation: pyyaml==6.0.1, pyxb-x==1.2.6.1 | Various: numpy==1.25.2 | Deep Learning: dill==0.3.7, grpcio==1.57.0, protobuf==3.19.4, hyperopt==0.2.5, tensorflow == 2.10, scikit-learn==1.3.0 | XGBoost: pandas==2.0.3, xgboost==1.7.6 | KNN: scipy==1.11.2 | Unit tests: pgsanity | | # Test Orhan's dev branch | git clone https://github.com/orhankislal/madlib.git ~/apache-madlib-orhankislal | cd ~/apache-madlib-orhankislal | git checkout test-pg15 | | # Apache MADlib release | cd ~/apache-madlib-2.1.0-src | | rm -rf build; mkdir build; cd build | | cmake -DCMAKE_INSTALL_PREFIX=$HOME/madlib-v2.1.0 \ | -DGREENPLUM_6_PG_CONFIG=/usr/local/greenplum-db/bin/pg_config \ | -DGREENPLUM_7_PG_CONFIG=/usr/local/greenplum-db/bin/pg_config \ | -DPOSTGRESQL_15_PG_CONFIG=$HOME/pg15.4-bin/bin/pg_config \ | -DPOSTGRESQL_14_PG_CONFIG=$HOME/pg14.9-bin/bin/pg_config \ | -DPOSTGRESQL_13_PG_CONFIG=$HOME/pg13.12-bin/bin/pg_config \ | -DPOSTGRESQL_12_PG_CONFIG=$HOME/pg12.16-bin/bin/pg_config \ | -DPYTHON_EXECUTABLE:FILEPATH=/usr/bin/python3.9 \ | -DCMAKE_BUILD_TYPE=Release \ | .. | | make -j$(nproc) | | rm -rf $HOME/madlib-v2.1.0 | make install | | export PGVER=15.4 | export PGVER=14.9 | export PGVER=13.12 | export PGVER=12.16 | export PATH=$HOME/pg${PGVER}-bin/bin:$PATH | export PGDATA=$HOME/pg${PGVER}-data | export PGPORT=5432 | | pg_ctl -D ${PGDATA} -l ${PGDATA}/logfile start | pg_ctl -D ${PGDATA} -l ${PGDATA}/logfile stop | pg_ctl -D ${PGDATA} -l ${PGDATA}/logfile status | | psql template1 -c 'select version()' | psql template1 -c 'DROP DATABASE IF EXISTS madlibtest' | psql template1 -c 'CREATE DATABASE madlibtest' | | $HOME/madlib-v2.1.0/bin/madpack -c localhost:${PGPORT}/madlibtest -s madlib -p postgres install | $HOME/madlib-v2.1.0/bin/madpack -c localhost:${PGPORT}/madlibtest -s madlib -p postgres install-check | $HOME/madlib-v2.1.0/bin/madpack -c localhost:${PGPORT}/madlibtest -s madlib -p postgres dev-check | | $HOME/madlib-v2.1.0/bin/madpack -c localhost:${PGPORT}/madlibtest -s madlib -p greenplum install | $HOME/madlib-v2.1.0/bin/madpack -c localhost:${PGPORT}/madlibtest -s madlib -p greenplum install-check | $HOME/madlib-v2.1.0/bin/madpack -c localhost:${PGPORT}/madlibtest -s madlib -p greenplum dev-check | | grep ERROR /tmp/madlib.zmbupzpw/*/*.log `---- 2.7 Rocky Linux 9.2 ~~~~~~~~~~~~~~~~~~~ ,---- | cat /etc/redhat-release | Rocky Linux release 9.2 (Blue Onyx) `---- ,---- | pip install protobuf==3.19.4 tensorflow==2.10.0 xgboost pyyaml scipy dill pgsanity hyperopt==0.2.5 scikit-learn pandas numpy rtree mock pyyaml pyxb | | # Test Orhan's dev branch | git clone https://github.com/orhankislal/madlib.git ~/apache-madlib-orhankislal | cd ~/apache-madlib-orhankislal | git checkout test-pg15 | | # Apache MADlib release | cd ~/apache-madlib-2.1.0-src | | rm -rf build; mkdir build; cd build | | cmake -DCMAKE_INSTALL_PREFIX=$HOME/madlib-v2.1.0 \ | -DPOSTGRESQL_15_PG_CONFIG=$HOME/pg15.4-bin/bin/pg_config \ | -DPOSTGRESQL_14_PG_CONFIG=$HOME/pg14.9-bin/bin/pg_config \ | -DPOSTGRESQL_13_PG_CONFIG=$HOME/pg13.12-bin/bin/pg_config \ | -DCMAKE_BUILD_TYPE=Release \ | .. | | make -j$(nproc) | | rm -rf $HOME/madlib-v2.1.0 | make install | | export PGVER=15.4 | export PGVER=14.9 | export PGVER=13.12 | export PATH=$HOME/pg${PGVER}-bin/bin:$PATH | export PGDATA=$HOME/pg${PGVER}-data | export PGPORT=5432 | | pg_ctl -D ${PGDATA} -l ${PGDATA}/logfile start | pg_ctl -D ${PGDATA} -l ${PGDATA}/logfile stop | pg_ctl -D ${PGDATA} -l ${PGDATA}/logfile status | | psql template1 -c 'select version()' | psql template1 -c 'DROP DATABASE IF EXISTS madlibtest' | psql template1 -c 'CREATE DATABASE madlibtest' | | $HOME/madlib-v2.1.0/bin/madpack -c localhost:${PGPORT}/madlibtest -s madlib -p postgres install | $HOME/madlib-v2.1.0/bin/madpack -c localhost:${PGPORT}/madlibtest -s madlib -p postgres install-check | $HOME/madlib-v2.1.0/bin/madpack -c localhost:${PGPORT}/madlibtest -s madlib -p postgres dev-check `---- 2.8 Reproduction ~~~~~~~~~~~~~~~~ 2.8.1 Ubuntu 22.04 ------------------ ,---- | # Access - Ubuntu 22.04 host | gcloud compute ssh --zone "us-west1-b" "eespino-ubuntu22" --project "data-gpdb-server" | | # Become "eespino" where the environment is available | sudo su - eespino | | # Show Ubuntu version | lsb_release -d | | # PostgreSQL 15 environment variables | export PATH=$HOME/pgbin15/bin:$PATH | export PGDATA=$HOME/pgdata15 | export PGPORT=5432 | | # Manipulate the PG service | pg_ctl -D ${PGDATA} -l ${PGDATA}/logfile start | pg_ctl -D ${PGDATA} -l ${PGDATA}/logfile stop | pg_ctl -D ${PGDATA} -l ${PGDATA}/logfile status | | # Environment - Postgres 15 - info | Source: $HOME/postgresql-15.4 | Install dir: $HOME/pgbin15 | Data dir: $HOME/pgdata15 | | # Environment - MADlib info | Source: $HOME/apache-madlib-2.1.0-src | Install dir: $HOME/madlib-v2.1.0 | | # MADlib commands | $HOME/madlib-v2.1.0/bin/madpack -c localhost:${PGPORT}/madlibtest -s madlib -p postgres install | $HOME/madlib-v2.1.0/bin/madpack -c localhost:${PGPORT}/madlibtest -s madlib -p postgres install-check | $HOME/madlib-v2.1.0/bin/madpack -c localhost:${PGPORT}/madlibtest -s madlib -p postgres dev-check | | # Pip packages | pip3 install protobuf==3.19.4 tensorflow==2.10.0 xgboost pyyaml scipy dill pgsanity hyperopt==0.2.5 scikit-learn pandas numpy rtree mock pyyaml pyxb | | # MADlib build command | cmake -DCMAKE_INSTALL_PREFIX=$HOME/madlib-v2.1.0 \ | -DPOSTGRESQL_15_PG_CONFIG=$HOME/pgbin15/bin/pg_config \ | -DPYTHON_EXECUTABLE:FILEPATH=/usr/bin/python3 \ | -DCMAKE_BUILD_TYPE=Release \ | .. | | # GCP VM creation & interaction | gcloud compute instances create eespino-ubuntu22 \ | --project=data-gpdb-server \ | --zone=us-west1-b \ | --machine-type=e2-standard-32 \ | --network-interface=network-tier=PREMIUM,stack-type=IPV4_ONLY,subnet=default \ | --maintenance-policy=MIGRATE \ | --provisioning-model=STANDARD \ | --service-account=612089704425-comp...@developer.gserviceaccount.com \ | --scopes=https://www.googleapis.com/auth/devstorage.read_only,https://www.googleapis.com/auth/logging.write,https://www.googleapis.com/auth/monitoring.write,https://www.googleapis.com/auth/servicecontrol,https://www.googleapis.com/auth/service.management.readonly,https://www.googleapis.com/auth/trace.append \ | --create-disk=auto-delete=yes,boot=yes,device-name=instance-1,image=projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727,mode=rw,size=100,type=projects/data-gpdb-server/zones/us-central1-a/diskTypes/pd-balanced \ | --no-shielded-secure-boot \ | --shielded-vtpm \ | --shielded-integrity-monitoring \ | --labels=goog-ec-src=vm_add-gcloud \ | --reservation-affinity=any | | gcloud compute ssh --zone "us-west1-b" "eespino-ubuntu22" --project "data-gpdb-server" | gcloud compute instances list --filter="name=('eespino-ubuntu22')" --project "data-gpdb-server" | gcloud compute instances suspend --zone "us-west1-b" "eespino-ubuntu22" --project "data-gpdb-server" --quiet | gcloud compute instances resume --zone "us-west1-b" "eespino-ubuntu22" --project "data-gpdb-server" --quiet | gcloud compute instances delete --zone "us-west1-b" "eespino-ubuntu22" --project "data-gpdb-server" --quiet | | # Ubuntu packages | sudo apt-get update | sudo NEEDRESTART_SUSPEND=1 apt-get install -qq -y wget cmake gcc g++ flex bison patch libreadline-dev zlib1g-dev libxml2-dev libpython3-dev m4 bat `---- 2.8.2 Rocky 8 ------------- ,---- | PG15 | Install dir: ~gpadmin/pg15.4-bin | Data dir: ~gpadmin/pg15.4-data | | PG14 | Install dir: ~gpadmin/pg14.9-bin | Data dir: ~gpadmin/pg14.9-data | | PG13 | Install dir: ~gpadmin/pg13.12-bin | Data dir: ~gpadmin/pg13.12-data | | PG12 | Install dir: ~gpadmin/pg12.16-bin | Data dir: ~gpadmin/pg12.16-data | | export PGVER=15.4 | export PATH=$HOME/pg${PGVER}-bin/bin:$PATH | export PGDATA=$HOME/pg${PGVER}-data | export PGPORT=5432 | | export PGVER=14.9 | export PATH=$HOME/pg${PGVER}-bin/bin:$PATH | export PGDATA=$HOME/pg${PGVER}-data | export PGPORT=5432 | | export PGVER=13.12 | export PATH=$HOME/pg${PGVER}-bin/bin:$PATH | export PGDATA=$HOME/pg${PGVER}-data | export PGPORT=5432 | | export PGVER=12.16 | export PATH=$HOME/pg${PGVER}-bin/bin:$PATH | export PGDATA=$HOME/pg${PGVER}-data | export PGPORT=5432 | | pg_ctl -D ${PGDATA} -l ${PGDATA}/logfile start | pg_ctl -D ${PGDATA} -l ${PGDATA}/logfile stop | pg_ctl -D ${PGDATA} -l ${PGDATA}/logfile status | | $HOME/madlib-v2.1.0/bin/madpack -c localhost:5432/madlibtest -s madlib -p postgres install | $HOME/madlib-v2.1.0/bin/madpack -c localhost:5432/madlibtest -s madlib -p postgres install-check | $HOME/madlib-v2.1.0/bin/madpack -c localhost:5432/madlibtest -s madlib -p postgres dev-check | | $HOME/madlib-v2.1.0/bin/madpack -c localhost:5432/madlibtest -s madlib -p greenplum install | $HOME/madlib-v2.1.0/bin/madpack -c localhost:5432/madlibtest -s madlib -p greenplum install-check | $HOME/madlib-v2.1.0/bin/madpack -c localhost:5432/madlibtest -s madlib -p greenplum dev-check `----
