On Mon, 2017-09-04 at 01:14 +1000, Daniel Axtens wrote:
> This allows us to easily test against PostgreSQL using the same
> tooling we normally use. This is helpful in (for example) shaking
> out the test failures that were observed on ozlabs.org
>
> To use it:
> docker-compose -f docker-compose-pg.yml <usual arguments>
>
> (You may find it necessary to do a 'docker-compose down' first,
> depending on what state the system is in and what command you're
> running.)
>
> Signed-off-by: Daniel Axtens <d...@axtens.net>
Some comments below.
> ---
> docker-compose-pg.yml | 30 ++++++++++++++++++++++++++
> requirements-test.txt | 3 ++-
> tools/docker/Dockerfile | 2 +-
> tools/docker/entrypoint.sh | 53 +++++++++++++++++++++++++++++++++++++-------
> --
> 4 files changed, 76 insertions(+), 12 deletions(-)
> create mode 100644 docker-compose-pg.yml
>
> diff --git a/docker-compose-pg.yml b/docker-compose-pg.yml
> new file mode 100644
> index 000000000000..34900418375e
> --- /dev/null
> +++ b/docker-compose-pg.yml
> @@ -0,0 +1,30 @@
> +# the version of docker-compose shipped in ubuntu 16.04 is
> +# 1.5.2, which doesn't support version 2 syntax. Yay!
> +# also, v1 doesn't support explicit build args, so if you're not
> +# uid 1000, you will either need to manually hack the Dockerfile
> +# or upgrade to v2 and use the build-arg to override it.
Only 7 months to 18.04, guys!
> +
> +db:
> + image: postgres
Any particular version needed?
> + environment:
> + - POSTGRES_PASSWORD=super_sekrit_postgres_passwd
> + volumes:
> + - ./tools/docker/db/postdata:/var/lib/postgresql/data
> +
> +web:
> + build: .
> + dockerfile: ./tools/docker/Dockerfile
> + command: python3 manage.py runserver 0.0.0.0:8000
> + volumes:
> + - .:/home/patchwork/patchwork/
> + ports:
> + - "8000:8000"
> + links:
> + - db
> + environment:
> + - PGPASSWORD=super_sekrit_postgres_passwd
> + - PW_TEST_DB_HOST=db
> + - PW_TEST_DB_PORT=5432
> + - PW_TEST_DB_TYPE=postgres
> + - PW_TEST_DB_USER=postgres
> + - PW_TEST_DB_PASS=super_sekrit_postgres_passwd
Any harm in just using 'password', like we do for mysql. It is for dev
purposes, after all.
> diff --git a/requirements-test.txt b/requirements-test.txt
> index cead336c7c77..141cf661663d 100644
> --- a/requirements-test.txt
> +++ b/requirements-test.txt
> @@ -1,4 +1,5 @@
> -mysqlclient>=1.3,<1.4 # replace this with psycopg2 for a PostgreSQL backend
> +mysqlclient>=1.3,<1.4
> +psycopg2>=2.7,<2.8
Are you sure this works? Last I checked, you needed PostgreSQL installed for
psycopg2 to install correctly (think I spotted it when adding Vagrant support).
Could you try rebuilding the MySQL-based container to validate this?
> django-debug-toolbar==1.8
> python-dateutil>2.0,<3.0
> selenium>=3.0,<4.0
> diff --git a/tools/docker/Dockerfile b/tools/docker/Dockerfile
> index ff05707a6049..946c646188e5 100644
> --- a/tools/docker/Dockerfile
> +++ b/tools/docker/Dockerfile
> @@ -21,7 +21,7 @@ RUN apt-get update -qq && \
> python3.5-dev python3-pip python3-setuptools python3-wheel \
> python3.4-dev findutils=4.4.2-7 \
> libmysqlclient-dev mysql-client curl unzip xvfb chromium-chromedriver \
> - chromium-browser build-essential git && \
> + chromium-browser build-essential git postgresql-client && \
...although maybe this resolves the issue. I'll wait to see if you tested this
:)
> ln -s /usr/lib/chromium-browser/chromedriver /usr/bin/
>
> # User
> diff --git a/tools/docker/entrypoint.sh b/tools/docker/entrypoint.sh
> index 2f413b02e83e..4b656adaeb35 100755
> --- a/tools/docker/entrypoint.sh
> +++ b/tools/docker/entrypoint.sh
> @@ -1,13 +1,31 @@
> #!/bin/bash
> -set -euo pipefail
> +set -eo pipefail
> +
> +if [ x${PW_TEST_DB_TYPE} = x ]; then
> + PW_TEST_DB_TYPE=mysql
> +fi
> +
> +set -u
Why do we need to move the '-u' (undefined variable?) down here? Does Bash not
provide an equivalent of Python's 'getattr(ATTR, DEFAULT)'?
>
> # functions
>
> test_db_connection() {
> - mysqladmin -h $PW_TEST_DB_HOST -u patchwork --password=password ping >
> /dev/null 2> /dev/null
> + if [ ${PW_TEST_DB_TYPE} = "postgres" ]; then
> + echo ';' | psql -h $PW_TEST_DB_HOST -U postgres 2> /dev/null >
> /dev/null
> + else
> + mysqladmin -h $PW_TEST_DB_HOST -u patchwork --password=password ping
> > /dev/null 2> /dev/null
> + fi
> }
Looks good.
> -reset_data() {
> +test_database() {
> + if [ ${PW_TEST_DB_TYPE} = "postgres" ]; then
> + echo ';' | psql -h $PW_TEST_DB_HOST -U postgres patchwork 2>
> /dev/null
> + else
> + echo ';' | mysql -h $PW_TEST_DB_HOST -u patchwork -ppassword
> patchwork 2> /dev/null
> + fi
> +}
> +
Looks good.
> +reset_data_mysql() {
> mysql -u$db_user -p$db_pass -h $PW_TEST_DB_HOST << EOF
> DROP DATABASE IF EXISTS patchwork;
> CREATE DATABASE patchwork CHARACTER SET utf8;
> @@ -15,6 +33,21 @@ GRANT ALL ON patchwork.* TO 'patchwork' IDENTIFIED BY
> 'password';
> GRANT ALL PRIVILEGES ON test_patchwork.* TO 'patchwork'@'%';
> FLUSH PRIVILEGES;
> EOF
> +}
> +
> +reset_data_postgres() {
> + psql -h $PW_TEST_DB_HOST -U postgres <<EOF
> +DROP DATABASE IF EXISTS patchwork;
> +CREATE DATABASE patchwork WITH ENCODING = 'UTF8';
> +EOF
> +}
> +
> +reset_data() {
> + if [ x${PW_TEST_DB_TYPE} = x"postgres" ]; then
> + reset_data_postgres
> + else
> + reset_data_mysql
> + fi
>
> # load initial data
> python3 $PROJECT_HOME/manage.py migrate #> /dev/null
> @@ -46,13 +79,13 @@ for x in /tmp/requirements-*.txt; do
> fi
> done
>
> -# check if mysql is connected
> +# check if db is connected
> if ! test_db_connection; then
> - echo "MySQL seems not to be connected, or the patchwork user is broken"
> - echo "MySQL may still be starting. Waiting 5 seconds."
> + echo "The database seems not to be connected, or the patchwork user is
> broken"
> + echo "MySQL/Postgres may still be starting. Waiting 5 seconds."
> sleep 5
> if ! test_db_connection; then
> - echo "Still cannot connect to MySQL."
> + echo "Still cannot connect to database."
> echo "Maybe you are starting the db for the first time. Waiting up
> to 60 seconds."
> for i in {0..9}; do
> sleep 5
> @@ -61,19 +94,19 @@ if ! test_db_connection; then
> fi
> done
> if ! test_db_connection; then
> - echo "Still cannot connect to MySQL. Giving up."
> + echo "Still cannot connect to database. Giving up."
> echo "Are you using docker-compose? If not, have you set up the
> link correctly?"
> exit 1
> fi
> fi
> fi
>
> -# rebuild mysql db
> +# rebuild db
> # do this on --reset or if the db doesn't exist
> if [[ "$1" == "--reset" ]]; then
> shift
> reset_data
> -elif ! ( echo ';' | mysql -h db -u patchwork -ppassword patchwork 2>
> /dev/null ); then
> +elif ! test_database; then
> reset_data
> fi
>
Yup, all this is fine by me.
Stephen
_______________________________________________
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork
