Re: Proposal for changes in official Docker image

2018-03-31 Thread Максим Кольцов
2018-03-19 17:55 GMT+03:00 Dave Page :
> Hi
>
> On Mon, Mar 19, 2018 at 11:19 AM, Dave Page  wrote:
>>
>> Hi
>>
>> On Fri, Mar 16, 2018 at 8:52 PM, Максим Кольцов 
>> wrote:
>>>
>>>
>>> I've switched to Gunicorn, adding SSL support. It has the same
>>> interface as the original container: PGADMIN_ENABLE_TLS,
>>> /certs/server.key and /certs/server.cert.
>>> I also incorporated building of sphinx manual in Dockerfile, so now
>>> the image should be complete.
>>
>>
>> Nice!
>>
>>>
>>>
>>> I noticed that I can't use gunicorn forking worker with pgadmin4, this
>>> is probably caused by session implementation, but I'm not sure. You
>>> can investigate this by using e.g. `-w 4` in entrypoint.sh, otherwise
>>> it's working fine with single-process threaded worker.
>>
>>
>> This is expected, and the reason why is explained here:
>> https://www.pgadmin.org/docs/pgadmin4/2.x/server_deployment.html#requirements
>>
>>>
>>>
>>> I will make my work into a patch and send it to the mail list soon.
>>> Meanwhile, it'd be great if you tested the updated image at
>>> https://hub.docker.com/r/maksbotan/pgadmin4/
>>
>>
>> I'll try to take a look later in the week (busy prepping for a release at
>> the moment).
>
>
> I actually found some time for this today. It's looking pretty good - TLS
> vs. non-TLS seems to work well. The couple of things I did notice were:
>
> - PGADMIN_SERVER_NAME doesn't appear to be supported. This was added at user
> request, for security reasons (to help ensure the connection is going where
> expected). I'm not entirely convinced of the value of that, but if it's
> fairly painless to add, it may well be worth it.

Can you explain the meaning of this option, or maybe give me a link to
original feature request?
As far as I understand, Apache uses this to identify virtual hosts
based on HTTP Host header,
but there are no virtual hosts in Gunicorn, so no need for this.

> - If you don't map /var/lib/pgadmin (in my case, to a directory already
> containing a config), then I was seeing the following error. I would expect
> it to init within the container if the config directory isn't mapped:
>
> piranha:web dpage$ docker logs affectionate_spence
> sh: -z: unknown operand
> NOTE: Configuring authentication for SERVER mode.
>
> Enter the email address and password to use for the initial pgAdmin user
> account:
>
> Traceback (most recent call last):
>   File "run_pgadmin.py", line 4, in 
> from pgAdmin4 import app
>   File "/pgadmin4/pgAdmin4.py", line 67, in 
> app = create_app()
>   File "/pgadmin4/pgadmin/__init__.py", line 306, in create_app
> db_upgrade(app)
>   File "/pgadmin4/pgadmin/setup/db_upgrade.py", line 25, in db_upgrade
> flask_migrate.upgrade(migration_folder)
>   File "/usr/local/lib/python3.6/site-packages/flask_migrate/__init__.py",
> line 244, in upgrade
> command.upgrade(config, revision, sql=sql, tag=tag)
>   File "/usr/local/lib/python3.6/site-packages/alembic/command.py", line
> 254, in upgrade
> script.run_env()
>   File "/usr/local/lib/python3.6/site-packages/alembic/script/base.py", line
> 427, in run_env
> util.load_python_file(self.dir, 'env.py')
>   File "/usr/local/lib/python3.6/site-packages/alembic/util/pyfiles.py",
> line 81, in load_python_file
> module = load_module_py(module_id, path)
>   File "/usr/local/lib/python3.6/site-packages/alembic/util/compat.py", line
> 83, in load_module_py
> spec.loader.exec_module(module)
>   File "/pgadmin4/pgadmin/setup/../../migrations/env.py", line 94, in
> 
> run_migrations_online()
>   File "/pgadmin4/pgadmin/setup/../../migrations/env.py", line 87, in
> run_migrations_online
> context.run_migrations()
>   File "", line 8, in run_migrations
>   File
> "/usr/local/lib/python3.6/site-packages/alembic/runtime/environment.py",
> line 836, in run_migrations
> self.get_context().run_migrations(**kw)
>   File
> "/usr/local/lib/python3.6/site-packages/alembic/runtime/migration.py", line
> 330, in run_migrations
> step.migration_fn(**kw)
>   File "/pgadmin4/migrations/versions/fdc58d9bd449_.py", line 112, in
> upgrade
> email, password = user_info()
>   File "/pgadmin4/pgadmin/setup/user_info.py", line 55, in user_info
> email = input("Email address: ")
> EOFError: EOF when reading a line
> [2018-03-19 14:50:59 +] [1] [INFO] Starting gunicorn 19.7.1
> [2018-03-19 14:50:59 +] [1] [INFO] Listening at: http://0.0.0.0:8080 (1)
> [2018-03-19 14:50:59 +] [1] [INFO] Using worker: threads
> [2018-03-19 14:50:59 +] [14] [INFO] Booting worker with pid: 14
> [2018-03-19 14:50:59 +] [14] [ERROR] Exception in worker process
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.6/site-packages/gunicorn/arbiter.py", line
> 578, in spawn_worker
> worker.init_process()
>   File "/usr/local/lib/python3.6/site-packages/gunicorn/workers/gthread.py",
> line 109, in init_process
> super(ThreadWorker, self).init_process()
>   File "/usr/local/lib/python3.6/site-pack

[pgAdmin4][Patch] Remake Docker container packaging

2018-03-31 Thread Максим Кольцов
Hi all,

I've been discussing this with Dave for about a month now. Today I
finally present a proposed patch to update pgadmin4's Docker
packaging.

Key features of this update:
- Main image is based on python:3.6-alpine3.7.
  Using Alpine linux leads to much smaller image
- All build is done with Docker multi-stage build. First of all build
the frontend in node:6 image,
  then build Sphinx documentation in separate Python container and in
the end just install all
  dependencies in a clean python:3.6-alpine3.7 image, so that it does
not have any leftovers from the build
  process and we don't rely on any tools available on the host.
- Use Gunicorn (http://gunicorn.org) as lightweight HTTP / WSGI server.
  Gunicorn supports both HTTP and HTTPS.
- Install Alpine postgresql-client package, which includes pg_dump and
other tools and config
  PgAdmin to find these tools by default
- Byte-compile all PgAdmin Python code in Dockerfile with optimization
(-O) enabled. This way Python
   does not have to compile modules on each container restart and
consume space in overlay fs

Please find attached patch from "git format-patch".
From 67e387525a7c832958858c7ec1a1b7076382090e Mon Sep 17 00:00:00 2001
From: Maxim Koltsov 
Date: Sat, 31 Mar 2018 20:37:51 +0300
Subject: [PATCH] Re-make Docker container packaging

Key features of this update:
- Main image is based on python:3.6-alpine3.7.
  Using Alpine linux leads to much smaller image
- All build is done with Docker multi-stage build. First of all build the frontend in node:6 image,
  then build Sphinx documentation in separate Python container and in the end just install all
  dependencies in a clean python:3.6-alpine3.7 image, so that it does not have any leftovers from the build
  process and we don't rely on any tools available on the host.
- Use Gunicorn (http://gunicorn.org) as lightweight HTTP / WSGI server.
  Gunicorn supports both HTTP and HTTPS.
- Install Alpine postgresql-client package, which includes pg_dump and other tools and config
  PgAdmin to find these tools by default
- Byte-compile all PgAdmin Python code in Dockerfile with optimization (-O) enabled. This way Python
  does not have to compile modules on each container restart and consume space in overlay fs
---
 pkg/docker/Dockerfile   | 89 -
 pkg/docker/README   | 58 ++---
 pkg/docker/build.sh | 57 +
 pkg/docker/config_distro.py |  4 ++
 pkg/docker/entry.sh | 29 ---
 pkg/docker/entrypoint.sh| 21 +++
 pkg/docker/pgadmin4.conf.j2 | 43 --
 pkg/docker/run_pgadmin.py   |  4 ++
 8 files changed, 112 insertions(+), 193 deletions(-)
 create mode 100644 pkg/docker/config_distro.py
 delete mode 100644 pkg/docker/entry.sh
 create mode 100755 pkg/docker/entrypoint.sh
 delete mode 100644 pkg/docker/pgadmin4.conf.j2
 create mode 100644 pkg/docker/run_pgadmin.py

diff --git a/pkg/docker/Dockerfile b/pkg/docker/Dockerfile
index 1c1dde27..083dbd60 100644
--- a/pkg/docker/Dockerfile
+++ b/pkg/docker/Dockerfile
@@ -7,58 +7,63 @@
 #
 #
 
-# Get the basics out of the way
-FROM centos:latest
+# First of all, build frontend with NodeJS in a separate builder container
+# Node-6 with ABI v48 is supported by all needed C++ packages
+FROM node:6 AS node-builder
 
-LABEL name="pgAdmin 4" \
-vendor="The pgAdmin Development Team" \
-license="PostgreSQL"
+COPY ./pgadmin4/web/ /pgadmin4/web/
+WORKDIR /pgadmin4/web
 
-# We only need the web/ directory, and a few other things
-COPY web /var/www/pgadmin
-COPY requirements.txt /var/www/pgadmin
+RUN yarn install --cache-folder ./ycache --verbose && \
+yarn run bundle && \
+rm -rf ./ycache ./pgadmin/static/js/generated/.cache
 
-# Install everything we need. Use easy_install to get pip, to avoid setting up EPEL
-RUN yum install -y python-setuptools python-devel httpd mod_wsgi mod_ssl gcc
-RUN easy_install pip
-RUN pip install j2cli
+# Build Sphinx documentation in separate container
+FROM python:3.6-alpine3.7 as docs-builder
 
-# Now install the Python runtime dependencies
-RUN pip install -r /var/www/pgadmin/requirements.txt
+# Install only dependencies absolutely required for documentation building
+RUN apk add --no-cache make
+RUN pip install --no-cache-dir \
+sphinx flask_babel flask_security flask_paranoid python-dateutil flask_sqlalchemy \
+flask_gravatar simplejson
 
-# Create required directories for config
+COPY ./pgadmin4/ /pgadmin4
 
+RUN LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8 make -C /pgadmin4/docs/en_US -f Makefile.sphinx html
 
-# Create required directories for running
-RUN mkdir -p /var/log/pgadmin
-RUN chown -R apache /var/log/pgadmin
-RUN mkdir -p /var/lib/pgadmin
-RUN chown -R apache /var/lib/pgadmin
-RUN mkdir -p /certs
-RUN chown -R apache /certs
-RUN chmod 700 /certs
+# Then install backend, copy static file