Hi Marcin, I have seen the exact same problem where the portal-db container reported it is running but actually failed to initialise correctly. In order to debug it, I needed to watch the logs as it started up:
kubectl -n onap logs -f dev-portal-db-?????-????? Initializing database [...] Database initialized MySQL init process in progress... 2018-11-16 16:40:58 0 [Note] mysqld (mysqld 10.3.10-MariaDB-1:10.3.10+maria~bionic) starting as process 101 ... MySQL init process in progress... MySQL init process in progress... 2018-11-16 16:41:01 0 [Note] InnoDB: Using Linux native AIO 2018-11-16 16:41:01 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2018-11-16 16:41:01 0 [Note] InnoDB: Uses event mutexes 2018-11-16 16:41:01 0 [Note] InnoDB: Compressed tables use zlib 1.2.11 2018-11-16 16:41:01 0 [Note] InnoDB: Number of pools: 1 2018-11-16 16:41:01 0 [Note] InnoDB: Using SSE2 crc32 instructions 2018-11-16 16:41:01 0 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M 2018-11-16 16:41:01 0 [Note] InnoDB: Completed initialization of buffer pool 2018-11-16 16:41:01 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). MySQL init process in progress... 2018-11-16 16:41:01 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2018-11-16 16:41:01 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2018-11-16 16:41:01 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... MySQL init process in progress... MySQL init process in progress... MySQL init process in progress... [...] MySQL init process in progress... MySQL init process failed. When I looked at the portal-db log after the fact, the stdout messages are no longer present. The mariadb docker-entrypoint.sh script was failing here: https://github.com/docker-library/mariadb/blob/7384b25a0196d8a7e62173781df5a5bed1eb88d2/10.4/docker-entrypoint.sh#L103-L113 As you can see the script has failed before running any of the SQL from /docker-entrypoint-initdb.d/* so it was taking longer than 30s to perform the database init and writing 12MB to ibtmpl. The root cause for me was very slow performance of the NFS share for /dockerdata-nfs/ – which for me was hosted by another VM that was backed with Openstack block storage. I wasn't able to isolate the root cause of the poor performance, but I guess it could have been due to block storage latency or openstack networking latency. Note that the portal-db readiness probe waits for 450s (7.5 minutes) before it starts. Even when the db initialisation fails, the portal-db will be marked "running" and ready "1/1" after 7.5 minutes and then the portal-db-config job will start (and then fail). Hope this helps you. Donald. -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#14002): https://lists.onap.org/g/onap-discuss/message/14002 Mute This Topic: https://lists.onap.org/mt/28285637/21656 Group Owner: onap-discuss+ow...@lists.onap.org Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
