Hi ONAP experts, Trying to deploy VF-C using OOM El Alto release. I'm getting some error during the initialization of mariadb related pods. Pod vfc - vfc -mariadb-0 keeps at CrashLoopBackOff status. I would like to know if someone else experienced this behavior before and if yes, if could share the workaround. Thank you!
Please find below the logs from vfc-mariadb-0: onapit- vfc - vfc -catalog-9d8b769ff-d8zbv 0/2 Init:0/1 1 13m onapit- vfc - vfc -ems-driver-756654cccf-6gcqr 1/1 Running 0 13m onapit- vfc - vfc -generic-vnfm-driver-547f6c5678-2n7lc 2/2 Running 0 13m onapit- vfc - vfc -huawei-vnfm-driver-7b665b768d-6qz9f 2/2 Running 0 13m onapit- vfc - vfc -juju-vnfm-driver-554f4f8478-zfv57 2/2 Running 2 13m onapit- vfc - vfc -mariadb-0 0/1 CrashLoopBackOff 10 13m onapit- vfc - vfc -multivim-proxy-676d5c6bc9-mbvqm 1/1 Running 1 13m onapit- vfc - vfc -nokia-v2vnfm-driver-897545dc4-2b7d9 1/1 Running 1 13m onapit- vfc - vfc -nslcm-96cc687c9-bvp4s 0/2 Init:0/1 1 13m onapit- vfc - vfc -redis-687bd5866c-qvlqr 1/1 Running 0 13m onapit- vfc - vfc -resmgr-859b596b9d-294q7 2/2 Running 2 13m onapit- vfc - vfc -vnflcm-698b5df694-mqp96 0/2 Init:0/1 1 13m onapit- vfc - vfc -vnfmgr-89d5749b6-9dvtt 0/2 Init:0/1 1 13m onapit- vfc - vfc -vnfres-7545b49cff-lmxgz 0/2 Init:0/1 1 13m onapit- vfc - vfc -workflow-798d598bcd-brzwx 1/1 Running 1 13m onapit- vfc - vfc -workflow-engine-5c6d7d6495-4g8cv 1/1 Running 0 13m onapit- vfc - vfc -zte-sdnc-driver-c64b7cdfb-l6tb8 1/1 Running 0 13m onapit- vfc - vfc -zte-vnfm-driver-6695754b4c-dzswf 2/2 Running 2 13m >From logs of pod I got the following errors before restart the VM: 2020-04-03 0:10:26 139909286091008 [ERROR] InnoDB: Tried to read 16384 bytes at offset 802816. Was only able to read 0. 2020-04-03 00:10:26 7f3f2b4c5900 InnoDB: Operating system error number 5 in a file operation. InnoDB: Error number 5 means 'Input/output error'. InnoDB: Some operating system error numbers are described at InnoDB: http://dev.mysql.com/doc/refman/5.6/en/operating-system-error-codes.html InnoDB: Operation read to file /home/buildbot/buildbot/build/mariadb-10.1.24/storage/xtradb/os/os0file.cc and at line 3175 2020-04-03 0:10:26 139909286091008 [ERROR] InnoDB: File (unknown): 'read' returned OS error 205. Cannot continue operation 200403 0:10:26 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.1.24-MariaDB key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=0 max_threads=153 thread_count=2 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467125 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x0 thread_stack 0x48400 addr2line: 'mysqld': No such file mysqld(my_print_stacktrace+0x2e)[0x5584673e333e] mysqld(handle_fatal_signal+0x305)[0x558466f090a5] /lib64/libpthread.so.0(+0xf370)[0x7f3f2b0d7370] /lib64/libc.so.6(gsignal+0x37)[0x7f3f2942c1d7] /lib64/libc.so.6(abort+0x148)[0x7f3f2942d8c8] mysqld(+0x79b179)[0x5584670ec179] mysqld(+0x79fb6c)[0x5584670f0b6c] mysqld(+0x7a27db)[0x5584670f37db] mysqld(+0x8e9d67)[0x55846723ad67] mysqld(+0x8aa3ff)[0x5584671fb3ff] mysqld(+0x888339)[0x5584671d9339] mysqld(+0x864976)[0x5584671b5976] mysqld(+0x8738e2)[0x5584671c48e2] mysqld(+0x8c9423)[0x55846721a423] mysqld(+0x8cc5c3)[0x55846721d5c3] mysqld(+0x8cac65)[0x55846721bc65] mysqld(+0x8cc5c3)[0x55846721d5c3] mysqld(+0x8b294f)[0x55846720394f] mysqld(+0x8b53fe)[0x5584672063fe] mysqld(+0x81d35f)[0x55846716e35f] mysqld(+0x73cfed)[0x55846708dfed] mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x64)[0x558466f0b2e4] mysqld(+0x4471d5)[0x558466d981d5] mysqld(_Z11plugin_initPiPPci+0x9ca)[0x558466d99aca] mysqld(+0x3a0278)[0x558466cf1278] mysqld(_Z11mysqld_mainiPPc+0x1bb0)[0x558466cf4bb0] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f3f29418b35] mysqld(+0x397b5d)[0x558466ce8b5d] After restarting the vfc-mariadb-0 pod got a different error: 2020-04-03 0:29:54 139920503318784 [Note] InnoDB: Initializing buffer pool, size = 128.0M 2020-04-03 0:29:54 139920503318784 [Note] InnoDB: Completed initialization of buffer pool 2020-04-03 0:29:54 139920503318784 [Note] InnoDB: Highest supported file format is Barracuda. 2020-04-03 0:29:54 139920503318784 [Note] InnoDB: The log sequence numbers 0 and 0 in ibdata files do not match the log sequence number 59916 in t he ib_logfiles! 2020-04-03 0:29:54 139920503318784 [Note] InnoDB: Restoring possible half-written data pages from the doublewrite buffer... 2020-04-03 0:29:54 139920503318784 [Note] InnoDB: 1 rollback segment(s) are active. 2020-04-03 0:29:54 139920503318784 [Note] InnoDB: Waiting for purge to start 2020-04-03 0:29:54 139920503318784 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.36-82.0 started; log sequence number 59916 2020-04-03 0:29:54 139919498409728 [Note] InnoDB: Dumping buffer pool(s) not yet started 2020-04-03 0:29:54 139920503318784 [Note] Plugin 'FEEDBACK' is disabled. 2020-04-03 0:29:54 139920503318784 [ERROR] Could not open mysql.plugin table. Some plugins may be not loaded 2020-04-03 0:29:54 139920503318784 [Note] Recovering after a crash using tc.log 2020-04-03 0:29:54 139920503318784 [Note] Starting crash recovery... 2020-04-03 0:29:54 139920503318784 [Note] Crash recovery finished. 2020-04-03 0:29:54 139920503318784 [ERROR] Can't open and lock privilege tables: Table 'mysql.servers' doesn't exist 2020-04-03 0:29:54 139920503318784 [Note] Server socket created on IP: '0.0.0.0'. 2020-04-03 0:29:54 139920503318784 [ERROR] Fatal error: Can't open and lock privilege tables: Table 'mysql.user' doesn't exist Best Regards, -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#20473): https://lists.onap.org/g/onap-discuss/message/20473 Mute This Topic: https://lists.onap.org/mt/72739369/21656 Mute #elalto: https://lists.onap.org/mk?hashtag=elalto&subid=2740164 Mute #mariadb: https://lists.onap.org/mk?hashtag=mariadb&subid=2740164 Mute #oom: https://lists.onap.org/mk?hashtag=oom&subid=2740164 Mute #vfc: https://lists.onap.org/mk?hashtag=vfc&subid=2740164 Group Owner: [email protected] Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
