Hi Paulo Duarte,
You can try and perform the following steps to restart mariadb pod:
cd /dockerdata-nfs/dev-vfc/vfc/
rm -rf data*
kubectl -n onap get pod |grep vfc-mariadb|awk '{print $1}'|xargs kubectl -n
onap delete pod
BR,
Yan
发件人: [email protected] [mailto:[email protected]] 代表 Paulo
Duarte
发送时间: 2020年4月3日 8:39
收件人: [email protected]
主题: [onap-discuss] OOM VF-C Mariadb pods deployment issue #mariadb #oom #vfc
#elalto
Hi ONAP experts,
Trying to deploy VF-C using OOM El Alto release. I'm getting some error during
the initialization of mariadb related pods.
Pod vfc-vfc-mariadb-0 keeps at CrashLoopBackOff status.
I would like to know if someone else experienced this behavior before and if
yes, if could share the workaround.
Thank you!
Please find below the logs from vfc-mariadb-0:
onapit-vfc-vfc-catalog-9d8b769ff-d8zbv 0/2 Init:0/1 1 13m
onapit-vfc-vfc-ems-driver-756654cccf-6gcqr 1/1 Running 0 13m
onapit-vfc-vfc-generic-vnfm-driver-547f6c5678-2n7lc 2/2 Running 0 13m
onapit-vfc-vfc-huawei-vnfm-driver-7b665b768d-6qz9f 2/2 Running 0 13m
onapit-vfc-vfc-juju-vnfm-driver-554f4f8478-zfv57 2/2 Running 2 13m
onapit-vfc-vfc-mariadb-0 0/1 CrashLoopBackOff 10 13m
onapit-vfc-vfc-multivim-proxy-676d5c6bc9-mbvqm 1/1 Running 1 13m
onapit-vfc-vfc-nokia-v2vnfm-driver-897545dc4-2b7d9 1/1 Running 1 13m
onapit-vfc-vfc-nslcm-96cc687c9-bvp4s 0/2 Init:0/1 1 13m
onapit-vfc-vfc-redis-687bd5866c-qvlqr 1/1 Running 0 13m
onapit-vfc-vfc-resmgr-859b596b9d-294q7 2/2 Running 2 13m
onapit-vfc-vfc-vnflcm-698b5df694-mqp96 0/2 Init:0/1 1 13m
onapit-vfc-vfc-vnfmgr-89d5749b6-9dvtt 0/2 Init:0/1 1 13m
onapit-vfc-vfc-vnfres-7545b49cff-lmxgz 0/2 Init:0/1 1 13m
onapit-vfc-vfc-workflow-798d598bcd-brzwx 1/1 Running 1 13m
onapit-vfc-vfc-workflow-engine-5c6d7d6495-4g8cv 1/1 Running 0 13m
onapit-vfc-vfc-zte-sdnc-driver-c64b7cdfb-l6tb8 1/1 Running 0 13m
onapit-vfc-vfc-zte-vnfm-driver-6695754b4c-dzswf 2/2 Running 2 13m
>From logs of pod I got the following errors before restart the VM:
2020-04-03 0:10:26 139909286091008 [ERROR] InnoDB: Tried to read 16384 bytes at
offset 802816. Was only able to read 0. 2020-04-03 00:10:26 7f3f2b4c5900
InnoDB: Operating system error number 5 in a file operation. InnoDB: Error
number 5 means 'Input/output error'. InnoDB: Some operating system error
numbers are described at InnoDB:
http://dev.mysql.com/doc/refman/5.6/en/operating-system-error-codes.html
InnoDB: Operation read to file
/home/buildbot/buildbot/build/mariadb-10.1.24/storage/xtradb/os/os0file.cc and
at line 3175 2020-04-03 0:10:26 139909286091008 [ERROR] InnoDB: File (unknown):
'read' returned OS error 205. Cannot continue operation 200403 0:10:26 [ERROR]
mysqld got signal 6 ; This could be because you hit a bug. It is also possible
that this binary or one of the libraries it was linked against is corrupt,
improperly built, or misconfigured. This error can also be caused by
malfunctioning hardware. To report this bug, see
https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some
info that will hopefully help diagnose the problem, but since we have already
crashed, something is definitely wrong and this may fail. Server version:
10.1.24-MariaDB key_buffer_size=134217728 read_buffer_size=131072
max_used_connections=0 max_threads=153 thread_count=2 It is possible that
mysqld could use up to key_buffer_size + (read_buffer_size +
sort_buffer_size)*max_threads = 467125 K bytes of memory Hope that's ok; if
not, decrease some variables in the equation. Thread pointer: 0x0 Attempting
backtrace. You can use the following information to find out where mysqld died.
If you see no messages after this, something went terribly wrong...
stack_bottom = 0x0 thread_stack 0x48400 addr2line: 'mysqld': No such file
mysqld(my_print_stacktrace+0x2e)[0x5584673e333e]
mysqld(handle_fatal_signal+0x305)[0x558466f090a5]
/lib64/libpthread.so.0(+0xf370)[0x7f3f2b0d7370]
/lib64/libc.so.6(gsignal+0x37)[0x7f3f2942c1d7]
/lib64/libc.so.6(abort+0x148)[0x7f3f2942d8c8] mysqld(+0x79b179)[0x5584670ec179]
mysqld(+0x79fb6c)[0x5584670f0b6c] mysqld(+0x7a27db)[0x5584670f37db]
mysqld(+0x8e9d67)[0x55846723ad67] mysqld(+0x8aa3ff)[0x5584671fb3ff]
mysqld(+0x888339)[0x5584671d9339] mysqld(+0x864976)[0x5584671b5976]
mysqld(+0x8738e2)[0x5584671c48e2] mysqld(+0x8c9423)[0x55846721a423]
mysqld(+0x8cc5c3)[0x55846721d5c3] mysqld(+0x8cac65)[0x55846721bc65]
mysqld(+0x8cc5c3)[0x55846721d5c3] mysqld(+0x8b294f)[0x55846720394f]
mysqld(+0x8b53fe)[0x5584672063fe] mysqld(+0x81d35f)[0x55846716e35f]
mysqld(+0x73cfed)[0x55846708dfed]
mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x64)[0x558466f0b2e4]
mysqld(+0x4471d5)[0x558466d981d5]
mysqld(_Z11plugin_initPiPPci+0x9ca)[0x558466d99aca]
mysqld(+0x3a0278)[0x558466cf1278]
mysqld(_Z11mysqld_mainiPPc+0x1bb0)[0x558466cf4bb0]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f3f29418b35]
mysqld(+0x397b5d)[0x558466ce8b5d]
After restarting the vfc-mariadb-0 pod got a different error:
2020-04-03 0:29:54 139920503318784 [Note] InnoDB: Initializing buffer pool,
size = 128.0M
2020-04-03 0:29:54 139920503318784 [Note] InnoDB: Completed initialization of
buffer pool
2020-04-03 0:29:54 139920503318784 [Note] InnoDB: Highest supported file format
is Barracuda.
2020-04-03 0:29:54 139920503318784 [Note] InnoDB: The log sequence numbers 0
and 0 in ibdata files do not match the log sequence number 59916 in t
he ib_logfiles!
2020-04-03 0:29:54 139920503318784 [Note] InnoDB: Restoring possible
half-written data pages from the doublewrite buffer...
2020-04-03 0:29:54 139920503318784 [Note] InnoDB: 1 rollback segment(s) are
active.
2020-04-03 0:29:54 139920503318784 [Note] InnoDB: Waiting for purge to start
2020-04-03 0:29:54 139920503318784 [Note] InnoDB: Percona XtraDB
(http://www.percona.com) 5.6.36-82.0 started; log sequence number 59916
2020-04-03 0:29:54 139919498409728 [Note] InnoDB: Dumping buffer pool(s) not
yet started
2020-04-03 0:29:54 139920503318784 [Note] Plugin 'FEEDBACK' is disabled.
2020-04-03 0:29:54 139920503318784 [ERROR] Could not open mysql.plugin table.
Some plugins may be not loaded
2020-04-03 0:29:54 139920503318784 [Note] Recovering after a crash using tc.log
2020-04-03 0:29:54 139920503318784 [Note] Starting crash recovery...
2020-04-03 0:29:54 139920503318784 [Note] Crash recovery finished.
2020-04-03 0:29:54 139920503318784 [ERROR] Can't open and lock privilege
tables: Table 'mysql.servers' doesn't exist
2020-04-03 0:29:54 139920503318784 [Note] Server socket created on IP:
'0.0.0.0'.
2020-04-03 0:29:54 139920503318784 [ERROR] Fatal error: Can't open and lock
privilege tables: Table 'mysql.user' doesn't exist
Best Regards,
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#20474): https://lists.onap.org/g/onap-discuss/message/20474
Mute This Topic: https://lists.onap.org/mt/72740869/21656
Mute #oom: https://lists.onap.org/mk?hashtag=oom&subid=2740164
Mute #vfc: https://lists.onap.org/mk?hashtag=vfc&subid=2740164
Mute #elalto: https://lists.onap.org/mk?hashtag=elalto&subid=2740164
Mute #mariadb: https://lists.onap.org/mk?hashtag=mariadb&subid=2740164
Group Owner: [email protected]
Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
