Re: [Gluster-devel] gfapi 3.6.3 QEMU 2.3 Ubuntu 14.04 testing
Hi Josh Boon, Can you provide me the load test simulation steps in detailed so that I will try to reproduce the scenario. I am planning to run Iozone in VM, how about this? Well, I can also give a session of my machine through tmate, but that involves some process to be completed, so we can think of that later. I am expecting some kind of scripts that helped you to produce segfaults in QEMU 2.0 + Ubuntu 14.04 + gluster 3.6.3 (libgfapi). Thanks Regards, Prasanna Kumar K. - Original Message - From: Josh Boon glus...@joshboon.com To: Prasanna Kalever pkale...@redhat.com Cc: Gluster Devel gluster-devel@gluster.org Sent: Wednesday, July 29, 2015 8:15:02 PM Subject: Re: [Gluster-devel] gfapi 3.6.3 QEMU 2.3 Ubuntu 14.04 testing Hey Prasanna, Thanks for your help! One of the issues we've had is DD doesn't seem to reproduce it. Anything that logs and handles large volumes, think mail and web servers, tends to segfault the most frequently. I could write up a load test and we could put apache on it and try that as that's closet to what we run. Also if you don't object would I be able to get on the machine to figure out apparmor and do a writeup? Most folks probably won't be able to disable it completely. Best, Josh - Original Message - From: Prasanna Kalever pkale...@redhat.com To: Josh Boon glus...@joshboon.com Cc: Pranith Kumar Karampuri pkara...@redhat.com, Gluster Devel gluster-devel@gluster.org Sent: Wednesday, July 29, 2015 1:54:34 PM Subject: Re: [Gluster-devel] gfapi 3.6.3 QEMU 2.3 Ubuntu 14.04 testing Hi Josh Boon, Below are my setup details: # qemu-system-x86_64 --version QEMU emulator version 2.3.0 (Debian 1:2.3+dfsg-5ubuntu4), Copyright (c) 2003-2008 Fabrice Bellard # gluster --version glusterfs 3.6.3 built on Jul 29 2015 16:01:10 Repository revision: git://git.gluster.com/glusterfs.git Copyright (c) 2006-2011 Gluster Inc. http://www.gluster.com GlusterFS comes with ABSOLUTELY NO WARRANTY. # lsb_release -a You may redistribute copies of GlusterFS under the terms of the GNU General Public License. Distributor ID: Ubuntu Description:Ubuntu 14.04 LTS Release:14.04 Codename: trusty # gluster vol info Volume Name: vol1 Type: Replicate Volume ID: ad78ac6c-c55e-4f4a-8b1b-a11865f1d01e Status: Started Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: 10.70.1.156:/brick1
Re: [Gluster-devel] gfapi 3.6.3 QEMU 2.3 Ubuntu 14.04 testing
Hey Prasanna, Thanks for your help! One of the issues we've had is DD doesn't seem to reproduce it. Anything that logs and handles large volumes, think mail and web servers, tends to segfault the most frequently. I could write up a load test and we could put apache on it and try that as that's closet to what we run. Also if you don't object would I be able to get on the machine to figure out apparmor and do a writeup? Most folks probably won't be able to disable it completely. Best, Josh - Original Message - From: Prasanna Kalever pkale...@redhat.com To: Josh Boon glus...@joshboon.com Cc: Pranith Kumar Karampuri pkara...@redhat.com, Gluster Devel gluster-devel@gluster.org Sent: Wednesday, July 29, 2015 1:54:34 PM Subject: Re: [Gluster-devel] gfapi 3.6.3 QEMU 2.3 Ubuntu 14.04 testing Hi Josh Boon, Below are my setup details: # qemu-system-x86_64 --version QEMU emulator version 2.3.0 (Debian 1:2.3+dfsg-5ubuntu4), Copyright (c) 2003-2008 Fabrice Bellard # gluster --version glusterfs 3.6.3 built on Jul 29 2015 16:01:10 Repository revision: git://git.gluster.com/glusterfs.git Copyright (c) 2006-2011 Gluster Inc. http://www.gluster.com GlusterFS comes with ABSOLUTELY NO WARRANTY. # lsb_release -a You may redistribute copies of GlusterFS under the terms of the GNU General Public License. Distributor ID: Ubuntu Description:Ubuntu 14.04 LTS Release:14.04 Codename: trusty # gluster vol info Volume Name: vol1 Type: Replicate Volume ID: ad78ac6c-c55e-4f4a-8b1b-a11865f1d01e Status: Started Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: 10.70.1.156:/brick1 Brick2: 10.70.1.156:/brick2 Options Reconfigured: server.allow-insecure: on storage.owner-uid: 116 storage.owner-gid: 125
Re: [Gluster-devel] gfapi 3.6.3 QEMU 2.3 Ubuntu 14.04 testing
49152 Y 3726 Brick 10.70.1.156:/brick2 49153 Y 7014 NFS Server on localhost 2049Y 7028 Self-heal Daemon on localhost N/A Y 7035 Task Status of Volume vol1 -- There are no active volume tasks # du -sh /brick1/vm1.img 8.6G/brick1/vm1.img # du -sh /brick2/vm1.img 8.6G/brick2/vm1.img # cat vm1.xml ... devices emulator/usr/bin/qemu-system-x86_64/emulator disk type='network' device='disk' driver name='qemu' type='qcow2' cache='none'/ source protocol='gluster' name='vol1/vm1.img' host name='10.70.1.156' port='24007'/ /source target dev='vda' bus='virtio'/ alias name='virtio-disk0'/ address type='pci' domain='0x' bus='0x00' slot='0x04' function='0x0'/ /disk disk type='file' device='cdrom' driver name='qemu' type='raw'/ source file='/home/pkalever/work/qemu/ubuntu-14.04-server-amd64.iso'/ target dev='hdb' bus='ide'/ readonly/ alias name='ide0-0-1'/ address type='drive' controller='0' bus='0' target='0' unit='1'/ /disk ... /devices With the underline setup given above, Once the VM booted successfully I have wrote 5G using dd command but I have not encountered any crashes I feel qemu- 2.3.0 doesn't have the segfaults issue. Please revert back for any information if required. Thanks regards, Prasanna Kumar K. - Original Message - From: Prasanna Kalever pkale...@redhat.com To: Josh Boon glus...@joshboon.com Cc: Pranith Kumar Karampuri pkara...@redhat.com, SATHEESARAN sasun...@redhat.com Sent: Tuesday, July 28, 2015 5:04:46 PM Subject: Re: [Gluster-devel] gfapi 3.6.3 QEMU 2.3 Ubuntu 14.04 testing Hi Josh Boon, finally able to boot the VM for now I disabled apparmor by update-rc.d -f apparmor remove Thanks for the support, Now I shall try on reproducing the actual problem :) Best Regards, Prasanna Kumar K. - Original Message - From: Prasanna Kalever pkale...@redhat.com To: Josh Boon glus...@joshboon.com Cc: Pranith Kumar Karampuri pkara...@redhat.com, SATHEESARAN sasun...@redhat.com Sent: Tuesday, July 28, 2015 3:45:12 PM Subject: Re: [Gluster-devel] gfapi 3.6.3 QEMU 2.3 Ubuntu 14.04 testing Hi Josh Boon, apparmor trick didn't work
Re: [Gluster-devel] gfapi 3.6.3 QEMU 2.3 Ubuntu 14.04 testing
CC Prasanna who will be looking into it. On 07/06/2015 07:30 PM, Josh Boon wrote: Hey folks, Does anyone have test environment running Ubuntu 14.04, QEMU 2.0, and Gluster 3.6.3? I'm looking to have some folks test out QEMU 2.3 for stability and performance and see if it removes the segfault errors. Another group of folks are experiencing the same segfaults I still experience but looking over their logs my theory of it being related to a self-heal didn't work out. I've included the stack trace below from their environment which matches mine. I've already put together a PPA over at https://launchpad.net/~josh-boon/+archive/ubuntu/qemu-edge-glusterfs with QEMU 2.3 and deps built for trusty. If anyone has the time or the resources that I could get into I'd appreciate the support. I'd like to get this ironed out so I can give my full vote of confidence to Gluster again. Thanks, Josh Stack #0 0x7f369c95248c in ?? () No symbol table info available. #1 0x7f369bd2b3b1 in glfs_io_async_cbk (ret=optimized out, frame=optimized out, data=0x7f369ee536c0) at glfs-fops.c:598 gio = 0x7f369ee536c0 #2 0x7f369badb66a in syncopctx_setfspid (pid=0x7f369ee536c0) at syncop.c:191 opctx = 0x0 ret = -1 #3 0x00100011 in ?? () No symbol table info available. #4 0x7f36a5ae26b0 in ?? () No symbol table info available. #5 0x7f36a81e2800 in ?? () No symbol table info available. #6 0x7f36a5ae26b0 in ?? () No symbol table info available. #7 0x7f36a81e2800 in ?? () No symbol table info available. #8 0x in ?? () No symbol table info available. Full log attached. ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel
[Gluster-devel] gfapi 3.6.3 QEMU 2.3 Ubuntu 14.04 testing
Hey folks, Does anyone have test environment running Ubuntu 14.04, QEMU 2.0, and Gluster 3.6.3? I'm looking to have some folks test out QEMU 2.3 for stability and performance and see if it removes the segfault errors. Another group of folks are experiencing the same segfaults I still experience but looking over their logs my theory of it being related to a self-heal didn't work out. I've included the stack trace below from their environment which matches mine. I've already put together a PPA over at https://launchpad.net/~josh-boon/+archive/ubuntu/qemu-edge-glusterfs with QEMU 2.3 and deps built for trusty. If anyone has the time or the resources that I could get into I'd appreciate the support. I'd like to get this ironed out so I can give my full vote of confidence to Gluster again. Thanks, Josh Stack #0 0x7f369c95248c in ?? () No symbol table info available. #1 0x7f369bd2b3b1 in glfs_io_async_cbk (ret=optimized out, frame=optimized out, data=0x7f369ee536c0) at glfs-fops.c:598 gio = 0x7f369ee536c0 #2 0x7f369badb66a in syncopctx_setfspid (pid=0x7f369ee536c0) at syncop.c:191 opctx = 0x0 ret = -1 #3 0x00100011 in ?? () No symbol table info available. #4 0x7f36a5ae26b0 in ?? () No symbol table info available. #5 0x7f36a81e2800 in ?? () No symbol table info available. #6 0x7f36a5ae26b0 in ?? () No symbol table info available. #7 0x7f36a81e2800 in ?? () No symbol table info available. #8 0x in ?? () No symbol table info available. Full log attached. ProblemType: Crash Architecture: amd64 CrashCounter: 1 Date: Mon Jun 29 02:45:09 2015 Dependencies: acl 2.2.52-1 adduser 3.113+nmu3ubuntu3 apt-utils 1.0.1ubuntu2.6 attr 1:2.4.47-1ubuntu1 base-passwd 3.5.33 busybox-initramfs 1:1.21.0-1ubuntu1 ca-certificates 20141019ubuntu0.14.04.1 coreutils 8.21-1ubuntu5.1 cpio 2.11+dfsg-1ubuntu1.1 cpu-checker 0.7-0ubuntu4 dbus 1.6.18-0ubuntu4.3 debconf 1.5.51ubuntu2 debconf-i18n 1.5.51ubuntu2 debianutils 4.4 dmsetup 2:1.02.77-6ubuntu2 dpkg 1.17.5ubuntu5.3 e2fslibs 1.42.9-3ubuntu1.2 e2fsprogs 1.42.9-3ubuntu1.2 file 1:5.14-2ubuntu3.3 findutils 4.4.2-7 gcc-4.8-base 4.8.4-2ubuntu1~14.04 gcc-4.9-base 4.9.1-0ubuntu1 glusterfs-common 3.6.3-ubuntu1~trusty10 [origin: LP-PPA-gluster-glusterfs-3.6] ifupdown 0.7.47.2ubuntu4.1 initramfs-tools 0.103ubuntu4.2 initramfs-tools-bin 0.103ubuntu4.2 initscripts 2.88dsf-41ubuntu6 insserv 1.14.0-5ubuntu2 iproute2 3.12.0-2 ipxe-qemu 1.0.0+git-2013.c3d1e78-2ubuntu1.1 isc-dhcp-client 4.2.4-7ubuntu12 isc-dhcp-common 4.2.4-7ubuntu12 klibc-utils 2.0.3-0ubuntu1 kmod 15-0ubuntu6 krb5-locales 1.12+dfsg-2ubuntu5.1 libacl1 2.2.52-1 libaio1 0.3.109-4 libapparmor1 2.8.95~2430-0ubuntu5.1 libapt-inst1.5 1.0.1ubuntu2.6 libapt-pkg4.12 1.0.1ubuntu2.6 libasn1-8-heimdal 1.6~git20131207+dfsg-1ubuntu1.1 libasound2 1.0.27.2-3ubuntu7 libasound2-data 1.0.27.2-3ubuntu7 libasyncns0 0.8-4ubuntu2 libattr1 1:2.4.47-1ubuntu1 libaudit-common 1:2.3.2-2ubuntu1 libaudit1 1:2.3.2-2ubuntu1 libblkid1 2.20.1-5.1ubuntu20.4 libbluetooth3 4.101-0ubuntu13.1 libboost-system1.54.0 1.54.0-4ubuntu3.1 libboost-thread1.54.0 1.54.0-4ubuntu3.1 libbrlapi0.6 5.0-2ubuntu2 libbz2-1.0 1.0.6-5 libc6 2.19-0ubuntu6.6 libcaca0 0.99.beta18-1ubuntu5 libcap-ng0 0.7.3-1ubuntu2 libcap2 1:2.24-0ubuntu2 libcgmanager0 0.24-0ubuntu7.3 libcomerr2 1.42.9-3ubuntu1.2 libcurl3-gnutls 7.35.0-1ubuntu2.3 libdb5.3 5.3.28-3ubuntu3 libdbus-1-3 1.6.18-0ubuntu4.3 libdebconfclient0 0.187ubuntu1 libdevmapper-event1.02.1 2:1.02.77-6ubuntu2 libdevmapper1.02.1 2:1.02.77-6ubuntu2 libdrm2 2.4.56-1~ubuntu2 libexpat1 2.1.0-4ubuntu1 libfdt1 1.4.0+dfsg-1 libffi6 3.1~rc1+r3.0.13-12 libflac8 1.3.0-2ubuntu0.14.04.1 libgcc1 1:4.9.1-0ubuntu1 libgcrypt11 1.5.3-2ubuntu4.1 libglib2.0-0 2.40.2-0ubuntu1 libglib2.0-data 2.40.2-0ubuntu1 libgnutls26 2.12.23-12ubuntu2.2 libgpg-error0 1.12-0.2ubuntu1 libgpm2 1.20.4-6.1 libgssapi-krb5-2 1.12+dfsg-2ubuntu5.1 libgssapi3-heimdal 1.6~git20131207+dfsg-1ubuntu1.1 libhcrypto4-heimdal 1.6~git20131207+dfsg-1ubuntu1.1 libheimbase1-heimdal 1.6~git20131207+dfsg-1ubuntu1.1 libheimntlm0-heimdal 1.6~git20131207+dfsg-1ubuntu1.1 libhx509-5-heimdal 1.6~git20131207+dfsg-1ubuntu1.1 libibverbs1 1.1.7-1ubuntu1 libidn11 1.28-1ubuntu2 libjpeg-turbo8 1.3.0-0ubuntu2 libjpeg8 8c-2ubuntu8 libjson-c2 0.11-3ubuntu1.2 libjson0 0.11-3ubuntu1.2 libk5crypto3 1.12+dfsg-2ubuntu5.1 libkeyutils1 1.5.6-1 libklibc 2.0.3-0ubuntu1 libkmod2 15-0ubuntu6 libkrb5-26-heimdal 1.6~git20131207+dfsg-1ubuntu1.1 libkrb5-3 1.12+dfsg-2ubuntu5.1 libkrb5support0 1.12+dfsg-2ubuntu5.1 libldap-2.4-2 2.4.31-1+nmu2ubuntu8 liblocale-gettext-perl 1.05-7build3 liblvm2app2.2 2.02.98-6ubuntu2 liblzma5 5.1.1alpha+20120614-2ubuntu2 libmagic1 1:5.14-2ubuntu3.3 libmount1 2.20.1-5.1ubuntu20.4 libncurses5 5.9+20140118-1ubuntu1 libncursesw5 5.9+20140118-1ubuntu1 libnih-dbus1 1.0.3-4ubuntu25 libnih1 1.0.3-4ubuntu25 libnspr4
