On 2017-11-29 11:03, Vincent Tran <[email protected]> wrote: > You are going to have to dump the symbols using the RPMs that match this > version here: > /opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/impala/ > sbin-retail/impalad > > > Follow "Extracting from RPMs" section of this guide: > > https://cwiki.apache.org/confluence/display/IMPALA/Debugging+Impala+Minidumps > > On Wed, Nov 29, 2017 at 12:28 AM, [email protected] < > [email protected]> wrote: > > > > > > > On 2017-11-29 10:31, Vincent Tran <[email protected]> wrote: > > > Hi Sandish, > > > > > > Did you get the minidump from the minicluster of this private build? It > > > looks like the hash of your private build symbols and the one from the md > > > do not match. > > > > > > You can turn the md into a core to see the original binary version that > > > generated it: > > > > > > <breakpad-home>/bin/minidump-2-core > > > 11850fc8-c7ec-dd15-15ba7314-1926301a.dmp > > > > 11850fc8-c7ec-dd15-15ba7314-1926301a.core > > > If you open the resulting core file in gdb, what does the last line say? > > > > > > > > > > > > > > > > > > On Nov 28, 2017 11:36 PM, "[email protected]" <[email protected] > > > > > > wrote: > > > > > > > > > > > > > > > On 2017-11-28 21:01, Lars Volker <[email protected]> wrote: > > > > > The resolved minidump looks truncated, as does the output of minidump > > > > > stackwalk. This could indicate that the minidump got corrupted. How > > did > > > > you > > > > > generate the minidump? > > > > > > > > > > On Nov 28, 2017 02:20, "[email protected]" < > > [email protected]> > > > > > wrote: > > > > > > > > > > > > > > > > > Thank you, Lars Volker, for quick response's > > > > > > > > > > > > The end content of /tmp/resolved.txt file: > > > > > > > > > > > > [ec2-user@jenkins tmp]$ tail -20f /tmp/resolved.txt > > > > > > rbp = 0x00007fd7d4fdf220 rsp = 0x00007fc10d6ad270 > > > > > > r8 = 0x00007fd7d4fdf220 r9 = 0x00000000ffffffff > > > > > > r10 = 0x00007fc10d6ad3d0 r11 = 0x0000000000000202 > > > > > > r12 = 0x0000000000000006 r13 = 0x00007fc10d6ad3d0 > > > > > > r14 = 0xffffffffffffff92 r15 = 0x00007fc10d6ad400 > > > > > > rip = 0x00007fdb5e2ffa82 > > > > > > Found by: given as instruction pointer in context > > > > > > > > > > > > Thread 960 > > > > > > 0 0x7fdb5e2ffa82 > > > > > > rax = 0xfffffffffffffdfc rdx = 0x0000000000000001 > > > > > > rcx = 0xffffffffffffffff rbx = 0x0000000010d33e20 > > > > > > rsi = 0x0000000000000189 rdi = 0x0000000010d33e4c > > > > > > rbp = 0x00007fc0f3e7a740 rsp = 0x00007fc0f3e7a6b0 > > > > > > r8 = 0x0000000010d33e20 r9 = 0x00000000ffffffff > > > > > > r10 = 0x00007fc0f3e7a740 r11 = 0x0000000000000202 > > > > > > r12 = 0x0000000000000001 r13 = 0x00007fc0f3e7a740 > > > > > > r14 = 0xffffffffffffff92 r15 = 0x0000000022217000 > > > > > > rip = 0x00007fdb5e2ffa82 > > > > > > Found by: given as instruction pointer in context > > > > > > > > > > > > Output of find syms/impalad: > > > > > > [ec2-user@jenkins tmp]$ find syms/impalad > > > > > > syms/impalad > > > > > > syms/impalad/73318DB4729C07530B7338E67EA255AB0 > > > > > > syms/impalad/73318DB4729C07530B7338E67EA255AB0/impalad.sym > > > > > > > > > > > > These are the steps I had followed: > > > > > > Step-1: > > > > > > I followed exact steps from this https://cwiki.apache.org/confl > > > > > > uence/display/IMPALA/Building+Impala and also followed "build > > > > > > native-toolchain from scratch" https://cwiki.apache.org/confl > > > > > > uence/display/IMPALA/Building+native-toolchain+from+scratch+ > > > > > > and+using+with+Impala > > > > > > > > > > > > Step-2: > > > > > > Env: > > > > > > export IMPALA_HOME=/home/ec2-user/impala-setup/Impala > > > > > > export PATH=$PATH:$IMPALA_HOME/bin > > > > > > export KUDU_IS_SUPPORTED=false > > > > > > export KUDU_CLIENT_DIR=/opt/kudu/ > > > > > > > > > > > > Get breakpad symbols: > > > > > > dump_breakpad_symbols.py -b be/build/ --dump_syms > > > > > > $IMPALA_HOME/toolchain/breakpad-1b704857f1e78a864e6942e61345 > > > > 7e55f1aecb60-p3/bin/dump_syms > > > > > > -d /tmp/syms > > > > > > > > > > > > rm /tmp/resolved.txt > > > > > > > > > > > > Run minidump_stackwalk: > > > > > > $IMPALA_HOME/toolchain/breakpad-1b704857f1e78a864e6942e61345 > > > > > > 7e55f1aecb60-p3/bin/minidump_stackwalk > > /home/ec2-user/64811560-8bae-a > > > > 67f-22f961c1-45bf52a3.dmp > > > > > > /tmp/syms > /tmp/resolved.txt > > > > > > > > > > > > > > > > > > Please let me know If I have missed any steps?? > > > > > > > > > > > > > > > > > > > > > > > Thanks you Lars Volker, > > > > I took few new .dmp files and ran through stackwalk. > > > > here is last few lines from the resolver: > > > > > > > > Loaded modules: > > > > 0x00400000 - 0x0283efff impalad ??? (main) (WARNING: No symbols, > > > > impalad, 21AA452CEF0F8C034C7E78A639190ED40) > > > > 0x7fce0a33b000 - 0x7fce0a543fff libgssapiv2.so.3.0.0 ??? > > > > 0x7fce0a544000 - 0x7fce0a748fff libplain.so.3.0.0 ??? > > > > 0x7fce0a749000 - 0x7fce0a94dfff liblogin.so.3.0.0 ??? > > > > 0x7fce0a94e000 - 0x7fce0ad0bfff libdb-5.3.so ??? > > > > 0x7fce0ad0c000 - 0x7fce0af12fff libsasldb.so.3.0.0 ??? > > > > 0x7fce0af13000 - 0x7fce0b118fff libanonymous.so.3.0.0 ??? > > > > 0x7fce0b919000 - 0x7fce0bb79fff libpcre.so.1.2.0 ??? > > > > 0x7fce0bb7a000 - 0x7fce0bd9efff libselinux.so.1 ??? > > > > 0x7fce0bda1000 - 0x7fce0bfa4fff libkeyutils.so.1.5 ??? > > > > 0x7fce0bfa5000 - 0x7fce0c1b2fff libkrb5support.so.0.1 ??? > > > > 0x7fce0c1b3000 - 0x7fce0c3b5fff libfreebl3.so ??? > > > > 0x7fce0c3b6000 - 0x7fce0c5cbfff libz.so.1.2.7 ??? > > > > 0x7fce0c5cc000 - 0x7fce0c7fefff libk5crypto.so.3.1 ??? > > > > 0x7fce0c7ff000 - 0x7fce0ca02fff libcom_err.so.2.1 ??? > > > > 0x7fce0ca03000 - 0x7fce0cceafff libkrb5.so.3.3 ??? > > > > 0x7fce0cceb000 - 0x7fce0cf37fff libgssapi_krb5.so.2.2 ??? > > > > 0x7fce0cf38000 - 0x7fce0d140fff libcrypt-2.17.so ??? > > > > 0x7fce0d16f000 - 0x7fce0d386fff libresolv-2.17.so ??? > > > > 0x7fce0d389000 - 0x7fce0d746fff libc-2.17.so ??? (WARNING: No > > symbols, > > > > libc-2.17.so, E42D1DFF35CF1831960C5EA195CEB1360) > > > > 0x7fce0d74c000 - 0x7fce0d963fff libpthread-2.17.so ??? (WARNING: No > > > > symbols, libpthread-2.17.so, 9B139160D961AA018425DBC16B7F51D40) > > > > > > > > I think I'm still not getting symbols for impala?? > > > > > > > > > > > > > > > > > Hi Vincent Tran, > > > > Thanks for response. No the dmp file is from different CDH cluster. > > > > Core file last few lines: > > > > > > ??9?k ??ol ?I?k ??Ö?~W???eM?x86_64/opt/ > > cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/impala/ > > sbin-retail/impalad--flagfile=/run/cloudera-scm-agent/ > > process/2619-impala-IMPALAD/impala-conf/impalad_flagsCDH_ > > HCAT_HOME=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hive- > > hcatalogTOMCAT_HOME=/opt/cloudera/parcels/CDH-5.11.0-1. > > cdh5.11.0.p0.34/lib/bigtop-tomcatXDG_SESSION_ID=c4CDH_ > > PIG_HOME=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11. > > 0.p0.34/lib/pigCDH_SOLR_HOME=/opt/cloudera/parcels/CDH-5.11. > > 0-1.cdh5.11.0.p0.34/lib/solrPARCELS_ROOT=/opt/cloudera/parcelsSHELL=/bin/ > > bashCLOUDERA_MYSQL_CONNECTOR_JAR=/usr/share/java/mysql- > > connector-java.jarGLOG_logbuflevel=0CDH_HUE_PLUGINS_ > > HOME=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/ > > lib/hadoopCDH_HIVE_HOME=/opt/cloudera/parcels/CDH-5.11.0-1. > > cdh5.11.0.p0.34/lib/hiveSCM_DEFINES_SCRIPTS=/opt/cloudera/ > > parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/meta/cdh_env.shCDH_ > > AVRO_HOME=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11. > > 0.p0.34/lib/avroCLOUDERA_ORAC > > LE_CONNECTOR_JAR=/usr/share/java/oracle-connector-java. > > jarCDH_VERSION=5GLOG_minloglevel=0HIVE_DEFAULT_XML= > > /etc/hive/conf.dist/hive-default.xmlSEARCH_HOME=/opt/ > > cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/searchCMF_ > > CONF_DIR=/etc/cloudera-scm-agentMGMT_HOME=/usr/share/ > > cmfCGROUP_GROUP_BLKIO=IMPALA_BIN=/opt/cloudera/parcels/CDH- > > 5.11.0-1.cdh5.11.0.p0.34/lib/impala/sbin-retailPARCEL_ > > DIRNAMES=CDH-5.11.0-1.cdh5.11.0.p0.34USER=impalaLD_LIBRARY_ > > PATH=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/ > > lib/impala/lib:/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11. > > 0.p0.34/lib/impala/sbin-retail:/usr/java/jdk1.8.0_131/ > > jre/lib/amd64:/usr/java/jdk1.8.0_131/jre/lib/amd64/server:/ > > usr/java/jdk1.8.0_131/jre/lib/amd64/server:JSVC_HOME=/opt/ > > cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/bigtop- > > utilsCDH_SQOOP_HOME=/usr/lib/sqoopAUX_CLASSPATH=/usr/share/ > > java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql- > > 9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java. > > jarHBASE_HOME=/opt/cloudera/parcels/ > > CDH-5.11.0-1.cdh5.11.0.p0.34/bin/../lib/hbaseGLOG_max_log_ > > size=200CDH_IMPALA_HOME=/opt/cloudera/parcels/CDH-5.11.0-1. > > cdh5.11.0.p0.34/lib/impalaCDH_HTTPFS_HOME=/opt/cloudera/ > > parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-httpfsCDH_ > > MR2_HOME=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop- > > mapreduceCM_STATUS_CODES=STATUS_NONE HDFS_DFS_DIR_NOT_EMPTY > > HBASE_TABLE_DISABLED HBASE_TABLE_ENABLED JOBTRACKER_IN_STANDBY_MODE > > YARN_RM_IN_STANDBY_MODECDH_HUE_HOME=/opt/cloudera/ > > parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hueCDH_MR1_HOME=/ > > opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/ > > hadoop-0.20-mapreduceCGROUP_ROOT_MEMORY=/sys/fs/cgroup/ > > memoryGLOG_v=1PATH=/sbin:/usr/sbin:/bin:/usr/binCONF_DIR=/ > > run/cloudera-scm-agent/process/2619-impala-IMPALADCGROUP_GROUP_MEMORY= > > CDH_KAFKA_HOME=/usr/lib/kafkaCDH_SENTRY_HOME=/opt/ > > cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/ > > sentryHIVE_HOME=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11. > > 0.p0.34/bin/../lib/hiveCDH_PARQUET_HOME=/opt/cloudera/parcels/C > > DH-5.11.0-1.cdh5.11.0.p0.34/lib/parquetSUPERVISOR_GROUP_NAME=2619-impala- > > IMPALADIMPALA_HOME=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/ > > impalaHBASE_CONF_DIR=/run/cloudera-scm-agent/process/ > > 2619-impala-IMPALAD/hbase-confPWD=/run/cloudera-scm- > > agent/process/2619-impala-IMPALADJAVA_HOME=/usr/java/ > > jdk1.8.0_131LANG=en_US.UTF-8HADOOP_CONF_DIR=/run/ > > cloudera-scm-agent/process/2619-impala-IMPALAD/hadoop- > > confCGROUP_ROOT_CPUACCT=/sys/fs/cgroup/cpu,cpuacctCDH_ > > HADOOP_HOME=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11. > > 0.p0.34/lib/hadoopCGROUP_ROOT_BLKIO=/sys/fs/cgroup/ > > blkioCGROUP_ROOT_CPU=/sys/fs/cgroup/cpu,cpuacctCDH_SQOOP2_ > > HOME=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/ > > lib/sqoop2CDH_SPARK_HOME=/opt/cloudera/parcels/CDH-5.11.0-1. > > cdh5.11.0.p0.34/lib/sparkCDH_CRUNCH_HOME=/usr/lib/ > > crunchCDH_LLAMA_HOME=/opt/cloudera/parcels/CDH-5.11.0-1. > > cdh5.11.0.p0.34/lib/llamaJAVA_TOOL_OPTIONS=CDH_HADOOP_BIN=/ > > opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/ > > hadoop/bin/hadoopCDH_HDFS_HOME=/opt/ > > cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop- > > hdfsSUPERVISOR_ENABLED=1LIBHDFS_OPTS=-Djava.library. > > path=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/ > > bin/../lib/impala/libHOME=/var/lib/impalaSHLVL=2CDH_ > > OOZIE_HOME=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11. > > 0.p0.34/lib/oozieKEYTRUSTEE_SERVER_HOME=/usr/lib/ > > keytrustee-serverCDH_HBASE_INDEXER_HOME=/opt/cloudera/parcels > > > > > > -- > Vincent T. Tran > Customer Operations Engineer > Cloudera, Inc. > Hi Vincent Tran,
I got these RPMs impala-2.8.0+cdh5.11.0+0-1.cdh5.11.0.p0.42.el6.x86_64.rpm impala-debuginfo-2.8.0+cdh5.11.0+0-1.cdh5.11.0.p0.42.el6.x86_64.rpm and generated syms dump_breakpad_symbols.py -r impala-2.8.0+cdh5.11.0+0-1.cdh5.11.0.p0.42.el6.x86_64.rpm -s impala-debuginfo-2.8.0+cdh5.11.0+0-1.cdh5.11.0.p0.42.el6.x86_64.rpm --dump_syms $IMPALA_HOME/toolchain/breakpad-1b704857f1e78a864e6942e613457e55f1aecb60-p3/bin/dump_syms -d /tmp/syms for resolved.txt $IMPALA_HOME/toolchain/breakpad-1b704857f1e78a864e6942e613457e55f1aecb60-p3/bin/minidump_stackwalk /home/ec2-user/impalad/242b57e2-a70a-695b-10ee19a1-79f2bad3.dmp /tmp/syms > /tmp/resolved.txt I'm I missing anything?? Last few lines from /tmp/resolved.txt Loaded modules: 0x00400000 - 0x0283efff impalad ??? (main) (WARNING: No symbols, impalad, 21AA452CEF0F8C034C7E78A639190ED40) 0x7ffa54a7d000 - 0x7ffa54c85fff libgssapiv2.so.3.0.0 ??? 0x7ffa54c86000 - 0x7ffa54e8afff libplain.so.3.0.0 ??? 0x7ffa54e8b000 - 0x7ffa5508ffff liblogin.so.3.0.0 ??? 0x7ffa55090000 - 0x7ffa5544dfff libdb-5.3.so ??? 0x7ffa5544e000 - 0x7ffa55654fff libsasldb.so.3.0.0 ??? 0x7ffa55655000 - 0x7ffa5585afff libanonymous.so.3.0.0 ??? Still same results. Do I need to get exact version of CDH-5.11.0-1.cdh5.11.0.p0.34 RPM'S? can you suggest me some Cloudera repo links where I can find exact version impala rpm's Thanks
