You are going to have to dump the symbols using the RPMs that match this version here: /opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/impala/ sbin-retail/impalad
Follow "Extracting from RPMs" section of this guide: https://cwiki.apache.org/confluence/display/IMPALA/Debugging+Impala+Minidumps On Wed, Nov 29, 2017 at 12:28 AM, [email protected] < [email protected]> wrote: > > > On 2017-11-29 10:31, Vincent Tran <[email protected]> wrote: > > Hi Sandish, > > > > Did you get the minidump from the minicluster of this private build? It > > looks like the hash of your private build symbols and the one from the md > > do not match. > > > > You can turn the md into a core to see the original binary version that > > generated it: > > > > <breakpad-home>/bin/minidump-2-core > > 11850fc8-c7ec-dd15-15ba7314-1926301a.dmp > > > 11850fc8-c7ec-dd15-15ba7314-1926301a.core > > If you open the resulting core file in gdb, what does the last line say? > > > > > > > > > > > > On Nov 28, 2017 11:36 PM, "[email protected]" <[email protected] > > > > wrote: > > > > > > > > > > > On 2017-11-28 21:01, Lars Volker <[email protected]> wrote: > > > > The resolved minidump looks truncated, as does the output of minidump > > > > stackwalk. This could indicate that the minidump got corrupted. How > did > > > you > > > > generate the minidump? > > > > > > > > On Nov 28, 2017 02:20, "[email protected]" < > [email protected]> > > > > wrote: > > > > > > > > > > > > > > Thank you, Lars Volker, for quick response's > > > > > > > > > > The end content of /tmp/resolved.txt file: > > > > > > > > > > [ec2-user@jenkins tmp]$ tail -20f /tmp/resolved.txt > > > > > rbp = 0x00007fd7d4fdf220 rsp = 0x00007fc10d6ad270 > > > > > r8 = 0x00007fd7d4fdf220 r9 = 0x00000000ffffffff > > > > > r10 = 0x00007fc10d6ad3d0 r11 = 0x0000000000000202 > > > > > r12 = 0x0000000000000006 r13 = 0x00007fc10d6ad3d0 > > > > > r14 = 0xffffffffffffff92 r15 = 0x00007fc10d6ad400 > > > > > rip = 0x00007fdb5e2ffa82 > > > > > Found by: given as instruction pointer in context > > > > > > > > > > Thread 960 > > > > > 0 0x7fdb5e2ffa82 > > > > > rax = 0xfffffffffffffdfc rdx = 0x0000000000000001 > > > > > rcx = 0xffffffffffffffff rbx = 0x0000000010d33e20 > > > > > rsi = 0x0000000000000189 rdi = 0x0000000010d33e4c > > > > > rbp = 0x00007fc0f3e7a740 rsp = 0x00007fc0f3e7a6b0 > > > > > r8 = 0x0000000010d33e20 r9 = 0x00000000ffffffff > > > > > r10 = 0x00007fc0f3e7a740 r11 = 0x0000000000000202 > > > > > r12 = 0x0000000000000001 r13 = 0x00007fc0f3e7a740 > > > > > r14 = 0xffffffffffffff92 r15 = 0x0000000022217000 > > > > > rip = 0x00007fdb5e2ffa82 > > > > > Found by: given as instruction pointer in context > > > > > > > > > > Output of find syms/impalad: > > > > > [ec2-user@jenkins tmp]$ find syms/impalad > > > > > syms/impalad > > > > > syms/impalad/73318DB4729C07530B7338E67EA255AB0 > > > > > syms/impalad/73318DB4729C07530B7338E67EA255AB0/impalad.sym > > > > > > > > > > These are the steps I had followed: > > > > > Step-1: > > > > > I followed exact steps from this https://cwiki.apache.org/confl > > > > > uence/display/IMPALA/Building+Impala and also followed "build > > > > > native-toolchain from scratch" https://cwiki.apache.org/confl > > > > > uence/display/IMPALA/Building+native-toolchain+from+scratch+ > > > > > and+using+with+Impala > > > > > > > > > > Step-2: > > > > > Env: > > > > > export IMPALA_HOME=/home/ec2-user/impala-setup/Impala > > > > > export PATH=$PATH:$IMPALA_HOME/bin > > > > > export KUDU_IS_SUPPORTED=false > > > > > export KUDU_CLIENT_DIR=/opt/kudu/ > > > > > > > > > > Get breakpad symbols: > > > > > dump_breakpad_symbols.py -b be/build/ --dump_syms > > > > > $IMPALA_HOME/toolchain/breakpad-1b704857f1e78a864e6942e61345 > > > 7e55f1aecb60-p3/bin/dump_syms > > > > > -d /tmp/syms > > > > > > > > > > rm /tmp/resolved.txt > > > > > > > > > > Run minidump_stackwalk: > > > > > $IMPALA_HOME/toolchain/breakpad-1b704857f1e78a864e6942e61345 > > > > > 7e55f1aecb60-p3/bin/minidump_stackwalk > /home/ec2-user/64811560-8bae-a > > > 67f-22f961c1-45bf52a3.dmp > > > > > /tmp/syms > /tmp/resolved.txt > > > > > > > > > > > > > > > Please let me know If I have missed any steps?? > > > > > > > > > > > > > > > > > > > Thanks you Lars Volker, > > > I took few new .dmp files and ran through stackwalk. > > > here is last few lines from the resolver: > > > > > > Loaded modules: > > > 0x00400000 - 0x0283efff impalad ??? (main) (WARNING: No symbols, > > > impalad, 21AA452CEF0F8C034C7E78A639190ED40) > > > 0x7fce0a33b000 - 0x7fce0a543fff libgssapiv2.so.3.0.0 ??? > > > 0x7fce0a544000 - 0x7fce0a748fff libplain.so.3.0.0 ??? > > > 0x7fce0a749000 - 0x7fce0a94dfff liblogin.so.3.0.0 ??? > > > 0x7fce0a94e000 - 0x7fce0ad0bfff libdb-5.3.so ??? > > > 0x7fce0ad0c000 - 0x7fce0af12fff libsasldb.so.3.0.0 ??? > > > 0x7fce0af13000 - 0x7fce0b118fff libanonymous.so.3.0.0 ??? > > > 0x7fce0b919000 - 0x7fce0bb79fff libpcre.so.1.2.0 ??? > > > 0x7fce0bb7a000 - 0x7fce0bd9efff libselinux.so.1 ??? > > > 0x7fce0bda1000 - 0x7fce0bfa4fff libkeyutils.so.1.5 ??? > > > 0x7fce0bfa5000 - 0x7fce0c1b2fff libkrb5support.so.0.1 ??? > > > 0x7fce0c1b3000 - 0x7fce0c3b5fff libfreebl3.so ??? > > > 0x7fce0c3b6000 - 0x7fce0c5cbfff libz.so.1.2.7 ??? > > > 0x7fce0c5cc000 - 0x7fce0c7fefff libk5crypto.so.3.1 ??? > > > 0x7fce0c7ff000 - 0x7fce0ca02fff libcom_err.so.2.1 ??? > > > 0x7fce0ca03000 - 0x7fce0cceafff libkrb5.so.3.3 ??? > > > 0x7fce0cceb000 - 0x7fce0cf37fff libgssapi_krb5.so.2.2 ??? > > > 0x7fce0cf38000 - 0x7fce0d140fff libcrypt-2.17.so ??? > > > 0x7fce0d16f000 - 0x7fce0d386fff libresolv-2.17.so ??? > > > 0x7fce0d389000 - 0x7fce0d746fff libc-2.17.so ??? (WARNING: No > symbols, > > > libc-2.17.so, E42D1DFF35CF1831960C5EA195CEB1360) > > > 0x7fce0d74c000 - 0x7fce0d963fff libpthread-2.17.so ??? (WARNING: No > > > symbols, libpthread-2.17.so, 9B139160D961AA018425DBC16B7F51D40) > > > > > > I think I'm still not getting symbols for impala?? > > > > > > > > > > > > Hi Vincent Tran, > > Thanks for response. No the dmp file is from different CDH cluster. > > Core file last few lines: > > > ??9?k ??ol ?I?k ??ւ?~W???eM?x86_64/opt/ > cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/impala/ > sbin-retail/impalad--flagfile=/run/cloudera-scm-agent/ > process/2619-impala-IMPALAD/impala-conf/impalad_flagsCDH_ > HCAT_HOME=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hive- > hcatalogTOMCAT_HOME=/opt/cloudera/parcels/CDH-5.11.0-1. > cdh5.11.0.p0.34/lib/bigtop-tomcatXDG_SESSION_ID=c4CDH_ > PIG_HOME=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11. > 0.p0.34/lib/pigCDH_SOLR_HOME=/opt/cloudera/parcels/CDH-5.11. > 0-1.cdh5.11.0.p0.34/lib/solrPARCELS_ROOT=/opt/cloudera/parcelsSHELL=/bin/ > bashCLOUDERA_MYSQL_CONNECTOR_JAR=/usr/share/java/mysql- > connector-java.jarGLOG_logbuflevel=0CDH_HUE_PLUGINS_ > HOME=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/ > lib/hadoopCDH_HIVE_HOME=/opt/cloudera/parcels/CDH-5.11.0-1. > cdh5.11.0.p0.34/lib/hiveSCM_DEFINES_SCRIPTS=/opt/cloudera/ > parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/meta/cdh_env.shCDH_ > AVRO_HOME=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11. > 0.p0.34/lib/avroCLOUDERA_ORAC > LE_CONNECTOR_JAR=/usr/share/java/oracle-connector-java. > jarCDH_VERSION=5GLOG_minloglevel=0HIVE_DEFAULT_XML= > /etc/hive/conf.dist/hive-default.xmlSEARCH_HOME=/opt/ > cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/searchCMF_ > CONF_DIR=/etc/cloudera-scm-agentMGMT_HOME=/usr/share/ > cmfCGROUP_GROUP_BLKIO=IMPALA_BIN=/opt/cloudera/parcels/CDH- > 5.11.0-1.cdh5.11.0.p0.34/lib/impala/sbin-retailPARCEL_ > DIRNAMES=CDH-5.11.0-1.cdh5.11.0.p0.34USER=impalaLD_LIBRARY_ > PATH=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/ > lib/impala/lib:/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11. > 0.p0.34/lib/impala/sbin-retail:/usr/java/jdk1.8.0_131/ > jre/lib/amd64:/usr/java/jdk1.8.0_131/jre/lib/amd64/server:/ > usr/java/jdk1.8.0_131/jre/lib/amd64/server:JSVC_HOME=/opt/ > cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/bigtop- > utilsCDH_SQOOP_HOME=/usr/lib/sqoopAUX_CLASSPATH=/usr/share/ > java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql- > 9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java. > jarHBASE_HOME=/opt/cloudera/parcels/ > CDH-5.11.0-1.cdh5.11.0.p0.34/bin/../lib/hbaseGLOG_max_log_ > size=200CDH_IMPALA_HOME=/opt/cloudera/parcels/CDH-5.11.0-1. > cdh5.11.0.p0.34/lib/impalaCDH_HTTPFS_HOME=/opt/cloudera/ > parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-httpfsCDH_ > MR2_HOME=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop- > mapreduceCM_STATUS_CODES=STATUS_NONE HDFS_DFS_DIR_NOT_EMPTY > HBASE_TABLE_DISABLED HBASE_TABLE_ENABLED JOBTRACKER_IN_STANDBY_MODE > YARN_RM_IN_STANDBY_MODECDH_HUE_HOME=/opt/cloudera/ > parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hueCDH_MR1_HOME=/ > opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/ > hadoop-0.20-mapreduceCGROUP_ROOT_MEMORY=/sys/fs/cgroup/ > memoryGLOG_v=1PATH=/sbin:/usr/sbin:/bin:/usr/binCONF_DIR=/ > run/cloudera-scm-agent/process/2619-impala-IMPALADCGROUP_GROUP_MEMORY= > CDH_KAFKA_HOME=/usr/lib/kafkaCDH_SENTRY_HOME=/opt/ > cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/ > sentryHIVE_HOME=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11. > 0.p0.34/bin/../lib/hiveCDH_PARQUET_HOME=/opt/cloudera/parcels/C > DH-5.11.0-1.cdh5.11.0.p0.34/lib/parquetSUPERVISOR_GROUP_NAME=2619-impala- > IMPALADIMPALA_HOME=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/ > impalaHBASE_CONF_DIR=/run/cloudera-scm-agent/process/ > 2619-impala-IMPALAD/hbase-confPWD=/run/cloudera-scm- > agent/process/2619-impala-IMPALADJAVA_HOME=/usr/java/ > jdk1.8.0_131LANG=en_US.UTF-8HADOOP_CONF_DIR=/run/ > cloudera-scm-agent/process/2619-impala-IMPALAD/hadoop- > confCGROUP_ROOT_CPUACCT=/sys/fs/cgroup/cpu,cpuacctCDH_ > HADOOP_HOME=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11. > 0.p0.34/lib/hadoopCGROUP_ROOT_BLKIO=/sys/fs/cgroup/ > blkioCGROUP_ROOT_CPU=/sys/fs/cgroup/cpu,cpuacctCDH_SQOOP2_ > HOME=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/ > lib/sqoop2CDH_SPARK_HOME=/opt/cloudera/parcels/CDH-5.11.0-1. > cdh5.11.0.p0.34/lib/sparkCDH_CRUNCH_HOME=/usr/lib/ > crunchCDH_LLAMA_HOME=/opt/cloudera/parcels/CDH-5.11.0-1. > cdh5.11.0.p0.34/lib/llamaJAVA_TOOL_OPTIONS=CDH_HADOOP_BIN=/ > opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/ > hadoop/bin/hadoopCDH_HDFS_HOME=/opt/ > cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop- > hdfsSUPERVISOR_ENABLED=1LIBHDFS_OPTS=-Djava.library. > path=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/ > bin/../lib/impala/libHOME=/var/lib/impalaSHLVL=2CDH_ > OOZIE_HOME=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11. > 0.p0.34/lib/oozieKEYTRUSTEE_SERVER_HOME=/usr/lib/ > keytrustee-serverCDH_HBASE_INDEXER_HOME=/opt/cloudera/parcels > -- Vincent T. Tran Customer Operations Engineer Cloudera, Inc.
