On 2017-11-29 10:31, Vincent Tran <[email protected]> wrote: > Hi Sandish, > > Did you get the minidump from the minicluster of this private build? It > looks like the hash of your private build symbols and the one from the md > do not match. > > You can turn the md into a core to see the original binary version that > generated it: > > <breakpad-home>/bin/minidump-2-core > 11850fc8-c7ec-dd15-15ba7314-1926301a.dmp > > 11850fc8-c7ec-dd15-15ba7314-1926301a.core > If you open the resulting core file in gdb, what does the last line say? > > > > > > On Nov 28, 2017 11:36 PM, "[email protected]" <[email protected]> > wrote: > > > > > > > On 2017-11-28 21:01, Lars Volker <[email protected]> wrote: > > > The resolved minidump looks truncated, as does the output of minidump > > > stackwalk. This could indicate that the minidump got corrupted. How did > > you > > > generate the minidump? > > > > > > On Nov 28, 2017 02:20, "[email protected]" <[email protected]> > > > wrote: > > > > > > > > > > > Thank you, Lars Volker, for quick response's > > > > > > > > The end content of /tmp/resolved.txt file: > > > > > > > > [ec2-user@jenkins tmp]$ tail -20f /tmp/resolved.txt > > > > rbp = 0x00007fd7d4fdf220 rsp = 0x00007fc10d6ad270 > > > > r8 = 0x00007fd7d4fdf220 r9 = 0x00000000ffffffff > > > > r10 = 0x00007fc10d6ad3d0 r11 = 0x0000000000000202 > > > > r12 = 0x0000000000000006 r13 = 0x00007fc10d6ad3d0 > > > > r14 = 0xffffffffffffff92 r15 = 0x00007fc10d6ad400 > > > > rip = 0x00007fdb5e2ffa82 > > > > Found by: given as instruction pointer in context > > > > > > > > Thread 960 > > > > 0 0x7fdb5e2ffa82 > > > > rax = 0xfffffffffffffdfc rdx = 0x0000000000000001 > > > > rcx = 0xffffffffffffffff rbx = 0x0000000010d33e20 > > > > rsi = 0x0000000000000189 rdi = 0x0000000010d33e4c > > > > rbp = 0x00007fc0f3e7a740 rsp = 0x00007fc0f3e7a6b0 > > > > r8 = 0x0000000010d33e20 r9 = 0x00000000ffffffff > > > > r10 = 0x00007fc0f3e7a740 r11 = 0x0000000000000202 > > > > r12 = 0x0000000000000001 r13 = 0x00007fc0f3e7a740 > > > > r14 = 0xffffffffffffff92 r15 = 0x0000000022217000 > > > > rip = 0x00007fdb5e2ffa82 > > > > Found by: given as instruction pointer in context > > > > > > > > Output of find syms/impalad: > > > > [ec2-user@jenkins tmp]$ find syms/impalad > > > > syms/impalad > > > > syms/impalad/73318DB4729C07530B7338E67EA255AB0 > > > > syms/impalad/73318DB4729C07530B7338E67EA255AB0/impalad.sym > > > > > > > > These are the steps I had followed: > > > > Step-1: > > > > I followed exact steps from this https://cwiki.apache.org/confl > > > > uence/display/IMPALA/Building+Impala and also followed "build > > > > native-toolchain from scratch" https://cwiki.apache.org/confl > > > > uence/display/IMPALA/Building+native-toolchain+from+scratch+ > > > > and+using+with+Impala > > > > > > > > Step-2: > > > > Env: > > > > export IMPALA_HOME=/home/ec2-user/impala-setup/Impala > > > > export PATH=$PATH:$IMPALA_HOME/bin > > > > export KUDU_IS_SUPPORTED=false > > > > export KUDU_CLIENT_DIR=/opt/kudu/ > > > > > > > > Get breakpad symbols: > > > > dump_breakpad_symbols.py -b be/build/ --dump_syms > > > > $IMPALA_HOME/toolchain/breakpad-1b704857f1e78a864e6942e61345 > > 7e55f1aecb60-p3/bin/dump_syms > > > > -d /tmp/syms > > > > > > > > rm /tmp/resolved.txt > > > > > > > > Run minidump_stackwalk: > > > > $IMPALA_HOME/toolchain/breakpad-1b704857f1e78a864e6942e61345 > > > > 7e55f1aecb60-p3/bin/minidump_stackwalk /home/ec2-user/64811560-8bae-a > > 67f-22f961c1-45bf52a3.dmp > > > > /tmp/syms > /tmp/resolved.txt > > > > > > > > > > > > Please let me know If I have missed any steps?? > > > > > > > > > > > > > > > Thanks you Lars Volker, > > I took few new .dmp files and ran through stackwalk. > > here is last few lines from the resolver: > > > > Loaded modules: > > 0x00400000 - 0x0283efff impalad ??? (main) (WARNING: No symbols, > > impalad, 21AA452CEF0F8C034C7E78A639190ED40) > > 0x7fce0a33b000 - 0x7fce0a543fff libgssapiv2.so.3.0.0 ??? > > 0x7fce0a544000 - 0x7fce0a748fff libplain.so.3.0.0 ??? > > 0x7fce0a749000 - 0x7fce0a94dfff liblogin.so.3.0.0 ??? > > 0x7fce0a94e000 - 0x7fce0ad0bfff libdb-5.3.so ??? > > 0x7fce0ad0c000 - 0x7fce0af12fff libsasldb.so.3.0.0 ??? > > 0x7fce0af13000 - 0x7fce0b118fff libanonymous.so.3.0.0 ??? > > 0x7fce0b919000 - 0x7fce0bb79fff libpcre.so.1.2.0 ??? > > 0x7fce0bb7a000 - 0x7fce0bd9efff libselinux.so.1 ??? > > 0x7fce0bda1000 - 0x7fce0bfa4fff libkeyutils.so.1.5 ??? > > 0x7fce0bfa5000 - 0x7fce0c1b2fff libkrb5support.so.0.1 ??? > > 0x7fce0c1b3000 - 0x7fce0c3b5fff libfreebl3.so ??? > > 0x7fce0c3b6000 - 0x7fce0c5cbfff libz.so.1.2.7 ??? > > 0x7fce0c5cc000 - 0x7fce0c7fefff libk5crypto.so.3.1 ??? > > 0x7fce0c7ff000 - 0x7fce0ca02fff libcom_err.so.2.1 ??? > > 0x7fce0ca03000 - 0x7fce0cceafff libkrb5.so.3.3 ??? > > 0x7fce0cceb000 - 0x7fce0cf37fff libgssapi_krb5.so.2.2 ??? > > 0x7fce0cf38000 - 0x7fce0d140fff libcrypt-2.17.so ??? > > 0x7fce0d16f000 - 0x7fce0d386fff libresolv-2.17.so ??? > > 0x7fce0d389000 - 0x7fce0d746fff libc-2.17.so ??? (WARNING: No symbols, > > libc-2.17.so, E42D1DFF35CF1831960C5EA195CEB1360) > > 0x7fce0d74c000 - 0x7fce0d963fff libpthread-2.17.so ??? (WARNING: No > > symbols, libpthread-2.17.so, 9B139160D961AA018425DBC16B7F51D40) > > > > I think I'm still not getting symbols for impala?? > > > > > > > Hi Vincent Tran,
Thanks for response. No the dmp file is from different CDH cluster. Core file last few lines: ??9?k ??ol ?I?k ??Ö?~W???eM?x86_64/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/impala/sbin-retail/impalad--flagfile=/run/cloudera-scm-agent/process/2619-impala-IMPALAD/impala-conf/impalad_flagsCDH_HCAT_HOME=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hive-hcatalogTOMCAT_HOME=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/bigtop-tomcatXDG_SESSION_ID=c4CDH_PIG_HOME=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/pigCDH_SOLR_HOME=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/solrPARCELS_ROOT=/opt/cloudera/parcelsSHELL=/bin/bashCLOUDERA_MYSQL_CONNECTOR_JAR=/usr/share/java/mysql-connector-java.jarGLOG_logbuflevel=0CDH_HUE_PLUGINS_HOME=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoopCDH_HIVE_HOME=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hiveSCM_DEFINES_SCRIPTS=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/meta/cdh_env.shCDH_AVRO_HOME=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/avroCLOUDERA_ORAC LE_CONNECTOR_JAR=/usr/share/java/oracle-connector-java.jarCDH_VERSION=5GLOG_minloglevel=0HIVE_DEFAULT_XML=/etc/hive/conf.dist/hive-default.xmlSEARCH_HOME=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/searchCMF_CONF_DIR=/etc/cloudera-scm-agentMGMT_HOME=/usr/share/cmfCGROUP_GROUP_BLKIO=IMPALA_BIN=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/impala/sbin-retailPARCEL_DIRNAMES=CDH-5.11.0-1.cdh5.11.0.p0.34USER=impalaLD_LIBRARY_PATH=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/impala/lib:/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/impala/sbin-retail:/usr/java/jdk1.8.0_131/jre/lib/amd64:/usr/java/jdk1.8.0_131/jre/lib/amd64/server:/usr/java/jdk1.8.0_131/jre/lib/amd64/server:JSVC_HOME=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/bigtop-utilsCDH_SQOOP_HOME=/usr/lib/sqoopAUX_CLASSPATH=/usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jarHBASE_HOME=/opt/cloudera/parcels/ CDH-5.11.0-1.cdh5.11.0.p0.34/bin/../lib/hbaseGLOG_max_log_size=200CDH_IMPALA_HOME=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/impalaCDH_HTTPFS_HOME=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-httpfsCDH_MR2_HOME=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-mapreduceCM_STATUS_CODES=STATUS_NONE HDFS_DFS_DIR_NOT_EMPTY HBASE_TABLE_DISABLED HBASE_TABLE_ENABLED JOBTRACKER_IN_STANDBY_MODE YARN_RM_IN_STANDBY_MODECDH_HUE_HOME=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hueCDH_MR1_HOME=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-0.20-mapreduceCGROUP_ROOT_MEMORY=/sys/fs/cgroup/memoryGLOG_v=1PATH=/sbin:/usr/sbin:/bin:/usr/binCONF_DIR=/run/cloudera-scm-agent/process/2619-impala-IMPALADCGROUP_GROUP_MEMORY=CDH_KAFKA_HOME=/usr/lib/kafkaCDH_SENTRY_HOME=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/sentryHIVE_HOME=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/bin/../lib/hiveCDH_PARQUET_HOME=/opt/cloudera/parcels/C DH-5.11.0-1.cdh5.11.0.p0.34/lib/parquetSUPERVISOR_GROUP_NAME=2619-impala-IMPALADIMPALA_HOME=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/impalaHBASE_CONF_DIR=/run/cloudera-scm-agent/process/2619-impala-IMPALAD/hbase-confPWD=/run/cloudera-scm-agent/process/2619-impala-IMPALADJAVA_HOME=/usr/java/jdk1.8.0_131LANG=en_US.UTF-8HADOOP_CONF_DIR=/run/cloudera-scm-agent/process/2619-impala-IMPALAD/hadoop-confCGROUP_ROOT_CPUACCT=/sys/fs/cgroup/cpu,cpuacctCDH_HADOOP_HOME=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoopCGROUP_ROOT_BLKIO=/sys/fs/cgroup/blkioCGROUP_ROOT_CPU=/sys/fs/cgroup/cpu,cpuacctCDH_SQOOP2_HOME=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/sqoop2CDH_SPARK_HOME=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/sparkCDH_CRUNCH_HOME=/usr/lib/crunchCDH_LLAMA_HOME=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/llamaJAVA_TOOL_OPTIONS=CDH_HADOOP_BIN=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop/bin/hadoopCDH_HDFS_HOME=/opt/ cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/hadoop-hdfsSUPERVISOR_ENABLED=1LIBHDFS_OPTS=-Djava.library.path=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/bin/../lib/impala/libHOME=/var/lib/impalaSHLVL=2CDH_OOZIE_HOME=/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/oozieKEYTRUSTEE_SERVER_HOME=/usr/lib/keytrustee-serverCDH_HBASE_INDEXER_HOME=/opt/cloudera/parcels
