It looks like you built the debug Impala with dynamic linking (-build_shared_libs or -so) whereas the CDH build uses static linking - that might explain why the release build worked ok.
It's expected that your binary is much larger, in the CDH packages the debug symbols were stripped out and put in a separate impalad.debug file, whereas just building a dev environment doesn't do that step. Should work fine though. On Tue, Sep 13, 2016 at 4:44 PM, Pradeep Nayak <pradeep1...@gmail.com> wrote: > Well when I built the debug one and replaced it my cluster it seemed to > have more dependencies which it was not able to find; > > ubuntu@ip-172-30-2-102:/opt/cloudera/parcels/CDH-5.8.0-1. > cdh5.8.0.p0.42/lib/impala/sbin-debug$ ldd impalad > ./impalad: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version > `GLIBCXX_3.4.20' not found (required by ./impalad) > linux-vdso.so.1 => (0x00007ffd87dc6000) > libjsig.so => not found > libTestUtil.so => not found > libUtil.so => not found > libRuntime.so => not found > libExec.so => not found > libCodeGen.so => not found > libExprs.so => not found > libRpc.so => not found > libService.so => not found > libStatestore.so => not found > libCatalog.so => not found > libResourceBroker.so => not found > libImpalaThrift.so => not found > libGlobalFlags.so => not found > libCommon.so => not found > libUdf.so => not found > libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 > (0x00007fd4661be000) > libThriftSaslTransport.so => not found > libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 > (0x00007fd465f5f000) > libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 > (0x00007fd465b83000) > libgutil.so => not found > libhdfs.so.0.0.0 => not found > libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fd46596a000) > libjvm.so => not found > libkudu_client.so.0 => not found > librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 > (0x00007fd465762000) > libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 > (0x00007fd46555e000) > libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 > (0x00007fd46525a000) > libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fd464f54000) > libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 > (0x00007fd464d3e000) > libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 > (0x00007fd464b20000) > libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd46475b000) > /lib64/ld-linux-x86-64.so.2 (0x00007fd4663d9000) > > > > I went ahead and build the retail one and that was OKAY. > > ubuntu@ip-172-30-2-102:/opt/cloudera/parcels/CDH-5.8.0-1. > cdh5.8.0.p0.42/lib/impala/sbin-retail$ ldd impalad > ./impalad: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version > `CXXABI_1.3.8' not found (required by ./impalad) > ./impalad: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version > `GLIBCXX_3.4.20' not found (required by ./impalad) > linux-vdso.so.1 => (0x00007fffc2679000) > libjsig.so => not found > libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 > (0x00007f153629d000) > libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 > (0x00007f153603e000) > libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 > (0x00007f1535c62000) > libjvm.so => not found > libkudu_client.so.0 => not found > librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 > (0x00007f1535a5a000) > libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 > (0x00007f1535856000) > libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 > (0x00007f1535552000) > libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f153524c000) > libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 > (0x00007f1535036000) > libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 > (0x00007f1534e18000) > libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1534a53000) > /lib64/ld-linux-x86-64.so.2 (0x00007f15364b8000) > > > However I see the sizes being different. The one which came with cloudera > parcels was of 37M and one which I built was 384MB!! Why is that ? > > ubuntu@ip-172-30-2-102:/opt/cloudera/parcels/CDH-5.8.0-1. > cdh5.8.0.p0.42/lib/impala/sbin-retail$ du -sh impalad > 348M impalad > ubuntu@ip-172-30-2-102:/opt/cloudera/parcels/CDH-5.8.0-1. > cdh5.8.0.p0.42/lib/impala/sbin-retail$ du -sh ~/impala.retail.orig > 37M /home/ubuntu/impala.retail.orig > ubuntu@ip-172-30-2-102:/opt/cloudera/parcels/CDH-5.8.0-1. > cdh5.8.0.p0.42/lib/impala/sbin-retail$ du -sh libfesupport.so > 346M libfesupport.so > ubuntu@ip-172-30-2-102:/opt/cloudera/parcels/CDH-5.8.0-1. > cdh5.8.0.p0.42/lib/impala/sbin-retail$ du -sh > ~/libfesupport.so.retail.orig > 39M /home/ubuntu/libfesupport.so.retail.orig > > > To conclude I am able to verify the changes which I made in the retail > build but not with the debug build. We can probably document what needs to > be replaced to test changes in a real cluster with someone's custom impala > binaries. > > 1. impalad in /opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/ > impala/sbin-retail > 2. libfesupport.so in /opt/cloudera/parcels/CDH- > 5.8.0-1.cdh5.8.0.p0.42/lib/impala/sbin-retail > 3. impala-frontend-0.1-SNAPSHOT.jar in > /opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/jars/. > The impala-frontend-0.1-SNAPSHOT.jar file in > /opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/impala/lib/ symlinks > to the one in /opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/jars/. > > This is all which is need to test your changes. > > Regards, > Pradeep Nayak > > http://pradeepnayak.in > http://twitter.com/_prdp > > On Tue, Sep 13, 2016 at 4:13 PM, Pradeep Nayak <pradeep1...@gmail.com> > wrote: > >> Okay thanks Tim. I tried that. >> >> I am seeing the below error, once I repalced the impalad and >> libfesupport.so. Same errors for statestore and catalogd. >> >> + true >> + export >> IMPALA_BIN=/opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/impala/sbin-debug >> + >> IMPALA_BIN=/opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/impala/sbin-debug >> + [[ true = '' ]] >> + '[' impalad = impalad ']' >> + exec >> /opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/impala/../../bin/impalad >> >> --flagfile=/run/cloudera-scm-agent/process/92-impala-IMPALAD/impala-conf/impalad_flags >> /opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/impala/sbin-debug/impalad: >> error while loading shared libraries: libTestUtil.so: cannot open shared >> object file: No such file or directory >> >> >> Regards, >> Pradeep Nayak >> >> http://pradeepnayak.in >> http://twitter.com/_prdp >> >> On Tue, Sep 13, 2016 at 2:16 PM, Tim Armstrong <tarmstr...@cloudera.com> >> wrote: >> >>> If the statestored and catalogd symlinks point to the new impalad binary >>> then you're good. >>> >>> On Tue, Sep 13, 2016 at 2:02 PM, Pradeep Nayak <pradeep1...@gmail.com> >>> wrote: >>> >>>> My confusion should I update the bin location or the sbin location here >>>> : /opt/cloudera/parcels/CDH/lib/impala/sbin. Here the catalogd and >>>> statestored appear to be symlinks to impalad. >>>> >>>> Regards, >>>> Pradeep Nayak >>>> >>>> http://pradeepnayak.in >>>> http://twitter.com/_prdp >>>> >>>> On Tue, Sep 13, 2016 at 1:57 PM, Pradeep Nayak <pradeep1...@gmail.com> >>>> wrote: >>>> >>>>> Okay, I can override impalad at /opt/cloudera/parcels/CDH-5 >>>>> .8.0-1.cdh5.8.0.p0.42/bin/impalad with the one which I have built. >>>>> How about catalogd and statestored ? >>>>> >>>>> Regards, >>>>> Pradeep Nayak >>>>> >>>>> http://pradeepnayak.in >>>>> http://twitter.com/_prdp >>>>> >>>>> On Mon, Sep 12, 2016 at 3:41 PM, Matthew Jacobs <m...@cloudera.com> >>>>> wrote: >>>>> >>>>>> Correct, impala-data-source-api*.jar rarely changes, but it will when >>>>>> we rename our java packages: https://gerrit.cloudera.org/#/c/3937/ >>>>>> >>>>>> If you go through this process again with a build after that change >>>>>> goes in, then you'll need to update that jar. >>>>>> >>>>>> You can build it by running mvn package in Impala/ext-data-source. >>>>>> >>>>>> On Mon, Sep 12, 2016 at 3:29 PM, Tim Armstrong < >>>>>> tarmstr...@cloudera.com> wrote: >>>>>> > It looks like impala-data-source* may not be built in all case. It >>>>>> doesn't >>>>>> > really change much if at all so I doubt you will need to update it >>>>>> anyway. >>>>>> > >>>>>> > P.S. I agree with Jim, this is exactly the kind of question this >>>>>> list is >>>>>> > for. >>>>>> > >>>>>> > On Mon, Sep 12, 2016 at 2:08 PM, Pradeep Nayak < >>>>>> pradeep1...@gmail.com> >>>>>> > wrote: >>>>>> > >>>>>> >> So here is what I understand: >>>>>> >> >>>>>> >> I need to pick the files built here: /home/ubuntu/Impala/be/ >>>>>> >> build/debug/service >>>>>> >> >>>>>> >> ubuntu@ip-172-30-2-163:~/Impala/be/build/debug/service$ ls -l >>>>>> >> total 330928 >>>>>> >> -rwxrwxr-x 1 ubuntu ubuntu 59342696 Sep 12 19:25 hs2-util-test >>>>>> >> *-rwxrwxr-x 1 ubuntu ubuntu 58803080 Sep 12 19:24 impalad* >>>>>> >> *-rwxrwxr-x 1 ubuntu ubuntu 63169736 Sep 12 19:25 libfesupport.so* >>>>>> >> *-rwxrwxr-x 1 ubuntu ubuntu 37056896 Sep 6 21:41 libService.so* >>>>>> >> -rwxrwxr-x 1 ubuntu ubuntu 59526680 Sep 12 19:25 query-options-test >>>>>> >> -rwxrwxr-x 1 ubuntu ubuntu 60960616 Sep 12 19:25 >>>>>> session-expiry-test >>>>>> >> ubuntu@ip-172-30-2-163:~/Impala/be/build/debug/service$ >>>>>> >> >>>>>> >> and replace it with one on my test cluster for backend part. >>>>>> >> >>>>>> >> >>>>>> >> For the frontend, i need to pick the below file: >>>>>> >> >>>>>> >> */home/ubuntu/Impala/fe/target/impala-frontend-0.1-SNAPSHOT.jar * >>>>>> >> >>>>>> >> >>>>>> >> I couldn't find the impala-data-source-api*.jar file >>>>>> >> >>>>>> >> >>>>>> >> Regards, >>>>>> >> Pradeep Nayak >>>>>> >> >>>>>> >> http://pradeepnayak.in >>>>>> >> http://twitter.com/_prdp >>>>>> >> >>>>>> >> On Mon, Sep 12, 2016 at 1:26 PM, Tim Armstrong < >>>>>> tarmstr...@cloudera.com> >>>>>> >> wrote: >>>>>> >> >>>>>> >>> I don't think this is documented anywhere. You would need to >>>>>> replace the >>>>>> >>> impalad binary and libfesupport.so under sbin-retail or >>>>>> sbin-debug for the >>>>>> >>> release or debug builds respectively. You also need to replace >>>>>> >>> impala-frontend-*.jar and impala-data-source-api*.jar. >>>>>> >>> >>>>>> >>> You also need to make sure the statestored and catalogd symlinks >>>>>> point to >>>>>> >>> the impalad binary, >>>>>> >>> >>>>>> >>> On Mon, Sep 12, 2016 at 1:06 PM, Pradeep Nayak < >>>>>> pradeep1...@gmail.com> >>>>>> >>> wrote: >>>>>> >>> >>>>>> >>>> This question may sound noob, but apologies in advance. >>>>>> >>>> >>>>>> >>>> I have changes in the impala source code (both backend and >>>>>> frontend >>>>>> >>>> components). How do I go about installing this version of Impala >>>>>> on my >>>>>> >>>> cloudera test cluster ? I know the path where I can replace the >>>>>> front end >>>>>> >>>> built jar file, but not sure of the backend parts. >>>>>> >>>> >>>>>> >>>> Regards, >>>>>> >>>> Pradeep Nayak >>>>>> >>>> >>>>>> >>>> http://pradeepnayak.in >>>>>> >>>> http://twitter.com/_prdp >>>>>> >>>> >>>>>> >>> >>>>>> >>> >>>>>> >> >>>>>> >>>>> >>>>> >>>> >>> >> >
