Re: GVO machines dependencies
Glad I could help! The fix is merged now: https://github.com/awleblang/impala-setup/commit/1fcff44887de76fdb5f86948635c073d0e615f1c On Thu, May 25, 2017 at 10:00 AM, Laszlo Gaal wrote: > Thanks a lot for fixing this, Sailesh, this has been bugging me for a while > when spinning up cloud-based workstations. > > On Thu, May 25, 2017 at 6:18 PM, Jim Apple wrote: > > > Nice find, Sailesh! > > > > On Wed, May 24, 2017 at 6:39 PM, Sailesh Mukil > > wrote: > > > The issue was that there was a bug in the install.sh script from the > > > impala-setup/ repo that always assumed that the repo would reside in > > > /home/$user. > > > > > > The Jenkins AMIs seem to have had a very old checkout (8 months old) at > > > /home/ubuntu/ which was being used for the GVOs. The Jenkins job would > > make > > > a fresh checkout in /tmp/ and run install.sh from that checkout. But > due > > to > > > the above bug, the script would change its sources to the old repo in > > > /home/ubuntu. So, the new dependencies added to the project didn't get > > > picked up. I've submitted a fix, and will redo the GVO once it's merged > > > into impala-setup. > > > > > > Thanks for your help Michael and Jim! > > > > > > On Wed, May 24, 2017 at 4:58 PM, Michael Brown > > wrote: > > > > > >> "from-scratch" is a misnomer: only the first build to run on a given > > node > > >> is truly from-scratch, because running impala-setup takes effect > > >> system-wide, and is run once, when the node is created. > > >> > > >> For http://jenkins.impala.io:8080/job/ubuntu-14.04-from-scratch/1371/ > , > > >> the > > >> worker's build history suggests it has been up roughly 17 hours. > > >> http://jenkins.impala.io:8080/computer/ub1404-c4.4xl-gp2%20( > > >> i-0d76efbc8de26926c)/builds > > >> This means your recent change hasn't taken effect on "older" workers. > > >> > > >> Any new workers that get created should get the change. > > >> > > >> > > >> > > >> > > >> > > >> On Wed, May 24, 2017 at 4:34 PM, Sailesh Mukil > > >> wrote: > > >> > > >> > Thanks Michael and Jim, > > >> > > > >> > It looks like adding to bin/bootstrap_build.sh makes the > > ub14-build-only > > >> > job work fine, however, updating the impala-setup still doesn't seem > > to > > >> > work. I submitted a pull request that was merged by Dimitris here: > > >> > https://github.com/awleblang/impala-setup/commits/master > > >> > > > >> > Also, the impala-setup chef logs don't seem to get printed in the > > jenkins > > >> > console output. So I'm not sure if it's actually being run or not. > > >> > > > >> > On Wed, May 24, 2017 at 3:37 PM, Michael Brown > > >> wrote: > > >> > > > >> > > Thanks Jim; I had forgotten bin/bootstrap_build.sh. > > >> > > > > >> > > On Wed, May 24, 2017 at 3:14 PM, Jim Apple > > >> wrote: > > >> > > > > >> > > > I'd recommend against #2 using the node setup to install Impala > > >> > > > dependencies. The only reason it installs openjdk is that > Jenkins > > >> > > > needs Java to talk to it. All of the other Impala dependencies > are > > >> > > > installed in the jobs themselves. > > >> > > > > > >> > > > On Wed, May 24, 2017 at 3:11 PM, Michael Brown < > > mi...@cloudera.com> > > >> > > wrote: > > >> > > > >> Is there a way to add a new dependency to these machines? > > >> > > > > > > >> > > > > Our builds use two worker labels: > > >> > > > > 1. ubuntu14.04-c4.4xlarge-gp2 > > >> > > > > 2. ub14-build-only > > >> > > > > > > >> > > > > 1 uses https://github.com/awleblang/impala-setup, and I think > > you > > >> > > should > > >> > > > > add libffi to there. It's something you should do anyway so > new > > >> users > > >> > > get > > >> > > > > set up with the right dependencies. > > >> > > > > > > >> > > > > 2 is a different story, but I see that at > > >> > > > > http://jenkins.impala.io:8080/configure that worker is > already > > >> > > > installing a > > >> > > > > JDK via apt-get. You could use a similar pattern to install > > libffi. > > >> > > > > > >> > > > > >> > > > >> > > >
Re: GVO machines dependencies
Thanks a lot for fixing this, Sailesh, this has been bugging me for a while when spinning up cloud-based workstations. On Thu, May 25, 2017 at 6:18 PM, Jim Apple wrote: > Nice find, Sailesh! > > On Wed, May 24, 2017 at 6:39 PM, Sailesh Mukil > wrote: > > The issue was that there was a bug in the install.sh script from the > > impala-setup/ repo that always assumed that the repo would reside in > > /home/$user. > > > > The Jenkins AMIs seem to have had a very old checkout (8 months old) at > > /home/ubuntu/ which was being used for the GVOs. The Jenkins job would > make > > a fresh checkout in /tmp/ and run install.sh from that checkout. But due > to > > the above bug, the script would change its sources to the old repo in > > /home/ubuntu. So, the new dependencies added to the project didn't get > > picked up. I've submitted a fix, and will redo the GVO once it's merged > > into impala-setup. > > > > Thanks for your help Michael and Jim! > > > > On Wed, May 24, 2017 at 4:58 PM, Michael Brown > wrote: > > > >> "from-scratch" is a misnomer: only the first build to run on a given > node > >> is truly from-scratch, because running impala-setup takes effect > >> system-wide, and is run once, when the node is created. > >> > >> For http://jenkins.impala.io:8080/job/ubuntu-14.04-from-scratch/1371/ , > >> the > >> worker's build history suggests it has been up roughly 17 hours. > >> http://jenkins.impala.io:8080/computer/ub1404-c4.4xl-gp2%20( > >> i-0d76efbc8de26926c)/builds > >> This means your recent change hasn't taken effect on "older" workers. > >> > >> Any new workers that get created should get the change. > >> > >> > >> > >> > >> > >> On Wed, May 24, 2017 at 4:34 PM, Sailesh Mukil > >> wrote: > >> > >> > Thanks Michael and Jim, > >> > > >> > It looks like adding to bin/bootstrap_build.sh makes the > ub14-build-only > >> > job work fine, however, updating the impala-setup still doesn't seem > to > >> > work. I submitted a pull request that was merged by Dimitris here: > >> > https://github.com/awleblang/impala-setup/commits/master > >> > > >> > Also, the impala-setup chef logs don't seem to get printed in the > jenkins > >> > console output. So I'm not sure if it's actually being run or not. > >> > > >> > On Wed, May 24, 2017 at 3:37 PM, Michael Brown > >> wrote: > >> > > >> > > Thanks Jim; I had forgotten bin/bootstrap_build.sh. > >> > > > >> > > On Wed, May 24, 2017 at 3:14 PM, Jim Apple > >> wrote: > >> > > > >> > > > I'd recommend against #2 using the node setup to install Impala > >> > > > dependencies. The only reason it installs openjdk is that Jenkins > >> > > > needs Java to talk to it. All of the other Impala dependencies are > >> > > > installed in the jobs themselves. > >> > > > > >> > > > On Wed, May 24, 2017 at 3:11 PM, Michael Brown < > mi...@cloudera.com> > >> > > wrote: > >> > > > >> Is there a way to add a new dependency to these machines? > >> > > > > > >> > > > > Our builds use two worker labels: > >> > > > > 1. ubuntu14.04-c4.4xlarge-gp2 > >> > > > > 2. ub14-build-only > >> > > > > > >> > > > > 1 uses https://github.com/awleblang/impala-setup, and I think > you > >> > > should > >> > > > > add libffi to there. It's something you should do anyway so new > >> users > >> > > get > >> > > > > set up with the right dependencies. > >> > > > > > >> > > > > 2 is a different story, but I see that at > >> > > > > http://jenkins.impala.io:8080/configure that worker is already > >> > > > installing a > >> > > > > JDK via apt-get. You could use a similar pattern to install > libffi. > >> > > > > >> > > > >> > > >> >
Re: GVO machines dependencies
Nice find, Sailesh! On Wed, May 24, 2017 at 6:39 PM, Sailesh Mukil wrote: > The issue was that there was a bug in the install.sh script from the > impala-setup/ repo that always assumed that the repo would reside in > /home/$user. > > The Jenkins AMIs seem to have had a very old checkout (8 months old) at > /home/ubuntu/ which was being used for the GVOs. The Jenkins job would make > a fresh checkout in /tmp/ and run install.sh from that checkout. But due to > the above bug, the script would change its sources to the old repo in > /home/ubuntu. So, the new dependencies added to the project didn't get > picked up. I've submitted a fix, and will redo the GVO once it's merged > into impala-setup. > > Thanks for your help Michael and Jim! > > On Wed, May 24, 2017 at 4:58 PM, Michael Brown wrote: > >> "from-scratch" is a misnomer: only the first build to run on a given node >> is truly from-scratch, because running impala-setup takes effect >> system-wide, and is run once, when the node is created. >> >> For http://jenkins.impala.io:8080/job/ubuntu-14.04-from-scratch/1371/ , >> the >> worker's build history suggests it has been up roughly 17 hours. >> http://jenkins.impala.io:8080/computer/ub1404-c4.4xl-gp2%20( >> i-0d76efbc8de26926c)/builds >> This means your recent change hasn't taken effect on "older" workers. >> >> Any new workers that get created should get the change. >> >> >> >> >> >> On Wed, May 24, 2017 at 4:34 PM, Sailesh Mukil >> wrote: >> >> > Thanks Michael and Jim, >> > >> > It looks like adding to bin/bootstrap_build.sh makes the ub14-build-only >> > job work fine, however, updating the impala-setup still doesn't seem to >> > work. I submitted a pull request that was merged by Dimitris here: >> > https://github.com/awleblang/impala-setup/commits/master >> > >> > Also, the impala-setup chef logs don't seem to get printed in the jenkins >> > console output. So I'm not sure if it's actually being run or not. >> > >> > On Wed, May 24, 2017 at 3:37 PM, Michael Brown >> wrote: >> > >> > > Thanks Jim; I had forgotten bin/bootstrap_build.sh. >> > > >> > > On Wed, May 24, 2017 at 3:14 PM, Jim Apple >> wrote: >> > > >> > > > I'd recommend against #2 using the node setup to install Impala >> > > > dependencies. The only reason it installs openjdk is that Jenkins >> > > > needs Java to talk to it. All of the other Impala dependencies are >> > > > installed in the jobs themselves. >> > > > >> > > > On Wed, May 24, 2017 at 3:11 PM, Michael Brown >> > > wrote: >> > > > >> Is there a way to add a new dependency to these machines? >> > > > > >> > > > > Our builds use two worker labels: >> > > > > 1. ubuntu14.04-c4.4xlarge-gp2 >> > > > > 2. ub14-build-only >> > > > > >> > > > > 1 uses https://github.com/awleblang/impala-setup, and I think you >> > > should >> > > > > add libffi to there. It's something you should do anyway so new >> users >> > > get >> > > > > set up with the right dependencies. >> > > > > >> > > > > 2 is a different story, but I see that at >> > > > > http://jenkins.impala.io:8080/configure that worker is already >> > > > installing a >> > > > > JDK via apt-get. You could use a similar pattern to install libffi. >> > > > >> > > >> > >>
Re: GVO machines dependencies
The issue was that there was a bug in the install.sh script from the impala-setup/ repo that always assumed that the repo would reside in /home/$user. The Jenkins AMIs seem to have had a very old checkout (8 months old) at /home/ubuntu/ which was being used for the GVOs. The Jenkins job would make a fresh checkout in /tmp/ and run install.sh from that checkout. But due to the above bug, the script would change its sources to the old repo in /home/ubuntu. So, the new dependencies added to the project didn't get picked up. I've submitted a fix, and will redo the GVO once it's merged into impala-setup. Thanks for your help Michael and Jim! On Wed, May 24, 2017 at 4:58 PM, Michael Brown wrote: > "from-scratch" is a misnomer: only the first build to run on a given node > is truly from-scratch, because running impala-setup takes effect > system-wide, and is run once, when the node is created. > > For http://jenkins.impala.io:8080/job/ubuntu-14.04-from-scratch/1371/ , > the > worker's build history suggests it has been up roughly 17 hours. > http://jenkins.impala.io:8080/computer/ub1404-c4.4xl-gp2%20( > i-0d76efbc8de26926c)/builds > This means your recent change hasn't taken effect on "older" workers. > > Any new workers that get created should get the change. > > > > > > On Wed, May 24, 2017 at 4:34 PM, Sailesh Mukil > wrote: > > > Thanks Michael and Jim, > > > > It looks like adding to bin/bootstrap_build.sh makes the ub14-build-only > > job work fine, however, updating the impala-setup still doesn't seem to > > work. I submitted a pull request that was merged by Dimitris here: > > https://github.com/awleblang/impala-setup/commits/master > > > > Also, the impala-setup chef logs don't seem to get printed in the jenkins > > console output. So I'm not sure if it's actually being run or not. > > > > On Wed, May 24, 2017 at 3:37 PM, Michael Brown > wrote: > > > > > Thanks Jim; I had forgotten bin/bootstrap_build.sh. > > > > > > On Wed, May 24, 2017 at 3:14 PM, Jim Apple > wrote: > > > > > > > I'd recommend against #2 using the node setup to install Impala > > > > dependencies. The only reason it installs openjdk is that Jenkins > > > > needs Java to talk to it. All of the other Impala dependencies are > > > > installed in the jobs themselves. > > > > > > > > On Wed, May 24, 2017 at 3:11 PM, Michael Brown > > > wrote: > > > > >> Is there a way to add a new dependency to these machines? > > > > > > > > > > Our builds use two worker labels: > > > > > 1. ubuntu14.04-c4.4xlarge-gp2 > > > > > 2. ub14-build-only > > > > > > > > > > 1 uses https://github.com/awleblang/impala-setup, and I think you > > > should > > > > > add libffi to there. It's something you should do anyway so new > users > > > get > > > > > set up with the right dependencies. > > > > > > > > > > 2 is a different story, but I see that at > > > > > http://jenkins.impala.io:8080/configure that worker is already > > > > installing a > > > > > JDK via apt-get. You could use a similar pattern to install libffi. > > > > > > > > > >
Re: GVO machines dependencies
"from-scratch" is a misnomer: only the first build to run on a given node is truly from-scratch, because running impala-setup takes effect system-wide, and is run once, when the node is created. For http://jenkins.impala.io:8080/job/ubuntu-14.04-from-scratch/1371/ , the worker's build history suggests it has been up roughly 17 hours. http://jenkins.impala.io:8080/computer/ub1404-c4.4xl-gp2%20(i-0d76efbc8de26926c)/builds This means your recent change hasn't taken effect on "older" workers. Any new workers that get created should get the change. On Wed, May 24, 2017 at 4:34 PM, Sailesh Mukil wrote: > Thanks Michael and Jim, > > It looks like adding to bin/bootstrap_build.sh makes the ub14-build-only > job work fine, however, updating the impala-setup still doesn't seem to > work. I submitted a pull request that was merged by Dimitris here: > https://github.com/awleblang/impala-setup/commits/master > > Also, the impala-setup chef logs don't seem to get printed in the jenkins > console output. So I'm not sure if it's actually being run or not. > > On Wed, May 24, 2017 at 3:37 PM, Michael Brown wrote: > > > Thanks Jim; I had forgotten bin/bootstrap_build.sh. > > > > On Wed, May 24, 2017 at 3:14 PM, Jim Apple wrote: > > > > > I'd recommend against #2 using the node setup to install Impala > > > dependencies. The only reason it installs openjdk is that Jenkins > > > needs Java to talk to it. All of the other Impala dependencies are > > > installed in the jobs themselves. > > > > > > On Wed, May 24, 2017 at 3:11 PM, Michael Brown > > wrote: > > > >> Is there a way to add a new dependency to these machines? > > > > > > > > Our builds use two worker labels: > > > > 1. ubuntu14.04-c4.4xlarge-gp2 > > > > 2. ub14-build-only > > > > > > > > 1 uses https://github.com/awleblang/impala-setup, and I think you > > should > > > > add libffi to there. It's something you should do anyway so new users > > get > > > > set up with the right dependencies. > > > > > > > > 2 is a different story, but I see that at > > > > http://jenkins.impala.io:8080/configure that worker is already > > > installing a > > > > JDK via apt-get. You could use a similar pattern to install libffi. > > > > > >
Re: GVO machines dependencies
Thanks Michael and Jim, It looks like adding to bin/bootstrap_build.sh makes the ub14-build-only job work fine, however, updating the impala-setup still doesn't seem to work. I submitted a pull request that was merged by Dimitris here: https://github.com/awleblang/impala-setup/commits/master Also, the impala-setup chef logs don't seem to get printed in the jenkins console output. So I'm not sure if it's actually being run or not. On Wed, May 24, 2017 at 3:37 PM, Michael Brown wrote: > Thanks Jim; I had forgotten bin/bootstrap_build.sh. > > On Wed, May 24, 2017 at 3:14 PM, Jim Apple wrote: > > > I'd recommend against #2 using the node setup to install Impala > > dependencies. The only reason it installs openjdk is that Jenkins > > needs Java to talk to it. All of the other Impala dependencies are > > installed in the jobs themselves. > > > > On Wed, May 24, 2017 at 3:11 PM, Michael Brown > wrote: > > >> Is there a way to add a new dependency to these machines? > > > > > > Our builds use two worker labels: > > > 1. ubuntu14.04-c4.4xlarge-gp2 > > > 2. ub14-build-only > > > > > > 1 uses https://github.com/awleblang/impala-setup, and I think you > should > > > add libffi to there. It's something you should do anyway so new users > get > > > set up with the right dependencies. > > > > > > 2 is a different story, but I see that at > > > http://jenkins.impala.io:8080/configure that worker is already > > installing a > > > JDK via apt-get. You could use a similar pattern to install libffi. > > >
Re: GVO machines dependencies
Thanks Jim; I had forgotten bin/bootstrap_build.sh. On Wed, May 24, 2017 at 3:14 PM, Jim Apple wrote: > I'd recommend against #2 using the node setup to install Impala > dependencies. The only reason it installs openjdk is that Jenkins > needs Java to talk to it. All of the other Impala dependencies are > installed in the jobs themselves. > > On Wed, May 24, 2017 at 3:11 PM, Michael Brown wrote: > >> Is there a way to add a new dependency to these machines? > > > > Our builds use two worker labels: > > 1. ubuntu14.04-c4.4xlarge-gp2 > > 2. ub14-build-only > > > > 1 uses https://github.com/awleblang/impala-setup, and I think you should > > add libffi to there. It's something you should do anyway so new users get > > set up with the right dependencies. > > > > 2 is a different story, but I see that at > > http://jenkins.impala.io:8080/configure that worker is already > installing a > > JDK via apt-get. You could use a similar pattern to install libffi. >
Re: GVO machines dependencies
I'd recommend against #2 using the node setup to install Impala dependencies. The only reason it installs openjdk is that Jenkins needs Java to talk to it. All of the other Impala dependencies are installed in the jobs themselves. On Wed, May 24, 2017 at 3:11 PM, Michael Brown wrote: >> Is there a way to add a new dependency to these machines? > > Our builds use two worker labels: > 1. ubuntu14.04-c4.4xlarge-gp2 > 2. ub14-build-only > > 1 uses https://github.com/awleblang/impala-setup, and I think you should > add libffi to there. It's something you should do anyway so new users get > set up with the right dependencies. > > 2 is a different story, but I see that at > http://jenkins.impala.io:8080/configure that worker is already installing a > JDK via apt-get. You could use a similar pattern to install libffi.
Re: GVO machines dependencies
> Is there a way to add a new dependency to these machines? Our builds use two worker labels: 1. ubuntu14.04-c4.4xlarge-gp2 2. ub14-build-only 1 uses https://github.com/awleblang/impala-setup, and I think you should add libffi to there. It's something you should do anyway so new users get set up with the right dependencies. 2 is a different story, but I see that at http://jenkins.impala.io:8080/configure that worker is already installing a JDK via apt-get. You could use a similar pattern to install libffi.
Re: GVO machines dependencies
They run https://github.com/awleblang/impala-setup. One of the jobs run as part of this is http://jenkins.impala.io:8080/view/Utility/job/ubuntu-14.04-build-only/ which runs ./bin/bootstrap_build.sh (in the Apache Impala repo), so you'll need to update that file, too, I suspect. On Wed, May 24, 2017 at 2:56 PM, Sailesh Mukil wrote: > I have a question regarding dependencies on the jenkins machines. ( > http://jenkins.impala.io:8080/job/ubuntu-14.04-from-scratch/) > > How do the Jenkins machines used to GVO have dependencies pre-packaged in > them? For eg., openssl, etc. Are they baked into an AMI? > > I have a patch (https://gerrit.cloudera.org/#/c/6910/) that requires a new > dependency for Ubuntu (libffi), and therefore won't pass GVO. This didn't > show up in all my testing since the machines I used already had this > dependency in them. > > Is there a way to add a new dependency to these machines?
GVO machines dependencies
I have a question regarding dependencies on the jenkins machines. ( http://jenkins.impala.io:8080/job/ubuntu-14.04-from-scratch/) How do the Jenkins machines used to GVO have dependencies pre-packaged in them? For eg., openssl, etc. Are they baked into an AMI? I have a patch (https://gerrit.cloudera.org/#/c/6910/) that requires a new dependency for Ubuntu (libffi), and therefore won't pass GVO. This didn't show up in all my testing since the machines I used already had this dependency in them. Is there a way to add a new dependency to these machines?
