Ah, haven't followed this thread - goes to another email id. But thanks for the info, I'll update the scripts over the weekend (i've downgraded to Whirr 0.3)
Thanks Saptarshi On Thu, Mar 10, 2011 at 11:37 AM, Tom White <t...@cloudera.com> wrote: > On Thu, Mar 10, 2011 at 11:27 AM, Benjamin Clark <b...@daltonclark.com> wrote: >> So if we grab install_cdh_hadoop.sh from the source tree, and put a >> customized version in our own bucket, and set whirr.run-url-base to the root >> of that bucket, it should work, even in 4.0 and after? > > whirr.run-url-base is no longer used in 0.4.0 and after. Instead, > scripts are read locally and copied to the nodes as needed. See > https://issues.apache.org/jira/browse/WHIRR-225 for more details, also > CHANGES.txt and the recipes directory. > > So if you use trunk (or 0.4.0) you should just be able to set > > whirr.hadoop-install-function=install_cdh_hadoop > whirr.hadoop-configure-function=configure_cdh_hadoop > > and it will work without having to involve S3 or a remote webserver. > > Cheers, > Tom > >> >> Based on the FAQ I tried a few of these to attempt to verify I was on the >> right track: >> >> wget http://whirr.s3.amazonaws.com/install_cdh_hadoop >> wget http://whirr.s3.amazonaws.com/0.4/install_cdh_hadoop >> wget http://whirr.s3.amazonaws.com/0.4.0/install_cdh_hadoop >> wget http://whirr.s3.amazonaws.com/install_cdh_hadoop.sh >> wget http://whirr.s3.amazonaws.com/0.4/install_cdh_hadoop.sh >> wget http://whirr.s3.amazonaws.com/0.4.0/install_cdh_hadoop.sh >> >> but all give 404s. >> >> >> >> On Mar 10, 2011, at 12:01 AM, Tom White wrote: >> >>> Sorry I missed this thread. On 0.4.0 and later >>> >>> whirr.hadoop-install-runurl=cloudera/cdh/install >>> whirr.hadoop-configure-runurl=cloudera/cdh/post-configure >>> >>> changes to >>> >>> whirr.hadoop-install-function=install_cdh_hadoop >>> whirr.hadoop-configure-function=configure_cdh_hadoop >>> >>> Cheers, >>> Tom >>> >>> On Wed, Mar 9, 2011 at 8:23 AM, Sebastian Schoenherr >>> <sebastian.schoenh...@uibk.ac.at> wrote: >>>> Hi Saptarshi, >>>> I tried to execute my working whirr 0.3.0 configuration (identical to your >>>> property file, using cloudera scripts) on branch-0.4 and the same issues >>>> arised for me. Unfortunately I'm not sure yet why it's not working with >>>> branch-0.4. Is using branch-0.3 an option for you? >>>> Any other guesses? >>>> cheers, >>>> sebastian >>>> >>>> >>>> On 08.03.2011 05:41, Saptarshi Guha wrote: >>>>> >>>>> once again, i've changed the secret identity .. >>>>> >>>>> On Mon, Mar 7, 2011 at 8:41 PM, Saptarshi Guha >>>>> <saptar...@revolutionanalytics.com> wrote: >>>>>> >>>>>> Hello >>>>>> >>>>>> No such luck on my end This is my script file, you can test that the >>>>>> scripts download. But when I log in >>>>>> hadoop version is. (I pulled the latest git). Also my scripts (you can >>>>>> confirm if you download) echo a small line to files in /tmp. >>>>>> They are not being created. >>>>>> >>>>>> Hadoop 0.20.2 >>>>>> Subversion >>>>>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 >>>>>> -r 911707 >>>>>> Compiled by chrisdo on Fri Feb 19 08:07:34 UTC 2010 >>>>>> >>>>>> whirr.cluster-name=revotesting2 >>>>>> whirr.service-name=hadoop >>>>>> whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,2 >>>>>> hadoop-datanode+hadoop-tasktracker >>>>>> whirr.provider=aws-ec2 >>>>>> whirr.identity= AKIAJH5JBSI5KJ7YZQ6A >>>>>> whirr.credential= b/kqLJAHOdRA4L30n7Zt8Edz383B1ARtPI3wiyD6 >>>>>> whirr.location-id=us-east-1 >>>>>> whirr.hardware-id=c1.xlarge >>>>>> whirr.run-url-base=http://ml.stat.purdue.edu/whirr-scripts/ >>>>>> whirr.hadoop-install-runurl=cloudera/cdh/install >>>>>> whirr.hadoop-configure-runurl=cloudera/cdh/post-configure >>>>>> >>>>>> ## Rightscales CentOS AMI >>>>>> >>>>>> ##http://support.rightscale.com/18-Release_Notes/02-AMI/RightImages_Release_Notes >>>>>> jclouds.ec2.ami-owners=411009282317 >>>>>> whirr.image-id=us-east-1/ami-ccb35ea5 >>>>>> >>>>>> >>>>>> On Mon, Mar 7, 2011 at 9:59 AM, Benjamin Clark<b...@daltonclark.com> >>>>>> wrote: >>>>>>> >>>>>>> In my experience you need >>>>>>> >>>>>>> whirr.run-url-base=http://name-of-my-bucket-with-customized-scripts/ >>>>>>> whirr.hadoop-install-runurl=cloudera/cdh/install >>>>>>> >>>>>>> and then you *also* need to have a copy of sun/java/install in that same >>>>>>> bucket. >>>>>>> >>>>>>> And both of those scripts need to be public-readable. >>>>>>> >>>>>>> So in the end you should be able to do >>>>>>> curl >>>>>>> http://name-of-my-bucket-with-customized-scripts.s3.amazonaws.com/cloudera/cdh/install >>>>>>> >>>>>>> and >>>>>>> curl >>>>>>> http://name-of-my-bucket-with-customized-scripts.s3.amazonaws.com/sun/java/install >>>>>>> >>>>>>> Even if you haven't customizied sun/java/install, it needs to be there. >>>>>>> >>>>>>> If you do all that, the scripts will run and you will have the versions >>>>>>> you asked for. >>>>>>> >>>>>>> `hadoop version` on the name node then says, in my case: >>>>>>> >>>>>>> Hadoop 0.20.2-CDH3B4 >>>>>>> >>>>>>> On Mar 7, 2011, at 12:41 PM, Saptarshi Guha wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> Fixed the security slip up. Did the hadoop version thing and got this >>>>>>>> >>>>>>>> [root@domU-12-31-39-0B-CC-41 ~]# hadoop version >>>>>>>> Hadoop 0.20.2 >>>>>>>> Subversion >>>>>>>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 >>>>>>>> -r 911707 >>>>>>>> Compiled by chrisdo on Fri Feb 19 08:07:34 UTC 2010 >>>>>>>> >>>>>>>> So i guess its not CDH. >>>>>>>> >>>>>>>> Thanks >>>>>>>> Saptarshi >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Mar 7, 2011 at 9:22 AM, Saptarshi Guha >>>>>>>> <saptar...@revolutionanalytics.com> wrote: >>>>>>>>> >>>>>>>>> dear me! thanks, will do right away. >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Mar 7, 2011 at 1:46 AM, Sebastian Schoenherr >>>>>>>>> <sebastian.schoenh...@uibk.ac.at> wrote: >>>>>>>>>> >>>>>>>>>> Hi Saptarshi, >>>>>>>>>> Try to execute "hadoop version" on your namenode, if the output is >>>>>>>>>> Hadoop >>>>>>>>>> 0.20.2-CDH3B4, the current cloudera distribution has been installed. >>>>>>>>>> Btw, I would recommend to set your current Access Key ID and Secret >>>>>>>>>> Key >>>>>>>>>> inactive, since you posted it in your prop file. >>>>>>>>>> cheers >>>>>>>>>> sebastian >>>>>>>>>> >>>>>>>>>> On 06.03.2011 06:54, Saptarshi Guha wrote: >>>>>>>>>>> >>>>>>>>>>> Hello, >>>>>>>>>>> >>>>>>>>>>> I did git clone of the latest whirr and copied cloudera scripts into >>>>>>>>>>> the script directory (copied over >>>>>>>>>>> from whirr-0.3-incubating). >>>>>>>>>>> >>>>>>>>>>> My properties file is at the end of this email. >>>>>>>>>>> >>>>>>>>>>> However, I don't think the scripts are being run because the >>>>>>>>>>> jobtracker >>>>>>>>>>> is the default Apache hadoop jobtracker and not the cloudera >>>>>>>>>>> jobtracker. >>>>>>>>>>> >>>>>>>>>>> Have i missed something? >>>>>>>>>>> >>>>>>>>>>> Thanks in advance >>>>>>>>>>> >>>>>>>>>>> Saptarshi >>>>>>>>>>> >>>>>>>>>>> ## Properties >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> whirr.cluster-name=revotesting >>>>>>>>>>> whirr.service-name=hadoop >>>>>>>>>>> whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,2 >>>>>>>>>>> hadoop-datanode+hadoop-tasktracker >>>>>>>>>>> whirr.provider=aws-ec2 >>>>>>>>>>> whirr.identity= AKIAI3FUFFXAPYLE7CJA >>>>>>>>>>> whirr.credential= 2Yq3Ar2HSxK/hbwZHs6aN6yrh0yfGNSPTpVw3t2n >>>>>>>>>>> whirr.location-id=us-east-1 >>>>>>>>>>> whirr.hardware-id=c1.xlarge >>>>>>>>>>> >>>>>>>>>>> ## Rightscales CentOS AMI >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> http://support.rightscale.com/18-Release_Notes/02-AMI/RightImages_Release_Notes >>>>>>>>>>> jclouds.ec2.ami-owners=411009282317 >>>>>>>>>>> whirr.image-id=us-east-1/ami-ccb35ea5 >>>>>>>>>>> >>>>>>>>>>> whirr.hadoop-install-runurl=cloudera/cdh/install >>>>>>>>>>> whirr.hadoop-configure-runurl=cloudera/cdh/post-configure >>>>>>>>>> >>>>>>> >>>> >>>> >> >> >
