Hi Andrei, Many thanks again. Will see those links and try to create a cluster with the tools I need.
Will be happy to contribute in any way I can. Will see the Wiki on how to contribute and will proceed from there. /PD. On Sun, Dec 4, 2011 at 1:10 PM, Andrei Savu <[email protected]> wrote: > The best way would be to define a new service for each one with new > install / configure scripts. > > See this presentation about creating a new Whirr service: > http://www.oscon.com/oscon2011/public/schedule/detail/19214 > > For Flume the following repo contains some working code: > https://github.com/tomwhite/whirr-service-example > > For Hive we are tracking progress here: > https://issues.apache.org/jira/browse/WHIRR-343 > > Currently there is no open issue for adding Sqoop as a service. > > I strongly recommend you to submit patches / pull requests as early as > possible and I will give as > much assistance as needed. See the following wiki page to learn more about > how to contribute: > https://cwiki.apache.org/confluence/display/WHIRR/How+To+Contribute > > On Sun, Dec 4, 2011 at 11:00 PM, Periya.Data <[email protected]>wrote: > >> Thanks much. Will follow your suggestions. A quick question: >> >> If I have to install Hive, Sqoop and Flume on the my cluster nodes, what >> is the suggested way? >> --> Log into each cluster node and install them manually? >> --> Maybe the I have to hack the install script >> (/whirr/services/cdh/src/main/resources/functions) or create a new one for >> each of the tools? >> >> Thanks, >> PD. >> >> >> On Sun, Dec 4, 2011 at 12:44 PM, Andrei Savu <[email protected]>wrote: >> >>> Here are a few advices: >>> >>> * use Ubuntu 10.04 LTS 64bit on the cluster nodes - this is the OS >>> version we are testing against before making a release >>> * whirr.hadoop.tarball.url is irrelevant if you specify >>> whirr.hadoop.install-function. Same is true for whirr.hadoop.version. >>> * install_cdh_hadoop will install by default the latest release. >>> We've implemented a way of selecting the version in 0.7.0: >>> https://issues.apache.org/jira/browse/WHIRR-415 >>> * use something bigger than t1.micro - this is just too small to run >>> Hadoop >>> >>> Let me know if you are still having problems. >>> >>> -- Andrei Savu / andreisavu.ro >>> >>> >>> On Sun, Dec 4, 2011 at 9:44 PM, Periya.Data <[email protected]>wrote: >>> >>>> Hi, >>>> I am trying Whirr to spin-up EC2 clusters. Here are the specs: >>>> >>>> >>>> - Local laptop - Ubuntu 11.10 Oneiric Ocelot + Hadoop-0.20.2-cdh3u2 >>>> - EC2 Whirr config : >>>> - >>>> - whirr.hadoop.install-function=install_cdh_hadoop >>>> - whirr.hadoop.configure-function=configure_cdh_hadoop >>>> - whirr.hadoop.version=0.20.2 >>>> - whirr.hadoop.tarball.url= >>>> http://archive.cloudera.com/cdh/3/hadoop-0.20.2-cdh3u2.tar.gz >>>> - whirr.hardware-id=t1.micro >>>> # Ubuntu 11.10 Oneiric EBS boot from alestic >>>> - whirr.image-id=us-east-1/ami-a7f539ce >>>> - whirr.location-id=us-east-1 >>>> >>>> >>>> *Problem:* The cluster seems to be up and running. But, cannot see >>>> Java and Hadoop on the instance. >>>> >>>> *Question:* Is this a problem or should I clearly mention in my Whirr >>>> config/properties file what version of JDK to be installed and its download >>>> URL? >>>> >>>> >>>> *Some logs:* (I ssh-ed into one of the nodes and tried to see the logs) >>>> >>>> sri@domU-12-31-39-02-21-20:/tmp$ ls -al >>>> total 40 >>>> drwxrwxrwt 4 root root 4096 2011-12-04 06:25 . >>>> drwxr-xr-x 22 root root 4096 2011-12-04 05:46 .. >>>> -rw-r--r-- 1 root root 938 2011-12-04 05:46 core-site.xml >>>> -rw-r--r-- 1 root root 701 2011-12-04 05:46 hdfs-site.xml >>>> drwxr-xr-x 2 root root 4096 2011-12-04 05:46 >>>> jclouds-script-1322977554131 >>>> drwxr-xr-x 2 root root 4096 2011-12-04 05:44 logs >>>> -rw-r--r-- 1 root root 1447 2011-12-04 05:46 mapred-site.xml >>>> -rwxr--r-- 1 root root 11212 2011-12-04 05:44 setup-sri.sh >>>> sri@domU-12-31-39-02-21-20:/tmp$ >>>> sri@domU-12-31-39-02-21-20:/tmp$ >>>> sri@domU-12-31-39-02-21-20:/tmp$ >>>> sri@domU-12-31-39-02-21-20:/tmp$ cd logs/ >>>> sri@domU-12-31-39-02-21-20:/tmp/logs$ ls -l >>>> total 12 >>>> -rw-r--r-- 1 root root 117 2011-12-04 05:45 stderr.log >>>> -rw-r--r-- 1 root root 4535 2011-12-04 05:45 stdout.log >>>> sri@domU-12-31-39-02-21-20:/tmp/logs$ more stderr.log >>>> E: Package 'sun-java6-jdk' has no installation candidate >>>> /tmp/setup-sri.sh: line 161: java: command not found >>>> sri@domU-12-31-39-02-21-20:/tmp/logs$ >>>> >>>> ---------- >>>> >>>> Get:36 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/universe >>>> i386 Packages [57.2 kB] >>>> Get:37 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/multiverse >>>> i386 Packages [4396 B] >>>> Get:38 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/main >>>> TranslationIndex [73 B] >>>> Get:39 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/multiverse >>>> TranslationIndex [72 B >>>> ] >>>> Get:40 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/universe >>>> TranslationIndex [73 B] >>>> Get:41 http://us-east-1.ec2.archive.ubuntu.com oneiric/main >>>> Translation-en [701 kB] >>>> Hit http://us-east-1.ec2.archive.ubuntu.com oneiric/multiverse >>>> Translation-en >>>> Get:42 http://us-east-1.ec2.archive.ubuntu.com oneiric/universe >>>> Translation-en [3165 kB] >>>> Get:43 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/main >>>> Translation-en [99.2 kB] >>>> Get:44 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/multiverse >>>> Translation-en [2419 B >>>> ] >>>> Get:45 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/universe >>>> Translation-en [35.4 kB] >>>> Fetched 16.2 MB in 12s (1290 kB/s) >>>> Reading package lists... >>>> Reading package lists... >>>> Building dependency tree... >>>> Reading state information... >>>> Package sun-java6-jdk is not available, but is referred to by another >>>> package. >>>> This may mean that the package is missing, has been obsoleted, or >>>> is only available from another source >>>> >>>> sri@domU-12-31-39-02-21-20:/tmp/logs$ >>>> Get:36 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/universe >>>> i386 Packages [57.2 kB] >>>> Get:37 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/multiverse >>>> i386 Packages [4396 B] >>>> Get:38 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/main >>>> TranslationIndex [73 B] >>>> Get:39 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/multiverse >>>> TranslationIndex [72 B] >>>> Get:40 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/universe >>>> TranslationIndex [73 B] >>>> Get:41 http://us-east-1.ec2.archive.ubuntu.com oneiric/main >>>> Translation-en [701 kB] >>>> Hit http://us-east-1.ec2.archive.ubuntu.com oneiric/multiverse >>>> Translation-en >>>> Get:42 http://us-east-1.ec2.archive.ubuntu.com oneiric/universe >>>> Translation-en [3165 kB] >>>> Get:43 http://us-east-1.ec2.archive.ubuntu.com oneiric-updates/main >>>> Translation-en [99.2 kB] >>>> Get:44 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/multiverse >>>> Translation-en [2419 B] >>>> Get:45 http://us-east-1.ec2.archive.ubuntu.comoneiric-updates/universe >>>> Translation-en [35.4 kB] >>>> Fetched 16.2 MB in 12s (1290 kB/s) >>>> Reading package lists... >>>> Reading package lists... >>>> Building dependency tree... >>>> Reading state information... >>>> Package sun-java6-jdk is not available, but is referred to by another >>>> package. >>>> This may mean that the package is missing, has been obsoleted, or >>>> is only available from another source >>>> >>>> sri@domU-12-31-39-02-21-20:/tmp/logs$ >>>> >>>> *[On a different cluster spun-up at a later time]* >>>> >>>> sri@domU-12-31-39-10-6D-38:~$ java >>>> The program 'java' can be found in the following packages: >>>> * gcj-4.4-jre-headless >>>> * gcj-4.6-jre-headless >>>> * openjdk-6-jre-headless >>>> * gcj-4.5-jre-headless >>>> * openjdk-7-jre-headless >>>> Ask your administrator to install one of them >>>> sri@domU-12-31-39-10-6D-38:~$ java -version >>>> The program 'java' can be found in the following packages: >>>> * gcj-4.4-jre-headless >>>> * gcj-4.6-jre-headless >>>> * openjdk-6-jre-headless >>>> * gcj-4.5-jre-headless >>>> * openjdk-7-jre-headless >>>> Ask your administrator to install one of them >>>> sri@domU-12-31-39-10-6D-38:~$ >>>> >>>> sri@domU-12-31-39-10-6D-38:~$ hadoop version >>>> hadoop: command not found >>>> sri@domU-12-31-39-10-6D-38:~$ >>>> >>>> =========================================== >>>> >>>> Finally, a quick Googling pointed to a JIRA issue: >>>> https://issues.apache.org/jira/browse/WHIRR-435 >>>> >>>> >>>> I am seeing the same stuff. Any updates? If this is a real issues, I do >>>> not have any problems in using Ubuntu 11.04 AMI for now. >>>> >>>> >>>> Your help is very much appreciated. >>>> >>>> >>>> Thanks, >>>> >>>> PD. >>>> >>> >>> >> >
