Hi Gary, finally the hello world program started and finished successfully.
The issue could be identified as a java version mismatch. The pom.xml was configured to java use 1.7 code and binaries whereas the cluster runs oracles jdk 1.6. DOH... Best regards, Hendrik -----Ursprüngliche Nachricht----- Von: helt [mailto:[email protected]] Gesendet: Sonntag, 15. Juni 2014 14:56 An: [email protected] Betreff: AW: Twill application development requirements for windows Hi Gary, Thank you for your response. In the short run, I want to develop a twill application on windows and then run it on a cluster. In the long run, I probably will develop & test on windows using a local Hadoop, but first, I would like to run hello world as a minimum working example. Then I can build on this to do my own stuff, but before I do that, I want to go through the examples to verify/fix that my setup works correctly, and that it will run on the cluster correctly… Which brings me to my actual question: What do I have to do to get the hello world example to run on the real cluster? I managed to get around a few issues. But it seems like the Hello World example is optimized towards a local cluster installation. So far I managed to fix (snippet at EOF): - Connection to yarn fixed by setting the yarn resource manager address to sth != 0.0.0.0:8032 - Connection to hdfs fixed by providing a defaultFS. - Minimize the apps resource requirements, to make sure that yarn is able to provide the needed resources. Hello World will start, but not finish, nor print the log messages… Open issues are therefore (from what I can see from here with my limited experience): - Fix log4j configurations so it stops complaining about “No appenders” somehow. Wrap-up: Hello World configuration for running on a real cluster seems to be quite cumbersome and is still unkown. Any advice on this? Best regards, Hendrik Twill configuration to get it running: public static void main(String[] args) { String zkConnectStr = "machine1:2181"; Configuration conf = new Configuration(); conf.set("yarn.resourcemanager.address", "machine1:8032"); conf.set("fs.defaultFS", "hdfs://machine1:8020"); YarnConfiguration yc = new YarnConfiguration(conf); final TwillRunnerService twillRunner = new YarnTwillRunnerService(yc, zkConnectStr); twillRunner.startAndWait(); TwillPreparer twillPreparer = twillRunner.prepare(new HelloWorldRunnable(), ResourceSpecification.Builder.with() .setVirtualCores(1) .setMemory(64, SizeUnit.MEGA) .setInstances(1) .build()) .addLogHandler( new PrinterLogHandler(new PrintWriter(System.out, true))); final TwillController controller = twillPreparer.start(); -----Ursprüngliche Nachricht----- Von: Gary Helmling [mailto:[email protected]] Gesendet: Freitag, 6. Juni 2014 18:39 An: [email protected] Betreff: Re: Twill application development requirements for windows Hi Hendrik, Are you talking about developing a twill application on windows and then running it on a cluster? That should be possible with just the Hadoop client library jars (dependencies of twill). Or are you talking about developing and testing locally on a pseudo-distributed Hadoop cluster running on your windows box? Please describe in a bit more detail what you want to do. Gary On Thu, Jun 5, 2014 at 11:19 AM, Hendrik Lücke-Tieke <[email protected]> wrote: > Hi, > > > is it possible to develop and start a twill application on windows > without having to install hadoop binaries? Or a node needed on the local > machine? > > > Ty, > > Hendrik
