So twill doesn’t check for "binary compatibility" with the cluster? If that is possible, I would (try to) provide a patch for that^^ If youre willing to lend me a hand and tell me which direction to search...
-----Ursprüngliche Nachricht----- Von: Gary Helmling [mailto:[email protected]] Gesendet: Donnerstag, 19. Juni 2014 19:27 An: [email protected] Betreff: Re: Twill application development requirements for windows Ah, that would be a problem. Glad that you worked out the issue and have things up and running now! On Thu, Jun 19, 2014 at 10:18 AM, helt <[email protected]> wrote: > Hi Gary, > > finally the hello world program started and finished successfully. > > The issue could be identified as a java version mismatch. The pom.xml > was configured to java use 1.7 code and binaries whereas the cluster > runs oracles jdk 1.6. > > DOH... > > Best regards, > Hendrik > > > > -----Ursprüngliche Nachricht----- > Von: helt [mailto:[email protected]] > Gesendet: Sonntag, 15. Juni 2014 14:56 > An: [email protected] > Betreff: AW: Twill application development requirements for windows > > Hi Gary, > > > > > > Thank you for your response. In the short run, I want to develop a > twill application on windows and then run it on a cluster. In the long > run, I probably will develop & test on windows using a local Hadoop, > but first, I would like to run hello world as a minimum working example. > > Then I can build on this to do my own stuff, but before I do that, I > want to go through the examples to verify/fix that my setup works > correctly, and that it will run on the cluster correctly… > > > > Which brings me to my actual question: > > What do I have to do to get the hello world example to run on the real > cluster? I managed to get around a few issues. But it seems like the > Hello World example is optimized towards a local cluster installation. > > > > So far I managed to fix (snippet at EOF): > > - Connection to yarn fixed by setting the yarn resource manager > address to sth != 0.0.0.0:8032 > > - Connection to hdfs fixed by providing a defaultFS. > > - Minimize the apps resource requirements, to make sure that yarn > is able to provide the needed resources. > > > > Hello World will start, but not finish, nor print the log messages… > > Open issues are therefore (from what I can see from here with my > limited > experience): > > - Fix log4j configurations so it stops complaining about “No > appenders” somehow. > > > > Wrap-up: Hello World configuration for running on a real cluster seems > to be quite cumbersome and is still unkown. Any advice on this? > > > > Best regards, > > Hendrik > > > > > > Twill configuration to get it running: > > public static void main(String[] args) { > > > > String zkConnectStr = "machine1:2181"; > > Configuration conf = new Configuration(); > > conf.set("yarn.resourcemanager.address", "machine1:8032"); > > conf.set("fs.defaultFS", "hdfs://machine1:8020"); > > > > YarnConfiguration yc = new YarnConfiguration(conf); > > final TwillRunnerService twillRunner = new > YarnTwillRunnerService(yc, zkConnectStr); > > twillRunner.startAndWait(); > > TwillPreparer twillPreparer = twillRunner.prepare(new > HelloWorldRunnable(), ResourceSpecification.Builder.with() > > .setVirtualCores(1) > > .setMemory(64, > SizeUnit.MEGA) > > .setInstances(1) > > .build()) > > .addLogHandler( > > new PrinterLogHandler(new PrintWriter(System.out, true))); > > final TwillController controller = twillPreparer.start(); > > > > > > > > -----Ursprüngliche Nachricht----- > > Von: Gary Helmling [mailto:[email protected]] > > Gesendet: Freitag, 6. Juni 2014 18:39 > > An: [email protected] > > Betreff: Re: Twill application development requirements for windows > > > > Hi Hendrik, > > > > Are you talking about developing a twill application on windows and > then running it on a cluster? That should be possible with just the > Hadoop client library jars (dependencies of twill). > > > > Or are you talking about developing and testing locally on a > pseudo-distributed Hadoop cluster running on your windows box? > > > > Please describe in a bit more detail what you want to do. > > > > Gary > > > > > > > > On Thu, Jun 5, 2014 at 11:19 AM, Hendrik Lücke-Tieke > <[email protected]> > > wrote: > > > > > Hi, > > > > > > > > > is it possible to develop and start a twill application on windows > > > without having to install hadoop binaries? Or a node needed on the > > local > machine? > > > > > > > > > Ty, > > > > > > Hendrik > >
