Hey,
I'm trying to find time to test again with your instructions. I started the YARN session before. I thing the main thing I forgot was the "-dist". Robert On Tue, Feb 3, 2015 at 6:29 PM, Leonidas Fegaras <[email protected]> wrote: > Oops, I forgot: In flink mode on a yarn cluster you need first to start > the Flink Yarn application in a seperate terminal: > ${FLINK_HOME}/bin/yarn-session.sh -n 10 > > > On 2/3/15 11:21 AM, Leonidas Fegaras wrote: > >> Hi Robert, >> Thanks for testing MRQL. >> I will open a new JIRA ticket to improve the configuration environment >> (this will be in the next release). >> I will also add more instructions to the MRQL web site about running >> MRQL in Flink mode. >> Like the other modes (MR, Spark, and Hama), you need to run MRQL in >> either -local or -dist mode. >> Runing without -local or -dist will run the queries in memory using >> plain Java on local files; this is for testing the query logic only (it >> doesn't use Flink). >> Your query failed because there is no tmp directory inside the directory >> you run the query (it tries to dump data to tmp/graph.bin but there is >> no tmp/). >> To run Flink on a Yarn cluster, say on 10 nodes, you need to compile >> MRQL with -Pyarn and then run: >> ./bin/mrql.flink -dist -nodes 10 ./queries/RMAT.mrql 100000 1000000 >> Best regards, >> Leonidas >> >> On 2/3/15 10:55 AM, Robert Metzger wrote: >> >>> Hi, >>> >>> I've tried out the release with Flink on YARN on Google Compute. >>> >>> Some general feedback regarding MRQL: >>> I found it a bit annoying to configure MRQL with all the variables in the >>> mrql-env.sh. For example the JAVA_HOME variable is set in the google >>> cloud >>> environment. Still, I have to set it in the file. >>> Also, the HADOOP_HOME variable is usually set. >>> >>> Also, the standard JRE on GCE doesn't have the "jar" utility located in >>> $JAVA_HOME/bin/jar, so the mrql scripts didn't work. I just replaced the >>> lines with "jar" instead of $JAVA_HOME/bin/jar and it worked. >>> >>> I found it confusing that " ./bin/mrql.flink" says "compiled in-memory >>> MapReduce mode". >>> >>> I don't know what I did wrong, but I got the following error when trying >>> to >>> run PageRank with Flink on YARN. >>> >>> robert@hadoop-m-hpfb:~/apache-mrql-0.9.4-incubating$ ./bin/mrql.flink >>> ./queries/RMAT.mrql 100000 1000000 >>> Apache MRQL version 0.9.4 (compiled in-memory MapReduce mode) >>> Query type: ( int, int, int, int ) -> ( int, int ) >>> Query type: !bag(( int, int )) >>> Physical plan: >>> MapReduce: >>> input: Generator >>> Exception in thread "main" java.lang.Error: >>> java.io.FileNotFoundException: >>> tmp/graph.bin.type (No such file or directory) >>> at org.apache.mrql.TopLevel.dump(TopLevel.java:195) >>> at org.apache.mrql.TopLevel.evaluate_top_level(TopLevel.java:324) >>> at org.apache.mrql.Translator.top_level(Translator.java:443) >>> at >>> org.apache.mrql.CUP$MRQLParser$actions.CUP$MRQLParser$do_action( >>> MRQLParser.java:4583) >>> at org.apache.mrql.MRQLParser.do_action(MRQLParser.java:2029) >>> at java_cup.runtime.lr_parser.parse(lr_parser.java:584) >>> at org.apache.mrql.Main.main(Main.java:184) >>> Caused by: java.io.FileNotFoundException: tmp/graph.bin.type (No such >>> file >>> or directory) >>> at java.io.FileOutputStream.open(Native Method) >>> at java.io.FileOutputStream.<init>(FileOutputStream.java:221) >>> at java.io.FileOutputStream.<init>(FileOutputStream.java:110) >>> at java.io.PrintStream.<init>(PrintStream.java:208) >>> at org.apache.mrql.MapReduceAlgebra.dump(MapReduceAlgebra.java:766) >>> at org.apache.mrql.TopLevel.dump(TopLevel.java:193) >>> ... 6 more >>> >>> >>> I didn't really verify the release itself, but maybe the feedback I gave >>> here is already helpful for the project. >>> >>> >>> Best, >>> Robert >>> >>> >>> >>> >>> On Mon, Feb 2, 2015 at 6:56 PM, Leonidas Fegaras <[email protected]> >>> wrote: >>> >>> Hello, >>>> A candidate for the Apache MRQL 0.9.4-incubating release is now >>>> available for vote. Please vote on releasing this package as Apache >>>> MRQL 0.9.4-incubating. The vote is open for at least 72 hours and >>>> passes if a majority of at least three +1 PPMC votes are cast. >>>> >>>> [ ] +1 Release this package as Apache MRQL 0.9.4-incubating >>>> [ ] 0 I don't feel strongly about it, but I'm okay with the release >>>> [ ] -1 Do not release this package because... >>>> >>>> The release tarballs, including signatures, digests, etc can be found >>>> at: >>>> https://dist.apache.org/repos/dist/dev/incubator/mrql/0.9.4- >>>> incubating-RC1/ >>>> The release candidate consists of the following source distribution >>>> archives: >>>> - apache-mrql-0.9.4-incubating-src.[tar.gz|zip] >>>> SHA1 of TGZ: D3A1 C4AB BD64 5565 B1CE 87DE FC8A 69BE CEDC 4A05 >>>> SHA1 of ZIP: 9B87 2EA0 BC81 3C58 9234 FBED 4D91 C293 DF52 59E7 >>>> You can compile the sources using 'mvn clean install'. >>>> In addition, the following supplementary binary distributions are >>>> provided for user convenience at the same location: >>>> - apache-mrql-0.9.4-incubating-bin.[tar.gz|zip] >>>> SHA1 of TGZ: 8240 E762 AEDB 608F 2BDD 33CB 038C 0382 1651 12D1 >>>> SHA1 of ZIP: 5C7B BDBE FC80 A20D 2776 4BDE A433 B46A 7628 A1D5 >>>> >>>> A staged Maven repository is available for review at: >>>> https://repository.apache.org/content/repositories/orgapachemrql-1002/ >>>> >>>> The release candidate has been signed through the key 798764F1 in: >>>> http://www.apache.org/dist/incubator/mrql/KEYS >>>> >>>> The release candidate is based on the sources tagged with >>>> MRQL-0.9.4-incubating-RC1 in: >>>> https://git-wip-us.apache.org/repos/asf?p=incubator-mrql. >>>> git;a=commit;h= >>>> ef7021da0c8e9999fea2698f61c5a0d119c6bf7b >>>> >>>> The list of fixed issues: >>>> https://git-wip-us.apache.org/repos/asf?p=incubator-mrql. >>>> git;a=blob_plain;f=RELEASE_NOTES;hb=ef7021da0c8e9999fea2698f61c5a0 >>>> d119c6bf7b >>>> >>>> Thanks, >>>> Leonidas Fegaras >>>> >>>> >>>> >
