Hi Robert,
Thanks for testing MRQL.
I will open a new JIRA ticket to improve the configuration environment
(this will be in the next release).
I will also add more instructions to the MRQL web site about running
MRQL in Flink mode.
Like the other modes (MR, Spark, and Hama), you need to run MRQL in
either -local or -dist mode.
Runing without -local or -dist will run the queries in memory using
plain Java on local files; this is for testing the query logic only (it
doesn't use Flink).
Your query failed because there is no tmp directory inside the directory
you run the query (it tries to dump data to tmp/graph.bin but there is
no tmp/).
To run Flink on a Yarn cluster, say on 10 nodes, you need to compile
MRQL with -Pyarn and then run:
./bin/mrql.flink -dist -nodes 10 ./queries/RMAT.mrql 100000 1000000
Best regards,
Leonidas
On 2/3/15 10:55 AM, Robert Metzger wrote:
Hi,
I've tried out the release with Flink on YARN on Google Compute.
Some general feedback regarding MRQL:
I found it a bit annoying to configure MRQL with all the variables in the
mrql-env.sh. For example the JAVA_HOME variable is set in the google cloud
environment. Still, I have to set it in the file.
Also, the HADOOP_HOME variable is usually set.
Also, the standard JRE on GCE doesn't have the "jar" utility located in
$JAVA_HOME/bin/jar, so the mrql scripts didn't work. I just replaced the
lines with "jar" instead of $JAVA_HOME/bin/jar and it worked.
I found it confusing that " ./bin/mrql.flink" says "compiled in-memory
MapReduce mode".
I don't know what I did wrong, but I got the following error when trying to
run PageRank with Flink on YARN.
robert@hadoop-m-hpfb:~/apache-mrql-0.9.4-incubating$ ./bin/mrql.flink
./queries/RMAT.mrql 100000 1000000
Apache MRQL version 0.9.4 (compiled in-memory MapReduce mode)
Query type: ( int, int, int, int ) -> ( int, int )
Query type: !bag(( int, int ))
Physical plan:
MapReduce:
input: Generator
Exception in thread "main" java.lang.Error: java.io.FileNotFoundException:
tmp/graph.bin.type (No such file or directory)
at org.apache.mrql.TopLevel.dump(TopLevel.java:195)
at org.apache.mrql.TopLevel.evaluate_top_level(TopLevel.java:324)
at org.apache.mrql.Translator.top_level(Translator.java:443)
at
org.apache.mrql.CUP$MRQLParser$actions.CUP$MRQLParser$do_action(MRQLParser.java:4583)
at org.apache.mrql.MRQLParser.do_action(MRQLParser.java:2029)
at java_cup.runtime.lr_parser.parse(lr_parser.java:584)
at org.apache.mrql.Main.main(Main.java:184)
Caused by: java.io.FileNotFoundException: tmp/graph.bin.type (No such file
or directory)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
at java.io.FileOutputStream.<init>(FileOutputStream.java:110)
at java.io.PrintStream.<init>(PrintStream.java:208)
at org.apache.mrql.MapReduceAlgebra.dump(MapReduceAlgebra.java:766)
at org.apache.mrql.TopLevel.dump(TopLevel.java:193)
... 6 more
I didn't really verify the release itself, but maybe the feedback I gave
here is already helpful for the project.
Best,
Robert
On Mon, Feb 2, 2015 at 6:56 PM, Leonidas Fegaras <[email protected]>
wrote:
Hello,
A candidate for the Apache MRQL 0.9.4-incubating release is now
available for vote. Please vote on releasing this package as Apache
MRQL 0.9.4-incubating. The vote is open for at least 72 hours and
passes if a majority of at least three +1 PPMC votes are cast.
[ ] +1 Release this package as Apache MRQL 0.9.4-incubating
[ ] 0 I don't feel strongly about it, but I'm okay with the release
[ ] -1 Do not release this package because...
The release tarballs, including signatures, digests, etc can be found at:
https://dist.apache.org/repos/dist/dev/incubator/mrql/0.9.4-
incubating-RC1/
The release candidate consists of the following source distribution
archives:
- apache-mrql-0.9.4-incubating-src.[tar.gz|zip]
SHA1 of TGZ: D3A1 C4AB BD64 5565 B1CE 87DE FC8A 69BE CEDC 4A05
SHA1 of ZIP: 9B87 2EA0 BC81 3C58 9234 FBED 4D91 C293 DF52 59E7
You can compile the sources using 'mvn clean install'.
In addition, the following supplementary binary distributions are
provided for user convenience at the same location:
- apache-mrql-0.9.4-incubating-bin.[tar.gz|zip]
SHA1 of TGZ: 8240 E762 AEDB 608F 2BDD 33CB 038C 0382 1651 12D1
SHA1 of ZIP: 5C7B BDBE FC80 A20D 2776 4BDE A433 B46A 7628 A1D5
A staged Maven repository is available for review at:
https://repository.apache.org/content/repositories/orgapachemrql-1002/
The release candidate has been signed through the key 798764F1 in:
http://www.apache.org/dist/incubator/mrql/KEYS
The release candidate is based on the sources tagged with
MRQL-0.9.4-incubating-RC1 in:
https://git-wip-us.apache.org/repos/asf?p=incubator-mrql.git;a=commit;h=
ef7021da0c8e9999fea2698f61c5a0d119c6bf7b
The list of fixed issues:
https://git-wip-us.apache.org/repos/asf?p=incubator-mrql.
git;a=blob_plain;f=RELEASE_NOTES;hb=ef7021da0c8e9999fea2698f61c5a0
d119c6bf7b
Thanks,
Leonidas Fegaras
