----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26769/ -----------------------------------------------------------
Review request for drill, Aditya Kishore, Jinfeng Ni, Mehant Baid, and Timothy Chen. Repository: drill-git Description ------- Dev Patch - Only for review - Not Commit ready - ------------------------------------------------- This review comment is only for getting review comments. Current dev is just a poc without best practices in place. Would be implementing all the comments as I keep adding any code. - Support for Load: Reads text files only. Picks delimeter provided in PigStorage(). Default \t. - Support for store: Only dumps on --SCREEN-- for now. - Support for LIMIT. - Support for JOIN: supports all joins by Pig apart from FULL OUTER(not supported by drill currently). Only supports alias based joins not index based($0 etc). - Does not support un terminated statements. ie. Operator that does not output/used in SINK operator. Known Issues: - Sqlline Hack: Need to take a call on this. One solution is suggested by Julian to have a custom Sqlline with both versions of JLine - so as to have compatibility with Pig's Sqlline - Drill's jdbc-all's build failure: Not able to pass through jdbc-all's Obfuscation failure error. Solution suggested by Aditya did not help me (removed jetty's jars along with other dependencies). Any tips would be appreciated here. Diffs ----- common/src/main/java/org/apache/drill/common/expression/UnquotedFieldReference.java PRE-CREATION common/src/main/java/org/apache/drill/common/logical/data/Join.java df7e583 exec/java-exec/pom.xml d05e4c6 exec/java-exec/src/main/java/org/apache/drill/exec/pigparser/parser/ExtendedPigServer.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/pigparser/parser/PigLatinParser.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/pigparser/parser/PigParserUtil.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/pigparser/parser/PigParsingException.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/pigparser/parser/PigPlanTranslator.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/pigparser/planconverter/TestPigParsing.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillImplementor.java 7fff6e9 exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/QueryWrapper.java 7bf3909 exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java 0a34a22 exec/java-exec/src/main/resources/rest/query/query.ftl 5033aca exec/java-exec/src/test/resources/pigparsing/sample-script-micro.txt PRE-CREATION exec/java-exec/src/test/resources/pigparsing/sample-script-mini.txt PRE-CREATION exec/java-exec/src/test/resources/pigparsing/sample-script.txt PRE-CREATION exec/java-exec/src/test/resources/pigparsing/test-pig-logical-plan.json PRE-CREATION exec/java-exec/src/test/resources/store/text/data/nations_caratseparated.txt PRE-CREATION exec/java-exec/src/test/resources/store/text/data/nations_tsv.txt PRE-CREATION exec/jdbc-all/pom.xml 349366b exec/pom.xml f27320e pom.xml 70abb75 protocol/src/main/java/org/apache/drill/exec/proto/UserBitShared.java 4f01053 protocol/src/main/java/org/apache/drill/exec/proto/beans/QueryType.java 6c8f3c5 protocol/src/main/java/org/apache/drill/exec/proto/beans/RpcType.java 3f1f9fd protocol/src/main/protobuf/UserBitShared.proto 3ad9a09 Diff: https://reviews.apache.org/r/26769/diff/ Testing ------- Testing done on Drill Rest Interface only via PigLatin script. Sample Pig Script: ----------------------------------------------------------------------------------- regions = LOAD '/home/yash/work/git/drill-win-forked/incubator-drill/exec/java-exec/src/test/resources/store/text/data/regions.csv' USING PigStorage(',') AS ( zone_id : int, zone_name : chararray, comment : chararray ); nations = LOAD '/home/yash/work/git/drill-win-forked/incubator-drill/exec/java-exec/src/test/resources/store/text/data/nations_caratseparated.txt' USING PigStorage('^') AS ( nation_id : int, nation_name : chararray, zone_id : int, comment : chararray ); r = limit regions 5; r1 = limit r 5; r = limit r1 2; n = limit nations 3; n1 = limit n 3; n = limit n1 2; -- JOIN Currently only works with named types e = JOIN r BY zone_id left outer , n BY zone_id; f = limit e 5; store f into 'some_location'; ----------------------------------------------------------------------------------- Logical Plan Generated: ----------------------------------------------------------------------------------- { "head" : { "version" : 1, "generator" : { "type" : "org.apache.drill.exec.pigparser.parser.PigPlanTranslator$PlanTranslator", "info" : "drill-pig-parser" }, "type" : "APACHE_DRILL_LOGICAL", "options" : null, "queue" : 0, "resultMode" : "EXEC" }, "storage" : { "cp" : { "type" : "file", "enabled" : true, "connection" : "file:///", "workspaces" : null, "formats" : { "delim^" : { "type" : "text", "extensions" : [ null ], "delimiter" : "^" }, "csv" : { "type" : "text", "extensions" : [ "csv" ], "delimiter" : "," } } } }, "query" : [ { "op" : "scan", "@id" : 1, "storageengine" : "cp", "selection" : { "format" : { "type" : "named", "name" : "csv" }, "files" : [ "file:/home/yash/work/git/drill-win-forked/incubator-drill/exec/java-exec/src/test/resources/store/text/data/regions.csv" ] } }, { "op" : "project", "@id" : 2, "input" : 1, "projections" : [ { "ref" : "`zone_id`", "expr" : "`columns`[0]" }, { "ref" : "`zone_name`", "expr" : "`columns`[1]" }, { "ref" : "`comment`", "expr" : "`columns`[2]" } ] }, { "op" : "limit", "@id" : 3, "input" : 2, "first" : 0, "last" : 5 }, { "op" : "limit", "@id" : 4, "input" : 3, "first" : 0, "last" : 5 }, { "op" : "limit", "@id" : 5, "input" : 4, "first" : 0, "last" : 2 }, { "op" : "project", "@id" : 6, "input" : 5, "projections" : [ { "ref" : "`r::zone_name`", "expr" : "`zone_name`" }, { "ref" : "`r::zone_id`", "expr" : "`zone_id`" }, { "ref" : "`r::comment`", "expr" : "`comment`" } ] }, { "op" : "scan", "@id" : 7, "storageengine" : "cp", "selection" : { "format" : { "type" : "named", "name" : "delim^" }, "files" : [ "file:/home/yash/work/git/drill-win-forked/incubator-drill/exec/java-exec/src/test/resources/store/text/data/nations_caratseparated.txt" ] } }, { "op" : "project", "@id" : 8, "input" : 7, "projections" : [ { "ref" : "`nation_id`", "expr" : "`columns`[0]" }, { "ref" : "`nation_name`", "expr" : "`columns`[1]" }, { "ref" : "`zone_id`", "expr" : "`columns`[2]" }, { "ref" : "`comment`", "expr" : "`columns`[3]" } ] }, { "op" : "limit", "@id" : 9, "input" : 8, "first" : 0, "last" : 3 }, { "op" : "limit", "@id" : 10, "input" : 9, "first" : 0, "last" : 3 }, { "op" : "limit", "@id" : 11, "input" : 10, "first" : 0, "last" : 2 }, { "op" : "project", "@id" : 12, "input" : 11, "projections" : [ { "ref" : "`n::zone_id`", "expr" : "`zone_id`" }, { "ref" : "`n::nation_id`", "expr" : "`nation_id`" }, { "ref" : "`n::comment`", "expr" : "`comment`" }, { "ref" : "`n::nation_name`", "expr" : "`nation_name`" } ] }, { "op" : "join", "@id" : 13, "left" : 6, "right" : 12, "conditions" : [ { "relationship" : "==", "left" : "`r::zone_id`", "right" : "`n::zone_id`" } ], "type" : "LEFT" }, { "op" : "project", "@id" : 14, "input" : 13, "projections" : [ { "ref" : "`r::zone_name`", "expr" : "`r::zone_name`" }, { "ref" : "`r::zone_id`", "expr" : "`r::zone_id`" }, { "ref" : "`r::comment`", "expr" : "`r::comment`" }, { "ref" : "`n::zone_id`", "expr" : "`n::zone_id`" }, { "ref" : "`n::nation_id`", "expr" : "`n::nation_id`" }, { "ref" : "`n::comment`", "expr" : "`n::comment`" }, { "ref" : "`n::nation_name`", "expr" : "`n::nation_name`" } ] }, { "op" : "limit", "@id" : 15, "input" : 14, "first" : 0, "last" : 5 }, { "op" : "store", "@id" : 16, "input" : 15, "target" : null, "storageEngine" : "--SCREEN--" } ] } ----------------------------------------------------------------------------------- Thanks, Yash Sharma
