[
https://issues.apache.org/jira/browse/PIG-4621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16150432#comment-16150432
]
Adam Szita commented on PIG-4621:
---------------------------------
[~YaShock]: the tests returned a few errors:
# Spark mode:
#* All rank related tests: this is because a missing nullcheck in the
RankConverter around {{illustrator.getData().clear();
illustrator.getEquivalenceClasses().get(0).clear();}}
#* TestPigContext#testImportList: the issue is due to POLoad getting passed in
LoadConverter#ToTupleFunction#constructor. Looks like if a POLoad instance is
passed, then for some reason the SparkEngineConf instance (which is also passed
here) is not deserialized on the Spark end in time and we need the UDF import
list before we would have an idea what classes we need. This results in Pig
being unable to resolve TestUDF2 UDF. I propose we make an exception for POLoad
and do the work of sparkIllustrate method inside the
LoadConverter#ToTupleFunction. (We can pass the operatorkey of POLoad insteadof
the op itself so that we can find the appropriate Illustrator instance)
# Tez mode:
#* we're missing some assumeFalse's in TestGrunt#testIllustrateScript(1-7),
these cases should be skipped in Tez mode: {{Assume.assumeFalse("Skip this test
for TEZ. See PIG-3993", Util.isTezExecType(cluster.getExecType()));}}
# MR mode:
#* no failures
+ a few smaller comments:
* POSort#sparkIllustrate method is indented with tabs in some lines, please
change to spaces
* POJoinGroupSpark#sparkIllustratorMarkup when catching ExecException let's log
an error instead of printStackTrace
* TezExecutionEngine please remove unused import LocalSparkSimulator
Can you please upload a patch with these fixes?
> Enable Illustrate in spark
> --------------------------
>
> Key: PIG-4621
> URL: https://issues.apache.org/jira/browse/PIG-4621
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Reporter: liyunzhang_intel
> Assignee: Jakov Rabinovits
> Labels: MissingFeature
> Fix For: spark-branch
>
> Attachments: PIG-4621.patch
>
>
> Current we don't support illustrate in spark mode.
> How illustrate works
> see:http://pig.apache.org/docs/r0.7.0/piglatin_ref2.html#ILLUSTRATE
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
