[
https://issues.apache.org/jira/browse/PIG-4168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14173544#comment-14173544
]
liyunzhang_intel commented on PIG-4168:
---------------------------------------
Hi [~rohini]:
according to your comment, do some modications and upload PIG-4168_4.patch.
Following are modifications:
1. remove LowercaseUDF.java
2. return spark in getExecType in SparkMiniCluster
{code}
@Override
public ExecType getExecType() {
return new SparkExecType();
}
{code}
3. reuse existing unit tests. Now only add "TestUDF.java" in test/spark-tests.
Some other unit tests can not pass because of features not are completed in
spark mode.
We can add more to test/spark-tests when some features are completed.
> Initial implementation of unit tests for Pig on Spark
> -----------------------------------------------------
>
> Key: PIG-4168
> URL: https://issues.apache.org/jira/browse/PIG-4168
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Reporter: Praveen Rachabattuni
> Assignee: liyunzhang_intel
> Attachments: PIG-4168.patch, PIG-4168_1.patch, PIG-4168_2.patch,
> PIG-4168_3.patch, PIG-4168_4.patch
>
>
> 1.ant clean jar; pig-0.14.0-SNAPSHOT-core-h1.jar will be generated by the
> command
> 2.export SPARK_PIG_JAR=$PIG_HOME/pig-0.14.0-SNAPSHOT-core-h1.jar
> 3.build hadoop1 and spark env.spark run in local mode
> jps:
> 11647 Master #spark master runs
> 6457 DataNode #hadoop datanode runs
> 22374 Jps
> 11705 Worker# spark worker runs
> 27009 JobTracker #hadoop jobtracker runs
> 26602 NameNode #hadoop namenode runs
> 29486 org.eclipse.equinox.launcher_1.3.0.v20120522-1813.jar
> 19692 Main
>
> 4.ant test-spark
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
