[
https://issues.apache.org/jira/browse/PIG-4754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15048317#comment-15048317
]
Xianda Ke commented on PIG-4754:
--------------------------------
Hi [~kellyzly] & [~mohitsabharwal], patch is attached. please help review.
Summaries:
When run javascript or python script, BoundScript.java will create multipule
threads, each thread will create a SparkLauncher. Race condition occurs!
1. Only one SparkContext may be active per JVM (SPARK-2243),
SparkLauncher.sparkContext is a static member, should be initialized only once
2. Delete and Copy a Jar file should be inside a critical section.
Otherwise, java.nio.file.FileAlreadyExistsException is thrown out.
How to test:
ant -Dhadoopversion=23 -Dexectype=spark -Dtestcase=TestScriptLanguage test
> Fix UT failures in TestScriptLanguage
> -------------------------------------
>
> Key: PIG-4754
> URL: https://issues.apache.org/jira/browse/PIG-4754
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Reporter: Xianda Ke
> Assignee: Xianda Ke
> Fix For: spark-branch
>
> Attachments: PIG-4754.patch
>
>
> org.apache.pig.test.TestScriptLanguage.runParallelTest2
> Error Message
> job should succeed
> Stacktrace
> junit.framework.AssertionFailedError: job should succeed
> at
> org.apache.pig.test.TestScriptLanguage.runPigRunner(TestScriptLanguage.java:96)
> at
> org.apache.pig.test.TestScriptLanguage.runPigRunner(TestScriptLanguage.java:105)
> at
> org.apache.pig.test.TestScriptLanguage.runParallelTest2(TestScriptLanguage.java:311)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
