[jira] [Commented] (PIG-4681) Enable Pig on Spark to run on Yarn Cluster mode
[ https://issues.apache.org/jira/browse/PIG-4681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14909168#comment-14909168 ] Srikanth Sundarrajan commented on PIG-4681: --- In the cluster mode the Spark Context will be created on the AM and all the pig execution need to move to the AM. This requires us to initialize pigContext on the AM as well. Given that pig has always had a thick client which controlled the execution from this client, am inclined to mark this as "Wont Fix". I intend to add necessary notes in Docs/FAQs as a patch as part of this JIRA for future reference. [~xuefuz], What are your thoughts? > Enable Pig on Spark to run on Yarn Cluster mode > --- > > Key: PIG-4681 > URL: https://issues.apache.org/jira/browse/PIG-4681 > Project: Pig > Issue Type: Sub-task > Components: spark >Affects Versions: spark-branch >Reporter: Srikanth Sundarrajan >Assignee: Srikanth Sundarrajan > Labels: spork > Fix For: spark-branch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (PIG-4236) Avoid packaging spark specific jars into pig fat jar
[ https://issues.apache.org/jira/browse/PIG-4236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14909169#comment-14909169 ] Srikanth Sundarrajan commented on PIG-4236: --- Am assuming PIG-4667 has already addressed this by shipping spark specific jars for execution and avoids bundling this in pig uber jar. [~praveenr019], [~xuefuz], Do you think there is anything left to be done on this ? > Avoid packaging spark specific jars into pig fat jar > > > Key: PIG-4236 > URL: https://issues.apache.org/jira/browse/PIG-4236 > Project: Pig > Issue Type: Sub-task > Components: spark >Reporter: Praveen Rachabattuni > Fix For: spark-branch > > > Spark and its dependencies jars shouldn't be packaged into the pig jar as it > would be redundant to do so. > Pig dependencies should be either added from lib directory or use the legacy > jar which contains pig along with its dependencies. > Please refer to PIG-4047 for more info. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] Subscription: PIG patch available
Issue Subscription Filter: PIG patch available (30 issues) Subscriber: pigdaily Key Summary PIG-4684Exception should be changed to warning when job diagnostics cannot be fetched https://issues.apache.org/jira/browse/PIG-4684 PIG-4677Display failure information on stop on failure https://issues.apache.org/jira/browse/PIG-4677 PIG-4670Embedded Python scripts still parse line by line https://issues.apache.org/jira/browse/PIG-4670 PIG-4656Improve String serialization and comparator performance in BinInterSedes https://issues.apache.org/jira/browse/PIG-4656 PIG-4641Print the instance of Object without using toString() https://issues.apache.org/jira/browse/PIG-4641 PIG-4598Allow user defined plan optimizer rules https://issues.apache.org/jira/browse/PIG-4598 PIG-4581thread safe issue in NodeIdGenerator https://issues.apache.org/jira/browse/PIG-4581 PIG-4554Compress pig.script before encoding https://issues.apache.org/jira/browse/PIG-4554 PIG-4539New PigUnit https://issues.apache.org/jira/browse/PIG-4539 PIG-4515org.apache.pig.builtin.Distinct throws ClassCastException https://issues.apache.org/jira/browse/PIG-4515 PIG-4468Pig's jackson version conflicts with that of hadoop 2.6.0 https://issues.apache.org/jira/browse/PIG-4468 PIG-4455Should use DependencyOrderWalker instead of DepthFirstWalker in MRPrinter https://issues.apache.org/jira/browse/PIG-4455 PIG-4417Pig's register command should support automatic fetching of jars from repo. https://issues.apache.org/jira/browse/PIG-4417 PIG-4373Implement PIG-3861 in Tez https://issues.apache.org/jira/browse/PIG-4373 PIG-4341Add CMX support to pig.tmpfilecompression.codec https://issues.apache.org/jira/browse/PIG-4341 PIG-4323PackageConverter hanging in Spark https://issues.apache.org/jira/browse/PIG-4323 PIG-4313StackOverflowError in LIMIT operation on Spark https://issues.apache.org/jira/browse/PIG-4313 PIG-4251Pig on Storm https://issues.apache.org/jira/browse/PIG-4251 PIG-4208Make merge-sparse join work with Spark https://issues.apache.org/jira/browse/PIG-4208 PIG-4111Make Pig compiles with avro-1.7.7 https://issues.apache.org/jira/browse/PIG-4111 PIG-4002Disable combiner when map-side aggregation is used https://issues.apache.org/jira/browse/PIG-4002 PIG-3952PigStorage accepts '-tagSplit' to return full split information https://issues.apache.org/jira/browse/PIG-3952 PIG-3911Define unique fields with @OutputSchema https://issues.apache.org/jira/browse/PIG-3911 PIG-3877Getting Geo Latitude/Longitude from Address Lines https://issues.apache.org/jira/browse/PIG-3877 PIG-3873Geo distance calculation using Haversine https://issues.apache.org/jira/browse/PIG-3873 PIG-3866Create ThreadLocal classloader per PigContext https://issues.apache.org/jira/browse/PIG-3866 PIG-3864ToDate(userstring, format, timezone) computes DateTime with strange handling of Daylight Saving Time with location based timezones https://issues.apache.org/jira/browse/PIG-3864 PIG-3851Upgrade jline to 2.11 https://issues.apache.org/jira/browse/PIG-3851 PIG-3668COR built-in function when atleast one of the coefficient values is NaN https://issues.apache.org/jira/browse/PIG-3668 PIG-3587add functionality for rolling over dates https://issues.apache.org/jira/browse/PIG-3587 You may edit this subscription at: https://issues.apache.org/jira/secure/FilterSubscription!default.jspa?subId=16328&filterId=12322384
