[
https://issues.apache.org/jira/browse/PIG-1265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12839938#action_12839938
]
Ashutosh Chauhan commented on PIG-1265:
---------------------------------------
Looked at the diff of first and second patch. +1 for that part.
One thing unrelated to patch I want to highlight is: these kind of issues
creeps in because our api is too wide, there are multiple ways of getting same
thing done in Pig (executing script through Java api in this case), each
exercising different code paths. So, first thing we need is to establish what
is part of our api and what is meant for Pig's internal purposes only? Is
every single public method part of api ? Or, only those which have properly
documented Javadocs are supported ? Or only those documented in wiki page ?
First we establish what constitutes our public api, then we should
systematically start decreasing the visibility of those methods which are
public but not meant as an api, then deprecate them and eventually remove them.
> Change LoadMetadata and StoreMetadata to use Job instead of Configuraiton and
> add a cleanupOnFailure method to StoreFuncInterface
> ---------------------------------------------------------------------------------------------------------------------------------
>
> Key: PIG-1265
> URL: https://issues.apache.org/jira/browse/PIG-1265
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.7.0
> Reporter: Pradeep Kamath
> Assignee: Pradeep Kamath
> Fix For: 0.7.0
>
> Attachments: PIG-1265-2.patch, PIG-1265.patch
>
>
> Speaking to the hadoop team folks, the direction in hadoop is to use Job
> instead of Configuration - for example InputFormat/OutputFormat
> implementations use Job to store input/output location. So pig should also do
> the same in LoadMetadata and StoreMetadata to be closer to hadoop.
> Currently when a job fails, pig assumes the output locations (corresponding
> to the stores in the job) are hdfs locations and attempts to delete them.
> Since output locations could be non hdfs locations, this cleanup should be
> delegated to the StoreFuncInterface implementation - hence a new method -
> cleanupOnFailure() should be introduced in StoreFuncInterface and a default
> implementation should be provided in the StoreFunc abstract class which
> checks if the location exists on hdfs and deletes it if so.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
