[jira] Commented: (PIG-259) allow store to overwrite existing directroy
[ https://issues.apache.org/jira/browse/PIG-259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12863469#action_12863469 ] Olga Natkovich commented on PIG-259: Jeff, are you still planning to work on this issue for 0.8.0 release or should we unlink it? > allow store to overwrite existing directroy > --- > > Key: PIG-259 > URL: https://issues.apache.org/jira/browse/PIG-259 > Project: Pig > Issue Type: Sub-task >Affects Versions: 0.8.0 >Reporter: Olga Natkovich >Assignee: Jeff Zhang > Fix For: 0.8.0 > > Attachments: Pig_259.patch, Pig_259_2.patch, Pig_259_3.patch, > Pig_259_4.patch > > > we have users who are asking for a flag to overwrite existing directory -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-259) allow store to overwrite existing directroy
[ https://issues.apache.org/jira/browse/PIG-259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840703#action_12840703 ] Jeff Zhang commented on PIG-259: Actually I have different thoughts about the class hierarchy structure of LoadFunc and StoreFunc, I will attach some diagram to illustrate my idea int Pig-966 later. > allow store to overwrite existing directroy > --- > > Key: PIG-259 > URL: https://issues.apache.org/jira/browse/PIG-259 > Project: Pig > Issue Type: Sub-task >Affects Versions: 0.8.0 >Reporter: Olga Natkovich >Assignee: Jeff Zhang > Fix For: 0.8.0 > > Attachments: Pig_259.patch, Pig_259_2.patch, Pig_259_3.patch, > Pig_259_4.patch > > > we have users who are asking for a flag to overwrite existing directory -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-259) allow store to overwrite existing directroy
[ https://issues.apache.org/jira/browse/PIG-259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840178#action_12840178 ] Jeff Zhang commented on PIG-259: I also think it make sense to have the StoreFuncInterface contain the overwriting interface. For users it is easy to understand the interface and easy to maintenance. > allow store to overwrite existing directroy > --- > > Key: PIG-259 > URL: https://issues.apache.org/jira/browse/PIG-259 > Project: Pig > Issue Type: Sub-task >Affects Versions: 0.8.0 >Reporter: Olga Natkovich >Assignee: Jeff Zhang > Fix For: 0.8.0 > > Attachments: Pig_259.patch, Pig_259_2.patch, Pig_259_3.patch, > Pig_259_4.patch > > > we have users who are asking for a flag to overwrite existing directory -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-259) allow store to overwrite existing directroy
[ https://issues.apache.org/jira/browse/PIG-259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12839926#action_12839926 ] Olga Natkovich commented on PIG-259: I am wondering if this is an issue we need to resolve in Pig at all. Openning new interfaces is costly and something we would need to maintain for a long time. Are the use cases strong enough to pay the cost? > allow store to overwrite existing directroy > --- > > Key: PIG-259 > URL: https://issues.apache.org/jira/browse/PIG-259 > Project: Pig > Issue Type: Sub-task >Affects Versions: 0.8.0 >Reporter: Olga Natkovich >Assignee: Jeff Zhang > Fix For: 0.8.0 > > Attachments: Pig_259.patch, Pig_259_2.patch, Pig_259_3.patch, > Pig_259_4.patch > > > we have users who are asking for a flag to overwrite existing directory -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-259) allow store to overwrite existing directroy
[
https://issues.apache.org/jira/browse/PIG-259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12839887#action_12839887
]
Alan Gates commented on PIG-259:
Thanks for the explanation, now I understand what you're doing. I'd like to
make one more suggestion. Could we do this in a separate interface that store
functions could choose to implement? One of our goals with PIG-966 was to make
the easy case of store functions easy, and put less commonly used features in
separate interfaces. This way implementing a new store function is less of a
burden. So could we add an interface like:
{code}
interface OverwritingLoadFunc {
/**
* whether the implementation support overwrite
*
* @return
*/
public boolean isOverWrite();
/**
* cleanup the old output if you want to overwrite
*
* @param store, the store information you would like to delete
* @param context, used for deletion operation
* @throws IOException
*/
public void cleanupOutput(POStore store, PigContext context) throws
IOException;
}
{code}
Also, I'm guessing your cleanupOutput method will need a Job and a store
location (like the recently added cleanupOnFailure call, see PIG-1265).
> allow store to overwrite existing directroy
> ---
>
> Key: PIG-259
> URL: https://issues.apache.org/jira/browse/PIG-259
> Project: Pig
> Issue Type: Sub-task
>Affects Versions: 0.8.0
>Reporter: Olga Natkovich
>Assignee: Jeff Zhang
> Fix For: 0.8.0
>
> Attachments: Pig_259.patch, Pig_259_2.patch, Pig_259_3.patch,
> Pig_259_4.patch
>
>
> we have users who are asking for a flag to overwrite existing directory
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-259) allow store to overwrite existing directroy
[ https://issues.apache.org/jira/browse/PIG-259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12839737#action_12839737 ] Hadoop QA commented on PIG-259: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12437000/Pig_259_4.patch against trunk revision 916793. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/216/console This message is automatically generated. > allow store to overwrite existing directroy > --- > > Key: PIG-259 > URL: https://issues.apache.org/jira/browse/PIG-259 > Project: Pig > Issue Type: Sub-task >Affects Versions: 0.8.0 >Reporter: Olga Natkovich >Assignee: Jeff Zhang > Fix For: 0.8.0 > > Attachments: Pig_259.patch, Pig_259_2.patch, Pig_259_3.patch, > Pig_259_4.patch > > > we have users who are asking for a flag to overwrite existing directory -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-259) allow store to overwrite existing directroy
[ https://issues.apache.org/jira/browse/PIG-259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12838656#action_12838656 ] Jeff Zhang commented on PIG-259: Alan, In the patch I created a PigStorage constructor with a second argument indicate whether to overwrite. And the reason I change the StoreFuncInterface is that I'd like the users who want to create customized StoreFunc has the option to support overwrite. Besides I need the overwrite information in InputOutputFileVisitor and JobControlCompiler, But in these two classes I only can get StoreFuncInterface. One way to work around is to check the class whether it is PigStorage, and then cast it to PigStorage, but it seems to me not so friendly and extensible. > allow store to overwrite existing directroy > --- > > Key: PIG-259 > URL: https://issues.apache.org/jira/browse/PIG-259 > Project: Pig > Issue Type: Sub-task >Affects Versions: 0.8.0 >Reporter: Olga Natkovich >Assignee: Jeff Zhang > Fix For: 0.8.0 > > Attachments: Pig_259.patch, Pig_259_2.patch, Pig_259_3.patch, > Pig_259_4.patch > > > we have users who are asking for a flag to overwrite existing directory -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-259) allow store to overwrite existing directroy
[ https://issues.apache.org/jira/browse/PIG-259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12838321#action_12838321 ] Hadoop QA commented on PIG-259: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12436962/Pig_259_3.patch against trunk revision 916065. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/221/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/221/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/221/console This message is automatically generated. > allow store to overwrite existing directroy > --- > > Key: PIG-259 > URL: https://issues.apache.org/jira/browse/PIG-259 > Project: Pig > Issue Type: Sub-task >Affects Versions: 0.8.0 >Reporter: Olga Natkovich >Assignee: Jeff Zhang > Fix For: 0.8.0 > > Attachments: Pig_259.patch, Pig_259_2.patch, Pig_259_3.patch > > > we have users who are asking for a flag to overwrite existing directory -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-259) allow store to overwrite existing directroy
[ https://issues.apache.org/jira/browse/PIG-259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12831628#action_12831628 ] Olga Natkovich commented on PIG-259: +1 on passing the information in the constructor. Since we need the store function to to the validation, we don't have control over the semantics and it is better not to have constructs in the language whose semantics are not well defined. One thing we need to provide to the store function writer is guidence on when the information they get in the constructor can be acted on. > allow store to overwrite existing directroy > --- > > Key: PIG-259 > URL: https://issues.apache.org/jira/browse/PIG-259 > Project: Pig > Issue Type: Sub-task >Affects Versions: 0.8.0 >Reporter: Olga Natkovich >Assignee: Jeff Zhang > Fix For: 0.8.0 > > Attachments: Pig_259.patch, Pig_259_2.patch > > > we have users who are asking for a flag to overwrite existing directory -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-259) allow store to overwrite existing directroy
[ https://issues.apache.org/jira/browse/PIG-259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12831487#action_12831487 ] Jeff Zhang commented on PIG-259: Sorry, I mean I can buy in your suggestion. > allow store to overwrite existing directroy > --- > > Key: PIG-259 > URL: https://issues.apache.org/jira/browse/PIG-259 > Project: Pig > Issue Type: Sub-task >Affects Versions: 0.8.0 >Reporter: Olga Natkovich >Assignee: Jeff Zhang > Fix For: 0.8.0 > > Attachments: Pig_259.patch, Pig_259_2.patch > > > we have users who are asking for a flag to overwrite existing directory -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-259) allow store to overwrite existing directroy
[ https://issues.apache.org/jira/browse/PIG-259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12831484#action_12831484 ] Jeff Zhang commented on PIG-259: Response to Dmitriy, Thanks for your suggestion of implementing overwrite on the StoreFunc level rather than on language level. I can bug in this. AndI think another advantage of putting it in StoreFunc is that it's more flexible than putting it in language. We have more control on StoreFunc than pig latin. > allow store to overwrite existing directroy > --- > > Key: PIG-259 > URL: https://issues.apache.org/jira/browse/PIG-259 > Project: Pig > Issue Type: Sub-task >Affects Versions: 0.8.0 >Reporter: Olga Natkovich >Assignee: Jeff Zhang > Fix For: 0.8.0 > > Attachments: Pig_259.patch, Pig_259_2.patch > > > we have users who are asking for a flag to overwrite existing directory -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-259) allow store to overwrite existing directroy
[ https://issues.apache.org/jira/browse/PIG-259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12831481#action_12831481 ] Jeff Zhang commented on PIG-259: Response to Alan, I agree that it makes more sense to do the overwrite in StoreFunc, and I notice that there's a JIAR PIG-1216 which is related with this. > allow store to overwrite existing directroy > --- > > Key: PIG-259 > URL: https://issues.apache.org/jira/browse/PIG-259 > Project: Pig > Issue Type: Sub-task >Affects Versions: 0.8.0 >Reporter: Olga Natkovich >Assignee: Jeff Zhang > Fix For: 0.8.0 > > Attachments: Pig_259.patch, Pig_259_2.patch > > > we have users who are asking for a flag to overwrite existing directory -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-259) allow store to overwrite existing directroy
[ https://issues.apache.org/jira/browse/PIG-259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12831237#action_12831237 ] Dmitriy V. Ryaboy commented on PIG-259: --- Yeah I think it makes more sense on that level. I think it's worth our while to add it to PigStorage and any other built-in engines though. Otherwise it's not clear what the overwrite keyword means. If it's HBase, does it mean drop the table and overwrite? Or delete old values when there is an old confilct? Does that mean we don't delete them otherwise? What about appends? This request made more sense when Pig maintained strong control over file creation and such, but now that it's pushed into IOFormats, I think it's ok to push this functionality there as well. It's just a lot of stuff I think we don't want to be in the business of creating interfaces for. Different storage engines have different options for what they can and can't do. > allow store to overwrite existing directroy > --- > > Key: PIG-259 > URL: https://issues.apache.org/jira/browse/PIG-259 > Project: Pig > Issue Type: Sub-task >Affects Versions: 0.8.0 >Reporter: Olga Natkovich >Assignee: Jeff Zhang > Fix For: 0.8.0 > > Attachments: Pig_259.patch, Pig_259_2.patch > > > we have users who are asking for a flag to overwrite existing directory -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-259) allow store to overwrite existing directroy
[ https://issues.apache.org/jira/browse/PIG-259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12831222#action_12831222 ] Alan Gates commented on PIG-259: If we make overwrite part of the language (as the JIRA proposes and Jeff's patch implements) then we need to make it part of the interface, at least to inform the store func that overwrite was set. Are you suggesting that individual store funcs that want to support overwrite instead support it as an argument and not make it a part of the language? > allow store to overwrite existing directroy > --- > > Key: PIG-259 > URL: https://issues.apache.org/jira/browse/PIG-259 > Project: Pig > Issue Type: Sub-task >Affects Versions: 0.8.0 >Reporter: Olga Natkovich >Assignee: Jeff Zhang > Fix For: 0.8.0 > > Attachments: Pig_259.patch, Pig_259_2.patch > > > we have users who are asking for a flag to overwrite existing directory -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-259) allow store to overwrite existing directroy
[
https://issues.apache.org/jira/browse/PIG-259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12831212#action_12831212
]
Dmitriy V. Ryaboy commented on PIG-259:
---
Doesn't the StoreFunc take care of resource creation/validation in LSR?
If so, no new method is needed (since as you say, what exactly it means is
indeterminate). StoreFuncs that want to can use initialization variables to
determine their behavior (eg: store into '/my/path' using PigStorage('\t',
'-overwrite:true, -writeHeader:true, -writeSchema:false');
> allow store to overwrite existing directroy
> ---
>
> Key: PIG-259
> URL: https://issues.apache.org/jira/browse/PIG-259
> Project: Pig
> Issue Type: Sub-task
>Affects Versions: 0.8.0
>Reporter: Olga Natkovich
>Assignee: Jeff Zhang
> Fix For: 0.8.0
>
> Attachments: Pig_259.patch, Pig_259_2.patch
>
>
> we have users who are asking for a flag to overwrite existing directory
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-259) allow store to overwrite existing directroy
[ https://issues.apache.org/jira/browse/PIG-259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12831205#action_12831205 ] Alan Gates commented on PIG-259: Sorry, I missed that it was already for load-store redesign. In the new load/store scheme we can't assume that outputs are files. So putting a DFS remove statement into JobControlCompiler won't be the right way to do this. What does overwrite mean when we're storing to HBase instead of DFS? It seems that this needs to be defined on the per store function basis, which implies it needs to somehow be part of the store function interface. This could be done by adding a boolean to validate or by adding a new method to the StoreFunc interface to overwrite. > allow store to overwrite existing directroy > --- > > Key: PIG-259 > URL: https://issues.apache.org/jira/browse/PIG-259 > Project: Pig > Issue Type: Sub-task >Affects Versions: 0.8.0 >Reporter: Olga Natkovich >Assignee: Jeff Zhang > Fix For: 0.8.0 > > Attachments: Pig_259.patch, Pig_259_2.patch > > > we have users who are asking for a flag to overwrite existing directory -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-259) allow store to overwrite existing directroy
[ https://issues.apache.org/jira/browse/PIG-259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12830170#action_12830170 ] Jeff Zhang commented on PIG-259: Thanks, Alan. I have moved it under Pig-966. And the new patch is for load-store-redesign branch not for trunk. > allow store to overwrite existing directroy > --- > > Key: PIG-259 > URL: https://issues.apache.org/jira/browse/PIG-259 > Project: Pig > Issue Type: Sub-task >Affects Versions: 0.8.0 >Reporter: Olga Natkovich >Assignee: Jeff Zhang > Fix For: 0.8.0 > > Attachments: Pig_259.patch, Pig_259_2.patch > > > we have users who are asking for a flag to overwrite existing directory -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-259) allow store to overwrite existing directroy
[ https://issues.apache.org/jira/browse/PIG-259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12830161#action_12830161 ] Alan Gates commented on PIG-259: Changes look good. This patch is still against trunk. Was your plan to apply this to trunk and then provide a different patch for load/store redesign or just to take this patch and change it to work against load/store? I think the latter is better, as it avoids the need to undo this patch when we merge in load/store. If you want to be able to apply the patch now to trunk (without checking it in) we can still leave your existing patch on this JIRA and only commit a patch against load/store redesign. As for attaching this to load/store, you can click on the edit link on the left side and this will take you to a page where you can make it a sub-task of PIG-966. (You may not have had this edit link before, as I had forgotten to add you to the list of committers on JIRA. I fixed that, so you should see the link now.) > allow store to overwrite existing directroy > --- > > Key: PIG-259 > URL: https://issues.apache.org/jira/browse/PIG-259 > Project: Pig > Issue Type: Bug >Affects Versions: 0.8.0 >Reporter: Olga Natkovich >Assignee: Jeff Zhang > Fix For: 0.8.0 > > Attachments: Pig_259.patch, Pig_259_2.patch > > > we have users who are asking for a flag to overwrite existing directory -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-259) allow store to overwrite existing directroy
[ https://issues.apache.org/jira/browse/PIG-259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12829982#action_12829982 ] Jeff Zhang commented on PIG-259: Alan, Should I create a new sub task under Pig-966 ? or is there any way to move this task under Pig-966 ? > allow store to overwrite existing directroy > --- > > Key: PIG-259 > URL: https://issues.apache.org/jira/browse/PIG-259 > Project: Pig > Issue Type: Bug >Affects Versions: 0.8.0 >Reporter: Olga Natkovich >Assignee: Jeff Zhang > Fix For: 0.8.0 > > Attachments: Pig_259.patch, Pig_259_2.patch > > > we have users who are asking for a flag to overwrite existing directory -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-259) allow store to overwrite existing directroy
[
https://issues.apache.org/jira/browse/PIG-259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12829981#action_12829981
]
Jeff Zhang commented on PIG-259:
Response to Alan regarding his comments,
1. I put the logic of deleting output file in JobControlCompiler, then it is
easy for me to delay the deletion util the dependent job is done.
2. I prefer using keywords rather than string, because if using string, the
following statement: {code} store a into 'output' 'overwrite'; {code} has two
consecutive string, it looks a little weird in my opinion.
3. I think the semantic of overwrite is the same as it is in file system. In
file system, when we overwrite file using java api, it won't complain even the
file does not exist
> allow store to overwrite existing directroy
> ---
>
> Key: PIG-259
> URL: https://issues.apache.org/jira/browse/PIG-259
> Project: Pig
> Issue Type: Bug
>Affects Versions: 0.8.0
>Reporter: Olga Natkovich
>Assignee: Jeff Zhang
> Fix For: 0.8.0
>
> Attachments: Pig_259.patch, Pig_259_2.patch
>
>
> we have users who are asking for a flag to overwrite existing directory
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-259) allow store to overwrite existing directroy
[ https://issues.apache.org/jira/browse/PIG-259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12829728#action_12829728 ] Hadoop QA commented on PIG-259: --- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12434801/Pig_259.patch against trunk revision 906326. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/192/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/192/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/192/console This message is automatically generated. > allow store to overwrite existing directroy > --- > > Key: PIG-259 > URL: https://issues.apache.org/jira/browse/PIG-259 > Project: Pig > Issue Type: Bug >Affects Versions: 0.8.0 >Reporter: Olga Natkovich >Assignee: Jeff Zhang > Fix For: 0.8.0 > > Attachments: Pig_259.patch > > > we have users who are asking for a flag to overwrite existing directory -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-259) allow store to overwrite existing directroy
[ https://issues.apache.org/jira/browse/PIG-259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12829697#action_12829697 ] Alan Gates commented on PIG-259: A few comments and questions on this: 1) We should make this work against the load/store branch instead of trunk. We're hoping to merge load/store into trunk in a week or two, so it makes more sense to put it there. This will also have implications for load/store. One, it will need to communicate to the new validate function that it's ok if the file (or whatever is being overwritten) exists. Two, load implementations will need to handle removing the file (or whatever) if necessary. For example, PigStorage will need to handle removing the file so MR doesn't complain. 2) Should we have overwrite be a keyword (as originally proposed and in the patch) or should it be string, like hints in join? I don't have a strong opinion one way or another but I think it's worth considering which we want. 3) Is the semantic of overwrite that it saves whether the file is there or not, or that it's an error if the file is not there to write? Write whether there or not makes more sense to me, but I wanted to make sure we all agree on it. 4) What happens when a user requests overwrite and the job fails before it runs? In the current implementation the file will be removed up front, so any planning errors will still result in the file being removed. Also, the file will be removed up front, even if the job remains in Hadoop's queue for a long time waiting to run. At the very least, I think Pig should delay removing the file until it is ready to launch the job so that type checking errors or whatever don't result in the file being removed when the job is not run. > allow store to overwrite existing directroy > --- > > Key: PIG-259 > URL: https://issues.apache.org/jira/browse/PIG-259 > Project: Pig > Issue Type: Bug >Affects Versions: 0.8.0 >Reporter: Olga Natkovich >Assignee: Jeff Zhang > Fix For: 0.8.0 > > Attachments: Pig_259.patch > > > we have users who are asking for a flag to overwrite existing directory -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
