[jira] [Commented] (PIG-3199) Expose LogicalPlan via PigServer API
[ https://issues.apache.org/jira/browse/PIG-3199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13773529#comment-13773529 ] Daniel Dai commented on PIG-3199: - Looks fine for me. We are not exposing LogicalPlan, but only loader/storer name in the new patch. I will commit the patch with Cheolsoo's suggested change shortly. > Expose LogicalPlan via PigServer API > > > Key: PIG-3199 > URL: https://issues.apache.org/jira/browse/PIG-3199 > Project: Pig > Issue Type: Improvement > Components: impl >Affects Versions: 0.10.0 >Reporter: Prashant Kommireddi >Assignee: Prashant Kommireddi > Fix For: 0.12 > > Attachments: PIG-3199_2.patch, PIG-3199.patch > > > LogicalPlan could be exposed to user in order for one to make validations > based on it. For eg, one could get Load/Store paths or other operators and be > able to perform checks such as whether I/O paths are valid etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-3199) Expose LogicalPlan via PigServer API
[
https://issues.apache.org/jira/browse/PIG-3199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13637373#comment-13637373
]
Cheolsoo Park commented on PIG-3199:
The patch seems fine to me except the following line:
{code}
+assertTrue("Source must end with input.txt",
lData.getSinks().get(0).endsWith("output"));
{code}
It should say "Sink must end with output" instead.
Nevertheless, I will wait for more feedback from other committers given this is
quite controversial.
> Expose LogicalPlan via PigServer API
>
>
> Key: PIG-3199
> URL: https://issues.apache.org/jira/browse/PIG-3199
> Project: Pig
> Issue Type: Improvement
> Components: impl
>Affects Versions: 0.10.0
>Reporter: Prashant Kommireddi
>Assignee: Prashant Kommireddi
> Fix For: 0.12
>
> Attachments: PIG-3199_2.patch, PIG-3199.patch
>
>
> LogicalPlan could be exposed to user in order for one to make validations
> based on it. For eg, one could get Load/Store paths or other operators and be
> able to perform checks such as whether I/O paths are valid etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-3199) Expose LogicalPlan via PigServer API
[ https://issues.apache.org/jira/browse/PIG-3199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13637351#comment-13637351 ] Prashant Kommireddi commented on PIG-3199: -- Ping! > Expose LogicalPlan via PigServer API > > > Key: PIG-3199 > URL: https://issues.apache.org/jira/browse/PIG-3199 > Project: Pig > Issue Type: Improvement > Components: impl >Affects Versions: 0.10.0 >Reporter: Prashant Kommireddi >Assignee: Prashant Kommireddi > Fix For: 0.12 > > Attachments: PIG-3199_2.patch, PIG-3199.patch > > > LogicalPlan could be exposed to user in order for one to make validations > based on it. For eg, one could get Load/Store paths or other operators and be > able to perform checks such as whether I/O paths are valid etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-3199) Expose LogicalPlan via PigServer API
[ https://issues.apache.org/jira/browse/PIG-3199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13591527#comment-13591527 ] Prashant Kommireddi commented on PIG-3199: -- [~dvryaboy] please let me know if you think otherwise? > Expose LogicalPlan via PigServer API > > > Key: PIG-3199 > URL: https://issues.apache.org/jira/browse/PIG-3199 > Project: Pig > Issue Type: Improvement > Components: impl >Affects Versions: 0.10.0 >Reporter: Prashant Kommireddi >Assignee: Prashant Kommireddi > Fix For: 0.12 > > Attachments: PIG-3199.patch > > > LogicalPlan could be exposed to user in order for one to make validations > based on it. For eg, one could get Load/Store paths or other operators and be > able to perform checks such as whether I/O paths are valid etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-3199) Expose LogicalPlan via PigServer API
[ https://issues.apache.org/jira/browse/PIG-3199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13587510#comment-13587510 ] Prashant Kommireddi commented on PIG-3199: -- Mainly because we don't need the extra steps of running the optimizer, generating Physical plan, generating MR plan to get to this information. It just feels querying LP for source/sink or load/store funcs is more efficient. Would be happy to get your thoughts? > Expose LogicalPlan via PigServer API > > > Key: PIG-3199 > URL: https://issues.apache.org/jira/browse/PIG-3199 > Project: Pig > Issue Type: Improvement > Components: impl >Affects Versions: 0.10.0 >Reporter: Prashant Kommireddi >Assignee: Prashant Kommireddi > Fix For: 0.12 > > Attachments: PIG-3199.patch > > > LogicalPlan could be exposed to user in order for one to make validations > based on it. For eg, one could get Load/Store paths or other operators and be > able to perform checks such as whether I/O paths are valid etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-3199) Expose LogicalPlan via PigServer API
[ https://issues.apache.org/jira/browse/PIG-3199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13586418#comment-13586418 ] Dmitriy V. Ryaboy commented on PIG-3199: Seriously though, why not PPNL? > Expose LogicalPlan via PigServer API > > > Key: PIG-3199 > URL: https://issues.apache.org/jira/browse/PIG-3199 > Project: Pig > Issue Type: Improvement > Components: impl >Affects Versions: 0.10.0 >Reporter: Prashant Kommireddi >Assignee: Prashant Kommireddi > Fix For: 0.12 > > Attachments: PIG-3199.patch > > > LogicalPlan could be exposed to user in order for one to make validations > based on it. For eg, one could get Load/Store paths or other operators and be > able to perform checks such as whether I/O paths are valid etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-3199) Expose LogicalPlan via PigServer API
[ https://issues.apache.org/jira/browse/PIG-3199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13584853#comment-13584853 ] Alan Gates commented on PIG-3199: - Keep this one, that way the history of the discussion is all together. > Expose LogicalPlan via PigServer API > > > Key: PIG-3199 > URL: https://issues.apache.org/jira/browse/PIG-3199 > Project: Pig > Issue Type: Improvement > Components: impl >Affects Versions: 0.10.0 >Reporter: Prashant Kommireddi >Assignee: Prashant Kommireddi > Fix For: 0.12 > > Attachments: PIG-3199.patch > > > LogicalPlan could be exposed to user in order for one to make validations > based on it. For eg, one could get Load/Store paths or other operators and be > able to perform checks such as whether I/O paths are valid etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-3199) Expose LogicalPlan via PigServer API
[ https://issues.apache.org/jira/browse/PIG-3199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13584851#comment-13584851 ] Prashant Kommireddi commented on PIG-3199: -- That makes sense, I will work on implementing the above proposal. Thanks Alan. Should we change this JIRA title to reflect that or open a new one? > Expose LogicalPlan via PigServer API > > > Key: PIG-3199 > URL: https://issues.apache.org/jira/browse/PIG-3199 > Project: Pig > Issue Type: Improvement > Components: impl >Affects Versions: 0.10.0 >Reporter: Prashant Kommireddi >Assignee: Prashant Kommireddi > Fix For: 0.12 > > Attachments: PIG-3199.patch > > > LogicalPlan could be exposed to user in order for one to make validations > based on it. For eg, one could get Load/Store paths or other operators and be > able to perform checks such as whether I/O paths are valid etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-3199) Expose LogicalPlan via PigServer API
[ https://issues.apache.org/jira/browse/PIG-3199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13584825#comment-13584825 ] Alan Gates commented on PIG-3199: - If the PigNotificationListener doesn't work for you I think getLoadPaths()/getStorePaths() is fine. I was thinking of proposing that when I remembered the initialPlanNotification stuff. You might want to return a class so it can contain the names of the load/store funcs as well and so you can include more info later. > Expose LogicalPlan via PigServer API > > > Key: PIG-3199 > URL: https://issues.apache.org/jira/browse/PIG-3199 > Project: Pig > Issue Type: Improvement > Components: impl >Affects Versions: 0.10.0 >Reporter: Prashant Kommireddi >Assignee: Prashant Kommireddi > Fix For: 0.12 > > Attachments: PIG-3199.patch > > > LogicalPlan could be exposed to user in order for one to make validations > based on it. For eg, one could get Load/Store paths or other operators and be > able to perform checks such as whether I/O paths are valid etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-3199) Expose LogicalPlan via PigServer API
[
https://issues.apache.org/jira/browse/PIG-3199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13584821#comment-13584821
]
Prashant Kommireddi commented on PIG-3199:
--
What are your thoughts on exposing the Load/Store paths as strings instead of
exposing any of the internal interfaces? That way we have a very neat way of
exposing these to users
{code}
public List getLoadPaths();
public List getStorePaths();
{code}
> Expose LogicalPlan via PigServer API
>
>
> Key: PIG-3199
> URL: https://issues.apache.org/jira/browse/PIG-3199
> Project: Pig
> Issue Type: Improvement
> Components: impl
>Affects Versions: 0.10.0
>Reporter: Prashant Kommireddi
>Assignee: Prashant Kommireddi
> Fix For: 0.12
>
> Attachments: PIG-3199.patch
>
>
> LogicalPlan could be exposed to user in order for one to make validations
> based on it. For eg, one could get Load/Store paths or other operators and be
> able to perform checks such as whether I/O paths are valid etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-3199) Expose LogicalPlan via PigServer API
[ https://issues.apache.org/jira/browse/PIG-3199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13584808#comment-13584808 ] Alan Gates commented on PIG-3199: - Take a look at PigNotificationListener.initialPlanNotification. It seems like this will give you what you want, since each of the sources and sinks for the MR jobs are in here. To my chagrin this exposes the MR plan. Someone snuck that past me. > Expose LogicalPlan via PigServer API > > > Key: PIG-3199 > URL: https://issues.apache.org/jira/browse/PIG-3199 > Project: Pig > Issue Type: Improvement > Components: impl >Affects Versions: 0.10.0 >Reporter: Prashant Kommireddi >Assignee: Prashant Kommireddi > Fix For: 0.12 > > Attachments: PIG-3199.patch > > > LogicalPlan could be exposed to user in order for one to make validations > based on it. For eg, one could get Load/Store paths or other operators and be > able to perform checks such as whether I/O paths are valid etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-3199) Expose LogicalPlan via PigServer API
[
https://issues.apache.org/jira/browse/PIG-3199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13584795#comment-13584795
]
Prashant Kommireddi commented on PIG-3199:
--
Ok, I can understand the concerns and agree with you on not exposing LP.
What I stated earlier is exactly the info I need - using PigServer register
script/query and be able to determine the source/sinks. For eg, in the
following snippet I read a script "script.pig". I don't control the contents of
the script (written by other users) and as an admin would like to make sure I
can read the I/O paths being used here.
{code}
pigServer = new PigServer(ExecType.MAPREDUCE, conf);
InputStream in = fs.open("/foo/script.pig");
pigServer.registerScript(in);
{code}
script.pig
{code}
A = load '/apache/pig/*';
B = store A into '/google';
{code}
The idea with this patch was to be able to determine source (/apache/pig) and
sink (/google) and perform certain operations on it. Would be great if you
think it can be done in a way better than exposing any of LP/Operator.
> Expose LogicalPlan via PigServer API
>
>
> Key: PIG-3199
> URL: https://issues.apache.org/jira/browse/PIG-3199
> Project: Pig
> Issue Type: Improvement
> Components: impl
>Affects Versions: 0.10.0
>Reporter: Prashant Kommireddi
>Assignee: Prashant Kommireddi
> Fix For: 0.12
>
> Attachments: PIG-3199.patch
>
>
> LogicalPlan could be exposed to user in order for one to make validations
> based on it. For eg, one could get Load/Store paths or other operators and be
> able to perform checks such as whether I/O paths are valid etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-3199) Expose LogicalPlan via PigServer API
[ https://issues.apache.org/jira/browse/PIG-3199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13584784#comment-13584784 ] Alan Gates commented on PIG-3199: - Even making Operator public is dangerous. These are internal structures. It would help to understand who you want to expose these to and why. Then we can see if there's a way to get you the information you need. I don't want to stand in the way of innovation but I also don't want Pig's internals exposed to the point that the next time we make a change to our Operator class someone complains because we broke his tool. > Expose LogicalPlan via PigServer API > > > Key: PIG-3199 > URL: https://issues.apache.org/jira/browse/PIG-3199 > Project: Pig > Issue Type: Improvement > Components: impl >Affects Versions: 0.10.0 >Reporter: Prashant Kommireddi >Assignee: Prashant Kommireddi > Fix For: 0.12 > > Attachments: PIG-3199.patch > > > LogicalPlan could be exposed to user in order for one to make validations > based on it. For eg, one could get Load/Store paths or other operators and be > able to perform checks such as whether I/O paths are valid etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-3199) Expose LogicalPlan via PigServer API
[
https://issues.apache.org/jira/browse/PIG-3199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13584776#comment-13584776
]
Prashant Kommireddi commented on PIG-3199:
--
It's not exposed at the moment but a part of this patch. I was not aware of
plans to make changes to LogicalPlan at the interface level. Just noticed this
for OperatorPlan and it makes sense.
{code}
@InterfaceAudience.Private
@InterfaceStability.Unstable
{code}
If that's the case, can we expose only a couple methods to get a handle on
source/sinks?
{code}
public List getSources();
public List getSinks();
{code}
> Expose LogicalPlan via PigServer API
>
>
> Key: PIG-3199
> URL: https://issues.apache.org/jira/browse/PIG-3199
> Project: Pig
> Issue Type: Improvement
> Components: impl
>Affects Versions: 0.10.0
>Reporter: Prashant Kommireddi
>Assignee: Prashant Kommireddi
> Fix For: 0.12
>
> Attachments: PIG-3199.patch
>
>
> LogicalPlan could be exposed to user in order for one to make validations
> based on it. For eg, one could get Load/Store paths or other operators and be
> able to perform checks such as whether I/O paths are valid etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-3199) Expose LogicalPlan via PigServer API
[ https://issues.apache.org/jira/browse/PIG-3199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13584772#comment-13584772 ] Jonathan Coveney commented on PIG-3199: --- Alan, This patch almost exclusively exists to make the LogicalPlan publicly available. Prashant, Want to discuss further how you want to handle this? I agree that we don't want to put ourselves into a difficult position when it comes to changing things in the future, but maybe there's a compromise. > Expose LogicalPlan via PigServer API > > > Key: PIG-3199 > URL: https://issues.apache.org/jira/browse/PIG-3199 > Project: Pig > Issue Type: Improvement > Components: impl >Affects Versions: 0.10.0 >Reporter: Prashant Kommireddi >Assignee: Prashant Kommireddi > Fix For: 0.12 > > Attachments: PIG-3199.patch > > > LogicalPlan could be exposed to user in order for one to make validations > based on it. For eg, one could get Load/Store paths or other operators and be > able to perform checks such as whether I/O paths are valid etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-3199) Expose LogicalPlan via PigServer API
[ https://issues.apache.org/jira/browse/PIG-3199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13584751#comment-13584751 ] Alan Gates commented on PIG-3199: - When you say "now that [the logical plan] is public" do you mean that's already true or it would be true with this patch? If it's already true, where are we exposing it? If it's not true yet, I'm -1 at this point in exposing it. Making that a public interface will severely restrict our ability to make changes at that layer, which we'd like to be to do. > Expose LogicalPlan via PigServer API > > > Key: PIG-3199 > URL: https://issues.apache.org/jira/browse/PIG-3199 > Project: Pig > Issue Type: Improvement > Components: impl >Affects Versions: 0.10.0 >Reporter: Prashant Kommireddi >Assignee: Prashant Kommireddi > Fix For: 0.12 > > Attachments: PIG-3199.patch > > > LogicalPlan could be exposed to user in order for one to make validations > based on it. For eg, one could get Load/Store paths or other operators and be > able to perform checks such as whether I/O paths are valid etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-3199) Expose LogicalPlan via PigServer API
[ https://issues.apache.org/jira/browse/PIG-3199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13584304#comment-13584304 ] Jonathan Coveney commented on PIG-3199: --- Given that a logical plan is only available under certain conditions, now that it is public it's probably worth testing for those conditions and throwing an error if they are not met. Aside from that this is a pretty small but potentially useful change. > Expose LogicalPlan via PigServer API > > > Key: PIG-3199 > URL: https://issues.apache.org/jira/browse/PIG-3199 > Project: Pig > Issue Type: Improvement > Components: impl >Affects Versions: 0.10.0 >Reporter: Prashant Kommireddi >Assignee: Prashant Kommireddi > Fix For: 0.12 > > Attachments: PIG-3199.patch > > > LogicalPlan could be exposed to user in order for one to make validations > based on it. For eg, one could get Load/Store paths or other operators and be > able to perform checks such as whether I/O paths are valid etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
