[jira] [Commented] (SPARK-6320) Adding new query plan strategy to SQLContext
[
https://issues.apache.org/jira/browse/SPARK-6320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15308824#comment-15308824
]
Apache Spark commented on SPARK-6320:
-
User 'ueshin' has created a pull request for this issue:
https://github.com/apache/spark/pull/13426
> Adding new query plan strategy to SQLContext
>
>
> Key: SPARK-6320
> URL: https://issues.apache.org/jira/browse/SPARK-6320
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.3.0
>Reporter: Youssef Hatem
>Priority: Minor
>
> Hi,
> I would like to add a new strategy to {{SQLContext}}. To do this I created a
> new class which extends {{Strategy}}. In my new class I need to call
> {{planLater}} function. However this method is defined in {{SparkPlanner}}
> (which itself inherits the method from {{QueryPlanner}}).
> To my knowledge the only way to make {{planLater}} function visible to my new
> strategy is to define my strategy inside another class that extends
> {{SparkPlanner}} and inherits {{planLater}} as a result, by doing so I will
> have to extend the {{SQLContext}} such that I can override the {{planner}}
> field with the new {{Planner}} class I created.
> It seems that this is a design problem because adding a new strategy seems to
> require extending {{SQLContext}} (unless I am doing it wrong and there is a
> better way to do it).
> Thanks a lot,
> Youssef
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6320) Adding new query plan strategy to SQLContext
[
https://issues.apache.org/jira/browse/SPARK-6320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15286339#comment-15286339
]
Apache Spark commented on SPARK-6320:
-
User 'ueshin' has created a pull request for this issue:
https://github.com/apache/spark/pull/13147
> Adding new query plan strategy to SQLContext
>
>
> Key: SPARK-6320
> URL: https://issues.apache.org/jira/browse/SPARK-6320
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.3.0
>Reporter: Youssef Hatem
>Priority: Minor
>
> Hi,
> I would like to add a new strategy to {{SQLContext}}. To do this I created a
> new class which extends {{Strategy}}. In my new class I need to call
> {{planLater}} function. However this method is defined in {{SparkPlanner}}
> (which itself inherits the method from {{QueryPlanner}}).
> To my knowledge the only way to make {{planLater}} function visible to my new
> strategy is to define my strategy inside another class that extends
> {{SparkPlanner}} and inherits {{planLater}} as a result, by doing so I will
> have to extend the {{SQLContext}} such that I can override the {{planner}}
> field with the new {{Planner}} class I created.
> It seems that this is a design problem because adding a new strategy seems to
> require extending {{SQLContext}} (unless I am doing it wrong and there is a
> better way to do it).
> Thanks a lot,
> Youssef
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6320) Adding new query plan strategy to SQLContext
[
https://issues.apache.org/jira/browse/SPARK-6320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14499542#comment-14499542
]
Edoardo Vacchi commented on SPARK-6320:
---
My use case is customizing strategies so that operations can be
pruned/rewritten, depending on metadata in a custom catalog. For instance,
consider the case of a DataFrame where a column X is known to be non-nullable;
if the query contains the filter {{where('x is not null)}} the filter can be
pruned with a rule of the form:
{code:java}
case logical.Filter(IsNotNull(e: NamedExpression), child)
if customCatalog.isNonNull(e) => {
planLater(child) :: Nil
{code}
I wonder why is {{SparkPlanner}} nested inside {{SQLContext}}. Cass
{{SparkPlanner}} may require the {{SQLContext}}, {{SQLConf}} as constructor
parameters and be defined in its own class (and the same could be said for
QueryExecution). suggestion would be to refactor out {{SparkPlanner}}.
Advanced users would then be able to extend it like any other class.
> Adding new query plan strategy to SQLContext
>
>
> Key: SPARK-6320
> URL: https://issues.apache.org/jira/browse/SPARK-6320
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.3.0
>Reporter: Youssef Hatem
>Priority: Minor
>
> Hi,
> I would like to add a new strategy to {{SQLContext}}. To do this I created a
> new class which extends {{Strategy}}. In my new class I need to call
> {{planLater}} function. However this method is defined in {{SparkPlanner}}
> (which itself inherits the method from {{QueryPlanner}}).
> To my knowledge the only way to make {{planLater}} function visible to my new
> strategy is to define my strategy inside another class that extends
> {{SparkPlanner}} and inherits {{planLater}} as a result, by doing so I will
> have to extend the {{SQLContext}} such that I can override the {{planner}}
> field with the new {{Planner}} class I created.
> It seems that this is a design problem because adding a new strategy seems to
> require extending {{SQLContext}} (unless I am doing it wrong and there is a
> better way to do it).
> Thanks a lot,
> Youssef
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6320) Adding new query plan strategy to SQLContext
[
https://issues.apache.org/jira/browse/SPARK-6320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14376434#comment-14376434
]
Michael Armbrust commented on SPARK-6320:
-
If that can be done in a minimally invasive way that sounds reasonable to me.
> Adding new query plan strategy to SQLContext
>
>
> Key: SPARK-6320
> URL: https://issues.apache.org/jira/browse/SPARK-6320
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.3.0
>Reporter: Youssef Hatem
>Priority: Minor
>
> Hi,
> I would like to add a new strategy to {{SQLContext}}. To do this I created a
> new class which extends {{Strategy}}. In my new class I need to call
> {{planLater}} function. However this method is defined in {{SparkPlanner}}
> (which itself inherits the method from {{QueryPlanner}}).
> To my knowledge the only way to make {{planLater}} function visible to my new
> strategy is to define my strategy inside another class that extends
> {{SparkPlanner}} and inherits {{planLater}} as a result, by doing so I will
> have to extend the {{SQLContext}} such that I can override the {{planner}}
> field with the new {{Planner}} class I created.
> It seems that this is a design problem because adding a new strategy seems to
> require extending {{SQLContext}} (unless I am doing it wrong and there is a
> better way to do it).
> Thanks a lot,
> Youssef
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6320) Adding new query plan strategy to SQLContext
[
https://issues.apache.org/jira/browse/SPARK-6320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14368683#comment-14368683
]
Santiago M. Mola commented on SPARK-6320:
-
[~marmbrus] We could change strategies so that they take a SparkPlanner in
their constructor. This should provide enough flexibility for [~H.Youssef]]'s
use case and might improve code organization of the core strategies in the
future.
> Adding new query plan strategy to SQLContext
>
>
> Key: SPARK-6320
> URL: https://issues.apache.org/jira/browse/SPARK-6320
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.3.0
>Reporter: Youssef Hatem
>Priority: Minor
>
> Hi,
> I would like to add a new strategy to {{SQLContext}}. To do this I created a
> new class which extends {{Strategy}}. In my new class I need to call
> {{planLater}} function. However this method is defined in {{SparkPlanner}}
> (which itself inherits the method from {{QueryPlanner}}).
> To my knowledge the only way to make {{planLater}} function visible to my new
> strategy is to define my strategy inside another class that extends
> {{SparkPlanner}} and inherits {{planLater}} as a result, by doing so I will
> have to extend the {{SQLContext}} such that I can override the {{planner}}
> field with the new {{Planner}} class I created.
> It seems that this is a design problem because adding a new strategy seems to
> require extending {{SQLContext}} (unless I am doing it wrong and there is a
> better way to do it).
> Thanks a lot,
> Youssef
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6320) Adding new query plan strategy to SQLContext
[
https://issues.apache.org/jira/browse/SPARK-6320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14365274#comment-14365274
]
Youssef Hatem commented on SPARK-6320:
--
Thank you Michael for your response,
Actually I am trying to extend SQL syntax with custom constructs which would
require altering the workflow of {{SparkSQL}} all the way starting from the
lexer and ending with the optimization and physical planning.
I thought this should be possible without having to extend many classes.
However not being able to use {{planLater}} forces me to do seemingly
unnecessary extension of {{SparkPlanner}} specially that there seem to be some
logic to handle these scenarios (i.e. the {{extraStrategies}} sequence).
> Adding new query plan strategy to SQLContext
>
>
> Key: SPARK-6320
> URL: https://issues.apache.org/jira/browse/SPARK-6320
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.3.0
>Reporter: Youssef Hatem
>Priority: Minor
>
> Hi,
> I would like to add a new strategy to {{SQLContext}}. To do this I created a
> new class which extends {{Strategy}}. In my new class I need to call
> {{planLater}} function. However this method is defined in {{SparkPlanner}}
> (which itself inherits the method from {{QueryPlanner}}).
> To my knowledge the only way to make {{planLater}} function visible to my new
> strategy is to define my strategy inside another class that extends
> {{SparkPlanner}} and inherits {{planLater}} as a result, by doing so I will
> have to extend the {{SQLContext}} such that I can override the {{planner}}
> field with the new {{Planner}} class I created.
> It seems that this is a design problem because adding a new strategy seems to
> require extending {{SQLContext}} (unless I am doing it wrong and there is a
> better way to do it).
> Thanks a lot,
> Youssef
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6320) Adding new query plan strategy to SQLContext
[
https://issues.apache.org/jira/browse/SPARK-6320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14364345#comment-14364345
]
Michael Armbrust commented on SPARK-6320:
-
Hmm, interesting. So far I had only considered this interface for planning
leaves of the query plan. Can you tell me more about what you are trying to
optimize?
> Adding new query plan strategy to SQLContext
>
>
> Key: SPARK-6320
> URL: https://issues.apache.org/jira/browse/SPARK-6320
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.3.0
>Reporter: Youssef Hatem
>Priority: Minor
>
> Hi,
> I would like to add a new strategy to {{SQLContext}}. To do this I created a
> new class which extends {{Strategy}}. In my new class I need to call
> {{planLater}} function. However this method is defined in {{SparkPlanner}}
> (which itself inherits the method from {{QueryPlanner}}).
> To my knowledge the only way to make {{planLater}} function visible to my new
> strategy is to define my strategy inside another class that extends
> {{SparkPlanner}} and inherits {{planLater}} as a result, by doing so I will
> have to extend the {{SQLContext}} such that I can override the {{planner}}
> field with the new {{Planner}} class I created.
> It seems that this is a design problem because adding a new strategy seems to
> require extending {{SQLContext}} (unless I am doing it wrong and there is a
> better way to do it).
> Thanks a lot,
> Youssef
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
