Re: Enumerable rules lost hint info

[Danny Chan]

We did have some hints for our physical rules to apply some strategies, the
very common case is the JOIN algorithm, such as the NESTED_LOOP,
SHUFFLE_HASH, SORT_MERGE and so on.

The only reason that I do not implement the physical hints for Calcite is
that I do not have enough time for that, sorry.

Jia Tao, what kind of physical hints do you expect to have for Calcite
physical operators ?

Best,
Danny Chan

Julian Hyde  于2021年5月20日周四 上午1:06写道：

> Sure, there’s a slot to hold the hints. But is there a benefit to keeping
> them in the physical algebra? There is certainly a cost to storing them,
> propagating them, fixing them up as we apply rules.
>
> The main source of hints is formatted comments in the SQL. As we get
> further from the SQL source and the AST, the benefit decreases.
>
> Danny,
>
> Were you envisioning hints being propagated into physical algebra? There’s
> no explicit mention in
> https://docs.google.com/document/d/1mykz-w2t1Yw7CH6NjUWpWqCAf_6YNKxSc59gXafrNCs/edit
> <
> https://docs.google.com/document/d/1mykz-w2t1Yw7CH6NjUWpWqCAf_6YNKxSc59gXafrNCs/edit>
> or https://issues.apache.org/jira/browse/CALCITE-482 <
> https://issues.apache.org/jira/browse/CALCITE-482>.
>
> Julian
>
>
> > On May 18, 2021, at 7:49 PM, JiaTao Tao  wrote:
> >
> > Hi Julian
> >
> > When hints were introduced, was that part of the plan?
> >
> > "hint" is a field in the base class, so the type of the node doesn't
> matter
> >
> > public abstract class Project extends SingleRel implements Hintable {
> >  //~ Instance fields
> 
> >
> >  protected final ImmutableList exps;
> >
> >  protected final ImmutableList hints;
> >
> >
> > And in RelBuilder#project, we just construct a project without
> > hints(ImmutableList.of()), so it will lose hints too.
> >
> >  public RelBuilder project(Iterable nodes,
> >  Iterable fieldNames, boolean force) {
> >return project_(nodes, fieldNames, ImmutableList.of(), force);
> >  }
> >
> >
> >
> > Regards!
> >
> > Aron Tao
> >
> >
> > Julian Hyde  于2021年5月19日周三 上午12:30写道：
> >
> >> Do we really need hints in physical operators? When hints were
> introduced,
> >> was that part of the plan?
> >>
> >> Julian
> >>
> >>> On May 17, 2021, at 11:50 PM, JiaTao Tao  wrote:
> >>>
> >>> Hi
> >>> Recently, I'm integrating Calcite's hint into our system, and I've
> found
> >>> the Enumerable rules just throw the hint info(ImmutableList.of()) away:
> >>>
> >>> public EnumerableProject(
> >>> RelOptCluster cluster,
> >>> RelTraitSet traitSet,
> >>> RelNode input,
> >>> List projects,
> >>> RelDataType rowType) {
> >>>   super(cluster, traitSet, ImmutableList.of(), input, projects,
> >> rowType);
> >>>   assert getConvention() instanceof EnumerableConvention;
> >>> }
> >>>
> >>>
> >>> Regards!
> >>>
> >>> Aron Tao
> >>
>
>

Re: Enumerable rules lost hint info

[Julian Hyde]

One principle we should apply here: if a so-called “hint” causes the physical 
operator to have a different cost then it’s not a hint; it is a parameter. An 
example would be the buffer size allocated to an external sort. You might 
conceivably have two instances of that operator with different buffer sizes, 
and the optimizer would choose between them based on cost.

> On May 21, 2021, at 1:30 AM, Stamatis Zampetakis  wrote:
> 
> There are use cases where it makes sense to keep/pass hints and others that
> I assume makes sense to discard them (not used by the operator, rules,
> implementors, etc).
> 
> Since the discussion was brought up for the Enumerable operators what's the
> benefit of retaining hints at this level?
> As far as I remember the Enumerable operators and existing Enumerable rules
> do not take any decision based on hints.
> 
> Best,
> Stamatis
> 
> On Fri, May 21, 2021 at 4:35 AM JiaTao Tao  wrote:
> 
>> +1 With Haisheng
>> Besides, seems we will lose hint even in the logical
>> plan,  in RelBuilder#project, we just construct a project without
>> hints(ImmutableList.of()).
>> 
>> Regards!
>> 
>> Aron Tao
>> 
>> 
>> Haisheng Yuan  于2021年5月21日周五 上午9:20写道：
>> 
>> The hint can be used to specify the degree of parallelism (DOP), MIN/MAX
>> memory allocated for the operator. In that case, we need to keep them in
>> the physical operators. But I am not sure whether there are downstream
>> projects that are using hints for physical resource.
>> 
>> On 2021/05/19 17:05:44, Julian Hyde  wrote:
>>> Sure, there’s a slot to hold the hints. But is there a benefit to keeping
>> them in the physical algebra? There is certainly a cost to storing them,
>> propagating them, fixing them up as we apply rules.
>>> 
>>> The main source of hints is formatted comments in the SQL. As we get
>> further from the SQL source and the AST, the benefit decreases.
>>> 
>>> Danny,
>>> 
>>> Were you envisioning hints being propagated into physical algebra?
>> There’s no explicit mention in
>> 
>> https://docs.google.com/document/d/1mykz-w2t1Yw7CH6NjUWpWqCAf_6YNKxSc59gXafrNCs/edit
>> <
>> 
>> https://docs.google.com/document/d/1mykz-w2t1Yw7CH6NjUWpWqCAf_6YNKxSc59gXafrNCs/edit
>>> 
>> or https://issues.apache.org/jira/browse/CALCITE-482 <
>> https://issues.apache.org/jira/browse/CALCITE-482>.
>>> 
>>> Julian
>>> 
>>> 
 On May 18, 2021, at 7:49 PM, JiaTao Tao  wrote:
 
 Hi Julian
 
 When hints were introduced, was that part of the plan?
 
 "hint" is a field in the base class, so the type of the node doesn't
>> matter
 
 public abstract class Project extends SingleRel implements Hintable {
 //~ Instance fields
>> 
 
 protected final ImmutableList exps;
 
 protected final ImmutableList hints;
 
 
 And in RelBuilder#project, we just construct a project without
 hints(ImmutableList.of()), so it will lose hints too.
 
 public RelBuilder project(Iterable nodes,
 Iterable fieldNames, boolean force) {
   return project_(nodes, fieldNames, ImmutableList.of(), force);
 }
 
 
 
 Regards!
 
 Aron Tao
 
 
 Julian Hyde  于2021年5月19日周三 上午12:30写道：
 
> Do we really need hints in physical operators? When hints were
>> introduced,
> was that part of the plan?
> 
> Julian
> 
>> On May 17, 2021, at 11:50 PM, JiaTao Tao 
>> wrote:
>> 
>> Hi
>> Recently, I'm integrating Calcite's hint into our system, and I've
>> found
>> the Enumerable rules just throw the hint info(ImmutableList.of())
>> away:
>> 
>> public EnumerableProject(
>>RelOptCluster cluster,
>>RelTraitSet traitSet,
>>RelNode input,
>>List projects,
>>RelDataType rowType) {
>>  super(cluster, traitSet, ImmutableList.of(), input, projects,
> rowType);
>>  assert getConvention() instanceof EnumerableConvention;
>> }
>> 
>> 
>> Regards!
>> 
>> Aron Tao
> 
>>> 
>>> 
>> 

Re: Enumerable rules lost hint info

[Stamatis Zampetakis]

There are use cases where it makes sense to keep/pass hints and others that
I assume makes sense to discard them (not used by the operator, rules,
implementors, etc).

Since the discussion was brought up for the Enumerable operators what's the
benefit of retaining hints at this level?
As far as I remember the Enumerable operators and existing Enumerable rules
do not take any decision based on hints.

Best,
Stamatis

On Fri, May 21, 2021 at 4:35 AM JiaTao Tao  wrote:

> +1 With Haisheng
> Besides, seems we will lose hint even in the logical
> plan,  in RelBuilder#project, we just construct a project without
> hints(ImmutableList.of()).
>
> Regards!
>
> Aron Tao
>
>
> Haisheng Yuan  于2021年5月21日周五 上午9:20写道：
>
> The hint can be used to specify the degree of parallelism (DOP), MIN/MAX
> memory allocated for the operator. In that case, we need to keep them in
> the physical operators. But I am not sure whether there are downstream
> projects that are using hints for physical resource.
>
> On 2021/05/19 17:05:44, Julian Hyde  wrote:
> > Sure, there’s a slot to hold the hints. But is there a benefit to keeping
> them in the physical algebra? There is certainly a cost to storing them,
> propagating them, fixing them up as we apply rules.
> >
> > The main source of hints is formatted comments in the SQL. As we get
> further from the SQL source and the AST, the benefit decreases.
> >
> > Danny,
> >
> > Were you envisioning hints being propagated into physical algebra?
> There’s no explicit mention in
>
> https://docs.google.com/document/d/1mykz-w2t1Yw7CH6NjUWpWqCAf_6YNKxSc59gXafrNCs/edit
> <
>
> https://docs.google.com/document/d/1mykz-w2t1Yw7CH6NjUWpWqCAf_6YNKxSc59gXafrNCs/edit
> >
> or https://issues.apache.org/jira/browse/CALCITE-482 <
> https://issues.apache.org/jira/browse/CALCITE-482>.
> >
> > Julian
> >
> >
> > > On May 18, 2021, at 7:49 PM, JiaTao Tao  wrote:
> > >
> > > Hi Julian
> > >
> > > When hints were introduced, was that part of the plan?
> > >
> > > "hint" is a field in the base class, so the type of the node doesn't
> matter
> > >
> > > public abstract class Project extends SingleRel implements Hintable {
> > >  //~ Instance fields
> 
> > >
> > >  protected final ImmutableList exps;
> > >
> > >  protected final ImmutableList hints;
> > >
> > >
> > > And in RelBuilder#project, we just construct a project without
> > > hints(ImmutableList.of()), so it will lose hints too.
> > >
> > >  public RelBuilder project(Iterable nodes,
> > >  Iterable fieldNames, boolean force) {
> > >return project_(nodes, fieldNames, ImmutableList.of(), force);
> > >  }
> > >
> > >
> > >
> > > Regards!
> > >
> > > Aron Tao
> > >
> > >
> > > Julian Hyde  于2021年5月19日周三 上午12:30写道：
> > >
> > >> Do we really need hints in physical operators? When hints were
> introduced,
> > >> was that part of the plan?
> > >>
> > >> Julian
> > >>
> > >>> On May 17, 2021, at 11:50 PM, JiaTao Tao 
> wrote:
> > >>>
> > >>> Hi
> > >>> Recently, I'm integrating Calcite's hint into our system, and I've
> found
> > >>> the Enumerable rules just throw the hint info(ImmutableList.of())
> away:
> > >>>
> > >>> public EnumerableProject(
> > >>> RelOptCluster cluster,
> > >>> RelTraitSet traitSet,
> > >>> RelNode input,
> > >>> List projects,
> > >>> RelDataType rowType) {
> > >>>   super(cluster, traitSet, ImmutableList.of(), input, projects,
> > >> rowType);
> > >>>   assert getConvention() instanceof EnumerableConvention;
> > >>> }
> > >>>
> > >>>
> > >>> Regards!
> > >>>
> > >>> Aron Tao
> > >>
> >
> >
>

Re: Enumerable rules lost hint info

[JiaTao Tao]

+1 With Haisheng
Besides, seems we will lose hint even in the logical
plan,  in RelBuilder#project, we just construct a project without
hints(ImmutableList.of()).

Regards!

Aron Tao


Haisheng Yuan  于2021年5月21日周五 上午9:20写道：

The hint can be used to specify the degree of parallelism (DOP), MIN/MAX
memory allocated for the operator. In that case, we need to keep them in
the physical operators. But I am not sure whether there are downstream
projects that are using hints for physical resource.

On 2021/05/19 17:05:44, Julian Hyde  wrote:
> Sure, there’s a slot to hold the hints. But is there a benefit to keeping
them in the physical algebra? There is certainly a cost to storing them,
propagating them, fixing them up as we apply rules.
>
> The main source of hints is formatted comments in the SQL. As we get
further from the SQL source and the AST, the benefit decreases.
>
> Danny,
>
> Were you envisioning hints being propagated into physical algebra?
There’s no explicit mention in
https://docs.google.com/document/d/1mykz-w2t1Yw7CH6NjUWpWqCAf_6YNKxSc59gXafrNCs/edit
<
https://docs.google.com/document/d/1mykz-w2t1Yw7CH6NjUWpWqCAf_6YNKxSc59gXafrNCs/edit>
or https://issues.apache.org/jira/browse/CALCITE-482 <
https://issues.apache.org/jira/browse/CALCITE-482>.
>
> Julian
>
>
> > On May 18, 2021, at 7:49 PM, JiaTao Tao  wrote:
> >
> > Hi Julian
> >
> > When hints were introduced, was that part of the plan?
> >
> > "hint" is a field in the base class, so the type of the node doesn't
matter
> >
> > public abstract class Project extends SingleRel implements Hintable {
> >  //~ Instance fields

> >
> >  protected final ImmutableList exps;
> >
> >  protected final ImmutableList hints;
> >
> >
> > And in RelBuilder#project, we just construct a project without
> > hints(ImmutableList.of()), so it will lose hints too.
> >
> >  public RelBuilder project(Iterable nodes,
> >  Iterable fieldNames, boolean force) {
> >return project_(nodes, fieldNames, ImmutableList.of(), force);
> >  }
> >
> >
> >
> > Regards!
> >
> > Aron Tao
> >
> >
> > Julian Hyde  于2021年5月19日周三 上午12:30写道：
> >
> >> Do we really need hints in physical operators? When hints were
introduced,
> >> was that part of the plan?
> >>
> >> Julian
> >>
> >>> On May 17, 2021, at 11:50 PM, JiaTao Tao  wrote:
> >>>
> >>> Hi
> >>> Recently, I'm integrating Calcite's hint into our system, and I've
found
> >>> the Enumerable rules just throw the hint info(ImmutableList.of())
away:
> >>>
> >>> public EnumerableProject(
> >>> RelOptCluster cluster,
> >>> RelTraitSet traitSet,
> >>> RelNode input,
> >>> List projects,
> >>> RelDataType rowType) {
> >>>   super(cluster, traitSet, ImmutableList.of(), input, projects,
> >> rowType);
> >>>   assert getConvention() instanceof EnumerableConvention;
> >>> }
> >>>
> >>>
> >>> Regards!
> >>>
> >>> Aron Tao
> >>
>
>

Re: Enumerable rules lost hint info

[Haisheng Yuan]

The hint can be used to specify the degree of parallelism (DOP), MIN/MAX memory 
allocated for the operator. In that case, we need to keep them in the physical 
operators. But I am not sure whether there are downstream projects that are 
using hints for physical resource.

On 2021/05/19 17:05:44, Julian Hyde  wrote: 
> Sure, there’s a slot to hold the hints. But is there a benefit to keeping 
> them in the physical algebra? There is certainly a cost to storing them, 
> propagating them, fixing them up as we apply rules.
> 
> The main source of hints is formatted comments in the SQL. As we get further 
> from the SQL source and the AST, the benefit decreases.
> 
> Danny,
> 
> Were you envisioning hints being propagated into physical algebra? There’s no 
> explicit mention in 
> https://docs.google.com/document/d/1mykz-w2t1Yw7CH6NjUWpWqCAf_6YNKxSc59gXafrNCs/edit
>  
> 
>  or https://issues.apache.org/jira/browse/CALCITE-482 
> .
> 
> Julian
> 
> 
> > On May 18, 2021, at 7:49 PM, JiaTao Tao  wrote:
> > 
> > Hi Julian
> > 
> > When hints were introduced, was that part of the plan?
> > 
> > "hint" is a field in the base class, so the type of the node doesn't matter
> > 
> > public abstract class Project extends SingleRel implements Hintable {
> >  //~ Instance fields 
> > 
> > 
> >  protected final ImmutableList exps;
> > 
> >  protected final ImmutableList hints;
> > 
> > 
> > And in RelBuilder#project, we just construct a project without
> > hints(ImmutableList.of()), so it will lose hints too.
> > 
> >  public RelBuilder project(Iterable nodes,
> >  Iterable fieldNames, boolean force) {
> >return project_(nodes, fieldNames, ImmutableList.of(), force);
> >  }
> > 
> > 
> > 
> > Regards!
> > 
> > Aron Tao
> > 
> > 
> > Julian Hyde  于2021年5月19日周三 上午12:30写道：
> > 
> >> Do we really need hints in physical operators? When hints were introduced,
> >> was that part of the plan?
> >> 
> >> Julian
> >> 
> >>> On May 17, 2021, at 11:50 PM, JiaTao Tao  wrote:
> >>> 
> >>> Hi
> >>> Recently, I'm integrating Calcite's hint into our system, and I've found
> >>> the Enumerable rules just throw the hint info(ImmutableList.of()) away:
> >>> 
> >>> public EnumerableProject(
> >>> RelOptCluster cluster,
> >>> RelTraitSet traitSet,
> >>> RelNode input,
> >>> List projects,
> >>> RelDataType rowType) {
> >>>   super(cluster, traitSet, ImmutableList.of(), input, projects,
> >> rowType);
> >>>   assert getConvention() instanceof EnumerableConvention;
> >>> }
> >>> 
> >>> 
> >>> Regards!
> >>> 
> >>> Aron Tao
> >> 
> 
> 

Re: Enumerable rules lost hint info

[Julian Hyde]

Sure, there’s a slot to hold the hints. But is there a benefit to keeping them 
in the physical algebra? There is certainly a cost to storing them, propagating 
them, fixing them up as we apply rules.

The main source of hints is formatted comments in the SQL. As we get further 
from the SQL source and the AST, the benefit decreases.

Danny,

Were you envisioning hints being propagated into physical algebra? There’s no 
explicit mention in 
https://docs.google.com/document/d/1mykz-w2t1Yw7CH6NjUWpWqCAf_6YNKxSc59gXafrNCs/edit
 

 or https://issues.apache.org/jira/browse/CALCITE-482 
.

Julian


> On May 18, 2021, at 7:49 PM, JiaTao Tao  wrote:
> 
> Hi Julian
> 
> When hints were introduced, was that part of the plan?
> 
> "hint" is a field in the base class, so the type of the node doesn't matter
> 
> public abstract class Project extends SingleRel implements Hintable {
>  //~ Instance fields 
> 
>  protected final ImmutableList exps;
> 
>  protected final ImmutableList hints;
> 
> 
> And in RelBuilder#project, we just construct a project without
> hints(ImmutableList.of()), so it will lose hints too.
> 
>  public RelBuilder project(Iterable nodes,
>  Iterable fieldNames, boolean force) {
>return project_(nodes, fieldNames, ImmutableList.of(), force);
>  }
> 
> 
> 
> Regards!
> 
> Aron Tao
> 
> 
> Julian Hyde  于2021年5月19日周三 上午12:30写道：
> 
>> Do we really need hints in physical operators? When hints were introduced,
>> was that part of the plan?
>> 
>> Julian
>> 
>>> On May 17, 2021, at 11:50 PM, JiaTao Tao  wrote:
>>> 
>>> Hi
>>> Recently, I'm integrating Calcite's hint into our system, and I've found
>>> the Enumerable rules just throw the hint info(ImmutableList.of()) away:
>>> 
>>> public EnumerableProject(
>>> RelOptCluster cluster,
>>> RelTraitSet traitSet,
>>> RelNode input,
>>> List projects,
>>> RelDataType rowType) {
>>>   super(cluster, traitSet, ImmutableList.of(), input, projects,
>> rowType);
>>>   assert getConvention() instanceof EnumerableConvention;
>>> }
>>> 
>>> 
>>> Regards!
>>> 
>>> Aron Tao
>> 

Re: Enumerable rules lost hint info

[JiaTao Tao]

Hi Julian

When hints were introduced, was that part of the plan?

"hint" is a field in the base class, so the type of the node doesn't matter

public abstract class Project extends SingleRel implements Hintable {
  //~ Instance fields 

  protected final ImmutableList exps;

  protected final ImmutableList hints;


And in RelBuilder#project, we just construct a project without
hints(ImmutableList.of()), so it will lose hints too.

  public RelBuilder project(Iterable nodes,
  Iterable fieldNames, boolean force) {
return project_(nodes, fieldNames, ImmutableList.of(), force);
  }



Regards!

Aron Tao


Julian Hyde  于2021年5月19日周三 上午12:30写道：

> Do we really need hints in physical operators? When hints were introduced,
> was that part of the plan?
>
> Julian
>
> > On May 17, 2021, at 11:50 PM, JiaTao Tao  wrote:
> >
> > Hi
> > Recently, I'm integrating Calcite's hint into our system, and I've found
> > the Enumerable rules just throw the hint info(ImmutableList.of()) away:
> >
> >  public EnumerableProject(
> >  RelOptCluster cluster,
> >  RelTraitSet traitSet,
> >  RelNode input,
> >  List projects,
> >  RelDataType rowType) {
> >super(cluster, traitSet, ImmutableList.of(), input, projects,
> rowType);
> >assert getConvention() instanceof EnumerableConvention;
> >  }
> >
> >
> > Regards!
> >
> > Aron Tao
>

Re: Enumerable rules lost hint info

[Julian Hyde]

Do we really need hints in physical operators? When hints were introduced, was 
that part of the plan?

Julian

> On May 17, 2021, at 11:50 PM, JiaTao Tao  wrote:
> 
> Hi
> Recently, I'm integrating Calcite's hint into our system, and I've found
> the Enumerable rules just throw the hint info(ImmutableList.of()) away:
> 
>  public EnumerableProject(
>  RelOptCluster cluster,
>  RelTraitSet traitSet,
>  RelNode input,
>  List projects,
>  RelDataType rowType) {
>super(cluster, traitSet, ImmutableList.of(), input, projects, rowType);
>assert getConvention() instanceof EnumerableConvention;
>  }
> 
> 
> Regards!
> 
> Aron Tao