Re: How to convert a logical operator into multiple physical operators

[Bhavya Aggarwal]

Thanks for the reply, so I looked at the example and created a Nested
RelNode for my TableScanOperator as shown below

override def onMatch(call: RelOptRuleCall): Unit = {
val tableScan : LogicalTableScan = call.rel[LogicalTableScan](0);
val builder = call.builder()
val newOp = new ReadTable(
  tableScan.getCluster,
  RelTraitSet.createEmpty
.plus(SqreamRel.CONVENTION),
  List.empty,
  tableScan.getTable
)

val input = builder.push(tableScan).empty().build()

val cpuDecompress  = new CpuDecompress(tableScan.getCluster,
 RelTraitSet.createEmpty
  .plus(SqreamRel.CONVENTION), newOp)


val collationListBuilder = new ImmutableList.Builder[RelFieldCollation]

val cpuToGpu = new CpuToGpu(tableScan.getCluster,
  RelTraitSet.createEmpty().plus(SqreamRel.CONVENTION),
  cpuDecompress,RelCollationImpl.of(collationListBuilder.build()))

val gpuTransform = new
GpuTransform(tableScan.getCluster,RelTraitSet.createEmpty().plus(SqreamRel.CONVENTION),
cpuToGpu )

builder.push(gpuTransform)
call.transformTo( builder.build())

  }

But in this case, I am getting an exception from the Planner which is
related to ReadTable which is extending the tablescan. How can I wrap my
operators around the table scan operator.

Exception in thread "main" java.lang.ClassCastException: class
com.sqream.calcite.aux.rel.ReadTable cannot be cast to class
org.apache.calcite.plan.hep.HepRelVertex
(com.sqream.calcite.aux.rel.ReadTable and
org.apache.calcite.plan.hep.HepRelVertex are in unnamed module of loader
'app')
at org.apache.calcite.plan.hep.HepPlanner.addRelToGraph(HepPlanner.java:849)
at org.apache.calcite.plan.hep.HepPlanner.addRelToGraph(HepPlanner.java:819)
at org.apache.calcite.plan.hep.HepPlanner.addRelToGraph(HepPlanner.java:819)
at
org.apache.calcite.plan.hep.HepPlanner.applyTransformationResults(HepPlanner.java:754)
at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:565)
at org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:427)
at
org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:264)
at
org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:127)
at
org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:223)
at org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:210)
at TestSqream$.delayedEndpoint$TestSqream$1(TestSqream.scala:77)
at TestSqream$delayedInit$body.apply(TestSqream.scala:24)
at scala.Function0.apply$mcV$sp(Function0.scala:39)
at scala.Function0.apply$mcV$sp$(Function0.scala:39)
at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
at scala.App.$anonfun$main$1$adapted(App.scala:80)
at scala.collection.immutable.List.foreach(List.scala:431)
at scala.App.main(App.scala:80)
at scala.App.main$(App.scala:78)
at TestSqream$.main(TestSqream.scala:24)
at TestSqream.main(TestSqream.scala)


>
>
> On Thu, Nov 26, 2020 at 1:20 PM Bhavya Aggarwal 
> wrote:
>
>> Hi All,
>>
>> We are trying to use Apache Calcite as and CBO for our homegrown DB and
>> there one Logical operator will translate into multiple Physical Operations
>> at our database e.g. a simple TableScan will convert into the given
>> structure. In the example above we have to decompress the data, then
>> rechunk it. However, I am able to create 1:1 conversion. Please let me know
>> how to proceed. Any help will be much appreciated.
>>
>> select id from users where id = 5;
>> AddToNetworkQueue 1
>>   Rechunk 2
>> ReorderInput 3
>> ReorderInput 4
>>   Filter 5
>>  Rechunk 6
>>   Decompress 7
>>ReadTable 8
>>
>> Thanks and regards
>> Bhavya
>> --
>> *Bhavya Aggarwal*
>> CTO & Partner
>> Knoldus Inc. 
>> +91-9910483067
>> Canada - USA - India - Singapore
>>  
>>  
>>
>
>
>

-- 
Your feedback matters - At Knoldus we aim to be very professional in our 
quality of work, commitment to results, and proactive communication. If you 
feel otherwise please share your feedback 
 and we would work on it. 

Re: How to convert a logical operator into multiple physical operators

[Chunwei Lei]

Hi, Bhavya.

I think all you need to do is to create the node correctly. You can
refer to how the EnumerableMergeJoin is created[1].

[1]
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/adapter/enumerable/EnumerableJoinRule.java#L86


Best,
Chunwei


On Fri, Nov 27, 2020 at 9:23 AM Michael Mior  wrote:

> Do you need to use CBO for the physical operators as well? If not, you
> could certainly just keep the single TableScan and perform  whatever
> operations you want at query execution time. If you want to use CBO
> with your physical operators, there is no requirement that an
> optimizer rule produce the same number of RelNodes. You could write a
> rule which converts a TableScan into the tree of operators you showed
> and then write further rules which manipulate the physical operators.
> --
> Michael Mior
> mm...@apache.org
>
> Le jeu. 26 nov. 2020 à 16:53, Bhavya Aggarwal  a
> écrit :
> >
> > Hi All,
> >
> > We are trying to use Apache Calcite as and CBO for our homegrown DB and
> > there one Logical operator will translate into multiple Physical
> Operations
> > at our database e.g. a simple TableScan will convert into the given
> > structure. In the example above we have to decompress the data, then
> > rechunk it. However, I am able to create 1:1 conversion. Please let me
> know
> > how to proceed. Any help will be much appreciated.
> >
> > select id from users where id = 5;
> > AddToNetworkQueue 1
> >   Rechunk 2
> > ReorderInput 3
> > ReorderInput 4
> >   Filter 5
> >  Rechunk 6
> >   Decompress 7
> >ReadTable 8
> >
> > Thanks and regards
> > Bhavya
> > --
> > *Bhavya Aggarwal*
> > CTO & Partner
> > Knoldus Inc. 
> > +91-9910483067
> > Canada - USA - India - Singapore
> >   >
> >  
> >
> > --
> > Your feedback matters - At Knoldus we aim to be very professional in our
> > quality of work, commitment to results, and proactive communication. If
> you
> > feel otherwise please share your feedback
> >  and we would work on it.
>

Re: How to convert a logical operator into multiple physical operators

[Michael Mior]

Do you need to use CBO for the physical operators as well? If not, you
could certainly just keep the single TableScan and perform  whatever
operations you want at query execution time. If you want to use CBO
with your physical operators, there is no requirement that an
optimizer rule produce the same number of RelNodes. You could write a
rule which converts a TableScan into the tree of operators you showed
and then write further rules which manipulate the physical operators.
--
Michael Mior
mm...@apache.org

Le jeu. 26 nov. 2020 à 16:53, Bhavya Aggarwal  a écrit :
>
> Hi All,
>
> We are trying to use Apache Calcite as and CBO for our homegrown DB and
> there one Logical operator will translate into multiple Physical Operations
> at our database e.g. a simple TableScan will convert into the given
> structure. In the example above we have to decompress the data, then
> rechunk it. However, I am able to create 1:1 conversion. Please let me know
> how to proceed. Any help will be much appreciated.
>
> select id from users where id = 5;
> AddToNetworkQueue 1
>   Rechunk 2
> ReorderInput 3
> ReorderInput 4
>   Filter 5
>  Rechunk 6
>   Decompress 7
>ReadTable 8
>
> Thanks and regards
> Bhavya
> --
> *Bhavya Aggarwal*
> CTO & Partner
> Knoldus Inc. 
> +91-9910483067
> Canada - USA - India - Singapore
>  
>  
>
> --
> Your feedback matters - At Knoldus we aim to be very professional in our
> quality of work, commitment to results, and proactive communication. If you
> feel otherwise please share your feedback
>  and we would work on it.

How to convert a logical operator into multiple physical operators

[Bhavya Aggarwal]

Hi All,

We are trying to use Apache Calcite as and CBO for our homegrown DB and
there one Logical operator will translate into multiple Physical Operations
at our database e.g. a simple TableScan will convert into the given
structure. In the example above we have to decompress the data, then
rechunk it. However, I am able to create 1:1 conversion. Please let me know
how to proceed. Any help will be much appreciated.

select id from users where id = 5;
AddToNetworkQueue 1
  Rechunk 2
ReorderInput 3
ReorderInput 4
  Filter 5
 Rechunk 6
  Decompress 7
   ReadTable 8

Thanks and regards
Bhavya
-- 
*Bhavya Aggarwal*
CTO & Partner
Knoldus Inc. 
+91-9910483067
Canada - USA - India - Singapore
 
 

-- 
Your feedback matters - At Knoldus we aim to be very professional in our 
quality of work, commitment to results, and proactive communication. If you 
feel otherwise please share your feedback 
 and we would work on it.