I'm trying to use Drill with a proprietary datasource that is very fast in
applying data joins (i.e. SQL joins) and query filters (i.e. SQL where
conditions).
To connect to that datasource, I first have to write a storage plugin, but
I'm not sure if my main goal is applicable.
May main goal is to
Hi Muhammad,
It seems that the goal for filters should be possible; I’m not familiar enough
with the code to know if joins are currently supported, or if this is where
you’d have to make some contributions to Drill.
The storage plugin is called at various places in the planning process, and can
The JDBC storage plugin does attempt to do pushdowns of joins. However, the
Drill optimizer will evaluate different query plans. In doing so, it may
choose an alternative plan that does not do a full pushdown if it believes
that’s a less costly plan than a full pushdown. There are a number of
I am working on pushing down joins to Druid storage plugin. To my
experience, you need to write a rule to know whether the joins could be
pushed down by your storage plugin metadata first,then if ok ,you transfer
the join node to the scan node with the query relevant information in the
scan node. T
Priceless information ! Thank you all.
I managed to debug Drill in Eclipse hoping to get a better understanding
but I can't get my head around some stuff:
- What is the purpose of these clases\interfaces:
- ConverterRule
- DrillRel
- Prel
- JdbcStoragePlugin.JdbcPrule
I'm focusing on JOINs now, specially a query such as this: *SELECT * FROM
TABLE1, TABLE2*, drill plans to transform this into 2 separate full scan
queries and then perform the cartesian product join on it's own. I'm trying
to make drill send the query as it is in a single scan (group scan ?)
@weij
my suggestion is you define a rule which matches the DrillJoinRel RelNode ,
then at the onMatch method ,you traverse the join children to find the
ScanRel nodes . You define a new ScanRel which include the ScanRel nodes
you find last step. Then transform the JoinRel to this equivalent new
ScanRel.
to avoid misunderstanding , the new equivalent ScanRel is to have the
joined ScanRel nodes's GroupScans, as the GroupScans indirectly hold the
underlying storage information.
On Wed, Mar 29, 2017 at 10:15 AM, weijie tong
wrote:
>
> my suggestion is you define a rule which matches the DrillJoinRe
Thanks a lot Weijie, I believe I'm very close now. I hope you don't mind
few more questions please:
1. The new rule you are mentioning is a physical rule ? So I should
implement the Prel interface ?
2. By "traversing the join to find the ScanRel"
- This sounds like I have to "sear
I mean the rule you write could be placed in the PlannerPhase.JOIN_PlANNING
which uses the HepPlanner. This phase is to solve the logical relnode .
Hope to help you.
Muhammad Gelbana 于2017年3月30日 周四上午12:07写道：
> Thanks a lot Weijie, I believe I'm very close now. I hope you don't mind
> few more que
*This is my rule class*
public class CartesianProductJoinRule extends RelOptRule {
public static final CartesianProductJoinRule INSTANCE = new
CartesianProductJoinRule(DrillJoinRel.class);
public CartesianProductJoinRule(Class clazz) {
super(operand(clazz, operand(RelNode.class,
your code seems right , just to implement the 'call.transformTo()' ,but the
left detail , maybe I think I can't express the left things so precisely,
just as @Paul Rogers mentioned the plugin detail is a little trivial.
1. drillScanRel.getGroupScan .
2. you need to extend the AbstractGroupScan ,
So I intend to use this constructor for the new *RelNode*:
*org.apache.drill.exec.planner.logical.DrillScanRel.DrillScanRel(RelOptCluster,
RelTraitSet, RelOptTable, GroupScan, RelDataType, List)*
How can I provide it's parameters ?
1. *RelOptCluster*: Can I pass *DrillJoinRel.getCluster()* ?
I've succeeded, theoretically, in what I wanted to do because I had to send
the selected columns manually to my datasource. Would someone please tell
me how can I identify the selected columns in the join ? I searched a lot
without success.
*-*
*Muhammad Gelbana*
http://www.lin
some tips:
1. you need to know the RexInputRef index relationship between the
JoinRel's and its inputs's .
join ( 1,2 ,3,4,5)
left input(1,2,3) right input (1,2)
1,2,3, ===> left input (1 ,2,3)
4,5 >right input (1,2)
2. you capture the index map relationship when you iterate over your
I have done it. Thanks a lot Weijie and all of you for your time.
*-*
*Muhammad Gelbana*
http://www.linkedin.com/in/mgelbana
On Thu, Apr 6, 2017 at 3:15 PM, weijie tong wrote:
> some tips:
> 1. you need to know the RexInputRef index relationship between the
> JoinRel's and
16 matches
Mail list logo
