[jira] [Commented] (DRILL-5977) predicate pushdown support kafkaMsgOffset
[
https://issues.apache.org/jira/browse/DRILL-5977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16421956#comment-16421956
]
B Anil Kumar commented on DRILL-5977:
-
Sounds good [~aravi5] . Please feel free to assign this JIRA to yourself.
> predicate pushdown support kafkaMsgOffset
> -
>
> Key: DRILL-5977
> URL: https://issues.apache.org/jira/browse/DRILL-5977
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: B Anil Kumar
>Assignee: Bhallamudi Venkata Siva Kamesh
>Priority: Major
> Fix For: 1.14.0
>
>
> As part of Kafka storage plugin review, below is the suggestion from Paul.
> {noformat}
> Does it make sense to provide a way to select a range of messages: a starting
> point or a count? Perhaps I want to run my query every five minutes, scanning
> only those messages since the previous scan. Or, I want to limit my take to,
> say, the next 1000 messages. Could we use a pseudo-column such as
> "kafkaMsgOffset" for that purpose? Maybe
> SELECT * FROM WHERE kafkaMsgOffset > 12345
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-5977) predicate pushdown support kafkaMsgOffset
[
https://issues.apache.org/jira/browse/DRILL-5977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16418387#comment-16418387
]
B Anil Kumar commented on DRILL-5977:
-
[~aravi5] Thanks for looking into this feature and providing the documentation.
Your approach looks good to me. But, just to note, in other storage plugin's
like Mongo plugin, we are converting the entire filter condition
expression(combination of all predicates) into Mongo filter. But in the case of
Kafka, it is not possible to achieve it.
So mostly, we might need apply predicate pushdown only in few cases.
* If predicates are on *kafkaMsgOffset* and/or *kafkaMsgTimestamp*.
* If predicates has AND condition with case 1. Example: select * from topic1
where kafkaMsgTimestamp > x AND (v1='' OR v2 = '')
And queries like select * from kafkaMsgTimestamp > x OR eventTimeStamp < y can
result in full scan.
> predicate pushdown support kafkaMsgOffset
> -
>
> Key: DRILL-5977
> URL: https://issues.apache.org/jira/browse/DRILL-5977
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: B Anil Kumar
>Assignee: Bhallamudi Venkata Siva Kamesh
>Priority: Major
> Fix For: 1.14.0
>
>
> As part of Kafka storage plugin review, below is the suggestion from Paul.
> {noformat}
> Does it make sense to provide a way to select a range of messages: a starting
> point or a count? Perhaps I want to run my query every five minutes, scanning
> only those messages since the previous scan. Or, I want to limit my take to,
> say, the next 1000 messages. Could we use a pseudo-column such as
> "kafkaMsgOffset" for that purpose? Maybe
> SELECT * FROM WHERE kafkaMsgOffset > 12345
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (DRILL-5940) Avro with schema registry support for Kafka
[ https://issues.apache.org/jira/browse/DRILL-5940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] B Anil Kumar reassigned DRILL-5940: --- Assignee: B Anil Kumar (was: Bhallamudi Venkata Siva Kamesh) > Avro with schema registry support for Kafka > --- > > Key: DRILL-5940 > URL: https://issues.apache.org/jira/browse/DRILL-5940 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - Avro, Storage - Other >Reporter: B Anil Kumar >Assignee: B Anil Kumar >Priority: Major > > Support Avro messages with Schema registry for Kafka storage plugin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-4779) Kafka storage plugin support
[ https://issues.apache.org/jira/browse/DRILL-4779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16359637#comment-16359637 ] B Anil Kumar commented on DRILL-4779: - Doc looks great. Thanks [~bbevens]. > Kafka storage plugin support > > > Key: DRILL-4779 > URL: https://issues.apache.org/jira/browse/DRILL-4779 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - Other >Affects Versions: 1.11.0 >Reporter: B Anil Kumar >Assignee: B Anil Kumar >Priority: Major > Labels: doc-impacting, ready-to-commit > Fix For: 1.12.0 > > > Implement Kafka storage plugin will enable the strong SQL support for Kafka. > Initially implementation can target for supporting json and avro message types -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (DRILL-5977) predicate pushdown support kafkaMsgOffset
[
https://issues.apache.org/jira/browse/DRILL-5977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
B Anil Kumar reassigned DRILL-5977:
---
Assignee: Bhallamudi Venkata Siva Kamesh
> predicate pushdown support kafkaMsgOffset
> -
>
> Key: DRILL-5977
> URL: https://issues.apache.org/jira/browse/DRILL-5977
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: B Anil Kumar
>Assignee: Bhallamudi Venkata Siva Kamesh
>
> As part of Kafka storage plugin review, below is the suggestion from Paul.
> {noformat}
> Does it make sense to provide a way to select a range of messages: a starting
> point or a count? Perhaps I want to run my query every five minutes, scanning
> only those messages since the previous scan. Or, I want to limit my take to,
> say, the next 1000 messages. Could we use a pseudo-column such as
> "kafkaMsgOffset" for that purpose? Maybe
> SELECT * FROM WHERE kafkaMsgOffset > 12345
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (DRILL-5977) predicate pushdown support kafkaMsgOffset
B Anil Kumar created DRILL-5977:
---
Summary: predicate pushdown support kafkaMsgOffset
Key: DRILL-5977
URL: https://issues.apache.org/jira/browse/DRILL-5977
Project: Apache Drill
Issue Type: Improvement
Reporter: B Anil Kumar
As part of Kafka storage plugin review, below is the suggestion from Paul.
{noformat}
Does it make sense to provide a way to select a range of messages: a starting
point or a count? Perhaps I want to run my query every five minutes, scanning
only those messages since the previous scan. Or, I want to limit my take to,
say, the next 1000 messages. Could we use a pseudo-column such as
"kafkaMsgOffset" for that purpose? Maybe
SELECT * FROM WHERE kafkaMsgOffset > 12345
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (DRILL-5976) Kafka MessageReader config optimization
B Anil Kumar created DRILL-5976:
---
Summary: Kafka MessageReader config optimization
Key: DRILL-5976
URL: https://issues.apache.org/jira/browse/DRILL-5976
Project: Apache Drill
Issue Type: Improvement
Reporter: B Anil Kumar
Assignee: B Anil Kumar
Kafka storage plugin currently supports JSON message format, but going forward
it will support Avro etc.
Also there might be multiple scenarios where user have to implement their own
MessageReader (Corresponding their custom Deserializer/Decoder).
So, this JIRA is to brainstorm on whether to go with MessageReader as a *Plugin
config* OR system/session option.
Paul's suggestion as part review comment is as below.
{noformat}
Suppose this is two or three releases from now and we support other forms of
Kafka messages. Different topics use different formats.
If the message format is a system/session option, then I need to switch the
option before each query. Very cumbersome and error prone.
Instead, perhaps this information should be part of the storage plugin config.
Then, I can define different plugins: one for each message format.
Further, can I have multiple Kafka servers? If so, would I need different
plugin configs for each?
So, should we be thinking about encoding most properties as plugin config
properties?
Now, the plugin might have a format property, one of which is json. The JSON
config properties would be defined in the json format within the overall
storage plugin config.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (DRILL-4779) Kafka storage plugin support
[ https://issues.apache.org/jira/browse/DRILL-4779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242170#comment-16242170 ] B Anil Kumar commented on DRILL-4779: - For Avro support we have raised a separate ticket https://issues.apache.org/jira/browse/DRILL-5940 > Kafka storage plugin support > > > Key: DRILL-4779 > URL: https://issues.apache.org/jira/browse/DRILL-4779 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - Other >Affects Versions: 1.11.0 >Reporter: B Anil Kumar >Assignee: B Anil Kumar > Labels: doc-impacting > Fix For: 1.12.0 > > > Implement Kafka storage plugin will enable the strong SQL support for Kafka. > Initially implementation can target for supporting json and avro message types -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (DRILL-5940) Avro with schema registry support for Kafka
B Anil Kumar created DRILL-5940: --- Summary: Avro with schema registry support for Kafka Key: DRILL-5940 URL: https://issues.apache.org/jira/browse/DRILL-5940 Project: Apache Drill Issue Type: New Feature Components: Storage - Other Reporter: B Anil Kumar Assignee: Bhallamudi Venkata Siva Kamesh Support Avro messages with Schema registry for Kafka storage plugin -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (DRILL-4779) Kafka storage plugin support
[ https://issues.apache.org/jira/browse/DRILL-4779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] B Anil Kumar updated DRILL-4779: Description: Implement Kafka storage plugin will enable the strong SQL support for Kafka. Initially implementation can target for supporting json and avro message types was: Implement Kafka storage plugin will enable the strong SQL support for Kafka. Initially implementation can target for supporting text, json and avro message types > Kafka storage plugin support > > > Key: DRILL-4779 > URL: https://issues.apache.org/jira/browse/DRILL-4779 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - Other >Affects Versions: 1.11.0 >Reporter: B Anil Kumar >Assignee: B Anil Kumar > Labels: doc-impacting > Fix For: 1.12.0 > > > Implement Kafka storage plugin will enable the strong SQL support for Kafka. > Initially implementation can target for supporting json and avro message types -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-4779) Kafka storage plugin support
[ https://issues.apache.org/jira/browse/DRILL-4779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16212115#comment-16212115 ] B Anil Kumar commented on DRILL-4779: - Hi [~arina] , Definitely we can plan for Drill release 1.12. But currently the status of https://github.com/akumarb2010/incubator-drill/tree/master/contrib/storage-kafka is it's working with JSON message types, but with schema registry support still has some issues. If we are unable to fix Avro with schema registry issues, Is it OK to go with only JSON support in first version Kafka support? > Kafka storage plugin support > > > Key: DRILL-4779 > URL: https://issues.apache.org/jira/browse/DRILL-4779 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - Other >Reporter: B Anil Kumar >Assignee: B Anil Kumar > > Implement Kafka storage plugin will enable the strong SQL support for Kafka. > Initially implementation can target for supporting text, json and avro > message types -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-4779) Kafka storage plugin support
[ https://issues.apache.org/jira/browse/DRILL-4779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15729026#comment-15729026 ] B Anil Kumar commented on DRILL-4779: - Hi [~rkiss] [~kam_iitkgp] and I are working on this feature. But this work is going in bit slow pace. > Kafka storage plugin support > > > Key: DRILL-4779 > URL: https://issues.apache.org/jira/browse/DRILL-4779 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - Other >Reporter: B Anil Kumar >Assignee: B Anil Kumar > > Implement Kafka storage plugin will enable the strong SQL support for Kafka. > Initially implementation can target for supporting text, json and avro > message types -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (DRILL-4779) Kafka storage plugin support
[ https://issues.apache.org/jira/browse/DRILL-4779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] B Anil Kumar reassigned DRILL-4779: --- Assignee: B Anil Kumar > Kafka storage plugin support > > > Key: DRILL-4779 > URL: https://issues.apache.org/jira/browse/DRILL-4779 > Project: Apache Drill > Issue Type: New Feature > Components: Storage - Other >Reporter: B Anil Kumar >Assignee: B Anil Kumar > > Implement Kafka storage plugin will enable the strong SQL support for Kafka. > Initially implementation can target for supporting text, json and avro > message types -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-3522) IllegalStateException from Mongo storage plugin
[
https://issues.apache.org/jira/browse/DRILL-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15138868#comment-15138868
]
B Anil Kumar commented on DRILL-3522:
-
+1 on this patch.
[~dragoncurve] Can you please rebase this patch?
> IllegalStateException from Mongo storage plugin
> ---
>
> Key: DRILL-3522
> URL: https://issues.apache.org/jira/browse/DRILL-3522
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - MongoDB
>Affects Versions: 1.1.0
>Reporter: Adam Gilmore
>Assignee: Adam Gilmore
>Priority: Critical
> Attachments: DRILL-3522.1.patch.txt
>
>
> With a Mongo storage plugin enabled, we are sporadically getting the
> following exception when running queries (even not against the Mongo storage
> plugin):
> {code}
> SYSTEM ERROR: IllegalStateException: state should be: open
> (org.apache.drill.exec.work.foreman.ForemanException) Unexpected exception
> during fragment initialization:
> org.apache.drill.common.exceptions.DrillRuntimeException: state should be:
> open
> org.apache.drill.exec.work.foreman.Foreman.run():253
> java.util.concurrent.ThreadPoolExecutor.runWorker():1145
> java.util.concurrent.ThreadPoolExecutor$Worker.run():615
> java.lang.Thread.run():745
> Caused By (com.google.common.util.concurrent.UncheckedExecutionException)
> org.apache.drill.common.exceptions.DrillRuntimeException: state should be:
> open
> com.google.common.cache.LocalCache$Segment.get():2263
> com.google.common.cache.LocalCache.get():4000
> com.google.common.cache.LocalCache.getOrLoad():4004
> com.google.common.cache.LocalCache$LocalLoadingCache.get():4874
>
> org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory$MongoSchema.getSubSchemaNames():172
>
> org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory$MongoSchema.setHolder():159
>
> org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory.registerSchemas():127
> org.apache.drill.exec.store.mongo.MongoStoragePlugin.registerSchemas():86
>
> org.apache.drill.exec.store.StoragePluginRegistry$DrillSchemaFactory.registerSchemas():328
> org.apache.drill.exec.ops.QueryContext.getRootSchema():165
> org.apache.drill.exec.ops.QueryContext.getRootSchema():154
> org.apache.drill.exec.ops.QueryContext.getRootSchema():142
> org.apache.drill.exec.ops.QueryContext.getNewDefaultSchema():128
> org.apache.drill.exec.planner.sql.DrillSqlWorker.():91
> org.apache.drill.exec.work.foreman.Foreman.runSQL():901
> org.apache.drill.exec.work.foreman.Foreman.run():242
> java.util.concurrent.ThreadPoolExecutor.runWorker():1145
> java.util.concurrent.ThreadPoolExecutor$Worker.run():615
> java.lang.Thread.run():745
> Caused By (org.apache.drill.common.exceptions.DrillRuntimeException) state
> should be: open
>
> org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory$DatabaseLoader.load():98
>
> org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory$DatabaseLoader.load():82
> com.google.common.cache.LocalCache$LoadingValueReference.loadFuture():3599
> com.google.common.cache.LocalCache$Segment.loadSync():2379
> com.google.common.cache.LocalCache$Segment.lockedGetOrLoad():2342
> com.google.common.cache.LocalCache$Segment.get():2257
> com.google.common.cache.LocalCache.get():4000
> com.google.common.cache.LocalCache.getOrLoad():4004
> com.google.common.cache.LocalCache$LocalLoadingCache.get():4874
>
> org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory$MongoSchema.getSubSchemaNames():172
>
> org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory$MongoSchema.setHolder():159
>
> org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory.registerSchemas():127
> org.apache.drill.exec.store.mongo.MongoStoragePlugin.registerSchemas():86
>
> org.apache.drill.exec.store.StoragePluginRegistry$DrillSchemaFactory.registerSchemas():328
> org.apache.drill.exec.ops.QueryContext.getRootSchema():165
> org.apache.drill.exec.ops.QueryContext.getRootSchema():154
> org.apache.drill.exec.ops.QueryContext.getRootSchema():142
> org.apache.drill.exec.ops.QueryContext.getNewDefaultSchema():128
> org.apache.drill.exec.planner.sql.DrillSqlWorker.():91
> org.apache.drill.exec.work.foreman.Foreman.runSQL():901
> org.apache.drill.exec.work.foreman.Foreman.run():242
> java.util.concurrent.ThreadPoolExecutor.runWorker():1145
> java.util.concurrent.ThreadPoolExecutor$Worker.run():615
> java.lang.Thread.run():745
> Caused By (java.lang.IllegalStateException) state should be: open
> com.mongodb.assertions.Assertions.isTrue():70
> com.mongodb.connection.BaseCluster.selectServer():79
>
[jira] [Commented] (DRILL-3478) Bson Record Reader for Mongo storage plugin
[ https://issues.apache.org/jira/browse/DRILL-3478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112214#comment-15112214 ] B Anil Kumar commented on DRILL-3478: - [~michael.stucco] Thanks for reporting. Can you please raise a JIRA. And meanwhile please use *alter session set store.mongo.bson.record.reader = false;* to switch back to json reader mode. > Bson Record Reader for Mongo storage plugin > --- > > Key: DRILL-3478 > URL: https://issues.apache.org/jira/browse/DRILL-3478 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - MongoDB >Reporter: B Anil Kumar >Assignee: B Anil Kumar > Fix For: 1.5.0 > > Attachments: 0001-DRILL-3478_1-Review-comments-fixes.patch, > Test_queries_with_review_comment_fixes, drill_bson_sqlline_test_2015_1 > > > Improve the mongo query performance. > We are considering the suggestions provided by [~dragoncurve] and [~hgunes] > in drill mailing chain. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4173) Query did not return all documents if collection using a hashed shard key
[
https://issues.apache.org/jira/browse/DRILL-4173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15081057#comment-15081057
]
B Anil Kumar commented on DRILL-4173:
-
[~ITDonald] Thanks for reporting this issue.
We are also able to reproduce issue, the root cause is, in sharded env, we
construct a mapping b/w chunk and Drillbit and assigned Drillbit will read the
chunk based on minKey and maxKey range.
In the case of hashed sharding the minKey and maxKey is not matching with
actual shardKey values. That's why there is a data loss through Drill.
We are looking into this issue.
> Query did not return all documents if collection using a hashed shard key
> -
>
> Key: DRILL-4173
> URL: https://issues.apache.org/jira/browse/DRILL-4173
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - MongoDB
>Affects Versions: 1.3.0
> Environment: Windows 2012
>Reporter: Yuqing Tang
>Assignee: Jason Altekruse
>
> MongoDB 3.0.6
> If a collection used a hashed shard key ({ "shardkey": "hashed" }), queries
> like "select * from ..." may not return all documents which should be
> returned from the collection.
> Test Case:
> Create 3 mongos, 3 config, 3 replicaset each with 3 mongod.
> Create one collection with hashed shard key.
> Insert 6 documents to this collection with shard key value 1,2,3,4,5,6
> Do a query select * from
> Only 2,3,4 will be returned.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3478) Bson Record Reader for Mongo storage plugin
[ https://issues.apache.org/jira/browse/DRILL-3478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15067341#comment-15067341 ] B Anil Kumar commented on DRILL-3478: - [~jnadeau] Can you please review latest patch with review comments fixes? > Bson Record Reader for Mongo storage plugin > --- > > Key: DRILL-3478 > URL: https://issues.apache.org/jira/browse/DRILL-3478 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - MongoDB >Reporter: B Anil Kumar >Assignee: B Anil Kumar > Fix For: Future > > Attachments: 0001-DRILL-3478_1-Review-comments-fixes.patch, > Test_queries_with_review_comment_fixes, drill_bson_sqlline_test_2015_1 > > > Improve the mongo query performance. > We are considering the suggestions provided by [~dragoncurve] and [~hgunes] > in drill mailing chain. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-3478) Bson Record Reader for Mongo storage plugin
[
https://issues.apache.org/jira/browse/DRILL-3478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059664#comment-15059664
]
B Anil Kumar commented on DRILL-3478:
-
Uploaded the new patch with review comments fixes. Please review
https://reviews.apache.org/r/40182/
With new patch made BsonRecordReader as *default* and tested with below test
cases and attached queries.
*To run test cases with Bson Record Reader:*
{noformat}
1) For sharded replicated (default)
mvn test -Ddrill.mongo.tests.shardMode=true
2) For embedded
mvn test -Ddrill.mongo.tests.shardMode=false
{noformat}
*To run with jsonRecordReader:*
{noformat}
1) For sharded replicated (default)
mvn test -Ddrill.mongo.tests.shardMode=true
-Ddrill.mongo.tests.bson.reader=false
2) For embedded
mvn test -Ddrill.mongo.tests.shardMode=false
-Ddrill.mongo.tests.bson.reader=false
{noformat}
> Bson Record Reader for Mongo storage plugin
> ---
>
> Key: DRILL-3478
> URL: https://issues.apache.org/jira/browse/DRILL-3478
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - MongoDB
>Reporter: B Anil Kumar
>Assignee: B Anil Kumar
> Fix For: Future
>
> Attachments: drill_bson_sqlline_test_2015_1
>
>
> Improve the mongo query performance.
> We are considering the suggestions provided by [~dragoncurve] and [~hgunes]
> in drill mailing chain.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-3478) Bson Record Reader for Mongo storage plugin
[ https://issues.apache.org/jira/browse/DRILL-3478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] B Anil Kumar updated DRILL-3478: Attachment: 0001-DRILL-3478_1-Review-comments-fixes.patch > Bson Record Reader for Mongo storage plugin > --- > > Key: DRILL-3478 > URL: https://issues.apache.org/jira/browse/DRILL-3478 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - MongoDB >Reporter: B Anil Kumar >Assignee: B Anil Kumar > Fix For: Future > > Attachments: 0001-DRILL-3478_1-Review-comments-fixes.patch, > Test_queries_with_review_comment_fixes, drill_bson_sqlline_test_2015_1 > > > Improve the mongo query performance. > We are considering the suggestions provided by [~dragoncurve] and [~hgunes] > in drill mailing chain. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-3478) Bson Record Reader for Mongo storage plugin
[ https://issues.apache.org/jira/browse/DRILL-3478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] B Anil Kumar updated DRILL-3478: Attachment: Test_queries_with_review_comment_fixes > Bson Record Reader for Mongo storage plugin > --- > > Key: DRILL-3478 > URL: https://issues.apache.org/jira/browse/DRILL-3478 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - MongoDB >Reporter: B Anil Kumar >Assignee: B Anil Kumar > Fix For: Future > > Attachments: Test_queries_with_review_comment_fixes, > drill_bson_sqlline_test_2015_1 > > > Improve the mongo query performance. > We are considering the suggestions provided by [~dragoncurve] and [~hgunes] > in drill mailing chain. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-3152) Apache Drill 1.0 not able to query MongoDB 3.0.
[
https://issues.apache.org/jira/browse/DRILL-3152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005771#comment-15005771
]
B Anil Kumar commented on DRILL-3152:
-
Sorry for the delay in response.
[~alinw] Can you please try as below mongo config and check?
{noformat}
{
"type": "mongo",
"connection": "mongodb://:@localhost:27017/db1",
"enabled": true
}
{noformat}
[~ttelfer] Can you once verify your issue with current 1.3 release?
> Apache Drill 1.0 not able to query MongoDB 3.0.
>
>
> Key: DRILL-3152
> URL: https://issues.apache.org/jira/browse/DRILL-3152
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - MongoDB
>Affects Versions: 0.9.0, 1.0.0
> Environment: The environment is as follows:
> Windows 7
> MongoDB 3 Wiredtiger (installed locally)
> Apache Drill 1.0 (installed locally)
>Reporter: Trent Telfer
> Labels: mongodb, mongodb3, windows7, wiredtiger
> Fix For: Future
>
>
> I have been trying to get Apache Drill 1.0, and previously 0.9 to work with
> MongoDB 3.0 Wiredtiger. I have no problem starting Apache Drill using the
> following, but I am having problems querying MongoDB:
> *./sqlline.bat*
> *!connect jdbc:drill:zk=local*
> *SHOW DATABASES;*
> +-+
> | SCHEMA_NAME |
> +-+
> | INFORMATION_SCHEMA |
> | cp.default |
> | dfs.default |
> | dfs.root|
> | dfs.tmp |
> | mongo.admin |
> | mongo.alliance_db |
> | mongo.local |
> | sys |
> +-+
> *USE mongo.alliance_db;*
> +---++
> | ok |summary |
> +---++
> | true | Default schema changed to [mongo.alliance_db] |
> +---++
> 1 row selected (0.116 seconds)
> *SELECT * FROM price_daily_ngi;*
> May 20, 2015 11:14:40 AM
> org.apache.calcite.sql.validate.SqlValidatorException
> SEVERE: org.apache.calcite.sql.validate.SqlValidatorException: Table
> 'price_daily_ngi' not found
> May 20, 2015 11:14:40 AM org.apache.calcite.runtime.CalciteException
> SEVERE: org.apache.calcite.runtime.CalciteContextException: From line 1,
> column 15 to line 1, column 29: Table 'price_daily_ngi' not found
> Error: PARSE ERROR: From line 1, column 15 to line 1, column 29: Table
> 'price_daily_ngi' not found
> [Error Id: 6414a69d-55a0-4918-8f95-10a920e4dc6b on PCV:31010] (state=,code=0)
> MongoDB storage configuration:
> {
> "type": "mongo",
> "connection": "mongodb://localhost:27017",
> "enabled": true
> }
> The collection price_daily_ngi exists and works with normal MongoDB queries.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-3478) Bson Record Reader for Mongo storage plugin
[ https://issues.apache.org/jira/browse/DRILL-3478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] B Anil Kumar updated DRILL-3478: Attachment: drill_bson_sqlline_test_2015 > Bson Record Reader for Mongo storage plugin > --- > > Key: DRILL-3478 > URL: https://issues.apache.org/jira/browse/DRILL-3478 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - MongoDB >Reporter: B Anil Kumar >Assignee: B Anil Kumar > Fix For: Future > > Attachments: drill_bson_sqlline_test_2015 > > > Improve the mongo query performance. > We are considering the suggestions provided by [~dragoncurve] and [~hgunes] > in drill mailing chain. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-3478) Bson Record Reader for Mongo storage plugin
[ https://issues.apache.org/jira/browse/DRILL-3478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15000281#comment-15000281 ] B Anil Kumar commented on DRILL-3478: - please review the patch https://reviews.apache.org/r/40182/ > Bson Record Reader for Mongo storage plugin > --- > > Key: DRILL-3478 > URL: https://issues.apache.org/jira/browse/DRILL-3478 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - MongoDB >Reporter: B Anil Kumar >Assignee: B Anil Kumar > Fix For: Future > > Attachments: drill_bson_sqlline_test_2015 > > > Improve the mongo query performance. > We are considering the suggestions provided by [~dragoncurve] and [~hgunes] > in drill mailing chain. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-3478) Bson Record Reader for Mongo storage plugin
[ https://issues.apache.org/jira/browse/DRILL-3478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15000275#comment-15000275 ] B Anil Kumar commented on DRILL-3478: - With this current bson reader support, most of bson and extended json type related issues will be resolved. And also this will optimize the drill-mongo query performance(by skipping BSON to JSON serialization and directly parsing BSON). As of now we can enable bson reader by enabling *alter session set store.mongo.bson.record.reader = true;* and default is JSON Reader. Once we do thourough performance testing then we can make bson as default. We have tested this feature in both embedded and sharded modes and also executed successfully below test cases. Attaching sqlline test results. To run test cases with Json: 1) For sharded replicated (default) mvn test -Ddrill.mongo.tests.shardMode=true 2) For embedded mvn test -Ddrill.mongo.tests.shardMode=false To run with bson: 1) For sharded replicated (default) mvn test -Ddrill.mongo.tests.shardMode=true -Ddrill.mongo.tests.bson.reader=true 2) For embedded mvn test -Ddrill.mongo.tests.shardMode=false -Ddrill.mongo.tests.bson.reader=true > Bson Record Reader for Mongo storage plugin > --- > > Key: DRILL-3478 > URL: https://issues.apache.org/jira/browse/DRILL-3478 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - MongoDB >Reporter: B Anil Kumar >Assignee: B Anil Kumar > Fix For: Future > > Attachments: drill_bson_sqlline_test_2015 > > > Improve the mongo query performance. > We are considering the suggestions provided by [~dragoncurve] and [~hgunes] > in drill mailing chain. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-3478) Bson Record Reader for Mongo storage plugin
[ https://issues.apache.org/jira/browse/DRILL-3478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] B Anil Kumar updated DRILL-3478: Attachment: (was: drill_bson_sqlline_test_2015) > Bson Record Reader for Mongo storage plugin > --- > > Key: DRILL-3478 > URL: https://issues.apache.org/jira/browse/DRILL-3478 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - MongoDB >Reporter: B Anil Kumar >Assignee: B Anil Kumar > Fix For: Future > > Attachments: drill_bson_sqlline_test_2015_1 > > > Improve the mongo query performance. > We are considering the suggestions provided by [~dragoncurve] and [~hgunes] > in drill mailing chain. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-3478) Bson Record Reader for Mongo storage plugin
[ https://issues.apache.org/jira/browse/DRILL-3478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] B Anil Kumar updated DRILL-3478: Attachment: drill_bson_sqlline_test_2015_1 > Bson Record Reader for Mongo storage plugin > --- > > Key: DRILL-3478 > URL: https://issues.apache.org/jira/browse/DRILL-3478 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - MongoDB >Reporter: B Anil Kumar >Assignee: B Anil Kumar > Fix For: Future > > Attachments: drill_bson_sqlline_test_2015_1 > > > Improve the mongo query performance. > We are considering the suggestions provided by [~dragoncurve] and [~hgunes] > in drill mailing chain. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-3478) Bson Record Reader for Mongo storage plugin
[ https://issues.apache.org/jira/browse/DRILL-3478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] B Anil Kumar updated DRILL-3478: Target Version/s: 1.4.0 > Bson Record Reader for Mongo storage plugin > --- > > Key: DRILL-3478 > URL: https://issues.apache.org/jira/browse/DRILL-3478 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - MongoDB >Reporter: B Anil Kumar >Assignee: B Anil Kumar > Fix For: Future > > Attachments: drill_bson_sqlline_test_2015 > > > Improve the mongo query performance. > We are considering the suggestions provided by [~dragoncurve] and [~hgunes] > in drill mailing chain. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-4037) No schemas available using MongoDB 3.0.6 with authentication + wired Tiger
[ https://issues.apache.org/jira/browse/DRILL-4037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] B Anil Kumar updated DRILL-4037: Component/s: (was: Client - ODBC) Storage - MongoDB > No schemas available using MongoDB 3.0.6 with authentication + wired Tiger > -- > > Key: DRILL-4037 > URL: https://issues.apache.org/jira/browse/DRILL-4037 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - MongoDB >Affects Versions: 1.2.0 > Environment: Windows 7 - 64 bits >Reporter: W. >Priority: Minor > > From Drill Explorer, Browse schemas tab, unable to view anything about MongoDB > MongoDB 3.0.6 is configured as Master/Replica with wiredTiger and > authentication enabled -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (DRILL-1586) NPE when the collection being queried for does not exist in Mongo DB
[ https://issues.apache.org/jira/browse/DRILL-1586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] B Anil Kumar closed DRILL-1586. --- Resolution: Fixed Fix Version/s: (was: Future) > NPE when the collection being queried for does not exist in Mongo DB > > > Key: DRILL-1586 > URL: https://issues.apache.org/jira/browse/DRILL-1586 > Project: Apache Drill > Issue Type: Bug > Components: Storage - MongoDB >Affects Versions: 0.6.0 >Reporter: Bhallamudi Venkata Siva Kamesh >Priority: Minor > > NPE when the collection being queried for does not exist in Mongo DB. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (DRILL-3478) Bson Record Reader for Mongo storage plugin
[ https://issues.apache.org/jira/browse/DRILL-3478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] B Anil Kumar reassigned DRILL-3478: --- Assignee: B Anil Kumar > Bson Record Reader for Mongo storage plugin > --- > > Key: DRILL-3478 > URL: https://issues.apache.org/jira/browse/DRILL-3478 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - MongoDB >Reporter: B Anil Kumar >Assignee: B Anil Kumar > Fix For: Future > > > Improve the mongo query performance. > We are considering the suggestions provided by [~dragoncurve] and [~hgunes] > in drill mailing chain. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-3478) Bson Record Reader for Mongo storage plugin
B Anil Kumar created DRILL-3478: --- Summary: Bson Record Reader for Mongo storage plugin Key: DRILL-3478 URL: https://issues.apache.org/jira/browse/DRILL-3478 Project: Apache Drill Issue Type: Improvement Components: Storage - MongoDB Reporter: B Anil Kumar Assignee: B Anil Kumar -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-3478) Bson Record Reader for Mongo storage plugin
[ https://issues.apache.org/jira/browse/DRILL-3478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14619897#comment-14619897 ] B Anil Kumar commented on DRILL-3478: - We have started working on this to optimize the mongo query performance by skipping BSON to JSON serialization and also to avoid most of the current extended json related issues. We are targeting this for 1.2.0. Bson Record Reader for Mongo storage plugin --- Key: DRILL-3478 URL: https://issues.apache.org/jira/browse/DRILL-3478 Project: Apache Drill Issue Type: Improvement Components: Storage - MongoDB Reporter: B Anil Kumar Assignee: B Anil Kumar -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-3478) Bson Record Reader for Mongo storage plugin
[ https://issues.apache.org/jira/browse/DRILL-3478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] B Anil Kumar updated DRILL-3478: Description: Improve the mongo query performance. We are considering the suggestions provided by [~dragoncurve] and [~hgunes] in drill mailing chain. Bson Record Reader for Mongo storage plugin --- Key: DRILL-3478 URL: https://issues.apache.org/jira/browse/DRILL-3478 Project: Apache Drill Issue Type: Improvement Components: Storage - MongoDB Reporter: B Anil Kumar Assignee: B Anil Kumar Improve the mongo query performance. We are considering the suggestions provided by [~dragoncurve] and [~hgunes] in drill mailing chain. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-1666) Provide Test cases for Mongo Storage plugin
[ https://issues.apache.org/jira/browse/DRILL-1666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14616833#comment-14616833 ] B Anil Kumar commented on DRILL-1666: - We have added the testcases for mongo storage plugin and uploaded the patch on review board. Please review. Review link : [36262|https://reviews.apache.org/r/36262/] Provide Test cases for Mongo Storage plugin --- Key: DRILL-1666 URL: https://issues.apache.org/jira/browse/DRILL-1666 Project: Apache Drill Issue Type: Improvement Components: Storage - MongoDB Affects Versions: 0.6.0 Reporter: Bhallamudi Venkata Siva Kamesh Fix For: Future -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-1666) Provide Test cases for Mongo Storage plugin
[ https://issues.apache.org/jira/browse/DRILL-1666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] B Anil Kumar updated DRILL-1666: Affects Version/s: (was: 0.6.0) 1.1.0 Fix Version/s: (was: Future) 1.1.0 Provide Test cases for Mongo Storage plugin --- Key: DRILL-1666 URL: https://issues.apache.org/jira/browse/DRILL-1666 Project: Apache Drill Issue Type: Improvement Components: Storage - MongoDB Affects Versions: 1.1.0 Reporter: Bhallamudi Venkata Siva Kamesh Fix For: 1.1.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-3165) Sorting a Mongo table should leverage Mongo Indexes
[
https://issues.apache.org/jira/browse/DRILL-3165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14554634#comment-14554634
]
B Anil Kumar commented on DRILL-3165:
-
Hi Leandro,
As of now, operator pushdown is not implemented in mongo storage plugin.
We are working on group, sort, limit operators pushdown, with this it should be
fast.
Sorting a Mongo table should leverage Mongo Indexes
---
Key: DRILL-3165
URL: https://issues.apache.org/jira/browse/DRILL-3165
Project: Apache Drill
Issue Type: Improvement
Components: Storage - MongoDB
Affects Versions: 1.0.0
Reporter: Leandro DG
Assignee: B Anil Kumar
When doing a query using Mongo, sorting takes place entirely in Drill.
Getting the first 1000 rows from a 100 rows table, sorted by a field
which has an index takes a long time (about 45 seconds in our test
environment).
Sample drill query:
Select c.name from mongo.foo.json_customers c order by c.name limit 1000
Doing the same in mongo client takes less than a second.
Sample mongo query: db.json_customers.find().sort({name:1}).limit(1000)
Sorting by a field should leverage the existing mongo indexes if they exist.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-3152) Apache Drill 1.0 not able to query MongoDB 3.0.
[
https://issues.apache.org/jira/browse/DRILL-3152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14554621#comment-14554621
]
B Anil Kumar commented on DRILL-3152:
-
Hi,
I just verified drill-1.0.0 with mongo v3.0.3 and it is working fine.
Can you please verify by mentioning the db name in the query as below?
{noformat}
BANL122d28a3e:drill-latest abatchu$ bin/sqlline -u jdbc:drill:zk=localhost:2181
-n admin -p admin
apache drill 1.0.0
just drill it
0: jdbc:drill:zk=localhost:2181 SELECT * FROM mongo.employee.`empinfo` limit
10;
+--+---+-+--+--+++-+
| employee_id | full_name | first_name | last_name | position_id
| position | isFTE | salary |
+--+---+-+--+--+++-+
| 1101 | Steve Eurich | Steve | Eurich | 16
| Store T| true | 20.0|
| 1102 | Mary Pierson | Mary| Pierson | 16
| Store T| true | 30.0|
| 1103 | Leo Jones | Leo | Jones| 16
| Store Tem | true | 10.0|
| 1104 | Nancy Beatty | Nancy | Beatty | 16
| Store T| false | 40.0|
| 1105 | Clara McNight | Clara | McNight | 16
| Store | true | 50.0|
| 1106 | Marcella Isaacs | Marcella| Isaacs | 17
| Stor | false | 120.0 |
| 1107 | Charlotte Yonce | Charlotte | Yonce| 17
| Stor | true | 120.0 |
| 1108 | Benjamin Foster | Benjamin| Foster | 17
| Stor | false | 22.04 |
| 1109 | John Reed | John| Reed | 17
| Store Per | false | 60.0|
| 1110 | Lynn Kwiatkowski | Lynn| Kwiatkowski | 17
| St | true | 80.0|
+--+---+-+--+--+++-+
10 rows selected (0.175 seconds)
0: jdbc:drill:zk=localhost:2181
{noformat}
Apache Drill 1.0 not able to query MongoDB 3.0.
Key: DRILL-3152
URL: https://issues.apache.org/jira/browse/DRILL-3152
Project: Apache Drill
Issue Type: Bug
Components: Storage - MongoDB
Affects Versions: 0.9.0, 1.0.0
Environment: The environment is as follows:
Windows 7
MongoDB 3 Wiredtiger (installed locally)
Apache Drill 1.0 (installed locally)
Reporter: Trent Telfer
Assignee: B Anil Kumar
Labels: mongodb, mongodb3, windows7, wiredtiger
I have been trying to get Apache Drill 1.0, and previously 0.9 to work with
MongoDB 3.0 Wiredtiger. I have no problem starting Apache Drill using the
following, but I am having problems querying MongoDB:
*./sqlline.bat*
*!connect jdbc:drill:zk=local*
*SHOW DATABASES;*
+-+
| SCHEMA_NAME |
+-+
| INFORMATION_SCHEMA |
| cp.default |
| dfs.default |
| dfs.root|
| dfs.tmp |
| mongo.admin |
| mongo.alliance_db |
| mongo.local |
| sys |
+-+
*USE mongo.alliance_db;*
+---++
| ok |summary |
+---++
| true | Default schema changed to [mongo.alliance_db] |
+---++
1 row selected (0.116 seconds)
*SELECT * FROM price_daily_ngi;*
May 20, 2015 11:14:40 AM
org.apache.calcite.sql.validate.SqlValidatorException init
SEVERE: org.apache.calcite.sql.validate.SqlValidatorException: Table
'price_daily_ngi' not found
May 20, 2015 11:14:40 AM org.apache.calcite.runtime.CalciteException init
SEVERE: org.apache.calcite.runtime.CalciteContextException: From line 1,
column 15 to line 1, column 29: Table 'price_daily_ngi' not found
Error: PARSE ERROR: From line 1, column 15 to line 1, column 29: Table
'price_daily_ngi' not found
[Error Id: 6414a69d-55a0-4918-8f95-10a920e4dc6b on PCV:31010] (state=,code=0)
MongoDB storage configuration:
{
type: mongo,
connection: mongodb://localhost:27017,
enabled: true
}
The collection price_daily_ngi exists and works with normal MongoDB queries.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-2734) Predicate pushdown like operator into Mongo.
[ https://issues.apache.org/jira/browse/DRILL-2734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14513023#comment-14513023 ] B Anil Kumar commented on DRILL-2734: - +1 Predicate pushdown like operator into Mongo. Key: DRILL-2734 URL: https://issues.apache.org/jira/browse/DRILL-2734 Project: Apache Drill Issue Type: New Feature Components: Storage - MongoDB Reporter: Bhallamudi Venkata Siva Kamesh Assignee: B Anil Kumar Labels: features Attachments: 0001-refactoring-code-and-pushing-like-operator-into-mong.patch Predicate pushdown like operator into mongo db. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-1514) Joins are not working with mongo
[
https://issues.apache.org/jira/browse/DRILL-1514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14349032#comment-14349032
]
B Anil Kumar commented on DRILL-1514:
-
Thanks Hanfi.
I verified this issue with latest code. And the queries are working fine.
Joins are not working with mongo
Key: DRILL-1514
URL: https://issues.apache.org/jira/browse/DRILL-1514
Project: Apache Drill
Issue Type: Bug
Components: Query Planning Optimization
Environment: Joins with Mongo storage plugin
Reporter: B Anil Kumar
Assignee: B Anil Kumar
Fix For: 0.9.0
As pointed in review board(https://reviews.apache.org/r/25996/), joins are
not working with mongo storage plugin.
Few sample queries:
{noformat}
1) SELECT t1.first_name FROM mongo.employee.`empinfo` t1 JOIN
mongo.employee.`empinfo` t2 ON t1.`employee_id` = t2.`employee_id`
2) SELECT t1.first_name, t2.last_name FROM mongo.employee.`empinfo` t1 JOIN
cp.`employee.json` t2 ON t1.`employee_id` = t2.`employee_id` limit 10
{noformat}
The Error is as below:
{noformat}
2014-10-09 11:57:59,721 [d1785c2e-ac11-4f1e-9f7f-1f880aeceb7e:frag:0:0] DEBUG
o.a.d.e.w.fragment.FragmentExecutor - Error while initializing or executing
fragment
java.lang.UnsupportedOperationException: Failure finding function that
runtime code generation expected. Signature: compare_to(
BIGINT:OPTIONALVARCHAR:OPTIONAL, ) returns INT:REQUIRED
at
org.apache.drill.exec.expr.fn.FunctionGenerationHelper.getFunctionExpression(FunctionGenerationHelper.java:74)
~[drill-java-exec-0.7.0-incubating-SNAPSHOT-rebuffed.jar:0.7.0-incubating-SNAPSHOT]
at
org.apache.drill.exec.expr.fn.FunctionGenerationHelper.getComparator(FunctionGenerationHelper.java:45)
~[drill-java-exec-0.7.0-incubating-SNAPSHOT-rebuffed.jar:0.7.0-incubating-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.common.ChainedHashTable.setupIsKeyMatchInternal(ChainedHashTable.java:232)
~[drill-java-exec-0.7.0-incubating-SNAPSHOT-rebuffed.jar:0.7.0-incubating-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.common.ChainedHashTable.createAndSetupHashTable(ChainedHashTable.java:182)
~[drill-java-exec-0.7.0-incubating-SNAPSHOT-rebuffed.jar:0.7.0-incubating-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.join.HashJoinBatch.setupHashTable(HashJoinBatch.java:298)
~[drill-java-exec-0.7.0-incubating-SNAPSHOT-rebuffed.jar:0.7.0-incubating-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.join.HashJoinBatch.executeBuildPhase(HashJoinBatch.java:325)
~[drill-java-exec-0.7.0-incubating-SNAPSHOT-rebuffed.jar:0.7.0-incubating-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.join.HashJoinBatch.innerNext(HashJoinBatch.java:193)
~[drill-java-exec-0.7.0-incubating-SNAPSHOT-rebuffed.jar:0.7.0-incubating-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:105)
~[drill-java-exec-0.7.0-incubating-SNAPSHOT-rebuffed.jar:0.7.0-incubating-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:117)
~[drill-java-exec-0.7.0-incubating-SNAPSHOT-rebuffed.jar:0.7.0-incubating-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:85)
~[drill-java-exec-0.7.0-incubating-SNAPSHOT-rebuffed.jar:0.7.0-incubating-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:75)
~[drill-java-exec-0.7.0-incubating-SNAPSHOT-rebuffed.jar:0.7.0-incubating-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:50)
~[drill-java-exec-0.7.0-incubating-SNAPSHOT-rebuffed.jar:0.7.0-incubating-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:127)
~[drill-java-exec-0.7.0-incubating-SNAPSHOT-rebuffed.jar:0.7.0-incubating-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:105)
~[drill-java-exec-0.7.0-incubating-SNAPSHOT-rebuffed.jar:0.7.0-incubating-SNAPSHOT]
at
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:117)
~[drill-java-exec-0.7.0-incubating-SNAPSHOT-rebuffed.jar:0.7.0-incubating-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:85)
~[drill-java-exec-0.7.0-incubating-SNAPSHOT-rebuffed.jar:0.7.0-incubating-SNAPSHOT]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:75)
~[drill-java-exec-0.7.0-incubating-SNAPSHOT-rebuffed.jar:0.7.0-incubating-SNAPSHOT]
at
[jira] [Updated] (DRILL-1899) Consider the read preference set by users
[ https://issues.apache.org/jira/browse/DRILL-1899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] B Anil Kumar updated DRILL-1899: Priority: Major (was: Minor) Consider the read preference set by users - Key: DRILL-1899 URL: https://issues.apache.org/jira/browse/DRILL-1899 Project: Apache Drill Issue Type: Bug Components: Storage - MongoDB Reporter: B Anil Kumar Fix For: 0.8.0 Attachments: 0001-Drill-1899.patch As of now, Drillbits read data from nearest mongod. But users may set read preference through client uri. Consider the user's read preference. However, if the users want data locality, users have to set read preference as nearest. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-1971) Mongo Storage Plugin slowness issue
[ https://issues.apache.org/jira/browse/DRILL-1971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] B Anil Kumar updated DRILL-1971: Attachment: 0001-DRILL-1971.patch Mongo Storage Plugin slowness issue --- Key: DRILL-1971 URL: https://issues.apache.org/jira/browse/DRILL-1971 Project: Apache Drill Issue Type: Bug Components: Storage - MongoDB Affects Versions: 0.8.0 Reporter: B Anil Kumar Assignee: B Anil Kumar Priority: Blocker Attachments: 0001-DRILL-1971.patch As part of [DRILL-1774|https://issues.apache.org/jira/browse/DRILL-1774], MongoRecordReader got updated. This has caused MongoRecordReader to break the *for loop* after reading every record, which caused slowness in Mongo Storage Plugin. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (DRILL-1971) Mongo Storage Plugin slowness issue
B Anil Kumar created DRILL-1971: --- Summary: Mongo Storage Plugin slowness issue Key: DRILL-1971 URL: https://issues.apache.org/jira/browse/DRILL-1971 Project: Apache Drill Issue Type: Bug Components: Storage - MongoDB Affects Versions: 0.8.0 Reporter: B Anil Kumar Assignee: B Anil Kumar Priority: Blocker As part of [DRILL-1774|https://issues.apache.org/jira/browse/DRILL-1774], MongoRecordReader got updated. This has caused MongoRecordReader to break the *for loop* after reading every record, which caused slowness in Mongo Storage Plugin. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
