[jira] [Commented] (KYLIN-4491) Query push down to Presto
[ https://issues.apache.org/jira/browse/KYLIN-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17129513#comment-17129513 ] ASF GitHub Bot commented on KYLIN-4491: --- hit-lacus merged pull request #1211: URL: https://github.com/apache/kylin/pull/1211 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Query push down to Presto > - > > Key: KYLIN-4491 > URL: https://issues.apache.org/jira/browse/KYLIN-4491 > Project: Kylin > Issue Type: New Feature > Components: Integration, RDBMS Source >Reporter: Shao Feng Shi >Assignee: fanfanAlice >Priority: Major > Fix For: v3.1.0 > > > With the data source SDK, we can push down a query to Presto instead of Hive > or Spark to gain better performance; With the SDK, we can also convert > Kylin's SQL query to Presto SQL. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4491) Query push down to Presto
[ https://issues.apache.org/jira/browse/KYLIN-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128766#comment-17128766 ] ASF subversion and git services commented on KYLIN-4491: Commit 9038428408ed4dd918362a408ff569a59121d897 in kylin's branch refs/heads/document from XiaoxiangYu [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=9038428 ] KYLIN-4491 Doc for push down to Presto > Query push down to Presto > - > > Key: KYLIN-4491 > URL: https://issues.apache.org/jira/browse/KYLIN-4491 > Project: Kylin > Issue Type: New Feature > Components: Integration, RDBMS Source >Reporter: Shao Feng Shi >Assignee: fanfanAlice >Priority: Major > Fix For: v3.1.0 > > > With the data source SDK, we can push down a query to Presto instead of Hive > or Spark to gain better performance; With the SDK, we can also convert > Kylin's SQL query to Presto SQL. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4491) Query push down to Presto
[ https://issues.apache.org/jira/browse/KYLIN-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125985#comment-17125985 ] Xiaoxiang Yu commented on KYLIN-4491: - Merge in https://github.com/apache/kylin/pull/1209 > Query push down to Presto > - > > Key: KYLIN-4491 > URL: https://issues.apache.org/jira/browse/KYLIN-4491 > Project: Kylin > Issue Type: New Feature > Components: Integration, RDBMS Source >Reporter: Shao Feng Shi >Assignee: fanfanAlice >Priority: Major > Fix For: v3.1.0 > > > With the data source SDK, we can push down a query to Presto instead of Hive > or Spark to gain better performance; With the SDK, we can also convert > Kylin's SQL query to Presto SQL. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4491) Query push down to Presto
[ https://issues.apache.org/jira/browse/KYLIN-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125976#comment-17125976 ] Xiaoxiang Yu commented on KYLIN-4491: - https://github.com/apache/kylin/pull/1209 > Query push down to Presto > - > > Key: KYLIN-4491 > URL: https://issues.apache.org/jira/browse/KYLIN-4491 > Project: Kylin > Issue Type: New Feature > Components: Integration, RDBMS Source >Reporter: Shao Feng Shi >Assignee: fanfanAlice >Priority: Major > Fix For: v3.1.0 > > > With the data source SDK, we can push down a query to Presto instead of Hive > or Spark to gain better performance; With the SDK, we can also convert > Kylin's SQL query to Presto SQL. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4491) Query push down to Presto
[ https://issues.apache.org/jira/browse/KYLIN-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125964#comment-17125964 ] ASF subversion and git services commented on KYLIN-4491: Commit cef10935b101d7aaf642ac3c62c196af523f512b in kylin's branch refs/heads/master from fanfanAlice [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=cef1093 ] KYLIN-4491 Support presto pushdown (#1209) * KYLIN-4491 Support presto pushdown * KYLIN-4491 add function to default.xml * KYLIN-4491 modify default.xml config * KYLIN-4491 modify getJdbcConnector config Co-authored-by: fanfanAlice <18611532...@163.com> > Query push down to Presto > - > > Key: KYLIN-4491 > URL: https://issues.apache.org/jira/browse/KYLIN-4491 > Project: Kylin > Issue Type: New Feature > Components: Integration, RDBMS Source >Reporter: Shao Feng Shi >Assignee: fanfanAlice >Priority: Major > Fix For: v3.1.0 > > > With the data source SDK, we can push down a query to Presto instead of Hive > or Spark to gain better performance; With the SDK, we can also convert > Kylin's SQL query to Presto SQL. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4491) Query push down to Presto
[ https://issues.apache.org/jira/browse/KYLIN-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125965#comment-17125965 ] ASF subversion and git services commented on KYLIN-4491: Commit cef10935b101d7aaf642ac3c62c196af523f512b in kylin's branch refs/heads/master from fanfanAlice [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=cef1093 ] KYLIN-4491 Support presto pushdown (#1209) * KYLIN-4491 Support presto pushdown * KYLIN-4491 add function to default.xml * KYLIN-4491 modify default.xml config * KYLIN-4491 modify getJdbcConnector config Co-authored-by: fanfanAlice <18611532...@163.com> > Query push down to Presto > - > > Key: KYLIN-4491 > URL: https://issues.apache.org/jira/browse/KYLIN-4491 > Project: Kylin > Issue Type: New Feature > Components: Integration, RDBMS Source >Reporter: Shao Feng Shi >Assignee: fanfanAlice >Priority: Major > Fix For: v3.1.0 > > > With the data source SDK, we can push down a query to Presto instead of Hive > or Spark to gain better performance; With the SDK, we can also convert > Kylin's SQL query to Presto SQL. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4491) Query push down to Presto
[ https://issues.apache.org/jira/browse/KYLIN-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125963#comment-17125963 ] ASF subversion and git services commented on KYLIN-4491: Commit cef10935b101d7aaf642ac3c62c196af523f512b in kylin's branch refs/heads/master from fanfanAlice [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=cef1093 ] KYLIN-4491 Support presto pushdown (#1209) * KYLIN-4491 Support presto pushdown * KYLIN-4491 add function to default.xml * KYLIN-4491 modify default.xml config * KYLIN-4491 modify getJdbcConnector config Co-authored-by: fanfanAlice <18611532...@163.com> > Query push down to Presto > - > > Key: KYLIN-4491 > URL: https://issues.apache.org/jira/browse/KYLIN-4491 > Project: Kylin > Issue Type: New Feature > Components: Integration, RDBMS Source >Reporter: Shao Feng Shi >Assignee: fanfanAlice >Priority: Major > Fix For: v3.1.0 > > > With the data source SDK, we can push down a query to Presto instead of Hive > or Spark to gain better performance; With the SDK, we can also convert > Kylin's SQL query to Presto SQL. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4491) Query push down to Presto
[ https://issues.apache.org/jira/browse/KYLIN-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125960#comment-17125960 ] ASF GitHub Bot commented on KYLIN-4491: --- shaofengshi merged pull request #1209: URL: https://github.com/apache/kylin/pull/1209 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Query push down to Presto > - > > Key: KYLIN-4491 > URL: https://issues.apache.org/jira/browse/KYLIN-4491 > Project: Kylin > Issue Type: New Feature > Components: Integration, RDBMS Source >Reporter: Shao Feng Shi >Assignee: fanfanAlice >Priority: Major > Fix For: v3.1.0 > > > With the data source SDK, we can push down a query to Presto instead of Hive > or Spark to gain better performance; With the SDK, we can also convert > Kylin's SQL query to Presto SQL. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4491) Query push down to Presto
[ https://issues.apache.org/jira/browse/KYLIN-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125961#comment-17125961 ] ASF subversion and git services commented on KYLIN-4491: Commit cef10935b101d7aaf642ac3c62c196af523f512b in kylin's branch refs/heads/master from fanfanAlice [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=cef1093 ] KYLIN-4491 Support presto pushdown (#1209) * KYLIN-4491 Support presto pushdown * KYLIN-4491 add function to default.xml * KYLIN-4491 modify default.xml config * KYLIN-4491 modify getJdbcConnector config Co-authored-by: fanfanAlice <18611532...@163.com> > Query push down to Presto > - > > Key: KYLIN-4491 > URL: https://issues.apache.org/jira/browse/KYLIN-4491 > Project: Kylin > Issue Type: New Feature > Components: Integration, RDBMS Source >Reporter: Shao Feng Shi >Assignee: fanfanAlice >Priority: Major > Fix For: v3.1.0 > > > With the data source SDK, we can push down a query to Presto instead of Hive > or Spark to gain better performance; With the SDK, we can also convert > Kylin's SQL query to Presto SQL. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4491) Query push down to Presto
[ https://issues.apache.org/jira/browse/KYLIN-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125962#comment-17125962 ] ASF subversion and git services commented on KYLIN-4491: Commit cef10935b101d7aaf642ac3c62c196af523f512b in kylin's branch refs/heads/master from fanfanAlice [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=cef1093 ] KYLIN-4491 Support presto pushdown (#1209) * KYLIN-4491 Support presto pushdown * KYLIN-4491 add function to default.xml * KYLIN-4491 modify default.xml config * KYLIN-4491 modify getJdbcConnector config Co-authored-by: fanfanAlice <18611532...@163.com> > Query push down to Presto > - > > Key: KYLIN-4491 > URL: https://issues.apache.org/jira/browse/KYLIN-4491 > Project: Kylin > Issue Type: New Feature > Components: Integration, RDBMS Source >Reporter: Shao Feng Shi >Assignee: fanfanAlice >Priority: Major > Fix For: v3.1.0 > > > With the data source SDK, we can push down a query to Presto instead of Hive > or Spark to gain better performance; With the SDK, we can also convert > Kylin's SQL query to Presto SQL. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4491) Query push down to Presto
[ https://issues.apache.org/jira/browse/KYLIN-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123451#comment-17123451 ] ASF GitHub Bot commented on KYLIN-4491: --- coveralls commented on pull request #1209: URL: https://github.com/apache/kylin/pull/1209#issuecomment-637353838 ## Pull Request Test Coverage Report for [Build 5962](https://coveralls.io/builds/31178926) * **4** of **113** **(3.54%)** changed or added relevant lines in **5** files are covered. * **4366** unchanged lines in **84** files lost coverage. * Overall coverage increased (+**0.9%**) to **28.268%** --- | Changes Missing Coverage | Covered Lines | Changed/Added Lines | % | | :-|--||---: | | [datasource-sdk/src/main/java/org/apache/kylin/sdk/datasource/framework/SourceConnectorFactory.java](https://coveralls.io/builds/31178926/source?filename=datasource-sdk%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fsdk%2Fdatasource%2Fframework%2FSourceConnectorFactory.java#L58) | 4 | 5 | 80.0% | [query/src/main/java/org/apache/kylin/query/util/PushDownUtil.java](https://coveralls.io/builds/31178926/source?filename=query%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fquery%2Futil%2FPushDownUtil.java#L62) | 0 | 2 | 0.0% | [server-base/src/main/java/org/apache/kylin/rest/service/QueryService.java](https://coveralls.io/builds/31178926/source?filename=server-base%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Frest%2Fservice%2FQueryService.java#L1025) | 0 | 2 | 0.0% | [query/src/main/java/org/apache/kylin/query/util/PushDownExecutor.java](https://coveralls.io/builds/31178926/source?filename=query%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fquery%2Futil%2FPushDownExecutor.java#L54) | 0 | 4 | 0.0% | [datasource-sdk/src/main/java/org/apache/kylin/sdk/datasource/adaptor/PrestoAdaptor.java](https://coveralls.io/builds/31178926/source?filename=datasource-sdk%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fsdk%2Fdatasource%2Fadaptor%2FPrestoAdaptor.java#L33) | 0 | 100 | 0.0% | Files with Coverage Reduction | New Missed Lines | % | | :-|--|--: | | [cache/src/main/java/org/apache/kylin/cache/cachemanager/RemoteLocalFailOverCacheManager.java](https://coveralls.io/builds/31178926/source?filename=cache%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcache%2Fcachemanager%2FRemoteLocalFailOverCacheManager.java#L74) | 1 | 85.71% | | [engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/CalculateStatsFromBaseCuboidMapper.java](https://coveralls.io/builds/31178926/source?filename=engine-mr%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fengine%2Fmr%2Fsteps%2FCalculateStatsFromBaseCuboidMapper.java#L150) | 1 | 95.35% | | [query/src/main/java/org/apache/kylin/query/util/PushDownExecutor.java](https://coveralls.io/builds/31178926/source?filename=query%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fquery%2Futil%2FPushDownExecutor.java#L60) | 1 | 0% | | [query/src/main/java/org/apache/kylin/query/util/PushDownUtil.java](https://coveralls.io/builds/31178926/source?filename=query%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fquery%2Futil%2FPushDownUtil.java#L63) | 1 | 85.37% | | [core-cube/src/main/java/org/apache/kylin/cube/cuboid/TreeCuboidScheduler.java](https://coveralls.io/builds/31178926/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Fcuboid%2FTreeCuboidScheduler.java#L124) | 2 | 68.46% | | [core-metadata/src/main/java/org/apache/kylin/metadata/expression/ExpressionCountDistributor.java](https://coveralls.io/builds/31178926/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmetadata%2Fexpression%2FExpressionCountDistributor.java#L87) | 2 | 94.44% | | [core-cube/src/main/java/org/apache/kylin/cube/CubeDescManager.java](https://coveralls.io/builds/31178926/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2FCubeDescManager.java#L302) | 3 | 38.57% | | [source-hive/src/main/java/org/apache/kylin/source/hive/HiveTableMeta.java](https://coveralls.io/builds/31178926/source?filename=source-hive%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fsource%2Fhive%2FHiveTableMeta.java#L70) | 3 | 0% | | [core-cube/src/main/java/org/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://coveralls.io/builds/31178926/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Finmemcubing%2FMemDiskStore.java#L439) | 5 | 77.81% | | [core-metadata/src/main/java/org/apache/kylin/measure/bitmap/BitmapMeasureType.java](https://coveralls.io/builds/31178926/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmeasure%2Fbitmap%2FBitmapMeasureType.java#L160) | 5 | 18.52% | | Totals | [](https://coveralls.io/builds/31178926) | | :-- | --: | | Change from base [Build
[jira] [Commented] (KYLIN-4491) Query push down to Presto
[ https://issues.apache.org/jira/browse/KYLIN-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123443#comment-17123443 ] ASF GitHub Bot commented on KYLIN-4491: --- codecov-commenter commented on pull request #1209: URL: https://github.com/apache/kylin/pull/1209#issuecomment-637349643 # [Codecov](https://codecov.io/gh/apache/kylin/pull/1209?src=pr=h1) Report > Merging [#1209](https://codecov.io/gh/apache/kylin/pull/1209?src=pr=desc) into [master](https://codecov.io/gh/apache/kylin/commit/a17419409199fc2fca3116718ce658040e0d6bbf=desc) will **increase** coverage by `0.86%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/kylin/pull/1209?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#1209 +/- ## + Coverage 24.87% 25.74% +0.86% - Complexity 6264 6624 +360 Files 1453 1483 +30 Lines 8912091053+1933 Branches 1243812701 +263 + Hits 2217223441+1269 - Misses6477165317 +546 - Partials 2177 2295 +118 ``` | [Impacted Files](https://codecov.io/gh/apache/kylin/pull/1209?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...he/kylin/sdk/datasource/adaptor/PrestoAdaptor.java](https://codecov.io/gh/apache/kylin/pull/1209/diff?src=pr=tree#diff-ZGF0YXNvdXJjZS1zZGsvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3Nkay9kYXRhc291cmNlL2FkYXB0b3IvUHJlc3RvQWRhcHRvci5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | | | [...k/datasource/framework/SourceConnectorFactory.java](https://codecov.io/gh/apache/kylin/pull/1209/diff?src=pr=tree#diff-ZGF0YXNvdXJjZS1zZGsvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3Nkay9kYXRhc291cmNlL2ZyYW1ld29yay9Tb3VyY2VDb25uZWN0b3JGYWN0b3J5LmphdmE=) | `33.33% <0.00%> (-21.67%)` | `1.00 <1.00> (ø)` | | | [.../org/apache/kylin/query/util/PushDownExecutor.java](https://codecov.io/gh/apache/kylin/pull/1209/diff?src=pr=tree#diff-cXVlcnkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3F1ZXJ5L3V0aWwvUHVzaERvd25FeGVjdXRvci5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (ø)` | | | [...java/org/apache/kylin/query/util/PushDownUtil.java](https://codecov.io/gh/apache/kylin/pull/1209/diff?src=pr=tree#diff-cXVlcnkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3F1ZXJ5L3V0aWwvUHVzaERvd25VdGlsLmphdmE=) | `80.48% <0.00%> (ø)` | `4.00 <0.00> (ø)` | | | [...va/org/apache/kylin/rest/service/QueryService.java](https://codecov.io/gh/apache/kylin/pull/1209/diff?src=pr=tree#diff-c2VydmVyLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3Jlc3Qvc2VydmljZS9RdWVyeVNlcnZpY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (ø)` | | | [...e/kylin/storage/gtrecord/CubeScanRangePlanner.java](https://codecov.io/gh/apache/kylin/pull/1209/diff?src=pr=tree#diff-Y29yZS1zdG9yYWdlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9zdG9yYWdlL2d0cmVjb3JkL0N1YmVTY2FuUmFuZ2VQbGFubmVyLmphdmE=) | `43.86% <0.00%> (-7.24%)` | `26.00% <0.00%> (ø%)` | | | [.../cachemanager/RemoteLocalFailOverCacheManager.java](https://codecov.io/gh/apache/kylin/pull/1209/diff?src=pr=tree#diff-Y2FjaGUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2NhY2hlL2NhY2hlbWFuYWdlci9SZW1vdGVMb2NhbEZhaWxPdmVyQ2FjaGVNYW5hZ2VyLmphdmE=) | `78.57% <0.00%> (-6.05%)` | `7.00% <0.00%> (ø%)` | | | [.../apache/kylin/cube/cuboid/TreeCuboidScheduler.java](https://codecov.io/gh/apache/kylin/pull/1209/diff?src=pr=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL2N1Ym9pZC9UcmVlQ3Vib2lkU2NoZWR1bGVyLmphdmE=) | `63.84% <0.00%> (-2.31%)` | `0.00% <0.00%> (ø%)` | | | [...g/apache/kylin/gridtable/GTScanRequestBuilder.java](https://codecov.io/gh/apache/kylin/pull/1209/diff?src=pr=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9ncmlkdGFibGUvR1RTY2FuUmVxdWVzdEJ1aWxkZXIuamF2YQ==) | `56.16% <0.00%> (-1.81%)` | `13.00% <0.00%> (ø%)` | | | [.../apache/kylin/measure/bitmap/BitmapSerializer.java](https://codecov.io/gh/apache/kylin/pull/1209/diff?src=pr=tree#diff-Y29yZS1tZXRhZGF0YS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUva3lsaW4vbWVhc3VyZS9iaXRtYXAvQml0bWFwU2VyaWFsaXplci5qYXZh) | `27.77% <0.00%> (-1.64%)` | `5.00% <0.00%> (ø%)` | | | ... and [156 more](https://codecov.io/gh/apache/kylin/pull/1209/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/kylin/pull/1209?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by
[jira] [Commented] (KYLIN-4491) Query push down to Presto
[
https://issues.apache.org/jira/browse/KYLIN-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17109251#comment-17109251
]
ASF GitHub Bot commented on KYLIN-4491:
---
hit-lacus commented on a change in pull request #1209:
URL: https://github.com/apache/kylin/pull/1209#discussion_r426201246
##
File path:
query/src/main/java/org/apache/kylin/query/pushdown/PushDownRunnerOtherImpl.java
##
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kylin.query.pushdown;
+
+import org.apache.kylin.common.KylinConfig;
+import org.apache.kylin.common.util.DBUtils;
+import org.apache.kylin.metadata.querymeta.SelectedColumnMeta;
+import org.apache.kylin.sdk.datasource.framework.JdbcConnector;
+import org.apache.kylin.sdk.datasource.framework.SourceConnectorFactory;
+import org.apache.kylin.source.adhocquery.AbstractPushdownRunner;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.Connection;
+import java.sql.ResultSet;
+import java.sql.ResultSetMetaData;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.LinkedList;
+import java.util.List;
+
+public class PushDownRunnerOtherImpl extends AbstractPushdownRunner {
Review comment:
The name of `PushDownRunnerOtherImpl` is not well defined. Since it is
the the advanced version of elder `PushDownRunnerJdbcImpl`, and reuse some
logic in Data Source SDK. Maybe we change is name to something like
`PushDownRunnerJdbcAdvencedImpl`? What do you think @shaofengshi ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Query push down to Presto
> -
>
> Key: KYLIN-4491
> URL: https://issues.apache.org/jira/browse/KYLIN-4491
> Project: Kylin
> Issue Type: New Feature
> Components: Integration, RDBMS Source
>Reporter: Shao Feng Shi
>Assignee: fanfanAlice
>Priority: Major
> Fix For: v3.1.0
>
>
> With the data source SDK, we can push down a query to Presto instead of Hive
> or Spark to gain better performance; With the SDK, we can also convert
> Kylin's SQL query to Presto SQL.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-4491) Query push down to Presto
[
https://issues.apache.org/jira/browse/KYLIN-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17109252#comment-17109252
]
ASF GitHub Bot commented on KYLIN-4491:
---
hit-lacus commented on a change in pull request #1209:
URL: https://github.com/apache/kylin/pull/1209#discussion_r426201246
##
File path:
query/src/main/java/org/apache/kylin/query/pushdown/PushDownRunnerOtherImpl.java
##
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kylin.query.pushdown;
+
+import org.apache.kylin.common.KylinConfig;
+import org.apache.kylin.common.util.DBUtils;
+import org.apache.kylin.metadata.querymeta.SelectedColumnMeta;
+import org.apache.kylin.sdk.datasource.framework.JdbcConnector;
+import org.apache.kylin.sdk.datasource.framework.SourceConnectorFactory;
+import org.apache.kylin.source.adhocquery.AbstractPushdownRunner;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.Connection;
+import java.sql.ResultSet;
+import java.sql.ResultSetMetaData;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.LinkedList;
+import java.util.List;
+
+public class PushDownRunnerOtherImpl extends AbstractPushdownRunner {
Review comment:
The name of `PushDownRunnerOtherImpl` is not well defined. Since it is
the the advanced version of elder `PushDownRunnerJdbcImpl`, and reuse some
logic in Data Source SDK. Maybe we change is name to something like
`PushDownRunnerJdbcAdvancedImpl`? What do you think @shaofengshi ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Query push down to Presto
> -
>
> Key: KYLIN-4491
> URL: https://issues.apache.org/jira/browse/KYLIN-4491
> Project: Kylin
> Issue Type: New Feature
> Components: Integration, RDBMS Source
>Reporter: Shao Feng Shi
>Assignee: fanfanAlice
>Priority: Major
> Fix For: v3.1.0
>
>
> With the data source SDK, we can push down a query to Presto instead of Hive
> or Spark to gain better performance; With the SDK, we can also convert
> Kylin's SQL query to Presto SQL.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-4491) Query push down to Presto
[ https://issues.apache.org/jira/browse/KYLIN-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17107985#comment-17107985 ] ASF GitHub Bot commented on KYLIN-4491: --- hit-lacus commented on a change in pull request #1209: URL: https://github.com/apache/kylin/pull/1209#discussion_r425589938 ## File path: datasource-sdk/src/main/resources/datasource/default.xml ## @@ -170,6 +170,34 @@ + + Review comment: Leave it be. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Query push down to Presto > - > > Key: KYLIN-4491 > URL: https://issues.apache.org/jira/browse/KYLIN-4491 > Project: Kylin > Issue Type: New Feature > Components: Integration, RDBMS Source >Reporter: Shao Feng Shi >Assignee: fanfanAlice >Priority: Major > Fix For: v3.1.0 > > > With the data source SDK, we can push down a query to Presto instead of Hive > or Spark to gain better performance; With the SDK, we can also convert > Kylin's SQL query to Presto SQL. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4491) Query push down to Presto
[
https://issues.apache.org/jira/browse/KYLIN-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17107984#comment-17107984
]
ASF GitHub Bot commented on KYLIN-4491:
---
hit-lacus commented on a change in pull request #1209:
URL: https://github.com/apache/kylin/pull/1209#discussion_r425589828
##
File path:
datasource-sdk/src/main/java/org/apache/kylin/sdk/datasource/adaptor/PrestoAdaptor.java
##
@@ -0,0 +1,211 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kylin.sdk.datasource.adaptor;
+
+import java.sql.Connection;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Types;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import javax.sql.rowset.CachedRowSet;
+
+public class PrestoAdaptor extends DefaultAdaptor {
+
+private Pattern patternASYM = Pattern.compile("BETWEEN(\\s*)ASYMMETRIC");
+private Pattern patternSYM = Pattern.compile("BETWEEN(\\s*)SYMMETRIC");
+private Pattern patternTrim =
Pattern.compile("TRIM\\(.*BOTH.*FROM\\s+(.+)\\)");
+private Pattern patternOffset = Pattern.compile("(?i)OFFSET\\s\\d+");
+
+public PrestoAdaptor(AdaptorConfig config) throws Exception {
+super(config);
+}
+
+@Override
+public String fixSql(String sql) {
+sql = resolveBetweenAsymmetricSymmetric(sql);
+sql = convertTrim(sql);
+sql = convertOffset(sql);
+return sql;
+}
+
+@Override
+public int toKylinTypeId(String type, int typeId) {
+if (2000 == typeId) {
Review comment:
Looks like it is presto's specific behaviour of mapping `DECIMAL` to
`JAVA_OBJECT`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Query push down to Presto
> -
>
> Key: KYLIN-4491
> URL: https://issues.apache.org/jira/browse/KYLIN-4491
> Project: Kylin
> Issue Type: New Feature
> Components: Integration, RDBMS Source
>Reporter: Shao Feng Shi
>Assignee: fanfanAlice
>Priority: Major
> Fix For: v3.1.0
>
>
> With the data source SDK, we can push down a query to Presto instead of Hive
> or Spark to gain better performance; With the SDK, we can also convert
> Kylin's SQL query to Presto SQL.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-4491) Query push down to Presto
[ https://issues.apache.org/jira/browse/KYLIN-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17107879#comment-17107879 ] ASF GitHub Bot commented on KYLIN-4491: --- hit-lacus opened a new pull request #1211: URL: https://github.com/apache/kylin/pull/1211 ## Proposed changes Describe the big picture of your changes here to communicate to the maintainers why we should accept this pull request. If it fixes a bug or resolves a feature request, be sure to link to that issue. ## Types of changes What types of changes does your code introduce to Kylin? _Put an `x` in the boxes that apply_ - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation Update (if none of the other choices apply) ## Checklist _Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._ - [ ] I have create an issue on [Kylin's jira](https://issues.apache.org/jira/browse/KYLIN), and have described the bug/feature there in detail - [ ] Commit messages in my PR start with the related jira ID, like "KYLIN- Make Kylin project open-source" - [ ] Compiling and unit tests pass locally with my changes - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] If this change need a document change, I will prepare another pr against the `document` branch - [ ] Any dependent changes have been merged ## Further comments If this is a relatively large or complex change, kick off the discussion at user@kylin or dev@kylin by explaining why you chose the solution you did and what alternatives you considered, etc... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Query push down to Presto > - > > Key: KYLIN-4491 > URL: https://issues.apache.org/jira/browse/KYLIN-4491 > Project: Kylin > Issue Type: New Feature > Components: Integration, RDBMS Source >Reporter: Shao Feng Shi >Priority: Major > Fix For: v3.1.0 > > > With the data source SDK, we can push down a query to Presto instead of Hive > or Spark to gain better performance; With the SDK, we can also convert > Kylin's SQL query to Presto SQL. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4491) Query push down to Presto
[
https://issues.apache.org/jira/browse/KYLIN-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17107427#comment-17107427
]
ASF GitHub Bot commented on KYLIN-4491:
---
hit-lacus commented on a change in pull request #1209:
URL: https://github.com/apache/kylin/pull/1209#discussion_r425240706
##
File path:
datasource-sdk/src/main/java/org/apache/kylin/sdk/datasource/adaptor/PrestoAdaptor.java
##
@@ -0,0 +1,211 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kylin.sdk.datasource.adaptor;
+
+import java.sql.Connection;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Types;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import javax.sql.rowset.CachedRowSet;
+
+public class PrestoAdaptor extends DefaultAdaptor {
+
+private Pattern patternASYM = Pattern.compile("BETWEEN(\\s*)ASYMMETRIC");
+private Pattern patternSYM = Pattern.compile("BETWEEN(\\s*)SYMMETRIC");
+private Pattern patternTrim =
Pattern.compile("TRIM\\(.*BOTH.*FROM\\s+(.+)\\)");
+private Pattern patternOffset = Pattern.compile("(?i)OFFSET\\s\\d+");
+
+public PrestoAdaptor(AdaptorConfig config) throws Exception {
+super(config);
+}
+
+@Override
+public String fixSql(String sql) {
+sql = resolveBetweenAsymmetricSymmetric(sql);
+sql = convertTrim(sql);
+sql = convertOffset(sql);
+return sql;
+}
+
+@Override
+public int toKylinTypeId(String type, int typeId) {
+if (2000 == typeId) {
Review comment:
Could you please kindly explain the magic number `2000` means? I check
the [JDK7 Constant
value](https://docs.oracle.com/javase/7/docs/api/constant-values.html#java.sql.Types.JAVA_OBJECT),
2000 is refer to `JAVA_OBJECT`, not `DECIMAL`, am I right?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Query push down to Presto
> -
>
> Key: KYLIN-4491
> URL: https://issues.apache.org/jira/browse/KYLIN-4491
> Project: Kylin
> Issue Type: New Feature
> Components: Integration, RDBMS Source
>Reporter: Shao Feng Shi
>Priority: Major
> Fix For: v3.1.0
>
>
> With the data source SDK, we can push down a query to Presto instead of Hive
> or Spark to gain better performance; With the SDK, we can also convert
> Kylin's SQL query to Presto SQL.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-4491) Query push down to Presto
[
https://issues.apache.org/jira/browse/KYLIN-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17107424#comment-17107424
]
ASF GitHub Bot commented on KYLIN-4491:
---
hit-lacus commented on a change in pull request #1209:
URL: https://github.com/apache/kylin/pull/1209#discussion_r425240706
##
File path:
datasource-sdk/src/main/java/org/apache/kylin/sdk/datasource/adaptor/PrestoAdaptor.java
##
@@ -0,0 +1,211 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kylin.sdk.datasource.adaptor;
+
+import java.sql.Connection;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Types;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import javax.sql.rowset.CachedRowSet;
+
+public class PrestoAdaptor extends DefaultAdaptor {
+
+private Pattern patternASYM = Pattern.compile("BETWEEN(\\s*)ASYMMETRIC");
+private Pattern patternSYM = Pattern.compile("BETWEEN(\\s*)SYMMETRIC");
+private Pattern patternTrim =
Pattern.compile("TRIM\\(.*BOTH.*FROM\\s+(.+)\\)");
+private Pattern patternOffset = Pattern.compile("(?i)OFFSET\\s\\d+");
+
+public PrestoAdaptor(AdaptorConfig config) throws Exception {
+super(config);
+}
+
+@Override
+public String fixSql(String sql) {
+sql = resolveBetweenAsymmetricSymmetric(sql);
+sql = convertTrim(sql);
+sql = convertOffset(sql);
+return sql;
+}
+
+@Override
+public int toKylinTypeId(String type, int typeId) {
+if (2000 == typeId) {
Review comment:
Could you please kindly explain the magic number `2000` means? I check
the [JDK7 Constant
value](https://docs.oracle.com/javase/7/docs/api/constant-values.html), 2000 is
refer to `JAVA_OBJECT`, am I right?
##
File path: datasource-sdk/src/main/resources/datasource/default.xml
##
@@ -170,6 +170,34 @@
+
+
Review comment:
Should we start the `ID` from 84 other than 92?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Query push down to Presto
> -
>
> Key: KYLIN-4491
> URL: https://issues.apache.org/jira/browse/KYLIN-4491
> Project: Kylin
> Issue Type: New Feature
> Components: Integration, RDBMS Source
>Reporter: Shao Feng Shi
>Priority: Major
> Fix For: v3.1.0
>
>
> With the data source SDK, we can push down a query to Presto instead of Hive
> or Spark to gain better performance; With the SDK, we can also convert
> Kylin's SQL query to Presto SQL.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-4491) Query push down to Presto
[ https://issues.apache.org/jira/browse/KYLIN-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17107158#comment-17107158 ] ASF GitHub Bot commented on KYLIN-4491: --- codecov-io commented on pull request #1209: URL: https://github.com/apache/kylin/pull/1209#issuecomment-628525850 # [Codecov](https://codecov.io/gh/apache/kylin/pull/1209?src=pr=h1) Report > Merging [#1209](https://codecov.io/gh/apache/kylin/pull/1209?src=pr=desc) into [master](https://codecov.io/gh/apache/kylin/commit/a17419409199fc2fca3116718ce658040e0d6bbf=desc) will **decrease** coverage by `0.03%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/kylin/pull/1209?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#1209 +/- ## - Coverage 24.87% 24.83% -0.04% - Complexity 6264 6266 +2 Files 1453 1455 +2 Lines 8912089278 +158 Branches 1243812454 +16 + Hits 2217222176 +4 - Misses6477164927 +156 + Partials 2177 2175 -2 ``` | [Impacted Files](https://codecov.io/gh/apache/kylin/pull/1209?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...he/kylin/sdk/datasource/adaptor/PrestoAdaptor.java](https://codecov.io/gh/apache/kylin/pull/1209/diff?src=pr=tree#diff-ZGF0YXNvdXJjZS1zZGsvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3Nkay9kYXRhc291cmNlL2FkYXB0b3IvUHJlc3RvQWRhcHRvci5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | | | [...k/datasource/framework/SourceConnectorFactory.java](https://codecov.io/gh/apache/kylin/pull/1209/diff?src=pr=tree#diff-ZGF0YXNvdXJjZS1zZGsvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3Nkay9kYXRhc291cmNlL2ZyYW1ld29yay9Tb3VyY2VDb25uZWN0b3JGYWN0b3J5LmphdmE=) | `30.55% <0.00%> (-24.45%)` | `1.00 <0.00> (ø)` | | | [.../kylin/query/pushdown/PushDownRunnerOtherImpl.java](https://codecov.io/gh/apache/kylin/pull/1209/diff?src=pr=tree#diff-cXVlcnkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3F1ZXJ5L3B1c2hkb3duL1B1c2hEb3duUnVubmVyT3RoZXJJbXBsLmphdmE=) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | | | [.../org/apache/kylin/query/util/PushDownExecutor.java](https://codecov.io/gh/apache/kylin/pull/1209/diff?src=pr=tree#diff-cXVlcnkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3F1ZXJ5L3V0aWwvUHVzaERvd25FeGVjdXRvci5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (ø)` | | | [...java/org/apache/kylin/query/util/PushDownUtil.java](https://codecov.io/gh/apache/kylin/pull/1209/diff?src=pr=tree#diff-cXVlcnkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3F1ZXJ5L3V0aWwvUHVzaERvd25VdGlsLmphdmE=) | `80.48% <0.00%> (ø)` | `4.00 <0.00> (ø)` | | | [...va/org/apache/kylin/rest/service/QueryService.java](https://codecov.io/gh/apache/kylin/pull/1209/diff?src=pr=tree#diff-c2VydmVyLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3Jlc3Qvc2VydmljZS9RdWVyeVNlcnZpY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (ø)` | | | [.../apache/kylin/cube/cuboid/TreeCuboidScheduler.java](https://codecov.io/gh/apache/kylin/pull/1209/diff?src=pr=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL2N1Ym9pZC9UcmVlQ3Vib2lkU2NoZWR1bGVyLmphdmE=) | `63.84% <0.00%> (-2.31%)` | `0.00% <0.00%> (ø%)` | | | [...tasource/framework/utils/ExpressionComparator.java](https://codecov.io/gh/apache/kylin/pull/1209/diff?src=pr=tree#diff-ZGF0YXNvdXJjZS1zZGsvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3Nkay9kYXRhc291cmNlL2ZyYW1ld29yay91dGlscy9FeHByZXNzaW9uQ29tcGFyYXRvci5qYXZh) | `57.74% <0.00%> (+1.40%)` | `2.00% <0.00%> (ø%)` | | | ... and [3 more](https://codecov.io/gh/apache/kylin/pull/1209/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/kylin/pull/1209?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/kylin/pull/1209?src=pr=footer). Last update [a174194...9e95247](https://codecov.io/gh/apache/kylin/pull/1209?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Query push down to Presto >
[jira] [Commented] (KYLIN-4491) Query push down to Presto
[ https://issues.apache.org/jira/browse/KYLIN-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17107138#comment-17107138 ] ASF GitHub Bot commented on KYLIN-4491: --- hit-lacus commented on pull request #1209: URL: https://github.com/apache/kylin/pull/1209#issuecomment-628518289 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Query push down to Presto > - > > Key: KYLIN-4491 > URL: https://issues.apache.org/jira/browse/KYLIN-4491 > Project: Kylin > Issue Type: New Feature > Components: Integration, RDBMS Source >Reporter: Shao Feng Shi >Priority: Major > Fix For: v3.1.0 > > > With the data source SDK, we can push down a query to Presto instead of Hive > or Spark to gain better performance; With the SDK, we can also convert > Kylin's SQL query to Presto SQL. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4491) Query push down to Presto
[ https://issues.apache.org/jira/browse/KYLIN-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17107132#comment-17107132 ] ASF GitHub Bot commented on KYLIN-4491: --- hit-lacus edited a comment on pull request #1209: URL: https://github.com/apache/kylin/pull/1209#issuecomment-628438384 ## Configuration Modify project level configuration.  This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Query push down to Presto > - > > Key: KYLIN-4491 > URL: https://issues.apache.org/jira/browse/KYLIN-4491 > Project: Kylin > Issue Type: New Feature > Components: Integration, RDBMS Source >Reporter: Shao Feng Shi >Priority: Major > Fix For: v3.1.0 > > > With the data source SDK, we can push down a query to Presto instead of Hive > or Spark to gain better performance; With the SDK, we can also convert > Kylin's SQL query to Presto SQL. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4491) Query push down to Presto
[ https://issues.apache.org/jira/browse/KYLIN-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17107133#comment-17107133 ] ASF GitHub Bot commented on KYLIN-4491: --- hit-lacus edited a comment on pull request #1209: URL: https://github.com/apache/kylin/pull/1209#issuecomment-628438076 ## Test Prepare Put driver jar to `$KYLIN_HOME/ext` and restart Kylin instance. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Query push down to Presto > - > > Key: KYLIN-4491 > URL: https://issues.apache.org/jira/browse/KYLIN-4491 > Project: Kylin > Issue Type: New Feature > Components: Integration, RDBMS Source >Reporter: Shao Feng Shi >Priority: Major > Fix For: v3.1.0 > > > With the data source SDK, we can push down a query to Presto instead of Hive > or Spark to gain better performance; With the SDK, we can also convert > Kylin's SQL query to Presto SQL. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4491) Query push down to Presto
[ https://issues.apache.org/jira/browse/KYLIN-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17107131#comment-17107131 ] ASF GitHub Bot commented on KYLIN-4491: --- hit-lacus edited a comment on pull request #1209: URL: https://github.com/apache/kylin/pull/1209#issuecomment-628439230 ## Test Pushdown - Kylin Query https://user-images.githubusercontent.com/14030549/81917610-ffdf8e80-9607-11ea-82f4-4b95e0e3defe.png;> - Presto query history https://user-images.githubusercontent.com/14030549/81917733-2998b580-9608-11ea-9e37-874015707efb.png;> - kylin.log ``` 2020-05-14 05:19:37,244 INFO [http-bio-7225-exec-2] service.QueryService:377 : Check query permission in 0 ms. 2020-05-14 05:19:37,245 INFO [Query de5c3280-7ea0-b20c-2eff-ccb7dfe78bca-59] service.QueryService:418 : Using project: learn_kylin 2020-05-14 05:19:37,245 INFO [Query de5c3280-7ea0-b20c-2eff-ccb7dfe78bca-59] service.QueryService:419 : The original query: -- select count(*) as cnt from KYLIN_ACCOUNT SELECT DAYOFWEEK(KYLIN_CAL_DT.CAL_DT) as dow, count(*) as cnt FROM KYLIN_CAL_DT WHERE KYLIN_CAL_DT.DAY_OF_CAL_ID >=2 GROUP BY DAYOFWEEK(KYLIN_CAL_DT.CAL_DT) ORDER BY cnt desc 2020-05-14 05:19:37,254 INFO [Query de5c3280-7ea0-b20c-2eff-ccb7dfe78bca-59] service.QueryService:652 : The corrected query: select * from (SELECT DAYOFWEEK(KYLIN_CAL_DT.CAL_DT) as dow, count(*) as cnt FROM KYLIN_CAL_DT WHERE KYLIN_CAL_DT.DAY_OF_CAL_ID >=2 GROUP BY DAYOFWEEK(KYLIN_CAL_DT.CAL_DT) ORDER BY cnt desc) limit 5 2020-05-14 05:19:37,558 INFO [Query de5c3280-7ea0-b20c-2eff-ccb7dfe78bca-59] util.PushDownExecutor:71 : Query failed to utilize pre-calculation, routing to other engines java.sql.SQLException: Error while executing SQL "select * from (SELECT DAYOFWEEK(KYLIN_CAL_DT.CAL_DT) as dow, count(*) as cnt FROM KYLIN_CAL_DT WHERE KYLIN_CAL_DT.DAY_OF_CAL_ID >=2 GROUP BY DAYOFWEEK(KYLIN_CAL_DT.CAL_DT) ORDER BY cnt desc) limit 5": No model found for OLAPContext, CUBE_NOT_READY, rel#312:OLAPTableScan.OLAP.[](table=[DEFAULT, KYLIN_CAL_DT],ctx=,fields=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99]) at org.apache.calcite.avatica.Helper.createException(Helper.java:56) at org.apache.calcite.avatica.Helper.createException(Helper.java:41) at org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:163) at org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:227) at org.apache.kylin.rest.service.QueryService.executeRequest(QueryService.java:978) at org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:674) at org.apache.kylin.rest.service.QueryService.query(QueryService.java:219) at org.apache.kylin.rest.service.QueryService.queryAndUpdateCache(QueryService.java:484) at org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:444) at org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:378) at org.apache.kylin.rest.controller.QueryController.query(QueryController.java:93) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901) at
[jira] [Commented] (KYLIN-4491) Query push down to Presto
[ https://issues.apache.org/jira/browse/KYLIN-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17107127#comment-17107127 ] ASF GitHub Bot commented on KYLIN-4491: --- fanfanAlice commented on pull request #1209: URL: https://github.com/apache/kylin/pull/1209#issuecomment-628513092 Presto PushDown No matter what data source you use, you can specify the corresponding data source to push down the query kylin.query.pushdown.runner-class-name=org.apache.kylin.query.pushdown.PushDownRunnerOtherImpl kylin.source.jdbc.dialect=presto kylin.source.jdbc.adaptor=org.apache.kylin.sdk.datasource.adaptor.PrestoAdaptor kylin.query.pushdown.jdbc.url=jdbc:presto://10.1.1.70:8083/hive/default kylin.query.pushdown.jdbc.driver=com.facebook.presto.jdbc.PrestoDriver kylin.query.pushdown.jdbc.username=presto kylin.query.pushdown.jdbc.password= kylin.query.pushdown.runner-class-name=org.apache.kylin.query.pushdown.PushDownRunnerOtherImpl : whether to enable query pushdown kylin.source.jdbc.dialect=presto : specifies JDBC dialect. The default value is default kylin.source.jdbc.adaptor=org.apache.kylin.sdk.datasource.adaptor.PrestoAdaptor : specifies JDBC adaptor This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Query push down to Presto > - > > Key: KYLIN-4491 > URL: https://issues.apache.org/jira/browse/KYLIN-4491 > Project: Kylin > Issue Type: New Feature > Components: Integration, RDBMS Source >Reporter: Shao Feng Shi >Priority: Major > Fix For: v3.1.0 > > > With the data source SDK, we can push down a query to Presto instead of Hive > or Spark to gain better performance; With the SDK, we can also convert > Kylin's SQL query to Presto SQL. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4491) Query push down to Presto
[ https://issues.apache.org/jira/browse/KYLIN-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17107116#comment-17107116 ] ASF GitHub Bot commented on KYLIN-4491: --- hit-lacus edited a comment on pull request #1209: URL: https://github.com/apache/kylin/pull/1209#issuecomment-628438384 ## Configuration  This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Query push down to Presto > - > > Key: KYLIN-4491 > URL: https://issues.apache.org/jira/browse/KYLIN-4491 > Project: Kylin > Issue Type: New Feature > Components: Integration, RDBMS Source >Reporter: Shao Feng Shi >Priority: Major > Fix For: v3.1.0 > > > With the data source SDK, we can push down a query to Presto instead of Hive > or Spark to gain better performance; With the SDK, we can also convert > Kylin's SQL query to Presto SQL. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4491) Query push down to Presto
[ https://issues.apache.org/jira/browse/KYLIN-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17107117#comment-17107117 ] ASF GitHub Bot commented on KYLIN-4491: --- hit-lacus edited a comment on pull request #1209: URL: https://github.com/apache/kylin/pull/1209#issuecomment-628438076 ## Test Prepare Put driver jar to `$KYLIN_HOME/ext` . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Query push down to Presto > - > > Key: KYLIN-4491 > URL: https://issues.apache.org/jira/browse/KYLIN-4491 > Project: Kylin > Issue Type: New Feature > Components: Integration, RDBMS Source >Reporter: Shao Feng Shi >Priority: Major > Fix For: v3.1.0 > > > With the data source SDK, we can push down a query to Presto instead of Hive > or Spark to gain better performance; With the SDK, we can also convert > Kylin's SQL query to Presto SQL. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4491) Query push down to Presto
[ https://issues.apache.org/jira/browse/KYLIN-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17106998#comment-17106998 ] ASF GitHub Bot commented on KYLIN-4491: --- hit-lacus commented on pull request #1209: URL: https://github.com/apache/kylin/pull/1209#issuecomment-628439230 ## Test Pushdown This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Query push down to Presto > - > > Key: KYLIN-4491 > URL: https://issues.apache.org/jira/browse/KYLIN-4491 > Project: Kylin > Issue Type: New Feature > Components: Integration, RDBMS Source >Reporter: Shao Feng Shi >Priority: Major > Fix For: v3.1.0 > > > With the data source SDK, we can push down a query to Presto instead of Hive > or Spark to gain better performance; With the SDK, we can also convert > Kylin's SQL query to Presto SQL. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4491) Query push down to Presto
[ https://issues.apache.org/jira/browse/KYLIN-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17106995#comment-17106995 ] ASF GitHub Bot commented on KYLIN-4491: --- hit-lacus commented on pull request #1209: URL: https://github.com/apache/kylin/pull/1209#issuecomment-628438076 ## Test Prepare This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Query push down to Presto > - > > Key: KYLIN-4491 > URL: https://issues.apache.org/jira/browse/KYLIN-4491 > Project: Kylin > Issue Type: New Feature > Components: Integration, RDBMS Source >Reporter: Shao Feng Shi >Priority: Major > Fix For: v3.1.0 > > > With the data source SDK, we can push down a query to Presto instead of Hive > or Spark to gain better performance; With the SDK, we can also convert > Kylin's SQL query to Presto SQL. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4491) Query push down to Presto
[ https://issues.apache.org/jira/browse/KYLIN-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17106996#comment-17106996 ] ASF GitHub Bot commented on KYLIN-4491: --- hit-lacus commented on pull request #1209: URL: https://github.com/apache/kylin/pull/1209#issuecomment-628438384 ## Configuration This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Query push down to Presto > - > > Key: KYLIN-4491 > URL: https://issues.apache.org/jira/browse/KYLIN-4491 > Project: Kylin > Issue Type: New Feature > Components: Integration, RDBMS Source >Reporter: Shao Feng Shi >Priority: Major > Fix For: v3.1.0 > > > With the data source SDK, we can push down a query to Presto instead of Hive > or Spark to gain better performance; With the SDK, we can also convert > Kylin's SQL query to Presto SQL. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4491) Query push down to Presto
[ https://issues.apache.org/jira/browse/KYLIN-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17106970#comment-17106970 ] ASF GitHub Bot commented on KYLIN-4491: --- fanfanAlice opened a new pull request #1209: URL: https://github.com/apache/kylin/pull/1209 ## Proposed changes Describe the big picture of your changes here to communicate to the maintainers why we should accept this pull request. If it fixes a bug or resolves a feature request, be sure to link to that issue. ## Types of changes What types of changes does your code introduce to Kylin? _Put an `x` in the boxes that apply_ - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation Update (if none of the other choices apply) ## Checklist _Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._ - [ ] I have create an issue on [Kylin's jira](https://issues.apache.org/jira/browse/KYLIN), and have described the bug/feature there in detail - [ ] Commit messages in my PR start with the related jira ID, like "KYLIN- Make Kylin project open-source" - [ ] Compiling and unit tests pass locally with my changes - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] If this change need a document change, I will prepare another pr against the `document` branch - [ ] Any dependent changes have been merged ## Further comments If this is a relatively large or complex change, kick off the discussion at user@kylin or dev@kylin by explaining why you chose the solution you did and what alternatives you considered, etc... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Query push down to Presto > - > > Key: KYLIN-4491 > URL: https://issues.apache.org/jira/browse/KYLIN-4491 > Project: Kylin > Issue Type: New Feature > Components: Integration, RDBMS Source >Reporter: Shao Feng Shi >Priority: Major > Fix For: v3.1.0 > > > With the data source SDK, we can push down a query to Presto instead of Hive > or Spark to gain better performance; With the SDK, we can also convert > Kylin's SQL query to Presto SQL. -- This message was sent by Atlassian Jira (v8.3.4#803005)
