[GitHub] [kylin] codecov-io edited a comment on issue #662: KYLIN-3983 Add extra metadata for measure. Fix the dead lock cause by…
codecov-io edited a comment on issue #662: KYLIN-3983 Add extra metadata for measure. Fix the dead lock cause by… URL: https://github.com/apache/kylin/pull/662#issuecomment-496083947 # [Codecov](https://codecov.io/gh/apache/kylin/pull/662?src=pr=h1) Report > :exclamation: No coverage uploaded for pull request base (`dynamic-measure@9f21f0c`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit). > The diff coverage is `62.5%`. [](https://codecov.io/gh/apache/kylin/pull/662?src=pr=tree) ```diff @@Coverage Diff @@ ## dynamic-measure #662 +/- ## == Coverage ? 26.17% Complexity ? 6181 == Files ? 1391 Lines ?82930 Branches ?11599 == Hits ?21704 Misses ?59151 Partials ? 2075 ``` | [Impacted Files](https://codecov.io/gh/apache/kylin/pull/662?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...e/kylin/storage/gtrecord/CubeScanRangePlanner.java](https://codecov.io/gh/apache/kylin/pull/662/diff?src=pr=tree#diff-Y29yZS1zdG9yYWdlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9zdG9yYWdlL2d0cmVjb3JkL0N1YmVTY2FuUmFuZ2VQbGFubmVyLmphdmE=) | `45.14% <0%> (ø)` | `26 <0> (?)` | | | [...ache/kylin/storage/hbase/cube/v2/CubeHBaseRPC.java](https://codecov.io/gh/apache/kylin/pull/662/diff?src=pr=tree#diff-c3RvcmFnZS1oYmFzZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUva3lsaW4vc3RvcmFnZS9oYmFzZS9jdWJlL3YyL0N1YmVIQmFzZVJQQy5qYXZh) | `11.92% <0%> (ø)` | `5 <0> (?)` | | | [...c/main/java/org/apache/kylin/cube/CubeManager.java](https://codecov.io/gh/apache/kylin/pull/662/diff?src=pr=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL0N1YmVNYW5hZ2VyLmphdmE=) | `40.69% <100%> (ø)` | `51 <1> (?)` | | | [.../java/org/apache/kylin/measure/MeasureManager.java](https://codecov.io/gh/apache/kylin/pull/662/diff?src=pr=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9tZWFzdXJlL01lYXN1cmVNYW5hZ2VyLmphdmE=) | `62.78% <63.88%> (ø)` | `68 <29> (?)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/kylin/pull/662?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/kylin/pull/662?src=pr=footer). Last update [9f21f0c...fd5a1d9](https://codecov.io/gh/apache/kylin/pull/662?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] nichunen merged pull request #652: Removed duplicate code in KylinMeta
nichunen merged pull request #652: Removed duplicate code in KylinMeta URL: https://github.com/apache/kylin/pull/652 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] nichunen merged pull request #651: Additional small performance improvements
nichunen merged pull request #651: Additional small performance improvements URL: https://github.com/apache/kylin/pull/651 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] nichunen merged pull request #649: Fixed some look not friendly things
nichunen merged pull request #649: Fixed some look not friendly things URL: https://github.com/apache/kylin/pull/649 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] codecov-io commented on issue #671: KYLIN-4026: Avoid too many file append operations in HiveProducer of of hive metrics reporter
codecov-io commented on issue #671: KYLIN-4026: Avoid too many file append operations in HiveProducer of of hive metrics reporter URL: https://github.com/apache/kylin/pull/671#issuecomment-497724489 # [Codecov](https://codecov.io/gh/apache/kylin/pull/671?src=pr=h1) Report > :exclamation: No coverage uploaded for pull request base (`master@cfce8bb`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit). > The diff coverage is `0%`. [](https://codecov.io/gh/apache/kylin/pull/671?src=pr=tree) ```diff @@Coverage Diff@@ ## master #671 +/- ## = Coverage ? 25.77% Complexity? 6009 = Files ? 1382 Lines ?82257 Branches ?11528 = Hits ?21199 Misses?59012 Partials ? 2046 ``` | [Impacted Files](https://codecov.io/gh/apache/kylin/pull/671?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...ache/kylin/metrics/lib/impl/hive/HiveProducer.java](https://codecov.io/gh/apache/kylin/pull/671/diff?src=pr=tree#diff-bWV0cmljcy1yZXBvcnRlci1oaXZlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9tZXRyaWNzL2xpYi9pbXBsL2hpdmUvSGl2ZVByb2R1Y2VyLmphdmE=) | `0% <0%> (ø)` | `0 <0> (?)` | | | [...ache/kylin/metrics/lib/impl/BlockingReservoir.java](https://codecov.io/gh/apache/kylin/pull/671/diff?src=pr=tree#diff-Y29yZS1tZXRyaWNzL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9tZXRyaWNzL2xpYi9pbXBsL0Jsb2NraW5nUmVzZXJ2b2lyLmphdmE=) | `0% <0%> (ø)` | `0 <0> (?)` | | | [...n/metrics/lib/impl/hive/HiveReservoirReporter.java](https://codecov.io/gh/apache/kylin/pull/671/diff?src=pr=tree#diff-bWV0cmljcy1yZXBvcnRlci1oaXZlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9tZXRyaWNzL2xpYi9pbXBsL2hpdmUvSGl2ZVJlc2Vydm9pclJlcG9ydGVyLmphdmE=) | `0% <0%> (ø)` | `0 <0> (?)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/kylin/pull/671?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/kylin/pull/671?src=pr=footer). Last update [cfce8bb...8f70a07](https://codecov.io/gh/apache/kylin/pull/671?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] codecov-io commented on issue #670: KYLIN-4025: Add detail exception in kylin http response
codecov-io commented on issue #670: KYLIN-4025: Add detail exception in kylin http response URL: https://github.com/apache/kylin/pull/670#issuecomment-497700427 # [Codecov](https://codecov.io/gh/apache/kylin/pull/670?src=pr=h1) Report > :exclamation: No coverage uploaded for pull request base (`master@cfce8bb`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit). > The diff coverage is `0%`. [](https://codecov.io/gh/apache/kylin/pull/670?src=pr=tree) ```diff @@Coverage Diff@@ ## master #670 +/- ## = Coverage ? 25.78% Complexity? 6008 = Files ? 1382 Lines ?82205 Branches ?11519 = Hits ?21198 Misses?58960 Partials ? 2047 ``` | [Impacted Files](https://codecov.io/gh/apache/kylin/pull/670?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [.../apache/kylin/rest/controller/TableController.java](https://codecov.io/gh/apache/kylin/pull/670/diff?src=pr=tree#diff-c2VydmVyLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3Jlc3QvY29udHJvbGxlci9UYWJsZUNvbnRyb2xsZXIuamF2YQ==) | `0% <0%> (ø)` | `0 <0> (?)` | | | [...che/kylin/rest/controller/StreamingController.java](https://codecov.io/gh/apache/kylin/pull/670/diff?src=pr=tree#diff-c2VydmVyLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3Jlc3QvY29udHJvbGxlci9TdHJlYW1pbmdDb250cm9sbGVyLmphdmE=) | `0% <0%> (ø)` | `0 <0> (?)` | | | [...pache/kylin/rest/controller/ProjectController.java](https://codecov.io/gh/apache/kylin/pull/670/diff?src=pr=tree#diff-c2VydmVyLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3Jlc3QvY29udHJvbGxlci9Qcm9qZWN0Q29udHJvbGxlci5qYXZh) | `0% <0%> (ø)` | `0 <0> (?)` | | | [.../apache/kylin/rest/controller/ModelController.java](https://codecov.io/gh/apache/kylin/pull/670/diff?src=pr=tree#diff-c2VydmVyLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3Jlc3QvY29udHJvbGxlci9Nb2RlbENvbnRyb2xsZXIuamF2YQ==) | `0% <0%> (ø)` | `0 <0> (?)` | | | [...g/apache/kylin/rest/controller/CubeController.java](https://codecov.io/gh/apache/kylin/pull/670/diff?src=pr=tree#diff-c2VydmVyLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3Jlc3QvY29udHJvbGxlci9DdWJlQ29udHJvbGxlci5qYXZh) | `0% <0%> (ø)` | `0 <0> (?)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/kylin/pull/670?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/kylin/pull/670?src=pr=footer). Last update [cfce8bb...f3f01e1](https://codecov.io/gh/apache/kylin/pull/670?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-4026) Avoid too many file append operations in HiveProducer of hive metrics reporter
[ https://issues.apache.org/jira/browse/KYLIN-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852949#comment-16852949 ] ASF GitHub Bot commented on KYLIN-4026: --- lshmouse commented on pull request #671: KYLIN-4026: Avoid too many file append operations in HiveProducer of of hive metrics reporter URL: https://github.com/apache/kylin/pull/671 See: https://issues.apache.org/jira/browse/KYLIN-4026 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Avoid too many file append operations in HiveProducer of hive metrics reporter > -- > > Key: KYLIN-4026 > URL: https://issues.apache.org/jira/browse/KYLIN-4026 > Project: Kylin > Issue Type: Improvement >Reporter: Liu Shaohui >Priority: Major > > Currently for each write in HiveProducer, there will be a hdfs append > operation, which is heavy for HDFS. > A improvement is to keep a FSDataOutputStream in HiveProducer and write data > to it continuously > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [kylin] lshmouse opened a new pull request #671: KYLIN-4026: Avoid too many file append operations in HiveProducer of of hive metrics reporter
lshmouse opened a new pull request #671: KYLIN-4026: Avoid too many file append operations in HiveProducer of of hive metrics reporter URL: https://github.com/apache/kylin/pull/671 See: https://issues.apache.org/jira/browse/KYLIN-4026 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] asfgit commented on issue #671: KYLIN-4026: Avoid too many file append operations in HiveProducer of of hive metrics reporter
asfgit commented on issue #671: KYLIN-4026: Avoid too many file append operations in HiveProducer of of hive metrics reporter URL: https://github.com/apache/kylin/pull/671#issuecomment-497684530 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] coveralls commented on issue #669: KYLIN-3997: Add a health check job of Kylin
coveralls commented on issue #669: KYLIN-3997: Add a health check job of Kylin URL: https://github.com/apache/kylin/pull/669#issuecomment-497677938 ## Pull Request Test Coverage Report for [Build 4565](https://coveralls.io/builds/23716615) * **5** of **170** **(2.94%)** changed or added relevant lines in **9** files are covered. * **224** unchanged lines in **4** files lost coverage. * Overall coverage decreased (**-0.06%**) to **28.222%** --- | Changes Missing Coverage | Covered Lines | Changed/Added Lines | % | | :-|--||---: | | [core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java](https://coveralls.io/builds/23716615/source?filename=core-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcommon%2FKylinConfigBase.java#L2197) | 0 | 1 | 0.0% | [core-cube/src/main/java/org/apache/kylin/cube/CubeDescManager.java](https://coveralls.io/builds/23716615/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2FCubeDescManager.java#L168) | 0 | 1 | 0.0% | [core-metadata/src/main/java/org/apache/kylin/metadata/cachesync/CachedCrudAssist.java](https://coveralls.io/builds/23716615/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmetadata%2Fcachesync%2FCachedCrudAssist.java#L147) | 3 | 5 | 60.0% | [core-metadata/src/main/java/org/apache/kylin/metadata/model/DataModelManager.java](https://coveralls.io/builds/23716615/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmetadata%2Fmodel%2FDataModelManager.java#L109) | 0 | 2 | 0.0% | [core-common/src/main/java/org/apache/kylin/common/util/BufferedLogger.java](https://coveralls.io/builds/23716615/source?filename=core-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcommon%2Futil%2FBufferedLogger.java#L49) | 0 | 3 | 0.0% | [core-common/src/main/java/org/apache/kylin/common/util/SoutLogger.java](https://coveralls.io/builds/23716615/source?filename=core-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcommon%2Futil%2FSoutLogger.java#L34) | 0 | 3 | 0.0% | [tool/src/main/java/org/apache/kylin/tool/KylinHealthCheckJob.java](https://coveralls.io/builds/23716615/source?filename=tool%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Ftool%2FKylinHealthCheckJob.java#L23) | 0 | 4 | 0.0% | [server-base/src/main/java/org/apache/kylin/rest/job/KylinHealthCheckJob.java](https://coveralls.io/builds/23716615/source?filename=server-base%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Frest%2Fjob%2FKylinHealthCheckJob.java#L57) | 0 | 149 | 0.0% | Files with Coverage Reduction | New Missed Lines | % | | :-|--|--: | | [core-cube/src/main/java/org/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://coveralls.io/builds/23716615/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Finmemcubing%2FMemDiskStore.java#L553) | 1 | 78.12% | | [core-metadata/src/main/java/org/apache/kylin/metadata/cachesync/CachedCrudAssist.java](https://coveralls.io/builds/23716615/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmetadata%2Fcachesync%2FCachedCrudAssist.java#L148) | 1 | 64.52% | | [core-cube/src/main/java/org/apache/kylin/cube/cuboid/TreeCuboidScheduler.java](https://coveralls.io/builds/23716615/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Fcuboid%2FTreeCuboidScheduler.java#L124) | 2 | 68.46% | | [core-cube/src/main/java/org/apache/kylin/cube/CubeManager.java](https://coveralls.io/builds/23716615/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2FCubeManager.java#L204) | 220 | 42.6% | | Totals | [](https://coveralls.io/builds/23716615) | | :-- | --: | | Change from base [Build 4562](https://coveralls.io/builds/23709655): | -0.06% | | Covered Lines: | 23248 | | Relevant Lines: | 82374 | --- # - [Coveralls](https://coveralls.io) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] codecov-io commented on issue #669: KYLIN-3997: Add a health check job of Kylin
codecov-io commented on issue #669: KYLIN-3997: Add a health check job of Kylin URL: https://github.com/apache/kylin/pull/669#issuecomment-497677464 # [Codecov](https://codecov.io/gh/apache/kylin/pull/669?src=pr=h1) Report > Merging [#669](https://codecov.io/gh/apache/kylin/pull/669?src=pr=desc) into [master](https://codecov.io/gh/apache/kylin/commit/670c2a98a82a699481eceddb685168104d5b185c?src=pr=desc) will **decrease** coverage by `0.05%`. > The diff coverage is `2.94%`. [](https://codecov.io/gh/apache/kylin/pull/669?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #669 +/- ## - Coverage 25.78% 25.73% -0.06% Complexity 6008 6008 Files 1382 1384 +2 Lines 8220282374 +172 Branches 1151911550 +31 - Hits 2119821197 -1 - Misses5895859129 +171 - Partials 2046 2048 +2 ``` | [Impacted Files](https://codecov.io/gh/apache/kylin/pull/669?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...in/java/org/apache/kylin/cube/CubeDescManager.java](https://codecov.io/gh/apache/kylin/pull/669/diff?src=pr=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL0N1YmVEZXNjTWFuYWdlci5qYXZh) | `33.57% <0%> (-0.25%)` | `12 <0> (ø)` | | | [.../java/org/apache/kylin/common/KylinConfigBase.java](https://codecov.io/gh/apache/kylin/pull/669/diff?src=pr=tree#diff-Y29yZS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2NvbW1vbi9LeWxpbkNvbmZpZ0Jhc2UuamF2YQ==) | `13.02% <0%> (-0.03%)` | `42 <0> (ø)` | | | [.../apache/kylin/metadata/model/DataModelManager.java](https://codecov.io/gh/apache/kylin/pull/669/diff?src=pr=tree#diff-Y29yZS1tZXRhZGF0YS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUva3lsaW4vbWV0YWRhdGEvbW9kZWwvRGF0YU1vZGVsTWFuYWdlci5qYXZh) | `34.54% <0%> (-0.64%)` | `12 <0> (ø)` | | | [...a/org/apache/kylin/common/util/BufferedLogger.java](https://codecov.io/gh/apache/kylin/pull/669/diff?src=pr=tree#diff-Y29yZS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2NvbW1vbi91dGlsL0J1ZmZlcmVkTG9nZ2VyLmphdmE=) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [.../java/org/apache/kylin/common/util/SoutLogger.java](https://codecov.io/gh/apache/kylin/pull/669/diff?src=pr=tree#diff-Y29yZS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2NvbW1vbi91dGlsL1NvdXRMb2dnZXIuamF2YQ==) | `16.66% <0%> (-16.67%)` | `1 <0> (ø)` | | | [...org/apache/kylin/rest/job/KylinHealthCheckJob.java](https://codecov.io/gh/apache/kylin/pull/669/diff?src=pr=tree#diff-c2VydmVyLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3Jlc3Qvam9iL0t5bGluSGVhbHRoQ2hlY2tKb2IuamF2YQ==) | `0% <0%> (ø)` | `0 <0> (?)` | | | [...ava/org/apache/kylin/tool/KylinHealthCheckJob.java](https://codecov.io/gh/apache/kylin/pull/669/diff?src=pr=tree#diff-dG9vbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUva3lsaW4vdG9vbC9LeWxpbkhlYWx0aENoZWNrSm9iLmphdmE=) | `0% <0%> (ø)` | `0 <0> (?)` | | | [...c/main/java/org/apache/kylin/cube/CubeManager.java](https://codecov.io/gh/apache/kylin/pull/669/diff?src=pr=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL0N1YmVNYW5hZ2VyLmphdmE=) | `37.77% <100%> (-0.26%)` | `44 <1> (ø)` | | | [...che/kylin/metadata/cachesync/CachedCrudAssist.java](https://codecov.io/gh/apache/kylin/pull/669/diff?src=pr=tree#diff-Y29yZS1tZXRhZGF0YS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUva3lsaW4vbWV0YWRhdGEvY2FjaGVzeW5jL0NhY2hlZENydWRBc3Npc3QuamF2YQ==) | `56.98% <60%> (+0.17%)` | `15 <1> (ø)` | :arrow_down: | | [.../apache/kylin/cube/cuboid/TreeCuboidScheduler.java](https://codecov.io/gh/apache/kylin/pull/669/diff?src=pr=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL2N1Ym9pZC9UcmVlQ3Vib2lkU2NoZWR1bGVyLmphdmE=) | `63.84% <0%> (-2.31%)` | `0% <0%> (ø)` | | | ... and [6 more](https://codecov.io/gh/apache/kylin/pull/669/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/kylin/pull/669?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/kylin/pull/669?src=pr=footer). Last update [670c2a9...2da08ff](https://codecov.io/gh/apache/kylin/pull/669?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message,
[jira] [Updated] (KYLIN-4026) Avoid too many file append operations in HiveProducer of hive metrics reporter
[ https://issues.apache.org/jira/browse/KYLIN-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui updated KYLIN-4026: --- Summary: Avoid too many file append operations in HiveProducer of hive metrics reporter (was: Avoid too many file append operation in HiveProducer of hive metrics reporter) > Avoid too many file append operations in HiveProducer of hive metrics reporter > -- > > Key: KYLIN-4026 > URL: https://issues.apache.org/jira/browse/KYLIN-4026 > Project: Kylin > Issue Type: Improvement >Reporter: Liu Shaohui >Priority: Major > > Currently for each write in HiveProducer, there will be a hdfs append > operation, which is heavy for HDFS. > A improvement is to keep a FSDataOutputStream in HiveProducer and write data > to it continuously > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-4026) Avoid too many file append operation in HiveProducer of hive metrics reporter
Liu Shaohui created KYLIN-4026: -- Summary: Avoid too many file append operation in HiveProducer of hive metrics reporter Key: KYLIN-4026 URL: https://issues.apache.org/jira/browse/KYLIN-4026 Project: Kylin Issue Type: Improvement Reporter: Liu Shaohui Currently for each write in HiveProducer, there will be a hdfs append operation, which is heavy for HDFS. A improvement is to keep a FSDataOutputStream in HiveProducer and write data to it continuously -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [kylin] asfgit commented on issue #670: KYLIN-4025: Add detail exception in kylin http response
asfgit commented on issue #670: KYLIN-4025: Add detail exception in kylin http response URL: https://github.com/apache/kylin/pull/670#issuecomment-497673914 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] lshmouse opened a new pull request #670: KYLIN-4025: Add detail exception in kylin http response
lshmouse opened a new pull request #670: KYLIN-4025: Add detail exception in kylin http response URL: https://github.com/apache/kylin/pull/670 See: https://issues.apache.org/jira/browse/KYLIN-4025 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-4025) Add detail exception in kylin http response
[ https://issues.apache.org/jira/browse/KYLIN-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852920#comment-16852920 ] ASF GitHub Bot commented on KYLIN-4025: --- lshmouse commented on pull request #670: KYLIN-4025: Add detail exception in kylin http response URL: https://github.com/apache/kylin/pull/670 See: https://issues.apache.org/jira/browse/KYLIN-4025 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add detail exception in kylin http response > > > Key: KYLIN-4025 > URL: https://issues.apache.org/jira/browse/KYLIN-4025 > Project: Kylin > Issue Type: New Feature > Components: REST Service >Affects Versions: v2.5.2 >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Minor > > Currently there is no detailed in http response when the requests go wrong, > because InternalErrorException in controller wrap the execption and remove > the exception stack. > It's better add the detail exception in kylin http response > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-4025) Add detail exception in kylin http response
Liu Shaohui created KYLIN-4025: -- Summary: Add detail exception in kylin http response Key: KYLIN-4025 URL: https://issues.apache.org/jira/browse/KYLIN-4025 Project: Kylin Issue Type: New Feature Components: REST Service Affects Versions: v2.5.2 Reporter: Liu Shaohui Assignee: Liu Shaohui Currently there is no detailed in http response when the requests go wrong, because InternalErrorException in controller wrap the execption and remove the exception stack. It's better add the detail exception in kylin http response -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3994) StorageCleanupJob may delete data of newly built segment because of cube cache in CubeManager
[ https://issues.apache.org/jira/browse/KYLIN-3994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852889#comment-16852889 ] ASF GitHub Bot commented on KYLIN-3994: --- nichunen commented on pull request #633: KYLIN-3994: StorageCleanupJob may delete cube id data of new built segment because of cube cache in CubeManager URL: https://github.com/apache/kylin/pull/633 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > StorageCleanupJob may delete data of newly built segment because of cube > cache in CubeManager > - > > Key: KYLIN-3994 > URL: https://issues.apache.org/jira/browse/KYLIN-3994 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.5.2 >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Major > Fix For: v2.6.3 > > > In our production cluster, we found that the cube id data of a new-built > segment is deleted by the StorageCleanupJob. > After checking the code of cleanUnusedHdfsFiles in StorageCleanupJob, we > found that there is a bug here: CubeManager read all cube meta in > initiation and cache it for later > listAllCubes operations, the metadata will be out of data after listing the > hdfs working dir. > So the working directory of a finished job may be deleted unexpectedly. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3994) StorageCleanupJob may delete data of newly built segment because of cube cache in CubeManager
[ https://issues.apache.org/jira/browse/KYLIN-3994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852890#comment-16852890 ] ASF subversion and git services commented on KYLIN-3994: Commit cfce8bb628e0e4ade4e03ba9d544096859760907 in kylin's branch refs/heads/master from Liu Shaohui [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=cfce8bb ] KYLIN-3994: StorageCleanupJob may delete cube id data of new built segment because of cube cache in CubeManager (#633) * KYLIN-3994: StorageCleanupJob may delete cube id data of new built segment because of cube cache in CubeManager > StorageCleanupJob may delete data of newly built segment because of cube > cache in CubeManager > - > > Key: KYLIN-3994 > URL: https://issues.apache.org/jira/browse/KYLIN-3994 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.5.2 >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Major > Fix For: v2.6.3 > > > In our production cluster, we found that the cube id data of a new-built > segment is deleted by the StorageCleanupJob. > After checking the code of cleanUnusedHdfsFiles in StorageCleanupJob, we > found that there is a bug here: CubeManager read all cube meta in > initiation and cache it for later > listAllCubes operations, the metadata will be out of data after listing the > hdfs working dir. > So the working directory of a finished job may be deleted unexpectedly. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3994) StorageCleanupJob may delete data of newly built segment because of cube cache in CubeManager
[ https://issues.apache.org/jira/browse/KYLIN-3994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852888#comment-16852888 ] ASF subversion and git services commented on KYLIN-3994: Commit cfce8bb628e0e4ade4e03ba9d544096859760907 in kylin's branch refs/heads/master from Liu Shaohui [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=cfce8bb ] KYLIN-3994: StorageCleanupJob may delete cube id data of new built segment because of cube cache in CubeManager (#633) * KYLIN-3994: StorageCleanupJob may delete cube id data of new built segment because of cube cache in CubeManager > StorageCleanupJob may delete data of newly built segment because of cube > cache in CubeManager > - > > Key: KYLIN-3994 > URL: https://issues.apache.org/jira/browse/KYLIN-3994 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.5.2 >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Major > Fix For: v2.6.3 > > > In our production cluster, we found that the cube id data of a new-built > segment is deleted by the StorageCleanupJob. > After checking the code of cleanUnusedHdfsFiles in StorageCleanupJob, we > found that there is a bug here: CubeManager read all cube meta in > initiation and cache it for later > listAllCubes operations, the metadata will be out of data after listing the > hdfs working dir. > So the working directory of a finished job may be deleted unexpectedly. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [kylin] nichunen merged pull request #633: KYLIN-3994: StorageCleanupJob may delete cube id data of new built segment because of cube cache in CubeManager
nichunen merged pull request #633: KYLIN-3994: StorageCleanupJob may delete cube id data of new built segment because of cube cache in CubeManager URL: https://github.com/apache/kylin/pull/633 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] coveralls edited a comment on issue #633: KYLIN-3994: StorageCleanupJob may delete cube id data of new built segment because of cube cache in CubeManager
coveralls edited a comment on issue #633: KYLIN-3994: StorageCleanupJob may delete cube id data of new built segment because of cube cache in CubeManager URL: https://github.com/apache/kylin/pull/633#issuecomment-489959857 ## Pull Request Test Coverage Report for [Build 4563](https://coveralls.io/builds/23715106) * **3** of **6** **(50.0%)** changed or added relevant lines in **4** files are covered. * **5** unchanged lines in **1** file lost coverage. * Overall coverage decreased (**-0.002%**) to **28.278%** --- | Changes Missing Coverage | Covered Lines | Changed/Added Lines | % | | :-|--||---: | | [core-cube/src/main/java/org/apache/kylin/cube/CubeManager.java](https://coveralls.io/builds/23715106/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2FCubeManager.java#L201) | 0 | 3 | 0.0% | Files with Coverage Reduction | New Missed Lines | % | | :-|--|--: | | [core-cube/src/main/java/org/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://coveralls.io/builds/23715106/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Finmemcubing%2FMemDiskStore.java#L439) | 5 | 77.81% | | Totals | [](https://coveralls.io/builds/23715106) | | :-- | --: | | Change from base [Build 4562](https://coveralls.io/builds/23709655): | -0.002% | | Covered Lines: | 23246 | | Relevant Lines: | 82205 | --- # - [Coveralls](https://coveralls.io) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] codecov-io commented on issue #633: KYLIN-3994: StorageCleanupJob may delete cube id data of new built segment because of cube cache in CubeManager
codecov-io commented on issue #633: KYLIN-3994: StorageCleanupJob may delete cube id data of new built segment because of cube cache in CubeManager URL: https://github.com/apache/kylin/pull/633#issuecomment-497653401 # [Codecov](https://codecov.io/gh/apache/kylin/pull/633?src=pr=h1) Report > Merging [#633](https://codecov.io/gh/apache/kylin/pull/633?src=pr=desc) into [master](https://codecov.io/gh/apache/kylin/commit/670c2a98a82a699481eceddb685168104d5b185c?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `50%`. [](https://codecov.io/gh/apache/kylin/pull/633?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #633 +/- ## - Coverage 25.78% 25.78% -0.01% - Complexity 6008 6009 +1 Files 1382 1382 Lines 8220282205 +3 Branches 1151911519 - Hits 2119821197 -1 - Misses5895858962 +4 Partials 2046 2046 ``` | [Impacted Files](https://codecov.io/gh/apache/kylin/pull/633?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...c/main/java/org/apache/kylin/cube/CubeManager.java](https://codecov.io/gh/apache/kylin/pull/633/diff?src=pr=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL0N1YmVNYW5hZ2VyLmphdmE=) | `37.83% <0%> (-0.2%)` | `44 <0> (ø)` | | | [...pache/kylin/rest/job/StorageCleanJobHbaseUtil.java](https://codecov.io/gh/apache/kylin/pull/633/diff?src=pr=tree#diff-c2VydmVyLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3Jlc3Qvam9iL1N0b3JhZ2VDbGVhbkpvYkhiYXNlVXRpbC5qYXZh) | `49.38% <100%> (ø)` | `7 <0> (ø)` | :arrow_down: | | [.../org/apache/kylin/rest/job/MetadataCleanupJob.java](https://codecov.io/gh/apache/kylin/pull/633/diff?src=pr=tree#diff-c2VydmVyLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3Jlc3Qvam9iL01ldGFkYXRhQ2xlYW51cEpvYi5qYXZh) | `66.42% <100%> (ø)` | `26 <0> (ø)` | :arrow_down: | | [...a/org/apache/kylin/rest/job/StorageCleanupJob.java](https://codecov.io/gh/apache/kylin/pull/633/diff?src=pr=tree#diff-c2VydmVyLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3Jlc3Qvam9iL1N0b3JhZ2VDbGVhbnVwSm9iLmphdmE=) | `65.48% <100%> (ø)` | `31 <0> (ø)` | :arrow_down: | | [...rg/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://codecov.io/gh/apache/kylin/pull/633/diff?src=pr=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL2lubWVtY3ViaW5nL01lbURpc2tTdG9yZS5qYXZh) | `69.3% <0%> (-0.92%)` | `7% <0%> (ø)` | | | [...ream/core/storage/columnar/ColumnarStoreCache.java](https://codecov.io/gh/apache/kylin/pull/633/diff?src=pr=tree#diff-c3RyZWFtLWNvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3N0cmVhbS9jb3JlL3N0b3JhZ2UvY29sdW1uYXIvQ29sdW1uYXJTdG9yZUNhY2hlLmphdmE=) | `51.16% <0%> (+2.32%)` | `8% <0%> (+1%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/kylin/pull/633?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/kylin/pull/633?src=pr=footer). Last update [670c2a9...7ac7c45](https://codecov.io/gh/apache/kylin/pull/633?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-3997) Add a health check job of Kylin
[
https://issues.apache.org/jira/browse/KYLIN-3997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852866#comment-16852866
]
ASF GitHub Bot commented on KYLIN-3997:
---
lshmouse commented on pull request #669: KYLIN-3997: Add a health check job of
Kylin
URL: https://github.com/apache/kylin/pull/669
See: https://issues.apache.org/jira/browse/KYLIN-3997
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add a health check job of Kylin
> ---
>
> Key: KYLIN-3997
> URL: https://issues.apache.org/jira/browse/KYLIN-3997
> Project: Kylin
> Issue Type: Improvement
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Major
>
> Kylin has many inner meta data and outer dependencies. There may be
> inconsistent for bugs or failures. It's better to have a a health check job
> to find these inconsistent issues in advance。
> The inconsistent issues we found in our clusters are followings
> * {color:#808080}the cubeid data not exist for cube merging{color}
> * {color:#808080}hbase table not exist or online for a segment{color}
> * {color:#808080}there are holes in cube segments(The build of some days
> failed, but user not found it){color}
> * {color:#808080}Too many segment(hbase tables){color}
> * {color:#808080}metadata of stale segment left in cube{color}
> * {color:#808080}Some cubes have not be updated/built for a long time{color}
> * {color:#808080}Some important parameters are no set in cube desc{color}
> * {color:#808080}...{color}
> Suggestions are welcomed, thanks~
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [kylin] lshmouse opened a new pull request #669: KYLIN-3997: Add a health check job of Kylin
lshmouse opened a new pull request #669: KYLIN-3997: Add a health check job of Kylin URL: https://github.com/apache/kylin/pull/669 See: https://issues.apache.org/jira/browse/KYLIN-3997 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] asfgit commented on issue #669: KYLIN-3997: Add a health check job of Kylin
asfgit commented on issue #669: KYLIN-3997: Add a health check job of Kylin URL: https://github.com/apache/kylin/pull/669#issuecomment-497648997 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Assigned] (KYLIN-4005) Saving Cube of a aggregation Groups(40 Dimensions, Max Dimension Combination:5) may cause kylin server OOM
[
https://issues.apache.org/jira/browse/KYLIN-4005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liu Shaohui reassigned KYLIN-4005:
--
Assignee: Liu Shaohui
> Saving Cube of a aggregation Groups(40 Dimensions, Max Dimension
> Combination:5) may cause kylin server OOM
> --
>
> Key: KYLIN-4005
> URL: https://issues.apache.org/jira/browse/KYLIN-4005
> Project: Kylin
> Issue Type: Bug
> Components: REST Service
>Affects Versions: v2.5.2
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Critical
> Fix For: Future
>
>
> A user try to save a cube with a aggregation Groups(40 Dimensions, Max
> Dimension Combination:5) caused the kylin server OOM. The reason is that the
> DefaultCuboidScheduler will cost a lot memory when calculating all cube ids.
> The stack is following
> {code}
> http-bio-7070-exec-35
> at java.lang.OutOfMemoryError.()V (OutOfMemoryError.java:48)
> at java.util.HashMap.resize()[Ljava/util/HashMap$Node; (HashMap.java:704)
> at
> java.util.HashMap.putVal(ILjava/lang/Object;Ljava/lang/Object;ZZ)Ljava/lang/Object;
> (HashMap.java:663)
> at
> java.util.HashMap.put(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
> (HashMap.java:612)
> at java.util.HashSet.add(Ljava/lang/Object;)Z (HashSet.java:220)
> at java.util.AbstractCollection.addAll(Ljava/util/Collection;)Z
> (AbstractCollection.java:344)
> at
> org.apache.kylin.cube.cuboid.DefaultCuboidScheduler.getOnTreeParentsByLayer(Ljava/util/Collection;)Ljava/util/Set;
> (DefaultCuboidScheduler.java:240)
> at
> org.apache.kylin.cube.cuboid.DefaultCuboidScheduler.buildTreeBottomUp()Lorg/apache/kylin/common/util/Pair;
> (DefaultCuboidScheduler.java:183)
> at
> org.apache.kylin.cube.cuboid.DefaultCuboidScheduler.(Lorg/apache/kylin/cube/model/CubeDesc;)V
> (DefaultCuboidScheduler.java:58)
> at
> sun.reflect.GeneratedConstructorAccessor140.newInstance([Ljava/lang/Object;)Ljava/lang/Object;
> (Unknown Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance([Ljava/lang/Object;)Ljava/lang/Object;
> (DelegatingConstructorAccessorImpl.java:45)
> at
> java.lang.reflect.Constructor.newInstance([Ljava/lang/Object;)Ljava/lang/Object;
> (Constructor.java:423)
> at
> org.apache.kylin.cube.cuboid.CuboidScheduler.getInstance(Lorg/apache/kylin/cube/model/CubeDesc;)Lorg/apache/kylin/cube/cuboid/CuboidScheduler;
> (CuboidScheduler.java:41)
> at
> org.apache.kylin.cube.model.CubeDesc.getInitialCuboidScheduler()Lorg/apache/kylin/cube/cuboid/CuboidScheduler;
> (CubeDesc.java:750)
> at
> org.apache.kylin.cube.cuboid.CuboidCLI.simulateCuboidGeneration(Lorg/apache/kylin/cube/model/CubeDesc;Z)I
> (CuboidCLI.java:47)
> at
> org.apache.kylin.rest.service.CubeService.updateCubeAndDesc(Lorg/apache/kylin/cube/CubeInstance;Lorg/apache/kylin/cube/model/CubeDesc;Ljava/lang/String;Z)Lorg/apache/kylin/cube/model/CubeDesc;
> (CubeService.java:287)
> at
> org.apache.kylin.rest.service.CubeService$$FastClassBySpringCGLIB$$17a07c0e.invoke(ILjava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
> (Unknown Source)
> at
> org.springframework.cglib.proxy.MethodProxy.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
> (MethodProxy.java:204)
> at
> org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(Ljava/lang/Object;Ljava/lang/reflect/Method;[Ljava/lang/Object;Lorg/springframework/cglib/proxy/MethodProxy;)Ljava/lang/Object;
> (CglibAopProxy.java:669)
> at
> org.apache.kylin.rest.service.CubeService$$EnhancerBySpringCGLIB$$34de75c4.updateCubeAndDesc(Lorg/apache/kylin/cube/CubeInstance;Lorg/apache/kylin/cube/model/CubeDesc;Ljava/lang/String;Z)Lorg/apache/kylin/cube/model/CubeDesc;
> (Unknown Source)
> at
> org.apache.kylin.rest.controller.CubeController.updateCubeDesc(Lorg/apache/kylin/rest/request/CubeRequest;)Lorg/apache/kylin/rest/request/CubeReq
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [kylin] ZhengshuaiPENG commented on issue #633: KYLIN-3994: StorageCleanupJob may delete cube id data of new built segment because of cube cache in CubeManager
ZhengshuaiPENG commented on issue #633: KYLIN-3994: StorageCleanupJob may delete cube id data of new built segment because of cube cache in CubeManager URL: https://github.com/apache/kylin/pull/633#issuecomment-497637516 it's fine to me This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] lshmouse opened a new pull request #668: KYLIN-4005: Saving Cube of a aggregation Groups(40 Dimensions, Max Dimension Combination:5) may cause kylin server OOM
lshmouse opened a new pull request #668: KYLIN-4005: Saving Cube of a aggregation Groups(40 Dimensions, Max Dimension Combination:5) may cause kylin server OOM URL: https://github.com/apache/kylin/pull/668 See: https://issues.apache.org/jira/browse/KYLIN-4005 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-4005) Saving Cube of a aggregation Groups(40 Dimensions, Max Dimension Combination:5) may cause kylin server OOM
[
https://issues.apache.org/jira/browse/KYLIN-4005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852841#comment-16852841
]
ASF GitHub Bot commented on KYLIN-4005:
---
lshmouse commented on pull request #668: KYLIN-4005: Saving Cube of a
aggregation Groups(40 Dimensions, Max Dimension Combination:5) may cause kylin
server OOM
URL: https://github.com/apache/kylin/pull/668
See: https://issues.apache.org/jira/browse/KYLIN-4005
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Saving Cube of a aggregation Groups(40 Dimensions, Max Dimension
> Combination:5) may cause kylin server OOM
> --
>
> Key: KYLIN-4005
> URL: https://issues.apache.org/jira/browse/KYLIN-4005
> Project: Kylin
> Issue Type: Bug
> Components: REST Service
>Affects Versions: v2.5.2
>Reporter: Liu Shaohui
>Priority: Critical
> Fix For: Future
>
>
> A user try to save a cube with a aggregation Groups(40 Dimensions, Max
> Dimension Combination:5) caused the kylin server OOM. The reason is that the
> DefaultCuboidScheduler will cost a lot memory when calculating all cube ids.
> The stack is following
> {code}
> http-bio-7070-exec-35
> at java.lang.OutOfMemoryError.()V (OutOfMemoryError.java:48)
> at java.util.HashMap.resize()[Ljava/util/HashMap$Node; (HashMap.java:704)
> at
> java.util.HashMap.putVal(ILjava/lang/Object;Ljava/lang/Object;ZZ)Ljava/lang/Object;
> (HashMap.java:663)
> at
> java.util.HashMap.put(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
> (HashMap.java:612)
> at java.util.HashSet.add(Ljava/lang/Object;)Z (HashSet.java:220)
> at java.util.AbstractCollection.addAll(Ljava/util/Collection;)Z
> (AbstractCollection.java:344)
> at
> org.apache.kylin.cube.cuboid.DefaultCuboidScheduler.getOnTreeParentsByLayer(Ljava/util/Collection;)Ljava/util/Set;
> (DefaultCuboidScheduler.java:240)
> at
> org.apache.kylin.cube.cuboid.DefaultCuboidScheduler.buildTreeBottomUp()Lorg/apache/kylin/common/util/Pair;
> (DefaultCuboidScheduler.java:183)
> at
> org.apache.kylin.cube.cuboid.DefaultCuboidScheduler.(Lorg/apache/kylin/cube/model/CubeDesc;)V
> (DefaultCuboidScheduler.java:58)
> at
> sun.reflect.GeneratedConstructorAccessor140.newInstance([Ljava/lang/Object;)Ljava/lang/Object;
> (Unknown Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance([Ljava/lang/Object;)Ljava/lang/Object;
> (DelegatingConstructorAccessorImpl.java:45)
> at
> java.lang.reflect.Constructor.newInstance([Ljava/lang/Object;)Ljava/lang/Object;
> (Constructor.java:423)
> at
> org.apache.kylin.cube.cuboid.CuboidScheduler.getInstance(Lorg/apache/kylin/cube/model/CubeDesc;)Lorg/apache/kylin/cube/cuboid/CuboidScheduler;
> (CuboidScheduler.java:41)
> at
> org.apache.kylin.cube.model.CubeDesc.getInitialCuboidScheduler()Lorg/apache/kylin/cube/cuboid/CuboidScheduler;
> (CubeDesc.java:750)
> at
> org.apache.kylin.cube.cuboid.CuboidCLI.simulateCuboidGeneration(Lorg/apache/kylin/cube/model/CubeDesc;Z)I
> (CuboidCLI.java:47)
> at
> org.apache.kylin.rest.service.CubeService.updateCubeAndDesc(Lorg/apache/kylin/cube/CubeInstance;Lorg/apache/kylin/cube/model/CubeDesc;Ljava/lang/String;Z)Lorg/apache/kylin/cube/model/CubeDesc;
> (CubeService.java:287)
> at
> org.apache.kylin.rest.service.CubeService$$FastClassBySpringCGLIB$$17a07c0e.invoke(ILjava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
> (Unknown Source)
> at
> org.springframework.cglib.proxy.MethodProxy.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
> (MethodProxy.java:204)
> at
> org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(Ljava/lang/Object;Ljava/lang/reflect/Method;[Ljava/lang/Object;Lorg/springframework/cglib/proxy/MethodProxy;)Ljava/lang/Object;
> (CglibAopProxy.java:669)
> at
> org.apache.kylin.rest.service.CubeService$$EnhancerBySpringCGLIB$$34de75c4.updateCubeAndDesc(Lorg/apache/kylin/cube/CubeInstance;Lorg/apache/kylin/cube/model/CubeDesc;Ljava/lang/String;Z)Lorg/apache/kylin/cube/model/CubeDesc;
> (Unknown Source)
> at
> org.apache.kylin.rest.controller.CubeController.updateCubeDesc(Lorg/apache/kylin/rest/request/CubeRequest;)Lorg/apache/kylin/rest/request/CubeReq
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [kylin] asfgit commented on issue #668: KYLIN-4005: Saving Cube of a aggregation Groups(40 Dimensions, Max Dimension Combination:5) may cause kylin server OOM
asfgit commented on issue #668: KYLIN-4005: Saving Cube of a aggregation Groups(40 Dimensions, Max Dimension Combination:5) may cause kylin server OOM URL: https://github.com/apache/kylin/pull/668#issuecomment-497635797 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] lshmouse commented on a change in pull request #633: KYLIN-3994: StorageCleanupJob may delete cube id data of new built segment because of cube cache in CubeManager
lshmouse commented on a change in pull request #633: KYLIN-3994:
StorageCleanupJob may delete cube id data of new built segment because of cube
cache in CubeManager
URL: https://github.com/apache/kylin/pull/633#discussion_r289296808
##
File path:
server-base/src/main/java/org/apache/kylin/rest/job/MetadataCleanupJob.java
##
@@ -122,7 +122,7 @@ public MetadataCleanupJob(KylinConfig config) {
// exclude resources in use
Set activeResources = Sets.newHashSet();
-for (CubeInstance cube : cubeManager.listAllCubes()) {
+for (CubeInstance cube : cubeManager.reloadAndListAllCubes()) {
Review comment:
In extremely case, NEW_RESOURCE_THRESHOLD_MS (12 hours) may be not enough.
So it's better to keep it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [kylin] lshmouse commented on a change in pull request #633: KYLIN-3994: StorageCleanupJob may delete cube id data of new built segment because of cube cache in CubeManager
lshmouse commented on a change in pull request #633: KYLIN-3994:
StorageCleanupJob may delete cube id data of new built segment because of cube
cache in CubeManager
URL: https://github.com/apache/kylin/pull/633#discussion_r289290054
##
File path: core-cube/src/main/java/org/apache/kylin/cube/CubeManager.java
##
@@ -188,6 +188,11 @@ public void onEntityChange(Broadcaster broadcaster,
String entity, Event event,
}
}
+public List reloadAndListAllCubes() throws IOException {
+crud.reloadAll();
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (KYLIN-4018) apache kylin 连接powerbi出错
[ https://issues.apache.org/jira/browse/KYLIN-4018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852799#comment-16852799 ] jamie_lee commented on KYLIN-4018: -- thanks, I have tried the solution you suggested,but it seems that the driver works for kylingence not apache kylin, so still I can't connect apache kylin with powerbi with directquery mode ,is there any other solutions ,thanks a lot! > apache kylin 连接powerbi出错 > > > Key: KYLIN-4018 > URL: https://issues.apache.org/jira/browse/KYLIN-4018 > Project: Kylin > Issue Type: Wish > Components: Driver - ODBC > Environment: apache kylin，powerbi desktop >Reporter: jamie_lee >Priority: Major > > 我在将apache kylin的数据源连接到powebi > desktop时使用了KylinODBCDriver的驱动，好像只支持导入模式，不支持directquery模式，以至于虽然连接成功了，但是导入的数据好像格式有些问题(有些类型为bigint的字段显示为0)，所以现在apache > kylin和powerbi之间的odbc驱动不支持directquery模式了么？或者是不是我现在用的驱动不对？我应该用哪个驱动呢？求解答，谢谢！ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-4007) Kylin concurrently builds a global dictionary for too long
[ https://issues.apache.org/jira/browse/KYLIN-4007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu closed KYLIN-4007. --- Resolution: Cannot Reproduce User cannot submit stacktrace, so we cannot reproduce this issue. > Kylin concurrently builds a global dictionary for too long > -- > > Key: KYLIN-4007 > URL: https://issues.apache.org/jira/browse/KYLIN-4007 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.6.1 >Reporter: mouxingya >Assignee: XiaoXiang Yu >Priority: Minor > Attachments: WechatIMG20.png > > > When a dimension (user_id) is accurately calculated at the same time, the > original data is in the order of millions, and the rowkey has the encoding > type fixed_lengrh and length 100. It took a few hours or even failed in the > build dimension dictionary step. After investigation, it was found that there > are concurrent processes that want to obtain the lock corresponding to the > cube / dict path, resulting in a deadlock. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
