[jira] [Created] (KYLIN-4121) Cleanup hive view intermediate tables after job be finished
Shaofeng SHI created KYLIN-4121: --- Summary: Cleanup hive view intermediate tables after job be finished Key: KYLIN-4121 URL: https://issues.apache.org/jira/browse/KYLIN-4121 Project: Kylin Issue Type: Improvement Components: Job Engine Reporter: Shaofeng SHI Reported by community user: I have a cube with a fact table join a lookup table in hive, and both are hive view. I submit a job one time per hour. Kylin can drop the intermediate fact table , but doesn't drop the intermediate lookup table . I check the source code , and find out that , at '13 step: Hive Cleanup' , the source code has been commented. It is a legacy issue. Now KYLIN-3515 has fixed that, so the cleanup can be enabled. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (KYLIN-4063) Avoid repeatedly calling "string.toLowerCase" in TimedJsonStreamParser#parse
Shaofeng SHI created KYLIN-4063:
---
Summary: Avoid repeatedly calling "string.toLowerCase" in
TimedJsonStreamParser#parse
Key: KYLIN-4063
URL: https://issues.apache.org/jira/browse/KYLIN-4063
Project: Kylin
Issue Type: Improvement
Components: NRT Streaming
Reporter: Shaofeng SHI
In TimedJsonStreamParser#parse, it has this:
{code:java}
for (TblColRef column : allColumns) {
final String columnName = column.getName().toLowerCase(Locale.ROOT);
if (populateDerivedTimeColumns(columnName, result, t) == false) {
result.add(getValueByKey(column, root));
}
}
{code}
As this method will be invoked for each message, and then for each column it
will have a "toLowerCase(Locale.ROOT)", which is unnecessary, because the
"allColumns" won't change.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-4062) Too many "if else" clause in PushDownRunnerJdbcImpl#toSqlType
Shaofeng SHI created KYLIN-4062:
---
Summary: Too many "if else" clause in
PushDownRunnerJdbcImpl#toSqlType
Key: KYLIN-4062
URL: https://issues.apache.org/jira/browse/KYLIN-4062
Project: Kylin
Issue Type: Improvement
Components: Query Engine
Reporter: Shaofeng SHI
In this method, it has 30 "if else" which is low efficient; Should use a static
Hashmap, then only need 1 check.
{code:java}
if ("string".equalsIgnoreCase(type)) {
return Types.VARCHAR;
} else if ("varchar".equalsIgnoreCase(type)) {
return Types.VARCHAR;
} else if ("char".equalsIgnoreCase(type)) {
return Types.CHAR;
} else if
...{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3999) Enable dynamic column by default
Shaofeng SHI created KYLIN-3999: --- Summary: Enable dynamic column by default Key: KYLIN-3999 URL: https://issues.apache.org/jira/browse/KYLIN-3999 Project: Kylin Issue Type: Improvement Components: Query Engine Reporter: Shaofeng SHI More and more user expects to use "SUM(Case when)" feature, and got error. The reason is the dynamic column is disabled by default. We should consider to enable it by default: kylin.query.enable-dynamic-column=true -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3878) NPE to run sonar analysis
Shaofeng SHI created KYLIN-3878: --- Summary: NPE to run sonar analysis Key: KYLIN-3878 URL: https://issues.apache.org/jira/browse/KYLIN-3878 Project: Kylin Issue Type: Test Components: Tools, Build and Test Reporter: Shaofeng SHI mvn sonar:sonar -Dsonar.host.url=https://sonarcloud.io -Dsonar.organization=kylin -e [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 03:13 min [INFO] Finished at: 2019-03-15T14:42:16Z [INFO] [ERROR] Failed to execute goal org.sonarsource.scanner.maven:sonar-maven-plugin:3.6.0.1398:sonar (default-cli) on project kylin: null: MojoExecutionException: NullPointerException -> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.sonarsource.scanner.maven:sonar-maven-plugin:3.6.0.1398:sonar (default-cli) on project kylin: null at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:213) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:154) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:146) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81) at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192) at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105) at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956) at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:290) at org.apache.maven.cli.MavenCli.main (MavenCli.java:194) at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke (Method.java:498) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289) at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415) at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356) Caused by: org.apache.maven.plugin.MojoExecutionException at org.sonarsource.scanner.maven.bootstrap.ScannerBootstrapper.execute (ScannerBootstrapper.java:67) at org.sonarsource.scanner.maven.SonarQubeMojo.execute (SonarQubeMojo.java:104) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:208) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:154) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:146) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81) at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192) at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105) at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956) at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:290) at org.apache.maven.cli.MavenCli.main (MavenCli.java:194) at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke (Method.java:498) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289) at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415) at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356) Caused by: java.lang.NullPointerException at org.A.E.get (Un
[jira] [Created] (KYLIN-3862) Check the binary packages
Shaofeng SHI created KYLIN-3862: --- Summary: Check the binary packages Key: KYLIN-3862 URL: https://issues.apache.org/jira/browse/KYLIN-3862 Project: Kylin Issue Type: Task Reporter: Shaofeng SHI As to the approval of binary packages: It's not possible in general to check the exact contents of a binary, however there are some checks that should be made: - sigs and hashes must be OK - the package must contain the correct NOTICE and LICENSE files for the included content - the package must not contain any content not derived from the source. - in the case of bundled binaries, reviewers must check that all contents are represented in the LICENSE (and NOTICE file if required). The bundle must not contain any files that are prohibited from distribution (category X). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3863) Check the binary packages
Shaofeng SHI created KYLIN-3863: --- Summary: Check the binary packages Key: KYLIN-3863 URL: https://issues.apache.org/jira/browse/KYLIN-3863 Project: Kylin Issue Type: Task Reporter: Shaofeng SHI As to the approval of binary packages: It's not possible in general to check the exact contents of a binary, however there are some checks that should be made: - sigs and hashes must be OK - the package must contain the correct NOTICE and LICENSE files for the included content - the package must not contain any content not derived from the source. - in the case of bundled binaries, reviewers must check that all contents are represented in the LICENSE (and NOTICE file if required). The bundle must not contain any files that are prohibited from distribution (category X). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3826) MergeCuboidJob only uploads necessary segment's dictionary
Shaofeng SHI created KYLIN-3826: --- Summary: MergeCuboidJob only uploads necessary segment's dictionary Key: KYLIN-3826 URL: https://issues.apache.org/jira/browse/KYLIN-3826 Project: Kylin Issue Type: Improvement Components: Job Engine Reporter: Shaofeng SHI On yesterday's Kylin meetup, Zhang Wei mentioned that the "MergeCuboidJob" will upload all segment's metadata, which will take extra long time when the segment number is large. While this is unnecessary. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3825) Add ACL Rest APIs to document
Shaofeng SHI created KYLIN-3825: --- Summary: Add ACL Rest APIs to document Key: KYLIN-3825 URL: https://issues.apache.org/jira/browse/KYLIN-3825 Project: Kylin Issue Type: Improvement Reporter: Shaofeng SHI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3823) Release v2.6.1
Shaofeng SHI created KYLIN-3823: --- Summary: Release v2.6.1 Key: KYLIN-3823 URL: https://issues.apache.org/jira/browse/KYLIN-3823 Project: Kylin Issue Type: Task Reporter: Shaofeng SHI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3795) Submit Spark jobs via Apache Livy
Shaofeng SHI created KYLIN-3795: --- Summary: Submit Spark jobs via Apache Livy Key: KYLIN-3795 URL: https://issues.apache.org/jira/browse/KYLIN-3795 Project: Kylin Issue Type: New Feature Components: Spark Engine Reporter: Shaofeng SHI Livy is a rest service for Spark. Some users are using Livy as the interface for Spark. Kylin can have the capability to submit spark job via Livy. https://livy.incubator.apache.org/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3796) MongoDB as data source
Shaofeng SHI created KYLIN-3796: --- Summary: MongoDB as data source Key: KYLIN-3796 URL: https://issues.apache.org/jira/browse/KYLIN-3796 Project: Kylin Issue Type: New Feature Components: Others Reporter: Shaofeng SHI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3793) org.apache.kylin.source.kafka.util.KafkaSampleProducer exit after generating 1 message
Shaofeng SHI created KYLIN-3793: --- Summary: org.apache.kylin.source.kafka.util.KafkaSampleProducer exit after generating 1 message Key: KYLIN-3793 URL: https://issues.apache.org/jira/browse/KYLIN-3793 Project: Kylin Issue Type: Bug Components: NRT Streaming Affects Versions: v2.6.0 Reporter: Shaofeng SHI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3753) Delete old hbase table when refresh the segment
Shaofeng SHI created KYLIN-3753: --- Summary: Delete old hbase table when refresh the segment Key: KYLIN-3753 URL: https://issues.apache.org/jira/browse/KYLIN-3753 Project: Kylin Issue Type: Improvement Components: Job Engine Reporter: Shaofeng SHI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3702) Release 2.5.2
Shaofeng SHI created KYLIN-3702: --- Summary: Release 2.5.2 Key: KYLIN-3702 URL: https://issues.apache.org/jira/browse/KYLIN-3702 Project: Kylin Issue Type: Task Reporter: Shaofeng SHI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3693) TopN, Count distinct incorrect in Spark engine
Shaofeng SHI created KYLIN-3693:
---
Summary: TopN, Count distinct incorrect in Spark engine
Key: KYLIN-3693
URL: https://issues.apache.org/jira/browse/KYLIN-3693
Project: Kylin
Issue Type: Bug
Components: Spark Engine
Affects Versions: v2.5.0, v2.5.1
Reporter: Shaofeng SHI
{code:java}
[ERROR] Failures:
[ERROR]
ITCombinationTest>ITKylinQueryTest.testPreciselyDistinctCountQuery:284->KylinTestBase.execAndCompQuery:544->KylinTestBase.execAndCompQuery:647->KylinTestBase.assertTableEquals:691
value (table=query result of query01, row=0, col=USER_COUNT) expected:<8[46]>
but was:<8[52]>
[ERROR]
ITCombinationTest>ITKylinQueryTest.testTopNQuery:277->KylinTestBase.execAndCompQuery:544->KylinTestBase.execAndCompQuery:647->KylinTestBase.assertTableEquals:691
value (table=query result of query82, row=0, col=CAL_DT)
expected:<2012-01-[28]> but was:<2012-01-[05]>
[INFO]
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3679) Fetch Kafka topic with Spark streaming
Shaofeng SHI created KYLIN-3679: --- Summary: Fetch Kafka topic with Spark streaming Key: KYLIN-3679 URL: https://issues.apache.org/jira/browse/KYLIN-3679 Project: Kylin Issue Type: New Feature Components: Spark Engine Reporter: Shaofeng SHI Now Kylin uses a MR job to fetch Kafka messages in parallel and then persist to HDFS for subsequent processing. If user selects to use Spark engine, we can use Spark streaming API to do this. Spark streaming can read the Kafka message in a given offset range as a RDD, then it would be easy to process; https://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html With Spark streaming, Kylin can also easily connect with other data source like Kinesis, Flume, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3678) CacheStateChecker may remove a cache file that under building
Shaofeng SHI created KYLIN-3678:
---
Summary: CacheStateChecker may remove a cache file that under
building
Key: KYLIN-3678
URL: https://issues.apache.org/jira/browse/KYLIN-3678
Project: Kylin
Issue Type: Bug
Components: Job Engine
Affects Versions: v2.5.0, v2.4.1, v2.4.0, v2.5.1
Reporter: Shaofeng SHI
Kylin test failed with such error:
{code:java}
2018-11-09 02:15:24,379 DEBUG [main] cachesync.CachedCrudAssist:127 : Loaded 1
ExternalFilterDesc(s) out of 1 resource
2018-11-09 02:15:24,380 WARN [main] common.KylinConfigBase:77 : KYLIN_HOME was
not set
2018-11-09 02:15:24,380 INFO [main] cache.RocksDBLookupBuilder:66 : create new
rocksdb
folder:lookup_cache/rocksdb/DEFAULT.TEST_COUNTRY/f19bc17c-d41d-a4be-b561-f6bd275f4c90/db
for table cache:DEFAULT.TEST_COUNTRY
2018-11-09 02:15:24,380 INFO [main] cache.RocksDBLookupBuilder:69 : start to
build lookup table:DEFAULT.TEST_COUNTRY to rocks
db:lookup_cache/rocksdb/DEFAULT.TEST_COUNTRY/f19bc17c-d41d-a4be-b561-f6bd275f4c90/db
2018-11-09 02:15:26,814 WARN [lookup-cache-state-checker-1]
common.KylinConfigBase:77 : KYLIN_HOME was not set
2018-11-09 02:15:26,814 INFO [lookup-cache-state-checker-1]
cache.RocksDBLookupTableCache:334 : check snapshot local cache state, local
path:lookup_cache/rocksdb
2018-11-09 02:15:26,814 INFO [lookup-cache-state-checker-1]
cache.RocksDBLookupTableCache:361 : removed cache
file:/var/jenkins/workspace/kylin-manual-ci/core-dictionary/lookup_cache/rocksdb/DEFAULT.TEST_COUNTRY/f19bc17c-d41d-a4be-b561-f6bd275f4c90,
it is not referred by any cube
2018-11-09 02:15:28,474 ERROR [main] cache.RocksDBLookupBuilder:77 : error when
put data to rocksDB
org.rocksdb.RocksDBException: While open a file for random read:
lookup_cache/rocksdb/DEFAULT.TEST_COUNTRY/f19bc17c-d41d-a4be-b561-f6bd275f4c90/db/18.sst:
No such file or directory
at org.rocksdb.RocksDB.put(Native Method)
at org.rocksdb.RocksDB.put(RocksDB.java:453)
at
org.apache.kylin.dict.lookup.cache.RocksDBLookupBuilder.build(RocksDBLookupBuilder.java:74)
at
org.apache.kylin.dict.lookup.cache.RocksDBLookupTableCacheTest.testRestoreCacheFromFiles(RocksDBLookupTableCacheTest.java:115)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:379)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:340)
at
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:413)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3664) Hive metrics reporter HiveProducer doesn't support multiple instances on one host
Shaofeng SHI created KYLIN-3664:
---
Summary: Hive metrics reporter HiveProducer doesn't support
multiple instances on one host
Key: KYLIN-3664
URL: https://issues.apache.org/jira/browse/KYLIN-3664
Project: Kylin
Issue Type: Bug
Components: Metrics
Reporter: Shaofeng SHI
In HiveProducer.java, it uses a file named with the host name, this causing if
there are multiple Kylin instances on the same machine, will have conflict when
appending metrics to HDFS file:
{code:java}
String hostName;
try {
hostName = InetAddress.getLocalHost().getHostName();
} catch (UnknownHostException e) {
hostName = "UNKNOWN";
}
CONTENT_FILE_NAME = hostName + "-part-";
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3652) Segment auto-merge should not be triggered in some cases
Shaofeng SHI created KYLIN-3652: --- Summary: Segment auto-merge should not be triggered in some cases Key: KYLIN-3652 URL: https://issues.apache.org/jira/browse/KYLIN-3652 Project: Kylin Issue Type: Improvement Components: Metadata Reporter: Shaofeng SHI The segment auto-merge will be triggered whenever there is segment change: add, remove, etc. This causes some problem, for example, the merging job has problem, and the user wants to do something for an existing segment. But once he discarding the auto-generated merging job, a new job is generated, causing an endless loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3651) JDBCResourceStore doesn't list all resources
Shaofeng SHI created KYLIN-3651: --- Summary: JDBCResourceStore doesn't list all resources Key: KYLIN-3651 URL: https://issues.apache.org/jira/browse/KYLIN-3651 Project: Kylin Issue Type: Bug Components: Metadata Affects Versions: v2.5.0 Reporter: Shaofeng SHI The JDBC resource store persists metadata into two tables: kylin_default_instance kylin_default_instance_log The metadata entry like "/execute_output" will be saved to the _log table; But the "list" operation on "/" only lists the subfolders in the first table, no in the second table. This will cause partial metadata copy when use #ResourceTool.copy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3648) Kylin may not add "kylin.storage.hbase.cluster-hdfs-config-file" to configuration properly
Shaofeng SHI created KYLIN-3648:
---
Summary: Kylin may not add
"kylin.storage.hbase.cluster-hdfs-config-file" to configuration properly
Key: KYLIN-3648
URL: https://issues.apache.org/jira/browse/KYLIN-3648
Project: Kylin
Issue Type: Bug
Components: Storage - HBase
Reporter: Shaofeng SHI
From this blog, we know that Configuration.addResource should better use a
Path as the input instead of a string:
conf.addResource(new Path("/home/hadoop/conf/core-site.xml"));
[https://dzone.com/articles/debugging-%E2%80%9Cwrong-fs-expected]
But in Kylin's HBaseConnection, it add the hbase cluster fs configure file as
string:
{code:java}
String hdfsConfigFile =
KylinConfig.getInstanceFromEnv().getHBaseClusterHDFSConfigFile();
if (hdfsConfigFile == null || hdfsConfigFile.isEmpty()) {
return;
}
Configuration hdfsConf = new Configuration(false);
hdfsConf.addResource(hdfsConfigFile);
{code}
This may make it doesn't work.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3646) Add instruction about rowkey sequence in "Advanced setting" page
Shaofeng SHI created KYLIN-3646: --- Summary: Add instruction about rowkey sequence in "Advanced setting" page Key: KYLIN-3646 URL: https://issues.apache.org/jira/browse/KYLIN-3646 Project: Kylin Issue Type: Improvement Components: Web Reporter: Shaofeng SHI The position of dimension in rowkey is important for performance (both build and query); But this message is not clear for a normal user. We'd better add instruction on the web GUI, so that user knows to adjust it by some rule. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3641) Release 2.5.1
Shaofeng SHI created KYLIN-3641: --- Summary: Release 2.5.1 Key: KYLIN-3641 URL: https://issues.apache.org/jira/browse/KYLIN-3641 Project: Kylin Issue Type: Task Reporter: Shaofeng SHI Kylin 2.5.0 has released for 1 month, we collected a lot of good feedbacks and also several bug reportings. Now it is time to release 2.5.1 . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3637) Union All not working with count()
Shaofeng SHI created KYLIN-3637:
---
Summary: Union All not working with count()
Key: KYLIN-3637
URL: https://issues.apache.org/jira/browse/KYLIN-3637
Project: Kylin
Issue Type: Bug
Reporter: Shaofeng SHI
>From user@kylin mailing list:
When I ran sql "SELECT count(TRANS_ID) as TRANS_ID FROM KYLIN_SALES where
TRANS_ID <> 1 union all select count(TRANS_ID) as TRANS_ID FROM KYLIN_SALES"
on sample cube.
Got Error:
Error while compiling generated Java code: public static class Record2_1
implements java.io.Serializable { public Long TRANS_ID; public long
_KY_COUNT__; public Record2_1() {} public boolean equals(Object o) { if
(this == o) \{ return true; } if (!(o instanceof Record2_1)) { return false;
} return java.util.Objects.equals(this.TRANS_ID, ((Record2_1) o).TRANS_ID)
&& this._KY_COUNT__ == ((Record2_1) o)._KY_COUNT__; } public int hashCode()
{ int h = 0; h = org.apache.calcite.runtime.Utilities.hash(h,
this.TRANS_ID); h = org.apache.calcite.runtime.Utilities.hash(h,
this._KY_COUNT__); return h; } public int compareTo(Record2_1 that) { int c;
c = org.apache.calcite.runtime.Utilities.compareNullsLast(this.TRANS_ID,
that.TRANS_ID); if (c != 0) \{ return c; } c =
org.apache.calcite.runtime.Utilities.compare(this._KY_COUNT__,
that._KY_COUNT__); if (c != 0) \{ return c; } return 0; } public String
toString() { return "{TRANS_ID=" + this.TRANS_ID + ", _KY_COUNT__=" +
this._KY_COUNT__ + "}"; } } public static class Record1_0 implements
java.io.Serializable \{ public long f0; public Record1_0() {} public boolean
equals(Object o) \{ if (this == o) { return true; } if (!(o instanceof
Record1_0)) \{ return false; } return this.f0 == ((Record1_0) o).f0; } public
int hashCode() { int h = 0; h = org.apache.calcite.runtime.Utilities.hash(h,
this.f0); return h; } public int compareTo(Record1_0 that) { final int c; c
= org.apache.calcite.runtime.Utilities.compare(this.f0, that.f0); if (c !=
0) \{ return c; } return 0; } public String toString() { return "{f0=" +
this.f0 + "}"; } } org.apache.calcite.DataContext root; public
org.apache.calcite.linq4j.Enumerable bind(final
org.apache.calcite.DataContext root0) { root = root0; final
org.apache.calcite.linq4j.Enumerable _inputEnumerable =
((org.apache.kylin.query.schema.OLAPTable)
root.getRootSchema().getSubSchema("DEFAULT").getTable("KYLIN_SALES")).executeOLAPQuery(root,
1); final org.apache.calcite.linq4j.AbstractEnumerable child = new
org.apache.calcite.linq4j.AbstractEnumerable(){ public
org.apache.calcite.linq4j.Enumerator enumerator() { return new
org.apache.calcite.linq4j.Enumerator(){ public final
org.apache.calcite.linq4j.Enumerator inputEnumerator =
_inputEnumerable.enumerator(); public void reset() {
inputEnumerator.reset(); } public boolean moveNext() { while
(inputEnumerator.moveNext()) { final Long inp0_ = (Long) ((Object[])
inputEnumerator.current())[0]; if (inp0_ != null && inp0_.longValue() != 1L)
{ return true; } } return false; } public void close() {
inputEnumerator.close(); } public Object current() { final Object[] current
= (Object[]) inputEnumerator.current(); return new Object[] { current[0],
current[11]}; } }; } }; final org.apache.calcite.linq4j.Enumerable
_inputEnumerable0 = ((org.apache.kylin.query.schema.OLAPTable)
root.getRootSchema().getSubSchema("DEFAULT").getTable("KYLIN_SALES")).executeOLAPQuery(root,
2); final org.apache.calcite.linq4j.AbstractEnumerable child1 = new
org.apache.calcite.linq4j.AbstractEnumerable(){ public
org.apache.calcite.linq4j.Enumerator enumerator() { return new
org.apache.calcite.linq4j.Enumerator(){ public final
org.apache.calcite.linq4j.Enumerator inputEnumerator =
_inputEnumerable0.enumerator(); public void reset() {
inputEnumerator.reset(); } public boolean moveNext() { return
inputEnumerator.moveNext(); } public void close() { inputEnumerator.close();
} public Object current() { final Object[] current = (Object[])
inputEnumerator.current(); return new Record2_1( (Long) current[0],
org.apache.calcite.runtime.SqlFunctions.toLong(current[11])); } }; } };
return
org.apache.calcite.linq4j.Linq4j.singletonEnumerable(child.aggregate(new
org.apache.calcite.linq4j.function.Function0() { public Object apply() {
long a0s0; a0s0 = 0; Record1_0 record0; record0 = new Record1_0();
record0.f0 = a0s0; return record0; } } .apply(), new
org.apache.calcite.linq4j.function.Function2() { public Record1_0
apply(Record1_0 acc, Object[] in) { acc.f0 = acc.f0 +
org.apache.calcite.runtime.SqlFunctions.toLong(in[1]); return acc; } public
Record1_0 apply(Object acc, Object in) { return apply( (Record1_0) acc,
(Object[]) in); } } , new org.apache.calcite.linq4j.function.Function1() {
public long apply(Record1_0 acc) \{ return acc.f0; } public Object
apply(Object acc) \{ return apply( (Record1_0) acc); } }
)).concat(org.apache.calcite.linq4j.Linq4j.singletonEnumerable(child1.aggregate(new
org.apache.calcite.linq4j.function.Func
[jira] [Created] (KYLIN-3635) Percentile calculation on Spark engine is not thread safe
Shaofeng SHI created KYLIN-3635: --- Summary: Percentile calculation on Spark engine is not thread safe Key: KYLIN-3635 URL: https://issues.apache.org/jira/browse/KYLIN-3635 Project: Kylin Issue Type: Bug Affects Versions: v2.4.1, v2.4.0, v2.3.1, v2.3.0, v2.3.2, v2.5.0 Reporter: Shaofeng SHI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3626) Allow customization for Parquet path
Shaofeng SHI created KYLIN-3626: --- Summary: Allow customization for Parquet path Key: KYLIN-3626 URL: https://issues.apache.org/jira/browse/KYLIN-3626 Project: Kylin Issue Type: Sub-task Reporter: Shaofeng SHI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3624) Convert cuboid to Parquet in Spark
Shaofeng SHI created KYLIN-3624: --- Summary: Convert cuboid to Parquet in Spark Key: KYLIN-3624 URL: https://issues.apache.org/jira/browse/KYLIN-3624 Project: Kylin Issue Type: Sub-task Reporter: Shaofeng SHI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3625) Query engine for Parquet
Shaofeng SHI created KYLIN-3625: --- Summary: Query engine for Parquet Key: KYLIN-3625 URL: https://issues.apache.org/jira/browse/KYLIN-3625 Project: Kylin Issue Type: Sub-task Reporter: Shaofeng SHI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3623) Convert cuboid to Parquet in MR
Shaofeng SHI created KYLIN-3623: --- Summary: Convert cuboid to Parquet in MR Key: KYLIN-3623 URL: https://issues.apache.org/jira/browse/KYLIN-3623 Project: Kylin Issue Type: Sub-task Components: Job Engine Reporter: Shaofeng SHI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3622) Cube layout in Parquet
Shaofeng SHI created KYLIN-3622: --- Summary: Cube layout in Parquet Key: KYLIN-3622 URL: https://issues.apache.org/jira/browse/KYLIN-3622 Project: Kylin Issue Type: Sub-task Reporter: Shaofeng SHI The file and schema layout for Cubes in Parquet. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3621) Parquet as Cube storage
Shaofeng SHI created KYLIN-3621: --- Summary: Parquet as Cube storage Key: KYLIN-3621 URL: https://issues.apache.org/jira/browse/KYLIN-3621 Project: Kylin Issue Type: New Feature Components: Query Engine, Storage - HBase Reporter: Shaofeng SHI As discussed in dev@kylin, we plan to make a new storage engine based on Apache Parquet and Apache Spark. The discussion is: http://apache-kylin.74782.x6.nabble.com/DISCUSS-Columnar-storage-engine-for-Apache-Kylin-td11821.html#a11910 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3611) Upgrade Tomcat to 7.0.91, 8.5.34 or later
Shaofeng SHI created KYLIN-3611: --- Summary: Upgrade Tomcat to 7.0.91, 8.5.34 or later Key: KYLIN-3611 URL: https://issues.apache.org/jira/browse/KYLIN-3611 Project: Kylin Issue Type: Improvement Reporter: Shaofeng SHI h2. [SECURITY] CVE-2018-11784 Apache Tomcat - Open Redirect CVE-2018-11784 Apache Tomcat - Open Redirect Severity: Moderate Vendor: The Apache Software Foundation Versions Affected: Apache Tomcat 9.0.0.M1 to 9.0.11 Apache Tomcat 8.5.0 to 8.5.33 Apache Tomcat 7.0.23 to 7.0.90 The unsupported 8.0.x release line has not been analysed but is likely to be affected. Description: When the default servlet returned a redirect to a directory (e.g. redirecting to '/foo/' when the user requested '/foo') a specially crafted URL could be used to cause the redirect to be generated to any URI of the attackers choice. Mitigation: Users of the affected versions should apply one of the following mitigations: - Upgrade to Apache Tomcat 9.0.12 or later. - Upgrade to Apache Tomcat 8.5.34 or later. - Upgrade to Apache Tomcat 7.0.91 or later. - Use mapperDirectoryRedirectEnabled="true" and mapperContextRootRedirectEnabled="true" on the Context to ensure that redirects are issued by the Mapper rather than the default Servlet. See the Context configuration documentation for further important details. Credit: This vulnerability was found by Sergey Bobrov and reported responsibly to the Apache Tomcat Security Team. History: 2018-10-03 Original advisory References: [1] [http://tomcat.apache.org/security-9.html] [2] [http://tomcat.apache.org/security-8.html] [3] [http://tomcat.apache.org/security-7.html] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3606) Unstable KylinUserManagerTest
Shaofeng SHI created KYLIN-3606:
---
Summary: Unstable KylinUserManagerTest
Key: KYLIN-3606
URL: https://issues.apache.org/jira/browse/KYLIN-3606
Project: Kylin
Issue Type: Bug
Affects Versions: v2.6.0
Reporter: Shaofeng SHI
{code:java}
[[1;34mINFO[m] [1;32mTests run: [0;1;32m3[m, Failures: 0, Errors: 0,
Skipped: 0, Time elapsed: 4.187 s - in
org.apache.kylin.rest.security.[1mQueryWithTableACLTest[m
[[1;34mINFO[m] Running
org.apache.kylin.rest.security.[1mKylinUserManagerTest[m
2018-09-30 04:01:45,114 INFO [main] common.KylinConfig:113 : Destroy
KylinConfig
2018-09-30 04:01:45,160 INFO [main] common.KylinConfig:332 : Loading
kylin-defaults.properties from
/home/travis/build/apache/kylin/core-common/target/classes/kylin-defaults.properties
2018-09-30 04:01:45,169 INFO [main] common.KylinConfig:287 : Use
KYLIN_CONF=../examples/test_metadata/
2018-09-30 04:01:45,170 INFO [main] common.KylinConfig:99 : Initialized a new
KylinConfig from getInstanceFromEnv : 1932094362
2018-09-30 04:01:45,170 INFO [main] common.KylinConfigBase:198 : Kylin Config
was updated with kylin.metadata.url : ../examples/test_metadata/
2018-09-30 04:01:45,170 INFO [main] common.KylinConfigBase:198 : Kylin Config
was updated with kylin.env.hdfs-working-dir : file:///tmp/kylin
2018-09-30 04:01:45,193 INFO [main] util.MultiNodeManagerTestBase:50 : Chosen
port for CacheServiceTest is 45238
2018-09-30 04:01:45,194 INFO [main] common.KylinConfigBase:198 : Kylin Config
was updated with kylin.server.cluster-servers : localhost:45238
2018-09-30 04:01:45,194 INFO [main] common.KylinConfigBase:198 : Kylin Config
was updated with kylin.server.cluster-servers : localhost:45238
2018-09-30 04:01:45,194 INFO [main] common.KylinConfigBase:198 : Kylin Config
was updated with kylin.metadata.url : ../examples/test_metadata
2018-09-30 04:01:45,273 INFO [Thread-2-SendThread(localhost:2181)]
zookeeper.ClientCnxn:1029 : Opening socket connection to server
localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown
error)
2018-09-30 04:01:45,274 INFO [Thread-2-SendThread(localhost:2181)]
zookeeper.ClientCnxn:1166 : Socket error occurred: localhost/127.0.0.1:2181:
Connection refused
2018-09-30 04:01:45,275 INFO [main] util.log:192 : Logging initialized @92503ms
2018-09-30 04:01:46,021 INFO [main] server.Server:346 : jetty-9.3.22.v20171030
2018-09-30 04:01:46,225 INFO [main] handler.ContextHandler:781 : Started
o.e.j.s.ServletContextHandler@545311c{/,null,AVAILABLE}
2018-09-30 04:01:46,254 INFO [main] server.AbstractConnector:278 : Started
ServerConnector@4bdfeead{HTTP/1.1,[http/1.1]}{0.0.0.0:45238}
2018-09-30 04:01:46,254 INFO [main] server.Server:414 : Started @93482ms
2018-09-30 04:01:46,255 INFO [main] security.KylinUserManager:61 :
Initializing KylinUserManager with config ../examples/test_metadata/
2018-09-30 04:01:46,256 INFO [main] persistence.ResourceStore:88 : Using
metadata url ../examples/test_metadata/ for resource store
2018-09-30 04:01:46,256 DEBUG [main] cachesync.CachedCrudAssist:118 : Reloading
ManagedUser from
/home/travis/build/apache/kylin/server/../examples/test_metadata/user
2018-09-30 04:01:46,257 DEBUG [main] cachesync.CachedCrudAssist:127 : Loaded 0
ManagedUser(s) out of 0 resource
2018-09-30 04:01:46,257 INFO [main] common.KylinConfig:432 : Creating new
manager instance of class org.apache.kylin.metadata.cachesync.Broadcaster
2018-09-30 04:01:46,258 DEBUG [main] cachesync.Broadcaster:101 : 1 nodes in the
cluster: [localhost:45238]
2018-09-30 04:01:46,259 INFO [main] security.KylinUserManager:61 :
Initializing KylinUserManager with config ../examples/test_metadata
2018-09-30 04:01:46,259 INFO [main] persistence.ResourceStore:88 : Using
metadata url ../examples/test_metadata for resource store
2018-09-30 04:01:46,259 DEBUG [main] cachesync.CachedCrudAssist:118 : Reloading
ManagedUser from
/home/travis/build/apache/kylin/server/../examples/test_metadata/user
2018-09-30 04:01:46,260 DEBUG [main] cachesync.CachedCrudAssist:127 : Loaded 0
ManagedUser(s) out of 0 resource
2018-09-30 04:01:46,260 INFO [main] common.KylinConfig:432 : Creating new
manager instance of class org.apache.kylin.metadata.cachesync.Broadcaster
2018-09-30 04:01:46,260 DEBUG [main] cachesync.Broadcaster:101 : 1 nodes in the
cluster: [localhost:45238]
2018-09-30 04:01:46,261 DEBUG [main] cachesync.CachedCrudAssist:190 : Saving
ManagedUser at /user/U1
2018-09-30 04:01:46,263 DEBUG [pool-173-thread-1] cachesync.Broadcaster:113 :
Servers in the cluster: [localhost:45238]
2018-09-30 04:01:46,375 INFO [Thread-2-SendThread(localhost:2181)]
zookeeper.ClientCnxn:1029 : Opening socket connection to server
localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown
error)
2018-09-30 04:01:46,376 INFO [Thread-2-SendThread(localhost:2181)]
zookeeper.Clie
[jira] [Created] (KYLIN-3605) Fix security issues reported by snyk.io
Shaofeng SHI created KYLIN-3605: --- Summary: Fix security issues reported by snyk.io Key: KYLIN-3605 URL: https://issues.apache.org/jira/browse/KYLIN-3605 Project: Kylin Issue Type: Improvement Reporter: Shaofeng SHI [org.apache.commons:commons-compress|https://github.com/apache/commons-compress] defines an API for working with compression and archive formats. Affected versions of this package are vulnerable to Directory Traversal. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3597) Fix sonar reported static code issues
Shaofeng SHI created KYLIN-3597: --- Summary: Fix sonar reported static code issues Key: KYLIN-3597 URL: https://issues.apache.org/jira/browse/KYLIN-3597 Project: Kylin Issue Type: Improvement Components: Others Reporter: Shaofeng SHI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3577) Mvn dependency warnings
Shaofeng SHI created KYLIN-3577: --- Summary: Mvn dependency warnings Key: KYLIN-3577 URL: https://issues.apache.org/jira/browse/KYLIN-3577 Project: Kylin Issue Type: Improvement Affects Versions: v2.5.0 Reporter: Shaofeng SHI [INFO] *--- maven-dependency-plugin:2.10:analyze-only (analyze) @ kylin-core-common ---* [WARNING] Used undeclared dependencies found:[WARNING] com.fasterxml.jackson.core:jackson-annotations:jar:2.9.5:compile[WARNING] com.google.guava:guava:jar:14.0:provided[WARNING] commons-lang:commons-lang:jar:2.6:provided[WARNING] com.fasterxml.jackson.core:jackson-core:jar:2.9.5:compile[WARNING] org.apache.httpcomponents:httpcore:jar:4.2.4:compile[WARNING] io.dropwizard.metrics:metrics-core:jar:3.1.2:compile[WARNING] commons-io:commons-io:jar:2.4:provided[WARNING] commons-logging:commons-logging:jar:1.1.1:compile[WARNING] javax.xml.bind:jaxb-api:jar:2.2.2:provided[WARNING] commons-codec:commons-codec:jar:1.6:compile[WARNING] commons-cli:commons-cli:jar:1.2:provided[WARNING] Unused declared dependencies found:[WARNING]io.dropwizard.metrics:metrics-jvm:jar:3.1.2:compile[WARNING] log4j:log4j:jar:1.2.17:provided[WARNING] org.slf4j:slf4j-log4j12:jar:1.7.21:provided[WARNING] mysql:mysql-connector-java:jar:5.1.8:provided[WARNING] org.slf4j:jcl-over-slf4j:jar:1.7.21:compile [INFO] *--- maven-dependency-plugin:2.10:analyze-only (analyze) @ kylin-core-metadata ---* [WARNING] Used undeclared dependencies found:[WARNING] org.apache.commons:commons-lang3:jar:3.4:compile[WARNING] com.fasterxml.jackson.core:jackson-annotations:jar:2.9.5:compile[WARNING] org.mockito:mockito-core:jar:1.10.19:test[WARNING] com.fasterxml.jackson.core:jackson-core:jar:2.9.5:compile[WARNING] org.objenesis:objenesis:jar:2.2:provided[WARNING] org.apache.calcite:calcite-core:jar:1.13.0-kylin-r4:compile[WARNING] org.apache.calcite.avatica:avatica:jar:1.10.0:compile[WARNING] Unused declared dependencies found:[WARNING]org.slf4j:slf4j-api:jar:1.7.21:compile[WARNING] log4j:log4j:jar:1.2.17:provided[WARNING] org.slf4j:slf4j-log4j12:jar:1.7.21:provided[WARNING] org.apache.hadoop:hadoop-common:jar:2.7.1:provided[WARNING] org.apache.kylin:atopcalcite:jar:2.6.0-SNAPSHOT:compile[WARNING] commons-cli:commons-cli:jar:1.2:provided[WARNING] org.slf4j:jcl-over-slf4j:jar:1.7.21:compile [INFO] *--- maven-dependency-plugin:2.10:analyze-only (analyze) @ kylin-core-dictionary ---* [WARNING] Used undeclared dependencies found:[WARNING] com.fasterxml.jackson.core:jackson-annotations:jar:2.9.5:compile[WARNING] commons-lang:commons-lang:jar:2.6:provided[WARNING] com.google.guava:guava:jar:14.0:provided[WARNING] org.apache.kylin:kylin-core-common:jar:2.6.0-SNAPSHOT:compile[WARNING] commons-io:commons-io:jar:2.4:provided[WARNING] org.apache.calcite.avatica:avatica:jar:1.10.0:compile[WARNING] com.google.code.findbugs:jsr305:jar:3.0.1:provided[WARNING] Unused declared dependencies found:[WARNING]org.slf4j:slf4j-api:jar:1.7.21:compile[WARNING] log4j:log4j:jar:1.2.17:provided[WARNING] org.slf4j:slf4j-log4j12:jar:1.7.21:provided[WARNING] org.slf4j:jcl-over-slf4j:jar:1.7.21:compile [INFO] *--- maven-dependency-plugin:2.10:analyze-only (analyze) @ kylin-core-cube ---* [WARNING] Used undeclared dependencies found:[WARNING] org.apache.commons:commons-lang3:jar:3.4:compile[WARNING] com.fasterxml.jackson.core:jackson-annotations:jar:2.9.5:compile[WARNING] commons-codec:commons-codec:jar:1.4:compile[WARNING] com.fasterxml.jackson.core:jackson-core:jar:2.9.5:compile[WARNING] org.apache.kylin:kylin-core-common:jar:2.6.0-SNAPSHOT:compile[WARNING] org.apache.commons:commons-math3:jar:3.1.1:provided[WARNING] org.apache.calcite.avatica:avatica:jar:1.10.0:compile[WARNING] com.fasterxml.jackson.core:jackson-databind:jar:2.9.5:compile[WARNING] Unused declared dependencies found:[WARNING] org.slf4j:slf4j-api:jar:1.7.21:compile[WARNING] log4j:log4j:jar:1.2.17:provided[WARNING] org.slf4j:slf4j-log4j12:jar:1.7.21:provided[WARNING] com.esotericsoftware:kryo-shaded:jar:4.0.0:provided[WARNING] org.apache.hadoop:hadoop-common:jar:2.7.1:provided[WARNING] org.slf4j:jcl-over-slf4j:jar:1.7.21:compile [INFO] *--- maven-dependency-plugin:2.10:analyze-only (analyze) @ kylin-core-metrics ---* [WARNING] Unused declared dependencies found:[WARNING] log4j:log4j:jar:1.2.17:provided[WARNING] org.slf4j:slf4j-log4j12:jar:1.7.21:provided[WARNING] org.apache.hadoop:hadoop-common:jar:2.7.1:provided[WARNING] org.slf4j:jcl-over-slf4j:jar:1.7.21:compile [INFO] *--- maven-dependency-plugin:2.10:analyze-only (analyze) @ kylin-core-job ---* [WARNING] Used undeclared dependencies found:[WARNING] com.fasterxml.jackson.core:jackson-an
[jira] [Created] (KYLIN-3576) IllegalArgumentException: No enum constant org.apache.kylin.rest.service.JobService.JobSearchMode.
Shaofeng SHI created KYLIN-3576:
---
Summary: IllegalArgumentException: No enum constant
org.apache.kylin.rest.service.JobService.JobSearchMode.
Key: KYLIN-3576
URL: https://issues.apache.org/jira/browse/KYLIN-3576
Project: Kylin
Issue Type: Bug
Components: Job Engine
Reporter: Shaofeng SHI
{code:java}
2018-09-22 19:58:20,258 INFO [main] execution.ExecutableManager:69 : Using
metadata url: ../examples/test_metadata/
2018-09-22 19:58:20,265 ERROR [main] controller.JobController:81 : Invalid
value for JobSearchMode: '', skip it.
java.lang.IllegalArgumentException: No enum constant
org.apache.kylin.rest.service.JobService.JobSearchMode.
at java.lang.Enum.valueOf(Enum.java:238)
at
org.apache.kylin.rest.service.JobService$JobSearchMode.valueOf(JobService.java:1236)
at
org.apache.kylin.rest.controller.JobController.list(JobController.java:79)
at
org.apache.kylin.rest.controller.JobControllerTest.testBasics(JobControllerTest.java:103)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:75)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at
org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:86)
at
org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:84)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3571) Not build Spark in Kylin's binary package
Shaofeng SHI created KYLIN-3571: --- Summary: Not build Spark in Kylin's binary package Key: KYLIN-3571 URL: https://issues.apache.org/jira/browse/KYLIN-3571 Project: Kylin Issue Type: Improvement Components: Environment Reporter: Shaofeng SHI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3563) Web UI style upgrade
Shaofeng SHI created KYLIN-3563: --- Summary: Web UI style upgrade Key: KYLIN-3563 URL: https://issues.apache.org/jira/browse/KYLIN-3563 Project: Kylin Issue Type: Improvement Components: Web Reporter: Shaofeng SHI Kylin's web GUI style almost has no change since the first day. We can introduce some change to make it fresh. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3554) Spark job failed but Yarn shows SUCCEED, causing Kylin move to next step
Shaofeng SHI created KYLIN-3554: --- Summary: Spark job failed but Yarn shows SUCCEED, causing Kylin move to next step Key: KYLIN-3554 URL: https://issues.apache.org/jira/browse/KYLIN-3554 Project: Kylin Issue Type: Bug Components: Spark Engine Reporter: Shaofeng SHI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3551) Spark job failed with "FileNotFoundException"
Shaofeng SHI created KYLIN-3551: --- Summary: Spark job failed with "FileNotFoundException" Key: KYLIN-3551 URL: https://issues.apache.org/jira/browse/KYLIN-3551 Project: Kylin Issue Type: Bug Components: Spark Engine Reporter: Shaofeng SHI java.io.FileNotFoundException: File does not exist: hdfs://sandbox.hortonworks.com:8020/kylin/kylin_default_instance/kylin-a3e39298-8dc3-21f2-cf16-0aa5e451c777/kylin_sales_cube_clone_clone/counter at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1319) at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1311) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1311) at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1752) at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1776) at org.apache.kylin.common.util.HadoopUtil.readFromSequenceFile(HadoopUtil.java:218) at org.apache.kylin.common.util.HadoopUtil.readFromSequenceFile(HadoopUtil.java:233) at org.apache.kylin.engine.spark.SparkExecutable.doWork(SparkExecutable.java:319) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:163) at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:69) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:163) at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:113) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3550) "kylin.source.hive.flat-table-field-delimiter" has extra "\" when create intermediate flat table
Shaofeng SHI created KYLIN-3550: --- Summary: "kylin.source.hive.flat-table-field-delimiter" has extra "\" when create intermediate flat table Key: KYLIN-3550 URL: https://issues.apache.org/jira/browse/KYLIN-3550 Project: Kylin Issue Type: Bug Components: Job Engine Affects Versions: v2.4.0, v2.4.1 Reporter: Shaofeng SHI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3527) Hybrid couldn't save when there is only 1 cube
Shaofeng SHI created KYLIN-3527: --- Summary: Hybrid couldn't save when there is only 1 cube Key: KYLIN-3527 URL: https://issues.apache.org/jira/browse/KYLIN-3527 Project: Kylin Issue Type: Bug Components: Web Affects Versions: v2.5.0 Reporter: Shaofeng SHI Attachments: hybrid-not-saved.png I created a hybrid with two cubes; Later I want to remove one cube from it, but I found when there is only 1 cube, the "Submit" button was disabled. The hybrid should allows 0, 1 or more cubes. Besides, I found the move in and out icons were not properly displayed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3524) Clone cube reports "Overwriting conflict /project/learn_kylin.json" error
Shaofeng SHI created KYLIN-3524:
---
Summary: Clone cube reports "Overwriting conflict
/project/learn_kylin.json" error
Key: KYLIN-3524
URL: https://issues.apache.org/jira/browse/KYLIN-3524
Project: Kylin
Issue Type: Bug
Components: Metadata
Affects Versions: v2.5.0
Reporter: Shaofeng SHI
{code:java}
53956 2018-09-02 20:25:51,458 INFO [http-bio-7080-exec-4]
service.CubeService:218 : New cube kylin_sales_cube_shao2 has 161 cuboids
53957 2018-09-02 20:25:51,458 INFO [http-bio-7080-exec-4] cube.CubeManager:225
: Creating cube 'learn_kylin-->kylin_sales_cube_shao2' from desc
'kylin_sales_cube_shao2'
53958 2018-09-02 20:25:51,458 INFO [http-bio-7080-exec-4] cube.CubeManager:316
: Updating cube instance 'kylin_sales_cube_shao2'
53959 2018-09-02 20:25:51,459 DEBUG [http-bio-7080-exec-4]
cachesync.CachedCrudAssist:190 : Saving CubeInstance at
/cube/kylin_sales_cube_shao2.json
53960 2018-09-02 20:25:51,460 DEBUG [pool-16-thread-1]
cachesync.Broadcaster:113 : Servers in the cluster: [localhost:7070]
53961 2018-09-02 20:25:51,460 DEBUG [pool-16-thread-1]
cachesync.Broadcaster:123 : Announcing new broadcast to all:
BroadcastEvent{entity=cube, event=create, cacheKey=kylin_sales_cube_shao2}
53962 2018-09-02 20:25:51,461 DEBUG [http-bio-7080-exec-4]
cachesync.CachedCrudAssist:190 : Saving ProjectInstance at
/project/learn_kylin.json
53963 2018-09-02 20:25:51,462 ERROR [http-bio-7080-exec-4]
controller.BasicController:62 :
53964 org.apache.kylin.common.persistence.WriteConflictException: Overwriting
conflict /project/learn_kylin.json, expect old TS 1535685985266, but it is
1535695655000
53965 at
org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:325)
53966 at
org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:318)
53967 at
org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:303)
53968 at
org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:282)
53969 at
org.apache.kylin.metadata.cachesync.CachedCrudAssist.save(CachedCrudAssist.java:192)
53970 at
org.apache.kylin.metadata.project.ProjectManager.save(ProjectManager.java:371)
53971 at
org.apache.kylin.metadata.project.ProjectManager.addRealizationToProject(ProjectManager.java:283)
53972 at
org.apache.kylin.metadata.project.ProjectManager.moveRealizationToProject(ProjectManager.java:266)
53973 at org.apache.kylin.cube.CubeManager.createCube(CubeManager.java:232)
53974 at
org.apache.kylin.rest.service.CubeService.createCubeAndDesc(CubeService.java:220)
53975 at
org.apache.kylin.rest.service.CubeService$$FastClassBySpringCGLIB$$17a07c0e.invoke()
53976 at
org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
53977 at
org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3521) Enable Cube Planner by default
Shaofeng SHI created KYLIN-3521: --- Summary: Enable Cube Planner by default Key: KYLIN-3521 URL: https://issues.apache.org/jira/browse/KYLIN-3521 Project: Kylin Issue Type: Improvement Affects Versions: v2.5.0 Reporter: Shaofeng SHI Cube planner can significantly reduce the cuboid number that to build. As it wasn't enabled by default in 2.3 and 2.4, many users don't know that. To let more user to start using it, I suggest to enable it by default. As Cube planner only works when build the first segment, it only affect the cuboid scheduler of a new Cube. Old cubes will not be affected. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3518) Coprocessor reports NPE when execute a query on HBase 2.0
Shaofeng SHI created KYLIN-3518:
---
Summary: Coprocessor reports NPE when execute a query on HBase 2.0
Key: KYLIN-3518
URL: https://issues.apache.org/jira/browse/KYLIN-3518
Project: Kylin
Issue Type: Bug
Components: Storage - HBase
Reporter: Shaofeng SHI
On HDP 3.0, build a cube and then run a simple count query, NPE occurred:
{code:java}
2018-08-28 01:30:16,969 DEBUG [Query f7bf8004-b516-e372-18df-0d507075d471-71]
v2.CubeHBaseRPC:315 : hbase.rpc.timeout = 9 ms, use 81000 ms as timeout for
coprocessor
2018-08-28 01:30:16,983 INFO [Query f7bf8004-b516-e372-18df-0d507075d471-71]
v2.CubeHBaseEndpointRPC:141 : Serialized scanRequestBytes 522 bytes,
rawScanBytesString 44 bytes
2018-08-28 01:30:16,984 INFO [Query f7bf8004-b516-e372-18df-0d507075d471-71]
v2.CubeHBaseEndpointRPC:143 : The scan 67b41fc6 for segment
kylin_sales_cube_clone[2012010100_2013010100] is as below with 1
separate raw scans, shard part of start/end key is set to 0
2018-08-28 01:30:16,991 INFO [Query f7bf8004-b516-e372-18df-0d507075d471-71]
v2.CubeHBaseRPC:288 : Visiting hbase table KYLIN_5Q088VO5I0: cuboid require
post aggregation, from 0 to 16384 Start:
\x00\x00\x00\x00\x00\x00\x00\x00\x40\x00\x00\x00\x00
(\x00\x00\x00\x00\x00\x00\x00\x00@\x00\x00\x00\x00) Stop:
\x00\x00\x00\x00\x00\x00\x00\x00\x40\x00\xFF\xFF\xFF\x00
(\x00\x00\x00\x00\x00\x00\x00\x00@\x00\xFF\xFF\xFF\x00), No Fuzzy Key
2018-08-28 01:30:16,991 DEBUG [Query f7bf8004-b516-e372-18df-0d507075d471-71]
v2.CubeHBaseEndpointRPC:148 : Submitting rpc to 1 shards starting from shard 0,
scan range count 1
2018-08-28 01:30:17,010 INFO [Query f7bf8004-b516-e372-18df-0d507075d471-71]
common.KylinConfig:332 : Loading kylin-defaults.properties from
file:/root/shaofengshi/apache-kylin-2.5.0-SNAPSHOT-bin/tomcat/webapps/kylin/WEB-INF/lib/kylin-core-common-2.5.0-SNAPSHOT.jar!/kylin-defaults.properties
2018-08-28 01:30:17,033 DEBUG [Query f7bf8004-b516-e372-18df-0d507075d471-71]
common.KylinConfig:291 : KYLIN_CONF property was not set, will seek KYLIN_HOME
env variable
2018-08-28 01:30:17,051 INFO [pool-14-thread-1] hbase.HBaseConnection:110 :
Creating coprocessor thread pool with max of 2048, core of 2048
2018-08-28 01:30:17,094 INFO [Query f7bf8004-b516-e372-18df-0d507075d471-71]
gtrecord.SequentialCubeTupleIterator:73 : Using SortedIteratorMergerWithLimit
to merge segment results
2018-08-28 01:30:17,097 DEBUG [Query f7bf8004-b516-e372-18df-0d507075d471-71]
enumerator.OLAPEnumerator:117 : return TupleIterator...
2018-08-28 01:30:21,607 INFO [kylin-coproc--pool9-t1]
client.RpcRetryingCallerImpl:134 : Call exception, tries=6, retries=6,
started=4410 ms ago, cancelled=false, msg=java.io.IOException
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:468)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:131)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
Caused by: java.lang.NullPointerException
at
org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService.visitCube(CubeVisitService.java:253)
at
org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitService.callMethod(CubeVisitProtos.java:)
at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:8032)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:2426)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:2408)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42010)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)
... 3 more
, details=row '' on table 'KYLIN_5Q088VO5I0' at
region=KYLIN_5Q088VO5I0,,1535417272444.27b82cb4702db4557a98b9a7e60b7692.,
hostname=ignite03.com,16020,1534313612401, seqNum=2
2018-08-28 01:30:25,633 INFO [kylin-coproc--pool9-t1]
client.RpcRetryingCallerImpl:134 : Call exception, tries=7, retries=7,
started=8436 ms ago, cancelled=false, msg=java.io.IOException
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:468)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:131)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
Caused by: java.lang.NullPointerException
at
org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService.visitCube(CubeVisitService.java:253)
at
org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitService.callMethod(CubeVisitProtos.java:)
at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:8032)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegi
[jira] [Created] (KYLIN-3517) Couldn't update coprocessor on HBase 2.0
Shaofeng SHI created KYLIN-3517:
---
Summary: Couldn't update coprocessor on HBase 2.0
Key: KYLIN-3517
URL: https://issues.apache.org/jira/browse/KYLIN-3517
Project: Kylin
Issue Type: Bug
Components: Storage - HBase
Reporter: Shaofeng SHI
On HDP 3.0, run update coprocessor, got this error:
{code:java}
2018-08-28 00:24:26,683 ERROR [pool-7-thread-1] util.DeployCoprocessorCLI:383 :
Error processing KYLIN_O9JRT8XOQ9
java.lang.UnsupportedOperationException: HTableDescriptor is read-only
at
org.apache.hadoop.hbase.client.ImmutableHTableDescriptor.getDelegateeForModification(ImmutableHTableDescriptor.java:59)
at
org.apache.hadoop.hbase.HTableDescriptor.removeCoprocessor(HTableDescriptor.java:768)
at
org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI.resetCoprocessor(DeployCoprocessorCLI.java:300)
at
org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI$ResetCoprocessorWorker.run(DeployCoprocessorCLI.java:375)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3513) Release 2.5.0
Shaofeng SHI created KYLIN-3513: --- Summary: Release 2.5.0 Key: KYLIN-3513 URL: https://issues.apache.org/jira/browse/KYLIN-3513 Project: Kylin Issue Type: Task Reporter: Shaofeng SHI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3504) Release 2.4.1
Shaofeng SHI created KYLIN-3504: --- Summary: Release 2.4.1 Key: KYLIN-3504 URL: https://issues.apache.org/jira/browse/KYLIN-3504 Project: Kylin Issue Type: Task Reporter: Shaofeng SHI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3488) Support MySQL as Kylin metadata storage
Shaofeng SHI created KYLIN-3488: --- Summary: Support MySQL as Kylin metadata storage Key: KYLIN-3488 URL: https://issues.apache.org/jira/browse/KYLIN-3488 Project: Kylin Issue Type: New Feature Components: Metadata Reporter: Shaofeng SHI Kylin uses HBase as the metastore; But in some cases user expects the metadata not in HBase. Sonny Heer from mailing list mentioned: "I'm fairly certain anyone using Kylin with AWS EMR will benefit from this. Having multiple hbase clusters across AZs is a huge benefit. BTW only thing blocking at the moment is write operations happening from kylin query nodes." -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3472) TopN merge in Spark engine is slow
Shaofeng SHI created KYLIN-3472: --- Summary: TopN merge in Spark engine is slow Key: KYLIN-3472 URL: https://issues.apache.org/jira/browse/KYLIN-3472 Project: Kylin Issue Type: Bug Components: Spark Engine Reporter: Shaofeng SHI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3464) Less user confirmation
Shaofeng SHI created KYLIN-3464: --- Summary: Less user confirmation Key: KYLIN-3464 URL: https://issues.apache.org/jira/browse/KYLIN-3464 Project: Kylin Issue Type: Improvement Components: Web Reporter: Shaofeng SHI Attachments: 2018-07-22 下午3.07.10.png Kylin web has too many confirmation box with user; I need to repeatedly click "OK" to continue. Suggest to remove unnecessary notification, or notification without confirmation. See the attachement. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3462) "dfs.replication=2" and compression not work in Spark cube engine
Shaofeng SHI created KYLIN-3462: --- Summary: "dfs.replication=2" and compression not work in Spark cube engine Key: KYLIN-3462 URL: https://issues.apache.org/jira/browse/KYLIN-3462 Project: Kylin Issue Type: Bug Components: Spark Engine Affects Versions: v2.4.0, v2.3.1, v2.3.0 Reporter: Shaofeng SHI In a comparison between Spark and MR cubing, I noticed the cuboid files that Spark engine generated is 3x lager than MR, and took 4x larger more disk on HDFS than MR. The reason is, the "dfs.replication=2" didn't work when Spark save to HDFS. And by default no compression. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3452) Optimize spark cubing memory footprint
Shaofeng SHI created KYLIN-3452: --- Summary: Optimize spark cubing memory footprint Key: KYLIN-3452 URL: https://issues.apache.org/jira/browse/KYLIN-3452 Project: Kylin Issue Type: Improvement Components: Spark Engine Reporter: Shaofeng SHI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3449) Should allow deleting a segment in NEW status
Shaofeng SHI created KYLIN-3449: --- Summary: Should allow deleting a segment in NEW status Key: KYLIN-3449 URL: https://issues.apache.org/jira/browse/KYLIN-3449 Project: Kylin Issue Type: Improvement Reporter: Shaofeng SHI Sometimes the job is dropped or not successfully created, then the new segment becomes orphan. So far Kylin doesn't allow deleting a NEW status segment, that is bad. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3446) Convert to HFile in spark reports ZK connection refused
Shaofeng SHI created KYLIN-3446:
---
Summary: Convert to HFile in spark reports ZK connection refused
Key: KYLIN-3446
URL: https://issues.apache.org/jira/browse/KYLIN-3446
Project: Kylin
Issue Type: Bug
Components: Spark Engine
Reporter: Shaofeng SHI
Fix For: v2.5.0
{code:java}
to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL
(unknown error)
2018-07-12 18:51:21,001 INFO [Scheduler 1109292714 Job
62f42193-20ff-4ca9-b898-52978a473bce-864] spark.SparkExecutable:38 : 18/07/12
18:51:21 WARN zookeeper.ClientCnxn: Session 0x0 for server null, unexpected
error, closing socket connection and attempting reconnect
2018-07-12 18:51:21,002 INFO [Scheduler 1109292714 Job
62f42193-20ff-4ca9-b898-52978a473bce-864] spark.SparkExecutable:38 :
java.net.ConnectException: Connection refused
2018-07-12 18:51:21,002 INFO [Scheduler 1109292714 Job
62f42193-20ff-4ca9-b898-52978a473bce-864] spark.SparkExecutable:38 : at
sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
2018-07-12 18:51:21,002 INFO [Scheduler 1109292714 Job
62f42193-20ff-4ca9-b898-52978a473bce-864] spark.SparkExecutable:38 : at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
2018-07-12 18:51:21,002 INFO [Scheduler 1109292714 Job
62f42193-20ff-4ca9-b898-52978a473bce-864] spark.SparkExecutable:38 : at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
2018-07-12 18:51:21,002 INFO [Scheduler 1109292714 Job
62f42193-20ff-4ca9-b898-52978a473bce-864] spark.SparkExecutable:38 : at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3443) Optimize cube job in Spark
Shaofeng SHI created KYLIN-3443: --- Summary: Optimize cube job in Spark Key: KYLIN-3443 URL: https://issues.apache.org/jira/browse/KYLIN-3443 Project: Kylin Issue Type: Improvement Components: Spark Engine Reporter: Shaofeng SHI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3442) Fact distinct columns in Spark
Shaofeng SHI created KYLIN-3442: --- Summary: Fact distinct columns in Spark Key: KYLIN-3442 URL: https://issues.apache.org/jira/browse/KYLIN-3442 Project: Kylin Issue Type: Improvement Components: Spark Engine Reporter: Shaofeng SHI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3441) Merge cube segments in Spark
Shaofeng SHI created KYLIN-3441: --- Summary: Merge cube segments in Spark Key: KYLIN-3441 URL: https://issues.apache.org/jira/browse/KYLIN-3441 Project: Kylin Issue Type: Improvement Components: Spark Engine Reporter: Shaofeng SHI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3440) Couldn't merge if there is an empty segment
Shaofeng SHI created KYLIN-3440:
---
Summary: Couldn't merge if there is an empty segment
Key: KYLIN-3440
URL: https://issues.apache.org/jira/browse/KYLIN-3440
Project: Kylin
Issue Type: Bug
Components: Web
Affects Versions: v2.4.0, v2.3.1
Reporter: Shaofeng SHI
Attachments: 2018-07-04 上午8.42.08.png
I have a cube which has empty segment (actually the segment is not empty, just
because the job counter wasn't fetched). When I try to merge the segments,
Kylin reports error, asking me if I want to continue forcely; When click "yes",
the same error be reported. From the http request, I didn't see the
"forceMergeEmptySegment" parameter was set.
{code:java}
org.apache.kylin.rest.exception.InternalErrorException: Empty cube segment
found, couldn't merge unless 'forceMergeEmptySegment' set to true:
[2012010100_2012080100, 2012080100_2018070100]
at
org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:416)
at
org.apache.kylin.rest.controller.CubeController.rebuild(CubeController.java:366)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at
org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
at
org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
at
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
at
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
at
org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
at
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
at
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at
org.springframework.web.servlet.FrameworkServlet.doPut(FrameworkServlet.java:883)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:653)
at
org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3437) Optimize HBase connection pool in Kylin side
Shaofeng SHI created KYLIN-3437: --- Summary: Optimize HBase connection pool in Kylin side Key: KYLIN-3437 URL: https://issues.apache.org/jira/browse/KYLIN-3437 Project: Kylin Issue Type: Improvement Components: Query Engine Reporter: Shaofeng SHI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3435) Only keep base cuboid files on HDFS for future merge
Shaofeng SHI created KYLIN-3435: --- Summary: Only keep base cuboid files on HDFS for future merge Key: KYLIN-3435 URL: https://issues.apache.org/jira/browse/KYLIN-3435 Project: Kylin Issue Type: Improvement Components: Job Engine Reporter: Shaofeng SHI Today Kylin keeps all cuboids data in HDFS for future merge. When doing the merge, Kylin need re-encode the dimension values with the new dictionaries, for all cuboids. If we only keep the base cuboid, lots of disk space can be saved. On merge, after merge the base cuboid, calculate others from the new base cuboid. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3432) Merge segments from HBase snapshots
Shaofeng SHI created KYLIN-3432: --- Summary: Merge segments from HBase snapshots Key: KYLIN-3432 URL: https://issues.apache.org/jira/browse/KYLIN-3432 Project: Kylin Issue Type: Improvement Components: Job Engine Reporter: Shaofeng SHI HBase introduces the MultiTableSnapshotInputFormat since 1.2 ; Kylin can merge segments with this method, so that the cuboid files can be deleted. # HBASE-13356 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3427) Convert to HFile in Spark
Shaofeng SHI created KYLIN-3427: --- Summary: Convert to HFile in Spark Key: KYLIN-3427 URL: https://issues.apache.org/jira/browse/KYLIN-3427 Project: Kylin Issue Type: Improvement Components: Spark Engine Reporter: Shaofeng SHI Some references: https://www.opencore.com/blog/2016/10/efficient-bulk-load-of-hbase-using-spark/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3422) Support multi-path of domain for kylin connection
Shaofeng SHI created KYLIN-3422: --- Summary: Support multi-path of domain for kylin connection Key: KYLIN-3422 URL: https://issues.apache.org/jira/browse/KYLIN-3422 Project: Kylin Issue Type: Improvement Components: Driver - JDBC, Driver - ODBC Reporter: Shaofeng SHI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3419) Upgrade to Java 8
Shaofeng SHI created KYLIN-3419: --- Summary: Upgrade to Java 8 Key: KYLIN-3419 URL: https://issues.apache.org/jira/browse/KYLIN-3419 Project: Kylin Issue Type: Improvement Components: Environment Reporter: Shaofeng SHI Java 7 will not be supported. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3418) User interface for hybrid model
Shaofeng SHI created KYLIN-3418: --- Summary: User interface for hybrid model Key: KYLIN-3418 URL: https://issues.apache.org/jira/browse/KYLIN-3418 Project: Kylin Issue Type: Improvement Components: Web Reporter: Shaofeng SHI Hybrid model is useful for model change. While now there is no entry for it from GUI, this makes many users don't see such feature. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3415) Remove "external" module
Shaofeng SHI created KYLIN-3415: --- Summary: Remove "external" module Key: KYLIN-3415 URL: https://issues.apache.org/jira/browse/KYLIN-3415 Project: Kylin Issue Type: Task Components: Tools, Build and Test Reporter: Shaofeng SHI Kylin's "external" module was created to build shaded jar so that it can be referenced in "core-common" module. While this function is not in real use now, and it blocks the apache release process. So we can safely remove it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3414) Optimize the cleanup of project L2 cache
Shaofeng SHI created KYLIN-3414: --- Summary: Optimize the cleanup of project L2 cache Key: KYLIN-3414 URL: https://issues.apache.org/jira/browse/KYLIN-3414 Project: Kylin Issue Type: Improvement Components: Metadata Reporter: Shaofeng SHI Whenever a cube changed, all projects' level 2 cache was cleared. This can be optimized to only flush the specific project's cache. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3408) Release 2.4.0
Shaofeng SHI created KYLIN-3408: --- Summary: Release 2.4.0 Key: KYLIN-3408 URL: https://issues.apache.org/jira/browse/KYLIN-3408 Project: Kylin Issue Type: Task Reporter: Shaofeng SHI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3383) Document on how to query Kylin from Spark
Shaofeng SHI created KYLIN-3383: --- Summary: Document on how to query Kylin from Spark Key: KYLIN-3383 URL: https://issues.apache.org/jira/browse/KYLIN-3383 Project: Kylin Issue Type: Improvement Components: Documentation Reporter: Shaofeng SHI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3382) YARN job link wasn't displayed when job is running
Shaofeng SHI created KYLIN-3382: --- Summary: YARN job link wasn't displayed when job is running Key: KYLIN-3382 URL: https://issues.apache.org/jira/browse/KYLIN-3382 Project: Kylin Issue Type: Bug Components: Spark Engine Reporter: Shaofeng SHI Affected steps: "Build cube with Spark" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3381) Stop/abort cubing job doesn't stop Spark job
Shaofeng SHI created KYLIN-3381: --- Summary: Stop/abort cubing job doesn't stop Spark job Key: KYLIN-3381 URL: https://issues.apache.org/jira/browse/KYLIN-3381 Project: Kylin Issue Type: Bug Components: Spark Engine Reporter: Shaofeng SHI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3378) Support Kafka table join with Hive tables
Shaofeng SHI created KYLIN-3378: --- Summary: Support Kafka table join with Hive tables Key: KYLIN-3378 URL: https://issues.apache.org/jira/browse/KYLIN-3378 Project: Kylin Issue Type: Improvement Components: Streaming Reporter: Shaofeng SHI At this moment, if the data source is Kafka, only 1 table allowed in the data model. In some cases, joining the kafka stream with hive lookup tables is expected -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3369) Reduce the data size sink from Kafka topic to HDFS
Shaofeng SHI created KYLIN-3369: --- Summary: Reduce the data size sink from Kafka topic to HDFS Key: KYLIN-3369 URL: https://issues.apache.org/jira/browse/KYLIN-3369 Project: Kylin Issue Type: Improvement Components: Streaming Reporter: Shaofeng SHI When building a cube from Kafka topic, the first step is to sink the Kafka data to HDFS. In today's implementation, it will persist all the fields of a message to disk. While in many cases, only a couple of fields will be needed for cubing; Today's behavior wastes network bandwidth and disk space. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3368) "/kylin/kylin_metadata/metadata/" has many gargage for spark cubing
Shaofeng SHI created KYLIN-3368: --- Summary: "/kylin/kylin_metadata/metadata/" has many gargage for spark cubing Key: KYLIN-3368 URL: https://issues.apache.org/jira/browse/KYLIN-3368 Project: Kylin Issue Type: Bug Components: Spark Engine Affects Versions: v2.3.0, v2.2.0 Reporter: Shaofeng SHI If using Spark as cube engine, Kylin will dump metadata to HDFS in "/kylin/kylin_metadata/metadata/"; As time goes on, many files were left there. They should be deleted when the job is finished. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3355) Improve the HTTP return code of Rest API
Shaofeng SHI created KYLIN-3355: --- Summary: Improve the HTTP return code of Rest API Key: KYLIN-3355 URL: https://issues.apache.org/jira/browse/KYLIN-3355 Project: Kylin Issue Type: Improvement Reporter: Shaofeng SHI Kylin returns 500 error for some invalid input; for example an invalid cube name, when the job exceeds the maximum number, etc. We need to refine the behavior; For invalid user input, return 4XX http code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3353) Merge job should not be blocked by "kylin.cube.max-building-segments"
Shaofeng SHI created KYLIN-3353: --- Summary: Merge job should not be blocked by "kylin.cube.max-building-segments" Key: KYLIN-3353 URL: https://issues.apache.org/jira/browse/KYLIN-3353 Project: Kylin Issue Type: Improvement Components: Job Engine Reporter: Shaofeng SHI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3350) Build cube incrementally from a file list
Shaofeng SHI created KYLIN-3350: --- Summary: Build cube incrementally from a file list Key: KYLIN-3350 URL: https://issues.apache.org/jira/browse/KYLIN-3350 Project: Kylin Issue Type: New Feature Components: Job Engine Reporter: Shaofeng SHI This a new requirement. In some cases, upstream components dump the new data into files, and then expect Kylin can consume data from a given file list, instead of from Hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3304) Instable UT CacheServiceTest
Shaofeng SHI created KYLIN-3304: --- Summary: Instable UT CacheServiceTest Key: KYLIN-3304 URL: https://issues.apache.org/jira/browse/KYLIN-3304 Project: Kylin Issue Type: Test Components: Tools, Build and Test Reporter: Shaofeng SHI Kylin integration test is unstable, sometimes failed at the CacheServiceTest: *00:48:11* Failed tests: *00:48:11* CacheServiceTest.testCubeCRUD:237*00:48:11* Tests in error: *00:48:11* CacheServiceTest.testMetaCRUD:316->waitForCounterAndClear:156 Runtime timeout -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3302) Upgrade commons-email
Shaofeng SHI created KYLIN-3302: --- Summary: Upgrade commons-email Key: KYLIN-3302 URL: https://issues.apache.org/jira/browse/KYLIN-3302 Project: Kylin Issue Type: Improvement Components: Integration Affects Versions: v2.3.0, v2.2.0 Reporter: Shaofeng SHI Check [CVE-2017-9801|https://nvd.nist.gov/vuln/detail/CVE-2017-9801] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3301) Upgrade opensaml
Shaofeng SHI created KYLIN-3301: --- Summary: Upgrade opensaml Key: KYLIN-3301 URL: https://issues.apache.org/jira/browse/KYLIN-3301 Project: Kylin Issue Type: Improvement Components: Integration Affects Versions: v2.3.0 Reporter: Shaofeng SHI opensaml 2.6.4 is reported with security issue (CVE-2015-1796), need upgrade. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3300) Upgrade jackson-databind
Shaofeng SHI created KYLIN-3300: --- Summary: Upgrade jackson-databind Key: KYLIN-3300 URL: https://issues.apache.org/jira/browse/KYLIN-3300 Project: Kylin Issue Type: Improvement Components: Integration Reporter: Shaofeng SHI jackson-databind 2.6.3 and 2.6.5 are reported with security issue (CVE-2017-7525), need ugprade -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3294) Remove HBaseMROutput.java, RangeKeyDistributionJob.java and other sunset classes
Shaofeng SHI created KYLIN-3294: --- Summary: Remove HBaseMROutput.java, RangeKeyDistributionJob.java and other sunset classes Key: KYLIN-3294 URL: https://issues.apache.org/jira/browse/KYLIN-3294 Project: Kylin Issue Type: Improvement Components: Job Engine Reporter: Shaofeng SHI They were legacy classes, keeping them will add maintainence effort especially when upgrade HBase version. Should delete them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3285) "Value NNN not exists" error run executing query
Shaofeng SHI created KYLIN-3285: --- Summary: "Value NNN not exists" error run executing query Key: KYLIN-3285 URL: https://issues.apache.org/jira/browse/KYLIN-3285 Project: Kylin Issue Type: Bug Components: Query Engine Affects Versions: v2.3.0 Reporter: Shaofeng SHI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3265) Add "jobSearchMode" as a condition to "/kylin/jobs" API
Shaofeng SHI created KYLIN-3265: --- Summary: Add "jobSearchMode" as a condition to "/kylin/jobs" API Key: KYLIN-3265 URL: https://issues.apache.org/jira/browse/KYLIN-3265 Project: Kylin Issue Type: Improvement Components: REST Service Affects Versions: v2.3.0 Reporter: Shaofeng SHI In kylin 2.3, the "/kylin/jobs" API will return both cubing job and checkpoint job together, this changed the API's behavior as there isn't checkpoint job before. We should keep the api only return cubing jobs to be compitable as before, and extend the API to add another condition 'jobSearchMode' to determine which kind of job should be returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3232) Need document for ops tools
Shaofeng SHI created KYLIN-3232: --- Summary: Need document for ops tools Key: KYLIN-3232 URL: https://issues.apache.org/jira/browse/KYLIN-3232 Project: Kylin Issue Type: Improvement Components: Documentation Reporter: Shaofeng SHI Assignee: Shaofeng SHI Kylin has many good ops tool, like CubeMigrationCLI, CubeMetaIngester, etc; They can greatly improve administrators productivity. But now there is no document about them (you need check the code to know what they're doing) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3231) Improve the "how to upgrade" guide
Shaofeng SHI created KYLIN-3231: --- Summary: Improve the "how to upgrade" guide Key: KYLIN-3231 URL: https://issues.apache.org/jira/browse/KYLIN-3231 Project: Kylin Issue Type: Improvement Components: Documentation Reporter: Shaofeng SHI Assignee: Shaofeng SHI Some user complains that the "How to upgrade" document is not friendly: it need provide the detailed step for each version. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3230) Need documenation for the configurations/parameters
Shaofeng SHI created KYLIN-3230: --- Summary: Need documenation for the configurations/parameters Key: KYLIN-3230 URL: https://issues.apache.org/jira/browse/KYLIN-3230 Project: Kylin Issue Type: Improvement Components: Documentation Reporter: Shaofeng SHI Assignee: Shaofeng SHI Kylin has more and more configuration parameters. But there is no a central document to explain these configurations and the impact they have. We need add such a doc for users. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3229) Update tutorial for 2.3 release
Shaofeng SHI created KYLIN-3229: --- Summary: Update tutorial for 2.3 release Key: KYLIN-3229 URL: https://issues.apache.org/jira/browse/KYLIN-3229 Project: Kylin Issue Type: Sub-task Components: Documentation Reporter: Shaofeng SHI Assignee: Shaofeng SHI The step by step tutorial on kylin website is out of date. Need be updated together with the release. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3177) Merged Streaming cube segments has no start/end time
Shaofeng SHI created KYLIN-3177: --- Summary: Merged Streaming cube segments has no start/end time Key: KYLIN-3177 URL: https://issues.apache.org/jira/browse/KYLIN-3177 Project: Kylin Issue Type: Bug Components: Job Engine, Metadata Affects Versions: v2.2.0 Reporter: Shaofeng SHI Assignee: Shaofeng SHI Create a streaming cube, define auto merge thresholds; and then build it repeatedly. When some segments be merged into a bigger segment, the new segment's start time and end time be 0 (show 1970-01-01). This is a bug similar with KYLIN-3165, introduced to Kylin 2.2 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3161) Enforce global dictionary for bitmap count distinct column
Shaofeng SHI created KYLIN-3161: --- Summary: Enforce global dictionary for bitmap count distinct column Key: KYLIN-3161 URL: https://issues.apache.org/jira/browse/KYLIN-3161 Project: Kylin Issue Type: Improvement Components: Metadata, Web Reporter: Shaofeng SHI Assignee: Shaofeng SHI For bitmap based count distinct column (as the data type is not int), a Global dictionary is required. But now user can use normal dictionary, which may cause incorrect result. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KYLIN-3145) Support Kafka JSON message whose property name includes "_"
Shaofeng SHI created KYLIN-3145:
---
Summary: Support Kafka JSON message whose property name includes
"_"
Key: KYLIN-3145
URL: https://issues.apache.org/jira/browse/KYLIN-3145
Project: Kylin
Issue Type: Improvement
Components: Streaming
Reporter: Shaofeng SHI
So far Kylin doesn't support JSON message which has property name with "_",
because that would be conflict with Kylin's logic.
For example, the JSON message is :
{code}
{
"user" : {
"first_name" : "Tom",
"age" : "20"
}
}
{code}
When map this topic to a table, the "first_name" is mapped to
"user_first_name"; When Kylin parse the message, it separates by "_" and then
try to find "user" -> "first" -> "name"; as there is no "first" property, an
error is reported.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (KYLIN-3142) Too many temp partition files left in Hadoop /tmp folder
Shaofeng SHI created KYLIN-3142: --- Summary: Too many temp partition files left in Hadoop /tmp folder Key: KYLIN-3142 URL: https://issues.apache.org/jira/browse/KYLIN-3142 Project: Kylin Issue Type: Bug Components: Job Engine Reporter: Shaofeng SHI Assignee: Shaofeng SHI I have a cluster running several streaming cubes; After months running, there are many files left in Hadoop /tmp folder, and it has impacted on the DFS's performance. After cleanup it, the cluster is much stable than before. hadoop fs -ls /tmp/ SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/hdp/2.5.4.0-121/hadoop/lib/alluxio-core-client-runtime-1.6.1-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.5.4.0-121/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.5.4.0-121/hadoop/alluxio-core-client-runtime-1.6.1-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] Found 46 items -rw-r--r-- 1 root supergroup153 2017-12-29 11:33 /tmp/partitions_02e317ac-03de-4cec-8906-6b18bc974858 -rw-r--r-- 1 root supergroup153 2017-12-29 07:03 /tmp/partitions_03108f0b-7873-4610-9346-e1c12388b229 -rw-r--r-- 1 root supergroup153 2017-12-29 03:00 /tmp/partitions_032d22ed-2ed3-401f-8601-b4b9d6f7800b -rw-r--r-- 1 root supergroup153 2017-12-29 02:41 /tmp/partitions_0869a983-4b20-40b7-a1f9-4ca9413ecd8d -rw-r--r-- 1 root supergroup153 2017-12-29 04:03 /tmp/partitions_10c47970-d4c2-4af1-819b-93717591ff5d -rw-r--r-- 1 root supergroup153 2017-12-29 06:03 /tmp/partitions_172e6684-f825-4101-8860-06aaad920c59 -rw-r--r-- 1 root supergroup153 2017-12-29 03:12 /tmp/partitions_249a224d-101d-4eff-b223-5453865e34b8 -rw-r--r-- 1 root supergroup153 2017-12-29 03:01 /tmp/partitions_24b74354-3b5d-44ad-984d-d9d7849988f9 -rw-r--r-- 1 root supergroup 1341 2017-12-29 03:08 /tmp/partitions_27c0089b-ed0f-4a37-af4a-adc686203ac3 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KYLIN-3129) Startup error on EMR 5.8+
Shaofeng SHI created KYLIN-3129: --- Summary: Startup error on EMR 5.8+ Key: KYLIN-3129 URL: https://issues.apache.org/jira/browse/KYLIN-3129 Project: Kylin Issue Type: Bug Reporter: Shaofeng SHI Assignee: Shaofeng SHI -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KYLIN-3062) Hide RAW measure
Shaofeng SHI created KYLIN-3062: --- Summary: Hide RAW measure Key: KYLIN-3062 URL: https://issues.apache.org/jira/browse/KYLIN-3062 Project: Kylin Issue Type: Task Reporter: Shaofeng SHI See the discussion here: http://apache-kylin.74782.x6.nabble.com/Discuss-Disable-hide-quot-RAW-quot-measure-in-Kylin-web-GUI-td6636.html -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KYLIN-3050) AWS Kinesis as data source
Shaofeng SHI created KYLIN-3050: --- Summary: AWS Kinesis as data source Key: KYLIN-3050 URL: https://issues.apache.org/jira/browse/KYLIN-3050 Project: Kylin Issue Type: New Feature Reporter: Shaofeng SHI -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KYLIN-3032) Cube size shows 0 but actually it isn't empty
Shaofeng SHI created KYLIN-3032: --- Summary: Cube size shows 0 but actually it isn't empty Key: KYLIN-3032 URL: https://issues.apache.org/jira/browse/KYLIN-3032 Project: Kylin Issue Type: Bug Components: Job Engine Environment: AWS EMR Reporter: Shaofeng SHI Assignee: Dong Li On EMR, if set working-dir to S3, after build the cube, will find the "Cube size" shows 0, but "Source count" is not 0. This is a bug, when working-dir isn't the same file system as Hadoop default FS, will face such problem. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KYLIN-3028) Build cube error when set S3 as working-dir
Shaofeng SHI created KYLIN-3028: --- Summary: Build cube error when set S3 as working-dir Key: KYLIN-3028 URL: https://issues.apache.org/jira/browse/KYLIN-3028 Project: Kylin Issue Type: Bug Components: Job Engine Affects Versions: v2.2.0 Reporter: Shaofeng SHI Assignee: Shaofeng SHI -- This message was sent by Atlassian JIRA (v6.4.14#64029)
