[jira] [Commented] (KYLIN-3315) allow each project to set its own source in project level override configuration

2018-05-26 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/KYLIN-3315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491894#comment-16491894
 ] 

ASF subversion and git services commented on KYLIN-3315:


Commit a3d0517e02e4928e1feb3166090a8e15bb0addb5 in kylin's branch 
refs/heads/pr140 from [~lidong_sjtu]
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=a3d0517 ]

KYLIN-3315 fix table init to avoid deadlock on project manager

Signed-off-by: shaofengshi 


> allow each project to set its own source in project level override 
> configuration
> 
>
> Key: KYLIN-3315
> URL: https://issues.apache.org/jira/browse/KYLIN-3315
> Project: Kylin
>  Issue Type: Improvement
>  Components: Metadata
>Reporter: Dong Li
>Assignee: Dong Li
>Priority: Major
> Fix For: v2.4.0
>
>
> Currently, all projects connect to the same source which is set in 
> kylin.properties with kylin.source.default property.
> It's better to allow each project to set its own source in project level 
> override configuration.
> As the result, we can have project A connects to JDBC, and project B connects 
> to Hive.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3354) KeywordDefaultDirtyHack cannot handle double-quoted defaultCatalog identifier

2018-05-26 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/KYLIN-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491893#comment-16491893
 ] 

ASF subversion and git services commented on KYLIN-3354:


Commit 4917466bf102f6234d2738ad749ca39d5757c89e in kylin's branch 
refs/heads/pr140 from [~lidong_sjtu]
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=4917466 ]

KYLIN-3354 escape double-quoted defaultCatalog keyword

Signed-off-by: shaofengshi 


> KeywordDefaultDirtyHack cannot handle double-quoted defaultCatalog identifier
> -
>
> Key: KYLIN-3354
> URL: https://issues.apache.org/jira/browse/KYLIN-3354
> Project: Kylin
>  Issue Type: Improvement
>Reporter: Dong Li
>Assignee: Dong Li
>Priority: Major
>
> for example, following SQL cannot be escaped by KeywordDefaultDirtyHack:
> {quote}
> select count(*) from "defaultCatalog"."DEFAULT"."TABLE"
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3352) Segment pruning bug, e.g. date_col > "max_date+1"

2018-05-26 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/KYLIN-3352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491892#comment-16491892
 ] 

ASF subversion and git services commented on KYLIN-3352:


Commit 1d62824fd75d3ed6e17c3ce6df9802c6e39ca9b4 in kylin's branch 
refs/heads/pr140 from Li Yang
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=1d62824 ]

KYLIN-3352 better filter transform for better seg pruning

Signed-off-by: shaofengshi 


> Segment pruning bug, e.g. date_col > "max_date+1"
> -
>
> Key: KYLIN-3352
> URL: https://issues.apache.org/jira/browse/KYLIN-3352
> Project: Kylin
>  Issue Type: Bug
>Affects Versions: v2.1.0, v2.2.0, v2.3.0
>Reporter: liyang
>Assignee: liyang
>Priority: Major
> Fix For: v2.4.0
>
>
> Currently {{date_col > "max_date+1"}} is rounded down to {{date_col > 
> "max_date"}} during encoding and further evaluated as {{date_col >= 
> "max_date"}} during segment pruning. This causes a segment can be pruned is 
> not pruned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3348) "missing LastBuildJobID" error when building new cube segment

2018-05-26 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/KYLIN-3348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491891#comment-16491891
 ] 

ASF subversion and git services commented on KYLIN-3348:


Commit 85638207f49b117f6cc019720303f5f1786f1754 in kylin's branch 
refs/heads/pr140 from Li Yang
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=8563820 ]

KYLIN-3348 fix 'missing LastBuildJobID' error

Signed-off-by: shaofengshi 


> "missing LastBuildJobID" error when building new cube segment
> -
>
> Key: KYLIN-3348
> URL: https://issues.apache.org/jira/browse/KYLIN-3348
> Project: Kylin
>  Issue Type: Bug
>Affects Versions: v2.3.0
>Reporter: liyang
>Assignee: liyang
>Priority: Major
> Fix For: v2.4.0
>
>
> An unstable exception. Likely to happen when there are multiple concurrent 
> builds.
> {{2018-04-18 20:11:16,856 ERROR [pool-33-thread-11] 
> threadpool.DefaultScheduler : ExecuteException 
> job:cc08da19-f53e-4344-a6c5-05e764834924}}
>  {{ org.apache.kylin.job.exception.ExecuteException: 
> org.apache.kylin.job.exception.ExecuteException: 
> java.lang.IllegalStateException: For cube CUBE[name=cube2], segment 
> cube2[2018041423000_2018041423001] missing LastBuildJobID}}
>  \{{ at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:140)}}
>  \{{ at 
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:307)}}
>  \{{ at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)}}
>  \{{ at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)}}
>  \{{ at java.lang.Thread.run(Thread.java:748)}}
>  {{ Caused by: org.apache.kylin.job.exception.ExecuteException: 
> java.lang.IllegalStateException: For cube CUBE[name=cube2], segment 
> cube2[2018041423000_2018041423001] missing LastBuildJobID}}
>  \{{ at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:140)}}
>  \{{ at 
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:67)}}
>  \{{ at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:129)}}
>  \{{ ... 4 more}}
>  {{ Caused by: java.lang.IllegalStateException: For cube CUBE[name=cube2], 
> segment cube2[2018041423000_2018041423001] missing LastBuildJobID}}
>  \{{ at 
> org.apache.kylin.cube.CubeManager$SegmentAssist.promoteNewlyBuiltSegments(CubeManager.java:810)}}
>  \{{ at 
> org.apache.kylin.cube.CubeManager.promoteNewlyBuiltSegments(CubeManager.java:535)}}
>  \{{ at 
> org.apache.kylin.engine.mr.steps.UpdateCubeInfoAfterBuildStep.doWork(UpdateCubeInfoAfterBuildStep.java:78)}}
>  \{{ at 
> io.kyligence.kap.engine.mr.steps.KapUpdateCubeInfoAfterBuildStep.doWork(SourceFile:47)}}
>  \{{ at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:129)}}
>  \{{ ... 6 more}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3315) allow each project to set its own source in project level override configuration

2018-05-26 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/KYLIN-3315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491890#comment-16491890
 ] 

ASF subversion and git services commented on KYLIN-3315:


Commit e4ae5e50c534eff1eee427decb66c139330c2be2 in kylin's branch 
refs/heads/pr140 from [~lidong_sjtu]
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=e4ae5e5 ]

KYLIN-3315 fix table init

Signed-off-by: shaofengshi 


> allow each project to set its own source in project level override 
> configuration
> 
>
> Key: KYLIN-3315
> URL: https://issues.apache.org/jira/browse/KYLIN-3315
> Project: Kylin
>  Issue Type: Improvement
>  Components: Metadata
>Reporter: Dong Li
>Assignee: Dong Li
>Priority: Major
> Fix For: v2.4.0
>
>
> Currently, all projects connect to the same source which is set in 
> kylin.properties with kylin.source.default property.
> It's better to allow each project to set its own source in project level 
> override configuration.
> As the result, we can have project A connects to JDBC, and project B connects 
> to Hive.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3310) Use lint for maven-compiler-plugin

2018-05-26 Thread Ted Yu (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KYLIN-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated KYLIN-3310:
--
Description: 
lint helps identify structural problems.

We should enable lint for maven-compiler-plugin
{code}
  maven-compiler-plugin
  ${maven-compiler-plugin.version}
  
1.8
1.8

  -Xlint:all
  ${compiler.error.flag}
  
  -Xlint:-options
  
  -Xlint:-cast
  -Xlint:-deprecation
  -Xlint:-processing
  -Xlint:-rawtypes
  -Xlint:-serial
  -Xlint:-try
  -Xlint:-unchecked
  -Xlint:-varargs
  
  
  

true

false
  
{code}

  was:
lint helps identify structural problems.

We should enable lint for maven-compiler-plugin

{code}
  maven-compiler-plugin
  ${maven-compiler-plugin.version}
  
1.8
1.8

  -Xlint:all
  ${compiler.error.flag}
  
  -Xlint:-options
  
  -Xlint:-cast
  -Xlint:-deprecation
  -Xlint:-processing
  -Xlint:-rawtypes
  -Xlint:-serial
  -Xlint:-try
  -Xlint:-unchecked
  -Xlint:-varargs
  
  
  

true

false
  
{code}


> Use lint for maven-compiler-plugin
> --
>
> Key: KYLIN-3310
> URL: https://issues.apache.org/jira/browse/KYLIN-3310
> Project: Kylin
>  Issue Type: Improvement
>  Components: Tools, Build and Test
>Reporter: Ted Yu
>Priority: Major
>
> lint helps identify structural problems.
> We should enable lint for maven-compiler-plugin
> {code}
>   maven-compiler-plugin
>   ${maven-compiler-plugin.version}
>   
> 1.8
> 1.8
> 
>   -Xlint:all
>   ${compiler.error.flag}
>   
>   -Xlint:-options
>   
>   -Xlint:-cast
>   -Xlint:-deprecation
>   -Xlint:-processing
>   -Xlint:-rawtypes
>   -Xlint:-serial
>   -Xlint:-try
>   -Xlint:-unchecked
>   -Xlint:-varargs
>   
>   
>   
> 
> true
> 
> false
>   
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3295) Unused method SQLDigestUtil#appendTsFilterToExecute

2018-05-26 Thread Ted Yu (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KYLIN-3295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated KYLIN-3295:
--
Description: 
SQLDigestUtil#appendTsFilterToExecute is not called anywhere.
{code}
   T ret = action.apply(null);
{code}
Passing null to {{apply}} seems incorrect.

  was:
SQLDigestUtil#appendTsFilterToExecute is not called anywhere.

{code}
   T ret = action.apply(null);
{code}
Passing null to {{apply}} seems incorrect.


> Unused method SQLDigestUtil#appendTsFilterToExecute
> ---
>
> Key: KYLIN-3295
> URL: https://issues.apache.org/jira/browse/KYLIN-3295
> Project: Kylin
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: jiatao.tao
>Priority: Minor
> Fix For: v2.4.0
>
>
> SQLDigestUtil#appendTsFilterToExecute is not called anywhere.
> {code}
>T ret = action.apply(null);
> {code}
> Passing null to {{apply}} seems incorrect.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (KYLIN-3334) Prepare for Java 10

2018-05-26 Thread Ted Yu (JIRA) 

[ 
https://issues.apache.org/jira/browse/KYLIN-3334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16473200#comment-16473200
 ] 

Ted Yu edited comment on KYLIN-3334 at 5/27/18 2:06 AM:


Compiling against jdk 11 gives similar error .


was (Author: yuzhih...@gmail.com):
Compiling against jdk 11 gives similar error.

> Prepare for Java 10
> ---
>
> Key: KYLIN-3334
> URL: https://issues.apache.org/jira/browse/KYLIN-3334
> Project: Kylin
>  Issue Type: Task
>Reporter: Ted Yu
>Priority: Major
>
> When compiling with Java 10 , MapReduce Engine module fails with
> {code}
> [ERROR] Failed to execute goal on project kylin-engine-mr: Could not resolve 
> dependencies for project org.apache.kylin:kylin-engine-mr:jar:2.4.0-SNAPSHOT: 
> Could not find artifact jdk.tools:jdk.tools:jar:1.7 at specified path 
> /a/jdk-10/../lib/tools.jar -> [Help 1]
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3373) Some improvements for lookup table - UI part change

2018-05-26 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/KYLIN-3373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491875#comment-16491875
 ] 

ASF subversion and git services commented on KYLIN-3373:


Commit 1c104d4c0d835bbfb322bc81535a5758f47bc752 in kylin's branch 
refs/heads/KYLIN-3221 from liapan
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=1c104d4 ]

KYLIN-3373 Some improvements for lookup table - UI part change - Bug fix for 
add snapshot desc

Signed-off-by: shaofengshi 


> Some improvements for lookup table - UI part change
> ---
>
> Key: KYLIN-3373
> URL: https://issues.apache.org/jira/browse/KYLIN-3373
> Project: Kylin
>  Issue Type: Sub-task
>Reporter: Ma Gang
>Assignee: Pan, Julian
>Priority: Major
>
> UI part change



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KYLIN-3363) Wrong partition condition appended in JDBC Source

2018-05-26 Thread Dong Li (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KYLIN-3363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dong Li resolved KYLIN-3363.

   Resolution: Fixed
Fix Version/s: v2.4.0

> Wrong partition condition appended in JDBC Source
> -
>
> Key: KYLIN-3363
> URL: https://issues.apache.org/jira/browse/KYLIN-3363
> Project: Kylin
>  Issue Type: Bug
>  Components: RDBMS Source
>Affects Versions: v2.3.1
>Reporter: Dong Li
>Assignee: Dong Li
>Priority: Minor
> Fix For: v2.4.0
>
>
> create a cube, and select the partition column from lookup table, then select 
> shard by column from fact table.
> then an unexecutable sql will be created to build the flatten table with 
> sqoop:
> select ... from [FACT] where [LOOKUP].[PARTITION_COL] > ...



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KYLIN-3345) Use Apache Parent POM 19

2018-05-26 Thread Dong Li (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KYLIN-3345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dong Li resolved KYLIN-3345.

   Resolution: Fixed
Fix Version/s: v2.4.0

> Use Apache Parent POM 19
> 
>
> Key: KYLIN-3345
> URL: https://issues.apache.org/jira/browse/KYLIN-3345
> Project: Kylin
>  Issue Type: Improvement
>  Components: Tools, Build and Test
>Reporter: Ted Yu
>Assignee: Dong Li
>Priority: Major
> Fix For: v2.4.0
>
>
> Kylin is still using Apache Parent POM 16. Apache Parent POM 19 is out.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KYLIN-3380) Allow to configure sqoop for jdbc source with a kylin_sqoop_conf.xml like hive

2018-05-26 Thread Dong Li (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KYLIN-3380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dong Li resolved KYLIN-3380.

   Resolution: Fixed
Fix Version/s: v2.4.0

> Allow to configure sqoop for jdbc source with a kylin_sqoop_conf.xml like hive
> --
>
> Key: KYLIN-3380
> URL: https://issues.apache.org/jira/browse/KYLIN-3380
> Project: Kylin
>  Issue Type: Improvement
>  Components: RDBMS Source
>Affects Versions: v2.3.1
>Reporter: Dong Li
>Assignee: Dong Li
>Priority: Minor
> Fix For: v2.4.0
>
>
> In hive source, users can config hive with a kylin_hive_conf.xml under conf 
> directory. 
> But for jdbc source, where we use sqoop to make flatten table, users cannot 
> customize the configuration according to the real environment. To fix this, 
> we can have a kylin_sqoop_conf.xml file under conf directory, then users can 
> specify some configurations for sqoop.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3363) Wrong partition condition appended in JDBC Source

2018-05-26 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/KYLIN-3363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491861#comment-16491861
 ] 

ASF subversion and git services commented on KYLIN-3363:


Commit faf7064c2161109eb43b2566960ea58b8556a67b in kylin's branch 
refs/heads/master from [~lidong_sjtu]
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=faf7064 ]

KYLIN-3363 fix wrong partition condition appended in JDBC Source


> Wrong partition condition appended in JDBC Source
> -
>
> Key: KYLIN-3363
> URL: https://issues.apache.org/jira/browse/KYLIN-3363
> Project: Kylin
>  Issue Type: Bug
>  Components: RDBMS Source
>Affects Versions: v2.3.1
>Reporter: Dong Li
>Assignee: Dong Li
>Priority: Minor
>
> create a cube, and select the partition column from lookup table, then select 
> shard by column from fact table.
> then an unexecutable sql will be created to build the flatten table with 
> sqoop:
> select ... from [FACT] where [LOOKUP].[PARTITION_COL] > ...



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3345) Use Apache Parent POM 19

2018-05-26 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/KYLIN-3345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491860#comment-16491860
 ] 

ASF subversion and git services commented on KYLIN-3345:


Commit e20e2443d61a92beb85f481be241032fd674ac57 in kylin's branch 
refs/heads/master from [~lidong_sjtu]
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=e20e244 ]

KYLIN-3345 Use Apache Parent POM 19


> Use Apache Parent POM 19
> 
>
> Key: KYLIN-3345
> URL: https://issues.apache.org/jira/browse/KYLIN-3345
> Project: Kylin
>  Issue Type: Improvement
>  Components: Tools, Build and Test
>Reporter: Ted Yu
>Assignee: Dong Li
>Priority: Major
>
> Kylin is still using Apache Parent POM 16. Apache Parent POM 19 is out.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3380) Allow to configure sqoop for jdbc source with a kylin_sqoop_conf.xml like hive

2018-05-26 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/KYLIN-3380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491862#comment-16491862
 ] 

ASF subversion and git services commented on KYLIN-3380:


Commit 36810892442d51a1d073817714e0b0ace9963a56 in kylin's branch 
refs/heads/master from [~lidong_sjtu]
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=3681089 ]

KYLIN-3380 JDBC source - add a configuration file for sqoop


> Allow to configure sqoop for jdbc source with a kylin_sqoop_conf.xml like hive
> --
>
> Key: KYLIN-3380
> URL: https://issues.apache.org/jira/browse/KYLIN-3380
> Project: Kylin
>  Issue Type: Improvement
>  Components: RDBMS Source
>Affects Versions: v2.3.1
>Reporter: Dong Li
>Assignee: Dong Li
>Priority: Minor
>
> In hive source, users can config hive with a kylin_hive_conf.xml under conf 
> directory. 
> But for jdbc source, where we use sqoop to make flatten table, users cannot 
> customize the configuration according to the real environment. To fix this, 
> we can have a kylin_sqoop_conf.xml file under conf directory, then users can 
> specify some configurations for sqoop.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3345) Use Apache Parent POM 19

2018-05-26 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/KYLIN-3345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491859#comment-16491859
 ] 

ASF GitHub Bot commented on KYLIN-3345:
---

lidongsjtu closed pull request #141:  KYLIN-3345KYLIN-3363 KYLIN-3380
URL: https://github.com/apache/kylin/pull/141
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java 
b/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
index 3b2883c7f9..689d08fc79 100644
--- a/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
+++ b/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
@@ -852,6 +852,10 @@ public int getSqoopMapperNum() {
 return 
Integer.parseInt(getOptional("kylin.source.jdbc.sqoop-mapper-num", "4"));
 }
 
+public Map getSqoopConfigOverride() {
+return 
getPropertiesByPrefix("kylin.source.jdbc.sqoop-config-override.");
+}
+
 public String getJdbcSourceFieldDelimiter() {
 return getOptional("kylin.source.jdbc.field-delimiter", "|");
 }
diff --git 
a/core-common/src/main/java/org/apache/kylin/common/util/HiveCmdBuilder.java 
b/core-common/src/main/java/org/apache/kylin/common/util/HiveCmdBuilder.java
index 1823dfcc30..1c023aa8aa 100644
--- a/core-common/src/main/java/org/apache/kylin/common/util/HiveCmdBuilder.java
+++ b/core-common/src/main/java/org/apache/kylin/common/util/HiveCmdBuilder.java
@@ -31,7 +31,6 @@
 public class HiveCmdBuilder {
 public static final Logger logger = 
LoggerFactory.getLogger(HiveCmdBuilder.class);
 
-public static final String HIVE_CONF_FILENAME = "kylin_hive_conf";
 static final String CREATE_HQL_TMP_FILE_TEMPLATE = "cat >%s< hiveConfiguration = loadHiveConfiguration();
 Properties ret = new Properties();
@@ -53,27 +54,38 @@ public static Properties loadHiveJDBCProperties() {
 }
 
 public static Map loadHiveConfiguration() {
-Map hiveConfProps = new HashMap<>();
-File hiveConfFile;
-String hiveConfFileName = (HIVE_CONF_FILENAME + ".xml");
+return loadXmlConfiguration(HIVE_CONF_FILENAME, true);
+}
+
+public static Map loadSqoopConfiguration() {
+return loadXmlConfiguration(SQOOP_CONF_FILENAME, false);
+}
+
+private static Map loadXmlConfiguration(String filename, 
boolean checkExist) {
+Map confProps = new HashMap<>();
+File confFile;
+String xmlFileName = filename + ".xml";
 String path = System.getProperty(KylinConfig.KYLIN_CONF);
 
 if (StringUtils.isNotEmpty(path)) {
-hiveConfFile = new File(path, hiveConfFileName);
+confFile = new File(path, xmlFileName);
 } else {
 path

[jira] [Commented] (KYLIN-3345) Use Apache Parent POM 19

2018-05-26 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/KYLIN-3345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491707#comment-16491707
 ] 

ASF GitHub Bot commented on KYLIN-3345:
---

coveralls commented on issue #141:  KYLIN-3345KYLIN-3363 KYLIN-3380
URL: https://github.com/apache/kylin/pull/141#issuecomment-392269378
 
 
   ## Pull Request Test Coverage Report for [Build 
3168](https://coveralls.io/builds/17180502)
   
   * **12** of **45**   **(26.67%)**  changed or added relevant lines in **5** 
files are covered.
   * **3** unchanged lines in **1** file lost coverage.
   * Overall coverage decreased (**-0.005%**) to **23.876%**
   
   ---
   
   |  Changes Missing Coverage | Covered Lines | Changed/Added Lines | % |
   | :-|--||---: |
   | 
[core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java](https://coveralls.io/builds/17180502/source?filename=core-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcommon%2FKylinConfigBase.java#L856)
 | 0 | 1 | 0.0%
   | 
[source-hive/src/main/java/org/apache/kylin/source/hive/BeelineHiveClient.java](https://coveralls.io/builds/17180502/source?filename=source-hive%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fsource%2Fhive%2FBeelineHiveClient.java#L63)
 | 0 | 1 | 0.0%
   | 
[core-common/src/main/java/org/apache/kylin/common/util/SourceConfigurationUtil.java](https://coveralls.io/builds/17180502/source?filename=core-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcommon%2Futil%2FSourceConfigurationUtil.java#L39)
 | 11 | 18 | 61.11%
   | 
[source-hive/src/main/java/org/apache/kylin/source/jdbc/JdbcHiveMRInput.java](https://coveralls.io/builds/17180502/source?filename=source-hive%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fsource%2Fjdbc%2FJdbcHiveMRInput.java#L158)
 | 0 | 24 | 0.0%
   
   
   |  Files with Coverage Reduction | New Missed Lines | % |
   | :-|--|--: |
   | 
[source-hive/src/main/java/org/apache/kylin/source/jdbc/JdbcHiveMRInput.java](https://coveralls.io/builds/17180502/source?filename=source-hive%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fsource%2Fjdbc%2FJdbcHiveMRInput.java#L45)
 | 3 | 0.0% |
   
   
   |  Totals | [![Coverage 
Status](https://coveralls.io/builds/17180502/badge)](https://coveralls.io/builds/17180502)
 |
   | :-- | --: |
   | Change from base [Build 3162](https://coveralls.io/builds/17162746): |  
-0.005% |
   | Covered Lines: | 14247 |
   | Relevant Lines: | 59671 |
   
   ---
   #   - [Coveralls](https://coveralls.io)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Use Apache Parent POM 19
> 
>
> Key: KYLIN-3345
> URL: https://issues.apache.org/jira/browse/KYLIN-3345
> Project: Kylin
>  Issue Type: Improvement
>  Components: Tools, Build and Test
>Reporter: Ted Yu
>Assignee: Dong Li
>Priority: Major
>
> Kylin is still using Apache Parent POM 16. Apache Parent POM 19 is out.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3345) Use Apache Parent POM 19

2018-05-26 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/KYLIN-3345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491701#comment-16491701
 ] 

ASF GitHub Bot commented on KYLIN-3345:
---

codecov-io commented on issue #141:  KYLIN-3345KYLIN-3363 KYLIN-3380
URL: https://github.com/apache/kylin/pull/141#issuecomment-392268993
 
 
   # [Codecov](https://codecov.io/gh/apache/kylin/pull/141?src=pr=h1) Report
   > Merging [#141](https://codecov.io/gh/apache/kylin/pull/141?src=pr=desc) 
into 
[master](https://codecov.io/gh/apache/kylin/commit/877cab0ca3089cfcccfa2bebcf211322a63e2a57?src=pr=desc)
 will **decrease** coverage by `<.01%`.
   > The diff coverage is `24.44%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/kylin/pull/141/graphs/tree.svg?height=150=650=pr=JawVgbgsVo)](https://codecov.io/gh/apache/kylin/pull/141?src=pr=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master #141  +/-   ##
   
   - Coverage 21.86%   21.85%   -0.01% 
   - Complexity 3879 3880   +1 
   
 Files   985  985  
 Lines 5965159671  +20 
 Branches   8608 8614   +6 
   
   + Hits  1304113042   +1 
   - Misses4540845426  +18 
   - Partials   1202 1203   +1
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/kylin/pull/141?src=pr=tree) | Coverage Δ 
| Complexity Δ | |
   |---|---|---|---|
   | 
[.../org/apache/kylin/source/jdbc/JdbcHiveMRInput.java](https://codecov.io/gh/apache/kylin/pull/141/diff?src=pr=tree#diff-c291cmNlLWhpdmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3NvdXJjZS9qZGJjL0pkYmNIaXZlTVJJbnB1dC5qYXZh)
 | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[...rg/apache/kylin/source/hive/BeelineHiveClient.java](https://codecov.io/gh/apache/kylin/pull/141/diff?src=pr=tree#diff-c291cmNlLWhpdmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3NvdXJjZS9oaXZlL0JlZWxpbmVIaXZlQ2xpZW50LmphdmE=)
 | `4.68% <0%> (ø)` | `5 <0> (ø)` | :arrow_down: |
   | 
[.../java/org/apache/kylin/common/KylinConfigBase.java](https://codecov.io/gh/apache/kylin/pull/141/diff?src=pr=tree#diff-Y29yZS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2NvbW1vbi9LeWxpbkNvbmZpZ0Jhc2UuamF2YQ==)
 | `14.83% <0%> (-0.04%)` | `35 <0> (ø)` | |
   | 
[...a/org/apache/kylin/common/util/HiveCmdBuilder.java](https://codecov.io/gh/apache/kylin/pull/141/diff?src=pr=tree#diff-Y29yZS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2NvbW1vbi91dGlsL0hpdmVDbWRCdWlsZGVyLmphdmE=)
 | `83.33% <100%> (ø)` | `12 <0> (ø)` | :arrow_down: |
   | 
[...che/kylin/common/util/SourceConfigurationUtil.java](https://codecov.io/gh/apache/kylin/pull/141/diff?src=pr=tree#diff-Y29yZS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2NvbW1vbi91dGlsL1NvdXJjZUNvbmZpZ3VyYXRpb25VdGlsLmphdmE=)
 | `63.41% <55.55%> (ø)` | `7 <5> (?)` | |
   | 
[...a/org/apache/kylin/dict/Number2BytesConverter.java](https://codecov.io/gh/apache/kylin/pull/141/diff?src=pr=tree#diff-Y29yZS1kaWN0aW9uYXJ5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9kaWN0L051bWJlcjJCeXRlc0NvbnZlcnRlci5qYXZh)
 | `81.74% <0%> (-0.8%)` | `17% <0%> (-1%)` | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/kylin/pull/141?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/kylin/pull/141?src=pr=footer). Last 
update 
[877cab0...cfd022e](https://codecov.io/gh/apache/kylin/pull/141?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Use Apache Parent POM 19
> 
>
> Key: KYLIN-3345
> URL: https://issues.apache.org/jira/browse/KYLIN-3345
> Project: Kylin
>  Issue Type: Improvement
>  Components: Tools, Build and Test
>Reporter: Ted Yu
>Assignee: Dong Li
>Priority: Major
>
> Kylin is still using Apache Parent POM 16. Apache Parent POM 19 is out.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3345) Use Apache Parent POM 19

2018-05-26 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/KYLIN-3345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491693#comment-16491693
 ] 

ASF GitHub Bot commented on KYLIN-3345:
---

lidongsjtu opened a new pull request #141:  KYLIN-3345KYLIN-3363 
KYLIN-3380
URL: https://github.com/apache/kylin/pull/141
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Use Apache Parent POM 19
> 
>
> Key: KYLIN-3345
> URL: https://issues.apache.org/jira/browse/KYLIN-3345
> Project: Kylin
>  Issue Type: Improvement
>  Components: Tools, Build and Test
>Reporter: Ted Yu
>Assignee: Dong Li
>Priority: Major
>
> Kylin is still using Apache Parent POM 16. Apache Parent POM 19 is out.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3345) Use Apache Parent POM 19

2018-05-26 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/KYLIN-3345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491694#comment-16491694
 ] 

ASF GitHub Bot commented on KYLIN-3345:
---

asfgit commented on issue #141:  KYLIN-3345KYLIN-3363 KYLIN-3380
URL: https://github.com/apache/kylin/pull/141#issuecomment-392267519
 
 
   Can one of the admins verify this patch?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Use Apache Parent POM 19
> 
>
> Key: KYLIN-3345
> URL: https://issues.apache.org/jira/browse/KYLIN-3345
> Project: Kylin
>  Issue Type: Improvement
>  Components: Tools, Build and Test
>Reporter: Ted Yu
>Assignee: Dong Li
>Priority: Major
>
> Kylin is still using Apache Parent POM 16. Apache Parent POM 19 is out.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3359) Support sum(expression) if possible

2018-05-26 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/KYLIN-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491648#comment-16491648
 ] 

ASF subversion and git services commented on KYLIN-3359:


Commit 28df046a00b7bad9fc71fdb2642d811ba94de237 in kylin's branch 
refs/heads/KYLIN-3359 from Zhong
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=28df046 ]

APACHE-KYLIN-3359 minor fix

Signed-off-by: shaofengshi 


> Support sum(expression) if possible
> ---
>
> Key: KYLIN-3359
> URL: https://issues.apache.org/jira/browse/KYLIN-3359
> Project: Kylin
>  Issue Type: Sub-task
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
>Priority: Major
> Attachments: KYLIN-3359-Hive-query.png, KYLIN-3359-Kylin-query.png
>
>
> The expression can be as follows:
>  # a ~1~*col ~1~ + a ~2~*col ~2~ + ... + a ~n~*col ~n~ + b, if sum(col 
> ~1~),sum(col ~2~),...sum(col ~n~) are defined
>  # case when {{filter}} ~1~ then expr ~1~
>  when {{filter}} ~2~ then expr ~2~
>  ...
>  else expr ~N~
>  end, if {{filter}} ~1~,{{filter}} ~2~, ... {{filter}} ~N-1~, and expr 
> ~1~,expr ~2~,...expr ~N~ are supported 
> There's a constraint for the filter. That is it's able to push down the 
> related filters in case when.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3348) "missing LastBuildJobID" error when building new cube segment

2018-05-26 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/KYLIN-3348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491616#comment-16491616
 ] 

ASF subversion and git services commented on KYLIN-3348:


Commit 14b59e8e624c92bfd6b36dc183e34d82fa398ff8 in kylin's branch 
refs/heads/sync from Li Yang
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=14b59e8 ]

KYLIN-3348 fix 'missing LastBuildJobID' error


> "missing LastBuildJobID" error when building new cube segment
> -
>
> Key: KYLIN-3348
> URL: https://issues.apache.org/jira/browse/KYLIN-3348
> Project: Kylin
>  Issue Type: Bug
>Affects Versions: v2.3.0
>Reporter: liyang
>Assignee: liyang
>Priority: Major
> Fix For: v2.4.0
>
>
> An unstable exception. Likely to happen when there are multiple concurrent 
> builds.
> {{2018-04-18 20:11:16,856 ERROR [pool-33-thread-11] 
> threadpool.DefaultScheduler : ExecuteException 
> job:cc08da19-f53e-4344-a6c5-05e764834924}}
>  {{ org.apache.kylin.job.exception.ExecuteException: 
> org.apache.kylin.job.exception.ExecuteException: 
> java.lang.IllegalStateException: For cube CUBE[name=cube2], segment 
> cube2[2018041423000_2018041423001] missing LastBuildJobID}}
>  \{{ at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:140)}}
>  \{{ at 
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:307)}}
>  \{{ at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)}}
>  \{{ at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)}}
>  \{{ at java.lang.Thread.run(Thread.java:748)}}
>  {{ Caused by: org.apache.kylin.job.exception.ExecuteException: 
> java.lang.IllegalStateException: For cube CUBE[name=cube2], segment 
> cube2[2018041423000_2018041423001] missing LastBuildJobID}}
>  \{{ at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:140)}}
>  \{{ at 
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:67)}}
>  \{{ at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:129)}}
>  \{{ ... 4 more}}
>  {{ Caused by: java.lang.IllegalStateException: For cube CUBE[name=cube2], 
> segment cube2[2018041423000_2018041423001] missing LastBuildJobID}}
>  \{{ at 
> org.apache.kylin.cube.CubeManager$SegmentAssist.promoteNewlyBuiltSegments(CubeManager.java:810)}}
>  \{{ at 
> org.apache.kylin.cube.CubeManager.promoteNewlyBuiltSegments(CubeManager.java:535)}}
>  \{{ at 
> org.apache.kylin.engine.mr.steps.UpdateCubeInfoAfterBuildStep.doWork(UpdateCubeInfoAfterBuildStep.java:78)}}
>  \{{ at 
> io.kyligence.kap.engine.mr.steps.KapUpdateCubeInfoAfterBuildStep.doWork(SourceFile:47)}}
>  \{{ at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:129)}}
>  \{{ ... 6 more}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3315) allow each project to set its own source in project level override configuration

2018-05-26 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/KYLIN-3315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491619#comment-16491619
 ] 

ASF subversion and git services commented on KYLIN-3315:


Commit bb35118c9febb477ebaf274c06e2d508c551ae32 in kylin's branch 
refs/heads/sync from [~lidong_sjtu]
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=bb35118 ]

KYLIN-3315 fix table init to avoid deadlock on project manager


> allow each project to set its own source in project level override 
> configuration
> 
>
> Key: KYLIN-3315
> URL: https://issues.apache.org/jira/browse/KYLIN-3315
> Project: Kylin
>  Issue Type: Improvement
>  Components: Metadata
>Reporter: Dong Li
>Assignee: Dong Li
>Priority: Major
> Fix For: v2.4.0
>
>
> Currently, all projects connect to the same source which is set in 
> kylin.properties with kylin.source.default property.
> It's better to allow each project to set its own source in project level 
> override configuration.
> As the result, we can have project A connects to JDBC, and project B connects 
> to Hive.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3354) KeywordDefaultDirtyHack cannot handle double-quoted defaultCatalog identifier

2018-05-26 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/KYLIN-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491618#comment-16491618
 ] 

ASF subversion and git services commented on KYLIN-3354:


Commit 52657a94cde85139a2bebac7e5166080e371badb in kylin's branch 
refs/heads/sync from [~lidong_sjtu]
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=52657a9 ]

KYLIN-3354 escape double-quoted defaultCatalog keyword


> KeywordDefaultDirtyHack cannot handle double-quoted defaultCatalog identifier
> -
>
> Key: KYLIN-3354
> URL: https://issues.apache.org/jira/browse/KYLIN-3354
> Project: Kylin
>  Issue Type: Improvement
>Reporter: Dong Li
>Assignee: Dong Li
>Priority: Major
>
> for example, following SQL cannot be escaped by KeywordDefaultDirtyHack:
> {quote}
> select count(*) from "defaultCatalog"."DEFAULT"."TABLE"
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3315) allow each project to set its own source in project level override configuration

2018-05-26 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/KYLIN-3315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491615#comment-16491615
 ] 

ASF subversion and git services commented on KYLIN-3315:


Commit 0b2e794fe79d461f84852a3dc98f4429cfb5c3c1 in kylin's branch 
refs/heads/sync from [~lidong_sjtu]
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=0b2e794 ]

KYLIN-3315 fix table init


> allow each project to set its own source in project level override 
> configuration
> 
>
> Key: KYLIN-3315
> URL: https://issues.apache.org/jira/browse/KYLIN-3315
> Project: Kylin
>  Issue Type: Improvement
>  Components: Metadata
>Reporter: Dong Li
>Assignee: Dong Li
>Priority: Major
> Fix For: v2.4.0
>
>
> Currently, all projects connect to the same source which is set in 
> kylin.properties with kylin.source.default property.
> It's better to allow each project to set its own source in project level 
> override configuration.
> As the result, we can have project A connects to JDBC, and project B connects 
> to Hive.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3352) Segment pruning bug, e.g. date_col > "max_date+1"

2018-05-26 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/KYLIN-3352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491617#comment-16491617
 ] 

ASF subversion and git services commented on KYLIN-3352:


Commit 5599b9aea19759481b6412c170f43676793e8792 in kylin's branch 
refs/heads/sync from Li Yang
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=5599b9a ]

KYLIN-3352 better filter transform for better seg pruning


> Segment pruning bug, e.g. date_col > "max_date+1"
> -
>
> Key: KYLIN-3352
> URL: https://issues.apache.org/jira/browse/KYLIN-3352
> Project: Kylin
>  Issue Type: Bug
>Affects Versions: v2.1.0, v2.2.0, v2.3.0
>Reporter: liyang
>Assignee: liyang
>Priority: Major
> Fix For: v2.4.0
>
>
> Currently {{date_col > "max_date+1"}} is rounded down to {{date_col > 
> "max_date"}} during encoding and further evaluated as {{date_col >= 
> "max_date"}} during segment pruning. This causes a segment can be pruned is 
> not pruned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KYLIN-1729) Error at #3 Step Name: Build Dimension Dictionary Duration

2018-05-26 Thread liyang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KYLIN-1729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

liyang resolved KYLIN-1729.
---
   Resolution: Fixed
 Assignee: liyang  (was: Dong Li)
Fix Version/s: v2.0.0

This has been fixed long ago, however I cannot recall the exact fix version..

> Error at  #3 Step Name: Build Dimension Dictionary Duration
> ---
>
> Key: KYLIN-1729
> URL: https://issues.apache.org/jira/browse/KYLIN-1729
> Project: Kylin
>  Issue Type: Bug
>  Components: Job Engine
>Affects Versions: 1.x-HBase1.1.3, v1.5.0
>Reporter: Chaozhong Yang
>Assignee: liyang
>Priority: Major
> Fix For: v2.0.0
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> stack trace as follows:
> java.lang.NegativeArraySizeException
> at 
> org.apache.kylin.dict.TrieDictionaryBuilder.buildTrieBytes(TrieDictionaryBuilder.java:472)
> at 
> org.apache.kylin.dict.TrieDictionaryBuilder.build(TrieDictionaryBuilder.java:439)
> at 
> org.apache.kylin.dict.DictionaryGenerator.buildStringDict(DictionaryGenerator.java:163)
> at 
> org.apache.kylin.dict.DictionaryGenerator.buildDictionaryFromValueEnumerator(DictionaryGenerator.java:75)
> at 
> org.apache.kylin.dict.DictionaryGenerator.buildDictionary(DictionaryGenerator.java:108)
> at 
> org.apache.kylin.dict.DictionaryManager.buildDictionary(DictionaryManager.java:300)
> at org.apache.kylin.cube.CubeManager.buildDictionary(CubeManager.java:165)
> at 
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:51)
> at 
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:42)
> at 
> org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:56)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at 
> org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:60)
> at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
> at 
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50)
> at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
> at 
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:124)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> result code:2



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-2662) NegativeArraySizeException in "Extract Fact Table Distinct Columns"

2018-05-26 Thread liyang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KYLIN-2662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

liyang updated KYLIN-2662:
--
Fix Version/s: v2.4.0

> NegativeArraySizeException in "Extract Fact Table Distinct Columns"
> ---
>
> Key: KYLIN-2662
> URL: https://issues.apache.org/jira/browse/KYLIN-2662
> Project: Kylin
>  Issue Type: Bug
>Reporter: liyang
>Assignee: liyang
>Priority: Major
>  Labels: scope
> Fix For: v2.4.0
>
>
> The full exception. The root cause suspect to be a cell goes over 32 KB. Need 
> verification and fix.
> {code}
> java.lang.NegativeArraySizeException
> at 
> org.apache.kylin.dict.TrieDictionary.getValueBytesFromIdWithoutCache(SourceFile:239)
> at 
> org.apache.kylin.dict.TrieDictionaryForestBuilder.addTree(SourceFile:134)
> at 
> org.apache.kylin.dict.TrieDictionaryForestBuilder.build(SourceFile:110)
> at 
> org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder.build(SourceFile:218)
> at 
> org.apache.kylin.engine.mr.steps.FactDistinctColumnsReducer.doCleanup(SourceFile:231)
> at 
> org.apache.kylin.engine.mr.KylinReducer.cleanup(SourceFile:71)
> at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:179)
> at 
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)
> at 
> org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
> at 
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (KYLIN-1729) Error at #3 Step Name: Build Dimension Dictionary Duration

2018-05-26 Thread liyang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KYLIN-1729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

liyang closed KYLIN-1729.
-

> Error at  #3 Step Name: Build Dimension Dictionary Duration
> ---
>
> Key: KYLIN-1729
> URL: https://issues.apache.org/jira/browse/KYLIN-1729
> Project: Kylin
>  Issue Type: Bug
>  Components: Job Engine
>Affects Versions: 1.x-HBase1.1.3, v1.5.0
>Reporter: Chaozhong Yang
>Assignee: liyang
>Priority: Major
> Fix For: v2.0.0
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> stack trace as follows:
> java.lang.NegativeArraySizeException
> at 
> org.apache.kylin.dict.TrieDictionaryBuilder.buildTrieBytes(TrieDictionaryBuilder.java:472)
> at 
> org.apache.kylin.dict.TrieDictionaryBuilder.build(TrieDictionaryBuilder.java:439)
> at 
> org.apache.kylin.dict.DictionaryGenerator.buildStringDict(DictionaryGenerator.java:163)
> at 
> org.apache.kylin.dict.DictionaryGenerator.buildDictionaryFromValueEnumerator(DictionaryGenerator.java:75)
> at 
> org.apache.kylin.dict.DictionaryGenerator.buildDictionary(DictionaryGenerator.java:108)
> at 
> org.apache.kylin.dict.DictionaryManager.buildDictionary(DictionaryManager.java:300)
> at org.apache.kylin.cube.CubeManager.buildDictionary(CubeManager.java:165)
> at 
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:51)
> at 
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:42)
> at 
> org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:56)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at 
> org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:60)
> at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
> at 
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50)
> at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
> at 
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:124)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> result code:2



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (KYLIN-2662) NegativeArraySizeException in "Extract Fact Table Distinct Columns"

2018-05-26 Thread liyang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KYLIN-2662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

liyang reassigned KYLIN-2662:
-

Assignee: liyang

> NegativeArraySizeException in "Extract Fact Table Distinct Columns"
> ---
>
> Key: KYLIN-2662
> URL: https://issues.apache.org/jira/browse/KYLIN-2662
> Project: Kylin
>  Issue Type: Bug
>Reporter: liyang
>Assignee: liyang
>Priority: Major
>  Labels: scope
> Fix For: v2.4.0
>
>
> The full exception. The root cause suspect to be a cell goes over 32 KB. Need 
> verification and fix.
> {code}
> java.lang.NegativeArraySizeException
> at 
> org.apache.kylin.dict.TrieDictionary.getValueBytesFromIdWithoutCache(SourceFile:239)
> at 
> org.apache.kylin.dict.TrieDictionaryForestBuilder.addTree(SourceFile:134)
> at 
> org.apache.kylin.dict.TrieDictionaryForestBuilder.build(SourceFile:110)
> at 
> org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder.build(SourceFile:218)
> at 
> org.apache.kylin.engine.mr.steps.FactDistinctColumnsReducer.doCleanup(SourceFile:231)
> at 
> org.apache.kylin.engine.mr.KylinReducer.cleanup(SourceFile:71)
> at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:179)
> at 
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)
> at 
> org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
> at 
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)