[jira] [Created] (KYLIN-4000) Merge task hang at

[Tuo Zhu (JIRA)]

Tuo Zhu created KYLIN-4000:
--

 Summary: Merge task hang at 
 Key: KYLIN-4000
 URL: https://issues.apache.org/jira/browse/KYLIN-4000
 Project: Kylin
  Issue Type: Bug
  Components: Job Engine
Affects Versions: v2.6.1
Reporter: Tuo Zhu


Auto triggered merge tasks all hanged at step: Convert Cuboid Data to HFile. 
The last reduce step in "Kylin_HFile_Generator_mycube_Step".

Hadoop UI shows job status:
 attempt_1555603539483_1251_r_00_0 78.56 RUNNING reduce > reduce

>From the syslog, following outputs are printed at first few hours:

{code}
 2019-05-10 22:27:48,699 INFO [IPC Server handler 27 on 46084] 
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt 
attempt_1555603539483_1149_r_00_0 is : 0.8629564
 2019-05-10 22:28:24,743 INFO [Socket Reader #1 for port 46084|#1 for port 
46084] SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for 
job_1555603539483_1149 (auth:SIMPLE)
 2019-05-10 22:28:24,746 INFO [IPC Server handler 23 on 46084] 
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt 
attempt_1555603539483_1149_r_00_0 is : 0.86303306
 2019-05-10 22:29:00,794 INFO [Socket Reader #1 for port 46084|#1 for port 
46084] SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for 
job_1555603539483_1149 (auth:SIMPLE)
 2019-05-10 22:29:00,796 INFO [IPC Server handler 20 on 46084] 
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt 
attempt_1555603539483_1149_r_00_0 is : 0.86305153
 2019-05-10 22:29:36,833 INFO [Socket Reader #1 for port 46084|#1 for port 
46084] SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for 
job_1555603539483_1149 (auth:SIMPLE)
 2019-05-10 22:29:36,836 INFO [IPC Server handler 3 on 46084] 
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt 
attempt_1555603539483_1149_r_00_0 is : 0.8634469
 2019-05-10 22:30:12,883 INFO [Socket Reader #1 for port 46084|#1 for port 
46084] SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for 
job_1555603539483_1149 (auth:SIMPLE)
 2019-05-10 22:30:12,887 INFO [IPC Server handler 18 on 46084] 
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt 
attempt_1555603539483_1149_r_00_0 is : 0.8643631
{code}


 After that, only ZK logs keeps printed following lines constantly:

{code}
2019-05-11 00:02:23,445 WARN [Thread-1388778] 
org.apache.hadoop.hdfs.DataStreamer: These favored nodes were specified but not 
chosen: [fluorine:16020] Specified favored nodes: [fluorine:16020]
 2019-05-11 00:02:23,452 INFO [main] 
org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient: Connect 0x29758ab9 to 
boron:2181,carbon:2181,neon:2181 with session timeout=9ms, retries 6, retry 
interval 1000ms, keepAlive=6ms
 2019-05-11 00:02:23,452 INFO 
[ReadOnlyZKClient-boron:2181,carbon:2181,neon:2181@0x29758ab9] 
org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=bo
 ron:2181,carbon:2181,neon:2181 sessionTimeout=9 
watcher=org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$24/171414890@19fbce98
 2019-05-11 00:02:23,452 INFO 
[ReadOnlyZKClient-boron:2181,carbon:2181,neon:2181@0x29758ab9-SendThread(boron:2181)]
 org.apache.zookeeper.ClientCnxn: Opening socket connection to server 
boron/172.16.211.20:2181. Will not attempt to authenticate using SASL (unknown 
error)
 2019-05-11 00:02:23,453 INFO 
[ReadOnlyZKClient-boron:2181,carbon:2181,neon:2181@0x29758ab9-SendThread(boron:2181)]
 org.apache.zookeeper.ClientCnxn: Socket connection established, initiating 
session, client: /172.16.211.8:52214, server: boron/172.16.211.20:2181
 2019-05-11 00:02:23,453 INFO 
[ReadOnlyZKClient-boron:2181,carbon:2181,neon:2181@0x29758ab9-SendThread(boron:2181)]
 org.apache.zookeeper.ClientCnxn: Session establishment complete on server 
boron/172.16.211.20:2181, sessionid = 0x16a061f0770a354, negotiated timeout = 
6
 2019-05-11 00:02:23,455 INFO [main] 
org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient: Close zookeeper connection 
0x29758ab9 to boron:2181,carbon:2181,neon:2181
 2019-05-11 00:02:23,455 INFO [main] 
org.apache.hadoop.hbase.io.hfile.CacheConfig: Created cacheConfig: 
CacheConfig:disabled
 2019-05-11 00:02:23,456 INFO [main] 
org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2: 
Writer=hdfs://berylium:8020/kylin/kylin_metadata/kylin-772502e8-c126-fec2-5808-08aa2df1fdea/pbs_uv_daily/hfile/_temporary/1/_temporary/attempt_1555603539483_1149_r_00_0/F1/fdbf1bd8933242f2bd1da40a03c7a635,
 wrote=43
 2019-05-11 00:02:23,457 WARN [Thread-1388781] 
org.apache.hadoop.hdfs.DataStreamer: These favored nodes were specified but not 
chosen: [fluorine:16020] Specified favored nodes: [fluorine:16020]
{code}

It lasts for tens of hours. No error, no exceptions. In the end, it created too 
many blocks in HDFS. It only happens when merge task is triggered. Daily build 
jobs went 

[jira] [Updated] (KYLIN-4000) Merge task hang at step #5: Convert Cuboid Data to HFile

[Tuo Zhu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-4000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tuo Zhu updated KYLIN-4000:
---
Summary: Merge task hang at step #5: Convert Cuboid Data to HFile  (was: 
Merge task hang at )

> Merge task hang at step #5: Convert Cuboid Data to HFile
> 
>
> Key: KYLIN-4000
> URL: https://issues.apache.org/jira/browse/KYLIN-4000
> Project: Kylin
>  Issue Type: Bug
>  Components: Job Engine
>Affects Versions: v2.6.1
>Reporter: Tuo Zhu
>Priority: Blocker
>
> Auto triggered merge tasks all hanged at step: Convert Cuboid Data to HFile. 
> The last reduce step in "Kylin_HFile_Generator_mycube_Step".
> Hadoop UI shows job status:
>  attempt_1555603539483_1251_r_00_0 78.56 RUNNING reduce > reduce
> From the syslog, following outputs are printed at first few hours:
> {code}
>  2019-05-10 22:27:48,699 INFO [IPC Server handler 27 on 46084] 
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt 
> attempt_1555603539483_1149_r_00_0 is : 0.8629564
>  2019-05-10 22:28:24,743 INFO [Socket Reader #1 for port 46084|#1 for port 
> 46084] SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for 
> job_1555603539483_1149 (auth:SIMPLE)
>  2019-05-10 22:28:24,746 INFO [IPC Server handler 23 on 46084] 
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt 
> attempt_1555603539483_1149_r_00_0 is : 0.86303306
>  2019-05-10 22:29:00,794 INFO [Socket Reader #1 for port 46084|#1 for port 
> 46084] SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for 
> job_1555603539483_1149 (auth:SIMPLE)
>  2019-05-10 22:29:00,796 INFO [IPC Server handler 20 on 46084] 
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt 
> attempt_1555603539483_1149_r_00_0 is : 0.86305153
>  2019-05-10 22:29:36,833 INFO [Socket Reader #1 for port 46084|#1 for port 
> 46084] SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for 
> job_1555603539483_1149 (auth:SIMPLE)
>  2019-05-10 22:29:36,836 INFO [IPC Server handler 3 on 46084] 
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt 
> attempt_1555603539483_1149_r_00_0 is : 0.8634469
>  2019-05-10 22:30:12,883 INFO [Socket Reader #1 for port 46084|#1 for port 
> 46084] SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for 
> job_1555603539483_1149 (auth:SIMPLE)
>  2019-05-10 22:30:12,887 INFO [IPC Server handler 18 on 46084] 
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt 
> attempt_1555603539483_1149_r_00_0 is : 0.8643631
> {code}
>  After that, only ZK logs keeps printed following lines constantly:
> {code}
> 2019-05-11 00:02:23,445 WARN [Thread-1388778] 
> org.apache.hadoop.hdfs.DataStreamer: These favored nodes were specified but 
> not chosen: [fluorine:16020] Specified favored nodes: [fluorine:16020]
>  2019-05-11 00:02:23,452 INFO [main] 
> org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient: Connect 0x29758ab9 to 
> boron:2181,carbon:2181,neon:2181 with session timeout=9ms, retries 6, 
> retry interval 1000ms, keepAlive=6ms
>  2019-05-11 00:02:23,452 INFO 
> [ReadOnlyZKClient-boron:2181,carbon:2181,neon:2181@0x29758ab9] 
> org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=bo
>  ron:2181,carbon:2181,neon:2181 sessionTimeout=9 
> watcher=org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$24/171414890@19fbce98
>  2019-05-11 00:02:23,452 INFO 
> [ReadOnlyZKClient-boron:2181,carbon:2181,neon:2181@0x29758ab9-SendThread(boron:2181)]
>  org.apache.zookeeper.ClientCnxn: Opening socket connection to server 
> boron/172.16.211.20:2181. Will not attempt to authenticate using SASL 
> (unknown error)
>  2019-05-11 00:02:23,453 INFO 
> [ReadOnlyZKClient-boron:2181,carbon:2181,neon:2181@0x29758ab9-SendThread(boron:2181)]
>  org.apache.zookeeper.ClientCnxn: Socket connection established, initiating 
> session, client: /172.16.211.8:52214, server: boron/172.16.211.20:2181
>  2019-05-11 00:02:23,453 INFO 
> [ReadOnlyZKClient-boron:2181,carbon:2181,neon:2181@0x29758ab9-SendThread(boron:2181)]
>  org.apache.zookeeper.ClientCnxn: Session establishment complete on server 
> boron/172.16.211.20:2181, sessionid = 0x16a061f0770a354, negotiated timeout = 
> 6
>  2019-05-11 00:02:23,455 INFO [main] 
> org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient: Close zookeeper 
> connection 0x29758ab9 to boron:2181,carbon:2181,neon:2181
>  2019-05-11 00:02:23,455 INFO [main] 
> org.apache.hadoop.hbase.io.hfile.CacheConfig: Created cacheConfig: 
> CacheConfig:disabled
>  2019-05-11 00:02:23,456 INFO [main] 
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2: 
> 

[jira] [Commented] (KYLIN-3934) sqoop import param '--null-string' result in null value become blank string in hive table

[nichunen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16837096#comment-16837096
 ] 

nichunen commented on KYLIN-3934:
-

Committed to 2.6.x branch

> sqoop import param '--null-string' result in null value become blank string 
> in hive table
> -
>
> Key: KYLIN-3934
> URL: https://issues.apache.org/jira/browse/KYLIN-3934
> Project: Kylin
>  Issue Type: Bug
>  Components: Others
>Affects Versions: v2.6.0
>Reporter: wanghao
>Priority: Major
> Fix For: v2.6.2
>
>
> when column value from jdbc is null, sqoop will write it into hive table with 
> blank string.
> eg 
> jdbc:
> A | B
> 1 | 1
> 2 | 2
> a | null
>  
> hive table:
> A | B
> 1 | 1
> 2 | 2
> a |
> because of this, when I use count(distinct B), it return 3, not 2, and it can 
> lead to other problems
>  
>  
> {code:java}
> String cmd = String.format(Locale.ROOT,
> "%s/bin/sqoop import" + generateSqoopConfigArgString()
> + "--connect \"%s\" --driver %s --username %s --password %s --query \"%s AND 
> \\$CONDITIONS\" "
> + "--target-dir %s/%s --split-by %s --boundary-query \"%s\" --null-string '' "
> + "--fields-terminated-by '%s' --num-mappers %d",
> sqoopHome, connectionUrl, driverClass, jdbcUser, jdbcPass, selectSql, 
> jobWorkingDir, hiveTable,
> splitColumn, bquery, filedDelimiter, mapperNum);
> {code}
> the param '–null=string' should be '
> \\N' instead of blank string ''.
> I resolved this problem by replace the param. But it needs be configured in 
> kylin.properties
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3934) sqoop import param '--null-string' result in null value become blank string in hive table

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16837097#comment-16837097
 ] 

ASF subversion and git services commented on KYLIN-3934:


Commit 53123d3e5250041203f1c815ed93533bd03a7d2b in kylin's branch 
refs/heads/2.6.x from wanghao
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=53123d3 ]

KYLIN-3934 add config for sqoop config null-string and null-non-string


> sqoop import param '--null-string' result in null value become blank string 
> in hive table
> -
>
> Key: KYLIN-3934
> URL: https://issues.apache.org/jira/browse/KYLIN-3934
> Project: Kylin
>  Issue Type: Bug
>  Components: Others
>Affects Versions: v2.6.0
>Reporter: wanghao
>Priority: Major
> Fix For: v2.6.2
>
>
> when column value from jdbc is null, sqoop will write it into hive table with 
> blank string.
> eg 
> jdbc:
> A | B
> 1 | 1
> 2 | 2
> a | null
>  
> hive table:
> A | B
> 1 | 1
> 2 | 2
> a |
> because of this, when I use count(distinct B), it return 3, not 2, and it can 
> lead to other problems
>  
>  
> {code:java}
> String cmd = String.format(Locale.ROOT,
> "%s/bin/sqoop import" + generateSqoopConfigArgString()
> + "--connect \"%s\" --driver %s --username %s --password %s --query \"%s AND 
> \\$CONDITIONS\" "
> + "--target-dir %s/%s --split-by %s --boundary-query \"%s\" --null-string '' "
> + "--fields-terminated-by '%s' --num-mappers %d",
> sqoopHome, connectionUrl, driverClass, jdbcUser, jdbcPass, selectSql, 
> jobWorkingDir, hiveTable,
> splitColumn, bquery, filedDelimiter, mapperNum);
> {code}
> the param '–null=string' should be '
> \\N' instead of blank string ''.
> I resolved this problem by replace the param. But it needs be configured in 
> kylin.properties
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KYLIN-3812) optimize the child CompareTupleFilter in a CompareTupleFilter

[nichunen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nichunen resolved KYLIN-3812.
-
Resolution: Fixed

> optimize the child CompareTupleFilter in a CompareTupleFilter
> -
>
> Key: KYLIN-3812
> URL: https://issues.apache.org/jira/browse/KYLIN-3812
> Project: Kylin
>  Issue Type: Improvement
>  Components: Query Engine
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
>Priority: Major
> Fix For: v3.0.0
>
>
> Currently it's not well supported for CompareTupleFilter to have a child of 
> CompareTupleFilter. However, in some cases, it's better to support it.
> {code}
> where (colA = (1=1))
> {code}
> The *(1=1)* can be transformed to "true". And then this filter can be pushed 
> down to hbase. Otherwise, the filter *(colA = (1=1))* does not work in hbase.
> And it may return incorrect results for the following SQL:
> {code}
> select colA
>case
>when colB = (1 = 1) then 'B'
>when colC = (1 = 1) then 'C'
>when colD = (1 = 1) then 'D'
>else 'n/a'
>end as phase,
>count(*)
> from T
> where session_date between '2018-08-01' and '2018-08-31'
> group by colA
>case
>when colB = (1 = 1) then 'B'
>when colC = (1 = 1) then 'C'
>when colD = (1 = 1) then 'D'
>else 'n/a'
>end;
> {code}
> In the final result, all of the keys will become 'B'.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3926) Should set sourceRecordCount when updating statistics

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16837094#comment-16837094
 ] 

ASF subversion and git services commented on KYLIN-3926:


Commit edab859ee51128f94d4e192abc56dc1171c468a9 in kylin's branch 
refs/heads/2.6.x from Chunen Ni
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=edab859 ]

KYLIN-3926 Code review


> Should set sourceRecordCount when updating statistics
> -
>
> Key: KYLIN-3926
> URL: https://issues.apache.org/jira/browse/KYLIN-3926
> Project: Kylin
>  Issue Type: Improvement
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
>Priority: Major
> Fix For: v2.6.2
>
>
> Currently there're 3 parts to write cuboids statistics:
> * SaveStatisticsStep
> * MergeStatisticsStep
> * MergeStatisticsWithOldStep
> With feature [KYLIN-3453], the sourceRecordCount is introduced. However, only 
> *SaveStatisticsStep* set this value. The other two should also be updated 
> with this value.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3926) Should set sourceRecordCount when updating statistics

[nichunen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16837095#comment-16837095
 ] 

nichunen commented on KYLIN-3926:
-

Committed to 2.6.x branch

> Should set sourceRecordCount when updating statistics
> -
>
> Key: KYLIN-3926
> URL: https://issues.apache.org/jira/browse/KYLIN-3926
> Project: Kylin
>  Issue Type: Improvement
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
>Priority: Major
> Fix For: v2.6.2
>
>
> Currently there're 3 parts to write cuboids statistics:
> * SaveStatisticsStep
> * MergeStatisticsStep
> * MergeStatisticsWithOldStep
> With feature [KYLIN-3453], the sourceRecordCount is introduced. However, only 
> *SaveStatisticsStep* set this value. The other two should also be updated 
> with this value.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3926) Should set sourceRecordCount when updating statistics

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16837093#comment-16837093
 ] 

ASF subversion and git services commented on KYLIN-3926:


Commit dff3a3c3c9dd2802e44de8273b785339ce232441 in kylin's branch 
refs/heads/2.6.x from kyotoYaho
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=dff3a3c ]

KYLIN-3926 set sourceRecordCount when updating statistics


> Should set sourceRecordCount when updating statistics
> -
>
> Key: KYLIN-3926
> URL: https://issues.apache.org/jira/browse/KYLIN-3926
> Project: Kylin
>  Issue Type: Improvement
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
>Priority: Major
> Fix For: v2.6.2
>
>
> Currently there're 3 parts to write cuboids statistics:
> * SaveStatisticsStep
> * MergeStatisticsStep
> * MergeStatisticsWithOldStep
> With feature [KYLIN-3453], the sourceRecordCount is introduced. However, only 
> *SaveStatisticsStep* set this value. The other two should also be updated 
> with this value.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KYLIN-3813) don't do push down when both of the children of CompareTupleFilter are CompareTupleFilter with column included

[nichunen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nichunen resolved KYLIN-3813.
-
Resolution: Fixed

> don't do push down when both of the children of CompareTupleFilter are 
> CompareTupleFilter with column included
> --
>
> Key: KYLIN-3813
> URL: https://issues.apache.org/jira/browse/KYLIN-3813
> Project: Kylin
>  Issue Type: Improvement
>  Components: Query Engine
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
>Priority: Major
> Fix For: v3.0.0
>
>
> When dynamic column is enabled, kylin will try to push down group by case 
> when to hbase. However, in the following case, the push down should not be 
> enabled, since currently it's not well supported for CompareTupleFilter to 
> have a child of CompareTupleFilter.
> Sample SQL:
> {code}
> select colA
>case
>when (colB = (1 = 1)) = (colC = (1 = 1)) then 'B'
>when (colC = (1 = 1)) = (colD = (1 = 1)) then 'C'
>else 'n/a'
>end as phase,
>count(*)
> from T
> where session_date between '2018-08-01' and '2018-08-31'
> group by colA
>case
>when (colB = (1 = 1)) = (colC = (1 = 1)) then 'B'
>when (colC = (1 = 1)) = (colD = (1 = 1)) then 'C'
>else 'n/a'
>end;
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KYLIN-3926) Should set sourceRecordCount when updating statistics

[nichunen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nichunen resolved KYLIN-3926.
-
Resolution: Fixed

> Should set sourceRecordCount when updating statistics
> -
>
> Key: KYLIN-3926
> URL: https://issues.apache.org/jira/browse/KYLIN-3926
> Project: Kylin
>  Issue Type: Improvement
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
>Priority: Major
> Fix For: v2.6.2
>
>
> Currently there're 3 parts to write cuboids statistics:
> * SaveStatisticsStep
> * MergeStatisticsStep
> * MergeStatisticsWithOldStep
> With feature [KYLIN-3453], the sourceRecordCount is introduced. However, only 
> *SaveStatisticsStep* set this value. The other two should also be updated 
> with this value.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3995) config data type may make oom

[nichunen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nichunen updated KYLIN-3995:

Fix Version/s: v2.6.2

> config data type may make oom
> -
>
> Key: KYLIN-3995
> URL: https://issues.apache.org/jira/browse/KYLIN-3995
> Project: Kylin
>  Issue Type: Bug
>  Components: Others
>Reporter: Zhixiong Chen
>Assignee: Zhixiong Chen
>Priority: Major
> Fix For: v2.6.2
>
>
> getTrieDictionaryForestMaxTrieSizeMB()  data type is int
> however in TrieDictionaryForestBuilder  "maxTrieTreeSizeMB * 1024 * 1024" 
> it could over integer.max



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3995) config data type may make oom

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16837091#comment-16837091
 ] 

ASF subversion and git services commented on KYLIN-3995:


Commit 85c7388f2f92d907e03c537cf49ea2b42e61b94e in kylin's branch 
refs/heads/2.6.x from Zhixiong Chen
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=85c7388 ]

KYLIN-3995 config data type may make oom


> config data type may make oom
> -
>
> Key: KYLIN-3995
> URL: https://issues.apache.org/jira/browse/KYLIN-3995
> Project: Kylin
>  Issue Type: Bug
>  Components: Others
>Reporter: Zhixiong Chen
>Assignee: Zhixiong Chen
>Priority: Major
> Fix For: v2.6.2
>
>
> getTrieDictionaryForestMaxTrieSizeMB()  data type is int
> however in TrieDictionaryForestBuilder  "maxTrieTreeSizeMB * 1024 * 1024" 
> it could over integer.max



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KYLIN-3995) config data type may make oom

[nichunen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nichunen resolved KYLIN-3995.
-
Resolution: Fixed

> config data type may make oom
> -
>
> Key: KYLIN-3995
> URL: https://issues.apache.org/jira/browse/KYLIN-3995
> Project: Kylin
>  Issue Type: Bug
>  Components: Others
>Reporter: Zhixiong Chen
>Assignee: Zhixiong Chen
>Priority: Major
> Fix For: v2.6.2
>
>
> getTrieDictionaryForestMaxTrieSizeMB()  data type is int
> however in TrieDictionaryForestBuilder  "maxTrieTreeSizeMB * 1024 * 1024" 
> it could over integer.max



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3995) config data type may make oom

[nichunen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16837092#comment-16837092
 ] 

nichunen commented on KYLIN-3995:
-

Committed to 2.6.x branch

> config data type may make oom
> -
>
> Key: KYLIN-3995
> URL: https://issues.apache.org/jira/browse/KYLIN-3995
> Project: Kylin
>  Issue Type: Bug
>  Components: Others
>Reporter: Zhixiong Chen
>Assignee: Zhixiong Chen
>Priority: Major
> Fix For: v2.6.2
>
>
> getTrieDictionaryForestMaxTrieSizeMB()  data type is int
> however in TrieDictionaryForestBuilder  "maxTrieTreeSizeMB * 1024 * 1024" 
> it could over integer.max



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3965) When using DriverManager - No suitable driver found for jdbc:kylin://

[nichunen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16837088#comment-16837088
 ] 

nichunen commented on KYLIN-3965:
-

Committed to 2.6.x branch

> When using DriverManager - No suitable driver found for jdbc:kylin://
> -
>
> Key: KYLIN-3965
> URL: https://issues.apache.org/jira/browse/KYLIN-3965
> Project: Kylin
>  Issue Type: Bug
>  Components: Driver - JDBC
>Affects Versions: v2.6.1
>Reporter: Alexander
>Assignee: Alexander
>Priority: Minor
> Fix For: v2.6.2
>
> Attachments: KYLIN-3965.master.001.patch
>
>
> Caused by: java.sql.SQLException: No suitable driver found for jdbc:kylin://
>  
> This is because META-INF/services/java.sql.Driver got incorrect name
> org.apache.calcite.avatica.remote.Driver



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3965) When using DriverManager - No suitable driver found for jdbc:kylin://

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16837087#comment-16837087
 ] 

ASF subversion and git services commented on KYLIN-3965:


Commit fd7ca25f869419034f3cd52381db71bebaaf9153 in kylin's branch 
refs/heads/2.6.x from alexandr.sidorchuk
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=fd7ca25 ]

KYLIN-3965 JDBC fix.

- Driver manager compability.
- Add guava to uber-jar for standalone usage.


> When using DriverManager - No suitable driver found for jdbc:kylin://
> -
>
> Key: KYLIN-3965
> URL: https://issues.apache.org/jira/browse/KYLIN-3965
> Project: Kylin
>  Issue Type: Bug
>  Components: Driver - JDBC
>Affects Versions: v2.6.1
>Reporter: Alexander
>Assignee: Alexander
>Priority: Minor
> Fix For: v2.6.2
>
> Attachments: KYLIN-3965.master.001.patch
>
>
> Caused by: java.sql.SQLException: No suitable driver found for jdbc:kylin://
>  
> This is because META-INF/services/java.sql.Driver got incorrect name
> org.apache.calcite.avatica.remote.Driver



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3987) Give more reducer for UHC column in fact distinct job

[nichunen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16837077#comment-16837077
 ] 

nichunen commented on KYLIN-3987:
-

Committed to 2.6.x branch

> Give more reducer for UHC column in fact distinct job
> -
>
> Key: KYLIN-3987
> URL: https://issues.apache.org/jira/browse/KYLIN-3987
> Project: Kylin
>  Issue Type: Bug
>Reporter: Chao Long
>Assignee: Chao Long
>Priority: Major
> Fix For: v2.6.2
>
>
> The defualt value of the reducer number for UHC column in fact distinct job 
> is too small(default 1),  the fact distinct job may be very slow, even throw 
> OOM exception. We should give it a large value by default.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3987) Give more reducer for UHC column in fact distinct job

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16837076#comment-16837076
 ] 

ASF subversion and git services commented on KYLIN-3987:


Commit 9bbf520f1149cea59697460dad252ee9de2cad34 in kylin's branch 
refs/heads/2.6.x from chao long
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=9bbf520 ]

KYLIN-3987 Set a larger value of reducer num for fact distinct job


> Give more reducer for UHC column in fact distinct job
> -
>
> Key: KYLIN-3987
> URL: https://issues.apache.org/jira/browse/KYLIN-3987
> Project: Kylin
>  Issue Type: Bug
>Reporter: Chao Long
>Assignee: Chao Long
>Priority: Major
> Fix For: v2.6.2
>
>
> The defualt value of the reducer number for UHC column in fact distinct job 
> is too small(default 1),  the fact distinct job may be very slow, even throw 
> OOM exception. We should give it a large value by default.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3968) Customized precision doesn't work in web

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16837084#comment-16837084
 ] 

ASF subversion and git services commented on KYLIN-3968:


Commit 20b3721a88e52df447ca640236526d545145da11 in kylin's branch 
refs/heads/2.6.x from majie
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=20b3721 ]

KYLIN-3968 Customized precision doesn't work in web


> Customized precision doesn't work in web
> 
>
> Key: KYLIN-3968
> URL: https://issues.apache.org/jira/browse/KYLIN-3968
> Project: Kylin
>  Issue Type: Bug
>  Components: Web 
>Reporter: Jack
>Priority: Minor
> Fix For: v2.6.2
>
>
> In the cubeMeasures.js, It will withdraw precision and scale by using Regular 
> Expression. The scale parameter is ok, but precision use the magic number 19.
> So we fixed it, In cubeMeasures.js, around line 469:
> “var precision = 19;”  --> "var precision = returnValue[2] || 0;"
> and we test ok including building cube and querying when the column is 
> decimal(38,18).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3968) Customized precision doesn't work in web

[nichunen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16837085#comment-16837085
 ] 

nichunen commented on KYLIN-3968:
-

Committed to 2.6.x branch

> Customized precision doesn't work in web
> 
>
> Key: KYLIN-3968
> URL: https://issues.apache.org/jira/browse/KYLIN-3968
> Project: Kylin
>  Issue Type: Bug
>  Components: Web 
>Reporter: Jack
>Priority: Minor
> Fix For: v2.6.2
>
>
> In the cubeMeasures.js, It will withdraw precision and scale by using Regular 
> Expression. The scale parameter is ok, but precision use the magic number 19.
> So we fixed it, In cubeMeasures.js, around line 469:
> “var precision = 19;”  --> "var precision = returnValue[2] || 0;"
> and we test ok including building cube and querying when the column is 
> decimal(38,18).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3981) Auto Merge Job failed to execute on windows

[nichunen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nichunen updated KYLIN-3981:

Fix Version/s: (was: v2.6.2)
   v3.0.0

> Auto Merge Job failed to execute on windows
> ---
>
> Key: KYLIN-3981
> URL: https://issues.apache.org/jira/browse/KYLIN-3981
> Project: Kylin
>  Issue Type: Bug
>Affects Versions: v2.6.1
>Reporter: Na Zhai
>Assignee: Na Zhai
>Priority: Major
> Fix For: v3.0.0
>
>
> Auto Merge Job failed to execute on windows. It will throw the following 
> errors.
> {code:none}
> java.lang.IllegalStateException: Metadata uri : 
> C:\Users\NAD096~1.ZHA\AppData\Local\Temp\kylin_job_meta1467762575939435363\meta
>  is not recognized
>  at org.apache.kylin.common.KylinConfig.decideUriType(KylinConfig.java:211)
>  at 
> org.apache.kylin.common.KylinConfig.createInstanceFromUri(KylinConfig.java:221)
>  at 
> org.apache.kylin.engine.mr.common.JobRelatedMetaUtil.dumpResources(JobRelatedMetaUtil.java:68)
>  at 
> org.apache.kylin.engine.mr.common.JobRelatedMetaUtil.dumpAndUploadKylinPropsAndMetadata(JobRelatedMetaUtil.java:87)
>  at 
> org.apache.kylin.engine.mr.common.AbstractHadoopJob.attachSegmentsMetadataWithDict(AbstractHadoopJob.java:572)
>  at 
> org.apache.kylin.engine.mr.steps.MergeDictionaryJob.run(MergeDictionaryJob.java:104)
>  at 
> org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:131)
>  at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:179)
>  at 
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
>  at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:179)
>  at 
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
>  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>  at java.lang.Thread.run(Thread.java:748)
>  Suppressed: java.io.FileNotFoundException: File does not exist: 
> C:\Users\NAD096~1.ZHA\AppData\Local\Temp\kylin_job_meta1467762575939435363\meta
>  at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2275)
>  at 
> org.apache.kylin.common.persistence.AutoDeleteDirectory.close(AutoDeleteDirectory.java:56)
>  at 
> org.apache.kylin.engine.mr.common.JobRelatedMetaUtil.dumpAndUploadKylinPropsAndMetadata(JobRelatedMetaUtil.java:103)
>  ... 10 more
> Caused by: java.lang.IllegalStateException: Metadata uri : 
> C:\Users\NAD096~1.ZHA\AppData\Local\Temp\kylin_job_meta1467762575939435363\meta
>  is not a valid REST URI address
>  at org.apache.kylin.common.KylinConfig.decideUriType(KylinConfig.java:208)
>  ... 13 more 
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3939) Open csv files from query exported, the Chinese characters are garbled

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16837065#comment-16837065
 ] 

ASF subversion and git services commented on KYLIN-3939:


Commit 93cf901e07ea67e1612eafd2333e90c3998100ec in kylin's branch 
refs/heads/2.6.x from zhaojintaozhao
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=93cf901 ]

KYLIN-3939 Add BOM character to the exported csv (#602)



> Open csv files  from query exported, the  Chinese characters are garbled
> 
>
> Key: KYLIN-3939
> URL: https://issues.apache.org/jira/browse/KYLIN-3939
> Project: Kylin
>  Issue Type: Bug
>  Components: REST Service
>Affects Versions: v2.5.2
>Reporter: zhao jintao
>Assignee: zhao jintao
>Priority: Minor
> Fix For: v2.6.2
>
>
> Hi team;
> I query sql in kylin portal and  then press "Export" button to download the 
> result; the result is  csv files. I open the csv file using excel to get the 
> query results.
>  But I find that if the result has Chinese characters, the excel will show 
> Chinese characters garbled. 
>  Does any one face this problem?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KYLIN-3987) Give more reducer for UHC column in fact distinct job

[nichunen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nichunen resolved KYLIN-3987.
-
Resolution: Fixed

> Give more reducer for UHC column in fact distinct job
> -
>
> Key: KYLIN-3987
> URL: https://issues.apache.org/jira/browse/KYLIN-3987
> Project: Kylin
>  Issue Type: Bug
>Reporter: Chao Long
>Assignee: Chao Long
>Priority: Major
> Fix For: v2.6.2
>
>
> The defualt value of the reducer number for UHC column in fact distinct job 
> is too small(default 1),  the fact distinct job may be very slow, even throw 
> OOM exception. We should give it a large value by default.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3939) Open csv files from query exported, the Chinese characters are garbled

[nichunen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nichunen updated KYLIN-3939:

Fix Version/s: v2.6.2

> Open csv files  from query exported, the  Chinese characters are garbled
> 
>
> Key: KYLIN-3939
> URL: https://issues.apache.org/jira/browse/KYLIN-3939
> Project: Kylin
>  Issue Type: Bug
>  Components: REST Service
>Affects Versions: v2.5.2
>Reporter: zhao jintao
>Assignee: zhao jintao
>Priority: Minor
> Fix For: v2.6.2
>
>
> Hi team;
> I query sql in kylin portal and  then press "Export" button to download the 
> result; the result is  csv files. I open the csv file using excel to get the 
> query results.
>  But I find that if the result has Chinese characters, the excel will show 
> Chinese characters garbled. 
>  Does any one face this problem?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3936) MR/Spark task will still run after the job is stopped.

[nichunen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16837063#comment-16837063
 ] 

nichunen commented on KYLIN-3936:
-

Committed to 2.6.x branch

> MR/Spark task will still run after the job is stopped.
> --
>
> Key: KYLIN-3936
> URL: https://issues.apache.org/jira/browse/KYLIN-3936
> Project: Kylin
>  Issue Type: Bug
>  Components: Job Engine
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
>Priority: Major
> Fix For: v2.6.2
>
>
> The command "pause" only sets status of the job to "stopped" and does not 
> reset the status of the subtask.
> So, In SparkExecutable, we can't get the real status of the running task.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3936) MR/Spark task will still run after the job is stopped.

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16837062#comment-16837062
 ] 

ASF subversion and git services commented on KYLIN-3936:


Commit 21c1717302eaf92daeca11b16989d1c051ada987 in kylin's branch 
refs/heads/2.6.x from Guangxu Cheng
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=21c1717 ]

KYLIN-3936 MR/Spark task will still run after the job is stopped


> MR/Spark task will still run after the job is stopped.
> --
>
> Key: KYLIN-3936
> URL: https://issues.apache.org/jira/browse/KYLIN-3936
> Project: Kylin
>  Issue Type: Bug
>  Components: Job Engine
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
>Priority: Major
> Fix For: v2.6.2
>
>
> The command "pause" only sets status of the job to "stopped" and does not 
> reset the status of the subtask.
> So, In SparkExecutable, we can't get the real status of the running task.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KYLIN-3939) Open csv files from query exported, the Chinese characters are garbled

[nichunen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nichunen resolved KYLIN-3939.
-
Resolution: Fixed

> Open csv files  from query exported, the  Chinese characters are garbled
> 
>
> Key: KYLIN-3939
> URL: https://issues.apache.org/jira/browse/KYLIN-3939
> Project: Kylin
>  Issue Type: Bug
>  Components: REST Service
>Affects Versions: v2.5.2
>Reporter: zhao jintao
>Assignee: zhao jintao
>Priority: Minor
> Fix For: v2.6.2
>
>
> Hi team;
> I query sql in kylin portal and  then press "Export" button to download the 
> result; the result is  csv files. I open the csv file using excel to get the 
> query results.
>  But I find that if the result has Chinese characters, the excel will show 
> Chinese characters garbled. 
>  Does any one face this problem?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3943) system-cube.sh has some problems

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16837060#comment-16837060
 ] 

ASF subversion and git services commented on KYLIN-3943:


Commit 75b17bed0a97d6af326820223f130f026c089c8c in kylin's branch 
refs/heads/2.6.x from chao long
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=75b17be ]

KYLIN-3943 Fix some problems in system-cube.sh


> system-cube.sh has some problems
> 
>
> Key: KYLIN-3943
> URL: https://issues.apache.org/jira/browse/KYLIN-3943
> Project: Kylin
>  Issue Type: Bug
>Reporter: Chao Long
>Assignee: Chao Long
>Priority: Major
> Fix For: v2.6.2
>
> Attachments: image-2019-04-09-18-50-35-111.png, 
> image-2019-04-09-23-07-19-229.png
>
>
> 1, The name of system cube should not hard code. the suffix should get from 
> config.
> !image-2019-04-09-18-50-35-111.png!
> 2, server port should not hard code
> !image-2019-04-09-23-07-19-229.png!  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3857) Add parameter to change sql quoting character for sqoop

[nichunen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16837057#comment-16837057
 ] 

nichunen commented on KYLIN-3857:
-

Committed to 2.6.x branch

> Add parameter to change sql quoting character for sqoop
> ---
>
> Key: KYLIN-3857
> URL: https://issues.apache.org/jira/browse/KYLIN-3857
> Project: Kylin
>  Issue Type: Bug
>Reporter: Davide Malagoli
>Assignee: XiaoXiang Yu
>Priority: Major
> Fix For: v3.0.0, v2.6.2
>
> Attachments: image-2019-03-11-11-46-05-946.png, 
> image-2019-03-11-11-46-14-305.png, image-2019-03-11-11-46-20-664.png, 
> image-2019-03-11-11-46-31-966.png, sqoop-error.log, sqoop-script.sh
>
>
> I've set up  Kylin to access a SQL server using JDBC and do the ETL step as 
> described in
> [http://kylin.apache.org/docs/tutorial/setup_jdbc_datasource.html]
> but when the "Sqoop To Flat Hive Table" runs it terminates with an error 
> shown in the attached log.
> It seems that the root cause is the presence of the '`' character in the 
> query, which has no meaning for sql server.
>  
> My problem seems to related to this piece of code
>  
> public class FlatTableSqlQuoteUtils {
> public static final String QUOTE = "`";
> /**
>  * Quote identifier by default quote `
>  * @param identifier
>  * @return
>  */
>  public static String quoteIdentifier(String identifier){
>  return QUOTE + identifier + QUOTE;
>  }
>  
> The solution may be to add a parameter to change this character used for 
> quoting.
>  
> Just to be sure I made a copy of the original "sqoop" command to "sqoop.orig" 
> and replaced the original with a little script of mine (attached).
> This little script removes the "`" character from the original command and 
> the call "sqoop.orig" with the cleaned arguments.
> It works, but it is still an ugly workaround.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KYLIN-3960) Only update user when login in LDAP environment

[nichunen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nichunen resolved KYLIN-3960.
-
Resolution: Fixed

> Only update user when login in LDAP environment
> ---
>
> Key: KYLIN-3960
> URL: https://issues.apache.org/jira/browse/KYLIN-3960
> Project: Kylin
>  Issue Type: Improvement
>  Components: Security
>Reporter: Jiatao Tao
>Assignee: Jiatao Tao
>Priority: Minor
> Fix For: v3.0.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3960) Only update user when login in LDAP environment

[nichunen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nichunen updated KYLIN-3960:

Fix Version/s: v3.0.0

> Only update user when login in LDAP environment
> ---
>
> Key: KYLIN-3960
> URL: https://issues.apache.org/jira/browse/KYLIN-3960
> Project: Kylin
>  Issue Type: Improvement
>  Components: Security
>Reporter: Jiatao Tao
>Assignee: Jiatao Tao
>Priority: Minor
> Fix For: v3.0.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3957) Query system_cube get exception Cannot cast "java.math.BigDecimal" to "java.lang.Double"

[nichunen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16837055#comment-16837055
 ] 

nichunen commented on KYLIN-3957:
-

Committed to 2.6.x branch

> Query system_cube get exception Cannot cast "java.math.BigDecimal" to 
> "java.lang.Double"
> 
>
> Key: KYLIN-3957
> URL: https://issues.apache.org/jira/browse/KYLIN-3957
> Project: Kylin
>  Issue Type: Bug
>  Components: Query Engine
>Affects Versions: v2.6.1
>Reporter: Chao Long
>Assignee: Chao Long
>Priority: Major
> Fix For: v3.0.0, v2.6.2
>
>
> In system cube, the return dataType of column whose real dataType is Double 
> will be converted to Decimal in SUM measure.
> {code:java}
> FunctionDesc function = new FunctionDesc();
> function.setExpression(FunctionDesc.FUNC_SUM);
> function.setParameter(parameterDesc);
> 
> function.setReturnType(dataType.equals(HiveTableCreator.HiveTypeEnum.HDOUBLE.toString())
> ? HiveTableCreator.HiveTypeEnum.HDECIMAL.toString()
> : dataType);
> {code}
> but query with that measure will get exception:
> {code}
> Caused by: org.codehaus.commons.compiler.CompileException: Line 108, Column 
> 44: Cannot cast "java.math.BigDecimal" to "java.lang.Double"
>   at 
> org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:10092)
>   at org.codehaus.janino.UnitCompiler.compileGet2(UnitCompiler.java:3839)
>   at org.codehaus.janino.UnitCompiler.access$6400(UnitCompiler.java:183)
>   at org.codehaus.janino.UnitCompiler$10.visitCast(UnitCompiler.java:3246)
>   at org.codehaus.janino.Java$Cast.accept(Java.java:3802)
>   at org.codehaus.janino.UnitCompiler.compileGet(UnitCompiler.java:3278)
>   at org.codehaus.janino.UnitCompiler.compileGet2(UnitCompiler.java:3845)
>   at org.codehaus.janino.UnitCompiler.access$8600(UnitCompiler.java:183)
>   at 
> org.codehaus.janino.UnitCompiler$10.visitParenthesizedExpression(UnitCompiler.java:3274)
>   at 
> org.codehaus.janino.Java$ParenthesizedExpression.accept(Java.java:3830)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3788) Kylin's internal time zone conversions are various and confusing

[nichunen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16837051#comment-16837051
 ] 

nichunen commented on KYLIN-3788:
-

Committed to 2.6.x branch

>  Kylin's internal time zone conversions are various and confusing
> -
>
> Key: KYLIN-3788
> URL: https://issues.apache.org/jira/browse/KYLIN-3788
> Project: Kylin
>  Issue Type: Bug
>  Components: REST Service, Web 
>Affects Versions: v2.5.2
> Environment: Huawei FI
>Reporter: zhao jintao
>Assignee: zhao jintao
>Priority: Major
>  Labels: TimeZone
> Fix For: v2.6.2
>
>
> Hi, All team
> I found that Kylin has a variety of time zone conversions inside, which is 
> confusing. In some code, the time zone is "GMT" when converting time, and 
> some code uses the time zone specified in the configuration file 
> ("TimeZone.getTimeZone("GMT")" );
> For Example:
> in CubingJob.java, use TimeZone from configuration file:  
> format.setTimeZone(TimeZone.getTimeZone(
> config.getTimeZone())),
> but in TimeUtil.java, use GMT TimeZone:private static TimeZone gmt = 
> TimeZone.getTimeZone("GMT")
>  
> When inport data from Kafka Streaming Data，kylin generate "Data_Start", 
> "Hour_Start" uses the GMT time zone,  They are  some time zones later than 
> the real time. Why not use the time zone specified in the configuration file?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3950) Cube planner optimize job only use inmem algorithm

[nichunen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16837047#comment-16837047
 ] 

nichunen commented on KYLIN-3950:
-

Committed to 2.6.x branch

> Cube planner optimize job only use inmem algorithm
> --
>
> Key: KYLIN-3950
> URL: https://issues.apache.org/jira/browse/KYLIN-3950
> Project: Kylin
>  Issue Type: Bug
>Affects Versions: v2.6.1
>Reporter: Chao Long
>Assignee: Chao Long
>Priority: Major
> Fix For: v2.6.2
>
>
> Currently, cube planner optimize job only use inmem algorithm to optimize 
> cube, which should be configurable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3938) can't delete job OPTIMIZE CHECKPOINT

[nichunen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16837043#comment-16837043
 ] 

nichunen commented on KYLIN-3938:
-

Committed to 2.6.x branch

> can't delete job OPTIMIZE CHECKPOINT
> 
>
> Key: KYLIN-3938
> URL: https://issues.apache.org/jira/browse/KYLIN-3938
> Project: Kylin
>  Issue Type: Bug
>  Components: Job Engine
>Affects Versions: v2.6.1
>Reporter: kaishen
>Assignee: Chao Long
>Priority: Major
> Fix For: v2.6.2
>
>
> I'm using kylin-2.6.1-cdh57 ，when I use cube planner , it will generate tow 
> jobs one is : OPTIMIZE CUBE another is : OPTIMIZE CHECKPOINT which depend on 
> job OPTIMIZE CUBE .
>  now the first job OPTIMIZE CUBE was failed , and I want to discard the 
> second job , I got error :
>  java.lang.ClassCastException: 
> org.apache.kylin.job.execution.CheckpointExecutable cannot be cast to 
> org.apache.kylin.engine.mr.CubingJob



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3892) Set cubing job priority

[nichunen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16837040#comment-16837040
 ] 

nichunen commented on KYLIN-3892:
-

Committed to 2.6.x branch

> Set cubing job priority
> ---
>
> Key: KYLIN-3892
> URL: https://issues.apache.org/jira/browse/KYLIN-3892
> Project: Kylin
>  Issue Type: New Feature
>  Components: Job Engine
>Affects Versions: v2.4.0, v2.5.0, v2.6.0
>Reporter: Temple Zhou
>Assignee: Temple Zhou
>Priority: Minor
> Fix For: v2.6.2
>
>
> The cubing job with high priority will be delayed when there are too many 
> tasks running. 
> So I want to set the job priority for the important cubing jobs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KYLIN-3999) Enable dynamic column by default

[Shaofeng SHI (JIRA)]

Shaofeng SHI created KYLIN-3999:
---

 Summary: Enable dynamic column by default
 Key: KYLIN-3999
 URL: https://issues.apache.org/jira/browse/KYLIN-3999
 Project: Kylin
  Issue Type: Improvement
  Components: Query Engine
Reporter: Shaofeng SHI


More and more user expects to use "SUM(Case when)" feature, and got error. The 
reason is the dynamic column is disabled by default. We should consider to 
enable it by default:

 

kylin.query.enable-dynamic-column=true



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3922) Fail to update coprocessor when run DeployCoprocessorCLI

[nichunen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16837030#comment-16837030
 ] 

nichunen commented on KYLIN-3922:
-

Committed to 2.6.x branch

> Fail to update coprocessor when run DeployCoprocessorCLI
> 
>
> Key: KYLIN-3922
> URL: https://issues.apache.org/jira/browse/KYLIN-3922
> Project: Kylin
>  Issue Type: Bug
>  Components: Client - CLI
>Affects Versions: v2.6.0
>Reporter: nichunen
>Assignee: nichunen
>Priority: Major
> Fix For: v2.6.2, v3.0.0-alpha
>
>
> Update coprocessor by run
> {code:java}
> $KYLIN_HOME/bin/kylin.sh 
> org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI 
> $KYLIN_HOME/lib/kylin-coprocessor-*.jar all
> {code}
> ArrayIndexOutOfBoundsException thrown with log:
> {code:java}
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 2 at 
> org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI.main(DeployCoprocessorCLI.java:118)
>  
> 2019-03-27 20:08:02,105 INFO [close-hbase-conn] hbase.HBaseConnection:137 : 
> Closing HBase connections... 
> 2019-03-27 20:08:02,106 INFO [close-hbase-conn] 
> client.ConnectionManager$HConnectionImplementation:1847 : Closing zookeeper 
> sessionid=0x169a4aff61f4457 
> 2019-03-27 20:08:02,111 INFO [close-hbase-conn] zookeeper.ZooKeeper:684 : 
> Session: 0x169a4aff61f4457 closed 
> 2019-03-27 20:08:02,111 INFO [main-EventThread] zookeeper.ClientCnxn:512 : 
> EventThread shut down 2019-03-27 20:08:02,112 INFO [close-hbase-conn] 
> client.ConnectionManager$HConnectionImplementation:2297 : Closing master 
> protocol: MasterService 2019-03-27 20:08:02,124 INFO [close-hbase-conn] 
> client.ConnectionManager$HConnectionImplementation:1847 : Closing zookeeper 
> sessionid=0x469a4aff7c42d72 
> 2019-03-27 20:08:02,126 INFO [close-hbase-conn] zookeeper.ZooKeeper:684 : 
> Session: 0x469a4aff7c42d72 closed 2019-03-27 20:08:02,126 INFO 
> [main-EventThread] zookeeper.ClientCnxn:512 : EventThread shut down
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3922) Fail to update coprocessor when run DeployCoprocessorCLI

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16837019#comment-16837019
 ] 

ASF subversion and git services commented on KYLIN-3922:


Commit 3fd38827bbcfd892f77f685e78e344c99c1d98d6 in kylin's branch 
refs/heads/2.6.x from Chunen Ni
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=3fd3882 ]

KYLIN-3922 Fix coprocessor can not be updated


> Fail to update coprocessor when run DeployCoprocessorCLI
> 
>
> Key: KYLIN-3922
> URL: https://issues.apache.org/jira/browse/KYLIN-3922
> Project: Kylin
>  Issue Type: Bug
>  Components: Client - CLI
>Affects Versions: v2.6.0
>Reporter: nichunen
>Assignee: nichunen
>Priority: Major
> Fix For: v2.6.2, v3.0.0-alpha
>
>
> Update coprocessor by run
> {code:java}
> $KYLIN_HOME/bin/kylin.sh 
> org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI 
> $KYLIN_HOME/lib/kylin-coprocessor-*.jar all
> {code}
> ArrayIndexOutOfBoundsException thrown with log:
> {code:java}
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 2 at 
> org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI.main(DeployCoprocessorCLI.java:118)
>  
> 2019-03-27 20:08:02,105 INFO [close-hbase-conn] hbase.HBaseConnection:137 : 
> Closing HBase connections... 
> 2019-03-27 20:08:02,106 INFO [close-hbase-conn] 
> client.ConnectionManager$HConnectionImplementation:1847 : Closing zookeeper 
> sessionid=0x169a4aff61f4457 
> 2019-03-27 20:08:02,111 INFO [close-hbase-conn] zookeeper.ZooKeeper:684 : 
> Session: 0x169a4aff61f4457 closed 
> 2019-03-27 20:08:02,111 INFO [main-EventThread] zookeeper.ClientCnxn:512 : 
> EventThread shut down 2019-03-27 20:08:02,112 INFO [close-hbase-conn] 
> client.ConnectionManager$HConnectionImplementation:2297 : Closing master 
> protocol: MasterService 2019-03-27 20:08:02,124 INFO [close-hbase-conn] 
> client.ConnectionManager$HConnectionImplementation:1847 : Closing zookeeper 
> sessionid=0x469a4aff7c42d72 
> 2019-03-27 20:08:02,126 INFO [close-hbase-conn] zookeeper.ZooKeeper:684 : 
> Session: 0x469a4aff7c42d72 closed 2019-03-27 20:08:02,126 INFO 
> [main-EventThread] zookeeper.ClientCnxn:512 : EventThread shut down
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3885) Build dimension dictionary job costs too long when using Spark fact distinct

[nichunen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16837000#comment-16837000
 ] 

nichunen commented on KYLIN-3885:
-

Committed to 2.6.x branch

> Build dimension dictionary job costs too long when using Spark fact distinct
> 
>
> Key: KYLIN-3885
> URL: https://issues.apache.org/jira/browse/KYLIN-3885
> Project: Kylin
>  Issue Type: Bug
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Major
> Fix For: v2.6.2
>
>
> Build dimension dictionary job costs less than 20 minutes when using 
> mapreduce fact distinct,but but it costs more than 3 hours when using spark 
> fact distinct.
> {code:java}
> "Scheduler 542945608 Job 05c62aca-853f-396e-9653-f20c9ebd8ebc-329" #329 
> prio=5 os_prio=0 tid=0x7f312109c800 nid=0x2dc0b in Object.wait() 
> [0x7f30d8d24000]
>java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at org.apache.hadoop.ipc.Client.call(Client.java:1482)
> - locked <0x0005c3110fc0> (a org.apache.hadoop.ipc.Client$Call)
> at org.apache.hadoop.ipc.Client.call(Client.java:1427)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
> at com.sun.proxy.$Proxy33.delete(Unknown Source)
> at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:573)
> at sun.reflect.GeneratedMethodAccessor193.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:249)
> at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:107)
> at com.sun.proxy.$Proxy34.delete(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:2057)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:682)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:675)
> at 
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:696)
> at 
> org.apache.hadoop.fs.FilterFileSystem.delete(FilterFileSystem.java:232)
> at 
> org.apache.hadoop.fs.viewfs.ChRootedFileSystem.delete(ChRootedFileSystem.java:198)
> at 
> org.apache.hadoop.fs.viewfs.ViewFileSystem.delete(ViewFileSystem.java:334)
> at 
> org.apache.hadoop.hdfs.FederatedDFSFileSystem.delete(FederatedDFSFileSystem.java:232)
> at 
> org.apache.kylin.dict.global.GlobalDictHDFSStore.deleteSlice(GlobalDictHDFSStore.java:211)
> at 
> org.apache.kylin.dict.global.AppendTrieDictionaryBuilder.flushCurrentNode(AppendTrieDictionaryBuilder.java:137)
> at 
> org.apache.kylin.dict.global.AppendTrieDictionaryBuilder.addValue(AppendTrieDictionaryBuilder.java:97)
> at 
> org.apache.kylin.dict.GlobalDictionaryBuilder.addValue(GlobalDictionaryBuilder.java:85)
> at 
> org.apache.kylin.dict.DictionaryGenerator.buildDictionary(DictionaryGenerator.java:82)
> at 
> org.apache.kylin.dict.DictionaryManager.buildDictFromReadableTable(DictionaryManager.java:303)
> at 
> org.apache.kylin.dict.DictionaryManager.buildDictionary(DictionaryManager.java:290)
> at 
> org.apache.kylin.cube.CubeManager$DictionaryAssist.buildDictionary(CubeManager.java:1043)
> at 
> org.apache.kylin.cube.CubeManager.buildDictionary(CubeManager.java:1012)
> at 
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:72)
> at 
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:50)
> at 
> org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:73)
> at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:92)
> at 
> org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63)
> at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:178)
> at 
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
> at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:178)
> at 
> 

[jira] [Updated] (KYLIN-3907) Sort the cube list by create time in descending order.

[nichunen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nichunen updated KYLIN-3907:

Fix Version/s: v3.0.0

> Sort the cube list by create time in descending order.
> --
>
> Key: KYLIN-3907
> URL: https://issues.apache.org/jira/browse/KYLIN-3907
> Project: Kylin
>  Issue Type: Improvement
>  Components: REST Service
>Affects Versions: v2.5.2
>Reporter: Yuzhang QIU
>Priority: Minor
> Fix For: v3.0.0
>
>
> Hi team:
> Maybe there have a use experience problem in the Web UI of cube list. We 
> will create many cubes over time and need click "MORE" to show the lastest 
> cube when the number cubes increate to over 15.
>In most cases, I think, the older cube should be steady and the new cube 
> may need to be debuged. So, sort  the cube list by create time in descending 
> order may be better.
> How do you think about this?
> Best regards
> yuzhang



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3911) Check if HBase table is enabled before diabling table in DeployCoprocessorCLI

[nichunen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836994#comment-16836994
 ] 

nichunen commented on KYLIN-3911:
-

Committed to 2.6.x branch

> Check if HBase table is enabled before diabling table in DeployCoprocessorCLI
> -
>
> Key: KYLIN-3911
> URL: https://issues.apache.org/jira/browse/KYLIN-3911
> Project: Kylin
>  Issue Type: Bug
>  Components: Client - CLI
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Minor
> Fix For: v2.6.2, v3.0.0-alpha
>
>
> HBase tables may be disabled for operation issues and the previous 
> interrupted DeployCoprocessorCLI, which cause the new  DeployCoprocessorCLI 
> failed.
> {code:java}
> 2018-06-08 10:40:23,489 ERROR [pool-5-thread-6] util.DeployCoprocessorCLI:383 
> : Error processing kylin_bigdata_prod:KYLIN_A9520J93GU
> org.apache.hadoop.hbase.TableNotEnabledException: 
> org.apache.hadoop.hbase.TableNotEnabledException: 
> kylin_bigdata_prod:KYLIN_A9520J93GU
>     at 
> org.apache.hadoop.hbase.master.handler.DisableTableHandler.prepare(DisableTableHandler.java:102)
>     at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:2609)
>     at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:2619)
>     at 
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:44586)
>     at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2061)
>     at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:125)
>     at 
> org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:83)
>     at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>     at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>     at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>     at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>     at java.lang.Thread.run(Thread.java:748)
> @c3-hadoop-prc-ct36.bj/10.136.14.13:33500
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>     at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>     at 
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
>     at 
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
>     at 
> org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(RpcRetryingCaller.java:220)
> ingCaller.java:86)
>     at 
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3239)
>     at 
> org.apache.hadoop.hbase.client.HBaseAdmin.disableTableAsync(HBaseAdmin.java:919)
>     at 
> org.apache.hadoop.hbase.client.HBaseAdmin.disableTable(HBaseAdmin.java:948)
>     at 
> org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI.resetCoprocessor(DeployCoprocessorCLI.java:294)
>     at 
> org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI$ResetCoprocessorWorker.run(DeployCoprocessorCLI.java:375)
> ```{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3911) Check if HBase table is enabled before diabling table in DeployCoprocessorCLI

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836993#comment-16836993
 ] 

ASF subversion and git services commented on KYLIN-3911:


Commit b09815c17be7eeb70ec48b3973518ae3e3d30217 in kylin's branch 
refs/heads/2.6.x from Liu Shaohui
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=b09815c ]

KYLIN-3911: Check if HBase table is enabled before diabling table in 
DeployCoprocessorCLI


> Check if HBase table is enabled before diabling table in DeployCoprocessorCLI
> -
>
> Key: KYLIN-3911
> URL: https://issues.apache.org/jira/browse/KYLIN-3911
> Project: Kylin
>  Issue Type: Bug
>  Components: Client - CLI
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Minor
> Fix For: v2.6.2, v3.0.0-alpha
>
>
> HBase tables may be disabled for operation issues and the previous 
> interrupted DeployCoprocessorCLI, which cause the new  DeployCoprocessorCLI 
> failed.
> {code:java}
> 2018-06-08 10:40:23,489 ERROR [pool-5-thread-6] util.DeployCoprocessorCLI:383 
> : Error processing kylin_bigdata_prod:KYLIN_A9520J93GU
> org.apache.hadoop.hbase.TableNotEnabledException: 
> org.apache.hadoop.hbase.TableNotEnabledException: 
> kylin_bigdata_prod:KYLIN_A9520J93GU
>     at 
> org.apache.hadoop.hbase.master.handler.DisableTableHandler.prepare(DisableTableHandler.java:102)
>     at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:2609)
>     at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:2619)
>     at 
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:44586)
>     at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2061)
>     at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:125)
>     at 
> org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:83)
>     at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>     at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>     at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>     at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>     at java.lang.Thread.run(Thread.java:748)
> @c3-hadoop-prc-ct36.bj/10.136.14.13:33500
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>     at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>     at 
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
>     at 
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
>     at 
> org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(RpcRetryingCaller.java:220)
> ingCaller.java:86)
>     at 
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3239)
>     at 
> org.apache.hadoop.hbase.client.HBaseAdmin.disableTableAsync(HBaseAdmin.java:919)
>     at 
> org.apache.hadoop.hbase.client.HBaseAdmin.disableTable(HBaseAdmin.java:948)
>     at 
> org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI.resetCoprocessor(DeployCoprocessorCLI.java:294)
>     at 
> org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI$ResetCoprocessorWorker.run(DeployCoprocessorCLI.java:375)
> ```{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3909) Spark cubing job failed for MappeableRunContainer is not registered

[nichunen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836987#comment-16836987
 ] 

nichunen commented on KYLIN-3909:
-

Committed to 2.6.x branch

> Spark cubing job failed for MappeableRunContainer is not registered
> ---
>
> Key: KYLIN-3909
> URL: https://issues.apache.org/jira/browse/KYLIN-3909
> Project: Kylin
>  Issue Type: Bug
>  Components: Spark Engine
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Minor
> Fix For: v2.6.2, v3.0.0-alpha
>
>
> |Job aborted due to stage failure: Task 2 in stage 1.0 failed 4 times, most 
> recent failure: Lost task 2.3 in stage 1.0 (TID 2621, 
> zjy-hadoop-prc-st2587.bj, executor 53): 
> com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: 
> Class is not registered: 
> org.apache.kylin.job.shaded.org.roaringbitmap.buffer.MappeableRunContainer|



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3818) After Cube disabled, auto-merge cube job still running

[Liu Shaohui (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836980#comment-16836980
 ] 

Liu Shaohui commented on KYLIN-3818:


I encountered the same problem yesterday in kylin 2.5.2. Thanks for the fix.

> After Cube disabled, auto-merge cube job still running
> --
>
> Key: KYLIN-3818
> URL: https://issues.apache.org/jira/browse/KYLIN-3818
> Project: Kylin
>  Issue Type: Bug
>Affects Versions: v2.6.0
>Reporter: Na Zhai
>Assignee: Na Zhai
>Priority: Major
> Fix For: v2.6.2, v3.0.0-alpha
>
>
> *precondition*
> There is a Cube that turns on the auto-merge feature. And it satisfied the 
> condition of the auto-merge. Then the job of merging segments begins.
> After a few minutes, the job of merging segments goes into the error status, 
> so I discard the job. Then I disable this cube, but a new job of merging 
> segments begin to run.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KYLIN-3756) Support check-port-availability script for mac os x

[nichunen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nichunen resolved KYLIN-3756.
-
Resolution: Fixed

> Support check-port-availability script for mac os x
> ---
>
> Key: KYLIN-3756
> URL: https://issues.apache.org/jira/browse/KYLIN-3756
> Project: Kylin
>  Issue Type: Improvement
>  Components: Tools, Build and Test
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Major
> Fix For: v3.0.0
>
>
> {{bin/check-port-availability.sh}} internally uses "netstat -tlpn" to detect 
> network ports. Running this script under mac os x will report the following 
> exception:
> {code:java}
> netstat: n: unknown or uninstrumented protocol{code}
> The "netstat" command under mac os x does not support the "n,p" option. We 
> can achieve the same function under mac with the "lsof" command. More 
> details: 
> https://stackoverflow.com/questions/4421633/who-is-listening-on-a-given-tcp-port-on-mac-os-x



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3730) TableMetadataManager.reloadSourceTableQuietly is wrong

[nichunen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nichunen updated KYLIN-3730:

Fix Version/s: (was: v2.6.2)

> TableMetadataManager.reloadSourceTableQuietly is wrong
> --
>
> Key: KYLIN-3730
> URL: https://issues.apache.org/jira/browse/KYLIN-3730
> Project: Kylin
>  Issue Type: Bug
>  Components: Metadata
>Reporter: XiaoXiang Yu
>Assignee: XiaoXiang Yu
>Priority: Minor
> Fix For: v3.0.0
>
> Attachments: image-2018-12-19-18-37-48-831.png
>
>
> In TableMetadataManager.reloadSourceTableQuietly, we find that this method 
> decorate input resource name with prefix and postfix twice.  Such as 
> {color:#ff}+*/table/table/XXX.json.json.*+{color}
>  
> !image-2018-12-19-18-37-48-831.png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3730) TableMetadataManager.reloadSourceTableQuietly is wrong

[nichunen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nichunen updated KYLIN-3730:

Fix Version/s: v2.6.2
   v3.0.0

> TableMetadataManager.reloadSourceTableQuietly is wrong
> --
>
> Key: KYLIN-3730
> URL: https://issues.apache.org/jira/browse/KYLIN-3730
> Project: Kylin
>  Issue Type: Bug
>  Components: Metadata
>Reporter: XiaoXiang Yu
>Assignee: XiaoXiang Yu
>Priority: Minor
> Fix For: v3.0.0, v2.6.2
>
> Attachments: image-2018-12-19-18-37-48-831.png
>
>
> In TableMetadataManager.reloadSourceTableQuietly, we find that this method 
> decorate input resource name with prefix and postfix twice.  Such as 
> {color:#ff}+*/table/table/XXX.json.json.*+{color}
>  
> !image-2018-12-19-18-37-48-831.png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3756) Support check-port-availability script for mac os x

[nichunen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nichunen updated KYLIN-3756:

Fix Version/s: v3.0.0

> Support check-port-availability script for mac os x
> ---
>
> Key: KYLIN-3756
> URL: https://issues.apache.org/jira/browse/KYLIN-3756
> Project: Kylin
>  Issue Type: Improvement
>  Components: Tools, Build and Test
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Major
> Fix For: v3.0.0
>
>
> {{bin/check-port-availability.sh}} internally uses "netstat -tlpn" to detect 
> network ports. Running this script under mac os x will report the following 
> exception:
> {code:java}
> netstat: n: unknown or uninstrumented protocol{code}
> The "netstat" command under mac os x does not support the "n,p" option. We 
> can achieve the same function under mac with the "lsof" command. More 
> details: 
> https://stackoverflow.com/questions/4421633/who-is-listening-on-a-given-tcp-port-on-mac-os-x



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3474) Tableau 10.5 get malformed token (multi-query instance)

[nichunen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836969#comment-16836969
 ] 

nichunen commented on KYLIN-3474:
-

Committed to 2.6.x branch

> Tableau 10.5 get malformed token (multi-query instance)
> ---
>
> Key: KYLIN-3474
> URL: https://issues.apache.org/jira/browse/KYLIN-3474
> Project: Kylin
>  Issue Type: Bug
>  Components: Query Engine, Security
>Affects Versions: v2.3.0, v2.4.0, v2.5.0, v2.6.1
>Reporter: Temple Zhou
>Assignee: Temple Zhou
>Priority: Major
> Fix For: v2.6.2
>
> Attachments: KYLIN-3474.master.001.patch, kylin.log, 
> tableau-malformed-token.png
>
>
> I found that when I use Tableau 10.5 with Kylin 2.4.0(multi-query) , the 
> Tableau Server will get the malformed token error when do query via restful 
> API.
> However, when I use Tableau 10.5 with Kylin 2.4.0(single query instance), the 
> error will be gone.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3886) Missing argument for options for yarn command

[nichunen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836966#comment-16836966
 ] 

nichunen commented on KYLIN-3886:
-

Committed to 2.6.x branch

>  Missing argument for options for yarn command
> --
>
> Key: KYLIN-3886
> URL: https://issues.apache.org/jira/browse/KYLIN-3886
> Project: Kylin
>  Issue Type: Bug
>  Components: Spark Engine
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Major
> Fix For: v2.6.2
>
>
> The app id is empty which cause the yarn command failed for missing argument
>  
> {code:java}
> 2019-03-13 11:48:08,604 INFO [Scheduler 542945608 Job 
> f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 : Missing 
> argument for options
> 2019-03-13 11:48:08,606 INFO [Scheduler 542945608 Job 
> f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 : usage: 
> application
> 2019-03-13 11:48:08,606 INFO [Scheduler 542945608 Job 
> f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 : 
> -appStates  Works with -list to filter applications
> 2019-03-13 11:48:08,606 INFO [Scheduler 542945608 Job 
> f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 : based on 
> input comma-separated list of
> 2019-03-13 11:48:08,606 INFO [Scheduler 542945608 Job 
> f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 : 
> application states. The valid application
> 2019-03-13 11:48:08,606 INFO [Scheduler 542945608 Job 
> f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 : state 
> can be one of the following:
> 2019-03-13 11:48:08,606 INFO [Scheduler 542945608 Job 
> f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 : 
> ALL,NEW,NEW_SAVING,SUBMITTED,ACCEPTED,RUN
> 2019-03-13 11:48:08,606 INFO [Scheduler 542945608 Job 
> f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 : 
> NING,FINISHED,FAILED,KILLED
> 2019-03-13 11:48:08,606 INFO [Scheduler 542945608 Job 
> f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 : 
> -appTypes  Works with -list to filter applications
> 2019-03-13 11:48:08,606 INFO [Scheduler 542945608 Job 
> f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 : based on 
> input comma-separated list of
> 2019-03-13 11:48:08,607 INFO [Scheduler 542945608 Job 
> f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 : 
> application types.
> 2019-03-13 11:48:08,607 INFO [Scheduler 542945608 Job 
> f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 : -help 
> Displays help for all commands.
> 2019-03-13 11:48:08,607 INFO [Scheduler 542945608 Job 
> f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 : -kill 
>  Kills the application.
> 2019-03-13 11:48:08,607 INFO [Scheduler 542945608 Job 
> f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 : -list 
> List applications. Supports optional use
> 2019-03-13 11:48:08,607 INFO [Scheduler 542945608 Job 
> f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 : of 
> -appTypes to filter applications based{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3884) loading hfile to HBase failed for temporary dir in output path

[nichunen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836961#comment-16836961
 ] 

nichunen commented on KYLIN-3884:
-

Committed to 2.6.x branch

> loading hfile  to HBase failed for temporary dir in output path
> ---
>
> Key: KYLIN-3884
> URL: https://issues.apache.org/jira/browse/KYLIN-3884
> Project: Kylin
>  Issue Type: Bug
>  Components: Storage - HBase
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Major
> Fix For: v2.6.2
>
>
> {code:java}
> 2019-03-14 20:18:46,591 DEBUG [Scheduler 2084224398 Job 
> e48de76a-6e16-309f-a3a5-191c04071072-131] steps.BulkLoadJob:77 : Start to run 
> LoadIncrementalHFiles
> 2019-03-14 20:18:46,642 WARN  [Scheduler 2084224398 Job 
> e48de76a-6e16-309f-a3a5-191c04071072-131] mapreduce.LoadIncrementalHFiles:197 
> : Skipping non-directory 
> hdfs://zjyprc-xiaomi/user/s_kylin/kylin_zjyprc_bigdata_staging/kylin_zjyprc_bigdata_staging-kylin_metadata/kylin-e48de76a-6e16-309f-a3a5-191c04071072/total_user_cube/hfile/_SUCCESS
> 2019-03-14 20:18:46,650 ERROR [Scheduler 2084224398 Job 
> e48de76a-6e16-309f-a3a5-191c04071072-131] mapreduce.LoadIncrementalHFiles:352 
> : -
>   
> hdfs://zjyprc-xiaomi/user/s_kylin/kylin_zjyprc_bigdata_staging/kylin_zjyprc_bigdata_staging-kylin_metadata/kylin-e48de76a-6e16-309f-a3a5-191c04071072/total_user_cube/hfile/F1/4170d772384144848c1c10cba66152c3
>   
> hdfs://zjyprc-xiaomi/user/s_kylin/kylin_zjyprc_bigdata_staging/kylin_zjyprc_bigdata_staging-kylin_metadata/kylin-e48de76a-6e16-309f-a3a5-191c04071072/total_user_cube/hfile/F1/50ec331ff3c648e3b6e4f54a7b1fe7e9
>   
> hdfs://zjyprc-xiaomi/user/s_kylin/kylin_zjyprc_bigdata_staging/kylin_zjyprc_bigdata_staging-kylin_metadata/kylin-e48de76a-6e16-309f-a3a5-191c04071072/total_user_cube/hfile/F1/703ade3b535b4fedab39ee183e22aa7c
>   
> hdfs://zjyprc-xiaomi/user/s_kylin/kylin_zjyprc_bigdata_staging/kylin_zjyprc_bigdata_staging-kylin_metadata/kylin-e48de76a-6e16-309f-a3a5-191c04071072/total_user_cube/hfile/F1/82019f8ca00a4f16b9d2b45356a55a3a
>   
> hdfs://zjyprc-xiaomi/user/s_kylin/kylin_zjyprc_bigdata_staging/kylin_zjyprc_bigdata_staging-kylin_metadata/kylin-e48de76a-6e16-309f-a3a5-191c04071072/total_user_cube/hfile/F1/8cc8844bced24cb88fda52fecc7224d5
>   
> hdfs://zjyprc-xiaomi/user/s_kylin/kylin_zjyprc_bigdata_staging/kylin_zjyprc_bigdata_staging-kylin_metadata/kylin-e48de76a-6e16-309f-a3a5-191c04071072/total_user_cube/hfile/F1/cbac78e0c6d74b5c96a7b64f99e0d0b3
>   
> hdfs://zjyprc-xiaomi/user/s_kylin/kylin_zjyprc_bigdata_staging/kylin_zjyprc_bigdata_staging-kylin_metadata/kylin-e48de76a-6e16-309f-a3a5-191c04071072/total_user_cube/hfile/F1/e3844766a4d0486d89f287450034f378
>   
> hdfs://zjyprc-xiaomi/user/s_kylin/kylin_zjyprc_bigdata_staging/kylin_zjyprc_bigdata_staging-kylin_metadata/kylin-e48de76a-6e16-309f-a3a5-191c04071072/total_user_cube/hfile/_temporary/0
> 2019-03-14 20:18:46,651 ERROR [Scheduler 2084224398 Job 
> e48de76a-6e16-309f-a3a5-191c04071072-131] common.HadoopShellExecutable:65 : 
> error execute 
> HadoopShellExecutable{id=e48de76a-6e16-309f-a3a5-191c04071072-08, name=Load 
> HFile to HBase Table, state=RUNNING}
> java.io.FileNotFoundException: Path is not a file: 
> /user/s_kylin/kylin_zjyprc_bigdata_staging/kylin_zjyprc_bigdata_staging-kylin_metadata/kylin-e48de76a-6e16-309f-a3a5-191c04071072/total_user_cube/hfile/_temporary/0
> Caused by: 
> org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): Path is 
> not a file: 
> /user/s_kylin/kylin_zjyprc_bigdata_staging/kylin_zjyprc_bigdata_staging-kylin_metadata/kylin-e48de76a-6e16-309f-a3a5-191c04071072/total_user_cube/hfile/_temporary/0{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3839) Storage clean up after refreshing or deleting a segment

[nichunen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836960#comment-16836960
 ] 

nichunen commented on KYLIN-3839:
-

Committed to 2.6.x branch

> Storage clean up after refreshing or deleting a segment
> ---
>
> Key: KYLIN-3839
> URL: https://issues.apache.org/jira/browse/KYLIN-3839
> Project: Kylin
>  Issue Type: Improvement
>  Components: Others
>Reporter: Chao Long
>Assignee: Chao Long
>Priority: Major
> Fix For: v2.6.2, v3.0.0-alpha
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3888) TableNotDisabledException when running "Convert Lookup Table to HFile"

[nichunen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836959#comment-16836959
 ] 

nichunen commented on KYLIN-3888:
-

Committed to 2.6.x branch

> TableNotDisabledException when running "Convert Lookup Table to HFile"
> --
>
> Key: KYLIN-3888
> URL: https://issues.apache.org/jira/browse/KYLIN-3888
> Project: Kylin
>  Issue Type: Bug
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
>Priority: Major
> Fix For: v2.6.2, v3.0.0-alpha
>
>
> {code}
> org.apache.hadoop.hbase.TableNotDisabledException: 
> KYLIN_LOOKUP_DEFAULT.F_ADVERTISER_DIM_KYLIN_TEST_9WCVUDIXI5
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>   at 
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
>   at 
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
>   at 
> org.apache.hadoop.hbase.util.ForeignExceptionUtil.toIOException(ForeignExceptionUtil.java:45)
>   at 
> org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.convertResult(HBaseAdmin.java:4756)
>   at 
> org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4714)
>   at 
> org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4647)
>   at 
> org.apache.hadoop.hbase.client.HBaseAdmin.deleteTable(HBaseAdmin.java:904)
>   at 
> org.apache.kylin.storage.hbase.lookup.LookupTableToHFileJob.removeSnapshotIfExist(LookupTableToHFileJob.java:170)
>   at 
> org.apache.kylin.storage.hbase.lookup.LookupTableToHFileJob.run(LookupTableToHFileJob.java:104)
>   at 
> org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:131)
>   at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:165)
>   at 
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:70)
>   at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:165)
>   at 
> org.apache.kylin.job.impl.threadpool.DistributedScheduler$JobRunner.run(DistributedScheduler.java:110)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at java.lang.Thread.run(Thread.java:748)
> Caused by: 
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.TableNotDisabledException):
>  KYLIN_LOOKUP_DEFAULT.F_ADVERTISER_DIM_KYLIN_TEST_9WCVUDIXI5
>   at 
> org.apache.hadoop.hbase.master.HMaster.checkTableModifiable(HMaster.java:2266)
>   at 
> org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure.prepareDelete(DeleteTableProcedure.java:253)
>   at 
> org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure.executeFromState(DeleteTableProcedure.java:102)
>   at 
> org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure.executeFromState(DeleteTableProcedure.java:59)
>   at 
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:119)
>   at 
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:498)
>   at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1147)
>   at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:942)
>   at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:895)
>   at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:77)
>   at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:497)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [kylin] hit-lacus edited a comment on issue #624: Allow user-specified time format using real-time

[GitBox]

hit-lacus edited a comment on issue #624: Allow user-specified time format 
using real-time
URL: https://github.com/apache/kylin/pull/624#issuecomment-491174708
 
 
   Hi, could you please update your commit?  @GuoNingNing
   1. Use correct git user name/ user email which can link to your github 
profile page
   2. Merge your commit into two commit, one for backend change, another for 
front end change
   3. Create a JIRA, describe :
   - the problem you found
   - how you solve it
   - how to use it 
   - attach manual test screen shot
   4. Use KYLIN- in your git commit message 
   
   Thank you so much~


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (KYLIN-3880) DataType is incompatible in Kylin HBase coprocessor

[nichunen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836958#comment-16836958
 ] 

nichunen commented on KYLIN-3880:
-

Committed to 2.6.x branch

> DataType is incompatible in Kylin HBase coprocessor
> ---
>
> Key: KYLIN-3880
> URL: https://issues.apache.org/jira/browse/KYLIN-3880
> Project: Kylin
>  Issue Type: Bug
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Major
> Fix For: v2.6.2, v3.0.0-alpha
>
>
> During upgrade kylin from 2.4.1 to 2.5.2, the query will failed for the 
> incompatible class in Kylin HBase coprocessor
> {code:java}
> 2019-03-12,17:48:11,530 INFO 
> [FifoRWQ.default.readRpcServer.handler=197,queue=13,port=24600] 
> org.apache.hadoop.hdfs.DFSClient: Access token was invalid when connecting to 
> /10.152.33.45:22402 : 
> org.apache.hadoop.hdfs.security.token.block.InvalidBlockTokenException: Got 
> access token error for OP_READ_BLOCK, self=/10.152.33.44:55387, 
> remote=/10.152.33.45:22402, for file 
> /hbase/zjyprc-xiaomi/data/miui_sec/data/4b88a72f5bd37daca00efb842e676ca8/C/6593503eb213431998db117cf3dab3a6,
>  for pool BP-792581576-10.152.48.22-1510572454905 block 1899006034_825272806
> 2019-03-12,17:48:12,135 INFO 
> [FifoRWQ.default.readRpcServer.handler=231,queue=15,port=24600] 
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService: 
> start query dc0fadcf-3689-5508-9a45-559aaebfd4e0 in thread 
> FifoRWQ.default.readRpcServer.handler=231,queue=15,port=24600
> 2019-03-12,17:48:12,135 ERROR 
> [FifoRWQ.default.readRpcServer.handler=231,queue=15,port=24600] 
> org.apache.hadoop.ipc.RpcServer: Unexpected throwable object 
> java.lang.RuntimeException: java.io.InvalidClassException: 
> org.apache.kylin.metadata.datatype.DataType; local class incompatible: stream 
> classdesc serialVersionUID = -8891652700267537109, local class 
> serialVersionUID = -406124487097947
>   at 
> org.apache.kylin.cube.gridtable.TrimmedCubeCodeSystem.readDimensionEncoding(TrimmedCubeCodeSystem.java:87)
>   at 
> org.apache.kylin.cube.gridtable.TrimmedCubeCodeSystem$1.deserialize(TrimmedCubeCodeSystem.java:122)
>   at 
> org.apache.kylin.cube.gridtable.TrimmedCubeCodeSystem$1.deserialize(TrimmedCubeCodeSystem.java:91)
>   at org.apache.kylin.gridtable.GTInfo$1.deserialize(GTInfo.java:346)
>   at org.apache.kylin.gridtable.GTInfo$1.deserialize(GTInfo.java:307)
>   at 
> org.apache.kylin.gridtable.GTScanRequest$2.deserialize(GTScanRequest.java:466)
>   at 
> org.apache.kylin.gridtable.GTScanRequest$2.deserialize(GTScanRequest.java:412)
>   at 
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService.visitCube(CubeVisitService.java:259)
>   at 
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitService.callMethod(CubeVisitProtos.java:)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:6625)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.execServiceOnRegion(HRegionServer.java:4336)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.execService(HRegionServer.java:4318)
>   at 
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:34964)
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2059)
>   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:126)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:152)
>   at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:128)
>   at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.InvalidClassException: 
> org.apache.kylin.metadata.datatype.DataType; local class incompatible: stream 
> classdesc serialVersionUID = -8891652700267537109, local class 
> serialVersionUID = -406124487097947
>   at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:699)
>   at 
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1885)
>   at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1751)
>   at 
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2042)
>   at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
>   at 
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
>   at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
>   at 
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
>   at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
>   at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
>   at 
> 

[jira] [Commented] (KYLIN-3874) "Convert Cuboid Data to HFile" failed when HBase and MR use different HDFS clusters

[nichunen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836957#comment-16836957
 ] 

nichunen commented on KYLIN-3874:
-

Committed to 2.6.x branch

> "Convert Cuboid Data to HFile" failed when HBase and MR use different HDFS 
> clusters
> ---
>
> Key: KYLIN-3874
> URL: https://issues.apache.org/jira/browse/KYLIN-3874
> Project: Kylin
>  Issue Type: Bug
>  Components: Job Engine
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
>Priority: Major
> Fix For: v2.6.2
>
>
> {panel:title=exception}
> 2019-03-14 16:33:45,113 INFO [main] 
> org.apache.hadoop.service.AbstractService: Service 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state INITED; cause: 
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.IOException: 
> Couldn't create proxy provider class 
> org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
>  ...
>  Caused by: java.lang.RuntimeException: Could not find any configured 
> addresses for URI 
> hdfs://{color:#ff}hbase-ns-test{color}/kylin/kylin_metadata/kylin-cd4db32e-42b6-44f2-7fa9-ef6c46334249/derived_meta_global/hfile
>  at 
> org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider.(ConfiguredFailoverProxyProvider.java:116)
>  ... 31 more
> {panel}
> {color:#ff}hbase-ns-test{color} is the nameservice of HDFS which is used 
> by HBase. 
> In our kylin environment, HBase and MR/HIVE use different HDFS clusters.
> {code:java|title=CubeHFileJob.java|borderStyle=solid}
> // use current hbase configuration
> Configuration configuration = new 
> Configuration(HBaseConnection.getCurrentHBaseConfiguration());
> merge(configuration, getConf());
> {code}
> "getConf()" contains only the HDFS configuration of MR, and "configuration" 
> contains the HDFS configuration both HBase and MR. See: 
> [HBaseConnection#addHBaseClusterNNHAConfiguration()|https://github.com/apache/kylin/blob/d23d2bddab3263d39059492b91f71d8091198057/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/HBaseConnection.java#L205]
> So "merge(configuration, getConf())" will lead to HBASE's HDFS configuration 
> to be covered. Mainly parameter "dfs.nameservices"



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3882) kylin master build failed for pom issues

[nichunen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836954#comment-16836954
 ] 

nichunen commented on KYLIN-3882:
-

Committed to 2.6.x branch

> kylin master build failed for pom issues
> 
>
> Key: KYLIN-3882
> URL: https://issues.apache.org/jira/browse/KYLIN-3882
> Project: Kylin
>  Issue Type: Bug
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Major
> Fix For: v2.6.2
>
>
> As title.
> 1,  Kyligence repo id : nexus conflicts with local maven settings.xml
> {code:java}
> [ERROR] Failed to execute goal on project kylin-core-metadata: Could not 
> resolve dependencies for project 
> org.apache.kylin:kylin-core-metadata:jar:3.0.0-SNAPSHOT: Failure to find 
> org.apache.calcite:calcite-core:jar:1.16.0-kylin-r2 in 
> http://nexus.x./nexus/content/groups/public was cached in the local 
> repository, resolution will not be reattempted until the update interval of 
> nexus has elapsed or updates are forced -> [Help 1]
> {code}
>  
> 2, maven.compiler.source/target is not set
> {code:java}
> [INFO] Compiling 2 Scala sources and 18 Java sources to 
> /ssd/liushaohui/workspace/computing/kylin/engine-spark/target/classes ...
> [WARNING] [Warn] : bootstrap class path not set in conjunction with -source 
> 1.6
> [ERROR] [Error] 
> /ssd/liushaohui/workspace/computing/kylin/engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkBatchCubingJobBuilder2.java:148:
>  diamond operator is not supported in -source 1.6
>   (use -source 7 or higher to enable diamond operator)
> [ERROR] [Error] 
> /ssd/liushaohui/workspace/computing/kylin/engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkCubingByLayer.java:239:
>  try-with-resources is not supported in -source 1.6
>   (use -source 7 or higher to enable try-with-resources)
> [ERROR] [Error] 
> /ssd/liushaohui/workspace/computing/kylin/engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkCubingByLayer.java:251:
>  diamond operator is not supported in -source 1.6
>   (use -source 7 or higher to enable diamond operator){code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3571) Not build Spark in Kylin's binary package

[nichunen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836952#comment-16836952
 ] 

nichunen commented on KYLIN-3571:
-

Committed to 2.6.x branch

> Not build Spark in Kylin's binary package
> -
>
> Key: KYLIN-3571
> URL: https://issues.apache.org/jira/browse/KYLIN-3571
> Project: Kylin
>  Issue Type: Improvement
>  Components: Environment 
>Reporter: Shaofeng SHI
>Assignee: Chao Long
>Priority: Major
> Fix For: v2.6.2
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3830) Wrong result when 'SELECT SUM(dim1)' without set a relative metric of dim1.

[nichunen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836947#comment-16836947
 ] 

nichunen commented on KYLIN-3830:
-

Committed to 2.6.x branch

> Wrong result when 'SELECT SUM(dim1)' without set a relative metric of dim1.
> ---
>
> Key: KYLIN-3830
> URL: https://issues.apache.org/jira/browse/KYLIN-3830
> Project: Kylin
>  Issue Type: Bug
>Affects Versions: v2.5.2
>Reporter: Yuzhang QIU
>Assignee: Yuzhang QIU
>Priority: Major
> Fix For: v2.6.2, v3.0.0-alpha
>
>
> Hi, dear team:
>   I design an cube1 based on table table1 with dim1, dim2, dim3 and only 
> one metric count(1), and 'SELECT SUM(dim1) FROM table1 group by dim2', Kylin 
> process this SQL and return some result1. It seems ok. But as we know, Kylin 
> don't store the detail data, the dimensions' members have been encoded and 
> stored in Hbase as rowkey(cause I don't set any metric with an column). So, 
> is the result1 right?
>   Then, I clone cube1 to cube2, and set a metric SUM(dim1). the same SQL has 
> been passed to kylin and got result2. It's different from result1 at the 
> aggregation field. I also pass same SQL to hive and got result3, it's same 
> with result2.
>   Yes, I turn off the pushdown.
>   I think there are some problems.
>   I can't upload some picture of results for secret policy, sorry for that.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3817) Duration in Cube building is a negative number

[nichunen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836953#comment-16836953
 ] 

nichunen commented on KYLIN-3817:
-

Committed to 2.6.x branch

> Duration in Cube building is a negative number
> --
>
> Key: KYLIN-3817
> URL: https://issues.apache.org/jira/browse/KYLIN-3817
> Project: Kylin
>  Issue Type: Bug
>Affects Versions: v2.6.0
>Reporter: Na Zhai
>Assignee: Liu Shaohui
>Priority: Major
> Fix For: v2.6.2, v3.0.0-alpha
>
> Attachments: Duration error.png
>
>
> After building the cube for a few minutes, click the Refresh button and the 
> duration of the job becomes negative. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Issue Comment Deleted] (KYLIN-3571) Not build Spark in Kylin's binary package

[nichunen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nichunen updated KYLIN-3571:

Comment: was deleted

(was: Committed to 2.6.x branch)

> Not build Spark in Kylin's binary package
> -
>
> Key: KYLIN-3571
> URL: https://issues.apache.org/jira/browse/KYLIN-3571
> Project: Kylin
>  Issue Type: Improvement
>  Components: Environment 
>Reporter: Shaofeng SHI
>Assignee: Chao Long
>Priority: Major
> Fix For: v2.6.2
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3866) Whether to set mapreduce.application.classpath is determined by the user

[nichunen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836951#comment-16836951
 ] 

nichunen commented on KYLIN-3866:
-

Committed to 2.6.x branch

> Whether to set mapreduce.application.classpath is determined by the user
> 
>
> Key: KYLIN-3866
> URL: https://issues.apache.org/jira/browse/KYLIN-3866
> Project: Kylin
>  Issue Type: Bug
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
>Priority: Major
> Fix For: v2.6.2, v3.0.0-alpha
>
> Attachments: KYLIN-3866.master.001.patch
>
>
> In the kylin environment, if there is no configuration parameter 
> "mapreduce.application.classpath", then kylin will use the command "mapred 
> classpath" to get the local classpath and set it to 
> "mapreduce.application.classpath", However, it will cause the following two 
> problems:
> 1. The local classpath may not be the same as the MapReduce cluster.
> 2. There is no need to configure the parameter 
> "mapreduce.application.classpath" on the client side in some cluster.
> So, it may be a better choice for users to decide whether they need to 
> configure this parameter or not.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3571) Not build Spark in Kylin's binary package

[nichunen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nichunen updated KYLIN-3571:

Fix Version/s: (was: v2.6.1)
   v2.6.2

> Not build Spark in Kylin's binary package
> -
>
> Key: KYLIN-3571
> URL: https://issues.apache.org/jira/browse/KYLIN-3571
> Project: Kylin
>  Issue Type: Improvement
>  Components: Environment 
>Reporter: Shaofeng SHI
>Assignee: Chao Long
>Priority: Major
> Fix For: v2.6.2
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [kylin] hit-lacus edited a comment on issue #624: Allow user-specified time format using real-time

[GitBox]

hit-lacus edited a comment on issue #624: Allow user-specified time format 
using real-time
URL: https://github.com/apache/kylin/pull/624#issuecomment-491174708
 
 
   Hi, could you please update your commit?  @GuoNingNing
   1. Use correct git user name/ user email which can link to your github 
profile page
   2. Merge your commit into two commit, one for backend change, another for 
front end change
   3. Create a JIRA, describe the problem you found, how you solve it, and 
attach manual test screen shot
   4. Use KYLIN- in your git commit message 
   
   Thank you so much~


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Reopened] (KYLIN-3571) Not build Spark in Kylin's binary package

[nichunen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nichunen reopened KYLIN-3571:
-

Reopen to reset fix version

> Not build Spark in Kylin's binary package
> -
>
> Key: KYLIN-3571
> URL: https://issues.apache.org/jira/browse/KYLIN-3571
> Project: Kylin
>  Issue Type: Improvement
>  Components: Environment 
>Reporter: Shaofeng SHI
>Assignee: Chao Long
>Priority: Major
> Fix For: v2.6.1
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KYLIN-3571) Not build Spark in Kylin's binary package

[nichunen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nichunen resolved KYLIN-3571.
-
Resolution: Fixed

Committed to 2.6.x branch

> Not build Spark in Kylin's binary package
> -
>
> Key: KYLIN-3571
> URL: https://issues.apache.org/jira/browse/KYLIN-3571
> Project: Kylin
>  Issue Type: Improvement
>  Components: Environment 
>Reporter: Shaofeng SHI
>Assignee: Chao Long
>Priority: Major
> Fix For: v2.6.2
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3818) After Cube disabled, auto-merge cube job still running

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836938#comment-16836938
 ] 

ASF subversion and git services commented on KYLIN-3818:


Commit d40f062a31851da4784f4d007192e305ce28b628 in kylin's branch 
refs/heads/2.6.x from GinaZhai
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=d40f062 ]

KYLIN-3818 After Cube disabled, auto-merge cube job still running


> After Cube disabled, auto-merge cube job still running
> --
>
> Key: KYLIN-3818
> URL: https://issues.apache.org/jira/browse/KYLIN-3818
> Project: Kylin
>  Issue Type: Bug
>Affects Versions: v2.6.0
>Reporter: Na Zhai
>Assignee: Na Zhai
>Priority: Major
> Fix For: v3.0.0-alpha
>
>
> *precondition*
> There is a Cube that turns on the auto-merge feature. And it satisfied the 
> condition of the auto-merge. Then the job of merging segments begins.
> After a few minutes, the job of merging segments goes into the error status, 
> so I discard the job. Then I disable this cube, but a new job of merging 
> segments begin to run.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KYLIN-3818) After Cube disabled, auto-merge cube job still running

[nichunen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nichunen resolved KYLIN-3818.
-
Resolution: Fixed

Committed to 2.6.x branch

> After Cube disabled, auto-merge cube job still running
> --
>
> Key: KYLIN-3818
> URL: https://issues.apache.org/jira/browse/KYLIN-3818
> Project: Kylin
>  Issue Type: Bug
>Affects Versions: v2.6.0
>Reporter: Na Zhai
>Assignee: Na Zhai
>Priority: Major
> Fix For: v2.6.2, v3.0.0-alpha
>
>
> *precondition*
> There is a Cube that turns on the auto-merge feature. And it satisfied the 
> condition of the auto-merge. Then the job of merging segments begins.
> After a few minutes, the job of merging segments goes into the error status, 
> so I discard the job. Then I disable this cube, but a new job of merging 
> segments begin to run.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Reopened] (KYLIN-3818) After Cube disabled, auto-merge cube job still running

[nichunen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nichunen reopened KYLIN-3818:
-

Reopen to add fix version

> After Cube disabled, auto-merge cube job still running
> --
>
> Key: KYLIN-3818
> URL: https://issues.apache.org/jira/browse/KYLIN-3818
> Project: Kylin
>  Issue Type: Bug
>Affects Versions: v2.6.0
>Reporter: Na Zhai
>Assignee: Na Zhai
>Priority: Major
> Fix For: v3.0.0-alpha
>
>
> *precondition*
> There is a Cube that turns on the auto-merge feature. And it satisfied the 
> condition of the auto-merge. Then the job of merging segments begins.
> After a few minutes, the job of merging segments goes into the error status, 
> so I discard the job. Then I disable this cube, but a new job of merging 
> segments begin to run.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3818) After Cube disabled, auto-merge cube job still running

[nichunen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nichunen updated KYLIN-3818:

Fix Version/s: v2.6.2

> After Cube disabled, auto-merge cube job still running
> --
>
> Key: KYLIN-3818
> URL: https://issues.apache.org/jira/browse/KYLIN-3818
> Project: Kylin
>  Issue Type: Bug
>Affects Versions: v2.6.0
>Reporter: Na Zhai
>Assignee: Na Zhai
>Priority: Major
> Fix For: v2.6.2, v3.0.0-alpha
>
>
> *precondition*
> There is a Cube that turns on the auto-merge feature. And it satisfied the 
> condition of the auto-merge. Then the job of merging segments begins.
> After a few minutes, the job of merging segments goes into the error status, 
> so I discard the job. Then I disable this cube, but a new job of merging 
> segments begin to run.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [kylin] hit-lacus commented on issue #624: Allow user-specified time format using real-time

[GitBox]

hit-lacus commented on issue #624: Allow user-specified time format using 
real-time
URL: https://github.com/apache/kylin/pull/624#issuecomment-491174708
 
 
   Hi, could you please update your commit?
   
   1. Use correct git user name/ user email which can link to your github 
profile page
   2. Merge your commit into two commit, one for backend change, another for 
front end change
   3. Create a JIRA, describe the problem you found, how you solve it, and 
attach manual test screen shot
   4. Use KYLIN- in your git commit message 
   
   Thank you so much~


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Resolved] (KYLIN-2620) Check for "ORDER BY LIMIT" clause when rewrite SUM query as TOPN

[nichunen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-2620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nichunen resolved KYLIN-2620.
-
Resolution: Fixed

Committed to 2.6.x branch

> Check for "ORDER BY LIMIT" clause when rewrite SUM query as TOPN
> 
>
> Key: KYLIN-2620
> URL: https://issues.apache.org/jira/browse/KYLIN-2620
> Project: Kylin
>  Issue Type: Bug
>  Components: Measure - TopN
>Reporter: Lin Tingmao
>Assignee: Chao Long
>Priority: Major
> Fix For: v2.6.2
>
>
> When running the following query
> select sum(measure) from table group by col_id
> if there exists TOPN(measure, group by col_id)  measure, 
> TopNMeasureType.isTopNCompatibleSum()will pass, so the SUM is rewritten 
> to TOPN. This confuses the user since they may expect a accurate result for 
> every distinct value of group by column(s). 
> Kylin should check if "ORDER BY col_id LIMIT topncapacity" is present in the 
> query to determine whether to rewrite.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3838) Retry mechanism is invalid when build with spark

[nichunen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836922#comment-16836922
 ] 

nichunen commented on KYLIN-3838:
-

Committed to 2.6.x branch

> Retry mechanism is invalid when build with spark
> 
>
> Key: KYLIN-3838
> URL: https://issues.apache.org/jira/browse/KYLIN-3838
> Project: Kylin
>  Issue Type: Bug
>  Components: Spark Engine
>Reporter: Chao Long
>Assignee: Chao Long
>Priority: Major
> Fix For: v2.6.2
>
>
> description：http://mail-archives.apache.org/mod_mbox/kylin-dev/201902.mbox/%3C1550482835832-0.post%40n6.nabble.com%3E



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3808) RestAPI /api/jobs always returns 0 for exec_start_time and exec_end_time and exec_interrupt_time fields

[nichunen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836925#comment-16836925
 ] 

nichunen commented on KYLIN-3808:
-

Committed to 2.6.x branch

> RestAPI /api/jobs always returns 0 for exec_start_time and exec_end_time and 
> exec_interrupt_time fields
> ---
>
> Key: KYLIN-3808
> URL: https://issues.apache.org/jira/browse/KYLIN-3808
> Project: Kylin
>  Issue Type: Bug
>  Components: REST Service
>Affects Versions: v2.5.2
>Reporter: yaozq
>Assignee: yaozq
>Priority: Minor
> Fix For: v2.6.2
>
> Attachments: 
> 0001-KYLIN-3808-fix-RestAPI-api-jobs-always-returns-0-for.patch
>
>
> GET 
> [http://kylin-host:7070/kylin/api/jobs?cubeName=xxx=0=0=1]
> [
> {
> "uuid":"a89cf2d4-caaf-21fe-5cb9-63f1725d0e02",
> "last_modified":1549917430155,
> "version":"2.5.2.20500",
> "name":"BUILD CUBE - - 2019021100_2019021200 - GMT+08:00 
> 2019-02-12 04:10:32",
> "type":"BUILD",
> "duration":1583,
> "related_cube":"",
> "display_cube_name":"",
> "related_segment":"898fc7c8-5e30-927c-da26-816ab4dbff08",
> {color:#FF}"exec_start_time":0,{color}
> {color:#FF}"exec_end_time":0,{color}
> {color:#FF}"exec_interrupt_time":0,{color}
> "mr_waiting":179,
> ..
>  
>  exec_start_time, exec_end_time and exec_interrupt_time fields always return 0



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)