答复: kylin cube build

2018-10-18 Thread Ge Silas
It looks that there are some Hive jars version conflict in your system. Maybe 
you can run bin/kylin.sh org.apache.kylin.common.util.ClasspathScanner 
ShutdownHookManager.class to find out the wrong jar file and exclude it from 
classpath.

Thanks,
Silas

发件人: elbert.lau 
发送时间: 2018年10月18日 14:39
收件人: dev@kylin.apache.org
主题: Re: kylin cube build

Hi,
How did you solve it? I have the same problem. My environment: HDP 2.6.4
Kylin 2.5/2.4.1

--
Sent from: http://apache-kylin.74782.x6.nabble.com/


答复: regrading model construction apache kylin

2018-09-25 Thread Ge Silas
Hi Vishnu,

You can find terminology definition here: 
http://kylin.apache.org/docs/gettingstarted/terminology.html

If there is only one table involved in the model for analysis, you can set it 
to fact table.

Thanks,
Silas
Apache Kylin | 
Terminology
Terminology. Here are some domain terms we are using in Apache Kylin, please 
check them for your reference. They are basic knowledge of Apache Kylin which 
also will help to well understand such concept, term, knowledge, theory and 
others about Data Warehouse, Business Intelligence for analycits.
kylin.apache.org



发件人: vishnu vardhan g 
发送时间: 2018年9月24日 19:07
收件人: dev@kylin.apache.org
主题: regrading model construction apache kylin

what's the need of lookup table in creation of model in apache kylin. If
there is onle one table do we need lookup table even then?

Thank you.


regrads

Vishnu Vardhan G

Software Intern

Orzota



答复: Build sample cude error, failed to read kylin_hive_conf.xml

2018-09-25 Thread Ge Silas
Hi Simon,

Can you please create a JIRA for this issue, I see others mentioning similar 
issues...

Best regards,
Silas

发件人: simon1118 
发送时间: 2018年9月23日 9:17
收件人: dev@kylin.apache.org
主题: Build sample cude error, failed to read kylin_hive_conf.xml

The following error always occur when I try to build kylin_sales_cube at step
cleanup Hive table:
org.apache.kylin.job.exception.ExecuteException:
org.apache.kylin.job.exception.ExecuteException: java.lang.RuntimeException:
Failed to read kylin_hive_conf.xml
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:178)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:113)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.kylin.job.exception.ExecuteException:
java.lang.RuntimeException: Failed to read kylin_hive_conf.xml
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:178)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:69)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:163)
... 4 more
Caused by: java.lang.RuntimeException: Failed to read kylin_hive_conf.xml
at
org.apache.kylin.common.util.SourceConfigurationUtil.loadXmlConfiguration(SourceConfigurationUtil.java:83)
at
org.apache.kylin.common.util.SourceConfigurationUtil.loadHiveConfiguration(SourceConfigurationUtil.java:57)
at
org.apache.kylin.common.util.HiveCmdBuilder.(HiveCmdBuilder.java:46)
at
org.apache.kylin.source.hive.GarbageCollectionStep.cleanUpIntermediateFlatTable(GarbageCollectionStep.java:61)
at
org.apache.kylin.source.hive.GarbageCollectionStep.doWork(GarbageCollectionStep.java:48)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:163)
... 6 more

After that, when I restart the kylin server, and resume the cube job, it
works fine.
My environment are:
Hadoop 2.9.1
Hbase 1.4.7
Hive 1.2.2
JDK1.8
Kylin 2.5.0

All running in docker, could anyone help me out?

--
Sent from: http://apache-kylin.74782.x6.nabble.com/


答复: String型日期做为分区字段,在cube编译后,sql无法查询。请帮忙看看原因,谢谢!

2018-09-20 Thread Ge Silas
Hi Gang,

I cannot see any of your pictures in the email. Can you please resend?

Thanks,
Silas

发件人: 梁刚 
发送时间: 2018年9月18日 15:36
收件人: dev@kylin.apache.org
主题: String型日期做为分区字段,在cube编译后,sql无法查询。请帮忙看看原因,谢谢!


你好,

我在用kylin的过程中遇到了个问题,想咨询一下。Hive表的分区字段是string型的,在建Model时,我指定这个分区字段格式为-MM-dd, 
当cube编译完成后,client端输入测试SQL:

[cid:image001.png@01D44F64.E6FED010]



可以成功查到Day字段的值:



[cid:image002.png@01D44F64.E6FED010]



但是, 当使用以Day做为查询条件的sql时,都会报错:

[cid:image005.png@01D44F65.10DAC150]



[cid:image006.png@01D44F65.69803C90]



想咨询一下,是否string类型的日期,在kylin里不能做为分区字段所导致的吗?谢谢!




答复: kylin config beelline error

2018-09-19 Thread Ge Silas
Hi hzxieyufeng,

The exact problem should be "Cannot modify dfs.replication at runtime". You may 
try to comment this configuration in kylin_hive_conf.xml, or set Hive 
configuraiton hive.security.authorization.sqlstd.confwhitelist to allow such 
configuration to be changed at runtime.

Thanks,
Silas

发件人: hzxieyufeng 
发送时间: 2018年9月12日 17:48
收件人: dev
主题: kylin config beelline error

hi,
I can't get hive metastore or mysql. I only use  hive with beeline mode. when 
is run sample.sh 。I meet this problem。


Retrieving hadoop conf dir...
Loading sample data into HDFS tmp path: /tmp/kylin/sample_cube/data
put: `/tmp/kylin/sample_cube/data/DEFAULT.KYLIN_ACCOUNT.csv': File exists
put: `/tmp/kylin/sample_cube/data/DEFAULT.KYLIN_CAL_DT.csv': File exists
put: `/tmp/kylin/sample_cube/data/DEFAULT.KYLIN_CATEGORY_GROUPINGS.csv': File 
exists
put: `/tmp/kylin/sample_cube/data/DEFAULT.KYLIN_COUNTRY.csv': File exists
put: `/tmp/kylin/sample_cube/data/DEFAULT.KYLIN_SALES.csv': File exists
Going to create sample tables in hive to database DEFAULT by beeline
ls: cannot access /home/recsys/hive-jd/lib/hive-jdbc-*-standalone.jar: No such 
file or directory
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in 
[jar:file:/home/recsys/hive-jd/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in 
[jar:file:/home/recsys/hadoop-jd/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Connecting to 
jdbc:hive2://hive1.jd.163.org:12181,hive2.jd.163.org:12181,spark5.jd.163.org:12181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hive-cluster9;principal=hive/_h...@hadoop2.hz.netease.com
18/09/12 17:23:38 [main]: INFO jdbc.HiveConnection: Connected to 
hive3.jd.163.org:10010
18/09/12 17:23:38 [main]: ERROR jdbc.HiveConnection: Error opening session
org.apache.thrift.protocol.TProtocolException: Required field 
'serverProtocolVersion' is unset! 
Struct:TOpenSessionResp(status:TStatus(statusCode:ERROR_STATUS, 
infoMessages:[*org.apache.hive.service.cli.HiveSQLException:Failed to open new 
session: java.lang.IllegalArgumentException: Cannot modify dfs.replication at 
runtime. It is not in list of params that are allowed to be modified at 
runtime:14:13, 
org.apache.hive.service.cli.session.SessionManager:createSession:SessionManager.java:336,
 
org.apache.hive.service.cli.session.SessionManager:openSession:SessionManager.java:279,
 
org.apache.hive.service.cli.CLIService:openSessionWithImpersonation:CLIService.java:189,
 
org.apache.hive.service.cli.thrift.ThriftCLIService:getSessionHandle:ThriftCLIService.java:431,
 
org.apache.hive.service.cli.thrift.ThriftCLIService:OpenSession:ThriftCLIService.java:313,
 
org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession:getResult:TCLIService.java:1377,
 
org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession:getResult:TCLIService.java:1362,
 org.apache.thrift.ProcessFunction:process:ProcessFunction.java:39, 
org.apache.thrift.TBaseProcessor:process:TBaseProcessor.java:39, 
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor:process:HadoopThriftAuthBridge.java:599,
 
org.apache.thrift.server.TThreadPoolServer$WorkerProcess:run:TThreadPoolServer.java:286,
 
java.util.concurrent.ThreadPoolExecutor:runWorker:ThreadPoolExecutor.java:1149, 
java.util.concurrent.ThreadPoolExecutor$Worker:run:ThreadPoolExecutor.java:624, 
java.lang.Thread:run:Thread.java:748, 
*org.apache.hive.service.cli.HiveSQLException:java.lang.IllegalArgumentException:
 Cannot modify dfs.replication at runtime. It is not in list of params that are 
allowed to be modified at runtime:27:13, 
org.apache.hive.service.cli.session.HiveSessionImpl:configureSession:HiveSessionImpl.java:264,
 
org.apache.hive.service.cli.session.HiveSessionImpl:open:HiveSessionImpl.java:189,
 sun.reflect.GeneratedMethodAccessor180:invoke::-1, 
sun.reflect.DelegatingMethodAccessorImpl:invoke:DelegatingMethodAccessorImpl.java:43,
 java.lang.reflect.Method:invoke:Method.java:498, 
org.apache.hive.service.cli.session.HiveSessionProxy:invoke:HiveSessionProxy.java:78,
 
org.apache.hive.service.cli.session.HiveSessionProxy:access$000:HiveSessionProxy.java:36,
 
org.apache.hive.service.cli.session.HiveSessionProxy$1:run:HiveSessionProxy.java:63,
 java.security.AccessController:doPrivileged:AccessController.java:-2, 
javax.security.auth.Subject:doAs:Subject.java:422, 
org.apache.hadoop.security.UserGroupInformation:doAs:UserGroupInformation.java:1698,
 
org.apache.hive.service.cli.session.HiveSessionProxy:invoke:HiveSessionProxy.java:59,
 com.sun.proxy.$Proxy48:open::-1, 
org.apache.hive.service.cli.session.SessionManager:createSession:SessionManager.java:327,
 *java.lang.IllegalArgumentException:Cannot modify 

答复: 答复: kylin merge

2018-09-05 Thread Ge Silas
Can you try increasing the Map memory settings in kylin_job_conf.xml or try to 
merge fewer segments at a time?

Thanks,
Silas

发件人: yujunliang <609417...@qq.com>
发送时间: 2018年9月4日 14:11
收件人: dev@kylin.apache.org
主题: Re: 答复: kylin merge

it's stuck on MapReduce side.Every segment has about less than 1 MB
data(source count:about 400 thousland)

the logs below:
Error: java.lang.OutOfMemoryError: Java heap space at
java.util.Arrays.copyOf(Arrays.java:3236) at
java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118) at
java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153) at
org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1793) at
org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1769) at
org.apache.commons.io.IOUtils.copy(IOUtils.java:1744) at
org.apache.kylin.common.persistence.FileResourceStore.getResourceImpl(FileResourceStore.java:123)
at
org.apache.kylin.common.persistence.ResourceStore.getResource(ResourceStore.java:165)
at org.apache.kylin.dict.DictionaryManager.load(DictionaryManager.java:404)
at org.apache.kylin.dict.DictionaryManager$1.load(DictionaryManager.java:83)
at org.apache.kylin.dict.DictionaryManager$1.load(DictionaryManager.java:80)
at
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3568)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350)
at
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228) at
com.google.common.cache.LocalCache.get(LocalCache.java:3965) at
com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3969) at
com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4829)
at
org.apache.kylin.dict.DictionaryManager.getDictionaryInfo(DictionaryManager.java:104)
at
org.apache.kylin.cube.CubeManager$DictionaryAssist.getDictionary(CubeManager.java:1028)
at org.apache.kylin.cube.CubeManager.getDictionary(CubeManager.java:961) at
org.apache.kylin.cube.CubeSegment.getDictionary(CubeSegment.java:324) at
org.apache.kylin.cube.kv.CubeDimEncMap.getDictionary(CubeDimEncMap.java:86)
at org.apache.kylin.cube.kv.CubeDimEncMap.get(CubeDimEncMap.java:65) at
org.apache.kylin.cube.kv.RowKeyColumnIO.getColumnLength(RowKeyColumnIO.java:43)
at org.apache.kylin.cube.kv.RowKeyEncoder.(RowKeyEncoder.java:61) at
org.apache.kylin.cube.kv.RowKeyEncoderProvider.getRowkeyEncoder(RowKeyEncoderProvider.java:41)
at
org.apache.kylin.engine.mr.steps.MergeCuboidMapper.doMap(MergeCuboidMapper.java:151)
at
org.apache.kylin.engine.mr.steps.MergeCuboidMapper.doMap(MergeCuboidMapper.java:61)
at org.apache.kylin.engine.mr.KylinMapper.map(KylinMapper.java:77) at
org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)

--
Sent from: 
https://apc01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fapache-kylin.74782.x6.nabble.com%2Fdata=02%7C01%7C%7C0ec43a0d0a2b46fb207708d6122e59e1%7C84df9e7fe9f640afb435%7C1%7C0%7C636716387613014969sdata=HkzTxpuklmmmLYt1is2WiBk089Y3XHm%2B8Pxp25Jg2Oo%3Dreserved=0


答复: kylin merge

2018-08-28 Thread Ge Silas
Hi,

Is the Kylin merge job stuck on Kylin side or MapReduce side? If it's stuck on 
MapReduce side, are there any suspicious log?

How many segments are you trying to merge? What about the data size?

发件人: yujunliang <609417...@qq.com>
发送时间: 2018年8月27日 17:31
收件人: dev@kylin.apache.org
主题: kylin merge

hi,kylin group:the cube merge job keeps running and it doesn't seem to run
out, but there is no error message. This kind of imagination came out after
upgrading from  kylin 1.6.0 to kylin 2.3.0. What should I do?

--
Sent from: http://apache-kylin.74782.x6.nabble.com/


答复: Configuration Overwrites the container memory for MR task not work

2018-07-18 Thread Ge Silas
Hi Xixin,

Can you try to configure these memory settings in conf/kylin_hive_conf.xml?

Thanks,
Silas

发件人: Xixin 
发送时间: 2018年7月19日 11:24
收件人: dev@kylin.apache.org
主题: Configuration Overwrites the container memory for MR task not work

Hi Kylin users/developers
Kylin 2.3.0, hadoop 2.6.5 cdh 5.12
A error occurred when I built a cube at step 2 Redistribute Flat Hive Table.
I checked the MR log and the log shows that reducers containers is running
beyond physical memory limits
below is detail logs:
Diagnostic Messages for this Task:
Container [pid=50920,containerID=container_1518617687620_384770_01_000287]
is running beyond physical memory limits. Current usage: 1.0 GB of 1 GB
physical memory used; 2.6 GB of 2.1 GB virtual memory used. Killing
container.

then I try to overwrites the related config
kylin.engine.mr.config-override.mapreduce.reduce.memory.mb = 4096
kylin.engine.mr.config-override.mapreduce.reduce.java.opts = -Xmx3g

then built the cube again, the the same error came again.
I check the yarn config mapreduce.reduce.memory.mb = 2G. this is confused
me. where kylin get the 1GB physical memory config limit? why the overwrites
not work?




--
Sent from: http://apache-kylin.74782.x6.nabble.com/


答复: kylin monitor "failed to load jobs"

2018-06-11 Thread Ge Silas
Hello,

I cannot see your attached image. Can you send it again?

Also please reload the monitor page and check backend kylin.log whether it 
reports any errors. This should give you further idea what is happening.

Regards,
Silas


发件人: chaoqun.520 
发送时间: 2018年6月10日 8:04
收件人: dev@kylin.apache.org
主题: kylin monitor "failed to load jobs"


hello,please help
how to resove this issue "failed to load jobs" in monitor?
thanks a lot!

[cid:7f31eff3$1$163e701c4ca$Coremail$chaoqun.520$163.com]






答复: timestampadd has some error

2018-05-16 Thread Ge Silas
Hi Jun,

1.6 is quite an old version. I did a quick test in Apache Kylin 2.3.1 and the 
issue is no longer there.

[cid:2a57adc5-118c-42e7-8cf1-c58133ad32f8]

Thanks,
Silas


发件人: jun <1063489...@qq.com>
发送时间: 2018年5月15日 15:56
收件人: dev@kylin.apache.org
主题: timestampadd has some error

kylin-1.6.0 version

1. SQL:  select part_dt, timestampadd(MONTH, 1, PART_DT) from KYLIN_SALES
result: 2012-01-01, 2012-01-01
MONTH result not correct;

2. SQL: select part_dt, timestampadd(DAY, 1, PART_DT) from KYLIN_SALES
result: 2012-01-01, 2012-01-02
DAY result corrcet

I found that DAY, HOUR, MINUTE correct, but MONTH and YEAR not work.

--
Sent from: http://apache-kylin.74782.x6.nabble.com/


Re: kylin installation issues--at stage Extract Fact Table Distince Columns

2018-03-16 Thread Ge Silas
Hello,

It looks the pictures you attached are missing.

What’s your Hadoop version?

Thanks,
Silas

On 9 Mar 2018, at 8:12 AM, 
zhangz...@missfresh.cn wrote:

Dear Kylin Group,

I've been installing Kylin1.6 for one week, and keeping having problems on the 
same issue. The following is the details:
I tried to build the example cube and always failed at the stage--Extract Fact 
Table Distince Columns.
[cid:_Foxmail.1@c548f8c7-e75c-eec2-27eb-2192c8cbda05]
and then i  googled online ,and find it was the classpath issue, so i try to 
add the following info in mapred-site.xml and yarn-site.xml
[cid:_Foxmail.1@b728fcf0-f90b-22cd-6a82-73d1ea0318df]
but still no working, and coming with issue show lacking of 
/home/hadoop/contrib/capacity-scheduler/*.jar. But i can't find
this dir at all.
so i google online again, and find the following solutions:
[cid:_Foxmail.1@285ead0b-eaa0-5119-a37a-3b1cafbaed68]
but my hadoop cluster is not under test environment, so i don't want to do this 
and resume my hadoop cluster.

So, please tell me am i on the right way to solve this issue, and please 
provide your kind suggestions.
 if you are looking for more details or info, please let me know.
Million thanks.


zaki





zhangz...@missfresh.cn



Re: [VOTE] Release apache-kylin-2.3.0 (RC3)

2018-03-01 Thread Ge Silas
+1

On 1 Mar 2018, at 2:35 PM, Jianhua Peng  wrote:

+1

On 2018/03/01 03:35:31, Billy Liu  wrote: 
> Hi all,
> 
> Now the known issue has been fixed, and all unit tests and integration
> tests passed. I have created a new build for Apache Kylin 2.3.0,
> release candidate 3.
> 
> Changes highlights:
>   [KYLIN-2727] - Introduce cube planner able to select cost-effective
> cuboids to be built by cost-based algorithms
>   [KYLIN-2726] - Introduce a dashboard for showing kylin service
> related metrics, like query count, query latency, job count, etc
>   [KYLIN-3125] - Support SparkSql in Cube building step â?oCreate
> Intermediate Flat Hive Table�
>   [KYLIN-3044] - Support SQL Server as data source
>   [KYLIN-2960] - Support user/group and role authentication for LDAP
> 
> And more than 240 issues fixed.
> 
> Thanks to everyone who has contributed to this release.
> Here's release notes:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12341685=12316121
> 
> The commit to be voted upon:
> https://github.com/apache/kylin/commit/1179ee72d0a2a2c629a649751ef43ae9e82dc87a
> 
> Its hash is 1179ee72d0a2a2c629a649751ef43ae9e82dc87a
> 
> The artifacts to be voted on are located here:
> https://dist.apache.org/repos/dist/dev/kylin/apache-kylin-2.3.0-rc3/
> 
> The hashes of the artifacts are as follows:
> apache-kylin-2.3.0-src.tar.gz.md5 138a96260dc2c8b8f9947b0bcbb11152
> apache-kylin-2.3.0-src.tar.gz.sha1 b305bb91144d2d1470bd1a8578a111732d04c6f8
> 
> A staged Maven repository is available for review at:
> https://repository.apache.org/content/repositories/orgapachekylin-1048/
> 
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/billyliu.asc
> 
> Please vote on releasing this package as Apache Kylin 2.3.0
> 
> The vote is open for the next 72 hours and passes if a majority of at
> least three +1 PPMC votes are cast.
> 
> [ ] +1 Release this package as Apache Kylin 2.3.0
> [ ]  0  I don't feel strongly about it, but I'm okay with the release
> [ ] -1  Do not release this package because...
> 
> Here is my vote:
> 
> +1 (binding)
> 
> With Warm regards
> 
> Billy Liu
> 



Re: Extract Fact Table Distinct Columns error

2018-03-01 Thread Ge Silas
Hello,

Can you check if you HBase was running fine during that time? I met a similar 
issue when HBase was not stable so the job status was not able to be 
synchronized.

Thanks,
Silas

On 22 Feb 2018, at 10:09 AM, xuzhe  wrote:

Hi, I built cube met something error. I couldn't solve it.
   Hadoop version : 2.7.3
   Hadoop NameNode HA
   ResouceManager HA

  log below:
  2018-02-22 09:05:04,510 INFO  [pool-8-thread-1]
threadpool.DefaultScheduler:123 : Job Fetcher: 1 should running, 1 actual
running, 0 stopped, 0 ready, 8 already succeed, 25 error, 61 discarded, 0
others
2018-02-22 09:05:34,499 INFO  [pool-8-thread-1]
threadpool.DefaultScheduler:123 : Job Fetcher: 1 should running, 1 actual
running, 0 stopped, 0 ready, 8 already succeed, 25 error, 61 discarded, 0
others
2018-02-22 09:06:04,499 INFO  [pool-8-thread-1]
threadpool.DefaultScheduler:123 : Job Fetcher: 1 should running, 1 actual
running, 0 stopped, 0 ready, 8 already succeed, 25 error, 61 discarded, 0
others
2018-02-22 09:06:34,499 INFO  [pool-8-thread-1]
threadpool.DefaultScheduler:123 : Job Fetcher: 1 should running, 1 actual
running, 0 stopped, 0 ready, 8 already succeed, 25 error, 61 discarded, 0
others
2018-02-22 09:06:59,485 INFO  [Job
9948d6ae-715a-4543-9d3b-8f9da133cb17-6583] mapred.ClientServiceDelegate:278
: Application state is completed. FinalApplicationStatus=SUCCEEDED.
Redirecting to job history server
2018-02-22 09:06:59,498 ERROR [Job
9948d6ae-715a-4543-9d3b-8f9da133cb17-6583] common.HadoopJobStatusChecker:58
: error check status
java.io.IOException: Job status not available 
at org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:331)
at org.apache.hadoop.mapreduce.Job.getStatus(Job.java:338)
at
org.apache.kylin.engine.mr.common.HadoopJobStatusChecker.checkStatus(HadoopJobStatusChecker.java:38)
at
org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:152)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:65)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:141)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2018-02-22 09:06:59,500 ERROR [Job
9948d6ae-715a-4543-9d3b-8f9da133cb17-6583] common.MapReduceExecutable:196 :
error execute
MapReduceExecutable{id=9948d6ae-715a-4543-9d3b-8f9da133cb17-02, name=Extract
Fact Table Distinct Columns, state=RUNNING}
java.lang.NullPointerException
at org.apache.hadoop.mapreduce.Job.getTrackingURL(Job.java:363)
at
org.apache.kylin.engine.mr.common.HadoopCmdOutput.getInfo(HadoopCmdOutput.java:62)
at
org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:161)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:65)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:141)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2018-02-22 09:06:59,516 INFO  [Job
9948d6ae-715a-4543-9d3b-8f9da133cb17-6583] execution.ExecutableManager:425 :
job id:9948d6ae-715a-4543-9d3b-8f9da133cb17-02 from RUNNING to ERROR
2018-02-22 09:06:59,534 INFO  [Job
9948d6ae-715a-4543-9d3b-8f9da133cb17-6583] execution.ExecutableManager:425 :
job id:9948d6ae-715a-4543-9d3b-8f9da133cb17 from RUNNING to ERROR
2018-02-22 09:06:59,535 DEBUG [Job
9948d6ae-715a-4543-9d3b-8f9da133cb17-6583] execution.AbstractExecutable:259
: no need to send email, user list is empty




--
Sent from: http://apache-kylin.74782.x6.nabble.com/



Re: Log split

2018-02-11 Thread Ge Silas
Hi Fei,

You may want to submit a Jira to track this enhancement?

Thanks,
Silas

On 2 Feb 2018, at 10:20 AM, Fei Yi  wrote:

Hi
Whether to consider the server log and query log stored in two log files,
The current log mixed together to find the content is not easy



Re: How to solve JDBC driver timezone problem?

2018-01-14 Thread Ge Silas
Hi Rayan,

Is this post related? 
http://apache-kylin.74782.x6.nabble.com/JDBC-query-result-Date-column-get-wrong-value-td5370.html

Best regards,
Shuangyin

On 13 Jan 2018, at 2:41 PM, Rayan J <378968...@qq.com> 
wrote:

Hi,

  I connect FineBI to kylin via the jdbc driver .But,the time and data
mismatched.

How to solve this problem?
Thanks.

--
Sent from: http://apache-kylin.74782.x6.nabble.com/



Re: As get Error

2017-12-06 Thread Ge Silas
Hi Dongpeng,

What is the exact SQL after you modified and what is the complete error message?

Thanks,
Silas

> On 7 Dec 2017, at 9:49 AM, Dongpeng Wang <apache_...@163.com> wrote:
> 
> Hi,
> 
> 
> It doesn't work.
> 
> Encountered "as \'\u5e01\u79cd\u4ee3\u7801\'" at line 1, column 16. Was 
> expecting one of:  "ORDER" ... "LIMIT" ... "OFFSET" ... "FETCH" ... 
> "FROM" ... "," ... "AS"  ... "AS"  ... "AS" 
>  ... "AS"  ... "AS" 
>  ... "." ... "NOT" ... "IN" ... "BETWEEN" ... 
> "LIKE" ... "SIMILAR" ... "=" ... ">" ... "<" ... "<=" ... ">=" ... "<>" ... 
> "!=" ... "+" ... "-" ... "*" ... "/" ... "||" ... "AND" ... "OR" ... "IS" ... 
> "MEMBER" ... "SUBMULTISET" ... "CONTAINS" ... "OVERLAPS" ... "EQUALS" ... 
> "PRECEDES" ... "SUCCEEDS" ... "IMMEDIATELY" ... "MULTISET" ... "[" ... 
> "UNION" ... "INTERSECT" ... "EXCEPT" ... "MINUS" ... "(" ...
> 
> 
> 
> apache_...@163.com
> 
> From: Ge Silas
> Date: 2017-12-06 18:33
> To: dev@kylin.apache.org
> Subject: Re: As get Error
> Hi,
> 
> Can you try quoting your Chinese alias in ``? Like SELECT CURR_CD AS `哈哈`?
> 
> Thanks,
> Silas
> 
>> On 6 Dec 2017, at 4:10 PM, Dongpeng Wang <apache_...@163.com> wrote:
>> 
>> Hi,
>> 
>> when I run this Sql,it Error.
>> 
>> but when Replace Chinese in English,it's can work. what reason?
>> 
>> 2017-12-06 16:04:11,103 DEBUG [Query 
>> 584c2617-76e0-41d6-b5b3-d7280813b7f3-71] util.PushDownUtil:93 : Query 
>> Pushdown runner org.apache.kylin.query.adhoc.PushDownRunnerJdbcImpl@2f3e6f43
>> 2017-12-06 16:04:11,104 INFO  [Query 
>> 584c2617-76e0-41d6-b5b3-d7280813b7f3-71] util.PushDownUtil:106 : the query 
>> is converted to SELECT CURR_CD AS 哈哈, CURR_NM AS 名称, DATE_CD AS 日期, ORG_CD 
>> AS 代码, ORG_NM AS 名称 
>>   , PSN_DEP_TER_PLUS AS 加值 
>> FROM ( 
>>   SELECT A1_IDM_DEP_ORG.CURR_CD, DIM_CURR.CURR_NM, 
>> A1_IDM_DEP_ORG.DATE_CD, A1_IDM_DEP_ORG.ORG_CD, DIM_ORG.ORG_NM 
>>   , SUM(A1_IDM_DEP_ORG.ACCT_BAL) AS PSN_DEP_TER_PLUS 
>>   FROM METRIC_AIA.A1_IDM_DEP_ORG 
>>   INNER JOIN METRIC_AIA.DIM_ACC ON A1_IDM_DEP_ORG.ACC_CD = 
>> DIM_ACC.ACC_CD 
>>   INNER JOIN METRIC_AIA.DIM_CURR ON A1_IDM_DEP_ORG.CURR_CD = 
>> DIM_CURR.CURR_CD 
>>   INNER JOIN METRIC_AIA.DIM_ORG ON A1_IDM_DEP_ORG.ORG_CD = 
>> DIM_ORG.ORG_CD 
>>   WHERE A1_IDM_DEP_ORG.CIF_TYP_CD IN ('C', 'P') 
>>   AND A1_IDM_DEP_ORG.MAT_CD NOT IN ('000') 
>>   AND DIM_ACC.LV1_CD IN ('2013', '2020') 
>>   GROUP BY A1_IDM_DEP_ORG.CURR_CD, DIM_CURR.CURR_NM, 
>> A1_IDM_DEP_ORG.DATE_CD, A1_IDM_DEP_ORG.ORG_CD, DIM_ORG.ORG_NM 
>> ) PSN_DEP_TER_PLUS 
>> WHERE 1 = 1
>> LIMIT 5 after schema completion
>> 2017-12-06 16:04:11,146 ERROR [Query 
>> 584c2617-76e0-41d6-b5b3-d7280813b7f3-71] service.QueryService:867 : pushdown 
>> engine failed current query too
>> org.apache.hive.service.cli.HiveSQLException: Error while compiling 
>> statement: FAILED: ParseException line 1:15 Failed to recognize predicate 
>> 'AS'. Failed rule: 'identifier' in selection target
>>   at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:262)
>>   at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:248)
>>   at 
>> org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:300)
>>   at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:241)
>>   at 
>> org.apache.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:437)
>>   at 
>> org.apache.commons.dbcp.DelegatingStatement.executeQuery(DelegatingStatement.java:208)
>>   at 
>> org.apache.commons.dbcp.DelegatingStatement.executeQuery(DelegatingStatement.java:208)
>>   at 
>> org.apache.kylin.query.adhoc.PushDownRunnerJdbcImpl.executeQuery(PushDownRunnerJdbcImpl.java:59)
>>   at 
>> org.apache.kylin.query.util.PushDownUtil.tryPushDownQuery(PushDownUtil.java:124)
>>   at 
>> org.apache.kylin.query.util.PushDownUtil.tryPushDownSelectQuery(PushDownUtil.java:64)
&

Re: As get Error

2017-12-06 Thread Ge Silas
Hi,

Can you try quoting your Chinese alias in ``? Like SELECT CURR_CD AS `哈哈`?

Thanks,
Silas

> On 6 Dec 2017, at 4:10 PM, Dongpeng Wang  wrote:
> 
> Hi,
> 
> when I run this Sql,it Error.
> 
> but when Replace Chinese in English,it's can work. what reason?
> 
> 2017-12-06 16:04:11,103 DEBUG [Query 584c2617-76e0-41d6-b5b3-d7280813b7f3-71] 
> util.PushDownUtil:93 : Query Pushdown runner 
> org.apache.kylin.query.adhoc.PushDownRunnerJdbcImpl@2f3e6f43
> 2017-12-06 16:04:11,104 INFO  [Query 584c2617-76e0-41d6-b5b3-d7280813b7f3-71] 
> util.PushDownUtil:106 : the query is converted to SELECT CURR_CD AS 哈哈, 
> CURR_NM AS 名称, DATE_CD AS 日期, ORG_CD AS 代码, ORG_NM AS 名称 
>, PSN_DEP_TER_PLUS AS 加值 
> FROM ( 
>SELECT A1_IDM_DEP_ORG.CURR_CD, DIM_CURR.CURR_NM, 
> A1_IDM_DEP_ORG.DATE_CD, A1_IDM_DEP_ORG.ORG_CD, DIM_ORG.ORG_NM 
>, SUM(A1_IDM_DEP_ORG.ACCT_BAL) AS PSN_DEP_TER_PLUS 
>FROM METRIC_AIA.A1_IDM_DEP_ORG 
>INNER JOIN METRIC_AIA.DIM_ACC ON A1_IDM_DEP_ORG.ACC_CD = 
> DIM_ACC.ACC_CD 
>INNER JOIN METRIC_AIA.DIM_CURR ON A1_IDM_DEP_ORG.CURR_CD = 
> DIM_CURR.CURR_CD 
>INNER JOIN METRIC_AIA.DIM_ORG ON A1_IDM_DEP_ORG.ORG_CD = 
> DIM_ORG.ORG_CD 
>WHERE A1_IDM_DEP_ORG.CIF_TYP_CD IN ('C', 'P') 
>AND A1_IDM_DEP_ORG.MAT_CD NOT IN ('000') 
>AND DIM_ACC.LV1_CD IN ('2013', '2020') 
>GROUP BY A1_IDM_DEP_ORG.CURR_CD, DIM_CURR.CURR_NM, 
> A1_IDM_DEP_ORG.DATE_CD, A1_IDM_DEP_ORG.ORG_CD, DIM_ORG.ORG_NM 
> ) PSN_DEP_TER_PLUS 
> WHERE 1 = 1
> LIMIT 5 after schema completion
> 2017-12-06 16:04:11,146 ERROR [Query 584c2617-76e0-41d6-b5b3-d7280813b7f3-71] 
> service.QueryService:867 : pushdown engine failed current query too
> org.apache.hive.service.cli.HiveSQLException: Error while compiling 
> statement: FAILED: ParseException line 1:15 Failed to recognize predicate 
> 'AS'. Failed rule: 'identifier' in selection target
>at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:262)
>at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:248)
>at 
> org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:300)
>at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:241)
>at 
> org.apache.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:437)
>at 
> org.apache.commons.dbcp.DelegatingStatement.executeQuery(DelegatingStatement.java:208)
>at 
> org.apache.commons.dbcp.DelegatingStatement.executeQuery(DelegatingStatement.java:208)
>at 
> org.apache.kylin.query.adhoc.PushDownRunnerJdbcImpl.executeQuery(PushDownRunnerJdbcImpl.java:59)
>at 
> org.apache.kylin.query.util.PushDownUtil.tryPushDownQuery(PushDownUtil.java:124)
>at 
> org.apache.kylin.query.util.PushDownUtil.tryPushDownSelectQuery(PushDownUtil.java:64)
>at 
> org.apache.kylin.rest.service.QueryService.execute(QueryService.java:864)
>at 
> org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:561)
>at 
> org.apache.kylin.rest.service.QueryService.query(QueryService.java:181)
>at 
> org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:415)
>at 
> org.apache.kylin.rest.controller.QueryController.query(QueryController.java:78)
>at sun.reflect.GeneratedMethodAccessor281.invoke(Unknown Source)
>at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>at java.lang.reflect.Method.invoke(Method.java:498)
>at 
> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
>at 
> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
>at 
> org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
>at 
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
>at 
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
>at 
> org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
>at 
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
>at 
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
>at 
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
>at 
> org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
>at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
>at 
> 

Re: Cross segment query error

2017-12-05 Thread Ge Silas
Hello,

What makes you think that the issue is related with reading from cross segment?

Thanks,
Silas

> On 1 Dec 2017, at 4:06 PM, hankunlun <811653...@qq.com> wrote:
> 
>   
> 
> 
> 
> 
> 
> The results of these two statements are the same. What is the reason, is it
> because of the failure to cross the segment?
> 
> 
> --
> Sent from: http://apache-kylin.74782.x6.nabble.com/



Re: auto merge

2017-12-02 Thread Ge Silas
Looks this job 1bc8fc5b-1bde-40e0-b913-72f0cc519665 is ongoing without any 
issues. Did it finish all its associated steps a bit later?

Thanks,
Silas

> 在 2017年12月3日,上午8:43,赵天烁  写道:
> 
> already dig into kylin.log and found the only exception below which might 
> related to the merge issue,any clue?
> 
> 2017-12-03 08:38:39,156 DEBUG [Thread-23387] model.Segments:180 : Cube 
> CUBE_UXIP_EVENT_PARAM2 has 1 building segments
> 2017-12-03 08:38:39,157 ERROR [Thread-23387] service.CacheService:88 : Error 
> in updateOnNewSegmentReady()
> java.lang.IllegalArgumentException
>at 
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:76)
>at 
> org.apache.kylin.cube.CubeManager.mergeSegments(CubeManager.java:547)
>at 
> org.apache.kylin.rest.service.CubeService.mergeCubeSegment(CubeService.java:618)
>at 
> org.apache.kylin.rest.service.CubeService.updateOnNewSegmentReady(CubeService.java:566)
>at 
> org.apache.kylin.rest.service.CubeService$$FastClassBySpringCGLIB$$17a07c0e.invoke()
>at 
> org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
>at 
> org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:669)
>at 
> org.apache.kylin.rest.service.CubeService$$EnhancerBySpringCGLIB$$e2b75386.updateOnNewSegmentReady()
>at 
> org.apache.kylin.rest.service.CacheService$1$1.run(CacheService.java:86)
> 2017-12-03 08:38:39,240 DEBUG [http-bio-7070-exec-53] service.AclService:217 
> : ACL of org.springframework.security.acls.domain.ObjectIdentityImpl[Type: 
> org.apache.kylin.job.JobInstance; Identifier: 
> 5e3e766e-00a5-46d2-bd37-b74fb6d1f2b8] deleted successfully.
> 2017-12-03 08:38:39,246 INFO  [Scheduler 71286977 Job 
> 1bc8fc5b-1bde-40e0-b913-72f0cc519665-1125] execution.ExecutableManager:421 : 
> job id:1bc8fc5b-1bde-40e0-b913-72f0cc519665-01 from RUNNING to SUCCEED
> 2017-12-03 08:38:39,262 INFO  [Scheduler 71286977 Job 
> 1bc8fc5b-1bde-40e0-b913-72f0cc519665-1125] execution.ExecutableManager:421 : 
> job id:1bc8fc5b-1bde-40e0-b913-72f0cc519665 from RUNNING to READY
> 2017-12-03 08:38:39,366 DEBUG [http-bio-7070-exec-53] service.AclService:217 
> : ACL of org.springframework.security.acls.domain.ObjectIdentityImpl[Type: 
> org.apache.kylin.job.JobInstance; Identifier: 
> 5e4160ee-6154-46e4-8f0a-3ee2fa85a688] deleted successfully.
> 2017-12-03 08:38:39,455 DEBUG [http-bio-7070-exec-53] service.AclService:217 
> : ACL of org.springframework.security.acls.domain.ObjectIdentityImpl[Type: 
> org.apache.kylin.job.JobInstance; Identifier: 
> 5e47a8ad-fdb5-43ed-9bcb-dcca4f280b97] deleted successfully.
> 2017-12-03 08:38:39,545 DEBUG [http-bio-7070-exec-53] service.AclService:217 
> : ACL of org.springframework.security.acls.domain.ObjectIdentityImpl[Type: 
> org.apache.kylin.job.JobInstance; Identifier: 
> 5e5518ef-dfff-4772-a353-a1b79e0c3ee4] deleted successfully.
> 2017-12-03 08:38:39,634 DEBUG [http-bio-7070-exec-53] service.AclService:217 
> : ACL of org.springframework.security.acls.domain.ObjectIdentityImpl[Type: 
> org.apache.kylin.job.JobInstance; Identifier: 
> 5e7b8422-a784-4752-b4b5-c55068fb558a] deleted successfully.
> 2017-12-03 08:38:39,721 DEBUG [http-bio-7070-exec-53] service.AclService:217 
> : ACL of org.springframework.security.acls.domain.ObjectIdentityImpl[Type: 
> org.apache.kylin.job.JobInstance; Identifier: 
> 5e87f7f5-7000-4846-959c-8eddbab83ddc] deleted successfully.
> 2017-12-03 08:38:39,808 DEBUG [http-bio-7070-exec-53] service.AclService:217 
> : ACL of org.springframework.security.acls.domain.ObjectIdentityImpl[Type: 
> org.apache.kylin.job.JobInstance; Identifier: 
> 5e8a6bcc-2945-4a28-a4c7-8a3f1dcec968] deleted successfully.
> 2017-12-03 08:38:39,853 INFO  [pool-8-thread-1] 
> threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0 actual 
> running, 0 stopped, 1 ready, 2106 already succeed, 77 error, 184 discarded, 0 
> others
> 2017-12-03 08:38:39,894 DEBUG [http-bio-7070-exec-53] service.AclService:217 
> : ACL of org.springframework.security.acls.domain.ObjectIdentityImpl[Type: 
> org.apache.kylin.job.JobInstance; Identifier: 
> 5e915dfb-f57f-494e-9323-2813ce6e9a02] deleted successfully.
> 2017-12-03 08:38:39,981 DEBUG [http-bio-7070-exec-53] service.AclService:217 
> : ACL of org.springframework.security.acls.domain.ObjectIdentityImpl[Type: 
> org.apache.kylin.job.JobInstance; Identifier: 
> 5e933f95-2863-4979-8034-e78e8baf1b3e] deleted successfully.
> 2017-12-03 08:38:40,063 INFO  [pool-8-thread-1] 
> threadpool.DefaultScheduler:113 : 
> CubingJob{id=1bc8fc5b-1bde-40e0-b913-72f0cc519665, name=MERGE CUBE - 
> CUBE_UXIP_EVENT_PARAM2 - 2017111800_2017112500 - GMT+08:00 2017-12-03 
> 08:38:10, state=READY} prepare to schedule
> 2017-12-03 08:38:40,063 INFO  [pool-8-thread-1] 
> threadpool.DefaultScheduler:116 : 
> CubingJob{id=1bc8fc5b-1bde-40e0-b913-72f0cc519665, name=MERGE CUBE - 
>