Hi Shaofeng ,
Thanks for replying.
Yes I am checking the yarn exception, But I find that this error comes while
importing the hive table to kylin.
Even if this error comes , hive tables is exported successfully in kylin. Is
this the reason why cube saving has failed ?
Next step when I go on creating the cube for following schema I get error
at last step while saving and I am unable to add any measures , clicking on
measure option just dont pop up any dialog,
I am using star schema with fact_sales as fact table and dim_* as dimension
tables.
fact_sales:
storeid int
itemid int
custid int
qty int
price double
dim_customer
custid int
name string
dim_item
itemid int
category string
brand string
color string
dim_store
storeid int
city string
state string
JSON is as below.
The JSON is as below.
{
"name": "Retail_Cube",
"description": "",
"dimensions": [
{
"name": "RETAIL.FACT_SALES.STOREID",
"table": "RETAIL.FACT_SALES",
"hierarchy": false,
"derived": null,
"column": [
"STOREID"
],
"id": 1
},
{
"name": "RETAIL.FACT_SALES.ITEMID",
"table": "RETAIL.FACT_SALES",
"hierarchy": false,
"derived": null,
"column": [
"ITEMID"
],
"id": 2
},
{
"name": "RETAIL.FACT_SALES.CUSTID",
"table": "RETAIL.FACT_SALES",
"hierarchy": false,
"derived": null,
"column": [
"CUSTID"
],
"id": 3
}
],
"measures": [
{
"id": 1,
"name": "_COUNT_",
"function": {
"expression": "COUNT",
"returntype": "bigint",
"parameter": {
"type": "constant",
"value": 1
}
}
}
],
"rowkey": {
"rowkey_columns": [
{
"column": "STOREID",
"length": 0,
"dictionary": "true",
"mandatory": false
},
{
"column": "ITEMID",
"length": 0,
"dictionary": "true",
"mandatory": false
},
{
"column": "CUSTID",
"length": 0,
"dictionary": "true",
"mandatory": false
}
],
"aggregation_groups": [
[
"STOREID",
"ITEMID",
"CUSTID"
]
]
},
"notify_list": [],
"capacity": "",
"hbase_mapping": {
"column_family": [
{
"name": "f1",
"columns": [
{
"qualifier": "m",
"measure_refs": [
"_COUNT_"
]
}
]
}
]
},
"project": "RetailProject",
"model_name": "Retail_Cube"
}
Regards,
Santosh Akhilesh
Bangalore R&D
HUAWEI TECHNOLOGIES CO.,LTD.
www.huawei.com
-------------------------------------------------------------------------------------------------------------------------------------
This e-mail and its attachments contain confidential information from HUAWEI,
which
is intended only for the person or entity whose address is listed above. Any
use of the
information contained herein in any way (including, but not limited to, total
or partial
disclosure, reproduction, or dissemination) by persons other than the intended
recipient(s) is prohibited. If you receive this e-mail in error, please notify
the sender by
phone or email immediately and delete it!
________________________________________
From: Shi, Shaofeng [[email protected]]
Sent: Thursday, February 26, 2015 7:01 AM
To: [email protected]
Subject: Re: Error while making cube & Measure option is not responding on GUI
Hi Santosh,
It looks like hadoop failed to execute some shell command in the
container; You need dive into hadoop to see what¹s the concrete error. You
can use yarn logs command to fetch all logs:
yarn logs -applicationId <app_id>
On 2/25/15, 7:39 PM, "Santosh Akhilesh" <[email protected]> wrote:
>Hi Luke / Shaofeng ,
> Can you please help me to check this issue.
>Regards,
>Santosh Akhilesh
>
>On Tue, Feb 24, 2015 at 10:41 PM, Santosh Akhilesh <
>[email protected]> wrote:
>
>> Hi All ,
>> is it because of following error in map reduce job ? what could
>>be
>> way to resolve this , a google search says that its issue of Yarn class
>> path , but I am not sure what it is ?
>>
>> Kylin Hive Column Cardinality Job table=RETAIL.FACT_SALES
>> output=/tmp/cardinality/RETAIL.FACT_SALES
>>
>> Application application_1424791969399_0008 failed 2 times due to AM
>> Container for appattempt_1424791969399_0008_000002 exited with
>>exitCode: 1
>> For more detailed output, check application tracking page:
>> http://santosh:8088/proxy/application_1424791969399_0008/Then, click on
>> links to logs of each attempt.
>> Diagnostics: Exception from container-launch.
>> Container id: container_1424791969399_0008_02_000001
>> Exit code: 1
>> Stack trace: ExitCodeException exitCode=1:
>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
>> at org.apache.hadoop.util.Shell.run(Shell.java:455)
>> at
>>
>>org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
>> at
>>
>>org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launch
>>Container(DefaultContainerExecutor.java:211)
>> at
>>
>>org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Conta
>>inerLaunch.call(ContainerLaunch.java:302)
>> at
>>
>>org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.Conta
>>inerLaunch.call(ContainerLaunch.java:82)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at
>>
>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java
>>:1145)
>> at
>>
>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav
>>a:615)
>> at java.lang.Thread.run(Thread.java:745)
>> Container exited with a non-zero exit code 1
>> Failing this attempt. Failing the application.
>>
>> ---------- Forwarded message ----------
>> From: Santoshakhilesh <[email protected]>
>> Date: Tue, Feb 24, 2015 at 7:41 PM
>> Subject: FW: Error while making cube & Measure option is not responding
>>on
>> GUI
>> To: "[email protected]" <[email protected]>
>>
>>
>> hi ,
>> please someone give me a hand to resolve this issue , thanks.
>>
>> Regards,
>> Santosh Akhilesh
>> Bangalore R&D
>> HUAWEI TECHNOLOGIES CO.,LTD.
>>
>> www.huawei.com
>>
>>
>>-------------------------------------------------------------------------
>>------------------------------------------------------------
>> This e-mail and its attachments contain confidential information from
>> HUAWEI, which
>> is intended only for the person or entity whose address is listed above.
>> Any use of the
>> information contained herein in any way (including, but not limited to,
>> total or partial
>> disclosure, reproduction, or dissemination) by persons other than the
>> intended
>> recipient(s) is prohibited. If you receive this e-mail in error, please
>> notify the sender by
>> phone or email immediately and delete it!
>>
>> ________________________________________
>> From: Santoshakhilesh [[email protected]]
>> Sent: Tuesday, February 24, 2015 12:55 PM
>> To: [email protected]
>> Cc: Kulbhushan Rana
>> Subject: FW: Error while making cube & Measure option is not responding
>>on
>> GUI
>>
>> 2. If I ignore and continue and try to save the cube I get an exception
>> in Kylin.log , I have checked the path is set correctly and
>>HCatInputFormat
>> this file is present in hive-hcatalog-core-0.14.0.jar . Please let me
>>know
>> what can I do to resolve this ?
>>
>> -- This was path issue , now no more exception in kylin.log
>>
>> But saveing cube still fails with error. And still can't add measures.
>>
>> Error Message
>> Failed to take action.
>>
>> In log I can find no exception. Following is the last log in kylin.log
>>
>> [pool-3-thread-1]:[2015-02-24
>>
>>20:47:15,613][INFO][org.apache.kylin.job.impl.threadpool.DefaultScheduler
>>$FetcherRunner.run(DefaultScheduler.java:117)]
>> - Job Fetcher: 0 running, 0 actual running, 0 ready, 6 others
>> [http-bio-7070-exec-2]:[2015-02-24
>>
>>20:47:51,610][DEBUG][org.apache.kylin.rest.controller.CubeController.dese
>>rializeDataModelDesc(CubeController.java:459)]
>> - Saving cube {
>> "name": "",
>> "fact_table": "RETAIL.FACT_SALES",
>> "lookups": [],
>> "filter_condition": "",
>> "capacity": "SMALL",
>> "partition_desc": {
>> "partition_date_column": "",
>> "partition_date_start": 0,
>> "partition_type": "APPEND"
>> },
>> "last_modified": 0
>> }
>>
>>
>> local access logs all with 200 , so seems ok.
>>
>> 10.18.146.105 - - [24/Feb/2015:20:46:56 +0800] "GET
>> /kylin/api/user/authentication HTTP/1.1" 200 246
>> 10.18.146.105 - - [24/Feb/2015:20:47:07 +0800] "GET
>> /kylin/api/user/authentication HTTP/1.1" 200 246
>> 10.18.146.105 - - [24/Feb/2015:20:47:27 +0800] "GET
>> /kylin/api/user/authentication HTTP/1.1" 200 246
>> 10.18.146.105 - - [24/Feb/2015:20:47:28 +0800] "GET
>> /kylin/api/user/authentication HTTP/1.1" 200 246
>> 10.18.146.105 - - [24/Feb/2015:20:47:34 +0800] "GET
>> /kylin/api/user/authentication HTTP/1.1" 200 246
>> 10.18.146.105 - - [24/Feb/2015:20:47:48 +0800] "GET
>> /kylin/api/user/authentication HTTP/1.1" 200 246
>> 10.18.146.105 - - [24/Feb/2015:20:47:51 +0800] "POST /kylin/api/cubes
>> HTTP/1.1" 200 701
>>
>>
>> Regards,
>> Santosh Akhilesh
>> Bangalore R&D
>> HUAWEI TECHNOLOGIES CO.,LTD.
>>
>> www.huawei.com
>>
>>
>>-------------------------------------------------------------------------
>>------------------------------------------------------------
>> This e-mail and its attachments contain confidential information from
>> HUAWEI, which
>> is intended only for the person or entity whose address is listed above.
>> Any use of the
>> information contained herein in any way (including, but not limited to,
>> total or partial
>> disclosure, reproduction, or dissemination) by persons other than the
>> intended
>> recipient(s) is prohibited. If you receive this e-mail in error, please
>> notify the sender by
>> phone or email immediately and delete it!
>>
>> ________________________________________
>> From: Santoshakhilesh [[email protected]]
>> Sent: Tuesday, February 24, 2015 12:09 PM
>> To: [email protected]
>> Cc: Kulbhushan Rana
>> Subject: Error while making cube & Measure option is not responding on
>>GUI
>>
>> Hi All ,
>>
>> I am building a simple cube for test and using the binary build
>>0.7.1
>> . I have following hive tables with columns.
>>
>>
>>
>> fact_sales:
>>
>> storeid int
>> itemid int
>> custid int
>> qty int
>> price double
>>
>> dim_customer
>> custid int
>> name string
>>
>> dim_item
>>
>> itemid int
>> category string
>> brand string
>> color string
>>
>> dim_store
>>
>> storeid int
>> city string
>> state string
>>
>> Please help me to answer following issues;
>>
>>
>>
>> 1. When I go to measure section and click on measure option , there is
>>no
>> response , I want add measure on qty and price with sum
>>
>> 2. If I ignore and continue and try to save the cube I get an exception
>> in Kylin.log , I have checked the path is set correctly and
>>HCatInputFormat
>> this file is present in hive-hcatalog-core-0.14.0.jar . Please let me
>>know
>> what can I do to resolve this ?
>>
>> 3. Also I have another question since this is a test and data is small I
>> have not partitioned the fact table , is it ok to skip partition stage
>> while cube build ?
>>
>>
>>
>> Exception
>>
>> pool-4-thread-4]:[2015-02-24
>>
>>19:26:32,577][ERROR][org.apache.kylin.job.impl.threadpool.DefaultSchedule
>>r$JobRunner.run(DefaultScheduler.java:134)]
>> - ExecuteException job:c3532a6f-97ea-474a-b36a-218dd517cedb
>> org.apache.kylin.job.exception.ExecuteException:
>> org.apache.kylin.job.exception.ExecuteException:
>> java.lang.NoClassDefFoundError:
>> org/apache/hive/hcatalog/mapreduce/HCatInputFormat
>> at
>>
>>org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecuta
>>ble.java:102)
>> at
>>
>>org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(Defau
>>ltScheduler.java:132)
>> at
>>
>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java
>>:1145)
>> at
>>
>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav
>>a:615)
>> at java.lang.Thread.run(Thread.java:745)
>> Caused by: org.apache.kylin.job.exception.ExecuteException:
>> java.lang.NoClassDefFoundError:
>> org/apache/hive/hcatalog/mapreduce/HCatInputFormat
>>
>>
>>
>> The JSON is as below.
>>
>>
>>
>> {
>> "name": "Retail_Cube",
>> "description": "",
>> "dimensions": [
>> {
>> "name": "RETAIL.FACT_SALES.STOREID",
>> "table": "RETAIL.FACT_SALES",
>> "hierarchy": false,
>> "derived": null,
>> "column": [
>> "STOREID"
>> ],
>> "id": 1
>> },
>> {
>> "name": "RETAIL.FACT_SALES.ITEMID",
>> "table": "RETAIL.FACT_SALES",
>> "hierarchy": false,
>> "derived": null,
>> "column": [
>> "ITEMID"
>> ],
>> "id": 2
>> },
>> {
>> "name": "RETAIL.FACT_SALES.CUSTID",
>> "table": "RETAIL.FACT_SALES",
>> "hierarchy": false,
>> "derived": null,
>> "column": [
>> "CUSTID"
>> ],
>> "id": 3
>> }
>> ],
>> "measures": [
>> {
>> "id": 1,
>> "name": "_COUNT_",
>> "function": {
>> "expression": "COUNT",
>> "returntype": "bigint",
>> "parameter": {
>> "type": "constant",
>> "value": 1
>> }
>> }
>> }
>> ],
>> "rowkey": {
>> "rowkey_columns": [
>> {
>> "column": "STOREID",
>> "length": 0,
>> "dictionary": "true",
>> "mandatory": false
>> },
>> {
>> "column": "ITEMID",
>> "length": 0,
>> "dictionary": "true",
>> "mandatory": false
>> },
>> {
>> "column": "CUSTID",
>> "length": 0,
>> "dictionary": "true",
>> "mandatory": false
>> }
>> ],
>> "aggregation_groups": [
>> [
>> "STOREID",
>> "ITEMID",
>> "CUSTID"
>> ]
>> ]
>> },
>> "notify_list": [],
>> "capacity": "",
>> "hbase_mapping": {
>> "column_family": [
>> {
>> "name": "f1",
>> "columns": [
>> {
>> "qualifier": "m",
>> "measure_refs": [
>> "_COUNT_"
>> ]
>> }
>> ]
>> }
>> ]
>> },
>> "project": "RetailProject",
>> "model_name": "Retail_Cube"
>> }
>>
>>
>>
>>
>>
>> Regards,
>> Santosh Akhilesh
>> Bangalore R&D
>> HUAWEI TECHNOLOGIES CO.,LTD.
>>
>> www.huawei.com
>>
>>
>>-------------------------------------------------------------------------
>>------------------------------------------------------------
>> This e-mail and its attachments contain confidential information from
>> HUAWEI, which
>> is intended only for the person or entity whose address is listed above.
>> Any use of the
>> information contained herein in any way (including, but not limited to,
>> total or partial
>> disclosure, reproduction, or dissemination) by persons other than the
>> intended
>> recipient(s) is prohibited. If you receive this e-mail in error, please
>> notify the sender by
>> phone or email immediately and delete it!
>>
>>
>>
>> --
>> Regards,
>> Santosh Akhilesh
>> +91-0-9845482201
>>
>
>
>
>--
>Regards,
>Santosh Akhilesh
>+91-0-9845482201
