[jira] [Updated] (KYLIN-3035) How to use Kylin on EMR with S3 as hbase storage
[
https://issues.apache.org/jira/browse/KYLIN-3035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shawn Wang updated KYLIN-3035:
--
Description:
Can somebody give an example of how to use kylin on EMR with S3 as hbase
storage, which support reuse the previously built cube on new EMR after the
original EMR has been terminated.
My purpose is simple:
1. use transient EMR cluster to build cubes
2. use a persistent cluster to handle query requests
Of course, the culsters should share same hbase storage, so I setup the cluster
to use S3 as hbase storage, after 2.2.0 fix the "HFile not written to S3"
issue, I have been built a sample cube successfully, using configurations:
EMR:
{noformat}
[
{
"Classification": "hbase-site",
"Properties": {
"hbase.rootdir": "s3://kylin-emrfs/hbase-production"
}
},
{
"Classification": "hbase",
"Properties": {
"hbase.emr.storageMode": "s3"
}
},
{
"Classification": "emrfs-site",
"Properties": {
"fs.s3.consistent": "true",
"fs.s3.consistent.metadata.tableName":
"KylinEmrFSMetadata"
}
}
]
{noformat}
kylin.propertities:
{noformat}
kylin.env.hdfs-working-dir=s3://kylin/kylin-emrfs/kylin-working-dir
kylin.server.mode=all
{noformat}
Then I create a new cluster with same EMR configuration and query mode for
kylin, kylin just can't startup with errors:
{noformat}
2017-11-13 07:33:44,415 INFO
[main-SendThread(ip-172-31-1-10.cn-north-1.compute.internal:2181)]
zookeeper.ClientCnxn:876 : Socket connection established to
ip-172-31-1-10.cn-north-1.compute.internal/172.31.1.10:2181, initiating session
2017-11-13 07:33:44,422 INFO
[main-SendThread(ip-172-31-1-10.cn-north-1.compute.internal:2181)]
zookeeper.ClientCnxn:1299 : Session establishment complete on server
ip-172-31-1-10.cn-north-1.compute.internal/172.31.1.10:2181, sessionid =
0x15fb4173c100156, negotiated timeout = 4
2017-11-13 07:33:48,380 DEBUG [main] hbase.HBaseConnection:279 : HTable
'kylin_metadata' already exists
Exception in thread "main" java.lang.IllegalArgumentException: Failed to find
metadata store by url: kylin_metadata@hbase
at
org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:89)
at
org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:101)
at
org.apache.kylin.rest.service.AclTableMigrationTool.checkIfNeedMigrate(AclTableMigrationTool.java:94)
at
org.apache.kylin.tool.AclTableMigrationCLI.main(AclTableMigrationCLI.java:41)
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed
after attempts=1, exceptions:
Mon Nov 13 07:33:48 UTC 2017, RpcRetryingCaller{globalStartTime=1510558428667,
pause=100, retries=1}, java.net.ConnectException: 拒绝连接
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:159)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:864)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:830)
at
org.apache.kylin.storage.hbase.HBaseResourceStore.internalGetFromHTable(HBaseResourceStore.java:385)
at
org.apache.kylin.storage.hbase.HBaseResourceStore.getFromHTable(HBaseResourceStore.java:363)
at
org.apache.kylin.storage.hbase.HBaseResourceStore.existsImpl(HBaseResourceStore.java:116)
at
org.apache.kylin.common.persistence.ResourceStore.exists(ResourceStore.java:144)
at
org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:84)
... 3 more
Caused by: java.net.ConnectException: 拒绝连接
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
at
org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupConnection(RpcClientImpl.java:416)
at
org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:722)
at
org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:909)
at
org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:873)
at
org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1244)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
at
[jira] [Created] (KYLIN-3035) How to use Kylin on EMR with S3 as hbase storage
Shawn Wang created KYLIN-3035:
-
Summary: How to use Kylin on EMR with S3 as hbase storage
Key: KYLIN-3035
URL: https://issues.apache.org/jira/browse/KYLIN-3035
Project: Kylin
Issue Type: Bug
Components: Metadata
Affects Versions: v2.2.0
Environment: EMR 5.5.0, Kylin 2.2.0
Reporter: Shawn Wang
Assignee: Shaofeng SHI
Can somebody give an example of how to use kylin on EMR with S3 as hbase
storage, which support reuse the previously built cube on new EMR after the
original EMR has been terminated.
My purpose is simple:
1. use transient EMR cluster to build cubes
2. use a persistent cluster to handle query requests
Of course, the culsters should share same hbase storage, so I setup the cluster
to use S3 as hbase storage, after 2.2.0 fix the "HFile not written to S3"
issue, I have been built a sample cube successfully, using configurations:
EMR:
{noformat}
[
{
"Classification": "hbase-site",
"Properties": {
"hbase.rootdir": "s3://kylin-emrfs/hbase-production"
}
},
{
"Classification": "hbase",
"Properties": {
"hbase.emr.storageMode": "s3"
}
},
{
"Classification": "emrfs-site",
"Properties": {
"fs.s3.consistent": "true",
"fs.s3.consistent.metadata.tableName":
"KylinEmrFSMetadata"
}
}
]
{noformat}
kylin.propertities:
{noformat}
kylin.env.hdfs-working-dir=s3://kylin/kylin-emrfs/kylin-working-dir
kylin.server.mode=query
{noformat}
Then I create a new cluster with same EMR configuration and query mode for
kylin, kylin just can't startup with errors:
{noformat}
2017-11-13 07:33:44,415 INFO
[main-SendThread(ip-172-31-1-10.cn-north-1.compute.internal:2181)]
zookeeper.ClientCnxn:876 : Socket connection established to
ip-172-31-1-10.cn-north-1.compute.internal/172.31.1.10:2181, initiating session
2017-11-13 07:33:44,422 INFO
[main-SendThread(ip-172-31-1-10.cn-north-1.compute.internal:2181)]
zookeeper.ClientCnxn:1299 : Session establishment complete on server
ip-172-31-1-10.cn-north-1.compute.internal/172.31.1.10:2181, sessionid =
0x15fb4173c100156, negotiated timeout = 4
2017-11-13 07:33:48,380 DEBUG [main] hbase.HBaseConnection:279 : HTable
'kylin_metadata' already exists
Exception in thread "main" java.lang.IllegalArgumentException: Failed to find
metadata store by url: kylin_metadata@hbase
at
org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:89)
at
org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:101)
at
org.apache.kylin.rest.service.AclTableMigrationTool.checkIfNeedMigrate(AclTableMigrationTool.java:94)
at
org.apache.kylin.tool.AclTableMigrationCLI.main(AclTableMigrationCLI.java:41)
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed
after attempts=1, exceptions:
Mon Nov 13 07:33:48 UTC 2017, RpcRetryingCaller{globalStartTime=1510558428667,
pause=100, retries=1}, java.net.ConnectException: 拒绝连接
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:159)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:864)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:830)
at
org.apache.kylin.storage.hbase.HBaseResourceStore.internalGetFromHTable(HBaseResourceStore.java:385)
at
org.apache.kylin.storage.hbase.HBaseResourceStore.getFromHTable(HBaseResourceStore.java:363)
at
org.apache.kylin.storage.hbase.HBaseResourceStore.existsImpl(HBaseResourceStore.java:116)
at
org.apache.kylin.common.persistence.ResourceStore.exists(ResourceStore.java:144)
at
org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:84)
... 3 more
Caused by: java.net.ConnectException: 拒绝连接
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
at
org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupConnection(RpcClientImpl.java:416)
at
org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:722)
at
org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:909)
at
org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:873)
at
[jira] [Commented] (KYLIN-3020) org.apache.hadoop.util.ToolRunner is not threadsafe and misused in kylin HadoopShellExecutable
[ https://issues.apache.org/jira/browse/KYLIN-3020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249204#comment-16249204 ] Billy Liu commented on KYLIN-3020: -- +1 > org.apache.hadoop.util.ToolRunner is not threadsafe and misused in kylin > HadoopShellExecutable > -- > > Key: KYLIN-3020 > URL: https://issues.apache.org/jira/browse/KYLIN-3020 > Project: Kylin > Issue Type: Bug >Reporter: Zhong Yanghong >Assignee: Ma Gang > Attachments: KYLIN-3020.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-3020) org.apache.hadoop.util.ToolRunner is not threadsafe and misused in kylin HadoopShellExecutable
[ https://issues.apache.org/jira/browse/KYLIN-3020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ma Gang updated KYLIN-3020: --- Attachment: KYLIN-3020.patch The root cause is GenericOptionsParser class is not thread safe, and when multiple jobs are running concurrently will cause some concurrency issue, attach the fix patch. > org.apache.hadoop.util.ToolRunner is not threadsafe and misused in kylin > HadoopShellExecutable > -- > > Key: KYLIN-3020 > URL: https://issues.apache.org/jira/browse/KYLIN-3020 > Project: Kylin > Issue Type: Bug >Reporter: Zhong Yanghong >Assignee: Ma Gang > Attachments: KYLIN-3020.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-3020) org.apache.hadoop.util.ToolRunner is not threadsafe and misused in kylin HadoopShellExecutable
[ https://issues.apache.org/jira/browse/KYLIN-3020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249192#comment-16249192 ] Ma Gang commented on KYLIN-3020: Sometimes NPE throws in create flat table steps or bulkload hfile steps, and often resume job make it work: java.lang.NullPointerException at org.apache.hadoop.util.GenericOptionsParser.processGeneralOptions(GenericOptionsParser.java:283) at org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:487) at org.apache.hadoop.util.GenericOptionsParser.(GenericOptionsParser.java:170) at org.apache.hadoop.util.GenericOptionsParser.(GenericOptionsParser.java:153) at org.apache.kylin.engine.mr.MRUtil.getParser(MRUtil.java:100) at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:90) at org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:126) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:153) at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:52) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:153) at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:158) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745 > org.apache.hadoop.util.ToolRunner is not threadsafe and misused in kylin > HadoopShellExecutable > -- > > Key: KYLIN-3020 > URL: https://issues.apache.org/jira/browse/KYLIN-3020 > Project: Kylin > Issue Type: Bug >Reporter: Zhong Yanghong >Assignee: Ma Gang > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-3034) In the models tree, the "Edit(JSON)" option is missing partly.
[ https://issues.apache.org/jira/browse/KYLIN-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] peng.jianhua updated KYLIN-3034: Attachment: 0001-KYLIN-3034-In-the-models-tree-the-Edit-JSON-option-i.patch > In the models tree, the "Edit(JSON)" option is missing partly. > -- > > Key: KYLIN-3034 > URL: https://issues.apache.org/jira/browse/KYLIN-3034 > Project: Kylin > Issue Type: Bug > Components: Web >Reporter: peng.jianhua >Assignee: peng.jianhua > Labels: patch > Attachments: > 0001-KYLIN-3034-In-the-models-tree-the-Edit-JSON-option-i.patch, > model_tree_action_missing_in_full_screen.PNG, > model_tree_action_missing_in_non-full_screen.PNG > > > 1. Open the Model page; > 2. Choose one model in model table, and click the "Action" button on right > side, then you can find the "Edit(JSON)" option is missing partly, refer to > [^model_tree_action_missing_in_full_screen.PNG] in full screen, and refer to > [^model_tree_action_missing_in_non-full_screen.PNG] in non-full screen; > 3. You can also find a scrollbar at the bottom, but you cannot click on it, > because when you move away from the "Action", the scrollbar disappears. > So we can modify models_tree.html, add the margin-right setting for "Action" > div, which can give more space to options. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-3034) In the models tree, the "Edit(JSON)" option is missing partly.
[ https://issues.apache.org/jira/browse/KYLIN-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] peng.jianhua updated KYLIN-3034: Description: 1. Open the Model page; 2. Choose one model in model table, and click the "Action" button on right side, then you can find the "Edit(JSON)" option is missing partly, refer to [^model_tree_action_missing_in_full_screen.PNG] in full screen, and refer to [^model_tree_action_missing_in_non-full_screen.PNG] in non-full screen; 3. You can also find a scrollbar at the bottom, but you cannot click on it, because when you move away from the "Action", the scrollbar disappears. So we can modify models_tree.html, add the margin-right setting for "Action" div, which can give more space to options, refer to [^0001-KYLIN-3034-In-the-models-tree-the-Edit-JSON-option-i.patch] was: 1. Open the Model page; 2. Choose one model in model table, and click the "Action" button on right side, then you can find the "Edit(JSON)" option is missing partly, refer to [^model_tree_action_missing_in_full_screen.PNG] in full screen, and refer to [^model_tree_action_missing_in_non-full_screen.PNG] in non-full screen; 3. You can also find a scrollbar at the bottom, but you cannot click on it, because when you move away from the "Action", the scrollbar disappears. So we can modify models_tree.html, add the margin-right setting for "Action" div, which can give more space to options. > In the models tree, the "Edit(JSON)" option is missing partly. > -- > > Key: KYLIN-3034 > URL: https://issues.apache.org/jira/browse/KYLIN-3034 > Project: Kylin > Issue Type: Bug > Components: Web >Reporter: peng.jianhua >Assignee: peng.jianhua > Labels: patch > Attachments: > 0001-KYLIN-3034-In-the-models-tree-the-Edit-JSON-option-i.patch, > model_tree_action_missing_in_full_screen.PNG, > model_tree_action_missing_in_non-full_screen.PNG > > > 1. Open the Model page; > 2. Choose one model in model table, and click the "Action" button on right > side, then you can find the "Edit(JSON)" option is missing partly, refer to > [^model_tree_action_missing_in_full_screen.PNG] in full screen, and refer to > [^model_tree_action_missing_in_non-full_screen.PNG] in non-full screen; > 3. You can also find a scrollbar at the bottom, but you cannot click on it, > because when you move away from the "Action", the scrollbar disappears. > So we can modify models_tree.html, add the margin-right setting for "Action" > div, which can give more space to options, refer to > [^0001-KYLIN-3034-In-the-models-tree-the-Edit-JSON-option-i.patch] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KYLIN-3034) In the models tree, the "Edit(JSON)" option is missing partly.
peng.jianhua created KYLIN-3034: --- Summary: In the models tree, the "Edit(JSON)" option is missing partly. Key: KYLIN-3034 URL: https://issues.apache.org/jira/browse/KYLIN-3034 Project: Kylin Issue Type: Bug Components: Web Reporter: peng.jianhua Assignee: peng.jianhua Attachments: model_tree_action_missing_in_full_screen.PNG, model_tree_action_missing_in_non-full_screen.PNG 1. Open the Model page; 2. Choose one model in model table, and click the "Action" button on right side, then you can find the "Edit(JSON)" option is missing partly, refer to [^model_tree_action_missing_in_full_screen.PNG] in full screen, and refer to [^model_tree_action_missing_in_non-full_screen.PNG] in non-full screen; 3. You can also find a scrollbar at the bottom, but you cannot click on it, because when you move away from the "Action", the scrollbar disappears. So we can modify models_tree.html, add the margin-right setting for "Action" div, which can give more space to options. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-3030) In the cubes table, the options of last column action are missing partly.
[ https://issues.apache.org/jira/browse/KYLIN-3030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249019#comment-16249019 ] Shaofeng SHI commented on KYLIN-3030: - Thanks jianhua; [~bob123] Guosheng, please review the patch. > In the cubes table, the options of last column action are missing partly. > - > > Key: KYLIN-3030 > URL: https://issues.apache.org/jira/browse/KYLIN-3030 > Project: Kylin > Issue Type: Bug > Components: Web >Reporter: peng.jianhua >Assignee: peng.jianhua > Labels: patch > Attachments: > 0001-KYLIN-3030-In-the-cubes-table-the-options-of-last-co.patch, > action_item_is_missing_partly1.png, action_item_is_missing_partly2.png > > > 1. Open the Model page; > 2. Choose one cube in table, and click the "Action" button, then you can find > serveral options, just like "Edit CubeDesc", "View Cube"; > 3. Then you can find the "Edit CubeDesc" option is missing partly, refer to > [^action_item_is_missing_partly1.png]; > 4. If you zoom out the browser window, you can even find not only "Edit > CubeDesc" but also "View Cube" is missing partly, refer to > [^action_item_is_missing_partly2.png] > So we can modify cubes.html, add the margin-right setting for div, which can > give more space to options. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KYLIN-3033) Provide API compatibility for hbase 2.0 release
Ted Yu created KYLIN-3033: - Summary: Provide API compatibility for hbase 2.0 release Key: KYLIN-3033 URL: https://issues.apache.org/jira/browse/KYLIN-3033 Project: Kylin Issue Type: Improvement Reporter: Ted Yu Compiling against hbase 2.0.0-alpha4 release, I got the following compilation errors: https://pastebin.com/yfejnTBE We should start preparing migration to hbase 2.0 compatible APIs. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (KYLIN-2975) Unclosed Statement in test
[
https://issues.apache.org/jira/browse/KYLIN-2975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Li resolved KYLIN-2975.
Resolution: Fixed
Fix Version/s: v2.3.0
Thanks Shengping. Patch merged to master branch.
> Unclosed Statement in test
> --
>
> Key: KYLIN-2975
> URL: https://issues.apache.org/jira/browse/KYLIN-2975
> Project: Kylin
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Shengping
>Priority: Minor
> Fix For: v2.3.0
>
> Attachments: KYLIN-2975.patch
>
>
> Some Statement resource is not closed in tests.
> e.g. from QuerACLTestUtil.java :
> {code}
> Statement statement = conn.createStatement();
> return statement.executeQuery(sql);
> {code}
> The statement should be closed after use.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
