[jira] [Commented] (KYLIN-2262) Kafka Streaming Cube build error
[
https://issues.apache.org/jira/browse/KYLIN-2262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15735643#comment-15735643
]
QiLiFei commented on KYLIN-2262:
I created a newest streaming cube StreamingCube12 to test and still got
java.lang.NoSuchMethodError error ! Currently we sue the Kylin1.6 and
kafka_2.11-0.9.0-kafka-2.0.2.jar . Is that caused by the mis-mached version ?
Log and error as below :
[root@hadoop102 bin]# curl -X PUT --user ADMIN:KYLIN -H "Content-Type:
application/json;charset=utf-8" -d '{ "sourceOffsetStart": 0,
"sourceOffsetEnd": 9223372036854775807, "buildType": "BUILD"}'
http://172.31.18.12:7070/kylin/api/cubes/StreamingCube12/build2
Apache Tomcat/7.0.69 - Error report
HTTP Status 500 - Handler processing failed; nested exception
is java.lang.NoSuchMethodError:
org.apache.kafka.clients.consumer.KafkaConsumer.assign(Ljava/util/Collection;)Vtype Exception
reportmessage Handler processing failed; nested exception is
java.lang.NoSuchMethodError:
org.apache.kafka.clients.consumer.KafkaConsumer.assign(Ljava/util/Collection;)Vdescription
The server encountered an internal error that prevented it from fulfilling
this request.exception
org.springframework.web.util.NestedServletException: Handler processing
failed; nested exception is java.lang.NoSuchMethodError:
org.apache.kafka.clients.consumer.KafkaConsumer.assign(Ljava/util/Collection;)V
org.springframework.web.servlet.DispatcherServlet.triggerAfterCompletionWithError(DispatcherServlet.java:1275)
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:951)
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:867)
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:951)
org.springframework.web.servlet.FrameworkServlet.doPut(FrameworkServlet.java:864)
javax.servlet.http.HttpServlet.service(HttpServlet.java:653)
org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:827)
javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:201)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
org.springframework.security.web.authentication.ui.DefaultLoginPageGeneratingFilter.doFilter(DefaultLoginPageGeneratingFilter.java:91)
[jira] [Commented] (KYLIN-2261) Cleanup Hbase Storage issue
[ https://issues.apache.org/jira/browse/KYLIN-2261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15735639#comment-15735639 ] Billy Liu commented on KYLIN-2261: -- Thanks for pointing out. You are correct. The StorageCleanupJob is still under hbase component for 1.6.0. https://issues.apache.org/jira/browse/KYLIN-2144 did the refactor, but will be effective in next release 1.6.1. But the document was updated too early. > Cleanup Hbase Storage issue > --- > > Key: KYLIN-2261 > URL: https://issues.apache.org/jira/browse/KYLIN-2261 > Project: Kylin > Issue Type: Test > Components: Client - CLI >Affects Versions: v1.6.0 > Environment: CDH-5.7.2 > Hbase1.2 > Kylin1.6 >Reporter: QiLiFei >Priority: Critical > Labels: test > Original Estimate: 2h > Remaining Estimate: 2h > > When I try to run below command according doc > (http://kylin.apache.org/docs16/howto/howto_cleanup_storage.html), it will > always raise error "Error: Could not find or load main class > org.apache.kylin.tool.StorageCleanupJob" > Command : > /opt/kylin/bin/kylin.sh org.apache.kylin.tool.StorageCleanupJob --delete > false > > Is the class in the 'kylin-storage-hbase.jar ' ? > And it should be put into $KYLIN_HOME/lib/ , Right ? > I've put jar file in $KYLIN_HOME/lib & $HBase_Home/lib/ and set the 777 > authority . However it's still cannot working!! > If I'm wrong , please correct me ! Thanks. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-2261) Cleanup Hbase Storage issue
[ https://issues.apache.org/jira/browse/KYLIN-2261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15735631#comment-15735631 ] Billy Liu commented on KYLIN-2261: -- QiLiFei commented on KYLIN-2261: I downloaded the kylin source code and find that the org.apache.kylin.tool.StorageCleanupJob is not exist. It should be the one org.apache.kylin.storage.hbase.util.StorageCleanupJob . And finally the below command can work correctly in the CLI. I considered there will be a P2 bug assigned to the doc (http://kylin.apache.org/docs16/howto/howto_cleanup_storage.html)!!! Correct : ./kylin.sh org.apache.kylin.storage.hbase.util.StorageCleanupJob --delete false Invalid : ./kylin.sh org.apache.kylin.tool.StorageCleanupJob --delete false > Cleanup Hbase Storage issue > --- > > Key: KYLIN-2261 > URL: https://issues.apache.org/jira/browse/KYLIN-2261 > Project: Kylin > Issue Type: Test > Components: Client - CLI >Affects Versions: v1.6.0 > Environment: CDH-5.7.2 > Hbase1.2 > Kylin1.6 >Reporter: QiLiFei >Priority: Critical > Labels: test > Original Estimate: 2h > Remaining Estimate: 2h > > When I try to run below command according doc > (http://kylin.apache.org/docs16/howto/howto_cleanup_storage.html), it will > always raise error "Error: Could not find or load main class > org.apache.kylin.tool.StorageCleanupJob" > Command : > /opt/kylin/bin/kylin.sh org.apache.kylin.tool.StorageCleanupJob --delete > false > > Is the class in the 'kylin-storage-hbase.jar ' ? > And it should be put into $KYLIN_HOME/lib/ , Right ? > I've put jar file in $KYLIN_HOME/lib & $HBase_Home/lib/ and set the 777 > authority . However it's still cannot working!! > If I'm wrong , please correct me ! Thanks. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-2265) Adjust default config for resource-limit sandbox
[ https://issues.apache.org/jira/browse/KYLIN-2265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15735613#comment-15735613 ] Billy Liu commented on KYLIN-2265: -- I'd like to prepare a document introducing how to config Kylin, including sandbox and production environment. Some of them are already there in http://kylin.apache.org/docs16/install/advance_settings.html. But most of the config items are still not explained well. > Adjust default config for resource-limit sandbox > > > Key: KYLIN-2265 > URL: https://issues.apache.org/jira/browse/KYLIN-2265 > Project: Kylin > Issue Type: Improvement > Components: General >Affects Versions: v1.6.0 >Reporter: Billy Liu >Assignee: Billy Liu > Attachments: KYLIN-2265.patch > > > Most users deploy Kylin on their own sandbox for the first trial. Most > sandbox has most 8G memory. The most used sandboxes are HDP sandbox and CDH > sandbox. We'd better make the default kylin configuration convenient for > these sandbox environment. The suggestion includes reducing region-cut-gb, > hfile-size-gb, max-region-count, reduce-input-mb, > max-reducer-number,mapreduce.map.memory.mb,mapreduce.map.java.opts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-2264) Date error when use new streaming cube in Kylin1.6.0
[ https://issues.apache.org/jira/browse/KYLIN-2264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15735521#comment-15735521 ] Shaofeng SHI commented on KYLIN-2264: - I think this is expected. The property "kylin.rest.timezone" will be used to display the operation times on Web GUI, like last build time, etc. But it is just for Kylin's Web GUI, not affect the query result. In Cube, all the times has no timezone (it equals to GMT0); GMT0 time is also called UTC or Epoch time; Java's System.currentTimeMillis() value is UTC, which is used in streaming scenarios very often. You mentioned that the result is 8 hours earlier than your expectation, because you're comparing it with your local time which is GMT+8. If you adding the timezone diff, they're matched. In this post Julian has a good sharing about timestamp in Calciate and JDBC, suggest to check if you haven't: http://apache-kylin.74782.x6.nabble.com/JDBC-query-result-Date-column-get-wrong-value-td5370.html > Date error when use new streaming cube in Kylin1.6.0 > > > Key: KYLIN-2264 > URL: https://issues.apache.org/jira/browse/KYLIN-2264 > Project: Kylin > Issue Type: Bug > Components: streaming, Web >Affects Versions: v1.6.0 > Environment: Debian 3.2.54-2 x86_64 GNU/Linux >Reporter: WangSheng >Assignee: Shaofeng SHI > > I installed Kylin1.6.0 and built streaming cube successgfully.But I found two > problems which I didn't met in Kylin1.5.*. > First, segments' start/end time displayed on Kylin Web are earlier 8 hours > than my PC date, but streaming cube's Last Build Time and Create Time > displayed on Kylin Web are same with my PC date. Maybe something wrong when > Kylin Web transform the segments' start/end timestamp into date, but I'm not > sure. > Second, I did sql query from streaming cube, but the records' time related > columns like "HOUR_START" and "MINUTE_START" are all earlier 8 hours than my > PC time. I found that these time related columns' timestamp from HBase are > correct by remote debug, so I guess something wrong when Kylin server > transform these timestamp into date. > By the way, I only changed the "kylin.rest.timezone=GM+8" in file > "kylin.properties", and my PC date is same with my server date. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-2192) More Robust Global Dictionary
[ https://issues.apache.org/jira/browse/KYLIN-2192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15735518#comment-15735518 ] Dayue Gao commented on KYLIN-2192: -- Found a bug in AppendTrieDictionary, which cause Kylin to load all data of AppendTrieDictionary into memory when it's about to add values to it. Fixed in https://github.com/apache/kylin/commit/fbb7ed921a8b63c3b62cb85bf64fb79ba650431d > More Robust Global Dictionary > - > > Key: KYLIN-2192 > URL: https://issues.apache.org/jira/browse/KYLIN-2192 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Affects Versions: v1.5.4.1 >Reporter: Yerui Sun >Assignee: Yerui Sun > Fix For: v1.6.1 > > Attachments: KYLIN-2192.2.patch > > > Global dictionary have been released over 2 months, I've received some > feedbacks and bug reports. Here's the patch to make global dictionary more > robust, including some functional improvements. > * Break through 255 bytes limitation for value, but still recommend value > length less than 8K, avoiding stack overflow error; > * Fix 'Value not exists' or stack overflow error when dict size is larger > than 1GB, the root cause is similar with KYLIN-1834; A check tool also > provided for check corrupted or not of existing dict data; > * Support parallel dictionary building in one job server, used for parallel > segments building; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-2261) Cleanup Hbase Storage issue
[ https://issues.apache.org/jira/browse/KYLIN-2261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15735411#comment-15735411 ] QiLiFei commented on KYLIN-2261: I downloaded the kylin source code and find that the org.apache.kylin.tool.StorageCleanupJob is not exist. It should be the one org.apache.kylin.storage.hbase.util.StorageCleanupJob . And finally the below command can work correctly in the CLI. I considered there will be a P2 bug assigned to the doc (http://kylin.apache.org/docs16/howto/howto_cleanup_storage.html)!!! Correct : ./kylin.sh org.apache.kylin.storage.hbase.util.StorageCleanupJob --delete false Invalid : ./kylin.sh org.apache.kylin.tool.StorageCleanupJob --delete false > Cleanup Hbase Storage issue > --- > > Key: KYLIN-2261 > URL: https://issues.apache.org/jira/browse/KYLIN-2261 > Project: Kylin > Issue Type: Test > Components: Client - CLI >Affects Versions: v1.6.0 > Environment: CDH-5.7.2 > Hbase1.2 > Kylin1.6 >Reporter: QiLiFei >Priority: Critical > Labels: test > Original Estimate: 2h > Remaining Estimate: 2h > > When I try to run below command according doc > (http://kylin.apache.org/docs16/howto/howto_cleanup_storage.html), it will > always raise error "Error: Could not find or load main class > org.apache.kylin.tool.StorageCleanupJob" > Command : > /opt/kylin/bin/kylin.sh org.apache.kylin.tool.StorageCleanupJob --delete > false > > Is the class in the 'kylin-storage-hbase.jar ' ? > And it should be put into $KYLIN_HOME/lib/ , Right ? > I've put jar file in $KYLIN_HOME/lib & $HBase_Home/lib/ and set the 777 > authority . However it's still cannot working!! > If I'm wrong , please correct me ! Thanks. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (KYLIN-2262) Kafka Streaming Cube build error
[
https://issues.apache.org/jira/browse/KYLIN-2262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI reassigned KYLIN-2262:
---
Assignee: Shaofeng SHI
> Kafka Streaming Cube build error
> --
>
> Key: KYLIN-2262
> URL: https://issues.apache.org/jira/browse/KYLIN-2262
> Project: Kylin
> Issue Type: Test
> Components: Client - CLI
>Affects Versions: v1.6.0
> Environment: CDH1.5.7
> Kylin1.6
> KAFKA-2.0.2-1.2.0.2.p0.5
>Reporter: QiLiFei
>Assignee: Shaofeng SHI
>Priority: Blocker
> Labels: features
> Attachments: kylin.logError.txt
>
>
> When I build the kafka stream cube according to the doc
> (http://kylin.apache.org/docs16/tutorial/cube_streaming.html) , it always
> raise the error in the CLI
> {"url":"http://172.31.18.12:7070/kylin/api/cubes/StreamingCube9/build2","exception":null}
> From the kylin.log, there are only "Java.lang.NullPointerException"
> present!!I'm not sure what exactly happened there !!!Please give me some
> support !
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KYLIN-2262) Kafka Streaming Cube build error
[
https://issues.apache.org/jira/browse/KYLIN-2262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15735365#comment-15735365
]
Shaofeng SHI commented on KYLIN-2262:
-
>From the log with source code I see the cube named "StreamingCube9" was not
>found; Please double check the name and let us know whether it is true;
In the build API it doesn't pre-check this; we should add this and give an
meaningful error message. Thanks for the input!
> Kafka Streaming Cube build error
> --
>
> Key: KYLIN-2262
> URL: https://issues.apache.org/jira/browse/KYLIN-2262
> Project: Kylin
> Issue Type: Test
> Components: Client - CLI
>Affects Versions: v1.6.0
> Environment: CDH1.5.7
> Kylin1.6
> KAFKA-2.0.2-1.2.0.2.p0.5
>Reporter: QiLiFei
>Priority: Blocker
> Labels: features
> Attachments: kylin.logError.txt
>
>
> When I build the kafka stream cube according to the doc
> (http://kylin.apache.org/docs16/tutorial/cube_streaming.html) , it always
> raise the error in the CLI
> {"url":"http://172.31.18.12:7070/kylin/api/cubes/StreamingCube9/build2","exception":null}
> From the kylin.log, there are only "Java.lang.NullPointerException"
> present!!I'm not sure what exactly happened there !!!Please give me some
> support !
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KYLIN-2265) Adjust default config for resource-limit sandbox
[ https://issues.apache.org/jira/browse/KYLIN-2265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15735340#comment-15735340 ] Shaofeng SHI commented on KYLIN-2265: - Hi Billy, the change is okay for a sandbox environment; but the change is made on build/conf/* files, which are for making kylin binary builds, not just for sandbox; where will be ones for prodution env? > Adjust default config for resource-limit sandbox > > > Key: KYLIN-2265 > URL: https://issues.apache.org/jira/browse/KYLIN-2265 > Project: Kylin > Issue Type: Improvement > Components: General >Affects Versions: v1.6.0 >Reporter: Billy Liu >Assignee: Billy Liu > Attachments: KYLIN-2265.patch > > > Most users deploy Kylin on their own sandbox for the first trial. Most > sandbox has most 8G memory. The most used sandboxes are HDP sandbox and CDH > sandbox. We'd better make the default kylin configuration convenient for > these sandbox environment. The suggestion includes reducing region-cut-gb, > hfile-size-gb, max-region-count, reduce-input-mb, > max-reducer-number,mapreduce.map.memory.mb,mapreduce.map.java.opts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-2261) Cleanup Hbase Storage issue
[
https://issues.apache.org/jira/browse/KYLIN-2261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15734752#comment-15734752
]
QiLiFei commented on KYLIN-2261:
I did below all action :
1.cp the jar from tool to kylin/lib
2.Add " export HBASE_CLASSPATH_PREFIX=/etc/hbase/conf " & "export
HBASE_CLASSPATH=/opt/cloudera/parcels/CDH/lib/hbase"
into kylin.sh at the beginning
Do you mean the {HBASE_CLASSPATH_PREFIX} need a default value like your
"/usr/hdp/2.2.4.2-2/hbase/conf" ?I also added it however got the same error
to run ./kylin.sh org.apache.kylin.tool.StorageCleanupJob --delete false . I
paste the all info from CLI below.
KYLIN_HOME is set to /opt/kylin
KYLIN_HOME is set to /opt/kylin
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M;
support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: Using incremental CMS is deprecated
and will likely be removed in a future release
16/12/09 16:59:47 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree
module jar containing PrefixTreeCodec is not present. Continuing without it.
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M;
support was removed in 8.0
Logging initialized using configuration in
jar:file:/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/jars/hive-common-1.1.0-cdh5.7.2.jar!/hive-log4j.properties
HCAT_HOME not found, try to find hcatalog path from hadoop home
hive dependency:
[jira] [Issue Comment Deleted] (KYLIN-2261) Cleanup Hbase Storage issue
[
https://issues.apache.org/jira/browse/KYLIN-2261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
QiLiFei updated KYLIN-2261:
---
Comment: was deleted
(was: I did below all action :
1.cp the jar from tool to kylin/lib
2.Add " export HBASE_CLASSPATH_PREFIX=/etc/hbase/conf " & "export
HBASE_CLASSPATH=/opt/cloudera/parcels/CDH/lib/hbase"
into kylin.sh at the beginning
Do you mean the {HBASE_CLASSPATH_PREFIX} need a default value like your
"/usr/hdp/2.2.4.2-2/hbase/conf" ?I also added it however got the same error
to run ./kylin.sh org.apache.kylin.tool.StorageCleanupJob --delete false . I
paste the all info from CLI below.
KYLIN_HOME is set to /opt/kylin
KYLIN_HOME is set to /opt/kylin
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M;
support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: Using incremental CMS is deprecated
and will likely be removed in a future release
16/12/09 16:59:47 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree
module jar containing PrefixTreeCodec is not present. Continuing without it.
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M;
support was removed in 8.0
Logging initialized using configuration in
jar:file:/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/jars/hive-common-1.1.0-cdh5.7.2.jar!/hive-log4j.properties
HCAT_HOME not found, try to find hcatalog path from hadoop home
hive dependency:
[jira] [Commented] (KYLIN-2261) Cleanup Hbase Storage issue
[
https://issues.apache.org/jira/browse/KYLIN-2261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15734750#comment-15734750
]
QiLiFei commented on KYLIN-2261:
I did below all action :
1.cp the jar from tool to kylin/lib
2.Add " export HBASE_CLASSPATH_PREFIX=/etc/hbase/conf " & "export
HBASE_CLASSPATH=/opt/cloudera/parcels/CDH/lib/hbase"
into kylin.sh at the beginning
Do you mean the {HBASE_CLASSPATH_PREFIX} need a default value like your
"/usr/hdp/2.2.4.2-2/hbase/conf" ?I also added it however got the same error
to run ./kylin.sh org.apache.kylin.tool.StorageCleanupJob --delete false . I
paste the all info from CLI below.
KYLIN_HOME is set to /opt/kylin
KYLIN_HOME is set to /opt/kylin
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M;
support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: Using incremental CMS is deprecated
and will likely be removed in a future release
16/12/09 16:59:47 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree
module jar containing PrefixTreeCodec is not present. Continuing without it.
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M;
support was removed in 8.0
Logging initialized using configuration in
jar:file:/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/jars/hive-common-1.1.0-cdh5.7.2.jar!/hive-log4j.properties
HCAT_HOME not found, try to find hcatalog path from hadoop home
hive dependency:
[jira] [Issue Comment Deleted] (KYLIN-2261) Cleanup Hbase Storage issue
[
https://issues.apache.org/jira/browse/KYLIN-2261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
QiLiFei updated KYLIN-2261:
---
Comment: was deleted
(was: I did below action :
1.cp the jar from tool to kylin/lib
2.Add " export HBASE_CLASSPATH_PREFIX=/etc/hbase/conf " & "export
HBASE_CLASSPATH=/opt/cloudera/parcels/CDH/lib/hbase"
into kylin.sh at the beginning
Do you mean the {HBASE_CLASSPATH_PREFIX} need a default value like your
"/usr/hdp/2.2.4.2-2/hbase/conf" ?I also added it however got the same error
to run ./kylin.sh org.apache.kylin.tool.StorageCleanupJob --delete false . I
paste the all info from CLI below.
KYLIN_HOME is set to /opt/kylin
KYLIN_HOME is set to /opt/kylin
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M;
support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: Using incremental CMS is deprecated
and will likely be removed in a future release
16/12/09 16:59:47 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree
module jar containing PrefixTreeCodec is not present. Continuing without it.
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M;
support was removed in 8.0
Logging initialized using configuration in
jar:file:/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/jars/hive-common-1.1.0-cdh5.7.2.jar!/hive-log4j.properties
HCAT_HOME not found, try to find hcatalog path from hadoop home
hive dependency:
[jira] [Commented] (KYLIN-2261) Cleanup Hbase Storage issue
[
https://issues.apache.org/jira/browse/KYLIN-2261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15734747#comment-15734747
]
QiLiFei commented on KYLIN-2261:
I did below action :
1.cp the jar from tool to kylin/lib
2.Add " export HBASE_CLASSPATH_PREFIX=/etc/hbase/conf " & "export
HBASE_CLASSPATH=/opt/cloudera/parcels/CDH/lib/hbase"
into kylin.sh at the beginning
Do you mean the {HBASE_CLASSPATH_PREFIX} need a default value like your
"/usr/hdp/2.2.4.2-2/hbase/conf" ?I also added it however got the same error
to run ./kylin.sh org.apache.kylin.tool.StorageCleanupJob --delete false . I
paste the all info from CLI below.
KYLIN_HOME is set to /opt/kylin
KYLIN_HOME is set to /opt/kylin
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M;
support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: Using incremental CMS is deprecated
and will likely be removed in a future release
16/12/09 16:59:47 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree
module jar containing PrefixTreeCodec is not present. Continuing without it.
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M;
support was removed in 8.0
Logging initialized using configuration in
jar:file:/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/jars/hive-common-1.1.0-cdh5.7.2.jar!/hive-log4j.properties
HCAT_HOME not found, try to find hcatalog path from hadoop home
hive dependency:
[jira] [Updated] (KYLIN-2265) Adjust default config for resource-limit sandbox
[ https://issues.apache.org/jira/browse/KYLIN-2265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu updated KYLIN-2265: - Attachment: KYLIN-2265.patch Hi [~Shaofengshi], Could you help review this update patch? In this update, reduce region-cut-gb to 1 reduce hflie-size-gb to 1 reduce max-region-count to 100 reduce reduce-input-mb to 100 reduce max-reducer-number to 100 reduce mapper-input-rows to 20 reduce mapreduce.map.memory.mb to 1024 reduce mapreduce.map.java.opts to -Xmx700m reduce mapreduce.task.io.sort.mb to 100 > Adjust default config for resource-limit sandbox > > > Key: KYLIN-2265 > URL: https://issues.apache.org/jira/browse/KYLIN-2265 > Project: Kylin > Issue Type: Improvement > Components: General >Affects Versions: v1.6.0 >Reporter: Billy Liu >Assignee: Billy Liu > Attachments: KYLIN-2265.patch > > > Most users deploy Kylin on their own sandbox for the first trial. Most > sandbox has most 8G memory. The most used sandboxes are HDP sandbox and CDH > sandbox. We'd better make the default kylin configuration convenient for > these sandbox environment. The suggestion includes reducing region-cut-gb, > hfile-size-gb, max-region-count, reduce-input-mb, > max-reducer-number,mapreduce.map.memory.mb,mapreduce.map.java.opts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KYLIN-2265) Adjust default config for resource-limit sandbox
Billy Liu created KYLIN-2265: Summary: Adjust default config for resource-limit sandbox Key: KYLIN-2265 URL: https://issues.apache.org/jira/browse/KYLIN-2265 Project: Kylin Issue Type: Improvement Components: General Affects Versions: v1.6.0 Reporter: Billy Liu Assignee: Billy Liu Most users deploy Kylin on their own sandbox for the first trial. Most sandbox has most 8G memory. The most used sandboxes are HDP sandbox and CDH sandbox. We'd better make the default kylin configuration convenient for these sandbox environment. The suggestion includes reducing region-cut-gb, hfile-size-gb, max-region-count, reduce-input-mb, max-reducer-number,mapreduce.map.memory.mb,mapreduce.map.java.opts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
