[ https://issues.apache.org/jira/browse/KYLIN-4683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17173900#comment-17173900 ]
ASF GitHub Bot commented on KYLIN-4683: --------------------------------------- hit-lacus edited a comment on pull request #1351: URL: https://github.com/apache/kylin/pull/1351#issuecomment-671065519 1. Go to web page and re-enable Cube, you will see some failure, please don't be in a panic. ```sh 2020-08-09 23:23:10,437 INFO [qtp193625667-35] controller.AdminController:58 : receive assign request:AssignRequest{cubeName='UserAnalysisCube', partitions=[Partition [partitionId=0, partitionInfo=null], Partition [partitionId=5, partitionInfo=null], Partition [partitionId=10, partitionInfo=null], Partition [partitionId=15, partitionInfo=null], Partition [partitionId=13, partitionInfo=null], Partition [partitionId=2, partitionInfo=null], Partition [partitionId=8, partitionInfo=null], Partition [partitionId=12, partitionInfo=null], Partition [partitionId=14, partitionInfo=null], Partition [partitionId=9, partitionInfo=null], Partition [partitionId=11, partitionInfo=null], Partition [partitionId=1, partitionInfo=null], Partition [partitionId=4, partitionInfo=null], Partition [partitionId=6, partitionInfo=null], Partition [partitionId=7, partitionInfo=null], Partition [partitionId=3, partitionInfo=null]], startConsumers=true} 2020-08-09 23:23:10,437 INFO [qtp193625667-35] server.StreamingServer:351 : create and start new consumer for cube:UserAnalysisCube 2020-08-09 23:23:10,468 INFO [qtp193625667-35] kafka.KafkaSource:141 : Create StreamingConnector for Cube UserAnalysisCube, assignedPartitions [Partition [partitionId=0, partitionInfo=null], Partition [partitionId=5, partitionInfo=null], Partition [partitionId=10, partitionInfo=null], Partition [partitionId=15, partitionInfo=null], Partition [partitionId=13, partitionInfo=null], Partition [partitionId=2, partitionInfo=null], Partition [partitionId=8, partitionInfo=null], Partition [partitionId=12, partitionInfo=null], Partition [partitionId=14, partitionInfo=null], Partition [partitionId=9, partitionInfo=null], Partition [partitionId=11, partitionInfo=null], Partition [partitionId=1, partitionInfo=null], Partition [partitionId=4, partitionInfo=null], Partition [partitionId=6, partitionInfo=null], Partition [partitionId=7, partitionInfo=null], Partition [partitionId=3, partitionInfo=null]], startProtocol null 2020-08-09 23:23:10,470 INFO [qtp193625667-35] kafka.TimedJsonStreamParser:98 : Using parser field mapping by {content_list=content_list, act_type=act_type, event_ts_2=event_ts_2, event_ts=event_ts, devide_type=user_detail.devide_type, location_city=user_detail.location.city, network_type=user_detail.network_type, video_id=video_id, event_date_2=event_date_2, str_minute=str_minute, video_type=video_type, play_times=play_times, pageview_id=pageview_id, active_minutes=active_minutes, device_brand=device_brand, str_minute_second=str_minute_second, play_duration=play_duration, event_date=event_date, page_id=page_id, str_second=str_second, uid=uid} 2020-08-09 23:23:10,470 INFO [qtp193625667-35] kafka.TimedJsonStreamParser:120 : TimedJsonStreamParser with formatTs false tsColName event_date 2020-08-09 23:23:10,470 INFO [qtp193625667-35] consumer.ConsumerConfig:279 : ConsumerConfig values: auto.commit.interval.ms = 5000 auto.offset.reset = latest bootstrap.servers = [cdh-master:9092, cdh-worker-1:9092, cdh-worker-2:9092] check.crcs = true client.dns.lookup = default client.id = connections.max.idle.ms = 540000 default.api.timeout.ms = 60000 enable.auto.commit = false exclude.internal.topics = true fetch.max.bytes = 52428800 fetch.max.wait.ms = 500 fetch.min.bytes = 1 group.id = heartbeat.interval.ms = 3000 interceptor.classes = [] internal.leave.group.on.close = true isolation.level = read_uncommitted key.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer max.partition.fetch.bytes = 1048576 max.poll.interval.ms = 300000 max.poll.records = 500 metadata.max.age.ms = 300000 metric.reporters = [] metrics.num.samples = 2 metrics.recording.level = INFO metrics.sample.window.ms = 30000 partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor] receive.buffer.bytes = 65536 reconnect.backoff.max.ms = 1000 reconnect.backoff.ms = 50 request.timeout.ms = 30000 retry.backoff.ms = 100 sasl.client.callback.handler.class = null sasl.jaas.config = null sasl.kerberos.kinit.cmd = /usr/bin/kinit sasl.kerberos.min.time.before.relogin = 60000 sasl.kerberos.service.name = null sasl.kerberos.ticket.renew.jitter = 0.05 sasl.kerberos.ticket.renew.window.factor = 0.8 sasl.login.callback.handler.class = null sasl.login.class = null sasl.login.refresh.buffer.seconds = 300 sasl.login.refresh.min.period.seconds = 60 sasl.login.refresh.window.factor = 0.8 sasl.login.refresh.window.jitter = 0.05 sasl.mechanism = GSSAPI security.protocol = PLAINTEXT send.buffer.bytes = 131072 session.timeout.ms = 20000 ssl.cipher.suites = null ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1] ssl.endpoint.identification.algorithm = https ssl.key.password = null ssl.keymanager.algorithm = SunX509 ssl.keystore.location = null ssl.keystore.password = null ssl.keystore.type = JKS ssl.protocol = TLS ssl.provider = null ssl.secure.random.implementation = null ssl.trustmanager.algorithm = PKIX ssl.truststore.location = null ssl.truststore.password = null ssl.truststore.type = JKS value.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer 2020-08-09 23:23:10,473 INFO [qtp193625667-35] utils.AppInfoParser:109 : Kafka version : 2.1.1 2020-08-09 23:23:10,473 INFO [qtp193625667-35] utils.AppInfoParser:110 : Kafka commitId : 21234bee31165527 2020-08-09 23:23:10,475 INFO [qtp193625667-35] kafka.KafkaSource:192 : localConsumeStats from local checkpoint null, remoteConsumeStats from remote checkpoint {"0":363171,"1":363198,"2":363249,"3":363171,"4":363199,"5":363250,"6":363170,"7":363196,"8":363250,"9":363170} 2020-08-09 23:23:10,475 INFO [qtp193625667-35] kafka.KafkaSource:239 : start kafka connector from specified position:KafkaPosition{partitionOffsetMap={0=363171, 1=363198, 2=363249, 3=363171, 4=363199, 5=363250, 6=363170, 7=363196, 8=363250, 9=363170}} 2020-08-09 23:23:10,476 ERROR [qtp193625667-35] server.StreamingServer:357 : consumer start fail for cube:UserAnalysisCube java.lang.NullPointerException at org.apache.kylin.stream.source.kafka.consumer.KafkaConnector.open(KafkaConnector.java:86) at org.apache.kylin.stream.core.consumer.StreamingConsumerChannel.start(StreamingConsumerChannel.java:85) at org.apache.kylin.stream.server.StreamingServer.startConsumer(StreamingServer.java:355) at org.apache.kylin.stream.server.StreamingServer.startConsumers(StreamingServer.java:327) at org.apache.kylin.stream.server.rest.controller.AdminController.assign(AdminController.java:61) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:849) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:760) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:755) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:547) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:500) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) at java.lang.Thread.run(Thread.java:748) ``` 2. Let try doing a reassignment. ![image](https://user-images.githubusercontent.com/14030549/89735852-e8ddfa80-da97-11ea-8ae3-744e275dab0d.png) ![image](https://user-images.githubusercontent.com/14030549/89735807-9d2b5100-da97-11ea-82fb-c30ad15c24ab.png) <img width="1517" alt="image" src="https://user-images.githubusercontent.com/14030549/89735825-bd5b1000-da97-11ea-9944-ae54331002aa.png"> 3. I have to modify the source code and repackage. Here is my what I change: ```java @Override public void open() { if (partitions == null || partitions.size() <= 0) { throw new IllegalStateException("not assign partitions"); } List<TopicPartition> topicPartitions = Lists.newArrayList(); for (Partition partition : partitions) { topicPartitions.add(new TopicPartition(topic, partition.getPartitionId())); } kafkaConsumer.assign(topicPartitions); if (startMode == ConsumerStartMode.EARLIEST) { kafkaConsumer.seekToBeginning(topicPartitions); } else if (startMode == ConsumerStartMode.LATEST) { kafkaConsumer.seekToEnd(topicPartitions); } else { for (TopicPartition topicPartition : topicPartitions) { Long offset = partitionOffsets.getOrDefault(topicPartition.partition(), 0L); if (offset == 0L) { logger.warn("Assume {} is a new added partition, let's seek it to 0", topicPartition); } kafkaConsumer.seek(topicPartition, offset); } } } ``` ```sh mvn clean package -skipTests ... scp stream-receiver/target/kylin-stream-receiver-3.1.1-SNAPSHOT-all.jar root@cdh-master:/root/xiaoxiang.yu/older_version/apache-kylin-3.1.0-bin-cdh57-receiver/lib/streaming ``` In receiver side, remove old kylin-stream-receiver-3.1.0-all.jar . ``` [root@cdh-master apache-kylin-3.1.0-bin-cdh57-receiver]# mv lib/streaming/kylin-stream-receiver-3.1.0-all.jar kylin-stream-receiver-3.1.0-all.jar.bak ``` 4. Restart receiver and here is what we see: ``` 2020-08-09 23:59:10,436 INFO [main] server.StreamingServer:725 : Using /root/xiaoxiang.yu/older_version/apache-kylin-3.1.0-bin-cdh57-receiver/stream_index to store local segment cache. 2020-08-09 23:59:10,437 INFO [main] server.StreamingServer:301 : register receiver: cdh-master:7514 2020-08-09 23:59:10,438 WARN [main] coordinator.ZookeeperStreamMetadataStore:488 : /stream/receivers/cdh-master_7514 exists. 2020-08-09 23:59:10,458 INFO [main] server.StreamingServer:421 : add the node to the replicaSet:0, join the group leader election. 2020-08-09 23:59:10,482 INFO [main] common.KylinConfig:476 : Creating new manager instance of class org.apache.kylin.cube.CubeManager 2020-08-09 23:59:10,495 INFO [main] cube.CubeManager:136 : Initializing CubeManager with config REALTIME@jdbc,url=jdbc:mysql://localhost:3306/NightlyBuild,username=root,password=R00t@kylin,maxActive=10,maxIdle=10 2020-08-09 23:59:10,539 INFO [main] persistence.ResourceStore:90 : Using metadata url REALTIME@jdbc,url=jdbc:mysql://localhost:3306/NightlyBuild,username=root,password=R00t@kylin,maxActive=10,maxIdle=10 for resource store 2020-08-09 23:59:10,545 INFO [main] persistence.JDBCConnectionManager:92 : Connecting to Jdbc with url:jdbc:mysql://localhost:3306/NightlyBuild by user root 2020-08-09 23:59:10,559 INFO [Curator-LeaderSelector-0] server.ReplicaSetLeaderSelector:66 : become the leader of the replicaSet:0 2020-08-09 23:59:10,559 INFO [Curator-LeaderSelector-0] server.StreamingServer:575 : become leader of the replicaSet:0 2020-08-09 23:59:10,607 INFO [Curator-LeaderSelector-0] client.HttpCoordinatorClient:81 : send replicaSet lead change notification to coordinator 2020-08-09 23:59:11,194 INFO [main] persistence.JDBCConnectionManager:65 : Connected to MySQL 5.7.26 2020-08-09 23:59:11,212 INFO [main] persistence.JDBCResourceStore:115 : Table [REALTIME] already exists 2020-08-09 23:59:11,214 INFO [main] persistence.JDBCResourceStore:115 : Table [REALTIME_log] already exists 2020-08-09 23:59:11,217 DEBUG [main] cachesync.CachedCrudAssist:122 : Reloading CubeInstance from REALTIME(key='/cube')@REALTIME@jdbc,url=jdbc:mysql://localhost:3306/NightlyBuild,username=root,password=R00t@kylin,maxActive=10,maxIdle=10 2020-08-09 23:59:11,306 INFO [main] common.KylinConfig:476 : Creating new manager instance of class org.apache.kylin.cube.CubeDescManager 2020-08-09 23:59:11,307 INFO [main] cube.CubeDescManager:91 : Initializing CubeDescManager with config REALTIME@jdbc,url=jdbc:mysql://localhost:3306/NightlyBuild,username=root,password=R00t@kylin,maxActive=10,maxIdle=10 2020-08-09 23:59:11,307 DEBUG [main] cachesync.CachedCrudAssist:122 : Reloading CubeDesc from REALTIME(key='/cube_desc')@REALTIME@jdbc,url=jdbc:mysql://localhost:3306/NightlyBuild,username=root,password=R00t@kylin,maxActive=10,maxIdle=10 2020-08-09 23:59:11,358 INFO [main] common.KylinConfig:476 : Creating new manager instance of class org.apache.kylin.metadata.project.ProjectManager 2020-08-09 23:59:11,359 INFO [main] project.ProjectManager:85 : Initializing ProjectManager with metadata url REALTIME@jdbc,url=jdbc:mysql://localhost:3306/NightlyBuild,username=root,password=R00t@kylin,maxActive=10,maxIdle=10 2020-08-09 23:59:11,360 DEBUG [main] cachesync.CachedCrudAssist:122 : Reloading ProjectInstance from REALTIME(key='/project')@REALTIME@jdbc,url=jdbc:mysql://localhost:3306/NightlyBuild,username=root,password=R00t@kylin,maxActive=10,maxIdle=10 2020-08-09 23:59:11,369 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded 1 ProjectInstance(s) out of 1 resource with 0 errors 2020-08-09 23:59:11,371 INFO [main] common.KylinConfig:476 : Creating new manager instance of class org.apache.kylin.metadata.cachesync.Broadcaster 2020-08-09 23:59:11,373 DEBUG [main] cachesync.Broadcaster:102 : 1 nodes in the cluster: [localhost:7514] 2020-08-09 23:59:11,377 INFO [main] common.KylinConfig:476 : Creating new manager instance of class org.apache.kylin.metadata.model.DataModelManager 2020-08-09 23:59:11,381 INFO [main] common.KylinConfig:476 : Creating new manager instance of class org.apache.kylin.metadata.TableMetadataManager 2020-08-09 23:59:11,382 DEBUG [main] cachesync.CachedCrudAssist:122 : Reloading TableDesc from REALTIME(key='/table')@REALTIME@jdbc,url=jdbc:mysql://localhost:3306/NightlyBuild,username=root,password=R00t@kylin,maxActive=10,maxIdle=10 2020-08-09 23:59:11,400 INFO [main] measure.MeasureTypeFactory:121 : Checking custom measure types from kylin config 2020-08-09 23:59:11,401 INFO [main] measure.MeasureTypeFactory:150 : registering COUNT_DISTINCT(hllc), class org.apache.kylin.measure.hllc.HLLCMeasureType$Factory 2020-08-09 23:59:11,406 INFO [main] measure.MeasureTypeFactory:150 : registering COUNT_DISTINCT(bitmap), class org.apache.kylin.measure.bitmap.BitmapMeasureType$Factory 2020-08-09 23:59:11,410 INFO [main] measure.MeasureTypeFactory:150 : registering TOP_N(topn), class org.apache.kylin.measure.topn.TopNMeasureType$Factory 2020-08-09 23:59:11,412 INFO [main] measure.MeasureTypeFactory:150 : registering RAW(raw), class org.apache.kylin.measure.raw.RawMeasureType$Factory 2020-08-09 23:59:11,414 INFO [main] measure.MeasureTypeFactory:150 : registering EXTENDED_COLUMN(extendedcolumn), class org.apache.kylin.measure.extendedcolumn.ExtendedColumnMeasureType$Factory 2020-08-09 23:59:11,415 INFO [main] measure.MeasureTypeFactory:150 : registering PERCENTILE_APPROX(percentile), class org.apache.kylin.measure.percentile.PercentileMeasureType$Factory 2020-08-09 23:59:11,416 INFO [main] measure.MeasureTypeFactory:150 : registering COUNT_DISTINCT(dim_dc), class org.apache.kylin.measure.dim.DimCountDistinctMeasureType$Factory 2020-08-09 23:59:11,418 INFO [main] measure.MeasureTypeFactory:150 : registering STDDEV_SUM(stddev_sum), class org.apache.kylin.measure.stddev.StdDevSumMeasureType$Factory 2020-08-09 23:59:11,431 INFO [main] measure.MeasureTypeFactory:150 : registering COUNT_DISTINCT(bitmap_map), class org.apache.kylin.measure.map.bitmap.BitmapMapMeasureType$Factory 2020-08-09 23:59:11,438 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded 1 TableDesc(s) out of 1 resource with 0 errors 2020-08-09 23:59:11,439 DEBUG [main] cachesync.CachedCrudAssist:122 : Reloading TableExtDesc from REALTIME(key='/table_exd')@REALTIME@jdbc,url=jdbc:mysql://localhost:3306/NightlyBuild,username=root,password=R00t@kylin,maxActive=10,maxIdle=10 2020-08-09 23:59:11,442 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded 0 TableExtDesc(s) out of 0 resource with 0 errors 2020-08-09 23:59:11,443 DEBUG [main] cachesync.CachedCrudAssist:122 : Reloading ExternalFilterDesc from REALTIME(key='/ext_filter')@REALTIME@jdbc,url=jdbc:mysql://localhost:3306/NightlyBuild,username=root,password=R00t@kylin,maxActive=10,maxIdle=10 2020-08-09 23:59:11,445 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded 0 ExternalFilterDesc(s) out of 0 resource with 0 errors 2020-08-09 23:59:11,445 DEBUG [main] cachesync.CachedCrudAssist:122 : Reloading DataModelDesc from REALTIME(key='/model_desc')@REALTIME@jdbc,url=jdbc:mysql://localhost:3306/NightlyBuild,username=root,password=R00t@kylin,maxActive=10,maxIdle=10 2020-08-09 23:59:11,472 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded 1 DataModelDesc(s) out of 1 resource with 0 errors 2020-08-09 23:59:11,494 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded 1 CubeDesc(s) out of 1 resource with 0 errors 2020-08-09 23:59:11,495 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded 1 CubeInstance(s) out of 1 resource with 0 errors 2020-08-09 23:59:11,518 INFO [main] storage.StreamingSegmentManager:239 : restore segments from local files for cube:UserAnalysisCube 2020-08-09 23:59:11,522 WARN [main] storage.StreamingSegmentManager:253 : no checkpoint for cube:UserAnalysisCube store state, remove all local segments folders 2020-08-09 23:59:11,522 INFO [main] server.StreamingServer:351 : create and start new consumer for cube:UserAnalysisCube 2020-08-09 23:59:11,550 INFO [main] common.KylinConfig:476 : Creating new manager instance of class org.apache.kylin.cube.cuboid.CuboidManager 2020-08-09 23:59:11,566 INFO [main] kafka.KafkaSource:141 : Create StreamingConnector for Cube UserAnalysisCube, assignedPartitions [Partition [partitionId=0, partitionInfo=null], Partition [partitionId=5, partitionInfo=null], Partition [partitionId=10, partitionInfo=null], Partition [partitionId=15, partitionInfo=null], Partition [partitionId=13, partitionInfo=null], Partition [partitionId=2, partitionInfo=null], Partition [partitionId=8, partitionInfo=null], Partition [partitionId=12, partitionInfo=null], Partition [partitionId=14, partitionInfo=null], Partition [partitionId=9, partitionInfo=null], Partition [partitionId=11, partitionInfo=null], Partition [partitionId=1, partitionInfo=null], Partition [partitionId=4, partitionInfo=null], Partition [partitionId=6, partitionInfo=null], Partition [partitionId=7, partitionInfo=null], Partition [partitionId=3, partitionInfo=null]], startProtocol null 2020-08-09 23:59:11,571 INFO [main] kafka.TimedJsonStreamParser:98 : Using parser field mapping by {content_list=content_list, act_type=act_type, event_ts_2=event_ts_2, event_ts=event_ts, devide_type=user_detail.devide_type, location_city=user_detail.location.city, network_type=user_detail.network_type, video_id=video_id, event_date_2=event_date_2, str_minute=str_minute, video_type=video_type, play_times=play_times, pageview_id=pageview_id, active_minutes=active_minutes, device_brand=device_brand, str_minute_second=str_minute_second, play_duration=play_duration, event_date=event_date, page_id=page_id, str_second=str_second, uid=uid} 2020-08-09 23:59:11,584 INFO [main] kafka.TimedJsonStreamParser:120 : TimedJsonStreamParser with formatTs false tsColName event_date 2020-08-09 23:59:11,663 INFO [main] consumer.ConsumerConfig:279 : ConsumerConfig values: auto.commit.interval.ms = 5000 auto.offset.reset = latest bootstrap.servers = [cdh-master:9092, cdh-worker-1:9092, cdh-worker-2:9092] check.crcs = true client.dns.lookup = default client.id = connections.max.idle.ms = 540000 default.api.timeout.ms = 60000 enable.auto.commit = false exclude.internal.topics = true fetch.max.bytes = 52428800 fetch.max.wait.ms = 500 fetch.min.bytes = 1 group.id = heartbeat.interval.ms = 3000 interceptor.classes = [] internal.leave.group.on.close = true isolation.level = read_uncommitted key.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer max.partition.fetch.bytes = 1048576 max.poll.interval.ms = 300000 max.poll.records = 500 metadata.max.age.ms = 300000 metric.reporters = [] metrics.num.samples = 2 metrics.recording.level = INFO metrics.sample.window.ms = 30000 partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor] receive.buffer.bytes = 65536 reconnect.backoff.max.ms = 1000 reconnect.backoff.ms = 50 request.timeout.ms = 30000 retry.backoff.ms = 100 sasl.client.callback.handler.class = null sasl.jaas.config = null sasl.kerberos.kinit.cmd = /usr/bin/kinit sasl.kerberos.min.time.before.relogin = 60000 sasl.kerberos.service.name = null sasl.kerberos.ticket.renew.jitter = 0.05 sasl.kerberos.ticket.renew.window.factor = 0.8 sasl.login.callback.handler.class = null sasl.login.class = null sasl.login.refresh.buffer.seconds = 300 sasl.login.refresh.min.period.seconds = 60 sasl.login.refresh.window.factor = 0.8 sasl.login.refresh.window.jitter = 0.05 sasl.mechanism = GSSAPI security.protocol = PLAINTEXT send.buffer.bytes = 131072 session.timeout.ms = 20000 ssl.cipher.suites = null ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1] ssl.endpoint.identification.algorithm = https ssl.key.password = null ssl.keymanager.algorithm = SunX509 ssl.keystore.location = null ssl.keystore.password = null ssl.keystore.type = JKS ssl.protocol = TLS ssl.provider = null ssl.secure.random.implementation = null ssl.trustmanager.algorithm = PKIX ssl.truststore.location = null ssl.truststore.password = null ssl.truststore.type = JKS value.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer 2020-08-09 23:59:11,790 INFO [main] utils.AppInfoParser:109 : Kafka version : 2.1.1 2020-08-09 23:59:11,790 INFO [main] utils.AppInfoParser:110 : Kafka commitId : 21234bee31165527 2020-08-09 23:59:11,792 INFO [main] kafka.KafkaSource:192 : localConsumeStats from local checkpoint null, remoteConsumeStats from remote checkpoint {"0":363171,"1":363198,"2":363249,"3":363171,"4":363199,"5":363250,"6":363170,"7":363196,"8":363250,"9":363170} 2020-08-09 23:59:11,793 INFO [main] kafka.KafkaSource:239 : start kafka connector from specified position:KafkaPosition{partitionOffsetMap={0=363171, 1=363198, 2=363249, 3=363171, 4=363199, 5=363250, 6=363170, 7=363196, 8=363250, 9=363170}} 2020-08-09 23:59:11,798 WARN [main] consumer.KafkaConnector:91 : Assume useraction_xxyu-10 is a new added partition, let's seek it to 0 2020-08-09 23:59:11,799 WARN [main] consumer.KafkaConnector:91 : Assume useraction_xxyu-15 is a new added partition, let's seek it to 0 2020-08-09 23:59:11,799 WARN [main] consumer.KafkaConnector:91 : Assume useraction_xxyu-13 is a new added partition, let's seek it to 0 2020-08-09 23:59:11,799 WARN [main] consumer.KafkaConnector:91 : Assume useraction_xxyu-12 is a new added partition, let's seek it to 0 2020-08-09 23:59:11,799 WARN [main] consumer.KafkaConnector:91 : Assume useraction_xxyu-14 is a new added partition, let's seek it to 0 2020-08-09 23:59:11,799 WARN [main] consumer.KafkaConnector:91 : Assume useraction_xxyu-11 is a new added partition, let's seek it to 0 2020-08-09 23:59:11,801 INFO [main] metrics.StreamingMetrics:76 : Skip streaming metricRegistry because it is empty. 2020-08-09 23:59:11,819 INFO [main] util.log:169 : Logging initialized @3474ms to org.eclipse.jetty.util.log.Slf4jLog 2020-08-09 23:59:11,946 INFO [UserAnalysisCube_channel] clients.Metadata:285 : Cluster ID: W1YX_q9GRd-R0qwcwFJ9rg 2020-08-09 23:59:12,163 INFO [UserAnalysisCube_channel] storage.StreamingSegmentManager:189 : Create new segment:StreamingCubeSegment [cubeName=UserAnalysisCube, segmentName=20200809210000_20200809220000] 2020-08-09 23:59:12,826 INFO [UserAnalysisCube_channel] storage.StreamingSegmentManager:189 : Create new segment:StreamingCubeSegment [cubeName=UserAnalysisCube, segmentName=20200809230000_20200810000000] 2020-08-09 23:59:13,226 INFO [main] server.Server:359 : jetty-9.4.26.v20200117; built: 2020-01-17T12:35:33.676Z; git: 7b38981d25d14afb4a12ff1f2596756144edf695; jvm 1.8.0_171-b11 2020-08-09 23:59:13,260 INFO [main] ContextHandler.kylin:2223 : Initializing Spring FrameworkServlet 'org.springframework.web.servlet.DispatcherServlet-d2e6b0b' 2020-08-09 23:59:13,278 INFO [main] handler.ContextHandler:825 : Started o.e.j.s.ServletContextHandler@708eb310{/kylin,null,AVAILABLE} 2020-08-09 23:59:13,279 INFO [main] handler.ContextHandler:825 : Started o.e.j.s.h.ContextHandler@6ebc9573{/kylin/logs,null,AVAILABLE} 2020-08-09 23:59:13,294 INFO [main] server.AbstractConnector:330 : Started ServerConnector@369c9bb{HTTP/1.1,[http/1.1]}{0.0.0.0:7514} 2020-08-09 23:59:13,294 INFO [main] server.Server:399 : Started @4949ms 2020-08-09 23:59:15,572 INFO [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:106 : Start persist memory store for cube:UserAnalysisCube, segment:20200809210000_20200809220000, rowCnt:50000 2020-08-09 23:59:15,635 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:202 : cuboid-8191 transform to columnar, take 58 ms 2020-08-09 23:59:15,655 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous '00000000000000003640', current '00000000000000003638' 2020-08-09 23:59:15,719 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 15962 2020-08-09 23:59:15,720 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder 2020-08-09 23:59:15,720 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest 2020-08-09 23:59:15,725 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: 3640=>1219, 3638=>1218, 3630=>1212, 3631=>1213, 3632=>1214 2020-08-09 23:59:15,725 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : UID took : 88 ms 2020-08-09 23:59:15,738 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous 'page_19', current 'page_13' 2020-08-09 23:59:15,739 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 50 2020-08-09 23:59:15,739 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder 2020-08-09 23:59:15,740 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest 2020-08-09 23:59:15,740 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: page_19=>11, page_13=>5, page_14=>6, page_11=>3, page_12=>4 2020-08-09 23:59:15,740 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : PAGE_ID took : 13 ms 2020-08-09 23:59:15,751 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous 'stop', current 'pause' 2020-08-09 23:59:15,752 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 10 2020-08-09 23:59:15,752 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder 2020-08-09 23:59:15,752 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest 2020-08-09 23:59:15,752 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: stop=>9, pause=>5, play=>6, like=>4, start=>8 2020-08-09 23:59:15,752 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : ACT_TYPE took : 11 ms 2020-08-09 23:59:15,763 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 34 2020-08-09 23:59:15,763 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder 2020-08-09 23:59:15,763 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest 2020-08-09 23:59:15,764 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: 0=>0, 1=>1, 2=>2, 3=>3, 4=>4 2020-08-09 23:59:15,764 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : DEVIDE_TYPE took : 10 ms 2020-08-09 23:59:15,774 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous 'hongkong', current 'hangzhou' 2020-08-09 23:59:15,774 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 7 2020-08-09 23:59:15,774 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder 2020-08-09 23:59:15,775 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest 2020-08-09 23:59:15,775 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: hongkong=>3, hangzhou=>2, shenzhen=>6, berlin=>1, nanjing=>4 2020-08-09 23:59:15,775 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : LOCATION_CITY took : 10 ms 2020-08-09 23:59:15,785 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 4 2020-08-09 23:59:15,785 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder 2020-08-09 23:59:15,785 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest 2020-08-09 23:59:15,785 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: 2=>0, 3=>1, 4=>2, 5=>3 2020-08-09 23:59:15,785 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : NETWORK_TYPE took : 10 ms 2020-08-09 23:59:15,794 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous 'xiaomi', current 'vivo' 2020-08-09 23:59:15,795 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 5 2020-08-09 23:59:15,795 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder 2020-08-09 23:59:15,795 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest 2020-08-09 23:59:15,795 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: huawei=>0, xiaomi=>4, vivo=>3, meizu=>2, iPhone=>1 2020-08-09 23:59:15,795 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : DEVICE_BRAND took : 9 ms 2020-08-09 23:59:15,806 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous '21_14_59', current '21_14_50' 2020-08-09 23:59:15,810 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 2032 2020-08-09 23:59:15,810 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder 2020-08-09 23:59:15,810 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest 2020-08-09 23:59:15,810 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: 21_14_59=>355, 21_14_50=>346, 21_14_55=>351, 21_14_56=>352, 21_14_57=>353 2020-08-09 23:59:15,810 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : STR_MINUTE_SECOND took : 15 ms 2020-08-09 23:59:16,088 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:UID, took: 276ms 2020-08-09 23:59:16,171 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:PAGE_ID, took: 83ms 2020-08-09 23:59:16,254 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:ACT_TYPE, took: 82ms 2020-08-09 23:59:16,316 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:DEVIDE_TYPE, took: 61ms 2020-08-09 23:59:16,391 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:LOCATION_CITY, took: 75ms 2020-08-09 23:59:16,461 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:NETWORK_TYPE, took: 69ms 2020-08-09 23:59:16,532 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:DEVICE_BRAND, took: 70ms 2020-08-09 23:59:16,608 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:STR_MINUTE_SECOND, took: 75ms 2020-08-09 23:59:16,646 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:MONTH_START, took: 38ms 2020-08-09 23:59:16,673 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:WEEK_START, took: 26ms 2020-08-09 23:59:16,703 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:DAY_START, took: 29ms 2020-08-09 23:59:16,736 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:HOUR_START, took: 33ms 2020-08-09 23:59:16,779 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:MINUTE_START, took: 41ms 2020-08-09 23:59:16,808 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:_COUNT_, took: 29ms 2020-08-09 23:59:16,871 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:SUM_PLAY_TIMES, took: 63ms 2020-08-09 23:59:16,907 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:SUM_ACTIVE_MINUTES, took: 34ms 2020-08-09 23:59:16,932 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:SUM_PLAY_DURATION, took: 25ms 2020-08-09 23:59:16,993 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:MAX_PLAY_TIMES, took: 60ms 2020-08-09 23:59:17,027 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:MAX_ACTIVE_MINUTES, took: 33ms 2020-08-09 23:59:17,049 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:MAX_PLAY_DURATION, took: 22ms 2020-08-09 23:59:17,081 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:PV_STAT, took: 31ms 2020-08-09 23:59:17,089 INFO [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:110 : Finish persist memory store for cube:UserAnalysisCube segment:20200809210000_20200809220000, take: 1517ms 2020-08-09 23:59:18,954 INFO [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:106 : Start persist memory store for cube:UserAnalysisCube, segment:20200809210000_20200809220000, rowCnt:50000 2020-08-09 23:59:18,980 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:202 : cuboid-8191 transform to columnar, take 26 ms 2020-08-09 23:59:18,989 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous '00000000000000003640', current '00000000000000003638' 2020-08-09 23:59:19,008 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 16046 2020-08-09 23:59:19,008 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder 2020-08-09 23:59:19,008 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest 2020-08-09 23:59:19,008 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: 3640=>1240, 3638=>1238, 3639=>1239, 3630=>1231, 3631=>1232 2020-08-09 23:59:19,008 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : UID took : 28 ms 2020-08-09 23:59:19,017 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous 'page_19', current 'page_13' 2020-08-09 23:59:19,017 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 50 2020-08-09 23:59:19,017 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder 2020-08-09 23:59:19,017 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest 2020-08-09 23:59:19,018 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: page_19=>11, page_13=>5, page_14=>6, page_11=>3, page_12=>4 2020-08-09 23:59:19,018 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : PAGE_ID took : 8 ms 2020-08-09 23:59:19,025 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous 'stop', current 'pause' 2020-08-09 23:59:19,026 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 10 2020-08-09 23:59:19,026 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder 2020-08-09 23:59:19,026 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest 2020-08-09 23:59:19,026 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: stop=>9, pause=>5, play=>6, like=>4, start=>8 2020-08-09 23:59:19,026 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : ACT_TYPE took : 8 ms 2020-08-09 23:59:19,035 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 44 2020-08-09 23:59:19,035 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder 2020-08-09 23:59:19,035 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest 2020-08-09 23:59:19,035 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: 16=>0, 17=>1, 18=>2, 19=>3, 20=>4 2020-08-09 23:59:19,035 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : DEVIDE_TYPE took : 8 ms 2020-08-09 23:59:19,043 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous 'hongkong', current 'hangzhou' 2020-08-09 23:59:19,043 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 7 2020-08-09 23:59:19,043 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder 2020-08-09 23:59:19,043 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest 2020-08-09 23:59:19,043 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: hongkong=>3, hangzhou=>2, shenzhen=>6, berlin=>1, nanjing=>4 2020-08-09 23:59:19,044 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : LOCATION_CITY took : 8 ms 2020-08-09 23:59:19,052 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 4 2020-08-09 23:59:19,052 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder 2020-08-09 23:59:19,052 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest 2020-08-09 23:59:19,052 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: 2=>0, 3=>1, 4=>2, 5=>3 2020-08-09 23:59:19,052 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : NETWORK_TYPE took : 8 ms 2020-08-09 23:59:19,059 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous 'xiaomi', current 'vivo' 2020-08-09 23:59:19,060 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 5 2020-08-09 23:59:19,060 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder 2020-08-09 23:59:19,060 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest 2020-08-09 23:59:19,060 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: huawei=>0, xiaomi=>4, vivo=>3, meizu=>2, iPhone=>1 2020-08-09 23:59:19,060 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : DEVICE_BRAND took : 7 ms 2020-08-09 23:59:19,069 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous '21_55_9', current '21_55_0' 2020-08-09 23:59:19,077 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 2536 2020-08-09 23:59:19,077 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder 2020-08-09 23:59:19,077 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest 2020-08-09 23:59:19,077 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: 21_55_7=>2341, 21_55_8=>2342, 21_55_9=>2343, 21_55_0=>2284, 21_55_1=>2285 2020-08-09 23:59:19,077 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : STR_MINUTE_SECOND took : 17 ms 2020-08-09 23:59:19,185 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:UID, took: 107ms 2020-08-09 23:59:19,235 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:PAGE_ID, took: 49ms 2020-08-09 23:59:19,284 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:ACT_TYPE, took: 48ms 2020-08-09 23:59:19,335 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:DEVIDE_TYPE, took: 50ms 2020-08-09 23:59:19,396 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:LOCATION_CITY, took: 60ms 2020-08-09 23:59:19,455 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:NETWORK_TYPE, took: 59ms 2020-08-09 23:59:19,522 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:DEVICE_BRAND, took: 66ms 2020-08-09 23:59:19,592 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:STR_MINUTE_SECOND, took: 69ms 2020-08-09 23:59:19,610 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:MONTH_START, took: 17ms 2020-08-09 23:59:19,626 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:WEEK_START, took: 16ms 2020-08-09 23:59:19,642 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:DAY_START, took: 15ms 2020-08-09 23:59:19,662 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:HOUR_START, took: 19ms 2020-08-09 23:59:19,691 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:MINUTE_START, took: 28ms 2020-08-09 23:59:19,708 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:_COUNT_, took: 16ms 2020-08-09 23:59:19,766 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:SUM_PLAY_TIMES, took: 57ms 2020-08-09 23:59:19,783 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:SUM_ACTIVE_MINUTES, took: 17ms 2020-08-09 23:59:19,801 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:SUM_PLAY_DURATION, took: 17ms 2020-08-09 23:59:19,856 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:MAX_PLAY_TIMES, took: 54ms 2020-08-09 23:59:19,873 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:MAX_ACTIVE_MINUTES, took: 17ms 2020-08-09 23:59:19,890 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:MAX_PLAY_DURATION, took: 16ms 2020-08-09 23:59:19,908 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:PV_STAT, took: 18ms 2020-08-09 23:59:19,913 INFO [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:110 : Finish persist memory store for cube:UserAnalysisCube segment:20200809210000_20200809220000, take: 959ms 2020-08-09 23:59:20,175 INFO [UserAnalysisCube_channel] storage.StreamingSegmentManager:189 : Create new segment:StreamingCubeSegment [cubeName=UserAnalysisCube, segmentName=20200809220000_20200809230000] 2020-08-09 23:59:22,201 INFO [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:106 : Start persist memory store for cube:UserAnalysisCube, segment:20200809220000_20200809230000, rowCnt:50000 2020-08-09 23:59:22,235 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:202 : cuboid-8191 transform to columnar, take 33 ms 2020-08-09 23:59:22,245 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous '00000000000000003640', current '00000000000000003638' 2020-08-09 23:59:22,258 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 15977 2020-08-09 23:59:22,258 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder 2020-08-09 23:59:22,258 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest 2020-08-09 23:59:22,258 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: 3640=>1166, 3638=>1165, 3630=>1158, 3631=>1159, 3632=>1160 2020-08-09 23:59:22,258 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : UID took : 23 ms 2020-08-09 23:59:22,267 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous 'page_19', current 'page_13' 2020-08-09 23:59:22,268 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 50 2020-08-09 23:59:22,268 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder 2020-08-09 23:59:22,268 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest 2020-08-09 23:59:22,268 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: page_19=>11, page_13=>5, page_14=>6, page_11=>3, page_12=>4 2020-08-09 23:59:22,268 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : PAGE_ID took : 9 ms 2020-08-09 23:59:22,277 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous 'stop', current 'pause' 2020-08-09 23:59:22,277 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 10 2020-08-09 23:59:22,277 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder 2020-08-09 23:59:22,277 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest 2020-08-09 23:59:22,277 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: stop=>9, pause=>5, play=>6, like=>4, start=>8 2020-08-09 23:59:22,277 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : ACT_TYPE took : 9 ms 2020-08-09 23:59:22,285 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 34 2020-08-09 23:59:22,285 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder 2020-08-09 23:59:22,286 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest 2020-08-09 23:59:22,286 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: 0=>0, 1=>1, 2=>2, 3=>3, 4=>4 2020-08-09 23:59:22,286 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : DEVIDE_TYPE took : 8 ms 2020-08-09 23:59:22,294 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous 'hongkong', current 'hangzhou' 2020-08-09 23:59:22,294 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 7 2020-08-09 23:59:22,294 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder 2020-08-09 23:59:22,295 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest 2020-08-09 23:59:22,295 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: hongkong=>3, hangzhou=>2, shenzhen=>6, berlin=>1, nanjing=>4 2020-08-09 23:59:22,295 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : LOCATION_CITY took : 8 ms 2020-08-09 23:59:22,303 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 4 2020-08-09 23:59:22,303 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder 2020-08-09 23:59:22,303 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest 2020-08-09 23:59:22,303 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: 2=>0, 3=>1, 4=>2, 5=>3 2020-08-09 23:59:22,303 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : NETWORK_TYPE took : 8 ms 2020-08-09 23:59:22,311 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous 'xiaomi', current 'vivo' 2020-08-09 23:59:22,311 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 5 2020-08-09 23:59:22,311 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder 2020-08-09 23:59:22,311 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest 2020-08-09 23:59:22,311 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: huawei=>0, xiaomi=>4, vivo=>3, meizu=>2, iPhone=>1 2020-08-09 23:59:22,311 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : DEVICE_BRAND took : 8 ms 2020-08-09 23:59:22,321 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous '22_14_59', current '22_14_41' 2020-08-09 23:59:22,323 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 1985 2020-08-09 23:59:22,324 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder 2020-08-09 23:59:22,324 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest 2020-08-09 23:59:22,324 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: 22_14_58=>354, 22_14_59=>355, 22_14_41=>336, 22_14_42=>337, 22_14_40=>335 2020-08-09 23:59:22,324 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : STR_MINUTE_SECOND took : 12 ms 2020-08-09 23:59:22,426 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:UID, took: 102ms 2020-08-09 23:59:22,477 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:PAGE_ID, took: 50ms 2020-08-09 23:59:22,526 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:ACT_TYPE, took: 49ms 2020-08-09 23:59:22,574 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:DEVIDE_TYPE, took: 47ms 2020-08-09 23:59:22,706 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:LOCATION_CITY, took: 61ms 2020-08-09 23:59:22,768 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:NETWORK_TYPE, took: 61ms 2020-08-09 23:59:22,834 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:DEVICE_BRAND, took: 66ms 2020-08-09 23:59:22,903 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:STR_MINUTE_SECOND, took: 68ms 2020-08-09 23:59:22,920 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:MONTH_START, took: 16ms 2020-08-09 23:59:22,937 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:WEEK_START, took: 16ms 2020-08-09 23:59:22,953 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:DAY_START, took: 16ms 2020-08-09 23:59:22,971 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:HOUR_START, took: 17ms 2020-08-09 23:59:22,995 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:MINUTE_START, took: 24ms 2020-08-09 23:59:23,008 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:_COUNT_, took: 12ms 2020-08-09 23:59:23,064 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:SUM_PLAY_TIMES, took: 55ms 2020-08-09 23:59:23,082 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:SUM_ACTIVE_MINUTES, took: 17ms 2020-08-09 23:59:23,100 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:SUM_PLAY_DURATION, took: 17ms 2020-08-09 23:59:23,157 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:MAX_PLAY_TIMES, took: 56ms 2020-08-09 23:59:23,175 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:MAX_ACTIVE_MINUTES, took: 17ms 2020-08-09 23:59:23,193 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:MAX_PLAY_DURATION, took: 18ms 2020-08-09 23:59:23,213 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:PV_STAT, took: 19ms 2020-08-09 23:59:23,214 INFO [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:110 : Finish persist memory store for cube:UserAnalysisCube segment:20200809220000_20200809230000, take: 1012ms 2020-08-09 23:59:25,467 INFO [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:106 : Start persist memory store for cube:UserAnalysisCube, segment:20200809230000_20200810000000, rowCnt:50000 2020-08-09 23:59:25,495 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:202 : cuboid-8191 transform to columnar, take 26 ms 2020-08-09 23:59:25,504 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous '00000000000000003639', current '00000000000000003630' 2020-08-09 23:59:25,519 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 15936 2020-08-09 23:59:25,519 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder 2020-08-09 23:59:25,519 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest 2020-08-09 23:59:25,519 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: 3638=>1084, 3639=>1085, 3630=>1077, 3631=>1078, 3632=>1079 2020-08-09 23:59:25,519 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : UID took : 24 ms 2020-08-09 23:59:25,532 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous 'page_19', current 'page_13' 2020-08-09 23:59:25,533 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 50 2020-08-09 23:59:25,533 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder 2020-08-09 23:59:25,533 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest 2020-08-09 23:59:25,533 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: page_19=>11, page_13=>5, page_14=>6, page_11=>3, page_12=>4 2020-08-09 23:59:25,533 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : PAGE_ID took : 13 ms 2020-08-09 23:59:25,542 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous 'stop', current 'pause' 2020-08-09 23:59:25,543 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 10 2020-08-09 23:59:25,543 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder 2020-08-09 23:59:25,543 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest 2020-08-09 23:59:25,543 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: stop=>9, pause=>5, play=>6, like=>4, start=>8 2020-08-09 23:59:25,543 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : ACT_TYPE took : 9 ms 2020-08-09 23:59:25,553 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 41 2020-08-09 23:59:25,553 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder 2020-08-09 23:59:25,553 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest 2020-08-09 23:59:25,553 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: 19=>0, 20=>1, 21=>2, 22=>3, 23=>4 2020-08-09 23:59:25,553 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : DEVIDE_TYPE took : 10 ms 2020-08-09 23:59:25,562 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous 'hongkong', current 'hangzhou' 2020-08-09 23:59:25,563 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 7 2020-08-09 23:59:25,563 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder 2020-08-09 23:59:25,563 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest 2020-08-09 23:59:25,563 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: hongkong=>3, hangzhou=>2, shenzhen=>6, berlin=>1, nanjing=>4 2020-08-09 23:59:25,563 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : LOCATION_CITY took : 9 ms 2020-08-09 23:59:25,572 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 4 2020-08-09 23:59:25,573 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder 2020-08-09 23:59:25,573 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest 2020-08-09 23:59:25,573 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: 2=>0, 3=>1, 4=>2, 5=>3 2020-08-09 23:59:25,573 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : NETWORK_TYPE took : 9 ms 2020-08-09 23:59:25,582 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous 'xiaomi', current 'vivo' 2020-08-09 23:59:25,583 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 5 2020-08-09 23:59:25,583 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder 2020-08-09 23:59:25,583 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest 2020-08-09 23:59:25,583 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: huawei=>0, xiaomi=>4, vivo=>3, meizu=>2, iPhone=>1 2020-08-09 23:59:25,583 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : DEVICE_BRAND took : 10 ms 2020-08-09 23:59:25,594 INFO [UserAnalysisCube_channel] dict.TrieDictionaryForestBuilder:90 : values not in ascending order, previous '23_53_19', current '23_53_10' 2020-08-09 23:59:25,596 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:91 : Dictionary cardinality: 2412 2020-08-09 23:59:25,596 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:92 : Dictionary builder class: org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder 2020-08-09 23:59:25,596 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:93 : Dictionary class: org.apache.kylin.dict.TrieDictionaryForest 2020-08-09 23:59:25,596 DEBUG [UserAnalysisCube_channel] dict.DictionaryGenerator:102 : Dictionary value samples: 23_53_16=>2035, 23_53_17=>2036, 23_53_18=>2037, 23_53_19=>2038, 23_53_10=>2029 2020-08-09 23:59:25,596 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:403 : BuildDictionary for column : STR_MINUTE_SECOND took : 12 ms 2020-08-09 23:59:25,702 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:UID, took: 105ms 2020-08-09 23:59:25,757 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:PAGE_ID, took: 54ms 2020-08-09 23:59:25,812 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:ACT_TYPE, took: 54ms 2020-08-09 23:59:25,866 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:DEVIDE_TYPE, took: 54ms 2020-08-09 23:59:25,932 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:LOCATION_CITY, took: 65ms 2020-08-09 23:59:25,996 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:NETWORK_TYPE, took: 63ms 2020-08-09 23:59:26,065 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:DEVICE_BRAND, took: 68ms 2020-08-09 23:59:26,138 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:STR_MINUTE_SECOND, took: 73ms 2020-08-09 23:59:26,157 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:MONTH_START, took: 18ms 2020-08-09 23:59:26,175 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:WEEK_START, took: 18ms 2020-08-09 23:59:26,193 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:DAY_START, took: 17ms 2020-08-09 23:59:26,210 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:HOUR_START, took: 17ms 2020-08-09 23:59:26,235 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:326 : cuboid-8191 saved dimension:MINUTE_START, took: 24ms 2020-08-09 23:59:26,256 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:_COUNT_, took: 20ms 2020-08-09 23:59:26,316 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:SUM_PLAY_TIMES, took: 60ms 2020-08-09 23:59:26,336 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:SUM_ACTIVE_MINUTES, took: 19ms 2020-08-09 23:59:26,355 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:SUM_PLAY_DURATION, took: 18ms 2020-08-09 23:59:26,412 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:MAX_PLAY_TIMES, took: 57ms 2020-08-09 23:59:26,431 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:MAX_ACTIVE_MINUTES, took: 18ms 2020-08-09 23:59:26,450 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:MAX_PLAY_DURATION, took: 18ms 2020-08-09 23:59:26,470 DEBUG [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:379 : cuboid-8191 saved measure:PV_STAT, took: 20ms 2020-08-09 23:59:26,471 INFO [UserAnalysisCube_channel] columnar.ColumnarMemoryStorePersister:110 : Finish persist memory store for cube:UserAnalysisCube segment:20200809230000_20200810000000, take: 1003ms ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Fail to consume kafka when partition number get larger > ------------------------------------------------------ > > Key: KYLIN-4683 > URL: https://issues.apache.org/jira/browse/KYLIN-4683 > Project: Kylin > Issue Type: Bug > Affects Versions: v3.0.2 > Reporter: tianhui > Priority: Major > Attachments: image-2020-08-05-17-20-37-270.png > > > I run a testing streaming cube with kafka. At first, the topic has 3 > partitions, and the cube running smoothly. But after I alter kafka topic to 7 > partitions, all receivers stop consuming. !image-2020-08-05-17-20-37-270.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)