hi,
when I debug the following java codes , the breakpoint can be hit "final
BroadcastEvent broadcastEvent = broadcastEvents.takeFirst();" but, it
cannot hit the bp of the line "final String[] nodes =
KylinConfig.getInstanceFromEnv().getRestServers();", do anyone how to
make it hit?
private Broadcaster() {
Executors.newSingleThreadExecutor().execute(new Runnable() {
@Override
public void run() {
final String[] nodes =
KylinConfig.getInstanceFromEnv().getRestServers();
if (nodes == null || nodes.length < 1) {//TODO if the node
count is greater than 1, it means it is a cluster
logger.warn("There is no available rest server; check
the 'kylin.rest.servers' config");
return;
}
final List<RestClient> restClients = Lists.newArrayList();
for (String node : nodes) {
restClients.add(new RestClient(node));
}
final ExecutorService wipingCachePool =
Executors.newFixedThreadPool(restClients.size());
while (true) {
try {
final BroadcastEvent broadcastEvent =
broadcastEvents.takeFirst();
2015-05-07 21:07 GMT+08:00 dong wang <[email protected]>:
> @qianhao,
> 1, I will debug the code if available
> 2, just now, as observed, after "REFRESH" a cube, even though I click
> "Reload Metadata" under "Admin" tab on the WEB of the "QUERY" kylin
> instances, the cube status cannot be syncd with the "ALL" kylin
> instance(but if restarting the "QUERY" kylin instances, "QUERY" instances
> can sync the FRESH cube result info), another thing is that it works OK for
> "BUILD" a cube if just reloading the metadata without restarting the
> instance, thus,I guess, the above 2 correct URL put methods will not work
> for "REFERSH" as well
>
> 2015-05-07 20:17 GMT+08:00 Zhou, Qianhao <[email protected]>:
>
>> Thanks for the research, the root cause should be the wrong URL redirect
>>
>> 2015年5月7日 18:59于 dong wang <[email protected]>写道：
>> @qianzhao, as tested, another discovery is that if I use the URL like:
>>
>> http://test.abc.com:8082/kylin/api/cache/cube/lbs_map_active_user_fact/update
>> ,
>> the cube status of the "QUERY" kylin instance also works SUCCESSFULLY
>>
>> thus, the problem should be something wrong with the communication between
>> "ALL" mode instance and "QUERY" mode instances
>>
>> 2015-05-07 18:12 GMT+08:00 dong wang <[email protected]>:
>>
>> > @qianhao, @shaofeng,
>> >
>> > today, I tried to mockup a PUT request as you mentioned above with the
>> > following different URLs:
>> > 1, http://test.abc.com:8082/cube/lbs_map_new_user_fact/update
>> >
>> > nothing happened, even more, no log information received in the kylin
>> > instance log file deployed with port 8082
>> >
>> > 2, http://test.abc.com:8082/kylin/api/cache/all/all/update
>> >
>> > with this API, It works OK, the cube status is flushed SUCCESSFULLY,
>> log
>> > information received as shown below:
>> >
>> > [http-bio-8082-exec-7]:[2015-05-07
>> >
>> 17:54:57,469][INFO][org.apache.kylin.rest.controller.CacheController.wipeCache(CacheController.java:
>> > 64)] - wipe cache type: ALL event:UPDATE name:all
>> > [http-bio-8082-exec-7]:[2015-05-07
>> >
>> 17:54:57,488][DEBUG][org.apache.kylin.metadata.MetadataManager.reloadAllSourceTable(MetadataManager.
>> > java:244)] - Reloading SourceTable from folder
>> > kylin_metadata(key='/table')@kylin_metadata@hbase
>> > [http-bio-8082-exec-7]:[2015-05-07
>> >
>> 17:54:57,533][DEBUG][org.apache.kylin.metadata.MetadataManager.reloadAllSourceTable(MetadataManager.
>> > java:253)] - Loaded 8 SourceTable(s)
>> > [http-bio-8082-exec-7]:[2015-05-07
>> >
>> 17:54:57,534][DEBUG][org.apache.kylin.metadata.MetadataManager.reloadAllSourceTableExd(MetadataManag
>> > er.java:201)] - Reloading SourceTable exd info from folder
>> > kylin_metadata(key='/table_exd')@kylin_metadata@hbase
>> > [http-bio-8082-exec-7]:[2015-05-07
>> >
>> 17:54:57,562][DEBUG][org.apache.kylin.metadata.MetadataManager.reloadAllSourceTableExd(MetadataManag
>> > er.java:210)] - Loaded 8 SourceTable EXD(s)
>> > [http-bio-8082-exec-7]:[2015-05-07
>> >
>> 17:54:57,562][DEBUG][org.apache.kylin.metadata.MetadataManager.reloadAllDataModel(MetadataManager.ja
>> > va:291)] - Reloading DataModel from folder
>> > kylin_metadata(key='/model_desc')@kylin_metadata@hbase
>> > [http-bio-8082-exec-7]:[2015-05-07
>> >
>> 17:54:57,599][DEBUG][org.apache.kylin.metadata.MetadataManager.reloadAllDataModel(MetadataManager.ja
>> > va:305)] - Loaded 8 DataModel(s)
>> > [http-bio-8082-exec-7]:[2015-05-07
>> >
>> 17:54:57,604][INFO][org.apache.kylin.rest.service.BasicService.resetOLAPDataSources(BasicService.jav
>> > a:90)] - resetOLAPDataSources is called.
>> >
>> > 3, currently, since we deploy 3 kylin instance on the same machine with
>> > different ports, and I set export KYLIN_HOME=/path_to_kylin at the very
>> > beginning of the file bin/kylin.sh since we cannot set a global env
>> > KYLIN_HOME
>> >
>> > 2015-05-07 17:50 GMT+08:00 Tao Wong <[email protected]>:
>> >
>> >> Cube status on query mode instance still can not change automatically.
>> >>
>> >> the all mode node seem to broadcast the query mode nodes,however ,the
>> >> query
>> >> node run with error as above and below:
>> >>
>> >> Caused by: java.lang.ClassNotFoundException:
>> >> org.apache.kylin.query.schema.OLAPSchemaFactory
>> >> 733 at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>> >> 734 at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>> >> 735 at java.security.AccessController.doPrivileged(Native Method)
>> >> 736 at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>> >> 737 at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>> >> 738 at
>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>> >> 739 at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>> >> 740 at java.lang.Class.forName0(Native Method)
>> >> 741 at java.lang.Class.forName(Class.java:191)
>> >> 742 at
>> >> org.apache.calcite.model.ModelHandler.visit(ModelHandler.java:209)
>> >>
>> >> 2015-05-07 14:17 GMT+08:00 Tao Wong <[email protected]>:
>> >>
>> >> > After I build a new segment. the kylin.log on the query mode node.
>> >> >
>> >> > kylin.log:
>> >> >
>> >> > [http-bio-8082-exec-1]:[2015-05-07
>> >> >
>> >>
>> 13:53:13,681][DEBUG][org.apache.kylin.rest.service.AdminService.getConfigAsString(AdminService
>> >> > .java:91)] - Get Kylin Runtime Config
>> >> > 3261 [http-bio-8082-exec-1]:[2015-05-07
>> >> >
>> >>
>> 13:56:08,971][INFO][org.apache.kylin.metadata.project.ProjectL2Cache.loadCache(ProjectL2Cache.
>> >> > java:152)] - Loading L2 project cache for user
>> >> > 3262 [http-bio-8082-exec-1]:[2015-05-07
>> >> >
>> >>
>> 13:56:08,974][INFO][org.apache.kylin.rest.controller.TableController.getHiveTables(TableContro
>> >> > ller.java:74)] - Return all table metadata in 3 seconds
>> >> > 3263 [http-bio-8082-exec-4]:[2015-05-07
>> >>
>> 13:59:03,657][ERROR][org.apache.kylin.rest.controller.BasicController.handleError(BasicControl
>> >> > ler.java:52)] -
>> >> > 3264 java.lang.RuntimeException: Error instantiating
>> >> > JsonCustomSchema(name=USER)
>> >> > 3265 at
>> >> > org.apache.calcite.model.ModelHandler.visit(ModelHandler.java:217)
>> >> > 3266 at
>> >> >
>> >>
>> org.apache.calcite.model.JsonCustomSchema.accept(JsonCustomSchema.java:36)
>> >> > 3267 at
>> >> > org.apache.calcite.model.ModelHandler.visit(ModelHandler.java:144)
>> >> > 3268 at
>> >> > org.apache.calcite.model.ModelHandler.<init>(ModelHandler.java:87)
>> >> > 3269 at
>> >> > org.apache.calcite.jdbc.Driver$1.onConnectionInit(Driver.java:97)
>> >> > 3270 at
>> >> >
>> >>
>> org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:145)
>> >> > 3271 at
>> java.sql.DriverManager.getConnection(DriverManager.java:571)
>> >> > 3272 at
>> java.sql.DriverManager.getConnection(DriverManager.java:187)
>> >> > 3273 at
>> >> >
>> >>
>> org.springframework.jdbc.datasource.DriverManagerDataSource.getConnectionFromDriverManager(DriverManagerDataSource.java:17
>> >> > 3)
>> >> > 3274 at
>> >> >
>> >>
>> org.springframework.jdbc.datasource.DriverManagerDataSource.getConnectionFromDriver(DriverManagerDataSource.java:164)
>> >> > 3275 at
>> >> >
>> >>
>> org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnectionFromDriver(AbstractDriverBasedDataSource.ja
>> >> > va:149)
>> >> > 3276 at
>> >> >
>> >>
>> org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnection(AbstractDriverBasedDataSource.java:119)
>> >> >
>> >> > 2015-05-07 9:15 GMT+08:00 Shi, Shaofeng <[email protected]>:
>> >> >
>> >> >> Dong, you can download a build directly from
>> >> >> https://kylin.incubator.apache.org/download/;
>> >> >>
>> >> >> On 5/7/15, 7:26 AM, "周千昊" <[email protected]> wrote:
>> >> >>
>> >> >> >Hi, dong
>> >> >> > #1 you don't have to add "http://"; prefix for the
>> >> >> kylin.rest.servers
>> >> >> >config,
>> >> >> > according to the log, the Broadcaster event has been
>> >> >> >successfully
>> >> >> >created, however there maybe some network issues when "ALL" server
>> >> trying
>> >> >> >to reach "QUERY" server, can you try to mockup a PUT request like
>> >> >> >http://test.abc.com:8081/cube/lbs_map_new_user_fact/update to see
>> if
>> >> the
>> >> >> >network is OK?
>> >> >> > #2 yes you can git clone the "0.7.1-staging" branch to build
>> the
>> >> >> >binary
>> >> >> >
>> >> >> >dong wang <[email protected]>于2015年5月7日周四 上午12:10写道：
>> >> >> >
>> >> >> >> 1, please help to review the above 2 response first
>> >> >> >> 2, in kylin.properties,currently we set kylin.rest.servers=
>> >> >> >> test.abc.com:8080
>> >> >> >> ,test.abc.com:8081,test.abc.com:8082 and test.abc.com:8080 are
>> >> the
>> >> >> >>"ALL"
>> >> >> >> instance, the other 2 are the "QUERY" kylin instances
>> >> >> >> should we set with the 'http://' prefix like:
>> >> kylin.rest.servers=
>> >> >> >>
>> >> >> >>http://test.abc.com:8080,http://test.abc.com:8081,
>> >> >> http://test.abc.com:808
>> >> >> >>2
>> >> >> >>
>> >> >> >> 2015-05-06 23:40 GMT+08:00 dong wang <[email protected]>:
>> >> >> >>
>> >> >> >> > @shaofeng, another question is that can I git clone the branch
>> >> >> >> > "0.71-staging"(
>> >> https://github.com/KylinOLAP/Kylin/tree/0.7.1-staging
>> >> >> )
>> >> >> >>to
>> >> >> >> > get the latest code to build the binary?
>> >> >> >> >
>> >> >> >> > 2015-05-06 23:38 GMT+08:00 dong wang <[email protected]>:
>> >> >> >> >
>> >> >> >> >> I double checked one piece of the above log of the "ALL" mode
>> >> kylin
>> >> >> >> >> instance:
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >>
>> >> >>
>> >> >>
>> >>
>> >>17:36:21,910][INFO][org.apache.kylin.common.restclient.Broadcaster$1.run(
>> >> >> >>Broadcaster.java:71)]
>> >> >> >> >> - new broadcast event:BroadcastEvent{type=cube, name=lbs_ma
>> >> >> >> >> p_new_user_fact, action=update}
>> >> >> >> >> 2553 [pool-7-thread-10]:[2015-05-06
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >>
>> >> >>
>> >> >>
>> >>
>> >>17:36:21,911][DEBUG][org.apache.kylin.common.persistence.ResourceStore.pu
>> >> >> >>tResource(ResourceStore.java:171)]
>> >> >> >> >> - Saving resource /execute_output/f03d3c3c-68
>> >> >> >> >> f4-4593-b01d-2778fcaf098b-15 (Store kylin_metadata@hbase)
>> >> >> >> >> 2554 [pool-7-thread-10]:[2015-05-06
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >>
>> >> >>
>> >> >>
>> >>
>> >>17:36:21,921][DEBUG][org.apache.kylin.common.persistence.ResourceStore.pu
>> >> >> >>tResource(ResourceStore.java:171)]
>> >> >> >> >> - Saving resource /execute_output/f03d3c3c-68
>> >> >> >> >> f4-4593-b01d-2778fcaf098b-15 (Store kylin_metadata@hbase)
>> >> >> >> >> 2555 [http-bio-8081-exec-5]:[2015-05-06
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >>
>> >> >>
>> >> >>
>> >>
>> >>17:36:21,922][INFO][org.apache.kylin.rest.controller.CacheController.wipe
>> >> >> >>Cache(CacheController.java:63)]
>> >> >> >> >> - wipe cache type: CUBE event:UPDATE name:lb
>> >> s_map_new_user_fact
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> from the first line and the last line, it seems that it tries
>> to
>> >> >> call
>> >> >> >> >> sync something related to 'rest', however, when I check the
>> >> >> >>kylin.log
>> >> >> >> for
>> >> >> >> >> each of the "QUERY" mode kylin instances, no related log
>> reponse
>> >> to
>> >> >> >>the
>> >> >> >> >> rest sync
>> >> >> >> >>
>> >> >> >> >
>> >> >> >> >
>> >> >> >>
>> >> >>
>> >> >>
>> >> >
>> >>
>> >
>> >
>>
>
>
