Hi Dlmuthu, Could you give us a bit more context? What kind of operations are you doing and what are the characteristics (config settings) for these workflows?
On a cursory look, it seems that the workflow "Workflow_of_process_PROCESS_4660666a-6acf-477e-bf97-0671b666dd19-PRE-6a568552-f517-49a1-94c8-c1d3814c3244"'s WorkflowConfig is missing even though its IdealState exists. A few possible scenarios are as follows: 1. Some purge operation/delete operation was performed on this particular workflow; however, the WorkflowConfig was removed successfully, but the IdealState removal failed. This causes WorkflowRebalancer to print out the log. This is not very serious and you'll just see a lot of WARN logs. The fix is to force delete the IdealState that failed to be deleted. 2. Some fundamental assumptions about Helix was broken - (for example, CLUSTER/CONFIG/RESOURCE folder was removed and re-created) that caused CallbackHandler to be removed. This will break the Helix Controller and no workflows will be scheduled. In this case, you have to ensure the cluster is set up properly and re-start the Controller. Hope this helps, Hunter On Tue, Nov 20, 2018 at 7:12 AM DImuthu Upeksha <[email protected]> wrote: > Hi Folks, > > I'm seeing a set of log lines that are continuously printing in controller > and some errors in between. Do you have an explanation for this? > > 2018-11-20 07:53:44,127 [GenericHelixController-event_process] WARN > o.a.h.task.WorkflowRebalancer - Workflow configuration is NULL for > > Workflow_of_process_PROCESS_4660666a-6acf-477e-bf97-0671b666dd19-PRE-6a568552-f517-49a1-94c8-c1d3814c3244 > 2018-11-20 07:53:44,662 [GenericHelixController-event_process] WARN > o.a.h.task.WorkflowRebalancer - Workflow configuration is NULL for > > Workflow_of_process_PROCESS_4660666a-6acf-477e-bf97-0671b666dd19-PRE-6a568552-f517-49a1-94c8-c1d3814c3244 > 2018-11-20 07:53:45,215 [GenericHelixController-event_process] WARN > o.a.h.task.WorkflowRebalancer - Workflow configuration is NULL for > > Workflow_of_process_PROCESS_4660666a-6acf-477e-bf97-0671b666dd19-PRE-6a568552-f517-49a1-94c8-c1d3814c3244 > 2018-11-20 07:53:46,709 [GenericHelixController-event_process] WARN > o.a.h.task.WorkflowRebalancer - Workflow configuration is NULL for > > Workflow_of_process_PROCESS_4660666a-6acf-477e-bf97-0671b666dd19-PRE-6a568552-f517-49a1-94c8-c1d3814c3244 > 2018-11-20 07:53:47,177 [GenericHelixController-event_process] WARN > o.a.h.task.WorkflowRebalancer - Workflow configuration is NULL for > > Workflow_of_process_PROCESS_85d960c4-d4c5-4308-aa67-eed87adb7eae-POST-5d7ffcdd-aa19-444d-b5e0-a7d8ecf7a450 > 2018-11-20 07:53:47,271 [pool-2-thread-1] ERROR > o.a.h.m.zk.ZkBaseDataAccessor - Exception while updating path: > > /AiravataDemoCluster/IDEALSTATES/Workflow_of_process_PROCESS_1e38a959-744b-4124-a79b-ff74f0980416-POST-1cdd0eec-1e0a-4f96-b9ca-e40c85365d76 > org.I0Itec.zkclient.exception.ZkInterruptedException: > java.lang.InterruptedException > at > > org.apache.helix.manager.zk.zookeeper.ZkClient.retryUntilConnected(ZkClient.java:943) > at > > org.apache.helix.manager.zk.zookeeper.ZkClient.writeDataReturnStat(ZkClient.java:1141) > at > > org.apache.helix.manager.zk.zookeeper.ZkClient.writeDataGetStat(ZkClient.java:1160) > at > > org.apache.helix.manager.zk.ZkBaseDataAccessor.doUpdate(ZkBaseDataAccessor.java:262) > at > > org.apache.helix.manager.zk.ZkBaseDataAccessor.update(ZkBaseDataAccessor.java:237) > at > > org.apache.helix.manager.zk.ZKHelixDataAccessor.updateProperty(ZKHelixDataAccessor.java:190) > at > > org.apache.helix.manager.zk.ZKHelixDataAccessor.updateProperty(ZKHelixDataAccessor.java:170) > at > > org.apache.helix.controller.rebalancer.util.RebalanceScheduler.invokeRebalance(RebalanceScheduler.java:144) > at > > org.apache.helix.controller.rebalancer.util.RebalanceScheduler$RebalanceInvoker.run(RebalanceScheduler.java:129) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.lang.InterruptedException: null > at java.lang.Object.wait(Native Method) > at java.lang.Object.wait(Object.java:502) > at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1259) > at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1226) > at > org.I0Itec.zkclient.ZkConnection.writeDataReturnStat(ZkConnection.java:131) > at > org.apache.helix.manager.zk.zookeeper.ZkClient$12.call(ZkClient.java:1143) > at > > org.apache.helix.manager.zk.zookeeper.ZkClient.retryUntilConnected(ZkClient.java:931) > ... 15 common frames omitted > 2018-11-20 07:53:47,273 [pool-2-thread-1] WARN > o.a.h.c.r.u.RebalanceScheduler - Failed to invoke rebalance on resource > > Workflow_of_process_PROCESS_1e38a959-744b-4124-a79b-ff74f0980416-POST-1cdd0eec-1e0a-4f96-b9ca-e40c85365d76 > 2018-11-20 07:53:47,526 [GenericHelixController-event_process] WARN > o.a.h.task.WorkflowRebalancer - Workflow configuration is NULL for > > Workflow_of_process_PROCESS_4660666a-6acf-477e-bf97-0671b666dd19-PRE-6a568552-f517-49a1-94c8-c1d3814c3244 > 2018-11-20 07:53:48,229 [GenericHelixController-event_process] WARN > o.a.h.task.WorkflowRebalancer - Workflow configuration is NULL for > > Workflow_of_process_PROCESS_4660666a-6acf-477e-bf97-0671b666dd19-PRE-6a568552-f517-49a1-94c8-c1d3814c3244 > 2018-11-20 07:53:48,735 [GenericHelixController-event_process] WARN > o.a.h.task.WorkflowRebalancer - Workflow configuration is NULL for > > Workflow_of_process_PROCESS_4660666a-6acf-477e-bf97-0671b666dd19-PRE-6a568552-f517-49a1-94c8-c1d3814c3244 > 2018-11-20 07:53:49,220 [GenericHelixController-event_process] WARN > o.a.h.task.WorkflowRebalancer - Workflow configuration is NULL for > > Workflow_of_process_PROCESS_4660666a-6acf-477e-bf97-0671b666dd19-PRE-6a568552-f517-49a1-94c8-c1d3814c3244 > 2018-11-20 07:54:03,112 [GenericHelixController-event_process] WARN > o.a.h.task.WorkflowRebalancer - Workflow configuration is NULL for > > Workflow_of_process_PROCESS_4660666a-6acf-477e-bf97-0671b666dd19-PRE-6a568552-f517-49a1-94c8-c1d3814c3244 > 2018-11-20 07:54:03,898 [GenericHelixController-event_process] WARN > o.a.h.task.WorkflowRebalancer - Workflow configuration is NULL for > > Workflow_of_process_PROCESS_a83a532b-76db-40f1-a1f3-8b417a5c2471-POST-766b65e3-20fe-4335-8037-7ca990d2da2a > 2018-11-20 07:54:03,898 [GenericHelixController-event_process] WARN > o.a.h.task.WorkflowRebalancer - Workflow configuration is NULL for > > Workflow_of_process_PROCESS_4660666a-6acf-477e-bf97-0671b666dd19-PRE-6a568552-f517-49a1-94c8-c1d3814c3244 > 2018-11-20 07:54:04,449 [GenericHelixController-event_process] WARN > o.a.h.task.WorkflowRebalancer - Workflow configuration is NULL for > > Workflow_of_process_PROCESS_4660666a-6acf-477e-bf97-0671b666dd19-PRE-6a568552-f517-49a1-94c8-c1d3814c3244 > 2018-11-20 07:54:05,186 [GenericHelixController-event_process] WARN > o.a.h.task.WorkflowRebalancer - Workflow configuration is NULL for > > Workflow_of_process_PROCESS_4660666a-6acf-477e-bf97-0671b666dd19-PRE-6a568552-f517-49a1-94c8-c1d3814c3244 > 2018-11-20 07:54:05,704 [GenericHelixController-event_process] WARN > o.a.h.task.WorkflowRebalancer - Workflow configuration is NULL for > > Workflow_of_process_PROCESS_4660666a-6acf-477e-bf97-0671b666dd19-PRE-6a568552-f517-49a1-94c8-c1d3814c3244 > 2018-11-20 07:54:06,186 [GenericHelixController-event_process] WARN > o.a.h.task.WorkflowRebalancer - Workflow configuration is NULL for > > Workflow_of_process_PROCESS_4660666a-6acf-477e-bf97-0671b666dd19-PRE-6a568552-f517-49a1-94c8-c1d3814c3244 > 2018-11-20 07:54:06,639 [GenericHelixController-event_process] WARN > o.a.h.task.WorkflowRebalancer - Workflow configuration is NULL for > > Workflow_of_process_PROCESS_4660666a-6acf-477e-bf97-0671b666dd19-PRE-6a568552-f517-49a1-94c8-c1d3814c3244 > 2018-11-20 07:55:20,463 [GenericHelixController-event_process] WARN > o.a.h.task.WorkflowRebalancer - Workflow configuration is NULL for > > Workflow_of_process_PROCESS_4660666a-6acf-477e-bf97-0671b666dd19-PRE-6a568552-f517-49a1-94c8-c1d3814c3244 > 2018-11-20 07:55:21,100 [GenericHelixController-event_process] WARN > o.a.h.task.WorkflowRebalancer - Workflow configuration is NULL for > > Workflow_of_process_PROCESS_4660666a-6acf-477e-bf97-0671b666dd19-PRE-6a568552-f517-49a1-94c8-c1d3814c3244 > 2018-11-20 07:55:21,599 [GenericHelixController-event_process] WARN > o.a.h.task.WorkflowRebalancer - Workflow configuration is NULL for > > Workflow_of_process_PROCESS_4660666a-6acf-477e-bf97-0671b666dd19-PRE-6a568552-f517-49a1-94c8-c1d3814c3244 > 2018-11-20 07:55:22,099 [GenericHelixController-event_process] WARN > o.a.h.task.WorkflowRebalancer - Workflow configuration is NULL for > > Workflow_of_process_PROCESS_4660666a-6acf-477e-bf97-0671b666dd19-PRE-6a568552-f517-49a1-94c8-c1d3814c3244 > 2018-11-20 08:35:43,890 [GenericHelixController-event_process] WARN > o.a.h.task.WorkflowRebalancer - Workflow configuration is NULL for > > Workflow_of_process_PROCESS_4660666a-6acf-477e-bf97-0671b666dd19-PRE-6a568552-f517-49a1-94c8-c1d3814c3244 > 2018-11-20 08:35:44,385 [GenericHelixController-event_process] WARN > o.a.h.task.WorkflowRebalancer - Workflow configuration is NULL for > > Workflow_of_process_PROCESS_4660666a-6acf-477e-bf97-0671b666dd19-PRE-6a568552-f517-49a1-94c8-c1d3814c3244 > 2018-11-20 08:35:44,828 [GenericHelixController-event_process] WARN > o.a.h.task.WorkflowRebalancer - Workflow configuration is NULL for > > Workflow_of_process_PROCESS_4660666a-6acf-477e-bf97-0671b666dd19-PRE-6a568552-f517-49a1-94c8-c1d3814c3244 > 2018-11-20 08:35:45,362 [GenericHelixController-event_process] WARN > o.a.h.task.WorkflowRebalancer - Workflow configuration is NULL for > > Workflow_of_process_PROCESS_4660666a-6acf-477e-bf97-0671b666dd19-PRE-6a568552-f517-49a1-94c8-c1d3814c3244 > 2018-11-20 08:35:45,883 [GenericHelixController-event_process] WARN > o.a.h.task.WorkflowRebalancer - Workflow configuration is NULL for > > Workflow_of_process_PROCESS_4660666a-6acf-477e-bf97-0671b666dd19-PRE-6a568552-f517-49a1-94c8-c1d3814c3244 > 2018-11-20 08:35:46,321 [GenericHelixController-event_process] WARN > o.a.h.task.WorkflowRebalancer - Workflow configuration is NULL for > > Workflow_of_process_PROCESS_4660666a-6acf-477e-bf97-0671b666dd19-PRE-6a568552-f517-49a1-94c8-c1d3814c3244 > > Thanks > Dimuthu >
