I am also interested in how can you make the storm be connected with HDFS? have you modified the lib from storm? can you guys roughly describe the steps? thanks
2014-09-10 0:16 GMT-04:00 Jiang Jacky <jiang0...@gmail.com>: > the best solution is that we can add the multiple nimbus server in the > storm.yaml, those should be for failover, it also will be easy to configure > > 2014-09-09 22:19 GMT-04:00 潘臻轩 <zhenxuan...@gmail.com>: > > yes, I have implement this way.. and it ok in fact.. >> I implement a total ha solution for nimbus. >> and our team write a total scheduler for storm(such as yarn for support >> 700+ cluster) >> >> 2014-09-10 10:02 GMT+08:00 Ankit Toshniwal <ankitoshni...@gmail.com>: >> >>> Yes, that's a problem area, and we have been discussing it internally on >>> how we can handle it better. We are considering moving to an HDFS based >>> solution where Nimbus will upload the jars into hdfs instead of local disk >>> (as that is a single point of failure) and supervisors will be downloading >>> the jar's from hdfs as well. >>> >>> The other problem we ran into was nic saturation on Nimbus host since >>> too many machines were doing copy of the jar's (180MB in size) to worker >>> machines leading to the total increase in time. Thus, with moving to HDFS >>> based solution we can do this more effectively and faster plus it scales >>> better. >>> >>> We do not have a working prototype for it, but something we are >>> actively pursuing. >>> >>> Ankit >>> >>> On Tue, Sep 9, 2014 at 6:43 PM, 潘臻轩 <zhenxuan...@gmail.com> wrote: >>> >>>> I not agree Nathan, if just nimbus down, it is fail-fast.but if the >>>> machine happen error(such as disk error), this may lead >>>> topology clear. >>>> >>>> 2014-09-10 9:39 GMT+08:00 潘臻轩 <zhenxuan...@gmail.com>: >>>> >>>>> *According to my knowledge, is not the case。you should check it with >>>>> script or other way.* >>>>> >>>>> 2014-09-10 0:49 GMT+08:00 Jiang Jacky <jiang0...@gmail.com>: >>>>> >>>>>> Hi, I read the articles about the nimbus, it specifies the nimbus >>>>>> daemon is fail-fast. But I am not sure if it is like Hadoop, there is >>>>>> secondary server for failover, if the nimbus server is totally down, then >>>>>> the secondary server can be up. Thanks >>>>>> >>>>> >>>>> >>>> >>> >> >
