Kishore, thanks, it is what I need. While I'm thinking how to properly map the helix partition id to the logic data, will create another thread for the discussion.
2016-02-03 22:02 GMT+08:00 kishore g <[email protected]>: > Hi ShaoFeng, > > Here is another thread that gives more info on adding partitions > dynamically. > > > http://mail-archives.apache.org/mod_mbox/helix-user/201503.mbox/%3ccabyby7dgqxnzstfhcfkyzpe55jd6ialeaak2u4qbnzxd96k...@mail.gmail.com%3E > > > > On Wed, Feb 3, 2016 at 1:26 AM, ShaoFeng Shi <[email protected]> > wrote: > > > I read through the codes and now I know how to adjust the partition > number > > now, will try that. Thanks for your attention. > > > > 2016-02-03 10:44 GMT+08:00 ShaoFeng Shi <[email protected]>: > > > > > Hi Kishore, from Helix's tutorial, I don't see a way to dynamically > > > add/remove partition, which made me had thought the partitions need be > > > fixed when adding the resource. If that can be adjusted, it would match > > my > > > case very well. Could you give me a sample or the API name on that? > > Thanks! > > > > > > Greg, thanks for putting that to a JIRA, I has watched it and looks > > > forward for the fix in next release. > > > > > > 2016-02-03 3:05 GMT+08:00 Greg Brandt <[email protected]>: > > > > > >> Hey ShaoFeng, > > >> > > >> I think this is a bug in the UI... in > > >> ResourceStateSpec#getResourceStateTable > > >> < > > >> > > > https://github.com/apache/helix/blob/master/helix-ui/src/main/java/org/apache/helix/ui/api/ResourceStateSpec.java#L62 > > >> >, > > >> we should check the rebalance mode, and use the partitions from > external > > >> view if FULL_AUTO, as there's no ideal placement for them. > > >> > > >> I've created this ticket > > https://issues.apache.org/jira/browse/HELIX-627 > > >> for that fix. > > >> > > >> Thanks, > > >> -Greg > > >> > > >> On Tue, Feb 2, 2016 at 9:04 AM, kishore g <[email protected]> > wrote: > > >> > > >> > Hi ShaoFeng*,* > > >> > > > >> > The rebalancer works ensures even distribution within a resource. > > Across > > >> > resources, it uses hashing based on resource names. > > >> > So if you add few more resources, you might start seeing the > > >> distribution > > >> > change. > > >> > > > >> > Another option is to add one partition every 5 minutes to the same > > >> > resource. > > >> > > > >> > thanks, > > >> > Kishore G > > >> > > > >> > On Tue, Feb 2, 2016 at 6:43 AM, ShaoFeng Shi < > [email protected]> > > >> > wrote: > > >> > > > >> > > Hello Helix developers, > > >> > > > > >> > > I'm trying to use Helix (0.7.1) to manage some resources; every 5 > > >> > minutes, > > >> > > such a resource will be added to the cluster; Each resource only > > has 1 > > >> > > partition, and it's state model is "LEADER_STANDBY" (the leader > will > > >> do > > >> > > some exclusive job); the rebalance mode is FULL_AUTO. Now the > > cluster > > >> has > > >> > > two instances. The interesting thing I found is, all the LEADERs > > are > > >> > > assigned to the first node of my cluster, and the other cluster > > >> carries > > >> > all > > >> > > STANDBYs. Although when I disable the first node, all the LEADERs > > are > > >> > > transferred to the second node automatically, I expect there could > > be > > >> > some > > >> > > load balance among them: half leaders on the first node, the > others > > on > > >> > the > > >> > > second. Did I do something wrong, or this is expected? > > >> > > > > >> > > Another issue I faced is, on the helix-ui, it can show all > > resources, > > >> but > > >> > > when selecting a resource, the "partitions" tab couldn't show > which > > >> > > instances it be assigned to; the only message is "No partitions of > > >> > > <resource_name> are assigned!" > > >> > > > > >> > > I checked the UI code, it indicates the reason is the "listFields" > > is > > >> > > empty. That matches with what I see with the helix-admin.sh > > >> > > --listResourceInfo: > > >> > > > > >> > > IdealState for Resource_Stream_1454391900000_1454392200000: > > >> > > { > > >> > > "id" : "Resource_Stream_1454391900000_1454392200000", > > >> > > "mapFields" : { > > >> > > "Resource_Stream_1454391900000_1454392200000" : { > > >> > > } > > >> > > }, > > >> > > "listFields" : { > > >> > > "Resource_Stream_1454391900000_1454392200000" : [ ] > > >> > > }, > > >> > > "simpleFields" : { > > >> > > "IDEAL_STATE_MODE" : "AUTO_REBALANCE", > > >> > > "NUM_PARTITIONS" : "1", > > >> > > "REBALANCE_MODE" : "FULL_AUTO", > > >> > > "REPLICAS" : "3", > > >> > > "STATE_MODEL_DEF_REF" : "LeaderStandby", > > >> > > "STATE_MODEL_FACTORY_NAME" : "DEFAULT" > > >> > > } > > >> > > } > > >> > > > > >> > > ExternalView for Resource_Stream_1454391900000_1454392200000: > > >> > > { > > >> > > "id" : "Resource_Stream_1454391900000_1454392200000", > > >> > > "mapFields" : { > > >> > > "Resource_Stream_1454391900000_1454392200000" : { > > >> > > "kylin-dev3_8080" : "LEADER", > > >> > > "kylin-dev4_8080" : "STANDBY" > > >> > > } > > >> > > }, > > >> > > "listFields" : { > > >> > > }, > > >> > > "simpleFields" : { > > >> > > "BUCKET_SIZE" : "0" > > >> > > } > > >> > > } > > >> > > > > >> > > I checked the code, it seems for FULL_AUTO, the values for > > >> "listFields" > > >> > and > > >> > > "mapFields" are empty (at least at the begining). Will it be > updated > > >> at > > >> > > some point of time, or how could I trigger that? > > >> > > > > >> > > Thanks in advance for your help; I'm new to Helix so may not get > > some > > >> > > concepts correctly. Just correct me if I'm wrong, thanks! > > >> > > > > >> > > -- > > >> > > Best regards, > > >> > > > > >> > > Shaofeng Shi > > >> > > > > >> > > > >> > > > > > > > > > > > > -- > > > Best regards, > > > > > > Shaofeng Shi > > > > > > > > > > > > -- > > Best regards, > > > > Shaofeng Shi > > > -- Best regards, Shaofeng Shi
