Hi ShaoFeng, Here is another thread that gives more info on adding partitions dynamically.
http://mail-archives.apache.org/mod_mbox/helix-user/201503.mbox/%3ccabyby7dgqxnzstfhcfkyzpe55jd6ialeaak2u4qbnzxd96k...@mail.gmail.com%3E On Wed, Feb 3, 2016 at 1:26 AM, ShaoFeng Shi <[email protected]> wrote: > I read through the codes and now I know how to adjust the partition number > now, will try that. Thanks for your attention. > > 2016-02-03 10:44 GMT+08:00 ShaoFeng Shi <[email protected]>: > > > Hi Kishore, from Helix's tutorial, I don't see a way to dynamically > > add/remove partition, which made me had thought the partitions need be > > fixed when adding the resource. If that can be adjusted, it would match > my > > case very well. Could you give me a sample or the API name on that? > Thanks! > > > > Greg, thanks for putting that to a JIRA, I has watched it and looks > > forward for the fix in next release. > > > > 2016-02-03 3:05 GMT+08:00 Greg Brandt <[email protected]>: > > > >> Hey ShaoFeng, > >> > >> I think this is a bug in the UI... in > >> ResourceStateSpec#getResourceStateTable > >> < > >> > https://github.com/apache/helix/blob/master/helix-ui/src/main/java/org/apache/helix/ui/api/ResourceStateSpec.java#L62 > >> >, > >> we should check the rebalance mode, and use the partitions from external > >> view if FULL_AUTO, as there's no ideal placement for them. > >> > >> I've created this ticket > https://issues.apache.org/jira/browse/HELIX-627 > >> for that fix. > >> > >> Thanks, > >> -Greg > >> > >> On Tue, Feb 2, 2016 at 9:04 AM, kishore g <[email protected]> wrote: > >> > >> > Hi ShaoFeng*,* > >> > > >> > The rebalancer works ensures even distribution within a resource. > Across > >> > resources, it uses hashing based on resource names. > >> > So if you add few more resources, you might start seeing the > >> distribution > >> > change. > >> > > >> > Another option is to add one partition every 5 minutes to the same > >> > resource. > >> > > >> > thanks, > >> > Kishore G > >> > > >> > On Tue, Feb 2, 2016 at 6:43 AM, ShaoFeng Shi <[email protected]> > >> > wrote: > >> > > >> > > Hello Helix developers, > >> > > > >> > > I'm trying to use Helix (0.7.1) to manage some resources; every 5 > >> > minutes, > >> > > such a resource will be added to the cluster; Each resource only > has 1 > >> > > partition, and it's state model is "LEADER_STANDBY" (the leader will > >> do > >> > > some exclusive job); the rebalance mode is FULL_AUTO. Now the > cluster > >> has > >> > > two instances. The interesting thing I found is, all the LEADERs > are > >> > > assigned to the first node of my cluster, and the other cluster > >> carries > >> > all > >> > > STANDBYs. Although when I disable the first node, all the LEADERs > are > >> > > transferred to the second node automatically, I expect there could > be > >> > some > >> > > load balance among them: half leaders on the first node, the others > on > >> > the > >> > > second. Did I do something wrong, or this is expected? > >> > > > >> > > Another issue I faced is, on the helix-ui, it can show all > resources, > >> but > >> > > when selecting a resource, the "partitions" tab couldn't show which > >> > > instances it be assigned to; the only message is "No partitions of > >> > > <resource_name> are assigned!" > >> > > > >> > > I checked the UI code, it indicates the reason is the "listFields" > is > >> > > empty. That matches with what I see with the helix-admin.sh > >> > > --listResourceInfo: > >> > > > >> > > IdealState for Resource_Stream_1454391900000_1454392200000: > >> > > { > >> > > "id" : "Resource_Stream_1454391900000_1454392200000", > >> > > "mapFields" : { > >> > > "Resource_Stream_1454391900000_1454392200000" : { > >> > > } > >> > > }, > >> > > "listFields" : { > >> > > "Resource_Stream_1454391900000_1454392200000" : [ ] > >> > > }, > >> > > "simpleFields" : { > >> > > "IDEAL_STATE_MODE" : "AUTO_REBALANCE", > >> > > "NUM_PARTITIONS" : "1", > >> > > "REBALANCE_MODE" : "FULL_AUTO", > >> > > "REPLICAS" : "3", > >> > > "STATE_MODEL_DEF_REF" : "LeaderStandby", > >> > > "STATE_MODEL_FACTORY_NAME" : "DEFAULT" > >> > > } > >> > > } > >> > > > >> > > ExternalView for Resource_Stream_1454391900000_1454392200000: > >> > > { > >> > > "id" : "Resource_Stream_1454391900000_1454392200000", > >> > > "mapFields" : { > >> > > "Resource_Stream_1454391900000_1454392200000" : { > >> > > "kylin-dev3_8080" : "LEADER", > >> > > "kylin-dev4_8080" : "STANDBY" > >> > > } > >> > > }, > >> > > "listFields" : { > >> > > }, > >> > > "simpleFields" : { > >> > > "BUCKET_SIZE" : "0" > >> > > } > >> > > } > >> > > > >> > > I checked the code, it seems for FULL_AUTO, the values for > >> "listFields" > >> > and > >> > > "mapFields" are empty (at least at the begining). Will it be updated > >> at > >> > > some point of time, or how could I trigger that? > >> > > > >> > > Thanks in advance for your help; I'm new to Helix so may not get > some > >> > > concepts correctly. Just correct me if I'm wrong, thanks! > >> > > > >> > > -- > >> > > Best regards, > >> > > > >> > > Shaofeng Shi > >> > > > >> > > >> > > > > > > > > -- > > Best regards, > > > > Shaofeng Shi > > > > > > > -- > Best regards, > > Shaofeng Shi >
