Hi, Nianjun. BTW, about the single proxy bottleneck, we will try to resolve it with proxies cluster in control-panel module, you can track and join it. >5. ss-proxy is the first version, we will plug ss-scaling into ss-jdbc in the >future, so don't warry about the proxy bottleneck.
At 2020-06-15 15:53:15, "lucas" <[email protected]> wrote: >Hi, Sun Nianjun: > > >Think you for your suggesions. > > >1. Your understanding is pretty good. Change `offer` to `put` is a good idea, >you can do it. And I think queue size defalut as 10000 is ok, if it's too big, >I'm a litter warry memory. as you know, it is multithreading task andconsumer >can change it by configuration. > > >2. Breakpoint resume is a very important function, and there is other guy do >it now. > > >3. You are right, we are think about to use elastic-job to process scaling job. > > >4. You suggesion is very good, but we want all put all other functions (such >as encryption) into ss-jdbc, and scaling is only scaling. > > >5. ss-proxy is the first version, we will plug ss-scaling into ss-jdbc in the >future, so don't warry about the proxy bottleneck. > > >6. `/scaling/job/start` supports json formats, see: >org.apache.shardingsphere.shardingscaling.web.HttpServerHandler#startJob > > >You suggesions are very visionary. there are lot of things we should do it in >the future. and welcome to do it together. > > > > >-------- Forwarding messages -------- > >From: "Sun Nianjun" <[email protected]> >Date: 2020-06-14 23:54:38 >To: "[email protected]" <[email protected]> >Subject: some suggesions for sharding scaling >hi , commuity > > since last week, I got some experience of using sharding scaling, and I > have summarized some suggestions as follows, hope to discuss with the > community : > > > 1. dumper, importer, channel is a typical producer-consumer pattern， > mainly used BlockingQueue in Java，and I adjusted the details of the use a > little in my actual experience: I changed the 'offer' to 'put' in > MemoryChannel , change the queue size a little bit larger, because the dumper > will stuffed the queue full very fast, and dumper is relative slow, even > there is a timeout limitation for method `offer`, it's very easy to cause an > RuntimeException by a failed offer. > 2. I’m thinking about how to add a breakpoint resume function to scaling, > because once there is a problem during the migration process, it is very > troublesome。maybe we could record the scaling position through local log or > something like that, we have to make sure the scaling could resume through > breakpoint after broken down. > 3. since the elastic job is resurrected, why not seperate and handle the > job in scaling by elastic job ? a job system must be more Robust and Stable > 4. It's very common usage in shardingsphere to load an implementaion for > interface, that's SPI . I think the current dumper and importer should exact > as default implementation, and add more implementions and load by SPI . BTW , > importer is not just focus on import data into database, maybe for encryption. > 5. I believe proxy will be the bottleneck for current architecture , event > I tried to scaling more than 40 million data in a mysql to 8 new shard , > sharding proxy is a little bit 'slow' , if we prepare to scale billions data > from 16 sharding to 32 shard , I don't think proxy could withstand such > pressure，we may use sharding jdbc for scaling to reshard. > 6. currently, the parameters for `/scaling/job/start` are in yaml formats, > it's ok if that's for API invocation , but its not friendly for > debugging，should we add some function to translate yaml configuration to json > format in API invocation ？ > >shardingsphere has a very active and open community , I hope the metors could >give some suggestions for that . > >Regards > > > > > >
