Thanks! We're looking into one other emergent issue that we uncovered during the rollout of server side TLS on RegionServers. It seems nettyDirectMemory has increased substantially when under load with it enabled. Details in https://issues.apache.org/jira/browse/HBASE-27947.
On Thu, Jun 22, 2023 at 12:02 PM 张铎(Duo Zhang) <palomino...@gmail.com> wrote: > PR is ready > > https://github.com/apache/hbase/pull/5305 > > PTAL. > > Thanks. > > 张铎(Duo Zhang) <palomino...@gmail.com> 于2023年6月22日周四 21:40写道： > > > > Ah, missed your last comment on HBASE-27782. > > > > Let me take a look. > > > > Netty has some rules about how the exceptions are passed through the > > pipeline(especially the order, forward or backward...) but honestly I > > always forget it just a day later after I finished the code... > > > > Bryan Beaudreault <bbeaudrea...@apache.org> 于2023年6月17日周六 00:43写道： > > > > > > In terms of TLS: > > > > > > - All of our clients (many thousands) in production are using the > > > NettyRpcConnection with TLS enabled. However, these clients are > currently > > > connecting to the RegionServer/HMaster through an haproxy process > local to > > > each server which handles SSL termination. So not quite end-to-end yet. > > > - On the server side, most of our QA environment (a thousand > regionservers > > > and ~200 hmasters) are running it. So these are accepting TLS from > clients > > > and using TLS for intra-cluster communication. > > > > > > The migration is tricky for us due to the scale and the fact that we > need > > > to migrate off haproxy at the same time. Hopefully we should have some > of > > > production running end-to-end TLS within the next month or so. > > > > > > From what we've seen in QA so far, there have not been any major > issues. We > > > also couldn't discern any performance issues in testing, though we were > > > comparing against our legacy haproxy setup and can't really compare > against > > > kerberos. > > > > > > One outstanding issue is > https://issues.apache.org/jira/browse/HBASE-27782, > > > which we still see periodically. It doesn't seem to cause actual > issues, > > > since the RpcClient still handles it gracefully, but it does cause > noise > > > and may have implications. > > > > > > On Fri, Jun 16, 2023 at 11:41 AM 张铎(Duo Zhang) <palomino...@gmail.com> > > > wrote: > > > > > > > So any updates here? > > > > > > > > Do we have any good news about the TLS usage in production so we can > > > > move forward on release 2.6.x? > > > > > > > > Thanks. > > > > > > > > Andrew Purtell <apurt...@apache.org> 于2023年4月7日周五 09:37写道： > > > > > > > > > > Agreed, that sounds like a good plan. > > > > > > > > > > On Wed, Mar 29, 2023 at 7:31 AM 张铎(Duo Zhang) < > palomino...@gmail.com> > > > > wrote: > > > > > > > > > > > I think we could follow the old pattern when we cut a new release > > > > branch. > > > > > > That is, after the new release branch is cut and the new minor > release > > > > is > > > > > > out, we will do a final release of the oldest release line and > then > > > > mark it > > > > > > as EOL. > > > > > > > > > > > > So here, I think once we cut branch-2.6 and release 2.6.0, we > can do a > > > > > > final release for 2.4.x and mark 2.4.x as EOL. > > > > > > > > > > > > Thanks. > > > > > > > > > > > > Bryan Beaudreault <bbeaudrea...@apache.org> 于2023年3月27日周一 > 09:57写道： > > > > > > > > > > > > > Primary development on hbase-backup and TLS is complete. There > are a > > > > > > couple > > > > > > > minor things I may want to add to TLS in the future, such as > > > > pluggable > > > > > > cert > > > > > > > verification. But those are not needed for initial release IMO. > > > > > > > > > > > > > > We are almost ready integrating hbase-backup in production. > We’ve > > > > fixed a > > > > > > > few minor things (all committed) but otherwise it’s worked > well so > > > > far in > > > > > > > tests. > > > > > > > > > > > > > > We are a bit delayed in integrating TLS. I’m hopeful it will > happen > > > > in > > > > > > the > > > > > > > next 2-3 months. It’s a big project for us, so not quick, but > > > > definitely > > > > > > on > > > > > > > the roadmap. > > > > > > > > > > > > > > It seems like cloudera may be closer to integrating TLS in > > > > production. > > > > > > > Balazs recently filed and fixed HBASE-27673 related to mTLS. > Maybe > > > > he can > > > > > > > chime in on his status, or let me know if I am totally off > base :) > > > > > > > > > > > > > > On Sun, Mar 26, 2023 at 9:25 PM Andrew Purtell < > > > > andrew.purt...@gmail.com > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Before we open a new code line should we discuss EOL of 2.4? > After > > > > the > > > > > > > > first 2.6 release? It’s not required of course but cuts down > the > > > > amount > > > > > > > of > > > > > > > > labor to have two 2.x code lines (presumably, one as stable > and > > > > one as > > > > > > > > next) rather than three. Perhaps even before that, should we > move > > > > the > > > > > > > > stable pointer to the latest 2.5 release? > > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 26, 2023, at 5:59 PM, 张铎 <palomino...@gmail.com> > wrote: > > > > > > > > > > > > > > > > > > Bump. > > > > > > > > > > > > > > > > > > I believe the mTLS and backup related code have all been > > > > finished on > > > > > > > > > branch-2? > > > > > > > > > > > > > > > > > > Are there any other things which block us making the > branch-2.6 > > > > > > branch? > > > > > > > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > > > > > Mallikarjun <mallik.v.ar...@gmail.com> 于2022年10月17日周一 > 02:09写道： > > > > > > > > > > > > > > > > > >> On hbase-backup, we are using in production for more then > 1 > > > > year. I > > > > > > > can > > > > > > > > >> vouch for it to be stable enough to be in a release > version so > > > > that > > > > > > > more > > > > > > > > >> people can use it and polished it further. > > > > > > > > >> > > > > > > > > >>> On Sun, Oct 16, 2022, 11:25 PM Andrew Purtell < > > > > > > > > andrew.purt...@gmail.com> > > > > > > > > >>> wrote: > > > > > > > > >>> > > > > > > > > >>> My understanding is some folks evaluating and polishing > TLS for > > > > > > their > > > > > > > > >>> production are also considering hbase-backup in the same > way, > > > > which > > > > > > > is > > > > > > > > >> why > > > > > > > > >>> I linked them together. If that is incorrect then they > both are > > > > > > still > > > > > > > > >> worth > > > > > > > > >>> considering in my opinion but would have a more tenuous > link. > > > > > > > > >>> > > > > > > > > >>> Where we are with hbase-backup is it should probably be > ported > > > > to > > > > > > > where > > > > > > > > >>> more people would be inclined to evaluate it, in order > for it > > > > to > > > > > > make > > > > > > > > >> more > > > > > > > > >>> progress. A new minor releasing line would fit. On the > other > > > > hand > > > > > > if > > > > > > > it > > > > > > > > >> is > > > > > > > > >>> too unpolished then the experience would be poor. > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > >>>> On Oct 16, 2022, at 5:35 AM, 张铎 <palomino...@gmail.com> > > > > wrote: > > > > > > > > >>>> > > > > > > > > >>>> I believe the second one is still ongoing? > > > > > > > > >>>> > > > > > > > > >>>> Andrew Purtell <apurt...@apache.org> 于2022年10月14日周五 > 05:37写道： > > > > > > > > >>>>> > > > > > > > > >>>>> We will begin releasing activity for the 2.6 code line > and > > > > as a > > > > > > > > >>>>> prerequisite to that we shall need to make a new branch > > > > > > branch-2.6 > > > > > > > > >> from > > > > > > > > >>>>> branch-2. > > > > > > > > >>>>> > > > > > > > > >>>>> Before we do that let's make sure all commits for the > key > > > > > > features > > > > > > > of > > > > > > > > >>> 2.6 > > > > > > > > >>>>> are settled in branch-2 before the branching point. > Those key > > > > > > > > features > > > > > > > > >>> are: > > > > > > > > >>>>> - mTLS RPC > > > > > > > > >>>>> - hbase-backup backport > > > > > > > > >>>>> > > > > > > > > >>>>> -- > > > > > > > > >>>>> Best regards, > > > > > > > > >>>>> Andrew > > > > > > > > >>> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Best regards, > > > > > Andrew > > > > > > > > > > Unrest, ignorance distilled, nihilistic imbeciles - > > > > > It's what we’ve earned > > > > > Welcome, apocalypse, what’s taken you so long? > > > > > Bring us the fitting end that we’ve been counting on > > > > > - A23, Welcome, Apocalypse > > > > >
