Are folks OK to wait on that OWASP issue I documented over the weekend? afaict we are not affected but it would be good to get another pair of eyes on it.
Patrick On Mon, Dec 17, 2018 at 2:55 PM Andor Molnár <an...@apache.org> wrote: > Hi team, > > > I'm proudly announce that thanks to the joint effort from the community, > the 3.5 blockers list has become empty: > > "project = ZooKeeper AND resolution = Unresolved AND fixVersion = 3.5.5 > AND priority in (blocker, critical) ORDER BY priority DESC, key ASC" > > > Well... almost. All the blocker issues have gone, but we still have the > Maven migration to complete before the stable release. If you have some > free cycles, please join us testing the Maven build on this PR: > > https://github.com/apache/zookeeper/pull/708 > > I hope we can merge it pretty soon. > > > In terms of the builds, the weather at 3.5 branch is quite sunny nowadays: > > https://builds.apache.org/view/S-Z/view/ZooKeeper/ > > The Java 11 build is still having some difficulties, which hopefully I > can address before the holidays: > > https://issues.apache.org/jira/browse/ZOOKEEPER-3204 > > > If you happen to know about something which is important from 3.5's > perspective and missing from the above, please don't hesitate to share. > > > Happy ZooKeeping! > > Andor > > > > On 11/2/18 21:12, Fangmin Lv wrote: > > Andor, > > > > Here is the PR to port ZK-3104 from master to 3.4: > > https://github.com/apache/zookeeper/pull/685. > > > > Fangmin > > > > On Fri, Nov 2, 2018 at 11:46 AM Fangmin Lv <lvfang...@gmail.com> wrote: > > > >> Hi Andor, > >> > >> Is anyone working on ZK-2778? I can pick it up if there is no one > working > >> on it yet. > >> > >> I'll open a 3.5 PR for ZK-3104 today. > >> > >> Fangmin > >> > >> On Fri, Oct 26, 2018 at 3:33 AM Andor Molnar <an...@apache.org> wrote: > >> > >>> Hi folks, > >>> > >>> You’ve probably realised lots of update emails coming from Jira. Please > >>> be aware that we’ve updated a bunch of open blocker/critical 3.5 > tickets to > >>> reflect to what we discussed in this email. > >>> > >>> If you open up the following jira filter: > >>> > >>> project = ZooKeeper and resolution = Unresolved and fixVersion = 3.5.5 > >>> AND priority in (blocker, critical) ORDER BY priority DESC, key ASC > >>> > >>> You’ll see the most up-to-date list of tickets which need to be > addressed > >>> before the stable 3.5 release. > >>> > >>> Thank you for your efforts to get this done. > >>> > >>> Fangmin, ZK-3104 is waiting for backport, but ticket has already been > >>> resolved. Have you created a separate ticket for the backport or shall > I > >>> just reopen it with the right fix versions? > >>> > >>> Thanks, > >>> Andor > >>> > >>> > >>> > >>>> On 2018. Oct 8., at 12:34, Andor Molnar <an...@apache.org> wrote: > >>>> > >>>> Hi, > >>>> > >>>> Let me summarize and give a quick update on the outstanding issues for > >>> 3.5 GA: > >>>> - ZOOKEEPER-1818 (Fix don't care for trunk) > >>>> - ZOOKEEPER-2778 (Potential server deadlock between follower sync with > >>> leader and follower receiving external connection requests.) > >>>> - ZOOKEEPER-3021 Migrate project structure to Maven (ongoing) > >>>> - ZOOKEEPER-925 Docs generation to Maven > >>>> - ZOOKEEPER-3104 (waiting for backport) > >>>> - ZOOKEEPER-3125 (waiting for backport PR #647) > >>>> > >>>> The 2 Maven related tickets are no-brainers as well as the backports. > >>> ZK-2778 has been picked up by Maoling (thanks!) as far as I can see, > >>> ZK-1818 is the only one waiting for a volunteer. > >>>> Please correct me if I’ve missed something. > >>>> > >>>> Regards, > >>>> Andor > >>>> > >>>> > >>>> > >>>> > >>>>> On 2018. Sep 28., at 18:32, Tamas Penzes <tam...@cloudera.com.INVALID > > > >>> wrote: > >>>>> Hi All, > >>>>> > >>>>> I would add ZOOKEEPER-3021 > >>>>> <https://issues.apache.org/jira/browse/ZOOKEEPER-3021> Migrate > project > >>>>> structure to Maven build as a blocker too. Since the migration has > >>> started > >>>>> it would be good to finish before releasing ZK 3.5.x GA. > >>>>> > >>>>> ZOOKEEPER-925 <https://issues.apache.org/jira/browse/ZOOKEEPER-925> > >>> replace > >>>>> our forrest site and documentation generation might also be a good > >>> idea, > >>>>> since then we could deliver the new MarkDown based documentation. > >>>>> > >>>>> Regards, Tamaas > >>>>> > >>>>> On Fri, Sep 14, 2018 at 10:09 AM Fangmin Lv <lvfang...@gmail.com> > >>> wrote: > >>>>>> Oh, sorry for the confusion, I should provide more context. > >>>>>> > >>>>>> Leader will use on disk txn sync with followers to if the peer zxid > >>> is not > >>>>>> in it's in memory commit logs, the code is here: Leader on disk txn > >>> sync > >>>>>> < > >>>>>> > >>> > https://github.com/apache/zookeeper/blob/master/src/java/main/org/apache/zookeeper/server/quorum/LearnerHandler.java#L774 > >>>>>>> . > >>>>>> There is bug that potentially there will be gap in the txn files, > like > >>>>>> after snap sync, etc, so it's possible the peer will miss txns due > to > >>> this. > >>>>>> The option to disable it is snapshotSizeFactor > >>>>>> < > >>>>>> > >>> > https://github.com/apache/zookeeper/blob/master/src/java/main/org/apache/zookeeper/server/ZKDatabase.java#L81 > >>>>>>> , > >>>>>> set it to -1 will disable this feature. On 3.5, it's better to have > a > >>> PR to > >>>>>> set this to -1 by default. It might have more SNAP sync, but from > our > >>> prod > >>>>>> it doesn't seem to be a big problem to me. > >>>>>> > >>>>>> I can send out the diff to disable it by default on 3.5 if you guys > >>> think > >>>>>> this is the right way to do. > >>>>>> > >>>>>> Thanks, > >>>>>> Fangmin > >>>>>> > >>>>>> On Thu, Sep 13, 2018 at 1:58 AM Andor Molnar <an...@apache.org> > >>> wrote: > >>>>>>> What’s needed to turn it off? > >>>>>>> Do we need a PR or it’s just a config option? > >>>>>>> Shall we implement a feature switch for that and turn it off by > >>> default? > >>>>>>> Sorry I don’t have too much insight on disk txn sync. > >>>>>>> > >>>>>>> Andor > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>> On 2018. Sep 13., at 9:16, Fangmin Lv <lvfang...@gmail.com> > wrote: > >>>>>>>> > >>>>>>>> And to be clear, ZOOKEEPER-2418 is actually just one case of > >>>>>>> inconsistency > >>>>>>>> which could caused by on disk txn sync, as I mentioned in a newer > >>> JIRA > >>>>>>>> ZOOKEEPER-2846 < > >>> https://issues.apache.org/jira/browse/ZOOKEEPER-2846>, > >>>>>>> the > >>>>>>>> snap sync or txn sync could also leave txns gap in the txn file, > >>> which > >>>>>>> is a > >>>>>>>> more common case could trigger this issue. > >>>>>>>> > >>>>>>>> I would suggest to turn off the on disk txn sync by default for > now > >>> to > >>>>>>>> avoid this issue, after we finished ZOOKEEPER-3114, we can use > that > >>> to > >>>>>>>> validate the on disk txns during syncing. > >>>>>>>> > >>>>>>>> Thanks, > >>>>>>>> Fangmin > >>>>>>>> > >>>>>>>> On Wed, Sep 12, 2018 at 9:55 AM Fangmin Lv <lvfang...@gmail.com> > >>>>>> wrote: > >>>>>>>>> Andor, > >>>>>>>>> > >>>>>>>>> ZOOKEEPER-3114 is about adding real time digest checking to help > >>>>>>> detecting > >>>>>>>>> inconsistency, it's a new feature with amounts of code change. > I'll > >>>>>>> start > >>>>>>>>> upstream it part by part, but I don't expect it's being merged in > >>> the > >>>>>>> next > >>>>>>>>> few weeks. So yes, it's a nice to have, but definitely not a > block > >>> for > >>>>>>> 3.5. > >>>>>>>>> Thanks, > >>>>>>>>> Fangmin > >>>>>>>>> > >>>>>>>>> On Wed, Sep 12, 2018 at 2:55 AM Andor Molnar <an...@apache.org> > >>>>>> wrote: > >>>>>>>>>> Fangmin, > >>>>>>>>>> > >>>>>>>>>> Sorry, I just noticed that you want to include the consistency > >>> fixes > >>>>>> in > >>>>>>>>>> the stable version which is fine. Let’s finish the backports and > >>>>>> we’ll > >>>>>>> be > >>>>>>>>>> done with them. > >>>>>>>>>> > >>>>>>>>>> ZOOKEEPER-3114 is essentially a new feature, I wouldn’t block > 3.5 > >>>>>> with > >>>>>>>>>> that. What do you think? > >>>>>>>>>> > >>>>>>>>>> Andor > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>> On 2018. Sep 12., at 11:52, Andor Molnar <an...@apache.org> > >>> wrote: > >>>>>>>>>>> Cool, thanks for the clarification. > >>>>>>>>>>> > >>>>>>>>>>> The updated list is as follows: > >>>>>>>>>>> > >>>>>>>>>>> - ZOOKEEPER-236 (SSL/TLS support for Atomic Broadcast protocol) > >>>>>>>>>>> - ZOOKEEPER-1818 (Fix don't care for trunk) > >>>>>>>>>>> - ZOOKEEPER-2778 (Potential server deadlock between follower > sync > >>>>>> with > >>>>>>>>>> leader and follower receiving external connection requests.) > >>>>>>>>>>> The following are not critical and no blockers for the stable > >>>>>> release: > >>>>>>>>>>> Waiting for to be ported to 3.5: > >>>>>>>>>>> - ZOOKEEPER-3104 > >>>>>>>>>>> - ZOOKEEPER-3125 > >>>>>>>>>>> - ZOOKEEPER-3127 > >>>>>>>>>>> > >>>>>>>>>>> New feature: > >>>>>>>>>>> - ZOOKEEPER-3114 (fixes ZOOKEEPER-2184 too) > >>>>>>>>>>> > >>>>>>>>>>> Regards, > >>>>>>>>>>> Andor > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> On 2018. Sep 12., at 0:42, Fangmin Lv <lvfang...@gmail.com> > >>> wrote: > >>>>>>>>>>>> Hi Andor, > >>>>>>>>>>>> > >>>>>>>>>>>> That's the on disk txn feature, which was disabled internally > >>> after > >>>>>>> we > >>>>>>>>>>>> found the potentially inconsistent issue. The only solution we > >>> have > >>>>>>>>>> for now > >>>>>>>>>>>> is waiting for the new digest checking feature I mentioned in > >>>>>>>>>>>> ZOOKEEPER-3114. > >>>>>>>>>>>> > >>>>>>>>>>>> I think there are some other critical consistent issues we > just > >>>>>> fixed > >>>>>>>>>> on > >>>>>>>>>>>> master recently: ZOOKEEPER-3104, ZOOKEEPER-3125, > >>> ZOOKEEPER-3127, I > >>>>>>>>>> think we > >>>>>>>>>>>> should include that in the official 3.5 release as well. > >>>>>>>>>>>> > >>>>>>>>>>>> Thanks, > >>>>>>>>>>>> Fangmin > >>>>>>>>>>>> > >>>>>>>>>>>> On Tue, Sep 11, 2018 at 11:58 AM Andor Molnár < > an...@apache.org > >>>>>>>>>> wrote: > >>>>>>>>>>>>> Hi Jeelani, > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> Thanks for letting me know. I'm happy to remove it from the > >>> list > >>>>>> to > >>>>>>>>>> get > >>>>>>>>>>>>> closer to a stable release. :) > >>>>>>>>>>>>> > >>>>>>>>>>>>> What's the feature which can be disabled to avoid data > >>>>>>> inconsistency? > >>>>>>>>>>>>> > >>>>>>>>>>>>> Andor > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> On 09/10/2018 11:33 PM, Mohamed Jeelani wrote: > >>>>>>>>>>>>>> Thanks Andor for compiling this. Should we be ignoring > >>>>>>>>>> ZOOKEEPER-2418 as > >>>>>>>>>>>>> well? This exists in 3.4 as well and the feature can be > >>> disabled. > >>>>>> We > >>>>>>>>>> are > >>>>>>>>>>>>> working on a longer term fix for it in 3.6. > >>>>>>>>>>>>>> Regards, > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Jeelani > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On 9/10/18, 5:19 AM, "Andor Molnar" > >>> <an...@cloudera.com.INVALID > >>>>>>>>>> wrote: > >>>>>>>>>>>>>> Fine. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> I'm happy to ignore 1549, 2846 and 2930. Still we have the > >>> list > >>>>>>> of: > >>>>>>>>>>>>>> - ZOOKEEPER-236 (SSL/TLS support for Atomic Broadcast > >>> protocol) > >>>>>>>>>>>>>> - ZOOKEEPER-1818 (Fix don't care for trunk) > >>>>>>>>>>>>>> - ZOOKEEPER-2418 (txnlog diff sync can skip sending some > >>>>>>>>>>>>> transactions to > >>>>>>>>>>>>>> followers) > >>>>>>>>>>>>>> - ZOOKEEPER-2778 (Potential server deadlock between follower > >>>>>> sync > >>>>>>>>>>>>> with > >>>>>>>>>>>>>> leader and follower receiving external connection requests.) > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> SSL (ZK-236) is a feature which essential for the 3.5 > release, > >>>>>>>>>> hence > >>>>>>>>>>>>> I > >>>>>>>>>>>>>> wouldn't leave it out or postpone it for the next stable > >>>>>> release. > >>>>>>>>>> PR > >>>>>>>>>>>>> has > >>>>>>>>>>>>>> been out for a long time, get on reviewing please. > >>>>>>>>>>>>>> The rest are also long outstanding issues which have been > >>> found > >>>>>> in > >>>>>>>>>>>>> the 3.5 > >>>>>>>>>>>>>> branch. > >>>>>>>>>>>>>> ZK-1818 is something which was found in 3.4 and fixed in > 3.4, > >>>>>> but > >>>>>>>>>>>>> never has > >>>>>>>>>>>>>> been fixed in 3.5. Quite a serious issue if still present. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> I think we should at least run some manual testing and see > if > >>> we > >>>>>>>>>>>>> could > >>>>>>>>>>>>>> repro any of these issues before going ahead with a stable > >>>>>>> release. > >>>>>>>>>>>>>> Regards, > >>>>>>>>>>>>>> Andor > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On Fri, Sep 7, 2018 at 3:24 AM, Michael Han < > h...@apache.org> > >>>>>>>>>> wrote: > >>>>>>>>>>>>>>> I haven't went through the entire list, but looks like lots > >>> of > >>>>>> the > >>>>>>>>>>>>> JIRA > >>>>>>>>>>>>>>> issues listed in this thread, such as ZOOKEEPER-1549, 2846, > >>> also > >>>>>>>>>>>>> affects > >>>>>>>>>>>>>>> 3.4 releases. Should we scope these issues out? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I think historically the single outstanding blocking issue > >>> for a > >>>>>>>>>>>>> stable 3.5 > >>>>>>>>>>>>>>> release is the reconfig feature and security concerns > around > >>> it > >>>>>>>>>>>>> (somehow > >>>>>>>>>>>>>>> addressed in ZOOKEEPER-2014), and the alpha and beta > releases > >>>>>> were > >>>>>>>>>>>>> created > >>>>>>>>>>>>>>> to stabilize that feature. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>> > https://urldefense.proofpoint.com/v2/url?u=http-3A__zookeeper-2Duser.578899.n2.nabble.com_Zookeeper-2Dwith-2D&d=DwIBaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=Vl4oKanLQehvaulUvoKg8A&m=wqlhnot9c-pQLdkGkccSGNpELUNUnB-wy_h0iA3PRqI&s=_tGtL3nMWtuPrXKXDx27AIWOzyyT7W-CjIVLDFZwT0E&e= > >>>>>>>>>>>>>>> SSL-release-date-tt7581744.html > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> So it looks like we are in good shape to release. Something > >>>>>> might > >>>>>>>>>>>>> worth > >>>>>>>>>>>>>>> doing to claim the quality of 3.5 is on par with 3.4 > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> * Run Jepsen on 3.5 - 3.4 passed the test for the record > >>>>>>>>>>>>>>> > >>> > https://urldefense.proofpoint.com/v2/url?u=https-3A__aphyr.com_posts_291-2Djepsen-2Dzookeeper&d=DwIBaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=Vl4oKanLQehvaulUvoKg8A&m=wqlhnot9c-pQLdkGkccSGNpELUNUnB-wy_h0iA3PRqI&s=VjORkX5s7hrJyl8mW9Q4cfeSWF4qfTdyRjcuAiBt0y4&e= > >>>>>>>>>>>>>>> * Fix all flaky tests on 3.5 - 3.4 has little or no flaky > >>> tests > >>>>>> at > >>>>>>>>>>>>> all. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On Tue, Sep 4, 2018 at 1:48 AM, Andor Molnar > >>>>>>>>>>>>> <an...@cloudera.com.invalid> > >>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Thanks Maoling! That would be huge help, I appreciate it. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Andor > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>> > >>>>>>> > >>> >
