I think JUnit can group tests in categories with annotations (AFK). Gary
On Tue, Nov 28, 2023, 12:01 AM Ralph Goers <ralph.go...@dslextreme.com> wrote: > I would be -1 if the issues are going to be ignored or not tracked in any > way. I don’t know if GitHub has something like a Jira Epic or if they can > be tagged in some way so that they can be easily located but something like > that would be fine. Even tracking them in Confluence would be fine. > > It would also be great if only the failing tests could be run under a > profile making it easy to fix them. > > Hopefully you get what I mean. I am not looking for something complicated, > just a way to make it easy to find them when someone has the urge to fix > them. > > Ralph > > > On Nov 27, 2023, at 3:28 AM, Volkan Yazıcı <vol...@yazi.ci> wrote: > > > > Ralph did not agree, but did not strongly object either. Ralph, are you > -1 > > on disabling tests only Windows that are failing frequently on Windows > and > > capturing them in tickets to be addressed? > > > > On Thu, Nov 23, 2023 at 12:23 AM Christian Grobmeier < > grobme...@apache.org> > > wrote: > > > >> Ralph said, nobody would ever fix these tests if you do it like this. I > >> think you should create the ticket but keep the tests until we find the > >> issue. Otherwise there issues will rot > >> > >> On Wed, Nov 22, 2023, at 09:13, Volkan Yazıcı wrote: > >>> AFAIC, nobody[1] shows a strong opposition against the idea of > disabling > >>> frequently failing Windows tests only on Windows and creating a ticket > >> for > >>> each one. I will proceed with that. > >>> > >>> [1] Except Piotr, whom I discussed the issue with in Slack and he > agreed > >>> with the above shared approach. > >>> > >>> On Mon, Nov 20, 2023 at 12:57 PM Volkan Yazıcı <vol...@yazi.ci> wrote: > >>> > >>>> I am not asking to disable Windows tests. I am asking to disable tests > >>>> and only those tests that have a failure rate on Windows higher than, > >>>> say, 30%. To be precise, I think there are 2-3 of them dealing with > >>>> network sockets and rolling file appenders. I am not talking about > >>>> dozens or such. > >>>> > >>>> After disabling them, we can create a ticket referencing them. So that > >>>> interested parties can fix them. > >>>> > >>>> On Mon, Nov 20, 2023 at 12:25 PM Piotr P. Karwasz > >>>> <piotr.karw...@gmail.com> wrote: > >>>>> > >>>>> Hi Volkan, > >>>>> > >>>>> On Mon, 20 Nov 2023 at 09:36, Volkan Yazıcı <vol...@yazi.ci> wrote: > >>>>>> > >>>>>> As Gary (the only Windows user among the active Log4j maintainers, > >>>>>> AFAIK) has noticed several times, Log4j tests on Windows are pretty > >>>>>> unstable. It not only fails on Gary's laptop, but Piotr and I need > >> to > >>>>>> give Windows tests in CI a kick on a regular basis. Approximately > >> one > >>>>>> out of three CI runs fails on Windows. Piotr already improved the > >>>>>> situation extensively, though there are still several leftovers that > >>>>>> need attention. > >>>>>> > >>>>>> Unless somebody steps up to improve the unstable Windows tests, I > >>>>>> would like to disable those only for the WIndows platform. > >>>>> > >>>>> Please don't. Windows has an annoying file locking policy that > >>>>> prevents users from deleting files with open file descriptors, but > >>>>> that is one of the few ways to detect resource leakage we have. > >>>>> > >>>>> Tests running on *NIXes will ignore problems with open file > >>>>> descriptors and delete the log files, but on a production system > those > >>>>> leaks will accumulate and cause application crashes. We had such a > >>>>> leak, when we used `URLConnection#getLastModified` on a `jar:...` > URL. > >>>>> This call caused file descriptor exhaustion on both Windows and > >>>>> *NIXes, but only the Windows test was able to detect it. > >>>>> > >>>>> Piotr, > >>>>> who never thought would ever defend Microsoft Windows. > >>>>> > >>>>> PS: Gary reports the failures, but always runs the build again until > >>>>> it succeeds, even on Friday 13th, when he had to wait until Saturday > >>>>> 14th for the test run to succeed. > >>>> > >> > >
