Great analysis, thank you.
Could the test be better synchronized using a count down latch?
Gary
On Sun, Oct 13, 2024, 1:09 AM Emmanuel Lécharny <elecha...@gmail.com> wrote:
> AFAIU, when you fetch an ephemeral port with a new ServerSocket(0), what
> happens is that the address:port is in TIME_WAIT after the socket
> closure, but as it's also using SO_REUSEADDR, the process can use it,
> which guarantees it's available, and no other process can use it.
>
> You just need to be sure no other test is going to call the
> AvailablePortFinder.getNextAvailable() method.
>
> So I retract what I said about the lock being caused by the wrong port
> being used.
>
>
> I analysed further the test and the Mina code, and I found there is a
> race condition when using the read() command:
>
> - On one thread, we do write a message when the session is opened
> - On the other thread we set the session configuration UseReadOperation
> flag to true, then do a read.
>
>
> What happens - and I can reproduce the lock in debugging mode easily -
> is that the read is executed too late, the message being already
> propagated *before* the check for the UseReadOperation flag is switched
> to 'true', so when read() is called, there is nothing to be read.
>
> I can also reproduce the lock 100% by adding a Thread.sleep(10) before
> setting the session configuration to accept read operation:
>
> try {
> IoConnector connector = new NioSocketConnector();
> connector.setHandler(new IoHandlerAdapter());
> ConnectFuture connectFuture = connector.connect(new
> InetSocketAddress("localhost", port));
> connectFuture.awaitUninterruptibly();
>
> if (connectFuture.getException() != null) {
> throw connectFuture.getException();
> }
>
> Thread.sleep( 10 );
>
> connectFuture.getSession().getConfig().setUseReadOperation(true);
> ...
>
>
> To get it working with the sleep(10), it's enough to add a
> Thread.sleep(11) in the sessionOpened handler method:
>
> @Override
> public void sessionOpened(IoSession session) throws Exception
> {
> Thread.sleep( 11 );
> ...
>
> Bottom line, the test is flaky. We should never expect a read to succeed
> when the data being written are pushed by the sessionConnected()
> handler's method.
> There is no way we can get this fixed in MINA core code, because the
> session only exists when it's opened, so too late for a configuration to
> be injected in time - unless we are lucky - for it to be taken in
> consideration when trying to do a read() on the client side.
>
> In other words, this feature is badly designed. We should either
> configure it when we create the server, so that it's available when the
> session is created, or expect the user not to write anything in the
> sessionOpened/sessionCreated handler's methods and assume that it will
> be available on the client immediately...
>
> I'll create a ticket for that issue. In the mean time, I don't think
> it's a blocker for a release, because it's also present in every version
> of MINA :/
>
>
> On 12/10/2024 22:36, Gary Gregory wrote:
> > What I've done elsewhere (mostly at work) is to configure the main code
> > (like a server) to accept port 0 from a test, instead of a "free"
> > discovered port. This port ends up being used by a ServerSocket
> > constructor or equivalent SSL class. The main class under test needs to
> > have a getPort() method or methods depending on how deeply buried the
> > server socket is, that queries the allocated ServerSocket. The test can
> > then query the main code after it knows the socket is bound.
> >
> > HTH,
> > Gary
> >
> > On Sat, Oct 12, 2024, 3:16 PM Guillaume Nodet <gno...@apache.org
> > <mailto:gno...@apache.org>> wrote:
> >
> >
> >
> > Le sam. 12 oct. 2024 à 06:24, Emmanuel Lécharny <elecha...@gmail.com
> > <mailto:elecha...@gmail.com>> a écrit :
> >
> > Indeed. But the point is that you can't fetch an available port
> > and use
> > it immediately and be sure it's not used by another process
> > without some
> > kind of global lock on your system...
> >
> > I don't see any other way to grab an available port from the OS,
> > and
> > guarantee it's not used when you have to bind a socket on it and
> > release
> > it immediately for use by another socket.
> >
> >
> > If you bind a server socket on port 0, the OS will allocate a free
> > port. So just need to get the bound port and configure the client to
> > use it ?
> >
> > Back to the issue: can you grab a StackTrace to see exactly
> > where it
> > blocks, in case there is another issue?
> >
> > Many thanks!
> >
> > On 11/10/2024 13:59, Gary Gregory wrote:
> > > But it's not really doing that because it's NOT using the
> > allocated
> > > ServerSocket. Once that method exists, any other process in
> the
> > > underlying OS can grab that port for itself.
> > >
> > > Gary
> > >
> > > On Fri, Oct 11, 2024, 7:53 AM Emmanuel Lécharny
> > <elecha...@gmail.com <mailto:elecha...@gmail.com>
> > > <mailto:elecha...@gmail.com <mailto:elecha...@gmail.com>>>
> wrote:
> > >
> > >
> > >
> > > On 11/10/2024 10:01, Gary Gregory wrote:
> > > > Hi Emanuel,
> > > >
> > > > Why not use ephemeral ports?
> > >
> > > this is exactly what the
> > AvailablePortFinder.getNextAvailable()
> > > method does:
> > >
> > > public static int getNextAvailable() {
> > > try (ServerSocket serverSocket = new
> > ServerSocket(0)){
> > > // Here, we simply return an available port
> > found by
> > > the system
> > > return serverSocket.getLocalPort();
> > > } catch (IOException ioe) {
> > > throw new
> > NoSuchElementException(ioe.getMessage());
> > > }
> > >
> > >
> > > It might mean a bit of work update the
> > > > tests to query the port and then use it, but it seems
> > less probe to
> > > > problems. I'll try again in a few hours.
> > > >
> > > > Gary
> > > >
> > > > On Thu, Oct 10, 2024, 10:17 PM Emmanuel Lécharny
> > > <elecha...@gmail.com <mailto:elecha...@gmail.com>
> > <mailto:elecha...@gmail.com <mailto:elecha...@gmail.com>>
> > > > <mailto:elecha...@gmail.com
> > <mailto:elecha...@gmail.com> <mailto:elecha...@gmail.com
> > <mailto:elecha...@gmail.com>>>> wrote:
> > > >
> > > > Thanks Gary!
> > > >
> > > > DIRMINA777Test may fell due to some port collision
> > (we use a
> > > > AvailablePortFinder.getNextAvailable() call to
> > find a port
> > > that the
> > > > Acceptor can use, but as some other tests may run
> > > concurrently, the
> > > > same
> > > > port can be in use by another Acceptor, and the
> > readFuture
> > > can wait
> > > > forever because the IoHandler that responds to
> > connection is
> > > not doing
> > > > what is expected.
> > > >
> > > > It's not perfect, and the only solution is to run
> > the test
> > > another time.
> > > >
> > > > Could you give it another try?
> > > >
> > > > Side note: the
> > AvailablePortFinder.getNextAvailable() method,
> > > which is
> > > > only used for tests, should probably be
> > syncrhonized to avoid
> > > being
> > > > called by many threads at the same time.
> > > >
> > > > Thanks!
> > > >
> > > > On 10/10/2024 15:10, Gary Gregory wrote:
> > > > > + 1 Release MINA 2.1.9
> > > > >
> > > > > Release MINA 2.0.26: Running
> > > > > org.apache.mina.transport.socket.nio.x hangs or
> > takes
> > > > > forever, I killed it after 30 minutes.
> > > > >
> > > > > - Tested the src ZIPs
> > > > > - ASCs OK
> > > > > - SHA512 files are not machine verifiable with
> > `shasum
> > > --check`
> > > > due to
> > > > > their incompatible format.
> > > > > - mvn clean verify
> > > > > - Using:
> > > > > openjdk version "11.0.24" 2024-07-16
> > > > > OpenJDK Runtime Environment Homebrew (build
> > 11.0.24+0)
> > > > > OpenJDK 64-Bit Server VM Homebrew (build
> > 11.0.24+0, mixed
> > > mode)
> > > > >
> > > > > Apache Maven 3.9.9
> > (8e8579a9e76f7d015ee5ec7bfcdc97d260186937)
> > > > > Maven home:
> /usr/local/Cellar/maven/3.9.9/libexec
> > > > > Java version: 11.0.24, vendor: Homebrew,
> runtime:
> > > > >
> > > >
> > >
> > /usr/local/Cellar/openjdk@11
> /11.0.24/libexec/openjdk.jdk/Contents/Home
> > > > > Default locale: en_US, platform encoding: UTF-8
> > > > > OS name: "mac os x", version: "15.0.1", arch:
> > "x86_64",
> > > family: "mac"
> > > > >
> > > > > Darwin **** 24.0.0 Darwin Kernel Version
> > 24.0.0: Tue Sep 24
> > > > 23:36:30 PDT
> > > > > 2024; root:xnu-11215.1.12~1/RELEASE_X86_64
> x86_64
> > > > >
> > > > > Gary
> > > > >
> > > > >
> > > > > On Wed, Oct 9, 2024 at 10:53 AM Emmanuel
> Lécharny
> > > > <elecha...@gmail.com <mailto:elecha...@gmail.com>
> > <mailto:elecha...@gmail.com <mailto:elecha...@gmail.com>>
> > > <mailto:elecha...@gmail.com <mailto:elecha...@gmail.com>
> > <mailto:elecha...@gmail.com <mailto:elecha...@gmail.com>>>
> > > > > <mailto:elecha...@gmail.com
> > <mailto:elecha...@gmail.com> <mailto:elecha...@gmail.com
> > <mailto:elecha...@gmail.com>>
> > > <mailto:elecha...@gmail.com <mailto:elecha...@gmail.com>
> > <mailto:elecha...@gmail.com <mailto:elecha...@gmail.com>>>>>
> wrote:
> > > > >
> > > > > Hi!
> > > > >
> > > > > I have started this vote 9 days ago, still
> > haven't get any
> > > > vote yet...
> > > > >
> > > > > On 30/09/2024 19:21, Emmanuel Lecharny
> wrote:
> > > > > > hi!
> > > > > >
> > > > > > WARNING: there are 2 votes to cast!
> > > > > >
> > > > > >
> > > > > > This is a vote for a double release:
> > > > > > * MINA 2.1.9
> > > > > > * MINA 2.0.26
> > > > > >
> > > > > > Those versions are a maintenance
> > realase, fixing a
> > > bug in
> > > > the way we
> > > > > > treat Strings when reading a IoBuffer:
> > > > > >
> > > > > > DIRMINA-1181:Exception thrown when
> > attempting to decode
> > > > certain
> > > > > UTF-16 chars
> > > > > >
> > > > > >
> > > > > >
> > > > > > Temporary tags have been created (they
> > can be
> > > removed if
> > > > the vote
> > > > > is not
> > > > > > approved) :
> > > > > >
> > > > > > * MINA 2.1.9:
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/mina/commit/8df31da1597056b73f5d6dbf11c75ce13227ba60
> <
> https://github.com/apache/mina/commit/8df31da1597056b73f5d6dbf11c75ce13227ba60>
> <
> https://github.com/apache/mina/commit/8df31da1597056b73f5d6dbf11c75ce13227ba60
> <
> https://github.com/apache/mina/commit/8df31da1597056b73f5d6dbf11c75ce13227ba60>>
> <
> https://github.com/apache/mina/commit/8df31da1597056b73f5d6dbf11c75ce13227ba60
> <
> https://github.com/apache/mina/commit/8df31da1597056b73f5d6dbf11c75ce13227ba60>
> <
> https://github.com/apache/mina/commit/8df31da1597056b73f5d6dbf11c75ce13227ba60
> <
> https://github.com/apache/mina/commit/8df31da1597056b73f5d6dbf11c75ce13227ba60>>>
> <
> https://github.com/apache/mina/commit/8df31da1597056b73f5d6dbf11c75ce13227ba60
> <
> https://github.com/apache/mina/commit/8df31da1597056b73f5d6dbf11c75ce13227ba60>
> <
> https://github.com/apache/mina/commit/8df31da1597056b73f5d6dbf11c75ce13227ba60
> <
> https://github.com/apache/mina/commit/8df31da1597056b73f5d6dbf11c75ce13227ba60>>
> <
> https://github.com/apache/mina/commit/8df31da1597056b73f5d6dbf11c75ce13227ba60
> <
> https://github.com/apache/mina/commit/8df31da1597056b73f5d6dbf11c75ce13227ba60>
> <
> https://github.com/apache/mina/commit/8df31da1597056b73f5d6dbf11c75ce13227ba60
> <
> https://github.com/apache/mina/commit/8df31da1597056b73f5d6dbf11c75ce13227ba60
> >>>>
> > > > > >
> > > > > > * MINA 2.0.26:
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/mina/commit/4d1cf35024ae565827b63c11bd0b42a62a1c3e49
> <
> https://github.com/apache/mina/commit/4d1cf35024ae565827b63c11bd0b42a62a1c3e49>
> <
> https://github.com/apache/mina/commit/4d1cf35024ae565827b63c11bd0b42a62a1c3e49
> <
> https://github.com/apache/mina/commit/4d1cf35024ae565827b63c11bd0b42a62a1c3e49>>
> <
> https://github.com/apache/mina/commit/4d1cf35024ae565827b63c11bd0b42a62a1c3e49
> <
> https://github.com/apache/mina/commit/4d1cf35024ae565827b63c11bd0b42a62a1c3e49>
> <
> https://github.com/apache/mina/commit/4d1cf35024ae565827b63c11bd0b42a62a1c3e49
> <
> https://github.com/apache/mina/commit/4d1cf35024ae565827b63c11bd0b42a62a1c3e49>>>
> <
> https://github.com/apache/mina/commit/4d1cf35024ae565827b63c11bd0b42a62a1c3e49
> <
> https://github.com/apache/mina/commit/4d1cf35024ae565827b63c11bd0b42a62a1c3e49>
> <
> https://github.com/apache/mina/commit/4d1cf35024ae565827b63c11bd0b42a62a1c3e49
> <
> https://github.com/apache/mina/commit/4d1cf35024ae565827b63c11bd0b42a62a1c3e49>>
> <
> https://github.com/apache/mina/commit/4d1cf35024ae565827b63c11bd0b42a62a1c3e49
> <
> https://github.com/apache/mina/commit/4d1cf35024ae565827b63c11bd0b42a62a1c3e49>
> <
> https://github.com/apache/mina/commit/4d1cf35024ae565827b63c11bd0b42a62a1c3e49
> <
> https://github.com/apache/mina/commit/4d1cf35024ae565827b63c11bd0b42a62a1c3e49
> >>>>
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > The final artifacts are stored in a
> > staging repository:
> > > > > > * MINA 2.1.9:
> > > > > >
> > > > >
> > > >
> > >
> >
> https://repository.apache.org/content/repositories/orgapachemina-1105 <
> https://repository.apache.org/content/repositories/orgapachemina-1105> <
> https://repository.apache.org/content/repositories/orgapachemina-1105 <
> https://repository.apache.org/content/repositories/orgapachemina-1105>> <
> https://repository.apache.org/content/repositories/orgapachemina-1105 <
> https://repository.apache.org/content/repositories/orgapachemina-1105> <
> https://repository.apache.org/content/repositories/orgapachemina-1105 <
> https://repository.apache.org/content/repositories/orgapachemina-1105>>> <
> https://repository.apache.org/content/repositories/orgapachemina-1105 <
> https://repository.apache.org/content/repositories/orgapachemina-1105> <
> https://repository.apache.org/content/repositories/orgapachemina-1105 <
> https://repository.apache.org/content/repositories/orgapachemina-1105>> <
> https://repository.apache.org/content/repositories/orgapachemina-1105 <
> https://repository.apache.org/content/repositories/orgapachemina-1105> <
> https://repository.apache.org/content/repositories/orgapachemina-1105 <
> https://repository.apache.org/content/repositories/orgapachemina-1105>>>>
> > > > > > * MINA 2.0.26:
> > > > > >
> > > > >
> > > >
> > >
> >
> https://repository.apache.org/content/repositories/orgapachemina-1104 <
> https://repository.apache.org/content/repositories/orgapachemina-1104> <
> https://repository.apache.org/content/repositories/orgapachemina-1104 <
> https://repository.apache.org/content/repositories/orgapachemina-1104>> <
> https://repository.apache.org/content/repositories/orgapachemina-1104 <
> https://repository.apache.org/content/repositories/orgapachemina-1104> <
> https://repository.apache.org/content/repositories/orgapachemina-1104 <
> https://repository.apache.org/content/repositories/orgapachemina-1104>>> <
> https://repository.apache.org/content/repositories/orgapachemina-1104 <
> https://repository.apache.org/content/repositories/orgapachemina-1104> <
> https://repository.apache.org/content/repositories/orgapachemina-1104 <
> https://repository.apache.org/content/repositories/orgapachemina-1104>> <
> https://repository.apache.org/content/repositories/orgapachemina-1104 <
> https://repository.apache.org/content/repositories/orgapachemina-1104> <
> https://repository.apache.org/content/repositories/orgapachemina-1104 <
> https://repository.apache.org/content/repositories/orgapachemina-1104>>>>
> > > > > >
> > > > > >
> > > > > >
> > > > > > The distributions are available for
> > download on :
> > > > > > * MINA 2.1.9:
> > > > >
> > https://dist.apache.org/repos/dist/dev/mina/mina/2.1.9
> > <https://dist.apache.org/repos/dist/dev/mina/mina/2.1.9>
> > > <https://dist.apache.org/repos/dist/dev/mina/mina/2.1.9
> > <https://dist.apache.org/repos/dist/dev/mina/mina/2.1.9>>
> > > >
> > <https://dist.apache.org/repos/dist/dev/mina/mina/2.1.9
> > <https://dist.apache.org/repos/dist/dev/mina/mina/2.1.9>
> > > <https://dist.apache.org/repos/dist/dev/mina/mina/2.1.9
> > <https://dist.apache.org/repos/dist/dev/mina/mina/2.1.9>>>
> > > > >
> > > <https://dist.apache.org/repos/dist/dev/mina/mina/2.1.9
> > <https://dist.apache.org/repos/dist/dev/mina/mina/2.1.9>
> > > <https://dist.apache.org/repos/dist/dev/mina/mina/2.1.9
> > <https://dist.apache.org/repos/dist/dev/mina/mina/2.1.9>>
> > > >
> > <https://dist.apache.org/repos/dist/dev/mina/mina/2.1.9
> > <https://dist.apache.org/repos/dist/dev/mina/mina/2.1.9>
> > > <https://dist.apache.org/repos/dist/dev/mina/mina/2.1.9
> > <https://dist.apache.org/repos/dist/dev/mina/mina/2.1.9>>>>
> > > > > > * MINA 2.0.26:
> > > > >
> > https://dist.apache.org/repos/dist/dev/mina/mina/2.0.26
> > <https://dist.apache.org/repos/dist/dev/mina/mina/2.0.26>
> > > <https://dist.apache.org/repos/dist/dev/mina/mina/2.0.26
> > <https://dist.apache.org/repos/dist/dev/mina/mina/2.0.26>>
> > > >
> > <https://dist.apache.org/repos/dist/dev/mina/mina/2.0.26
> > <https://dist.apache.org/repos/dist/dev/mina/mina/2.0.26>
> > > <https://dist.apache.org/repos/dist/dev/mina/mina/2.0.26
> > <https://dist.apache.org/repos/dist/dev/mina/mina/2.0.26>>>
> > > > >
> > >
> > <https://dist.apache.org/repos/dist/dev/mina/mina/2.0.26
> > <https://dist.apache.org/repos/dist/dev/mina/mina/2.0.26>
> > > <https://dist.apache.org/repos/dist/dev/mina/mina/2.0.26
> > <https://dist.apache.org/repos/dist/dev/mina/mina/2.0.26>>
> > > >
> > <https://dist.apache.org/repos/dist/dev/mina/mina/2.0.26
> > <https://dist.apache.org/repos/dist/dev/mina/mina/2.0.26>
> > > <https://dist.apache.org/repos/dist/dev/mina/mina/2.0.26
> > <https://dist.apache.org/repos/dist/dev/mina/mina/2.0.26>>>>
> > > > > >
> > > > > >
> > > > > > Let us vote :
> > > > > > [ ] +1 | Release MINA 2.1.9
> > > > > > [ ] ± | Abstain
> > > > > > [ ] -1 | Do *NOT* release MINA 2.1.9
> > > > > >
> > > > > >
> > > > > > [ ] +1 | Release MINA 2.0.26
> > > > > > [ ] ± | Abstain
> > > > > > [ ] -1 | Do *NOT* release MINA 2.0.26
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > *Emmanuel Lécharny* P. +33 (0)6 08 33 32 61
> > > > > elecha...@apache.org
> > <mailto:elecha...@apache.org> <mailto:elecha...@apache.org
> > <mailto:elecha...@apache.org>>
> > > <mailto:elecha...@apache.org
> > <mailto:elecha...@apache.org> <mailto:elecha...@apache.org
> > <mailto:elecha...@apache.org>>>
> > > > <mailto:elecha...@apache.org
> > <mailto:elecha...@apache.org> <mailto:elecha...@apache.org
> > <mailto:elecha...@apache.org>>
> > > <mailto:elecha...@apache.org
> > <mailto:elecha...@apache.org> <mailto:elecha...@apache.org
> > <mailto:elecha...@apache.org>>>>
> > > > >
> > > > >
> > > >
> > >
> >
> ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail:
> > > dev-unsubscr...@mina.apache.org
> > <mailto:dev-unsubscr...@mina.apache.org>
> > <mailto:dev-unsubscr...@mina.apache.org
> > <mailto:dev-unsubscr...@mina.apache.org>>
> > > > <mailto:dev-unsubscr...@mina.apache.org
> > <mailto:dev-unsubscr...@mina.apache.org>
> > > <mailto:dev-unsubscr...@mina.apache.org
> > <mailto:dev-unsubscr...@mina.apache.org>>>
> > > > > <mailto:dev-unsubscr...@mina.apache.org
> > <mailto:dev-unsubscr...@mina.apache.org>
> > > <mailto:dev-unsubscr...@mina.apache.org
> > <mailto:dev-unsubscr...@mina.apache.org>>
> > > > <mailto:dev-unsubscr...@mina.apache.org
> > <mailto:dev-unsubscr...@mina.apache.org>
> > > <mailto:dev-unsubscr...@mina.apache.org
> > <mailto:dev-unsubscr...@mina.apache.org>>>>
> > > > > For additional commands, e-mail:
> > > dev-h...@mina.apache.org <mailto:dev-h...@mina.apache.org>
> > <mailto:dev-h...@mina.apache.org <mailto:
> dev-h...@mina.apache.org>>
> > > > <mailto:dev-h...@mina.apache.org
> > <mailto:dev-h...@mina.apache.org>
> > > <mailto:dev-h...@mina.apache.org
> > <mailto:dev-h...@mina.apache.org>>>
> > > > > <mailto:dev-h...@mina.apache.org
> > <mailto:dev-h...@mina.apache.org>
> > > <mailto:dev-h...@mina.apache.org
> > <mailto:dev-h...@mina.apache.org>>
> > > > <mailto:dev-h...@mina.apache.org
> > <mailto:dev-h...@mina.apache.org>
> > > <mailto:dev-h...@mina.apache.org
> > <mailto:dev-h...@mina.apache.org>>>>
> > > > >
> > > >
> > > > --
> > > > *Emmanuel Lécharny* P. +33 (0)6 08 33 32 61
> > > > elecha...@apache.org <mailto:elecha...@apache.org>
> > <mailto:elecha...@apache.org <mailto:elecha...@apache.org>>
> > > <mailto:elecha...@apache.org
> > <mailto:elecha...@apache.org> <mailto:elecha...@apache.org
> > <mailto:elecha...@apache.org>>>
> > > >
> > >
> > > --
> > > *Emmanuel Lécharny* P. +33 (0)6 08 33 32 61
> > > elecha...@apache.org <mailto:elecha...@apache.org>
> > <mailto:elecha...@apache.org <mailto:elecha...@apache.org>>
> > >
> >
> > --
> > *Emmanuel Lécharny* P. +33 (0)6 08 33 32 61
> > elecha...@apache.org <mailto:elecha...@apache.org>
> >
> >
> ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscr...@mina.apache.org
> > <mailto:dev-unsubscr...@mina.apache.org>
> > For additional commands, e-mail: dev-h...@mina.apache.org
> > <mailto:dev-h...@mina.apache.org>
> >
> >
> >
> > --
> > ------------------------
> > Guillaume Nodet
> >
>
> --
> *Emmanuel Lécharny* P. +33 (0)6 08 33 32 61
> elecha...@apache.org
>
