Also in TserverUtils:270, when the TNonblockingServerSocket is created it looks like it ends up using the default frame size. I am not sure if this is intentional or not.
On Thu, Dec 15, 2022 at 3:26 PM Vincent Russell <vincent.russ...@gmail.com> wrote: > Christopher, > > I am not sure if this issue is related to 3042 or not. > > On the client side it does look like TConfiguration ends up being called > with the default constructor. I am not sure if this is intentional or not. > > On the server side I see this stack, so it also looks like: > > at org.apache.thrift.TConfiguration.<init>(TConfiguration.java:36) > at org.apache.thrift.TConfiguration$Builder.build(TConfiguration.java:99) > at org.apache.thrift.TConfiguration.<clinit>(TConfiguration.java:65) > at > org.apache.thrift.transport.TNonblockingSocket.<init>(TNonblockingSocket.java:74) > at > org.apache.thrift.transport.TNonblockingSocket.<init>(TNonblockingSocket.java:68) > at > org.apache.thrift.transport.TNonblockingServerSocket.accept(TNonblockingServerSocket.java:135) > at > org.apache.thrift.transport.TNonblockingServerSocket.accept(TNonblockingServerSocket.java:36) > at > org.apache.thrift.server.TNonblockingServer$SelectAcceptThread.handleAccept(TNonblockingServer.java:218) > at > org.apache.thrift.server.TNonblockingServer$SelectAcceptThread.select(TNonblockingServer.java:186) > at > org.apache.thrift.server.TNonblockingServer$SelectAcceptThread.run(TNonblockingServer.java:142) > > > I see this in the server log so it does look like it should be using 1G: > 2022-09-01 16:59:41 INFO [org.apache.accumulo.tserver.TabletServer] > ServerUtil:124 - general.server.message.size.max = 1G > > Thanks, > Vincent > > On Thu, Dec 15, 2022 at 12:26 PM Vincent Russell < > vincent.russ...@gmail.com> wrote: > >> >> >> I had to make a stack trace with hacking together a remote debug instance: >> >> at >> org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.read(AbstractNonblockingServer.java:334) >> at >> org.apache.accumulo.server.rpc.CustomNonBlockingServer$CustomFrameBuffer.read(CustomNonBlockingServer.java:134) >> at >> org.apache.thrift.server.AbstractNonblockingServer$AbstractSelectThread.handleRead(AbstractNonblockingServer.java:187) >> at >> org.apache.thrift.server.TNonblockingServer$SelectAcceptThread.select(TNonblockingServer.java:189) >> at >> org.apache.thrift.server.TNonblockingServer$SelectAcceptThread.run(TNonblockingServer.java:142) >> >> On Thu, Dec 15, 2022 at 12:52 AM Christopher <ctubb...@apache.org> wrote: >> >>> From the numbers in the message, it looks like you're sending an 18MB >>> payload but something in Thrift is limiting things to 16384000 >>> (16000KB). I doubt you've overridden the default >>> general.server.message.size.max to be anything that low (the default >>> is 1G). Unless you're flushing after every mutation, it would not be >>> surprising to exceed the 16MB max frame size indicated in the error >>> message quite quickly. >>> >>> This value of 16384000 seemed weird. It looks like it's not using our >>> configuration, but using the built-in default value of >>> org.apache.thrift.TConfiguration.DEFAULT_MAX_FRAME_SIZE. It looks like >>> this can happen whenever `new TConfiguration()` is called without >>> parameters... and there's a fair amount of internal code, mostly in >>> libthrift itself, that does that. It's a bit tricky to track down the >>> one causing this particular issue. If you have a full stack trace, it >>> could help. >>> >>> Also, this might be the same issue seen reported in >>> https://github.com/apache/accumulo/issues/3042 >>> >>> On Wed, Dec 14, 2022 at 8:53 PM Vincent Russell >>> <vincent.russ...@gmail.com> wrote: >>> > >>> > I was able to work out all of my compilation issues; however when I >>> run an >>> > integration test with the Mini Accumulo Cluster that tests writing >>> > mutations with values of 5mb the flush hangs forever >>> > and I see the following logs in the TabletServer logs: >>> > >>> > 20:41:02.306 [Thread-7] ERROR >>> > o.a.a.s.r.CustomNonBlockingServer$CustomFrameBuffer - Read a frame >>> size of >>> > 18874697, which is bigger than the maximum allowable frame size >>> 16384000 >>> > for ALL connections. >>> > 20:41:03.582 [Thread-7] ERROR >>> > o.a.a.s.r.CustomNonBlockingServer$CustomFrameBuffer - Read a frame >>> size of >>> > 18874697, which is bigger than the maximum allowable frame size >>> 16384000 >>> > for ALL connections. >>> > 20:41:05.079 [Thread-7] ERROR >>> > o.a.a.s.r.CustomNonBlockingServer$CustomFrameBuffer - Read a frame >>> size of >>> > 18874697, which is bigger than the maximum allowable frame size >>> 16384000 >>> > for ALL connections. >>> > >>> > Other tests that write smaller amounts of data appear to work fine. >>> > >>> > Any idea what the issue might be? >>> > >>> > Thank you, >>> > Vincent >>> > >>> > On Tue, Dec 13, 2022 at 4:43 PM Vincent Russell < >>> vincent.russ...@gmail.com> >>> > wrote: >>> > >>> > > Thank you both for your responses. >>> > > >>> > > We are using an event store library from a sister project that was >>> written >>> > > for accumulo 1.10., which I have already upgraded to 2.0. >>> > > >>> > > I'll spend some time investigating how bad the usage of the internal >>> > > packages are and get back to you. >>> > > >>> > > Thanks again, >>> > > >>> > > On Tue, Dec 13, 2022 at 3:20 PM Christopher <ctubb...@apache.org> >>> wrote: >>> > > >>> > >> To add to Dave's answer, the public API is defined at >>> > >> https://accumulo.apache.org/api/ >>> > >> Anything else is not public and is subject to change without notice >>> on >>> > >> any release without any attempt to retain compatibility. >>> > >> >>> > >> On Tue, Dec 13, 2022 at 3:10 PM Dave Marion <dmario...@gmail.com> >>> wrote: >>> > >> > >>> > >> > There is no guide. You are using implementation classes (see >>> clientImpl >>> > >> in >>> > >> > the package name) vs. using the client api. If you can use the >>> client >>> > >> api >>> > >> > directly, then this should insulate you from changes in the future >>> > >> (except >>> > >> > during major versions). We can try and find where things might >>> have >>> > >> moved, >>> > >> > but a class may have been split into multiple pieces. If you could >>> > >> provide >>> > >> > class and method, that would be easier. >>> > >> > >>> > >> > On Tue, Dec 13, 2022 at 2:45 PM Vincent Russell < >>> > >> vincent.russ...@gmail.com> >>> > >> > wrote: >>> > >> > >>> > >> > > Is there a guide that shows where classes may have been moved >>> with >>> > >> moving >>> > >> > > from 2.0 to 2.1? For instance, I am having issues compiling, >>> because >>> > >> the >>> > >> > > following class doesn't exist: >>> > >> > > import org.apache.accumulo.core.clientImpl.Tables; >>> > >> > > >>> > >> > > I'm just getting started so I'm sure there are others. >>> > >> > > >>> > >> > > Thanks, >>> > >> > > Vincent >>> > >> > > >>> > >> > > >>> > >> > > >>> > >> > > On Fri, Dec 9, 2022 at 9:02 AM Vincent Russell < >>> > >> vincent.russ...@gmail.com> >>> > >> > > wrote: >>> > >> > > >>> > >> > > > I mean Christopher. >>> > >> > > > >>> > >> > > > Thanks again. >>> > >> > > > >>> > >> > > > On Fri, Dec 9, 2022 at 9:01 AM Vincent Russell < >>> > >> > > vincent.russ...@gmail.com> >>> > >> > > > wrote: >>> > >> > > > >>> > >> > > >> Thank you Chris. >>> > >> > > >> >>> > >> > > >> Will will upgrade to Accumulo 2.1 and ZooKeeper 3.7 or >>> later as >>> > >> soon as >>> > >> > > >> possible. >>> > >> > > >> >>> > >> > > >> On Thu, Dec 8, 2022 at 8:44 PM Christopher < >>> ctubb...@apache.org> >>> > >> wrote: >>> > >> > > >> >>> > >> > > >>> Hi Vincent, >>> > >> > > >>> >>> > >> > > >>> Version 2.0.1 is end of life as of the 2.1.0 LTM release, >>> and 2.0 >>> > >> is >>> > >> > > >>> not expected to receive any further updates. Version 2.1.0 >>> may >>> > >> work >>> > >> > > >>> with ZooKeeper 3.4, but was developed and tested against >>> 3.5 and >>> > >> later >>> > >> > > >>> versions. I believe the ZooKeeper community is currently >>> > >> considering >>> > >> > > >>> whether to make 3.6 end-of-life themselves, so I would >>> recommend >>> > >> using >>> > >> > > >>> Accumulo 2.1.0 with the latest ZooKeeper 3.7 or later to >>> have the >>> > >> best >>> > >> > > >>> chance of any kind of support, including JDK 17 support. >>> > >> > > >>> >>> > >> > > >>> As for your specific issues: >>> > >> > > >>> >>> > >> > > >>> 1. This is already fixed in 2.1.0 >>> > >> > > >>> 2/3. These issues are likely fixed in newer ZooKeeper >>> versions. I >>> > >> > > >>> haven't seen them anytime recently, anyway. Bugs in >>> ZooKeeper >>> > >> itself >>> > >> > > >>> are out of scope for the Accumulo developers, but I have >>> tried >>> > >> > > >>> building Accumulo 2.1.0 with JDK 17 and ZooKeeper 3.8.0 and >>> > >> haven't >>> > >> > > >>> observed any unresolved issues. However, it's difficult to >>> > >> actually >>> > >> > > >>> run it because I don't think Hadoop has good JDK 17 support >>> yet. >>> > >> So, >>> > >> > > >>> MiniAccumuloCluster seems to work with JDK 17, as does >>> Accumulo >>> > >> and ZK >>> > >> > > >>> 3.8, but I don't think a full Hadoop cluster would (yet). >>> > >> > > >>> >>> > >> > > >>> On Thu, Dec 8, 2022 at 12:28 PM Vincent Russell >>> > >> > > >>> <vincent.russ...@gmail.com> wrote: >>> > >> > > >>> > >>> > >> > > >>> > Hello, >>> > >> > > >>> > >>> > >> > > >>> > We are currently using accumulo 2.0.1. >>> > >> > > >>> > >>> > >> > > >>> > We are in the process of upgrading our source code to use >>> jdk 17 >>> > >> > > >>> however we >>> > >> > > >>> > are running into some problems with our tests and the >>> > >> > > >>> MiniAccumuloCluster. >>> > >> > > >>> > >>> > >> > > >>> > One of our developer encountered the following issues: >>> > >> > > >>> > >>> > >> > > >>> > 1. The MiniAccumumluoClusterImpl._exec is hardcoded >>> with the >>> > >> JVM >>> > >> > > arg >>> > >> > > >>> > -XX:+IUseConcMarkSweepGC, which is no longer tolerated >>> with >>> > >> JDK17. >>> > >> > > >>> > 2. In Zookeeper 3.4.14, ConetStringParser uses >>> > >> createUnresolved to >>> > >> > > >>> > make IPAddresses. >>> > >> > > >>> SaslServerPrincipal.WrapperInetSocketAddress.getAddress >>> > >> > > >>> > uses InetSocketAddess.getAddress, which returns null >>> because >>> > >> it's >>> > >> > > >>> not >>> > >> > > >>> > resolved, resulting in a failure to connect to the >>> > >> newly-started >>> > >> > > >>> zookeeper. >>> > >> > > >>> > 3. StaticHostProvider.getHostString() tries to extract >>> he >>> > >> hostname >>> > >> > > >>> by >>> > >> > > >>> > calling toString on the address and taking everything >>> before >>> > >> the >>> > >> > > >>> colon, but >>> > >> > > >>> > in JDK17, the string format changed to >>> > >> > > "localhost/<unresolved->:xx" >>> > >> > > >>> (where >>> > >> > > >>> > XX is still the port number). That's incorrect and it >>> can't >>> > >> > > >>> resolve the >>> > >> > > >>> > names. >>> > >> > > >>> > >>> > >> > > >>> > >>> > >> > > >>> > Has anyone come across/resolved these kinds of issues? >>> Is it >>> > >> not >>> > >> > > >>> possible >>> > >> > > >>> > to use java17 from a client perspective? Will upgrading >>> to >>> > >> accumulo >>> > >> > > >>> 2.1 >>> > >> > > >>> > help? >>> > >> > > >>> > >>> > >> > > >>> > Thanks, >>> > >> > > >>> > Vincent >>> > >> > > >>> >>> > >> > > >> >>> > >> > > >>> > >> >>> > > >>> >>
