On Sun, Feb 1, 2009 at 12:26 PM, Mateusz Berezecki <[email protected]> wrote: > Hi Luke, > > On Sun, Feb 1, 2009 at 4:05 AM, Luke <[email protected]> wrote: >> >> On Jan 31, 5:23 pm, Mateusz Berezecki <[email protected]> wrote: >>> On Sun, Feb 1, 2009 at 2:13 AM, Luke <[email protected]> wrote: >>> >>> > This should only happen when an old client is talking to a new server >>> > or vice versa. Make sure you kill all existing servers and restart the >>> > servers. >>> >>> Is this is the thrift broker you're mentioning? >>> Could you please be more verbose ? >>> I'm not sure I understand. >> >> Range servers talk to DFS brokers via dfs client. ThriftBroker talks >> to range servers/hyperspace/master via the hypertable client. You must >> make sure all the servers (DFS broker, hyperspace, range server, >> master and thrift broker) and hypertable client (native C++ client) >> are from the same build. >> >> It's OK for thrift client to be older (that's the advantage of the >> thrift interface) > > > I'm now 100% sure that I have everything from the same build compiled > from the newest source > and I've started diagnosing the problem. I still did not figure out > the root cause but here's a > little info on what I am seeing. > > I've added few debugging statements around the code and when e.g. > cleaning the database I see this > > ./clean-database.sh > 1233487312 INFO dfsclient : > (/home/mateusz/work/hypertable/src/cc/AsyncComm/CommHeader.cc:51) > Header size according to HT: 36 > 1233487312 INFO dfsclient : > (/home/mateusz/work/hypertable/src/cc/AsyncComm/CommHeader.cc:51) > Header size according to HT: 36 > Removed /hypertable/servers in DFS > Removed /hypertable/tables in DFS > Cleared hyperspace > Killing DfsBroker.local.pid 30270 > Shutdown hyperspace complete > Shutdown range server complete > Shutdown hypertable master complete > Shutdown thrift broker complete > Shutdown DFS broker complete > > But when I try running my thrift script in DfsBroker.local.log I see this > > 1233487321 INFO localBroker : > (/home/mateusz/work/hypertable/src/cc/AsyncComm/IOHandler.h:88) Event: > type=CONNECTION_ESTABLISHED from=127.0.0.1:38852 > 1233487321 INFO localBroker : > (/home/mateusz/work/hypertable/src/cc/AsyncComm/CommHeader.cc:51) > Header size according to HT: 36 > 1233487326 INFO localBroker : > (/home/mateusz/work/hypertable/src/cc/AsyncComm/IOHandler.h:88) Event: > type=CONNECTION_ESTABLISHED from=127.0.0.1:38857 > 1233487329 INFO localBroker : > (/home/mateusz/work/hypertable/src/cc/AsyncComm/IOHandler.h:88) Event: > type=CONNECTION_ESTABLISHED from=127.0.0.1:38863 > 1233487329 INFO localBroker : > (/home/mateusz/work/hypertable/src/cc/AsyncComm/CommHeader.cc:51) > Header size according to HT: 36 > 1233487329 INFO localBroker : > (/home/mateusz/work/hypertable/src/cc/AsyncComm/CommHeader.cc:51) > Header size according to HT: 36 > 1233487329 INFO localBroker : > (/home/mateusz/work/hypertable/src/cc/AsyncComm/CommHeader.cc:51) > Header size according to HT: 36 > 1233487329 INFO localBroker : > (/home/mateusz/work/hypertable/src/cc/AsyncComm/CommHeader.cc:51) > Header size according to HT: 36 > 1233487329 INFO localBroker : > (/home/mateusz/work/hypertable/src/cc/AsyncComm/CommHeader.cc:51) > Header size according to HT: 36 > 1233487329 INFO localBroker : > (/home/mateusz/work/hypertable/src/cc/AsyncComm/CommHeader.cc:51) > Header size according to HT: 36 > 1233487329 INFO localBroker : > (/home/mateusz/work/hypertable/src/cc/AsyncComm/CommHeader.cc:51) > Header size according to HT: 36 > 1233487329 INFO localBroker : > (/home/mateusz/work/hypertable/src/cc/AsyncComm/CommHeader.cc:51) > Header size according to HT: 36 > 1233487329 INFO localBroker : > (/home/mateusz/work/hypertable/src/cc/AsyncComm/CommHeader.cc:51) > Header size according to HT: 36 > 1233487329 INFO localBroker : > (/home/mateusz/work/hypertable/src/cc/AsyncComm/CommHeader.cc:51) > Header size according to HT: 36 > 1233487329 INFO localBroker : > (/home/mateusz/work/hypertable/src/cc/AsyncComm/CommHeader.cc:51) > Header size according to HT: 36 > 1233487329 INFO localBroker : > (/home/mateusz/work/hypertable/src/cc/AsyncComm/CommHeader.cc:51) > Header size according to HT: 36 > 1233487329 INFO localBroker : > (/home/mateusz/work/hypertable/src/cc/AsyncComm/CommHeader.cc:51) > Header size according to HT: 36 > 1233487329 INFO localBroker : > (/home/mateusz/work/hypertable/src/cc/AsyncComm/CommHeader.cc:51) > Header size according to HT: 36 > 1233487329 INFO localBroker : > (/home/mateusz/work/hypertable/src/cc/AsyncComm/CommHeader.cc:51) > Header size according to HT: 36 > 1233487329 INFO localBroker : > (/home/mateusz/work/hypertable/src/cc/AsyncComm/CommHeader.cc:51) > Header size according to HT: 36 > 1233487329 INFO localBroker : > (/home/mateusz/work/hypertable/src/cc/AsyncComm/CommHeader.cc:51) > Header size according to HT: 36 > 1233487329 INFO localBroker : > (/home/mateusz/work/hypertable/src/cc/AsyncComm/CommHeader.cc:51) > Header size according to HT: 36 > 1233487329 INFO localBroker : > (/home/mateusz/work/hypertable/src/cc/AsyncComm/CommHeader.cc:51) > Header size according to HT: 36 > 1233487329 INFO localBroker : > (/home/mateusz/work/hypertable/src/cc/AsyncComm/CommHeader.cc:51) > Header size according to HT: 36 > 1233487329 INFO localBroker : > (/home/mateusz/work/hypertable/src/cc/AsyncComm/CommHeader.cc:51) > Header size according to HT: 36 > 1233487329 INFO localBroker : > (/home/mateusz/work/hypertable/src/cc/AsyncComm/CommHeader.cc:51) > Header size according to HT: 36 > 1233487329 INFO localBroker : > (/home/mateusz/work/hypertable/src/cc/AsyncComm/CommHeader.cc:51) > Header size according to HT: 36 > 1233487329 INFO localBroker : > (/home/mateusz/work/hypertable/src/cc/AsyncComm/CommHeader.cc:51) > Header size according to HT: 36 > 1233487329 INFO localBroker : > (/home/mateusz/work/hypertable/src/cc/AsyncComm/CommHeader.cc:51) > Header size according to HT: 36 > 1233487329 INFO localBroker : > (/home/mateusz/work/hypertable/src/cc/AsyncComm/CommHeader.cc:51) > Header size according to HT: 36 > 1233487329 INFO localBroker : > (/home/mateusz/work/hypertable/src/cc/AsyncComm/CommHeader.cc:51) > Header size according to HT: 36 > 1233487329 INFO localBroker : > (/home/mateusz/work/hypertable/src/cc/AsyncComm/CommHeader.cc:51) > Header size according to HT: 36 > 1233487329 INFO localBroker : > (/home/mateusz/work/hypertable/src/cc/AsyncComm/CommHeader.cc:51) > Header size according to HT: 36 > 1233487329 INFO localBroker : > (/home/mateusz/work/hypertable/src/cc/AsyncComm/CommHeader.cc:51) > Header size according to HT: 36 > 1233487346 INFO localBroker : > (/home/mateusz/work/hypertable/src/cc/AsyncComm/IOHandler.h:88) Event: > type=CONNECTION_ESTABLISHED from=127.0.0.1:38972 > 1233487346 INFO localBroker : > (/home/mateusz/work/hypertable/src/cc/AsyncComm/CommHeader.cc:51) > Header size according to HT: 0 > 1233487346 FATAL localBroker : > (/home/mateusz/work/hypertable/src/cc/AsyncComm/CommHeader.cc:52) > failed expectation: *remainp >= FIXED_LENGTH > > I'll spend some time on this and try to find what's wrong.
Problem is now partially solved on my side. It turns out I had a set of scripts copied from a different configuration but the new configuration was using completely different set of ports. There was one port from the new configuration that was overlapping with the port from the old configuration, the one used in a copied script. This resulted in thrift client sending packets directly to DfsBroker, and causing it to crash, instead of sending them to ThriftBroker. I think it would be nice to have a preventive measure for bogus packets being sent to servers. I'll try to come up with a patch for this today if time permits. Mateusz --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Hypertable Development" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en -~----------~----~----~----~------~----~------~--~---
