#general
@iamtherealdarknight: @iamtherealdarknight has joined the channel
#random
@iamtherealdarknight: @iamtherealdarknight has joined the channel
#troubleshooting
@ysuo: Hi, what’s the common reason for errorCode 235? [ { “message”: “ServerSegmentMissing:\n17 segments [table_name__1__22__20220527T1518Z, table_name__1__10__20220516T1300Z, table_name__1__14__20220517T1900Z, missing on server: Server_pinot-server-28.pinot-server-headless.pinot.svc.cluster.local_8098", “errorCode”: 235 } ]
@mayanks: Check ideal-state and external-view for the table for these segments. This error message means that the server did not find these segments.
@ysuo: I checked the external-view, it said the segment is on another server instance. The table status shows good.
@mayanks: Hmm, that means the broker routing table is not updated (that shouldn’t happen). Can you rebuild broker resource in swagger, that should update the routing table.
@ysuo: You mean this one?
@mayanks: yes
@ysuo: Thanks. I’ll have a try.
@ysuo: ```{ "status": "Broker resource is not rebuilt because ideal state is the same for table: table_test_REALTIME" }```
@mayanks: I think there is api on broker to print the routing table, you can check that to see why the broker thinks these segments are on this server
@mayanks: `/debug/routingTable/{tableName}`
@mayanks: Also, are you sure the IS/EV say these segments are in another sever?
@ysuo: yes, ```, "table_test__1__585__20220605T0324Z": { "Server_pinot-server-27.pinot-server-headless.pinot.svc.cluster.local_8098": "ONLINE" }, "table_test__1__586__20220605T0354Z": { "Server_pinot-server-27.pinot-server-headless.pinot.svc.cluster.local_8098": "CONSUMING" },```
@ysuo: it’s on server 27. But query shows segment not on server 28.
@mayanks: ok, then check the broker routing table
@ysuo: There is only /debug/tables/{tableName}
@ysuo: I can’t find /debug/routingTable/{tableName}
@mayanks:
@mayanks: It is on broker
@ysuo: It said this segment is on server 28.
@ysuo: Is there anything I can do to solve this error?
@mayanks: Hmm, this is not expected. What operations were done on the cluster? For now, restarting the broker should fix this. But. I haven’t really seen this behavior in the past. cc: @jackie.jxt
@ysuo: I think I didn’t do any operation that may cause this problem.:sweat_smile:
@ysuo: edit server tag?
@mayanks: Ah, did you rebalance after that?
@ysuo: didn’t update any table config to use that new tag, so not rebalance operation.
@ysuo: restart broker didn’t help
@mayanks: Please run rebalance
@ysuo: rebalanced broker and server
@ysuo: same error there.
@mayanks: This does not make sense
@mayanks: Broker routing table is built from EV. If EV is correct I don’t understand why wrong server shows up in routing table
@mayanks: Did you confirm that routing table shows incorrect server
@ysuo: Three segments were missing from server 28. After restart broker, rebalance broker and servers, there is only one segment has this error. It’s a consuming status segment.
@mayanks: When you did rebalance, did you including consuming segment?
@mayanks: If not, do that.
@ysuo: just after you told me to run rebalance. I did check consuming.
@ysuo:
@ysuo: Should reassign instance be checked?
@mayanks: yeah try that
@ysuo: It seems a new consuming status segment has this issue on server 28.
@mayanks: Try `reset` segment api
@ysuo: I tried reset. reset segment success returned. Then I query this table, no error. But run the same query again, same error occurred.
@ysuo: :sweat:
@mayanks: After reset segment is the consuming segment CONSUMING or ERROR in EV?
@mayanks: And after reset + rebalance is it on server 27 or 28? And the new error is complaning about the same consuming segment to be on incorrect server?
@ysuo: Hi, i just found server-27 and server-28 have the same hostName.
@ysuo: I really don’t know how is that happening.
@mayanks: I am confused, if they have 27 and 28 in the name, how is the host name same?
@mayanks: But sounds like that is likely the issue, Your hostnames are messed up, but that is outside of Pinot
@ysuo: restarted server 27. I’ll check if there’s still trouble.
@ysuo: I’m confused too.
@ysuo: Hi, it’s ok now. Thank you so much.
@mayanks: Ok so restart or sever fixed it?
@ysuo: restarted server 27 which has a wrong hostName.
@ysuo: It’s strange. Edit server tag could cause this problem?
@mayanks: I’ll need to know the exact events, but seems like it
@ysuo: :sweat_smile:
@iamtherealdarknight: @iamtherealdarknight has joined the channel
#getting-started
@iamtherealdarknight: @iamtherealdarknight has joined the channel
#introductions
@iamtherealdarknight: @iamtherealdarknight has joined the channel
--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@pinot.apache.org For additional commands, e-mail: dev-h...@pinot.apache.org
