Hi, Iam trying to do some network control on the storage nodes. For this, I need to know the ports opened for communication by each OSD processes.
I got to know from the link http://ceph.com/docs/master/rados/configuration/network-config-ref/ , that each OSD process requires 3 ports and they from port 6800 it is reserved for OSD processes. However, when I do a ceph osd dump command, it lists 4 ports in use for each of the OSDs: *root@cephnode2:~# ceph osd dump | grep osd* *max_osd 4* *osd.0 up in weight 1 up_from 71 up_thru 71 down_at 68 last_clean_interval [4,70) 10.223.169.166:6800/83380 <http://10.223.169.166:6800/83380> 10.223.169.166:6810/1083380 <http://10.223.169.166:6810/1083380> 10.223.169.166:6811/1083380 <http://10.223.169.166:6811/1083380> 10.223.169.166:6812/1083380 <http://10.223.169.166:6812/1083380> exists,up fdbbc6eb-7d9f-4ad8-a8c3-caf995422528* *osd.1 up in weight 1 up_from 7 up_thru 71 down_at 0 last_clean_interval [0,0) 10.223.169.201:6800/83569 <http://10.223.169.201:6800/83569> 10.223.169.201:6801/83569 <http://10.223.169.201:6801/83569> 10.223.169.201:6802/83569 <http://10.223.169.201:6802/83569> 10.223.169.201:6803/83569 <http://10.223.169.201:6803/83569> exists,up db545fd7-071f-4671-b1c4-c57221f894a3* *osd.2 up in weight 1 up_from 64 up_thru 64 down_at 61 last_clean_interval [12,60) 10.223.169.166:6805/92402 <http://10.223.169.166:6805/92402> 10.223.169.166:6806/92402 <http://10.223.169.166:6806/92402> 10.223.169.166:6807/92402 <http://10.223.169.166:6807/92402> 10.223.169.166:6808/92402 <http://10.223.169.166:6808/92402> exists,up 594b73b9-1908-4757-b914-d887d850b386* *osd.3 up in weight 1 up_from 17 up_thru 71 down_at 0 last_clean_interval [0,0) 10.223.169.201:6805/84590 <http://10.223.169.201:6805/84590> 10.223.169.201:6806/84590 <http://10.223.169.201:6806/84590> 10.223.169.201:6807/84590 <http://10.223.169.201:6807/84590> 10.223.169.201:6808/84590 <http://10.223.169.201:6808/84590> exists,up 37536050-ef92-4eba-95a7-e7a099c6d059* *root@cephnode2:~# * I also, listed the ports listening on the above highlighted OSD process using lsof root@cephnode2:~/nethogs# lsof -i | grep ceph | grep 83380 ntpd 1627 ntp 19u IPv4 33890 0t0 UDP cephnode2.iind.intel.com:ntp *ceph-osd 83380 root 4u IPv4 4881747 0t0 TCP *:6800 (LISTEN)* *ceph-osd 83380 root 5u IPv4 5045544 0t0 TCP cephnode2.iind.intel.com:6810 <http://cephnode2.iind.intel.com:6810> (LISTEN)* *ceph-osd 83380 root 6u IPv4 5045545 0t0 TCP cephnode2.iind.intel.com:6811 <http://cephnode2.iind.intel.com:6811> (LISTEN)* *ceph-osd 83380 root 7u IPv4 5045546 0t0 TCP cephnode2.iind.intel.com:6812 <http://cephnode2.iind.intel.com:6812> (LISTEN)* *ceph-osd 83380 root 8u IPv4 4881751 0t0 TCP *:6804 (LISTEN)* ceph-osd 83380 root 19u IPv4 5101954 0t0 TCP cephnode2.iind.intel.com:6800->computeich.iind.intel.com:60781 (ESTABLISHED) ceph-osd 83380 root 23u IPv4 5013387 0t0 TCP cephnode2.iind.intel.com:41878->cephnode4.iind.intel.com:6803 (ESTABLISHED) ceph-osd 83380 root 25u IPv4 5037728 0t0 TCP cephnode2.iind.intel.com:44251->cephnode4.iind.intel.com:6802 (ESTABLISHED) ceph-osd 83380 root 83u IPv4 5025954 0t0 TCP cephnode2.iind.intel.com:47863->cephnode4.iind.intel.com:6808 (ESTABLISHED) ceph-osd 83380 root 111u IPv4 4850005 0t0 TCP cephnode2.iind.intel.com:43189->cephnode2.iind.intel.com:6807 (ESTABLISHED) ceph-osd 83380 root 112u IPv4 4850839 0t0 TCP cephnode2.iind.intel.com:59738->cephnode2.iind.intel.com:6808 (ESTABLISHED) ceph-osd 83380 root 130u IPv4 5037729 0t0 TCP cephnode2.iind.intel.com:41902->cephnode4.iind.intel.com:6807 (ESTABLISHED) ceph-osd 83380 root 152u IPv4 5013621 0t0 TCP cephnode2.iind.intel.com:34798->cephmon.iind.intel.com:6789 (ESTABLISHED) ceph-osd 83380 root 159u IPv4 5040569 0t0 TCP cephnode2.iind.intel.com:6811->cephnode4.iind.intel.com:35321 (ESTABLISHED) ceph-osd 83380 root 160u IPv4 5040570 0t0 TCP cephnode2.iind.intel.com:6812->cephnode4.iind.intel.com:42682 (ESTABLISHED) ceph-osd 83380 root 161u IPv4 5043767 0t0 TCP cephnode2.iind.intel.com:6812->cephnode4.iind.intel.com:42683 (ESTABLISHED) ceph-osd 83380 root 162u IPv4 5038664 0t0 TCP cephnode2.iind.intel.com:6811->cephnode4.iind.intel.com:35324 (ESTABLISHED) In the above list, it looks like it is listening to some additional ports(6810-6812) from what is listed in the "ceph osd dump" command. I would like to know, if there is any straight way of listing the ports used by each OSD process. Also, I would also like to understand the networking architecture of Ceph in more detail. Is there any link/doc for the same? Thanks in Advance, Sharmila
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
