Re: [ceph-users] Fwd: [Ceph-community]Improve Read Performance
Thanks for your reply. I intend use Ceph RBD as shared storage for Oracle Database RAC. My Ceph deployment has 3 nodes with 8 1TB 15k SAS per node, I do not have SSD at the moment, so I design every SAS will be Journal and OSD. Can you suggest me a way to get highest performance for Oracle Cluster with this deployment? Many thanks. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Fwd: [Ceph-community]Improve Read Performance
How heavy transaction are you expecting? Shinobu On Sun, Aug 30, 2015 at 8:33 PM, Le Quang Long longlq.openst...@gmail.com wrote: Thanks for your reply. I intend use Ceph RBD as shared storage for Oracle Database RAC. My Ceph deployment has 3 nodes with 8 1TB 15k SAS per node, I do not have SSD at the moment, so I design every SAS will be Journal and OSD. Can you suggest me a way to get highest performance for Oracle Cluster with this deployment? Many thanks. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Email: shin...@linux.com ski...@redhat.com Life with Distributed Computational System based on OpenSource http://i-shinobu.hatenablog.com/ ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Fwd: [Ceph-community]Improve Read Performance
And what kind of performance are you looking for? I assume your workload will be small block random read/write? Btw, without SSD journal write performance will be very bad specially when your cluster is small.. Sent from my iPhone On Aug 30, 2015, at 4:33 AM, Le Quang Long longlq.openst...@gmail.commailto:longlq.openst...@gmail.com wrote: Thanks for your reply. I intend use Ceph RBD as shared storage for Oracle Database RAC. My Ceph deployment has 3 nodes with 8 1TB 15k SAS per node, I do not have SSD at the moment, so I design every SAS will be Journal and OSD. Can you suggest me a way to get highest performance for Oracle Cluster with this deployment? Many thanks. PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies). ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] OSD activate hangs
Hi, I am trying to ceph-deploy with Hammer on rhel7. While trying to activate the OSD using ceph-deploy on admin-node, the below step hangs. I tried to run it manually on the osd-node and tried tracing using python -m trace --trace . It looks like it is stuck in some threading.py code. Can someone please help? [*ceph-vm-osd1*][*WARNIN*] INFO:ceph-disk:Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd create --concise c2d19639-f3ec-447d-9a7c-a180a226dded [*ceph-vm-osd1*][*WARNIN*] No data was received after 300 seconds, disconnecting... *Manual run with --verbose option:* [cloud-user@ceph-vm-osd1 ~]$ sudo /usr/bin/ceph --verbose --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd create c2d19639-f3ec-447d-9a7c-a180a226dded parsed_args: Namespace(admin_socket=None, admin_socket_nope=None, cephconf=None, client_id=None, client_name='client.bootstrap-osd', cluster='ceph', cluster_timeout=None, completion=False, help=False, input_file=None, output_file=None, output_format=None, status=False, verbose=True, version=False, watch=False, watch_debug=False, watch_error=False, watch_info=False, watch_sec=False, watch_warn=False), childargs: ['--keyring', '/var/lib/ceph/bootstrap-osd/ceph.keyring', 'osd', 'create', 'c2d19639-f3ec-447d-9a7c-a180a226dded'] ^CError connecting to cluster: InterruptedOrTimeoutError *Manual run with python -m trace --trace :* truncated --- modulename: threading, funcname: _note threading.py(64): if self.__verbose: threading.py(946): self.__block.acquire() threading.py(947): try: threading.py(948): if timeout is None: threading.py(954): deadline = _time() + timeout threading.py(955): while not self.__stopped: threading.py(956): delay = deadline - _time() threading.py(957): if delay = 0: threading.py(961): self.__block.wait(delay, balancing) --- modulename: threading, funcname: wait threading.py(331): if not self._is_owned(): --- modulename: threading, funcname: _is_owned threading.py(302): if self.__lock.acquire(0): threading.py(306): return True threading.py(333): waiter = _allocate_lock() threading.py(334): waiter.acquire() threading.py(335): self.__waiters.append(waiter) threading.py(336): saved_state = self._release_save() --- modulename: threading, funcname: _release_save threading.py(294): self.__lock.release() # No state to save threading.py(337): try:# restore state no matter what (e.g., KeyboardInterrupt) threading.py(338): if timeout is None: threading.py(348): endtime = _time() + timeout threading.py(349): delay = 0.0005 # 500 us - initial delay of 1 ms threading.py(350): while True: threading.py(351): gotit = waiter.acquire(0) threading.py(352): if gotit: threading.py(354): remaining = endtime - _time() threading.py(355): if remaining = 0: threading.py(357): if balancing: threading.py(358): delay = min(delay * 2, remaining, 0.05) threading.py(361): _sleep(delay) --- modulename: threading, funcname: _note threading.py(64): if self.__verbose: --- modulename: threading, funcname: __stop threading.py(870): if not hasattr(self, '_Thread__block'): threading.py(872): self.__block.acquire() threading.py(873): self.__stopped = True threading.py(874): self.__block.notify_all() --- modulename: threading, funcname: notifyAll threading.py(409): self.notify(len(self.__waiters)) --- modulename: threading, funcname: notify threading.py(385): if not self._is_owned(): --- modulename: threading, funcname: _is_owned threading.py(302): if self.__lock.acquire(0): threading.py(306): return True threading.py(387): __waiters = self.__waiters threading.py(388): waiters = __waiters[:n] threading.py(389): if not waiters: threading.py(393): self._note(%s.notify(): notifying %d waiter%s, self, n, threading.py(394):n!=1 and s or ) --- modulename: threading, funcname: _note threading.py(64): if self.__verbose: threading.py(395): for waiter in waiters: threading.py(396): waiter.release() threading.py(397): try: threading.py(398): __waiters.remove(waiter) threading.py(395): for waiter in waiters: threading.py(875): self.__block.release() threading.py(350): while True: threading.py(351): gotit = waiter.acquire(0) threading.py(352):
Re: [ceph-users] Fwd: [Ceph-community]Improve Read Performance
Yes, I will use Ceph RBD as shared Storage for Oracle Database Cluster, so I need high I/O read write random. With 3 nodes and 24 SAS 15K 1TB, what is the most optimized solution to get it ? On Aug 31, 2015 2:01 AM, Somnath Roy somnath@sandisk.com wrote: And what kind of performance are you looking for? I assume your workload will be small block random read/write? Btw, without SSD journal write performance will be very bad specially when your cluster is small.. Sent from my iPhone On Aug 30, 2015, at 4:33 AM, Le Quang Long longlq.openst...@gmail.com wrote: Thanks for your reply. I intend use Ceph RBD as shared storage for Oracle Database RAC. My Ceph deployment has 3 nodes with 8 1TB 15k SAS per node, I do not have SSD at the moment, so I design every SAS will be Journal and OSD. Can you suggest me a way to get highest performance for Oracle Cluster with this deployment? Many thanks. -- PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies). ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph-deploy error
In case someone else runs into the same issue in future: I came out of this issue by installing epel-release before installing ceph-deploy. If the order of installation is ceph-deploy followed by epel-release, the issue is being hit. Thanks, Pavana On Sat, Aug 29, 2015 at 10:02 AM, pavana bhat pavanakrishnab...@gmail.com wrote: Hi, I'm trying to install ceph for the first time following the quick installation guide. I'm getting the below error, can someone please help? ceph-deploy install --release=firefly ceph-vm-mon1 [*ceph_deploy.conf*][*DEBUG* ] found configuration file at: /home/cloud-user/.cephdeploy.conf [*ceph_deploy.cli*][*INFO* ] Invoked (1.5.28): /usr/bin/ceph-deploy install --release=firefly ceph-vm-mon1 [*ceph_deploy.cli*][*INFO* ] ceph-deploy options: [*ceph_deploy.cli*][*INFO* ] verbose : False [*ceph_deploy.cli*][*INFO* ] testing : None [*ceph_deploy.cli*][*INFO* ] cd_conf : ceph_deploy.conf.cephdeploy.Conf instance at 0xaab248 [*ceph_deploy.cli*][*INFO* ] cluster : ceph [*ceph_deploy.cli*][*INFO* ] install_mds : False [*ceph_deploy.cli*][*INFO* ] stable: None [*ceph_deploy.cli*][*INFO* ] default_release : False [*ceph_deploy.cli*][*INFO* ] username : None [*ceph_deploy.cli*][*INFO* ] adjust_repos : True [*ceph_deploy.cli*][*INFO* ] func : function install at 0x7f34b410e938 [*ceph_deploy.cli*][*INFO* ] install_all : False [*ceph_deploy.cli*][*INFO* ] repo : False [*ceph_deploy.cli*][*INFO* ] host : ['ceph-vm-mon1'] [*ceph_deploy.cli*][*INFO* ] install_rgw : False [*ceph_deploy.cli*][*INFO* ] repo_url : None [*ceph_deploy.cli*][*INFO* ] ceph_conf : None [*ceph_deploy.cli*][*INFO* ] install_osd : False [*ceph_deploy.cli*][*INFO* ] version_kind : stable [*ceph_deploy.cli*][*INFO* ] install_common: False [*ceph_deploy.cli*][*INFO* ] overwrite_conf: False [*ceph_deploy.cli*][*INFO* ] quiet : False [*ceph_deploy.cli*][*INFO* ] dev : master [*ceph_deploy.cli*][*INFO* ] local_mirror : None [*ceph_deploy.cli*][*INFO* ] release : firefly [*ceph_deploy.cli*][*INFO* ] install_mon : False [*ceph_deploy.cli*][*INFO* ] gpg_url : None [*ceph_deploy.install*][*DEBUG* ] Installing stable version firefly on cluster ceph hosts ceph-vm-mon1 [*ceph_deploy.install*][*DEBUG* ] Detecting platform for host ceph-vm-mon1 ... [*ceph-vm-mon1*][*DEBUG* ] connection detected need for sudo [*ceph-vm-mon1*][*DEBUG* ] connected to host: ceph-vm-mon1 [*ceph-vm-mon1*][*DEBUG* ] detect platform information from remote host [*ceph-vm-mon1*][*DEBUG* ] detect machine type [*ceph_deploy.install*][*INFO* ] Distro info: Red Hat Enterprise Linux Server 7.1 Maipo [*ceph-vm-mon1*][*INFO* ] installing Ceph on ceph-vm-mon1 [*ceph-vm-mon1*][*INFO* ] Running command: sudo yum clean all [*ceph-vm-mon1*][*DEBUG* ] Loaded plugins: fastestmirror, priorities [*ceph-vm-mon1*][*DEBUG* ] Cleaning repos: epel rhel-7-ha-rpms rhel-7-optional-rpms rhel-7-server-rpms [*ceph-vm-mon1*][*DEBUG* ] : rhel-7-supplemental-rpms [*ceph-vm-mon1*][*DEBUG* ] Cleaning up everything [*ceph-vm-mon1*][*DEBUG* ] Cleaning up list of fastest mirrors [*ceph-vm-mon1*][*INFO* ] Running command: sudo yum -y install epel-release [*ceph-vm-mon1*][*DEBUG* ] Loaded plugins: fastestmirror, priorities [*ceph-vm-mon1*][*DEBUG* ] Determining fastest mirrors [*ceph-vm-mon1*][*DEBUG* ] * epel: kdeforge2.unl.edu [*ceph-vm-mon1*][*DEBUG* ] * rhel-7-ha-rpms: rhel-repo.eu-biere-1.t-systems.cloud.cisco.com [*ceph-vm-mon1*][*DEBUG* ] * rhel-7-optional-rpms: rhel-repo.eu-biere-1.t-systems.cloud.cisco.com [*ceph-vm-mon1*][*DEBUG* ] * rhel-7-server-rpms: rhel-repo.eu-biere-1.t-systems.cloud.cisco.com [*ceph-vm-mon1*][*DEBUG* ] * rhel-7-supplemental-rpms: rhel-repo.eu-biere-1.t-systems.cloud.cisco.com [*ceph-vm-mon1*][*DEBUG* ] Package epel-release-7-5.noarch already installed and latest version [*ceph-vm-mon1*][*DEBUG* ] Nothing to do [*ceph-vm-mon1*][*INFO* ] Running command: sudo yum -y install yum-plugin-priorities [*ceph-vm-mon1*][*DEBUG* ] Loaded plugins: fastestmirror, priorities [*ceph-vm-mon1*][*DEBUG* ] Loading mirror speeds from cached hostfile [*ceph-vm-mon1*][*DEBUG* ] * epel: kdeforge2.unl.edu [*ceph-vm-mon1*][*DEBUG* ] * rhel-7-ha-rpms: rhel-repo.eu-biere-1.t-systems.cloud.cisco.com [*ceph-vm-mon1*][*DEBUG* ] * rhel-7-optional-rpms:
Re: [ceph-users] Is Ceph appropriate for small installations?
On 29 August 2015 at 00:53, Tony Nelson tnel...@starpoint.com wrote: I recently built a 3 node Proxmox cluster for my office. I’d like to get HA setup, and the Proxmox book recommends Ceph. I’ve been reading the documentation and watching videos, and I think I have a grasp on the basics, but I don’t need anywhere near a petabyte of storage. I’m considering servers w/ 12 drive bays, 2 SDD mirrored for the OS, 2 SDDs for journals and the other 8 for OSDs. I was going to purchase 3 identical servers, and use my 3 Proxmox servers as the monitors, with of course GB networking in between. Obviously this is very vague, but I’m just getting started on the research. I run a small 3 node Proxmox cluster for our office as well with Ceph, but I'd now recommend against using Ceph for small setups like ours. - Maintenance headache. Ceph requires a lot of tweaking to get started and a lot of ongoing monitoring, plus a fair bit of skill. If you're running the show yourself (as typical in small businesses) its quite stressful. Who's going to fix the ceph cluster when a osd goes down when you're on holiday? - Performance. Its terrible on small clusters. I've setup a iSCSI over ZFS for a server and its orders of magnitude better at I/O. And I haven't even configured multipath yet. - Flexibility. Much much easier to expand or replace disks on my ZFS server. The redundancy is good, I can reboot a ceph node for maintenance and it recovers very quickly (much quicker than glusterfs), but cluster performance suffers badly when a node is down so in practice its of limited utility. I'm coming to the realisation that for us performance and ease of administration is more valuable than 100% uptime. Worst case (Storage server dies) we could rebuild from backups in a day. Essentials could be restored in a hour. I could experiment with ongoing ZFS replications to a backup server that makes that even quicker. Thats for use - your requirements may be different. And of course once you get into truly large deployments, ceph comes into its own. -- Lindsay ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Storage node refurbishing, a freeze OSD feature would be nice
Hello, I'm about to add another storage node to small firefly cluster here and refurbish 2 existing nodes (more RAM, different OSD disks). Insert rant about not going to start using ceph-deploy as I would have to set the cluster to no-in since prepare also activates things due to the udev magic... This cluster is quite at the limits of its IOPS capacity (the HW was requested ages ago, but the mills here grind slowly and not particular fine either), so the plan is to: a) phase in the new node (lets call it C), one OSD at a time (in the dead of night) b) empty out old node A (weight 0), one OSD at a time. When done, refurbish and bring it back in, like above. c) repeat with 2nd old node B. Looking at this it's obvious where the big optimization in this procedure would be, having the ability to freeze the OSDs on node B. That is making them ineligible for any new PGs while preserving their current status. So that data moves from A to C (which is significantly faster than A or B) and then back to A when it is refurbished, avoiding any heavy lifting by B. Does that sound like something other people might find useful as well and is it feasible w/o upsetting the CRUSH applecart? Christian -- Christian BalzerNetwork/Systems Engineer ch...@gol.com Global OnLine Japan/Fusion Communications http://www.gol.com/ ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
