Ok, cool. It's worth noting that worker nodes are completely disposable and that if there are any not running jobs at the top of the hour (when billing happens) it's a waste. What you may want to do, at least until the event tomorrow, is scale back down to 1 master and 1 worker node and turn on autoscaling.
Cloudman's autoscaling will automatically kill idle instances at the end of the hour (prior to being billed for the next hour), and will start more if the job queue starts to get backed up (within the limits set in the admin interface, of course). On Mon, May 6, 2013 at 4:31 PM, Iry Witham <iry.wit...@jax.org> wrote: > I did add nodes about the same time I setup load balancing so that was > likely the fix. I will definitely set the head node not to run jobs. > > Thanks, > Iry > > From: Dannon Baker <dannon.ba...@gmail.com> > Date: Monday, May 6, 2013 4:19 PM > > To: Iry Witham <iry.wit...@jax.org> > Cc: "galaxy-dev@lists.bx.psu.edu" <galaxy-dev@lists.bx.psu.edu> > Subject: Re: [galaxy-dev] Resizing galaxy cloudman /mnt/galaxyData > partition > > Ahh, ok. So, unfortunately Cloudman isn't designed to be used with a > load balancer (at least at this point). In any given cluster you have > exactly one master node and all of the rest are workers (which do not run > web processes, hence the 1/9 health). > > What was slow that you were trying to improve the performance of? One > thing I'd definitely recommend for head node performance, if you do have > extra worker nodes, would be to go to the admin panel > (<your_ec2_instance>/cloud/admin) and disable job running on the head node > using the "Set master to not run jobs" option. > > > > On Mon, May 6, 2013 at 4:15 PM, Iry Witham <iry.wit...@jax.org> wrote: > >> I was not clear how the load balancer worked in this instance, but my >> goal was to improve performance. I am not sure this was the correct use, >> but it seemed to work. >> >> From: Dannon Baker <dannon.ba...@gmail.com> >> Date: Monday, May 6, 2013 4:12 PM >> >> To: Iry Witham <iry.wit...@jax.org> >> Cc: "galaxy-dev@lists.bx.psu.edu" <galaxy-dev@lists.bx.psu.edu> >> Subject: Re: [galaxy-dev] Resizing galaxy cloudman /mnt/galaxyData >> partition >> >> So you have a cluster with 9 instances and you're using ec2 load >> balancer with all of them to divert traffic? >> >> >> On Mon, May 6, 2013 at 3:56 PM, Iry Witham <iry.wit...@jax.org> wrote: >> >>> I have setup a load balancer, but it shows that only 1 of 9 instances >>> are 'In Service". I have set the health check to: >>> >>> Ping: http:80/ >>> Timeout: 40 seconds >>> Interval: 60 seconds >>> Unhealthy threshold: 2 >>> Healty threshold: 10 >>> >>> As far as autoscaling, I am not really interested since I see no real >>> need at this point. >>> >>> Thanks, >>> >>> Iry >>> >>> From: Dannon Baker <dannon.ba...@gmail.com> >>> Date: Monday, May 6, 2013 3:41 PM >>> >>> To: Iry Witham <iry.wit...@jax.org> >>> Cc: "galaxy-dev@lists.bx.psu.edu" <galaxy-dev@lists.bx.psu.edu> >>> Subject: Re: [galaxy-dev] Resizing galaxy cloudman /mnt/galaxyData >>> partition >>> >>> Great, I'm glad it's working, and I'm sorry you ran into this bug. >>> >>> As far as autoscaling, etc., what are you interested in? I should be >>> able to help out with just about anything related to Cloudman. >>> >>> >>> On Mon, May 6, 2013 at 3:03 PM, Iry Witham <iry.wit...@jax.org> wrote: >>> >>>> That seems to be working. >>>> >>>> From: Dannon Baker <dannon.ba...@gmail.com> >>>> Date: Monday, May 6, 2013 3:01 PM >>>> >>>> To: Iry Witham <iry.wit...@jax.org> >>>> Cc: "galaxy-dev@lists.bx.psu.edu" <galaxy-dev@lists.bx.psu.edu> >>>> Subject: Re: [galaxy-dev] Resizing galaxy cloudman /mnt/galaxyData >>>> partition >>>> >>>> The first thing that comes to mind is a bug that existed in a >>>> particular version of cloudman regarding the max size allowable -- can you >>>> try 999GB? >>>> >>>> >>>> On Mon, May 6, 2013 at 2:57 PM, Iry Witham <iry.wit...@jax.org> wrote: >>>> >>>>> Hi Dannon, >>>>> >>>>> I have attempted the resizing. I clicked on the disk and set it for >>>>> 1000GB. The instance restarted, but the volume remains at 100GB. Is >>>>> this due to the lack of disk space? I don't see a snapshot when I look at >>>>> the AWS console after the restart. Any ideas? >>>>> >>>>> Thanks, >>>>> >>>>> IRy >>>>> >>>>> From: Dannon Baker <dannon.ba...@gmail.com> >>>>> Date: Monday, May 6, 2013 2:06 PM >>>>> To: Iry Witham <iry.wit...@jax.org> >>>>> Cc: "galaxy-dev@lists.bx.psu.edu" <galaxy-dev@lists.bx.psu.edu> >>>>> Subject: Re: [galaxy-dev] Resizing galaxy cloudman /mnt/galaxyData >>>>> partition >>>>> >>>>> Hi Iry, >>>>> >>>>> This should work, I use it regularly. Just make sure you're not >>>>> ssh'ed in and 'in' /mnt/galaxyData or anything like that. There's not >>>>> really another way to do this that I'd recommend trying if you need the >>>>> instance to continue working for something tomorrow. >>>>> >>>>> Let me know if you have any issues and I can help. >>>>> >>>>> -Dannon >>>>> >>>>> >>>>> On Mon, May 6, 2013 at 1:54 PM, Iry Witham <iry.wit...@jax.org> wrote: >>>>> >>>>>> Hi Team, >>>>>> >>>>>> I urgently need to resize the data partition on my galaxy cloudman >>>>>> instance without loosing anything. I notice that there a gui on the >>>>>> cloudman admin page that will allow me to increase it to 1000GB. If I >>>>>> use >>>>>> that will it restart all of the processes? I need to maintain this as a >>>>>> functional instance since it needs to be available for use tomorrow at >>>>>> 8:00 >>>>>> AM and I do not have the time to rebuild it. The gui says it will >>>>>> create a >>>>>> snapshot. The following is what it says: >>>>>> >>>>>> "Expand Disk Space >>>>>> Through this form you may increase the disk space available to >>>>>> Galaxy. All of the cluster services (but not the cluster) WILL BE STOPPED >>>>>> until the new disk is ready, at which point they will all be restarted. >>>>>> This may result in Galaxy jobs that are currently running to fail. Note >>>>>> that the new disk size must be larger than the current disk size. >>>>>> >>>>>> During this process, a snapshot of your data volume will be >>>>>> created, which can optionally be left in your account. If you decide to >>>>>> leave the snapshot for reference, you may also provide a brief note that >>>>>> will later be visible in the snapshot's description. New Disk Size >>>>>> (minimum >>>>>> 100GB, maximum 1000GB): >>>>>> >>>>>> OK >>>>>> Note (optional): >>>>>> >>>>>> or delete the created snapshot after filesystem resizing? If >>>>>> checked, the created snapshot will not be kept" >>>>>> >>>>>> What is the likelihood that this will be successful? Has anyone >>>>>> utilized it? Is there another way to resize the /mnt/galaxyData/ >>>>>> and /mnt/galaxyIndices partitions that will allow me to do what I need >>>>>> to? >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Iry Witham >>>>>> >>>>>> The information in this email, including attachments, may be >>>>>> confidential and is intended solely for the addressee(s). If you believe >>>>>> you received this email by mistake, please notify the sender by return >>>>>> email as soon as possible. >>>>>> >>>>>> ___________________________________________________________ >>>>>> Please keep all replies on the list by using "reply all" >>>>>> in your mail client. To manage your subscriptions to this >>>>>> and other Galaxy lists, please use the interface at: >>>>>> http://lists.bx.psu.edu/ >>>>>> >>>>>> To search Galaxy mailing lists use the unified search at: >>>>>> http://galaxyproject.org/search/mailinglists/ >>>>>> >>>>> >>>>> The information in this email, including attachments, may be >>>>> confidential and is intended solely for the addressee(s). If you believe >>>>> you received this email by mistake, please notify the sender by return >>>>> email as soon as possible. >>>>> >>>> >>>> The information in this email, including attachments, may be >>>> confidential and is intended solely for the addressee(s). If you believe >>>> you received this email by mistake, please notify the sender by return >>>> email as soon as possible. >>>> >>> >>> The information in this email, including attachments, may be >>> confidential and is intended solely for the addressee(s). If you believe >>> you received this email by mistake, please notify the sender by return >>> email as soon as possible. >>> >> >> The information in this email, including attachments, may be >> confidential and is intended solely for the addressee(s). If you believe >> you received this email by mistake, please notify the sender by return >> email as soon as possible. >> > > The information in this email, including attachments, may be > confidential and is intended solely for the addressee(s). If you believe > you received this email by mistake, please notify the sender by return > email as soon as possible. >
___________________________________________________________ Please keep all replies on the list by using "reply all" in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
