Re: [foreman-dev] Redmine running slowly

2017-10-12 Thread Ewoud Kohl van Wijngaarden

On Thu, Oct 12, 2017 at 11:50:55AM +0100, Greg Sutcliffe wrote:

And we're back. Please test extensively and report issues ;)

In particular, we're now on Ruby 2.0 (up from 1.9 on Openshift) so I
suspect the plugins might have issues. Any issues, please report them
here and we'll take a look. I checked quickly and didn't see any
errors, but I was in a hurry :P

Things still to do:

* Add logrotate for the production.log files
* Add HTTPS (ewoud is on that)
* Upgrade to latest Redmine (I'm looking into it)
* Puppetize it so we can migrate more easily in future

We'll monitor to see if the resources are enough, let me know if you
see any issues (no, Backlogs does not count, that thing is *slow* :P)


We did have to do some changes to the importing of git repos. If you 
merge an issue with a keyword (refs or fixes) and it doesn't update the 
issue in an hour then please let us know.


--
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: [foreman-dev] Redmine running slowly

2017-10-12 Thread Greg Sutcliffe
And we're back. Please test extensively and report issues ;)

In particular, we're now on Ruby 2.0 (up from 1.9 on Openshift) so I
suspect the plugins might have issues. Any issues, please report them
here and we'll take a look. I checked quickly and didn't see any
errors, but I was in a hurry :P

Things still to do:

* Add logrotate for the production.log files
* Add HTTPS (ewoud is on that)
* Upgrade to latest Redmine (I'm looking into it)
* Puppetize it so we can migrate more easily in future

We'll monitor to see if the resources are enough, let me know if you
see any issues (no, Backlogs does not count, that thing is *slow* :P)

Thanks for your patience everyone.
Greg

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: [foreman-dev] Redmine running slowly

2017-10-12 Thread Greg Sutcliffe
We're ready to switch. I'll take Redmine offline in about 30mins (11.15
UK time) and we should be back up on the new host at 12 noon. Please
save your work ;)

Greg

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: [foreman-dev] Redmine running slowly

2017-10-12 Thread Greg Sutcliffe
On Wed, 2017-10-11 at 21:52 +0300, Ohad Levy wrote:
> 
> > redmine01.theforeman.org A 51.15.192.166
> > redmine01.theforeman.org  2001:bc8:4400:2300::5:e03
> 
> both should be resolved.

Thanks Ohad.

SMTP is now unblocked and confirmed working, so we're ready to switch
over. As soon as I can get a timeframe from Ohad for when the DNS
change can be done, I'll arrange a maintenance window.

Hang in there guys, we're almost done ;)

Greg

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: [foreman-dev] Redmine running slowly

2017-10-11 Thread Ohad Levy
On Wed, Oct 11, 2017 at 7:39 PM, Ewoud Kohl van Wijngaarden <
ew...@kohlvanwijngaarden.nl> wrote:

> On Wed, Oct 11, 2017 at 07:36:39PM +0300, Ohad Levy wrote:
>
>> On Oct 11, 2017 5:40 PM, "Greg Sutcliffe" 
>> wrote:
>>
>> Update on migration:
>>
>> http://51.15.192.166 is now live for your testing pleasure, with a copy
>>
>>
>> Any chance to update/enable https in the process?
>>
>
> That is the plan, but first we want to know it's working. If you can
> create DNS records we can already test the process with letsencrypt:
>
> redmine01.theforeman.org A 51.15.192.166
> redmine01.theforeman.org  2001:bc8:4400:2300::5:e03


both should be resolved.

Ohad

>
>
> Thanks!
>>
>> of the DB from a few days ago. Email is currently disabled, so you
>> can't spam anyone. There's still a few small tasks to sort out that
>> will keep me busy, but if you want to see if you can break it, go
>> ahead.
>>
>> Sadly, SMTP outbound is blocked by default on Scaleway, which I didn't
>> realise until about 20 mins ago. I've raised a ticket to lift this so
>> Redmine can send email, but until that's resolved we can't complete the
>> migration.
>>
>> Once email is confirmed working, we'll schedule a maintenance window,
>> where I will stop the Openshift instance and make a final DB dump. Ohad
>> will then do a DNS switch, and as soon as it comes back you should all
>> be good to go.
>>
>> Stay tuned for further updates. I'll announce the maintenance window
>> before I start.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: [foreman-dev] Redmine running slowly

2017-10-11 Thread Ewoud Kohl van Wijngaarden

On Wed, Oct 11, 2017 at 07:36:39PM +0300, Ohad Levy wrote:

On Oct 11, 2017 5:40 PM, "Greg Sutcliffe"  wrote:

Update on migration:

http://51.15.192.166 is now live for your testing pleasure, with a copy


Any chance to update/enable https in the process?


That is the plan, but first we want to know it's working. If you can 
create DNS records we can already test the process with letsencrypt:


redmine01.theforeman.org A 51.15.192.166
redmine01.theforeman.org  2001:bc8:4400:2300::5:e03


Thanks!

of the DB from a few days ago. Email is currently disabled, so you
can't spam anyone. There's still a few small tasks to sort out that
will keep me busy, but if you want to see if you can break it, go
ahead.

Sadly, SMTP outbound is blocked by default on Scaleway, which I didn't
realise until about 20 mins ago. I've raised a ticket to lift this so
Redmine can send email, but until that's resolved we can't complete the
migration.

Once email is confirmed working, we'll schedule a maintenance window,
where I will stop the Openshift instance and make a final DB dump. Ohad
will then do a DNS switch, and as soon as it comes back you should all
be good to go.

Stay tuned for further updates. I'll announce the maintenance window
before I start.


--
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: [foreman-dev] Redmine running slowly

2017-10-11 Thread Ohad Levy
On Oct 11, 2017 5:40 PM, "Greg Sutcliffe"  wrote:

Update on migration:

http://51.15.192.166 is now live for your testing pleasure, with a copy


Any chance to update/enable https in the process?

Thanks!

of the DB from a few days ago. Email is currently disabled, so you
can't spam anyone. There's still a few small tasks to sort out that
will keep me busy, but if you want to see if you can break it, go
ahead.

Sadly, SMTP outbound is blocked by default on Scaleway, which I didn't
realise until about 20 mins ago. I've raised a ticket to lift this so
Redmine can send email, but until that's resolved we can't complete the
migration.

Once email is confirmed working, we'll schedule a maintenance window,
where I will stop the Openshift instance and make a final DB dump. Ohad
will then do a DNS switch, and as soon as it comes back you should all
be good to go.

Stay tuned for further updates. I'll announce the maintenance window
before I start.

Greg

--
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: [foreman-dev] Redmine running slowly

2017-10-11 Thread Greg Sutcliffe
Update on migration:

http://51.15.192.166 is now live for your testing pleasure, with a copy
of the DB from a few days ago. Email is currently disabled, so you
can't spam anyone. There's still a few small tasks to sort out that
will keep me busy, but if you want to see if you can break it, go
ahead. 

Sadly, SMTP outbound is blocked by default on Scaleway, which I didn't
realise until about 20 mins ago. I've raised a ticket to lift this so
Redmine can send email, but until that's resolved we can't complete the
migration. 

Once email is confirmed working, we'll schedule a maintenance window,
where I will stop the Openshift instance and make a final DB dump. Ohad
will then do a DNS switch, and as soon as it comes back you should all
be good to go.

Stay tuned for further updates. I'll announce the maintenance window
before I start.

Greg

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: [foreman-dev] Redmine running slowly

2017-10-11 Thread Greg Sutcliffe
Things are no better today, and I'm out of patience with Openshift.
I've spun up an instance on Scaleway that should be able to cope and am
in the process of creating a copy of Redmine there. Once it's ready
I'll stop Openshift, re-import the db and cut over the DNS.

Continue to use Redmine as normal for now, I'll update when we're ready
to do the final cutover.

Greg

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: [foreman-dev] Redmine running slowly

2017-10-11 Thread Lukas Zapletal
Thanks guys for doing this.

If things go terribly wrong, we still have an account on EC2 where we
run our koji.

On Tue, Oct 10, 2017 at 9:21 PM, Greg Sutcliffe
 wrote:
> On Tue, 2017-10-10 at 10:50 -0400, Andrew Kofink wrote:
>> Me as well. It's quite difficult to work this way.
>
> Yeah, I know :(
>
> Openshift aren't saying much other than that this is mainly due to the
> number of people that decided to upgrade to Silver Tier to avoid the
> sunset of v2. That's putting a lot of load on the v2 cluster, which
> obviously is hitting us.
>
> As Ewoud said, we've made a change today in how we process the
> underlying cron jobs that should reduce the amount of IO we were doing
> - if there's any kind of quota-ing going on, that should help. We're
> seeing that bring the time taken to run the cron down to about 10mins
> (starting at the top of the hour). That should improve things during
> that period. Sadly I did make a mistake during a manual part of the
> changes that impacted the DB, but that should be resolved now.
>
> Base load now seems to be down to around 7-9 which is better but still
> too high. Sadly the v3 resources are unlikely to be available before
> November, which is a limiter. If things are not better in the next day
> or two, then on Thu or Fri I may migrate it to our Scaleway account
> anyway, as we have capacity there, although I'd rather not migrate
> twice...
>
> Greg
>
> --
> You received this message because you are subscribed to the Google Groups 
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: [foreman-dev] Redmine running slowly

2017-10-10 Thread Greg Sutcliffe
On Tue, 2017-10-10 at 10:50 -0400, Andrew Kofink wrote:
> Me as well. It's quite difficult to work this way.

Yeah, I know :(

Openshift aren't saying much other than that this is mainly due to the
number of people that decided to upgrade to Silver Tier to avoid the
sunset of v2. That's putting a lot of load on the v2 cluster, which
obviously is hitting us. 

As Ewoud said, we've made a change today in how we process the
underlying cron jobs that should reduce the amount of IO we were doing
- if there's any kind of quota-ing going on, that should help. We're
seeing that bring the time taken to run the cron down to about 10mins
(starting at the top of the hour). That should improve things during
that period. Sadly I did make a mistake during a manual part of the
changes that impacted the DB, but that should be resolved now.

Base load now seems to be down to around 7-9 which is better but still
too high. Sadly the v3 resources are unlikely to be available before
November, which is a limiter. If things are not better in the next day
or two, then on Thu or Fri I may migrate it to our Scaleway account
anyway, as we have capacity there, although I'd rather not migrate
twice...

Greg

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: [foreman-dev] Redmine running slowly

2017-10-10 Thread Ewoud Kohl van Wijngaarden
We deployed a new version but that took longer than expected. Now we use 
bare git clones rather than doing full checkouts. This should save a lot 
of IO which is generally a limiting factor. Hopefully this helps enough 
until we can migrate to the new platform.


https://github.com/theforeman/redmine/commit/cb4ccf049e0c892fcbba98861c904492e9833a67

On Tue, Oct 10, 2017 at 10:50:24AM -0400, Andrew Kofink wrote:

Me as well. It's quite difficult to work this way.

On Tue, Oct 10, 2017 at 10:40 AM, Dirk Götz  wrote:


Now Redmine seems to be down completely. Only getting 404 or 502 errors
since an half hour.


--
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: [foreman-dev] Redmine running slowly

2017-10-10 Thread Andrew Kofink
Me as well. It's quite difficult to work this way.

On Tue, Oct 10, 2017 at 10:40 AM, Dirk Götz  wrote:

> Now Redmine seems to be down completely. Only getting 404 or 502 errors
> since an half hour.
>
> Regards,
>  Dirk
>
> --
> You received this message because you are subscribed to the Google Groups
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Andrew Kofink
akof...@redhat.com
IRC: akofink
Associate Software Engineer
Red Hat Satellite

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: [foreman-dev] Redmine running slowly

2017-10-10 Thread Dirk Götz
Now Redmine seems to be down completely. Only getting 404 or 502 errors 
since an half hour.

Regards,
 Dirk

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: [foreman-dev] Redmine running slowly

2017-10-05 Thread Greg Sutcliffe
On Thu, 2017-10-05 at 14:08 +0200, Lukas Zapletal wrote:
> Ohad mentioned this on our coffee meeting, but why we insist on
> OpenShift? It's just a regular Rails app, we could easily drop it to
> one of our web hosts and maybe the performance will be even better
> than it was.
> 
> In other words, no OpenShift V3, our infra for now and forever. Not
> sure how much work is to puppetize this, but this could be, I know
> someone will eat me for these words, simply a pet server with no
> automatization.
> 
> /me runs away! :-)

Our infra is already overloaded (see other mials on this list), so
something else would have to give way for it. Openshift is free and a
good fit for anything "simple" and self-contained, which is why
Redmine, PrProcessor, and the Etherpad run on it.

If we cannot get this resolved, or if the extra v3 resources aren't
granted, then we'll have to do exactly that, but I'll take what free
handouts we can get :)

Current update BTW, the load is dropping (now sitting around 7) and
I've opened a support ticket. Preliminary feedback is that is likely
due to a large number of people upgrading their v2 accounts to Silver
to avoid the sunset, leading to increased load on the cluster.
Hopefully it'll settle down soon.

Greg

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: [foreman-dev] Redmine running slowly

2017-10-05 Thread Lukas Zapletal
Ohad mentioned this on our coffee meeting, but why we insist on
OpenShift? It's just a regular Rails app, we could easily drop it to
one of our web hosts and maybe the performance will be even better
than it was.

In other words, no OpenShift V3, our infra for now and forever. Not
sure how much work is to puppetize this, but this could be, I know
someone will eat me for these words, simply a pet server with no
automatization.

/me runs away! :-)

LZ

On Wed, Oct 4, 2017 at 6:18 PM, Greg Sutcliffe  wrote:
> Since the start of this week Redmine has been consistently slow.
> Console shows very high load (not dropping below 10, often above 20).
> We haven't chnaged anyhting, so my suspicions are on the v2 sunset that
> happened at the end of last week.
>
> I've opened a support case to see what's going on, but for now treat
> Redmine with care ;)
>
> Greg
>
> --
> You received this message because you are subscribed to the Google Groups 
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


[foreman-dev] Redmine running slowly

2017-10-04 Thread Greg Sutcliffe
Since the start of this week Redmine has been consistently slow.
Console shows very high load (not dropping below 10, often above 20).
We haven't chnaged anyhting, so my suspicions are on the v2 sunset that
happened at the end of last week.

I've opened a support case to see what's going on, but for now treat
Redmine with care ;)

Greg

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.