Thank you always, Shane!
On Thu, Oct 15, 2020 at 4:55 AM shane knapp ☠ wrote:
> everything's up and jenkins is slowly chewing through the queue! :)
>
> On Wed, Oct 14, 2020 at 12:00 PM Xiao Li wrote:
>
>> Thank you, Shane!
>>
>> Xiao
>>
>> On Wed, Oct 14, 2020 at 12:00 PM shane knapp ☠
>>
everything's up and jenkins is slowly chewing through the queue! :)
On Wed, Oct 14, 2020 at 12:00 PM Xiao Li wrote:
> Thank you, Shane!
>
> Xiao
>
> On Wed, Oct 14, 2020 at 12:00 PM shane knapp ☠
> wrote:
>
>> we're mostly back up, and just waiting for a couple of ubuntu boxes to
>> finish
Thank you, Shane!
Xiao
On Wed, Oct 14, 2020 at 12:00 PM shane knapp ☠ wrote:
> we're mostly back up, and just waiting for a couple of ubuntu boxes to
> finish booting... prb seem to be building now!
>
> On Wed, Oct 14, 2020 at 11:48 AM shane knapp ☠
> wrote:
>
>> i'm going to reboot the
we're mostly back up, and just waiting for a couple of ubuntu boxes to
finish booting... prb seem to be building now!
On Wed, Oct 14, 2020 at 11:48 AM shane knapp ☠ wrote:
> i'm going to reboot the primary and worker nodes, so it'll be a few
> minutes before everything is back up.
>
> shane
>
i'm going to reboot the primary and worker nodes, so it'll be a few minutes
before everything is back up.
shane
--
Shane Knapp
Computer Guy / Voice of Reason
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
quick update:
since kicking httpd on the jenkins master "fixes" the GUI hanging, i set up
a cron job to restart httpd 4 times per day.
this is not the final solution, but will definitely help over the weekend
as i'm heading out of town.
shane
On Fri, Mar 22, 2019 at 9:50 AM shane knapp wrote:
i was right to not hold my breath... while my apache changes seem to have
helped a bit, things are still slowing down after 10-12 hours.
i have a few other things i can look at, and will get as much done as
possible before the weekend. serious troubleshooting will begin anew
monday.
apologies
i tweaked some apache settings (MaxClients increased to fix an error i
found buried in the logs, and added 'retry' and 'acquire' to the reverse
proxy settings to hopefully combat the dreaded 502 response), restarted
httpd and things actually seem quite snappy right now!
i'm not holding my breath,
seems wedged again?
sorry for the bad news Shane, thanks for all the work on fixing it
On Mon, Mar 18, 2019 at 4:02 PM shane knapp wrote:
> ok, i dug through the logs and noticed that rsyslogd was dropping messages
> to do imuxsock being spammed by postfix... which i then tracked down to
>
ok, i dug through the logs and noticed that rsyslogd was dropping messages
to do imuxsock being spammed by postfix... which i then tracked down to
our installation of fail2ban being incorrectly configured and attempting to
send IP ban/unban status emails to 'em...@example.com'.
since we're a
i kicked the service. again. and jenkins seems happy for now.
nothing has changed (system config, packages, etc) on the master node.
i'll dive in to this tomorrow morning.
On Sun, Mar 17, 2019 at 9:46 AM Sean Owen wrote:
> It's wedged against since this morning. Something's clearly gone
>
It's wedged against since this morning. Something's clearly gone
wrong-er than usual; any recent changes that could be a culprit?
On Fri, Mar 15, 2019 at 12:08 PM shane knapp wrote:
>
> well, that box rebooted in record time! we're back up and building.
>
> and as always, i'll keep a close eye
On Fri, Mar 15, 2019 at 8:22 PM Sean Owen wrote:
> It's not responding again. Is there any way to kick it harder? I know
> it's well understood but this means not much can be merged in Spark
>
> it's back up and running now.
btw, the only way to kick it harder would be to do a complete
argh. kicking it again.
On Fri, Mar 15, 2019 at 8:22 PM Sean Owen wrote:
> It's not responding again. Is there any way to kick it harder? I know
> it's well understood but this means not much can be merged in Spark
>
> On Fri, Mar 15, 2019 at 12:08 PM shane knapp wrote:
> >
> > well, that box
It's not responding again. Is there any way to kick it harder? I know
it's well understood but this means not much can be merged in Spark
On Fri, Mar 15, 2019 at 12:08 PM shane knapp wrote:
>
> well, that box rebooted in record time! we're back up and building.
>
> and as always, i'll keep a
cool, thanks!
On Sat, Mar 16, 2019 at 1:08 AM shane knapp wrote:
> well, that box rebooted in record time! we're back up and building.
>
> and as always, i'll keep a close eye on things today... jenkins usually
> works great, until it doesn't. :\
>
> On Fri, Mar 15, 2019 at 9:52 AM shane
well, that box rebooted in record time! we're back up and building.
and as always, i'll keep a close eye on things today... jenkins usually
works great, until it doesn't. :\
On Fri, Mar 15, 2019 at 9:52 AM shane knapp wrote:
> as some of you may have noticed, jenkins got itself in a bad
as some of you may have noticed, jenkins got itself in a bad state multiple
times over the past couple of weeks. usually restarting the service is
sufficient, but it appears that i need to hit it w/the reboot hammer.
jenkins will be down for the next 20-30 minutes as the node reboots and
jenkins
things are building again!
On Thu, Mar 14, 2019 at 2:02 PM shane knapp wrote:
> if performance continues to drop off like this, i will reboot the box soon.
>
> builds should be back to normal in a few minutes... sorry for the
> inconvenience!
>
> --
> Shane Knapp
> UC Berkeley EECS Research /
if performance continues to drop off like this, i will reboot the box soon.
builds should be back to normal in a few minutes... sorry for the
inconvenience!
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
20 matches
Mail list logo
