Re: [pfSense] Suricata sync crashes WebConfigurator, and other issues

2016-01-18 Thread Steve Yates 
> Not sure offhand whether Suricata is even usable in 2.3, but that
> might be worth a shot.

Given that we're using CARP, if we install it on our router2 to test, how long 
would you recommend running router2 on 2.3 and router1 on 2.2.6?  Generally 
I've not waited more than a few minutes between upgrading, though we've usually 
upgraded our office router first and tested there.

Another question...for syncing Suricata, and/or the configuration sync, would 
you recommend using the pfSync interface, or the LAN interface?  Or does it 
matter?  I've tried both and it didn't help my issue...

Steve Yates
___
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold

Re: [pfSense] Suricata sync crashes WebConfigurator, and other issues

2016-01-17 Thread Steve Yates 
Chris Buechler wrote on Sat, Jan 16 2016 at 2:23 am:

> The fact you're hitting at least one lighttpd crash makes me think
> there's some other issue there, though no one else has seen any issues
> in 2.2.6, the issue in 2.2.5 wasn't replicable in most cases either.
> There's a reason nginx is now the web server in 2.3.
> 
> That could be an issue in the Suricata package, given the web server
> only crashed once it appears. Since you end up in a situation where
> you're stuck until restarting php-fpm, that points to the issue being
> in PHP, though an issue in lighttpd could impact PHP.

If I step back and look at the big picture it kind of got worse over 
time.  It started off that restarting webConfigurator seemed to fix it, at 
least letting me log in to the web GUI and syncing for a while afterwards.  
Then restarting webConfigurator had no effect and restarting PHP-FPM would 
immediately yield an HTTP error (usually 500).  And then Friday night it seemed 
like I had to restart the entire router to get to the web GUI.

Is it conceivable that a temporary problem would survive restarting 
webConfigurator and PHP-FPM?  I don't understand how.  I'd guess Suricata was 
left running but the log says "Restarting/Starting all packages" at every 
firewall sync.

I'd ask if someone with a couple of routers/VMs could install Suricata, 
enable some rule sets, disable all the rules in 
emerging-web_specific_apps.rules and try to duplicate it, but un-disabling them 
didn't fix the problem.  Although I probably had not yet restarted our router2 
at that point either, come to think of it.

It's even weirder that a "successful" sync can be 1-4 seconds or 3 
minutes.  It does make me think the issue is with Suricata, but ideally 
whatever the issue is shouldn't block access to the web GUI.  Luckily I can get 
to the router's console.

Is there a way to get lighttpd to log errors?  I was poking around 
while logged into the console but its log was blank (as I recall now).

> Not sure offhand whether Suricata is even usable in 2.3, but that
> might be worth a shot.

Hmmm, we don't have a long history with packages.  I was kind of 
assuming it would just work with new versions. :)  Will have to test it out 
first.  Usually I don't hurry to upgrade without a reason but I've never had a 
problem upgrading 2.x versions.  That said I read the changelog-in-progress for 
2.3 and it looks like a big overhaul.

--

Steve Yates
ITS, Inc.


___
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold

Re: [pfSense] Suricata sync crashes WebConfigurator, and other issues

2016-01-16 Thread Chris Buechler 
On Fri, Jan 15, 2016 at 11:59 PM, Steve Yates  wrote:
> I don't like leaving things not fully stable so I bit the bullet and 
> clicked "Remove Enable/Disable changes in the current Category" so it would 
> at least sync.  To my surprise it did not help, even after doing it on 
> router2 as well.  Then I noticed the CARP sync was also starting to fail.
>
> After thinking about it a bit I restarted router2 and syncing 
> immediately worked again.  That implies something was wrong with the XMLRPC 
> sync that wasn't fixed by restarting webConfigurator and/or PHP-FPM.  Notably 
> there was a config sync fix included in pfSense 2.2.6...
>

That was strictly the upgrade to lighttpd to fix a regression they
introduced in the updated version new in 2.2.5.
http://redmine.lighttpd.net/issues/2670

The fact you're hitting at least one lighttpd crash makes me think
there's some other issue there, though no one else has seen any issues
in 2.2.6, the issue in 2.2.5 wasn't replicable in most cases either.
There's a reason nginx is now the web server in 2.3.

That could be an issue in the Suricata package, given the web server
only crashed once it appears. Since you end up in a situation where
you're stuck until restarting php-fpm, that points to the issue being
in PHP, though an issue in lighttpd could impact PHP.

Not sure offhand whether Suricata is even usable in 2.3, but that
might be worth a shot.

If you want to troubleshoot the sync, maybe the easiest way is to
switch to HTTP temporarily, packet capture the config sync traffic,
follow TCP stream in Wireshark. That's usually telling to at least
narrow it down much more.
___
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold

Re: [pfSense] Suricata sync crashes WebConfigurator, and other issues

2016-01-15 Thread Steve Yates 
I don't like leaving things not fully stable so I bit the bullet and 
clicked "Remove Enable/Disable changes in the current Category" so it would at 
least sync.  To my surprise it did not help, even after doing it on router2 as 
well.  Then I noticed the CARP sync was also starting to fail.

After thinking about it a bit I restarted router2 and syncing 
immediately worked again.  That implies something was wrong with the XMLRPC 
sync that wasn't fixed by restarting webConfigurator and/or PHP-FPM.  Notably 
there was a config sync fix included in pfSense 2.2.6...

I noticed another interesting tidbit.  The first Suricata sync after 
the restart I used a hostname (to router2's LAN IP).  The sync took 4 seconds.  
I then changed to an IP address.  It succeeded but took just shy of 3 minutes.  
Back to the hostname...1 second.  Back to the IP...timeouts and "Code 2: 
Invalid return payload."  At that point I had to restart router2 again.

I can't imagine using a hostname makes any practical difference.  I had 
started with an IP for the Suricata sync because the High Availability Sync 
page says to use an IP.

I did notice that the pfSense config sync triggers a route reload and 
down/up of the OpenVPN interface (which isn't connected), and the OpenVPN 
down/up logs, in order:

/rc.newwanip: rc.newwanip: Info: starting on ovpns1.
/rc.newwanip: rc.newwanip: on (IP address: 192.168.199.1) (interface: []) (real 
interface: ovpns1).
check_reload_status: Reloading filter
php-fpm[49144]: /rc.newwanip: pfSense package system has detected an IP change 
or dynamic WAN reconnection - -> 192.168.199.1 - Restarting packages.
check_reload_status: Starting packages
/rc.start_packages: Restarting/Starting all packages.

...maybe "restarting packages" is interfering with the Suricata sync?

Or possibly the default Suricata sync timeout of 150 seconds needs to 
be a *lot* higher?

--

Steve Yates
ITS, Inc.


___
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold

Re: [pfSense] Suricata sync crashes WebConfigurator, and other issues

2016-01-15 Thread Steve Yates 
Steve Yates wrote on Tue, Jan 12 2016 at 1:25 am:

> 6) I started on pfSense 2.2.5 and upgraded both routers to 2.2.6 since it 
> said it
> fixed some sync issues.  On at least two occasions, with 2.2.6, I start 
> getting
> "unread notice" alerts for sync errors, and can't connect to the web GUI on
> router2.  Connecting to its console and choosing "Restart webConfigurator"
> (option 11) fixes both issues, as if the web browser crashed.

It happened just now and the General log on router2 shows:

Jan 15 18:37:23 kernel: pid 17318 (lighttpd), uid 0: exited on signal 
11 (core dumped)

...however that usually doesn't get logged, and I just see my restart 
("lighttpd[33922]: (log.c.194) server started").

At this point, if I open the Suricata Sync tab, click Save, and within 
a minute or so router2's web GUI crashes again.  Interestingly, the last few 
times if I restart webConfigurator I still can't connect but if I restart 
PHP-FPM I instantly get a 500 - Internal Server Error page. Does that imply a 
PHP problem?

I am thinking it can't handle having most of the rules in 
emerging-web_specific_apps.rules disabled...too many things to update?  A 
memory limit somewhere? (PHP's is 256 MB)

Does anyone know if "Enable all rules in the current Category" will 
reset the rule state back to default, or mark them all enabled (which won't 
help any, if my theory is correct)?  Is there a way to set "Disable all rules 
in the current Category" back to the default but keep any changes?  " Remove 
Enable/Disable changes in the current Category" sounds like it will undo all my 
changes.  :-/

--

Steve Yates
ITS, Inc.





___
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold

[pfSense] Suricata sync crashes WebConfigurator, and other issues

2016-01-11 Thread Steve Yates 
I've been working on implementing Suricata (package 2.1.9.1) on a CARP 
dual router setup, and Suricata is set to sync to router2 as well.  I have 
several issues, the worst of which ends with me unable to connect to router2 
via a browser (and of course sync fails).

1) Agonizingly slow page loads.
I'm trying to enable only certain emerging-web_specific_apps.rules rules. I 
disabled all rules, and am going through and enabling certain ones that apply.  
There are several thousand rules in that category, so it is a big page*.  If I 
enable a rule, sometimes the page reloads in a few seconds. Sometimes it takes 
several minutes.  Sometimes I can enable 20 in a row, fast, and then it slows 
down again.  I don't understand the discrepancy.  It is so slow I can watch the 
table draw if I scroll to the bottom of what's loaded.  While it's loading, 
other pages from the router load fine, e.g. the index.php page loads 
immediately and shows 0% CPU usage, 30% memory usage (it's a 4 CPU VM with 2 GB 
RAM, on a 100 Mbps connection).  Other connections *through* this router are 
normal.

2) I have found that despite two Apply buttons on the "Suricata: Interface WAN 
- Rules: " page it syncs every change to router2 anyway, every time a rule 
is enabled.  It seems slightly faster to turn off syncing but not several 
minutes faster (and then enable it at the end, which immediately syncs).

3) CARP syncs at every Suricata rule enable also , even though Suricata has its 
own sync.  QUESTION: do I need the Suricata sync enabled if the CARP sync is 
enabled?

4) If I disable the CARP configuration sync (leaving state sync enabled) the 
super slow page loads go away for a while.  However they come back so it does 
not 100% fix the problem of the several-minute page loads.

5) Occasionally, clicking on the Enable icon sends me directly to the router's 
index.php page as if something crashed.  I would say it is rare, but just now 
it happened 4 times inside of a few minutes.  It can happen even if I wait a 
couple minutes after the page loads before clicking an Enable icon.  What would 
cause this redirect?  Shouldn't pfSense show an error page if an error is 
happening?

6) I started on pfSense 2.2.5 and upgraded both routers to 2.2.6 since it said 
it fixed some sync issues.  On at least two occasions, with 2.2.6, I start 
getting "unread notice" alerts for sync errors, and can't connect to the web 
GUI on router2.  Connecting to its console and choosing "Restart 
webConfigurator" (option 11) fixes both issues, as if the web browser crashed.

7) I don't know if this is relevant but when each and every CARP sync happens, 
router2 logs the following.  The 192.168.199.1 IP address is in the tunnel 
network for OpenVPN, which is not connected.

Jan 12 00:39:47 php-fpm[26893]: /rc.start_packages: Restarting/Starting 
all packages.
Jan 12 00:39:46 check_reload_status: Starting packages
Jan 12 00:39:46 php-fpm[26893]: /rc.newwanip: pfSense package system 
has detected an IP change or dynamic WAN reconnection - -> 192.168.199.1 - 
Restarting packages.
Jan 12 00:39:46 check_reload_status: Reloading filter
Jan 12 00:39:46 php-fpm[26893]: /rc.newwanip: rc.newwanip: on (IP 
address: 192.168.199.1) (interface: []) (real interface: ovpns1).
Jan 12 00:39:46 php-fpm[26893]: /rc.newwanip: rc.newwanip: Info: 
starting on ovpns1.
Jan 12 00:39:45 check_reload_status: rc.newwanip starting ovpns1
Jan 12 00:39:45 kernel: ovpns1: link state changed to UP
Jan 12 00:39:44 check_reload_status: Reloading filter
Jan 12 00:39:44 kernel: ovpns1: link state changed to DOWN
Jan 12 00:39:44 php-fpm[19360]: /xmlrpc.php: Resyncing OpenVPN 
instances.
Jan 12 00:39:44 php-fpm[19360]: /xmlrpc.php: ROUTING: setting IPv6 
default route to [IPv6 WAN gateway]
Jan 12 00:39:44 php-fpm[19360]: /xmlrpc.php: ROUTING: setting default 
route to [IPv4 WAN gateway]
Jan 12 00:39:44 check_reload_status: Reloading filter
Jan 12 00:39:44 check_reload_status: Syncing firewall




* small JavaScript tip: define a function for document.getElementById like so 
and it will save a lot of repeated text on a page that big:
function x() {
return document.getElementById(arguments[0]);
}

--

Steve Yates
ITS, Inc.



___
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold