Re: [graylog2] Slow web interface and overly large JS files

[Richard S. Westmoreland]

Right now the JS client appears to be embedded within the graylog server
jar.  Have you considered moving the js client into its own /client
directory and having the server load the files on demand to serve them?
That way admins can alternatively point their preferred web server to the
client directory to serve it up that way.  This also gives advanced users
an opportunity to play around with the js files.

On Tue, Jan 24, 2017 at 10:06 PM, Richard S. Westmoreland <
richar...@gmail.com> wrote:

> Graylog 2.1.2+50e449a
>
> Accept-Rangesbytes
> Age306654
> Cache-Controlprivate, max-age=604800
> Content-Encodinggzip
> Content-Length704247
> Content-Typeapplication/javascript
> DateSat, 21 Jan 2017 13:16:11 GMT
> EtagW/"7aacedce323be9e3c5edb7dd3a1ad06c37755aefdd960b442379d8d96149
> e12f"
> Last-ModifiedFri, 04 Nov 2016 15:49:36 GMT
> Via1.1 varnish-v4
> X-Graylog-Node-IDnode1-gl
> X-Varnish229488 131075
>
> I'm adding the Cache-Control via haproxy, Content-Encoding via Varnish.
>
>
> On Tue, Jan 24, 2017 at 6:57 AM, Dennis Oelkers  wrote:
>
>> Which version of Graylog are you using? Can you paste the response
>> headers you get when your browser is requesting any of the JS assets (not
>> config.js though)?
>>
>> Kind regards,
>> D.
>>
>> > On 24.01.2017, at 12:11, Richard S. Westmoreland 
>> wrote:
>> >
>> > The compression works, the caching doesn't.  It is possible that
>> haproxy is messing up the etag which is invalidating the browser's cache?
>> >
>> > On Tue, Jan 24, 2017 at 5:54 AM, Dennis Oelkers 
>> wrote:
>> > Hey Richard,
>> >
>> > thanks for the extensive and very valuable feedback. We put some
>> thought into architecting the web interface in a way that it's still usable
>> on slower connections/lower bandwidths and/or high-latency links, but it
>> seems we are still not there. Doing this is always a tradeoff between
>> keeping assets small, providing plugin authors a degree of freedom and
>> providing a rich interface.
>> >
>> > What I am wondering though, is that compression and caching does not
>> work for you. Both have been introduced for the web interface for a long
>> time now (see https://github.com/Graylog2/gr
>> aylog2-server/commit/515c3a242a3fdff1dcb3a4a2fece8351d0060c02 and
>> https://github.com/Graylog2/graylog2-server/commit/68f0a0bc9
>> 508597cc5b81adc0c0b3496b5ac19eb). Do you have any special configuration
>> or components in your environment?
>> >
>> > Kind regards,
>> >
>> >   D.
>> >
>> > On Wednesday, 4 January 2017 04:43:27 UTC+1, Richard S. Westmoreland
>> wrote:
>> > Hello,
>> >
>> > I'm working on putting a GL cluster behind a reverse proxy and load
>> balancer and while troubleshooting a url redirect issue I discovered why
>> the interface has seemed really slow to me.  My servers are in another
>> country so latency becomes obvious for me.
>> >
>> > Just getting to the login screen are 9 requests totally 9.3 MB in
>> transfer.  That is huge.  Individually it may be less of a problem but on
>> the login screen the browser is preloading all these javascript files even
>> when they aren't needed yet, and it is waiting until most of them are
>> loaded before rendering the page.
>> >
>> > My short-term proposal:  At UI login, don't preload.  Just handle
>> authentication first.  Don't load certain JS files until the page that uses
>> it requires it.
>> >
>> > My long-term proposal:  I think the javascript is going to need some
>> refactoring.  For example, I'm looking at the MapWidget code and I see
>> multiple sections of code concatenated together, which is easy to spot
>> because of the sudden break of commented opensource license notices.  If
>> more of these are loaded individually, it may load faster since they will
>> be separate/parallel requests.
>> >
>> > In the mapwidget code I see jQuery UI Widget 1.12.1, which is also
>> being loaded by the enterprise widget, maybe more of them.  So there is a
>> lot of redundant code being downloaded and re-executed.
>> >
>> > Thanks for reading.
>> >
>> > Also is this the best place to post this?  Or is there a dev/eng
>> specific forum I can put these kinds of requests on?
>> >
>> > Thanks,
>> > Rick
>> >
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> Groups "Graylog Users" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> an email to graylog2+unsubscr...@googlegroups.com.
>> > To view this discussion on the web visit https://groups.google.com/d/ms
>> gid/graylog2/0d0cc565-f4b9-4b0a-8e77-cfc2c7f0b2f8%40googlegroups.com.
>> >
>> > For more options, visit https://groups.google.com/d/optout.
>> >
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> Groups "Graylog Users" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> an email to graylog2+unsubscr...@googlegroups.com.
>> > 

Re: [graylog2] Slow web interface and overly large JS files

[Richard S. Westmoreland]

Graylog 2.1.2+50e449a

Accept-Rangesbytes
Age306654
Cache-Controlprivate, max-age=604800
Content-Encodinggzip
Content-Length704247
Content-Typeapplication/javascript
DateSat, 21 Jan 2017 13:16:11 GMT
EtagW/"7aacedce323be9e3c5edb7dd3a1ad06c37755aefdd960b442379d8d96149e12f"
Last-ModifiedFri, 04 Nov 2016 15:49:36 GMT
Via1.1 varnish-v4
X-Graylog-Node-IDnode1-gl
X-Varnish229488 131075

I'm adding the Cache-Control via haproxy, Content-Encoding via Varnish.


On Tue, Jan 24, 2017 at 6:57 AM, Dennis Oelkers  wrote:

> Which version of Graylog are you using? Can you paste the response headers
> you get when your browser is requesting any of the JS assets (not config.js
> though)?
>
> Kind regards,
> D.
>
> > On 24.01.2017, at 12:11, Richard S. Westmoreland 
> wrote:
> >
> > The compression works, the caching doesn't.  It is possible that haproxy
> is messing up the etag which is invalidating the browser's cache?
> >
> > On Tue, Jan 24, 2017 at 5:54 AM, Dennis Oelkers  wrote:
> > Hey Richard,
> >
> > thanks for the extensive and very valuable feedback. We put some thought
> into architecting the web interface in a way that it's still usable on
> slower connections/lower bandwidths and/or high-latency links, but it seems
> we are still not there. Doing this is always a tradeoff between keeping
> assets small, providing plugin authors a degree of freedom and providing a
> rich interface.
> >
> > What I am wondering though, is that compression and caching does not
> work for you. Both have been introduced for the web interface for a long
> time now (see https://github.com/Graylog2/graylog2-server/commit/
> 515c3a242a3fdff1dcb3a4a2fece8351d0060c02 and https://github.com/Graylog2/
> graylog2-server/commit/68f0a0bc9508597cc5b81adc0c0b3496b5ac19eb). Do you
> have any special configuration or components in your environment?
> >
> > Kind regards,
> >
> >   D.
> >
> > On Wednesday, 4 January 2017 04:43:27 UTC+1, Richard S. Westmoreland
> wrote:
> > Hello,
> >
> > I'm working on putting a GL cluster behind a reverse proxy and load
> balancer and while troubleshooting a url redirect issue I discovered why
> the interface has seemed really slow to me.  My servers are in another
> country so latency becomes obvious for me.
> >
> > Just getting to the login screen are 9 requests totally 9.3 MB in
> transfer.  That is huge.  Individually it may be less of a problem but on
> the login screen the browser is preloading all these javascript files even
> when they aren't needed yet, and it is waiting until most of them are
> loaded before rendering the page.
> >
> > My short-term proposal:  At UI login, don't preload.  Just handle
> authentication first.  Don't load certain JS files until the page that uses
> it requires it.
> >
> > My long-term proposal:  I think the javascript is going to need some
> refactoring.  For example, I'm looking at the MapWidget code and I see
> multiple sections of code concatenated together, which is easy to spot
> because of the sudden break of commented opensource license notices.  If
> more of these are loaded individually, it may load faster since they will
> be separate/parallel requests.
> >
> > In the mapwidget code I see jQuery UI Widget 1.12.1, which is also being
> loaded by the enterprise widget, maybe more of them.  So there is a lot of
> redundant code being downloaded and re-executed.
> >
> > Thanks for reading.
> >
> > Also is this the best place to post this?  Or is there a dev/eng
> specific forum I can put these kinds of requests on?
> >
> > Thanks,
> > Rick
> >
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "Graylog Users" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to graylog2+unsubscr...@googlegroups.com.
> > To view this discussion on the web visit https://groups.google.com/d/
> msgid/graylog2/0d0cc565-f4b9-4b0a-8e77-cfc2c7f0b2f8%40googlegroups.com.
> >
> > For more options, visit https://groups.google.com/d/optout.
> >
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "Graylog Users" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to graylog2+unsubscr...@googlegroups.com.
> > To view this discussion on the web visit https://groups.google.com/d/
> msgid/graylog2/CAFnE7wPMyXQNCsnSoyYXPX6qN4UsY9YZLP9TvZQWLoshpHHoPg%40mail.
> gmail.com.
> > For more options, visit https://groups.google.com/d/optout.
>
> --
> Tel.: +49 (0)40 609 452 077
> Fax.: +49 (0)40 609 452 078
>
> TORCH GmbH - A Graylog company
> Poolstrasse 21
> 20355 Hamburg
> Germany
>
> Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175
> Geschäftsführer: Lennart Koopmann (CEO)
>
> --
> You received this message because you are subscribed to the Google Groups
> "Graylog Users" group.
> To unsubscribe from this group and stop 

Re: [graylog2] Slow web interface and overly large JS files

[Dennis Oelkers]

Which version of Graylog are you using? Can you paste the response headers you 
get when your browser is requesting any of the JS assets (not config.js though)?

Kind regards,
D.

> On 24.01.2017, at 12:11, Richard S. Westmoreland  wrote:
> 
> The compression works, the caching doesn't.  It is possible that haproxy is 
> messing up the etag which is invalidating the browser's cache?
> 
> On Tue, Jan 24, 2017 at 5:54 AM, Dennis Oelkers  wrote:
> Hey Richard,
> 
> thanks for the extensive and very valuable feedback. We put some thought into 
> architecting the web interface in a way that it's still usable on slower 
> connections/lower bandwidths and/or high-latency links, but it seems we are 
> still not there. Doing this is always a tradeoff between keeping assets 
> small, providing plugin authors a degree of freedom and providing a rich 
> interface.
> 
> What I am wondering though, is that compression and caching does not work for 
> you. Both have been introduced for the web interface for a long time now (see 
> https://github.com/Graylog2/graylog2-server/commit/515c3a242a3fdff1dcb3a4a2fece8351d0060c02
>  and 
> https://github.com/Graylog2/graylog2-server/commit/68f0a0bc9508597cc5b81adc0c0b3496b5ac19eb).
>  Do you have any special configuration or components in your environment?
> 
> Kind regards,
> 
>   D.
> 
> On Wednesday, 4 January 2017 04:43:27 UTC+1, Richard S. Westmoreland wrote:
> Hello,
> 
> I'm working on putting a GL cluster behind a reverse proxy and load balancer 
> and while troubleshooting a url redirect issue I discovered why the interface 
> has seemed really slow to me.  My servers are in another country so latency 
> becomes obvious for me.
> 
> Just getting to the login screen are 9 requests totally 9.3 MB in transfer.  
> That is huge.  Individually it may be less of a problem but on the login 
> screen the browser is preloading all these javascript files even when they 
> aren't needed yet, and it is waiting until most of them are loaded before 
> rendering the page.
> 
> My short-term proposal:  At UI login, don't preload.  Just handle 
> authentication first.  Don't load certain JS files until the page that uses 
> it requires it.
> 
> My long-term proposal:  I think the javascript is going to need some 
> refactoring.  For example, I'm looking at the MapWidget code and I see 
> multiple sections of code concatenated together, which is easy to spot 
> because of the sudden break of commented opensource license notices.  If more 
> of these are loaded individually, it may load faster since they will be 
> separate/parallel requests.
> 
> In the mapwidget code I see jQuery UI Widget 1.12.1, which is also being 
> loaded by the enterprise widget, maybe more of them.  So there is a lot of 
> redundant code being downloaded and re-executed.
> 
> Thanks for reading.
> 
> Also is this the best place to post this?  Or is there a dev/eng specific 
> forum I can put these kinds of requests on?
> 
> Thanks,
> Rick
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Graylog Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to graylog2+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/graylog2/0d0cc565-f4b9-4b0a-8e77-cfc2c7f0b2f8%40googlegroups.com.
> 
> For more options, visit https://groups.google.com/d/optout.
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Graylog Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to graylog2+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/graylog2/CAFnE7wPMyXQNCsnSoyYXPX6qN4UsY9YZLP9TvZQWLoshpHHoPg%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.

--
Tel.: +49 (0)40 609 452 077
Fax.: +49 (0)40 609 452 078

TORCH GmbH - A Graylog company
Poolstrasse 21
20355 Hamburg
Germany

Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175
Geschäftsführer: Lennart Koopmann (CEO)

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/2F866693-2B44-412C-9AD7-B1A0B84BC45C%40graylog.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Slow web interface and overly large JS files

[Richard S. Westmoreland]

Hello,

I'm working on putting a GL cluster behind a reverse proxy and load 
balancer and while troubleshooting a url redirect issue I discovered why 
the interface has seemed really slow to me.  My servers are in another 
country so latency becomes obvious for me.

Just getting to the login screen are 9 requests totally 9.3 MB in 
transfer.  That is huge.  Individually it may be less of a problem but on 
the login screen the browser is preloading all these javascript files even 
when they aren't needed yet, and it is waiting until most of them are 
loaded before rendering the page.

My short-term proposal:  At UI login, don't preload.  Just handle 
authentication first.  Don't load certain JS files until the page that uses 
it requires it.

My long-term proposal:  I think the javascript is going to need some 
refactoring.  For example, I'm looking at the MapWidget code and I see 
multiple sections of code concatenated together, which is easy to spot 
because of the sudden break of commented opensource license notices.  If 
more of these are loaded individually, it may load faster since they will 
be separate/parallel requests.

In the mapwidget code I see jQuery UI Widget 1.12.1, which is also being 
loaded by the enterprise widget, maybe more of them.  So there is a lot of 
redundant code being downloaded and re-executed.

Thanks for reading.

Also is this the best place to post this?  Or is there a dev/eng specific 
forum I can put these kinds of requests on?

Thanks,
Rick

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/f7e58985-66a8-4da1-93ee-f23a4a5a3289%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.