Re: Log filter / display dashboard - Looking for recommendation [Slightly off topic]

[Corneliu I. Tusnea]

Greg,

I've been looking into this as well and I'm not sure if there is a
one-size-fits all.

We've been using Seq in our app and we ingest ~1-6gb logs a day. Found seq
ok but if you search ... well, anything past 24-48h is very very slow. We
did various optimizations and move lots of the logs out of seq but still
not going where we wanted it.
I'm sure with a bigger machine we could have made it faster but I can't pay
$500 for a VM ingesting logs.
What I think seq is missing is a way to declare some sort of indexes
because now it indexes & searches everything. I only have 3-4 columns that
are my Primary Key and 99% of the searches have one of those columns in
them.

We've moved some of our logs into Azure Tables. Cheap and chips. Because we
have the PK as above we made those PK in Azure Tables. Search is fast,
maybe a bit limited as we have to search for stuff like contains in the
strings but cheap and scalable.

We area also using Azure Application Insighs for everything not log related
(events, metrics, ...) that we report from inside the app.

I'm now looking to deploy this on top of Application Insights:
https://www.microsoft.com/developerblog/2017/09/26/custom-analytics-dashboard-application-insights/

Looks very customizable and if you track the right data you migth pull out
the right reports.
I don't think you want to generate reports from logs. I would add telemetry
information inside the app using App Insights and use that for reports and
use the logs purely for diagnosing purposes.

Regards,
Corneliu.













On Tue, Mar 13, 2018 at 10:52 AM, Greg Harris 
wrote:

> Hi All,
>
> Question: Can you recommend a standard dashboard app that takes multiple
> logs, filters out the 99% that is not relevant, works out a status,
> displays a panel of Green, Amber, Red blobs with hyperlinks to detailed log
> info?
>
> Details:
>
> I am doing an architecture consulting gig for a client (a mostly MS
> Windows / Azure site, but there are some Unix servers out there as well).
>
> They have about a dozen core applications that the business is highly
> dependent on and another 100 or so that they are less dependent on.
>
> The problem being that when something goes wrong, vendor A says “the
> problem is with vendor B’s system” and you know what vendor B says.  What
> makes this worse is that the chain goes through multiple vendors and
> systems with long inter system data flows.
>
> I want to make the recommendation to them that they implement logging and
> a top level log display dashboard for the systems they are highly dependent
> on.  The dashboard would show:
>
> 1.   Dataflow around the systems
>
> 2.   Work backlog at each sub system
>
> 3.   Status at each sub system
>
> 4.   Time since last ping at each sub system
>
> This feels to me to be a standard sort of system that every major site
> needs!
>
> I am not sure what to recommend???
>
> Question: Can you recommend a standard dashboard app that takes multiple
> logs, filters out the 99% that is not relevant, works out a status,
> displays a panel of Green, Amber, Red blobs with hyperlinks to detailed log
> info?
>
> I am thinking of Nicholas Blumhardt’s Seq application (https://getseq.net/)
> or Datadog (https://www.datadoghq.com/) but neither of these feel quite
> complete to me.
>
> We will need a hierarchy solution:
>
> 1.   App logging (has to be baked into existing and new apps)
>
> 2.   Forwarded to Site Logger (maybe baked into the app, or as an add
> on)
>
> 3.   Site Logger
>
> 4.   Filter
>
> 5.   Analysis
>
> 6.   Dashboard display
>
> At level 1 if logging needs to be added to an existing or new system, I am
> thinking that the recommendation will be to use Serilog (
> https://github.com/serilog) or Datadog (https://www.datadoghq.com/).
>
> If the app already has logging, I am thinking that some form of log
> forwarder is needed, I want to avoid paying for apps to be modified to meet
> this need if they already have logging.
>
> Levels 2-6 could be should be a standard off the shelf app.
>
> Any advice would be greatly appreciated.
>
> Best Regards
>
> Greg Harris
> Greg Harris
> harris.gre...@gmail.com
>
>
>
>

Re: Log filter / display dashboard - Looking for recommendation [Slightly off topic]

[Stephen Price]

By coincidence, the latest dotnet rocks episode is all about dashboards.

Listened to half of it on the way to work today. Might help you...



From: ozdotnet-boun...@ozdotnet.com <ozdotnet-boun...@ozdotnet.com> on behalf 
of Greg Low <g...@greglow.com>
Sent: Tuesday, March 13, 2018 8:14:11 AM
To: ozDotNet
Subject: Re: Log filter / display dashboard - Looking for recommendation 
[Slightly off topic]

Have you looked at Azure Log Analytics ?

Easy to ingest bunches of logs, good query language, basic dashboards, easy to 
integrate with things like Power BI, has a REST based interface, cheap, etc

Regards,

Greg

Dr Greg Low
1300SQLSQL (1300 775 775) office | +61 419201410 mobile│ +61 3 8676 4913 fax
SQL Down Under | Web: www.sqldownunder.com

From: ozdotnet-boun...@ozdotnet.com <ozdotnet-boun...@ozdotnet.com> on behalf 
of Greg Harris <harris.gre...@gmail.com>
Sent: Tuesday, March 13, 2018 10:52:43 AM
To: ozDotNet
Subject: Log filter / display dashboard - Looking for recommendation [Slightly 
off topic]

Hi All,
Question: Can you recommend a standard dashboard app that takes multiple logs, 
filters out the 99% that is not relevant, works out a status, displays a panel 
of Green, Amber, Red blobs with hyperlinks to detailed log info?
Details:
I am doing an architecture consulting gig for a client (a mostly MS Windows / 
Azure site, but there are some Unix servers out there as well).
They have about a dozen core applications that the business is highly dependent 
on and another 100 or so that they are less dependent on.
The problem being that when something goes wrong, vendor A says “the problem is 
with vendor B’s system” and you know what vendor B says.  What makes this worse 
is that the chain goes through multiple vendors and systems with long inter 
system data flows.
I want to make the recommendation to them that they implement logging and a top 
level log display dashboard for the systems they are highly dependent on.  The 
dashboard would show:

1.   Dataflow around the systems

2.   Work backlog at each sub system

3.   Status at each sub system

4.   Time since last ping at each sub system
This feels to me to be a standard sort of system that every major site needs!
I am not sure what to recommend???
Question: Can you recommend a standard dashboard app that takes multiple logs, 
filters out the 99% that is not relevant, works out a status, displays a panel 
of Green, Amber, Red blobs with hyperlinks to detailed log info?
I am thinking of Nicholas Blumhardt’s Seq application (https://getseq.net/) or 
Datadog (https://www.datadoghq.com/) but neither of these feel quite complete 
to me.
We will need a hierarchy solution:

1.   App logging (has to be baked into existing and new apps)

2.   Forwarded to Site Logger (maybe baked into the app, or as an add on)

3.   Site Logger

4.   Filter

5.   Analysis

6.   Dashboard display
At level 1 if logging needs to be added to an existing or new system, I am 
thinking that the recommendation will be to use Serilog 
(https://github.com/serilog) or Datadog (https://www.datadoghq.com/).
If the app already has logging, I am thinking that some form of log forwarder 
is needed, I want to avoid paying for apps to be modified to meet this need if 
they already have logging.
Levels 2-6 could be should be a standard off the shelf app.
Any advice would be greatly appreciated.
Best Regards
Greg Harris

Greg Harris
harris.gre...@gmail.com<mailto:harris.gre...@gmail.com>

Re: Log filter / display dashboard - Looking for recommendation [Slightly off topic]

[Jorke Odolphi]

check out elastic - seems like a logstash/elasticsearch/kibana would produce 
what you need fairly easily.



> On 13 Mar 2018, at 10:52 am, Greg Harris  wrote:
> 
> Hi All,
> 
> Question: Can you recommend a standard dashboard app that takes multiple 
> logs, filters out the 99% that is not relevant, works out a status, displays 
> a panel of Green, Amber, Red blobs with hyperlinks to detailed log info?
> 
> Details:
> 
> I am doing an architecture consulting gig for a client (a mostly MS Windows / 
> Azure site, but there are some Unix servers out there as well).
> 
> They have about a dozen core applications that the business is highly 
> dependent on and another 100 or so that they are less dependent on.
> 
> The problem being that when something goes wrong, vendor A says “the problem 
> is with vendor B’s system” and you know what vendor B says.  What makes this 
> worse is that the chain goes through multiple vendors and systems with long 
> inter system data flows.
> 
> I want to make the recommendation to them that they implement logging and a 
> top level log display dashboard for the systems they are highly dependent on. 
>  The dashboard would show:
> 
> 1.   Dataflow around the systems
> 2.   Work backlog at each sub system
> 3.   Status at each sub system
> 4.   Time since last ping at each sub system
> 
> This feels to me to be a standard sort of system that every major site needs!
> 
> I am not sure what to recommend???
> 
> Question: Can you recommend a standard dashboard app that takes multiple 
> logs, filters out the 99% that is not relevant, works out a status, displays 
> a panel of Green, Amber, Red blobs with hyperlinks to detailed log info?
> 
> I am thinking of Nicholas Blumhardt’s Seq application (https://getseq.net/) 
> or Datadog (https://www.datadoghq.com/) but neither of these feel quite 
> complete to me.
> 
> We will need a hierarchy solution:
> 
> 1.   App logging (has to be baked into existing and new apps)
> 2.   Forwarded to Site Logger (maybe baked into the app, or as an add on)
> 3.   Site Logger
> 4.   Filter
> 5.   Analysis
> 6.   Dashboard display
> 
> At level 1 if logging needs to be added to an existing or new system, I am 
> thinking that the recommendation will be to use Serilog 
> (https://github.com/serilog) or Datadog (https://www.datadoghq.com/). 
> 
> If the app already has logging, I am thinking that some form of log forwarder 
> is needed, I want to avoid paying for apps to be modified to meet this need 
> if they already have logging.
> 
> Levels 2-6 could be should be a standard off the shelf app.
> 
> Any advice would be greatly appreciated.
> 
> Best Regards
> 
> Greg Harris
> 
> Greg Harris
> harris.gre...@gmail.com
> 
> 
>  

Re: Log filter / display dashboard - Looking for recommendation [Slightly off topic]

[Greg Low]

Have you looked at Azure Log Analytics ?

Easy to ingest bunches of logs, good query language, basic dashboards, easy to 
integrate with things like Power BI, has a REST based interface, cheap, etc

Regards,

Greg

Dr Greg Low
1300SQLSQL (1300 775 775) office | +61 419201410 mobile│ +61 3 8676 4913 fax
SQL Down Under | Web: www.sqldownunder.com

From: ozdotnet-boun...@ozdotnet.com <ozdotnet-boun...@ozdotnet.com> on behalf 
of Greg Harris <harris.gre...@gmail.com>
Sent: Tuesday, March 13, 2018 10:52:43 AM
To: ozDotNet
Subject: Log filter / display dashboard - Looking for recommendation [Slightly 
off topic]

Hi All,
Question: Can you recommend a standard dashboard app that takes multiple logs, 
filters out the 99% that is not relevant, works out a status, displays a panel 
of Green, Amber, Red blobs with hyperlinks to detailed log info?
Details:
I am doing an architecture consulting gig for a client (a mostly MS Windows / 
Azure site, but there are some Unix servers out there as well).
They have about a dozen core applications that the business is highly dependent 
on and another 100 or so that they are less dependent on.
The problem being that when something goes wrong, vendor A says “the problem is 
with vendor B’s system” and you know what vendor B says.  What makes this worse 
is that the chain goes through multiple vendors and systems with long inter 
system data flows.
I want to make the recommendation to them that they implement logging and a top 
level log display dashboard for the systems they are highly dependent on.  The 
dashboard would show:

1.   Dataflow around the systems

2.   Work backlog at each sub system

3.   Status at each sub system

4.   Time since last ping at each sub system
This feels to me to be a standard sort of system that every major site needs!
I am not sure what to recommend???
Question: Can you recommend a standard dashboard app that takes multiple logs, 
filters out the 99% that is not relevant, works out a status, displays a panel 
of Green, Amber, Red blobs with hyperlinks to detailed log info?
I am thinking of Nicholas Blumhardt’s Seq application (https://getseq.net/) or 
Datadog (https://www.datadoghq.com/) but neither of these feel quite complete 
to me.
We will need a hierarchy solution:

1.   App logging (has to be baked into existing and new apps)

2.   Forwarded to Site Logger (maybe baked into the app, or as an add on)

3.   Site Logger

4.   Filter

5.   Analysis

6.   Dashboard display
At level 1 if logging needs to be added to an existing or new system, I am 
thinking that the recommendation will be to use Serilog 
(https://github.com/serilog) or Datadog (https://www.datadoghq.com/).
If the app already has logging, I am thinking that some form of log forwarder 
is needed, I want to avoid paying for apps to be modified to meet this need if 
they already have logging.
Levels 2-6 could be should be a standard off the shelf app.
Any advice would be greatly appreciated.
Best Regards
Greg Harris

Greg Harris
harris.gre...@gmail.com<mailto:harris.gre...@gmail.com>

Re: Log filter / display dashboard - Looking for recommendation [Slightly off topic]

[Stephen Price]

My fairly limited use of seq is that it's more a centralised log collection and 
search tool than a dashboard. But it might work for you if it's just logs.

I have used SharePoint to present dashboards in the past, customising the parts 
to collect the desired info and present In a list.

Not familiar with any off the shelf solutions. We have plans to add health end 
points to the system which will call through each layer and report what's up 
and what's not. Will be a custom dashboard to display the health of each 
component but most of them are in our control. Not sure if there is a standard 
for reporting health of a system, would be useful.

Sorry if that's not much help

Cheers
Stephen




From: Greg Harris
Sent: Tuesday, 13 March, 7:52 am
Subject: Log filter / display dashboard - Looking for recommendation 
[Slightlyoff topic]
To: ozDotNet


Hi All,
Question: Can you recommend a standard dashboard app that takes multiple logs, 
filters out the 99% that is not relevant, works out a status, displays a panel 
of Green, Amber, Red blobs with hyperlinks to detailed log info?
Details:
I am doing an architecture consulting gig for a client (a mostly MS Windows / 
Azure site, but there are some Unix servers out there as well).
They have about a dozen core applications that the business is highly dependent 
on and another 100 or so that they are less dependent on.
The problem being that when something goes wrong, vendor A says “the problem is 
with vendor B’s system” and you know what vendor B says.  What makes this worse 
is that the chain goes through multiple vendors and systems with long inter 
system data flows.
I want to make the recommendation to them that they implement logging and a top 
level log display dashboard for the systems they are highly dependent on.  The 
dashboard would show:
1.   Dataflow around the systems
2.   Work backlog at each sub system
3.   Status at each sub system
4.   Time since last ping at each sub system
This feels to me to be a standard sort of system that every major site needs!
I am not sure what to recommend???
Question: Can you recommend a standard dashboard app that takes multiple logs, 
filters out the 99% that is not relevant, works out a status, displays a panel 
of Green, Amber, Red blobs with hyperlinks to detailed log info?
I am thinking of Nicholas Blumhardt’s Seq application (https://getseq.net/) or 
Datadog (https://www.datadoghq.com/) but neither of these feel quite complete 
to me.
We will need a hierarchy solution:
1.   App logging (has to be baked into existing and new apps)
2.   Forwarded to Site Logger (maybe baked into the app, or as an add on)
3.   Site Logger
4.   Filter
5.   Analysis
6.   Dashboard display
At level 1 if logging needs to be added to an existing or new system, I am 
thinking that the recommendation will be to use Serilog 
(https://github.com/serilog) or Datadog (https://www.datadoghq.com/).
If the app already has logging, I am thinking that some form of log forwarder 
is needed, I want to avoid paying for apps to be modified to meet this need if 
they already have logging.
Levels 2-6 could be should be a standard off the shelf app.
Any advice would be greatly appreciated.
Best Regards
Greg Harris
Greg Harris

harris.gre...@gmail.com

Log filter / display dashboard - Looking for recommendation [Slightly off topic]

[Greg Harris]

Hi All,

Question: Can you recommend a standard dashboard app that takes multiple
logs, filters out the 99% that is not relevant, works out a status,
displays a panel of Green, Amber, Red blobs with hyperlinks to detailed log
info?

Details:

I am doing an architecture consulting gig for a client (a mostly MS Windows
/ Azure site, but there are some Unix servers out there as well).

They have about a dozen core applications that the business is highly
dependent on and another 100 or so that they are less dependent on.

The problem being that when something goes wrong, vendor A says “the
problem is with vendor B’s system” and you know what vendor B says.  What
makes this worse is that the chain goes through multiple vendors and
systems with long inter system data flows.

I want to make the recommendation to them that they implement logging and a
top level log display dashboard for the systems they are highly dependent
on.  The dashboard would show:

1.   Dataflow around the systems

2.   Work backlog at each sub system

3.   Status at each sub system

4.   Time since last ping at each sub system

This feels to me to be a standard sort of system that every major site
needs!

I am not sure what to recommend???

Question: Can you recommend a standard dashboard app that takes multiple
logs, filters out the 99% that is not relevant, works out a status,
displays a panel of Green, Amber, Red blobs with hyperlinks to detailed log
info?

I am thinking of Nicholas Blumhardt’s Seq application (https://getseq.net/)
or Datadog (https://www.datadoghq.com/) but neither of these feel quite
complete to me.

We will need a hierarchy solution:

1.   App logging (has to be baked into existing and new apps)

2.   Forwarded to Site Logger (maybe baked into the app, or as an add
on)

3.   Site Logger

4.   Filter

5.   Analysis

6.   Dashboard display

At level 1 if logging needs to be added to an existing or new system, I am
thinking that the recommendation will be to use Serilog (
https://github.com/serilog) or Datadog (https://www.datadoghq.com/).

If the app already has logging, I am thinking that some form of log
forwarder is needed, I want to avoid paying for apps to be modified to meet
this need if they already have logging.

Levels 2-6 could be should be a standard off the shelf app.

Any advice would be greatly appreciated.

Best Regards

Greg Harris
Greg Harris
harris.gre...@gmail.com