Advise in optimizing an application

2014-11-14 Thread Les Mizzell
Evening, I'm still fighting with my Excel Import application a bit. Basically, it works pretty well, as long as there's a limited number of records being imported. Problem is, the client tried to import a list with almost 15,000 addresses today, and it eventually timed out or the server reset

Re: Advise in optimizing an application

2014-11-15 Thread Dan G. Switzer, II
You need to turn use some debugging tools to help you identify the key culprits. For example, if your query to check if a user has unsubscribed is taking 500ms, that alone would be the reason for the slowness. Either use the step debugger, a tool like FusionReactor or just turn on verbose debuggi

Re: Advise in optimizing an application

2014-11-15 Thread Les Mizzell
> I'm still fighting with my Excel Import application a > bit. Basically, it works pretty well, as long as > there's a limited number of records being imported. Was testing last night with the 15,000 record Excel sheet. If I comment out the two filter queries in the code, it will read in the s

Re: Advise in optimizing an application

2014-11-15 Thread Michael Grant
Do you have indexes on ml_email and groups_id columns? I would likely create an index that has both in it. There's a couple of suggestions about changing your architecture a bit: First, from the looks of it I think you could probably combine both these queries into one with a left join vw_mailLI

Re: Advise in optimizing an application

2014-11-15 Thread Roger Austin
Any chance to have the database engine do all that record logic? That would be the first thing I would try. Stored procedures are great for things like this. I don't know what RDMS you are using, but most have SQL that can do this. Complex database stuff is usually better off doing within the d

Re: Advise in optimizing an application

2014-11-15 Thread Les Mizzell
On 11/15/2014 12:42 PM, Roger Austin wrote: > Any chance to have the database engine do all that record logic? After killing myself on this, I finally realiaed I was doing it ALL WRONG! All those loops and everything to filter the array, then do the insertwas taking forever. I'm revising - BULK

Re: Advise in optimizing an application

2014-11-15 Thread Roger Austin
You might try something like this SELECT * FROM t1 WHERE NOT EXISTS (SELECT * FROM t2 WHERE t1.email = t2.email); Without being more familiar with your use case, it is very difficult to suggest much. Les Mizzell wrote: > > On 11/15/2014 12:42 PM, Roger Austin wrote: > > Any chance to

Re: Advise in optimizing an application

2014-11-16 Thread Byron Mann
This may be something like that you want. SELECT max(a.email_id) as email_id, a.ml_email FROM nl_mailgroups a inner join nl_catREL c on c.email_id = a.email_id WHERE c.groups_id = group by a.ml_email ORDER BY email_id Your other query may be taking a while due t

Re: Advise in optimizing an application

2014-11-16 Thread Les Mizzell
Ended up finally creating a VIEW in the database to handle the problem. Even on a 15,000 record list, runs in just a few seconds... | SELECT DISTINCT nl_catREL.groups_id, duplicates.ml_id FROM nl_catREL INNER JOIN nl_mailgroupsON nl_mailgroups.ml_id = nl_catREL.ml_id