subject:"Re\: Why are Tomcat's threads more costly than background threads\?"

i'm quite again using JMS for local asynchronism generally

not the goal of JMS IMO

Romain Manni-Bucau
Twitter: @rmannibucau
Blog: http://rmannibucau.wordpress.com/
LinkedIn: http://fr.linkedin.com/in/rmannibucau
Github: https://github.com/rmannibucau



2012/12/16 Anthony Fryer apfr...@hotmail.com:
 Another change you could make to improve the performance of the method
 invoked by the timer, is instead of performing the actual database insert
 for each email in that method, instead write a message to a JMS queue.  Then
 create a message driven bean that processes the JMS messages and performs
 the actual database insert.



 --
 View this message in context: 
 http://openejb.979440.n4.nabble.com/Re-Why-are-Tomcat-s-threads-more-costly-than-background-threads-tp4659429p4659697.html
 Sent from the OpenEJB User mailing list archive at Nabble.com.

Re: Why are Tomcat's threads more costly than background threads?

Anthony, thanks for the recommendation. I will have to give that a try.
On Dec 16, 2012 8:33 AM, Anthony Fryer apfr...@hotmail.com wrote:

 Another change you could make to improve the performance of the method
 invoked by the timer, is instead of performing the actual database insert
 for each email in that method, instead write a message to a JMS queue.
  Then
 create a message driven bean that processes the JMS messages and performs
 the actual database insert.



 --
 View this message in context:
 http://openejb.979440.n4.nabble.com/Re-Why-are-Tomcat-s-threads-more-costly-than-background-threads-tp4659429p4659697.html
 Sent from the OpenEJB User mailing list archive at Nabble.com.

Re: Why are Tomcat's threads more costly than background threads?

I have never used JMS/message-driven-beans as of yet, but considering it...

On Sun, Dec 16, 2012 at 8:35 AM, Romain Manni-Bucau
rmannibu...@gmail.comwrote:

 i'm quite again using JMS for local asynchronism generally

 not the goal of JMS IMO

 Romain Manni-Bucau
 Twitter: @rmannibucau
 Blog: http://rmannibucau.wordpress.com/
 LinkedIn: http://fr.linkedin.com/in/rmannibucau
 Github: https://github.com/rmannibucau



 2012/12/16 Anthony Fryer apfr...@hotmail.com:
  Another change you could make to improve the performance of the method
  invoked by the timer, is instead of performing the actual database insert
  for each email in that method, instead write a message to a JMS queue.
  Then
  create a message driven bean that processes the JMS messages and performs
  the actual database insert.
 
 
 
  --
  View this message in context:
 http://openejb.979440.n4.nabble.com/Re-Why-are-Tomcat-s-threads-more-costly-than-background-threads-tp4659429p4659697.html
  Sent from the OpenEJB User mailing list archive at Nabble.com.

Re: Why are Tomcat's threads more costly than background threads?

2012-12-16 Thread Stephen Connolly

Need to take care around transaction symmantics with JMS too.

On Sunday, 16 December 2012, Anthony Fryer wrote:

 Another change you could make to improve the performance of the method
 invoked by the timer, is instead of performing the actual database insert
 for each email in that method, instead write a message to a JMS queue.
  Then
 create a message driven bean that processes the JMS messages and performs
 the actual database insert.



 --
 View this message in context:
 http://openejb.979440.n4.nabble.com/Re-Why-are-Tomcat-s-threads-more-costly-than-background-threads-tp4659429p4659697.html
 Sent from the OpenEJB User mailing list archive at Nabble.com.

Re: Why are Tomcat's threads more costly than background threads?

2012-12-16 Thread Anthony Fryer

Romain, you're opinion means alot so i'm wondering why you are against JMS in
this case. This kind of pattern happens alot in my experience. In this
case a timer job executes that finds a list of jobs to execute, in this case
read a list of emails and for each email insert a row into a database.

In my experience, the timer job that executes wants to finish as quickly as
possible so you don't get overlapping timer executions and also to minimize
resource utilization. Putting a jms message on a queue is generally alot
quicker than performing an actual database insert. In addition JMS gives
you guaranteed message delivery, so you know that the message is going to
get processed and not get lost. It is part of the JMS spec that i believe
is very applicable in this case.

As for the transactional semantics that apply to the Message Driven Bean
that performs the actual work of inserting the record into the database,
again i think it works well in this case. If the MDB is executing within a
container managed transaction, then only if the database insert succeeds and
the transaction commits will the message be taken off the queue. This means
if the database insert fails, you haven't lost the message and can either
retry or configure the JMS to place the message on an error queue that can
be processed in a different way.

How would you handle a database insert failure in the current solution?

Also I don't think it matters if the MDB that performs the message
processing is local or not. Its irrelevant really. I wouldn't be using the
JMS just for asynchronous calls. I'd be using it for the guarenteed message
delivery and processing, performance AND asynchronous processing.

--
View this message in context:
http://openejb.979440.n4.nabble.com/Re-Why-are-Tomcat-s-threads-more-costly-than-background-threads-tp4659429p4659710.html
Sent from the OpenEJB User mailing list archive at Nabble.com.

Re: Why are Tomcat's threads more costly than background threads?

Good points Anthony and thanks for sharing all that. Stephen, right now, I
assume the same as what you stated. My experience with entirely @stateless
solution, IMHO, got me nowhere because @stateless seems to be application
scope and the database got locked application scope/wide, so that is why I
reverted to user session being responsible for the update. JMS seems a bit
@stateless in its nature to me. I honestly believe that entirely @stateless
can perform narrower scope transactions. Maybe one day I can achieve such a
thing. :-)
On Dec 16, 2012 7:07 PM, Anthony Fryer apfr...@hotmail.com wrote:

Romain, you're opinion means alot so i'm wondering why you are against JMS
in
this case. This kind of pattern happens alot in my experience. In this
case a timer job executes that finds a list of jobs to execute, in this
case
read a list of emails and for each email insert a row into a database.

As for the transactional semantics that apply to the Message Driven Bean
that performs the actual work of inserting the record into the database,
again i think it works well in this case. If the MDB is executing within a
container managed transaction, then only if the database insert succeeds
and
the transaction commits will the message be taken off the queue. This
means
if the database insert fails, you haven't lost the message and can either
retry or configure the JMS to place the message on an error queue that can
be processed in a different way.

How would you handle a database insert failure in the current solution?

Also I don't think it matters if the MDB that performs the message
processing is local or not. Its irrelevant really. I wouldn't be using
the
JMS just for asynchronous calls. I'd be using it for the guarenteed
message
delivery and processing, performance AND asynchronous processing.

Re: Why are Tomcat's threads more costly than background threads?

well i don't like it because for asynchronouism you now have some
tools in JavaEE. then i agree the retry mecanism can be interesting
but
1) did you already look what does the container (AMQ here) when you
send/receive a message? i wouldn't try to get any perf with such a lot
of job
2) AMQ blocks easily and it is as easy to get twice the same message
or lost it so i wouldn't bet on it

don't take it bad, AMQ is fine in general but got so much issues that
i wouldnt use it when i stay in the same JVM

@Asynchronous should be interesting

that's said when a server is really used async is useless (work has to
be done :p)

Romain Manni-Bucau
Twitter: @rmannibucau
Blog: http://rmannibucau.wordpress.com/
LinkedIn: http://fr.linkedin.com/in/rmannibucau
Github: https://github.com/rmannibucau

2012/12/17 Howard W. Smith, Jr. smithh032...@gmail.com:
Good points Anthony and thanks for sharing all that. Stephen, right now, I
assume the same as what you stated. My experience with entirely @stateless
solution, IMHO, got me nowhere because @stateless seems to be application
scope and the database got locked application scope/wide, so that is why I
reverted to user session being responsible for the update. JMS seems a bit
@stateless in its nature to me. I honestly believe that entirely @stateless
can perform narrower scope transactions. Maybe one day I can achieve such a
thing. :-)
On Dec 16, 2012 7:07 PM, Anthony Fryer apfr...@hotmail.com wrote:

Romain, you're opinion means alot so i'm wondering why you are against JMS
in
this case. This kind of pattern happens alot in my experience. In this
case a timer job executes that finds a list of jobs to execute, in this
case
read a list of emails and for each email insert a row into a database.

As for the transactional semantics that apply to the Message Driven Bean
that performs the actual work of inserting the record into the database,
again i think it works well in this case. If the MDB is executing within a
container managed transaction, then only if the database insert succeeds
and
the transaction commits will the message be taken off the queue. This
means
if the database insert fails, you haven't lost the message and can either
retry or configure the JMS to place the message on an error queue that can
be processed in a different way.

How would you handle a database insert failure in the current solution?

Also I don't think it matters if the MDB that performs the message
processing is local or not. Its irrelevant really. I wouldn't be using
the
JMS just for asynchronous calls. I'd be using it for the guarenteed
message
delivery and processing, performance AND asynchronous processing.

Re: Why are Tomcat's threads more costly than background threads?

2012-12-16 Thread Anthony Fryer

Ok so what i'm hearing is that you're against the idea because you don't
trust the implementation of JMS included in tomee?



--
View this message in context: 
http://openejb.979440.n4.nabble.com/Re-Why-are-Tomcat-s-threads-more-costly-than-background-threads-tp4659429p4659713.html
Sent from the OpenEJB User mailing list archive at Nabble.com.

Re: Why are Tomcat's threads more costly than background threads?

2012-12-16 Thread Anthony Fryer

Yes i've done this before a few times and had really good results using
weblogic.  I must say i have an element of doubt about it with Romain not
liking it though.  I've used ActiveMQ before and not encountered issues but
these guys are the tomee devs and I'd go with what they say.



--
View this message in context: 
http://openejb.979440.n4.nabble.com/Re-Why-are-Tomcat-s-threads-more-costly-than-background-threads-tp4659429p4659718.html
Sent from the OpenEJB User mailing list archive at Nabble.com.

Re: Why are Tomcat's threads more costly than background threads?

I used it a lot in javaee 5 (openejb 3) but i'm happy to leave it on the
road qince javaee 6, no more.

It works but i dont find it effective when it stays local

@others: am i alone?
Le 17 déc. 2012 06:53, Anthony Fryer apfr...@hotmail.com a écrit :

 Yes i've done this before a few times and had really good results using
 weblogic.  I must say i have an element of doubt about it with Romain not
 liking it though.  I've used ActiveMQ before and not encountered issues but
 these guys are the tomee devs and I'd go with what they say.



 --
 View this message in context:
 http://openejb.979440.n4.nabble.com/Re-Why-are-Tomcat-s-threads-more-costly-than-background-threads-tp4659429p4659718.html
 Sent from the OpenEJB User mailing list archive at Nabble.com.

Re: Why are Tomcat's threads more costly than background threads?

Very interesting discussion and responses here! Anthony motivates me to
jump on the bandwagon of using JMS/message-driven-beans, and at the same
time, Romain tells me, not much need (for local) in a Java EE 6
application. :)

Either way, i have not ruled out JMS. I have done some strategic coding on
my part to avoid @Schedule or use of timer service...using Date class to
postpone pushing data to google calendar, and only pushing when there is a
new date to update...application-wide. At first, i was pushing data to
google calendar after 'every' update request... sending over 1,000 requests
to google calendar sometimes or daily (adding and deleting events on google
calendar to keep google calendar in sync with the database), but I cut that
back (or decreased the # of requests) with the new code I developed 2+
months ago. just makes for a faster user experience since they requested to
have data updated via ajax on change/selection of this field, and that
field.

Right now, JMS would be similar to a nice science project for me... just to
see if I can do it and the success of it.


On Mon, Dec 17, 2012 at 1:57 AM, Romain Manni-Bucau
rmannibu...@gmail.comwrote:

 I used it a lot in javaee 5 (openejb 3) but i'm happy to leave it on the
 road qince javaee 6, no more.

 It works but i dont find it effective when it stays local

Re: Why are Tomcat's threads more costly than background threads?

Almost forgot...PrimeFaces peer/friend of mine informed me that JMS is good
for running two instances of an app, similar to load balancer.

Prior to using TomEE, I really wanted to use PrimeFaces Push (powered by
Atmosphere/websocket), and at that time, I was thinking that I could
possibly run Tomcat or TomEE just to push messages to the clients of my app
running on Glassfish, since Glassfish had issues with PrimeFaces Push
(Atmosphere). I think he said that tomEE and Glassfish could do that, but
they would have to communicate via JMS or the two servlets would have to
serve requests/responses, appropriately, via JMS.

I am probably not stating the above correctly as he said, but that's how I
understood what he told me.

Now that I'm only using tomEE, there is no need to do such a thing. Loving
TomEE!!!



On Mon, Dec 17, 2012 at 2:26 AM, Howard W. Smith, Jr. 
smithh032...@gmail.com wrote:

 Very interesting discussion and responses here! Anthony motivates me to
 jump on the bandwagon of using JMS/message-driven-beans, and at the same
 time, Romain tells me, not much need (for local) in a Java EE 6
 application. :)

 Either way, i have not ruled out JMS. I have done some strategic coding on
 my part to avoid @Schedule or use of timer service...using Date class to
 postpone pushing data to google calendar, and only pushing when there is a
 new date to update...application-wide. At first, i was pushing data to
 google calendar after 'every' update request... sending over 1,000 requests
 to google calendar sometimes or daily (adding and deleting events on google
 calendar to keep google calendar in sync with the database), but I cut that
 back (or decreased the # of requests) with the new code I developed 2+
 months ago. just makes for a faster user experience since they requested to
 have data updated via ajax on change/selection of this field, and that
 field.

 Right now, JMS would be similar to a nice science project for me... just
 to see if I can do it and the success of it.


 On Mon, Dec 17, 2012 at 1:57 AM, Romain Manni-Bucau rmannibu...@gmail.com
  wrote:

 I used it a lot in javaee 5 (openejb 3) but i'm happy to leave it on the
 road qince javaee 6, no more.

 It works but i dont find it effective when it stays local

Re: Why are Tomcat's threads more costly than background threads?

Side note: if your need is pushing regularly to the client just having some
client polling can make it far easier to manage on both sides...
Le 17 déc. 2012 08:33, Howard W. Smith, Jr. smithh032...@gmail.com a
écrit :

 Almost forgot...PrimeFaces peer/friend of mine informed me that JMS is good
 for running two instances of an app, similar to load balancer.

 Prior to using TomEE, I really wanted to use PrimeFaces Push (powered by
 Atmosphere/websocket), and at that time, I was thinking that I could
 possibly run Tomcat or TomEE just to push messages to the clients of my app
 running on Glassfish, since Glassfish had issues with PrimeFaces Push
 (Atmosphere). I think he said that tomEE and Glassfish could do that, but
 they would have to communicate via JMS or the two servlets would have to
 serve requests/responses, appropriately, via JMS.

 I am probably not stating the above correctly as he said, but that's how I
 understood what he told me.

 Now that I'm only using tomEE, there is no need to do such a thing. Loving
 TomEE!!!



 On Mon, Dec 17, 2012 at 2:26 AM, Howard W. Smith, Jr. 
 smithh032...@gmail.com wrote:

  Very interesting discussion and responses here! Anthony motivates me to
  jump on the bandwagon of using JMS/message-driven-beans, and at the same
  time, Romain tells me, not much need (for local) in a Java EE 6
  application. :)
 
  Either way, i have not ruled out JMS. I have done some strategic coding
 on
  my part to avoid @Schedule or use of timer service...using Date class to
  postpone pushing data to google calendar, and only pushing when there is
 a
  new date to update...application-wide. At first, i was pushing data to
  google calendar after 'every' update request... sending over 1,000
 requests
  to google calendar sometimes or daily (adding and deleting events on
 google
  calendar to keep google calendar in sync with the database), but I cut
 that
  back (or decreased the # of requests) with the new code I developed 2+
  months ago. just makes for a faster user experience since they requested
 to
  have data updated via ajax on change/selection of this field, and that
  field.
 
  Right now, JMS would be similar to a nice science project for me... just
  to see if I can do it and the success of it.
 
 
  On Mon, Dec 17, 2012 at 1:57 AM, Romain Manni-Bucau 
 rmannibu...@gmail.com
   wrote:
 
  I used it a lot in javaee 5 (openejb 3) but i'm happy to leave it on the
  road qince javaee 6, no more.
 
  It works but i dont find it effective when it stays local

Re: Why are Tomcat's threads more costly than background threads?

Polling is too much network traffic in my opinion. Websocket is instant
messaging. No need to add polling code and xhtml here there for different
type of messages coming from server and no need of an ajax request to
server throughout entire session just to get an update from the server.
 On Dec 17, 2012 2:47 AM, Romain Manni-Bucau rmannibu...@gmail.com
wrote:

 Side note: if your need is pushing regularly to the client just having some
 client polling can make it far easier to manage on both sides...
 Le 17 déc. 2012 08:33, Howard W. Smith, Jr. smithh032...@gmail.com a
 écrit :

  Almost forgot...PrimeFaces peer/friend of mine informed me that JMS is
 good
  for running two instances of an app, similar to load balancer.
 
  Prior to using TomEE, I really wanted to use PrimeFaces Push (powered by
  Atmosphere/websocket), and at that time, I was thinking that I could
  possibly run Tomcat or TomEE just to push messages to the clients of my
 app
  running on Glassfish, since Glassfish had issues with PrimeFaces Push
  (Atmosphere). I think he said that tomEE and Glassfish could do that, but
  they would have to communicate via JMS or the two servlets would have to
  serve requests/responses, appropriately, via JMS.
 
  I am probably not stating the above correctly as he said, but that's how
 I
  understood what he told me.
 
  Now that I'm only using tomEE, there is no need to do such a thing.
 Loving
  TomEE!!!
 
 
 
  On Mon, Dec 17, 2012 at 2:26 AM, Howard W. Smith, Jr. 
  smithh032...@gmail.com wrote:
 
   Very interesting discussion and responses here! Anthony motivates me to
   jump on the bandwagon of using JMS/message-driven-beans, and at the
 same
   time, Romain tells me, not much need (for local) in a Java EE 6
   application. :)
  
   Either way, i have not ruled out JMS. I have done some strategic coding
  on
   my part to avoid @Schedule or use of timer service...using Date class
 to
   postpone pushing data to google calendar, and only pushing when there
 is
  a
   new date to update...application-wide. At first, i was pushing data to
   google calendar after 'every' update request... sending over 1,000
  requests
   to google calendar sometimes or daily (adding and deleting events on
  google
   calendar to keep google calendar in sync with the database), but I cut
  that
   back (or decreased the # of requests) with the new code I developed 2+
   months ago. just makes for a faster user experience since they
 requested
  to
   have data updated via ajax on change/selection of this field, and that
   field.
  
   Right now, JMS would be similar to a nice science project for me...
 just
   to see if I can do it and the success of it.
  
  
   On Mon, Dec 17, 2012 at 1:57 AM, Romain Manni-Bucau 
  rmannibu...@gmail.com
wrote:
  
   I used it a lot in javaee 5 (openejb 3) but i'm happy to leave it on
 the
   road qince javaee 6, no more.
  
   It works but i dont find it effective when it stays local

Re: Why are Tomcat's threads more costly than background threads?

Yes and no, it makes the connection being from the client why is better for
it and it works - when websockets can have issues with proxies and so on.
Websocket are nice when you have a real protocol...just create one and ask
the client to ask for an update with a timer, that's already better and you
keep the full connected advantages
Le 17 déc. 2012 08:54, Howard W. Smith, Jr. smithh032...@gmail.com a
écrit :

 Polling is too much network traffic in my opinion. Websocket is instant
 messaging. No need to add polling code and xhtml here there for different
 type of messages coming from server and no need of an ajax request to
 server throughout entire session just to get an update from the server.
  On Dec 17, 2012 2:47 AM, Romain Manni-Bucau rmannibu...@gmail.com
 wrote:

  Side note: if your need is pushing regularly to the client just having
 some
  client polling can make it far easier to manage on both sides...
  Le 17 déc. 2012 08:33, Howard W. Smith, Jr. smithh032...@gmail.com a
  écrit :
 
   Almost forgot...PrimeFaces peer/friend of mine informed me that JMS is
  good
   for running two instances of an app, similar to load balancer.
  
   Prior to using TomEE, I really wanted to use PrimeFaces Push (powered
 by
   Atmosphere/websocket), and at that time, I was thinking that I could
   possibly run Tomcat or TomEE just to push messages to the clients of my
  app
   running on Glassfish, since Glassfish had issues with PrimeFaces Push
   (Atmosphere). I think he said that tomEE and Glassfish could do that,
 but
   they would have to communicate via JMS or the two servlets would have
 to
   serve requests/responses, appropriately, via JMS.
  
   I am probably not stating the above correctly as he said, but that's
 how
  I
   understood what he told me.
  
   Now that I'm only using tomEE, there is no need to do such a thing.
  Loving
   TomEE!!!
  
  
  
   On Mon, Dec 17, 2012 at 2:26 AM, Howard W. Smith, Jr. 
   smithh032...@gmail.com wrote:
  
Very interesting discussion and responses here! Anthony motivates me
 to
jump on the bandwagon of using JMS/message-driven-beans, and at the
  same
time, Romain tells me, not much need (for local) in a Java EE 6
application. :)
   
Either way, i have not ruled out JMS. I have done some strategic
 coding
   on
my part to avoid @Schedule or use of timer service...using Date class
  to
postpone pushing data to google calendar, and only pushing when there
  is
   a
new date to update...application-wide. At first, i was pushing data
 to
google calendar after 'every' update request... sending over 1,000
   requests
to google calendar sometimes or daily (adding and deleting events on
   google
calendar to keep google calendar in sync with the database), but I
 cut
   that
back (or decreased the # of requests) with the new code I developed
 2+
months ago. just makes for a faster user experience since they
  requested
   to
have data updated via ajax on change/selection of this field, and
 that
field.
   
Right now, JMS would be similar to a nice science project for me...
  just
to see if I can do it and the success of it.
   
   
On Mon, Dec 17, 2012 at 1:57 AM, Romain Manni-Bucau 
   rmannibu...@gmail.com
 wrote:
   
I used it a lot in javaee 5 (openejb 3) but i'm happy to leave it on
  the
road qince javaee 6, no more.
   
It works but i dont find it effective when it stays local

Re: Why are Tomcat's threads more costly than background threads?

2012-12-15 Thread Howard W. Smith, Jr.

David,

just wanted to let you know that I did what you recommended below, and it's
working great (no errors; will continue to monitor server log). I had to
make a few other code changes to this piece of software, and did some
testing as well.

Feel free to see my latest test results... once I have the data from each
email in memory/POJO, then it is updating data at an average of 4 to 6
seconds (2 airport shuttles per email).

https://issues.apache.org/jira/browse/OPENEJB-1968#comment-13532966

Thanks,
Howard

On Fri, Dec 14, 2012 at 3:47 PM, David Blevins david.blev...@gmail.comwrote:

 Now that things are working, I suggest converting your @Schedule bean from
 @Stateless to @Singleton for the reasons mentioned before.  If you someday
 (knock on wood) get totally nailed with a ton of emails/orders, the
 @Singleton with @Lock(WRITE) will prevent any situations where your timers
 start to overlap due to backlog and you accidentally process emails twice
 or more.

Re: Why are Tomcat's threads more costly than background threads?

2012-12-14 Thread Howard W. Smith, Jr.

David (and any/all interested),

Please click URL below to see my latest test results (along with code
changes).  It took about 12 seconds to perform the update on the Windows
Server 2003 32bit 4GB RAM.

https://issues.apache.org/jira/browse/OPENEJB-1968#comment-13532480


I know, I know, I need to consider moving away from Apache Derby database
to a 'real database', but I hope to do that, ASAP.

Thanks for all the responses/recommendations/etc...

Howard


On Thu, Dec 13, 2012 at 11:09 PM, Howard W. Smith, Jr. 
smithh032...@gmail.com wrote:

 David, I corrected that as you stated, and I made sure the code was as you
 recommended, and I've updated the JIRA with test results. Windows Server
 2008 64bit is still fast, Windows SErver 2003 32bit is 'still' slow and
 locking the database.

Re: Why are Tomcat's threads more costly than background threads?

2012-12-14 Thread David Blevins


On Dec 14, 2012, at 9:38 AM, Howard W. Smith, Jr. smithh032...@gmail.com 
wrote:

 David (and any/all interested),
 
 Please click URL below to see my latest test results (along with code
 changes).  It took about 12 seconds to perform the update on the Windows
 Server 2003 32bit 4GB RAM.
 
 https://issues.apache.org/jira/browse/OPENEJB-1968#comment-13532480
 
 
 I know, I know, I need to consider moving away from Apache Derby database
 to a 'real database', but I hope to do that, ASAP.
 
 Thanks for all the responses/recommendations/etc...

Absolutely amazing.  This is fantastic!!

Now that things are working, I suggest converting your @Schedule bean from 
@Stateless to @Singleton for the reasons mentioned before.  If you someday 
(knock on wood) get totally nailed with a ton of emails/orders, the @Singleton 
with @Lock(WRITE) will prevent any situations where your timers start to 
overlap due to backlog and you accidentally process emails twice or more.


-David

Re: Why are Tomcat's threads more costly than background threads?

2012-12-14 Thread Howard W. Smith, Jr.

Thanks David.

Before you introduced me to the @Lock(WRITE), I already came up with a
Boolean variable that is updated on @ApplicationScoped bean, every time
@schedule is triggered 'and' whenever a user clicks the commandLink to
check/get emails. The Boolean variable is checked whenever @schedule or
enduser is checking emails...if @schedule or enduser is 'already' checking
emails, then 'return' from the method.

The @Schedule is 'not' responsible for retrieving/downloading the data from
the email server. The enduser is 100% responsible for downloading the data
from the email server...via the commandLink.

The @schedule is only used for providing a notification to endusers...that
airport shuttle requests are available to download.

I will let you know, if I use Singleton and @Lock(WRITE). On the current
database (Apache Derby), the @Lock(WRITE) always results in a 'locked'
exception which tells me that @Lock(WRITE) does not work well with the
Apache Derby database.

It would be nice to know why all you recommended to me...does not work with
Apache Derby; I did read that row-level or table-level locking is (or can
be) enabled/disabled throughout the database. All I know...i have 'no'
issues updating the database when enduser initiates database updates via
managed-bean-to-stateless-ejb approach.

since I've spent too much time on this and since I have something
working-as-designed, I'm going to leave this alone for now and move on to
the next task...which is very similar to this task...another set of emails
to retrieve from the email server, but a different set of data. A lot of my
code can be reused, but still need to write separate software for those
emails. Looking forward to it...i'm loving me some Java!!!

It will be interesting, when i have time/bandwidth to migrate to another
database...as others, here, have suggested to me. :)


On Fri, Dec 14, 2012 at 3:47 PM, David Blevins david.blev...@gmail.comwrote:

 Now that things are working, I suggest converting your @Schedule bean from
 @Stateless to @Singleton for the reasons mentioned before.  If you someday
 (knock on wood) get totally nailed with a ton of emails/orders, the
 @Singleton with @Lock(WRITE) will prevent any situations where your timers
 start to overlap due to backlog and you accidentally process emails twice
 or more.

Re: Why are Tomcat's threads more costly than background threads?


On Dec 11, 2012, at 7:10 PM, Howard W. Smith, Jr. smithh032...@gmail.com 
wrote:

 Shaking my head... test results were not good at all.
 
 1. @StatelessEJB EmailStatelessBean has @Schedule getEmails()
 2. @EmailStatelessBean has multiple @EJB references to @Stateless
 (sessionfacade) classes that are the DAO classes
 3. EmailStatelessBean gets invoked when triggered by @Schedule
 4. EmailStatelessBean execution locks the tables, still, and locks up the
 entire app, and seems to take longer...since endusers (myself) are making
 requests against database (and web app).
 5. Last but not least, EmailStatelessBean failed to commit the changes;
 transaction rolled back for EmailStatelessBean as well as enduser requests.

The description of EmailStatelessBean still looking beans and failing rollback 
would imply no transactional semantics have actually changed.  Is it possible 
you can paste that bean?


-David

Re: Why are Tomcat's threads more costly than background threads?


On Dec 13, 2012, at 12:27 PM, Howard W. Smith, Jr. smithh032...@gmail.com 
wrote:

 David,
 
 Yes, please refer to the following (and see what I wrote at the beginning
 of that comment), thanks.
 
 https://issues.apache.org/jira/browse/OPENEJB-1968#comment-13531442
 

So first note, the @TransactionAttribute(value = 
TransactionAttributeType.NEVER) and @TransactionAttribute(value = 
TransactionAttributeType.REQUIRES_NEW) were two aspects of the same suggestion.

Second note is, were I in your shoes, the very first thing I'd do is revert all 
the @ApplicationScoped and @Asychronous stuff and go back to the code that only 
took 2 minutes -- I seem to recall that before it jumped to 30 minutes there 
was a version that ran faster (2 minute range).

Third, all versions of the code posted have the same flaw as the original code 
-- there is only one transaction for processing all emails.  This is the 
fundamental problem and there is no possible solution that involves *adding* 
code and not changing this one fundamental flaw.  This is why when you added 
the code that waited for a safe time to do the processing it made things worse 
-- it was too late a transaction was already started and the database tables 
already locked.

We can definitely get this working better, but the problem is there is simply 
too much code happing in one transaction.  No other fix will help.

If you're willing to go back to the original version of the code and do *only* 
the suggestions given, I think we can get this easily sorted out today.

Thoughts?


-David

Re: Why are Tomcat's threads more costly than background threads?

David,

1. definitely not using @Asynchronous; that was no good at all

2. Actually, it was 2 seconds on the Windows Server 2008 64-bit 16GB RAM...
always

3. Always 5 to 10 minutes to process 1 or 2 emails on Windows Server 2003
32-bit 4GB RAM

4. As noted on the JIRA, I tried the TransactionAttribute NEVER and
REQUIRES_NEW on @Stateless EJB that was all invoked via @Schedule, and same
test results, which are The EJB timer rolls back the transaction, and
every database operation that the EmailStatelessBean completed...prior to
@Schedule rolling back the transaction.



On Thu, Dec 13, 2012 at 4:06 PM, David Blevins david.blev...@gmail.comwrote:

  David,
 
  Yes, please refer to the following (and see what I wrote at the beginning
  of that comment), thanks.
 
  https://issues.apache.org/jira/browse/OPENEJB-1968#comment-13531442
 

 So first note, the @TransactionAttribute(value =
 TransactionAttributeType.NEVER) and @TransactionAttribute(value =
 TransactionAttributeType.REQUIRES_NEW) were two aspects of the same
 suggestion.

 Second note is, were I in your shoes, the very first thing I'd do is
 revert all the @ApplicationScoped and @Asychronous stuff and go back to the
 code that only took 2 minutes -- I seem to recall that before it jumped to
 30 minutes there was a version that ran faster (2 minute range).

Re: Why are Tomcat's threads more costly than background threads?


On Dec 13, 2012, at 1:17 PM, Howard W. Smith, Jr. smithh032...@gmail.com 
wrote:

 David,
 
 1. definitely not using @Asynchronous; that was no good at all
 
 2. Actually, it was 2 seconds on the Windows Server 2008 64-bit 16GB RAM...
 always
 
 3. Always 5 to 10 minutes to process 1 or 2 emails on Windows Server 2003
 32-bit 4GB RAM

Good info thanks.

 4. As noted on the JIRA, I tried the TransactionAttribute NEVER and
 REQUIRES_NEW on @Stateless EJB that was all invoked via @Schedule, and same
 test results, which are The EJB timer rolls back the transaction, and
 every database operation that the EmailStatelessBean completed...prior to
 @Schedule rolling back the transaction.

Right, that suggestion was never implemented as it was intended.  Changing the 
TransactionAttribute alone will do nothing.  What we need is to 1) prevent a 
transaction from being started and 2) creating a new and smaller transaction 
scope through a small code rework.

The ApplicationScoped check backfired because it has the effect of locking 
tables for much longer.  We want to process each email in an individual 
transaction and move away from one big transaction.  This is really the most 
critical step to take, no other steps or workarounds matter at this phase of 
the problem.

Let's try this:

  - Go back to the very original 5-10 minutes 32-bit code

  - Split EmailSessionBean into two beans

  - EmailSessionBean
 - will be marked @TransactionAttribute(NEVER)
 - will no longer have an @PersistenceContext EntityManager reference
  - AirportShuttleRequestProcessorBean
 - will be marked @TransactionAttribute(REQUIRES_NEW)
 - will have only one public method, `createOrderForAirportShuttle`
 - will do all persistence processing

The EmailSessionBean will have an @EJB reference to the new 
AirportShuttleRequestProcessorBean.  Effectively, we're splitting the code that 
does the contacting the of the email server from the code that contacts the 
database and we're creating several much smaller transactions.


-David

Re: Why are Tomcat's threads more costly than background threads?

David,

Thanks for the response. Guess what... the 2 beans along with the names and
all... is exactly what I did 'yesterday', the only thing I did 'not' do was
mark each of the beans according to what you mentioned.

I will try that, ASAP might not be able to report about it until a
little later. I want the @Schedule approach along with a user-initiated
approach. I'm coding the user initiated approach right now.

The original approach will definitely serve for background processing.

Thanks,
Howard

On Thu, Dec 13, 2012 at 4:53 PM, David Blevins david.blev...@gmail.comwrote:

 Let's try this:

   - Go back to the very original 5-10 minutes 32-bit code

   - Split EmailSessionBean into two beans

   - EmailSessionBean
  - will be marked @TransactionAttribute(NEVER)
  - will no longer have an @PersistenceContext EntityManager
 reference
   - AirportShuttleRequestProcessorBean
  - will be marked @TransactionAttribute(REQUIRES_NEW)
  - will have only one public method, `createOrderForAirportShuttle`
  - will do all persistence processing

 The EmailSessionBean will have an @EJB reference to the new
 AirportShuttleRequestProcessorBean.  Effectively, we're splitting the code
 that does the contacting the of the email server from the code that
 contacts the database and we're creating several much smaller transactions.

Re: Why are Tomcat's threads more costly than background threads?


On Dec 13, 2012, at 1:53 PM, David Blevins david.blev...@gmail.com wrote:

 
 On Dec 13, 2012, at 1:17 PM, Howard W. Smith, Jr. smithh032...@gmail.com 
 wrote:
 
 David,
 
 1. definitely not using @Asynchronous; that was no good at all
 
 2. Actually, it was 2 seconds on the Windows Server 2008 64-bit 16GB RAM...
 always
 
 3. Always 5 to 10 minutes to process 1 or 2 emails on Windows Server 2003
 32-bit 4GB RAM
 
 Good info thanks.
 
 4. As noted on the JIRA, I tried the TransactionAttribute NEVER and
 REQUIRES_NEW on @Stateless EJB that was all invoked via @Schedule, and same
 test results, which are The EJB timer rolls back the transaction, and
 every database operation that the EmailStatelessBean completed...prior to
 @Schedule rolling back the transaction.
 
 Right, that suggestion was never implemented as it was intended.  Changing 
 the TransactionAttribute alone will do nothing.  What we need is to 1) 
 prevent a transaction from being started and 2) creating a new and smaller 
 transaction scope through a small code rework.
 
 The ApplicationScoped check backfired because it has the effect of locking 
 tables for much longer.  We want to process each email in an individual 
 transaction and move away from one big transaction.  This is really the most 
 critical step to take, no other steps or workarounds matter at this phase of 
 the problem.
 
 Let's try this:
 
  - Go back to the very original 5-10 minutes 32-bit code
 
  - Split EmailSessionBean into two beans
 
  - EmailSessionBean
 - will be marked @TransactionAttribute(NEVER)
 - will no longer have an @PersistenceContext EntityManager reference
  - AirportShuttleRequestProcessorBean
 - will be marked @TransactionAttribute(REQUIRES_NEW)
 - will have only one public method, `createOrderForAirportShuttle`
 - will do all persistence processing
 
 The EmailSessionBean will have an @EJB reference to the new 
 AirportShuttleRequestProcessorBean.  Effectively, we're splitting the code 
 that does the contacting the of the email server from the code that contacts 
 the database and we're creating several much smaller transactions.

And one more tweak.  Let's change the EmailSessionBean to type @Singleton and 
add @Lock(WRITE) to our `getEmails()` method.

We're doing this to ensure that two `getEmail()` invocations cannot possibly 
happen at the same time.  This is not necessarily the case now, but this can 
easily occur if the `getEmails()` method takes longer than the configured time 
of every 10 minutes.  It can also occur if the time gets shorted, to say 2 
minutes.

With switching our EmailSessionBean bean to type @Singleton and ensuring 
@Lock(WRITE) on the `getEmails()` method, once things are working well we could 
effectively dramatically reduce the delay between firings to something far more 
aggressive than 10 minutes (say 2 or 5 minutes) and still be safe.  The 
combination of @Singleton/@Lock(WRITE) guarantees that method will only ever be 
active one-at-a-time in the entire application.


-David

Re: Why are Tomcat's threads more costly than background threads?


On Dec 13, 2012, at 2:01 PM, Howard W. Smith, Jr. smithh032...@gmail.com 
wrote:

 Thanks for the response. Guess what... the 2 beans along with the names and
 all... is exactly what I did 'yesterday', the only thing I did 'not' do was
 mark each of the beans according to what you mentioned.

Right, the annotations are a critical part.  Splitting the code and not 
changing the TransactionAttribute annotations as noted would essentially still 
result in one big transaction (i.e. no change).

Educational point; when no @TransactionAttribute is specified on an EJB, it 
defaults to @TransactionAttribute(REQUIRED)

REQUIRED basically means join the existing transaction if possible, otherwise 
create one.  So EmailSessionBean would create a transaction in the @Schedule 
method and every method invocation on AirportShuttleRequestProcessorBean would 
just join the existing transaction.

The result:
 - one big transaction that locks all tables and doesn't let them go till 
getEmails completes
 - if email 1 from Joe fails, all email processing will fail and rollback when 
getEmail completes

So aside from a performance problem, there's a very real business logic problem.

Splitting them *and* adjusting the transaction settings will fix the business 
problem and likely help with the performance problem.  The big difference of 2 
seconds on 64-bit vs several minutes on a 32-bit is alarming and very likely a 
system issue of some kind.  That said, going down this path of shrinking 
transaction scope will help zero in on where the problem occurs.


-David

Re: Why are Tomcat's threads more costly than background threads?

David, I did as you recommended. I got an exception related to the ejbTimer
and JTA. please see my latest 2 post on the JIRA.

https://issues.apache.org/jira/browse/OPENEJB-1968

Need to leave my desk for little bit. Thanks.

On Thu, Dec 13, 2012 at 4:06 PM, David Blevins david.blev...@gmail.comwrote:

 If you're willing to go back to the original version of the code and do
 *only* the suggestions given, I think we can get this easily sorted out
 today.

Re: Why are Tomcat's threads more costly than background threads?


On Dec 13, 2012, at 3:57 PM, Howard W. Smith, Jr. smithh032...@gmail.com 
wrote:

 David, I did as you recommended. I got an exception related to the ejbTimer
 and JTA. please see my latest 2 post on the JIRA.

My bad, should be @TransactionAttribute(NOT_SUPPORTED)  -- even hard for me to 
remember the right names :)

In case you're curious, reading for another day:

  http://tomee.apache.org/transaction-annotations.html


-David

Re: Why are Tomcat's threads more costly than background threads?


On Dec 13, 2012, at 2:41 PM, Howard W. Smith, Jr. smithh032...@gmail.com 
wrote:

 Question...
 
 My last/latest implementation/version of AirportShuttleProcessingBean is a
 @Stateless EJB that calls many other @Stateless EJB (DAO). Should I keep
 PersistentContext in AirportShuttleProcessingBean, or can
 AirportShuttleProcessingBean call any/all @Stateless EJB (DAO) for shortest
 transaction possiblel? I hear that a lot here in Apache/TomEE/Tomcat
 land... shortest transactions are best.

The rule of thumb is that you want the transaction to be as short as possible 
while also not jeopardizing data integrity/consistency.

As a small example, say you had to process an order from a customer that didn't 
yet exist.  Ideally, you would add the new customer information in one 
transaction, then add the order in another transaction.

The scope starts to become too short, when we get into say processing the 
order itself with several small transactions.  You wouldn't want to insert a 
new bare Order as one transaction, then add each Item in an order as 
separate transactions.  The result of a failure to complete is a half processed 
order which is clearly a non-recoverable situation.


One somewhat unrelated note, I did notice several flush() commands in your 
code.  All data is automatically flushed at the end of the transaction, so 
there should be no need for this can and in fact calling it excessively can 
cause performance issues.  It can mean the difference between one trip to the 
database and one hundred trips to the database.

Having said that, *do not* remove them now.  Wait till we are clearly at a 
state  where we have successfully split the transactions and things work.  Once 
we reach that milestone and all works and code is checked in, then give the 
remove excess flush() calls optimization a try.


-David

Re: Why are Tomcat's threads more costly than background threads?

David, I corrected that as you stated, and I made sure the code was as you
recommended, and I've updated the JIRA with test results. Windows Server
2008 64bit is still fast, Windows SErver 2003 32bit is 'still' slow and
locking the database.



On Thu, Dec 13, 2012 at 7:02 PM, David Blevins david.blev...@gmail.comwrote:

 My bad, should be @TransactionAttribute(NOT_SUPPORTED)  -- even hard for
 me to remember the right names :)

 In case you're curious, reading for another day:

   http://tomee.apache.org/transaction-annotations.html

Re: Why are Tomcat's threads more costly than background threads?

2012-12-12 Thread Jean-Louis MONTEIRO

I tend to recommend to set any finder methods to SUPPORT which is far
better in terms of performances and allow reusability of finder methods in
business methods.

JLouis


2012/12/11 David Blevins david.blev...@gmail.com

 Setting the LockModeType is good.  Understand, though, that all the values
 in LockModeType are not guaranteed to be supported in all databases.

 It doesn't address the core problem in that the transaction scope is
 likely too broad.  As a general rule *always* ensure your transactions are
 as *small* as possible.  In order to be very conscious about it it can help
 to actually document the code and justify why two or more operations need
 to be in the same transaction.  It'll force you to think about transactions
 more deliberately and ask yourself other questions like where are my
 transactions starting and stopping in the first place.

 In this particular situation, with default settings, you'll get one
 transaction started for you when the @Schedule method fires.  So
 essentially, you're batch processing unrelated data in one transaction.  If
 Joe and Jane both send email with data, you're processing Jane's data in
 Joe's transaction.  As as well your email fetching time is getting included
 in your transaction time.  If your email server hangs, your transaction
 hangs.

 So instead of:

 // TRANSACTION START
 @Schedule(...)
 public void processPendingEmail() {

 ListEmail emails = fetchEmail();
 for (Email email : emails) {
// parse email and insert records
 }
 }
 // TRANSACTION END

 You want:

 @Schedule(...)
 public void processPendingEmail() {

 ListEmail emails = fetchEmail();
 for (Email email : emails) {
// TRANSACTION START
// parse email and insert records
// TRANSACTION END
 }
 }

 This can be as easy as marking the `processPendingEmail` as
 `@TransactionAttribute(NEVER)` then moving the insert logic to another EJB
 method.  Just make sure that if you move that logic to another method of
 the same EJB that you don't invoke that method with a `this` reference or
 you won't get container-managed transactions on that method call. (see this
 reference for why):


 http://stackoverflow.com/questions/3381002/how-is-the-return-value-of-sessioncontext-getbusinessobject-different-from-th/3384128#3384128

 Hope this helps!


 -David

 On Dec 11, 2012, at 12:21 PM, Howard W. Smith, Jr. 
 smithh032...@gmail.com wrote:

  Prior to adding this @Stateless EJB (that's invoked via @Schedule every
 10
  minutes), my JSF web app has been running 100% okay without any deadlocks
  or anything else like that.
 
  So, is it okay if I just add optimistic locking, em.lock(entity,
  LockType.Optimistic) before any/all em.persist(...) that I have in the
  @Stateless EJB?
 
  JSF web app is working okay, because the JSF components call multiple
  @Stateless EJB that usually do any of the following, SELECT, UPDATE,
  DELETE,  on one table at time.
 
 
 
  On Tue, Dec 11, 2012 at 2:35 PM, Howard W. Smith, Jr. 
  smithh032...@gmail.com wrote:
 
  You're absolutely right, David. Thanks for the response.
 
  It is a transaction issue and the @Stateless EJB is holding a
 transaction
  open and writing into 6 to 10 tables in a single database, which is also
  the database accessed by JSF components (by endusers). The @Stateless
 EJB
  is on a timer ( @Schedule), checks email server for incoming requests
 from
  customers, and takes that data and inserts it into database, so endusers
  don't have to manually enter the data (that they receive from these
  formmail results).
 
  Below is the exception that I saw maybe an hour or two ago, and I
  confirmed that it's a locking/transaction issue, and from the
 stacktrace,
  it seems as though the JSF components cannot access the table, that is
  evidently locked by @Stateless EJB. The @Stateless @EJB updated all
 tables
  on my faster/dev/test server on average of 1 to 2 seconds, and/so in my
  opinion, the data is not that much at all, maybe a few kbytes of data
 being
  inserted into the database. Anyway, I've been reading up on pessismistic
  and optimistic locking, and that's where I was in my reading before I
  noticed your email. I really don't know what the database or JPA is
  defaulting to, because honestly, I have no code that specifies any
 locks.
  Recently, I only added query hint (readonly) to JPA queries.
 Personally, I
  feel as though I can use 'no locking' for these inserts, since this is
  brand new data, but the JSF components may be locking the entites/tables
  during simple queries (SELECT statements).  Feel free to change this
  email/topic to a more suitable topic/subject, if necessary. :)
 
 
 
 




-- 
Jean-Louis

Re: Why are Tomcat's threads more costly than background threads?

2012-12-12 Thread Howard W. Smith, Jr.

Jean-Louis, please clarify 'finder methods' and/provide a short example,
thanks.


On Wed, Dec 12, 2012 at 3:08 AM, Jean-Louis MONTEIRO jeano...@gmail.comwrote:

 I tend to recommend to set any finder methods to SUPPORT which is far
 better in terms of performances and allow reusability of finder methods in
 business methods.

 JLouis


 2012/12/11 David Blevins david.blev...@gmail.com

  Setting the LockModeType is good.  Understand, though, that all the
 values
  in LockModeType are not guaranteed to be supported in all databases.
 
  It doesn't address the core problem in that the transaction scope is
  likely too broad.  As a general rule *always* ensure your transactions
 are
  as *small* as possible.  In order to be very conscious about it it can
 help
  to actually document the code and justify why two or more operations need
  to be in the same transaction.  It'll force you to think about
 transactions
  more deliberately and ask yourself other questions like where are my
  transactions starting and stopping in the first place.
 
  In this particular situation, with default settings, you'll get one
  transaction started for you when the @Schedule method fires.  So
  essentially, you're batch processing unrelated data in one transaction.
  If
  Joe and Jane both send email with data, you're processing Jane's data in
  Joe's transaction.  As as well your email fetching time is getting
 included
  in your transaction time.  If your email server hangs, your transaction
  hangs.
 
  So instead of:
 
  // TRANSACTION START
  @Schedule(...)
  public void processPendingEmail() {
 
  ListEmail emails = fetchEmail();
  for (Email email : emails) {
 // parse email and insert records
  }
  }
  // TRANSACTION END
 
  You want:
 
  @Schedule(...)
  public void processPendingEmail() {
 
  ListEmail emails = fetchEmail();
  for (Email email : emails) {
 // TRANSACTION START
 // parse email and insert records
 // TRANSACTION END
  }
  }
 
  This can be as easy as marking the `processPendingEmail` as
  `@TransactionAttribute(NEVER)` then moving the insert logic to another
 EJB
  method.  Just make sure that if you move that logic to another method of
  the same EJB that you don't invoke that method with a `this` reference or
  you won't get container-managed transactions on that method call. (see
 this
  reference for why):
 
 
 
 http://stackoverflow.com/questions/3381002/how-is-the-return-value-of-sessioncontext-getbusinessobject-different-from-th/3384128#3384128
 
  Hope this helps!
 
 
  -David
 
  On Dec 11, 2012, at 12:21 PM, Howard W. Smith, Jr. 
  smithh032...@gmail.com wrote:
 
   Prior to adding this @Stateless EJB (that's invoked via @Schedule every
  10
   minutes), my JSF web app has been running 100% okay without any
 deadlocks
   or anything else like that.
  
   So, is it okay if I just add optimistic locking, em.lock(entity,
   LockType.Optimistic) before any/all em.persist(...) that I have in the
   @Stateless EJB?
  
   JSF web app is working okay, because the JSF components call multiple
   @Stateless EJB that usually do any of the following, SELECT, UPDATE,
   DELETE,  on one table at time.
  
  
  
   On Tue, Dec 11, 2012 at 2:35 PM, Howard W. Smith, Jr. 
   smithh032...@gmail.com wrote:
  
   You're absolutely right, David. Thanks for the response.
  
   It is a transaction issue and the @Stateless EJB is holding a
  transaction
   open and writing into 6 to 10 tables in a single database, which is
 also
   the database accessed by JSF components (by endusers). The @Stateless
  EJB
   is on a timer ( @Schedule), checks email server for incoming requests
  from
   customers, and takes that data and inserts it into database, so
 endusers
   don't have to manually enter the data (that they receive from these
   formmail results).
  
   Below is the exception that I saw maybe an hour or two ago, and I
   confirmed that it's a locking/transaction issue, and from the
  stacktrace,
   it seems as though the JSF components cannot access the table, that is
   evidently locked by @Stateless EJB. The @Stateless @EJB updated all
  tables
   on my faster/dev/test server on average of 1 to 2 seconds, and/so in
 my
   opinion, the data is not that much at all, maybe a few kbytes of data
  being
   inserted into the database. Anyway, I've been reading up on
 pessismistic
   and optimistic locking, and that's where I was in my reading before I
   noticed your email. I really don't know what the database or JPA is
   defaulting to, because honestly, I have no code that specifies any
  locks.
   Recently, I only added query hint (readonly) to JPA queries.
  Personally, I
   feel as though I can use 'no locking' for these inserts, since this is
   brand new data, but the JSF components may be locking the
 entites/tables
   during simple queries (SELECT statements).  Feel free to change

Re: Why are Tomcat's threads more costly than background threads?

2012-12-12 Thread Jean-Louis MONTEIRO

Finder methods are usually methods to retrieve an entity or a set of
entities.
Basically, it means Qurey.getSingleResult() or getResultList().

By default any method is REQUIRED in EJBs so that means a transaction will
be created whereas it's not required as per the spec.
When you just want to browse data, you actually don't need a transaction.

A special attention must be of course placed on JPA relationships and more
especially LAZY tagged ones.
As soon as the transaction finished or as soon as they are retrieved when
no transaction is used, they are in an unmanaged state which means that you
cannot navigate through lazy relationships.

So before retrieving an object graph to browse it in the front layer, you
basically need to know what you want to display in order to pre load all
relationships (using a left join fetch or so query).

Jean-Louis

2012/12/12 Howard W. Smith, Jr. smithh032...@gmail.com

Jean-Louis, please clarify 'finder methods' and/provide a short example,
thanks.

On Wed, Dec 12, 2012 at 3:08 AM, Jean-Louis MONTEIRO jeano...@gmail.com
wrote:

I tend to recommend to set any finder methods to SUPPORT which is far
better in terms of performances and allow reusability of finder methods
in
business methods.

JLouis

2012/12/11 David Blevins david.blev...@gmail.com

Setting the LockModeType is good. Understand, though, that all the
values
in LockModeType are not guaranteed to be supported in all databases.

It doesn't address the core problem in that the transaction scope is
likely too broad. As a general rule *always* ensure your transactions
are
as *small* as possible. In order to be very conscious about it it can
help
to actually document the code and justify why two or more operations
need
to be in the same transaction. It'll force you to think about
transactions
more deliberately and ask yourself other questions like where are my
transactions starting and stopping in the first place.

In this particular situation, with default settings, you'll get one
transaction started for you when the @Schedule method fires. So
essentially, you're batch processing unrelated data in one transaction.
If
Joe and Jane both send email with data, you're processing Jane's data
in
Joe's transaction. As as well your email fetching time is getting
included
in your transaction time. If your email server hangs, your transaction
hangs.

So instead of:

// TRANSACTION START
@Schedule(...)
public void processPendingEmail() {

ListEmail emails = fetchEmail();
for (Email email : emails) {
// parse email and insert records
}
}
// TRANSACTION END

You want:

@Schedule(...)
public void processPendingEmail() {

ListEmail emails = fetchEmail();
for (Email email : emails) {
// TRANSACTION START
// parse email and insert records
// TRANSACTION END
}
}

This can be as easy as marking the `processPendingEmail` as
`@TransactionAttribute(NEVER)` then moving the insert logic to another
EJB
method. Just make sure that if you move that logic to another method
of
the same EJB that you don't invoke that method with a `this` reference
or
you won't get container-managed transactions on that method call. (see
this
reference for why):

http://stackoverflow.com/questions/3381002/how-is-the-return-value-of-sessioncontext-getbusinessobject-different-from-th/3384128#3384128

Hope this helps!

-David

On Dec 11, 2012, at 12:21 PM, Howard W. Smith, Jr.
smithh032...@gmail.com wrote:

Prior to adding this @Stateless EJB (that's invoked via @Schedule
every
10
minutes), my JSF web app has been running 100% okay without any
deadlocks
or anything else like that.

So, is it okay if I just add optimistic locking, em.lock(entity,
LockType.Optimistic) before any/all em.persist(...) that I have in
the
@Stateless EJB?

JSF web app is working okay, because the JSF components call multiple
@Stateless EJB that usually do any of the following, SELECT, UPDATE,
DELETE, on one table at time.

On Tue, Dec 11, 2012 at 2:35 PM, Howard W. Smith, Jr.
smithh032...@gmail.com wrote:

You're absolutely right, David. Thanks for the response.

It is a transaction issue and the @Stateless EJB is holding a
transaction
open and writing into 6 to 10 tables in a single database, which is
also
the database accessed by JSF components (by endusers). The
@Stateless
EJB
is on a timer ( @Schedule), checks email server for incoming
requests
from
customers, and takes that data and inserts it into database, so
endusers
don't have to manually enter the data (that they receive from these
formmail results).

Re: Why are Tomcat's threads more costly than background threads?

2012-12-12 Thread Howard W. Smith, Jr.

okay, thanks. I was aware of that already. ;)

On Wed, Dec 12, 2012 at 4:03 AM, Jean-Louis MONTEIRO jeano...@gmail.comwrote:

Finder methods are usually methods to retrieve an entity or a set of
entities.
Basically, it means Qurey.getSingleResult() or getResultList().

So before retrieving an object graph to browse it in the front layer, you
basically need to know what you want to display in order to pre load all
relationships (using a left join fetch or so query).

Jean-Louis

2012/12/12 Howard W. Smith, Jr. smithh032...@gmail.com

Jean-Louis, please clarify 'finder methods' and/provide a short example,
thanks.

On Wed, Dec 12, 2012 at 3:08 AM, Jean-Louis MONTEIRO jeano...@gmail.com
wrote:

I tend to recommend to set any finder methods to SUPPORT which is far
better in terms of performances and allow reusability of finder methods
in
business methods.

JLouis

2012/12/11 David Blevins david.blev...@gmail.com

Setting the LockModeType is good. Understand, though, that all the
values
in LockModeType are not guaranteed to be supported in all databases.

It doesn't address the core problem in that the transaction scope is
likely too broad. As a general rule *always* ensure your
transactions
are
as *small* as possible. In order to be very conscious about it it
can
help
to actually document the code and justify why two or more operations
need
to be in the same transaction. It'll force you to think about
transactions
more deliberately and ask yourself other questions like where are my
transactions starting and stopping in the first place.

In this particular situation, with default settings, you'll get one
transaction started for you when the @Schedule method fires. So
essentially, you're batch processing unrelated data in one
transaction.
If
Joe and Jane both send email with data, you're processing Jane's data
in
Joe's transaction. As as well your email fetching time is getting
included
in your transaction time. If your email server hangs, your
transaction
hangs.

So instead of:

// TRANSACTION START
@Schedule(...)
public void processPendingEmail() {

ListEmail emails = fetchEmail();
for (Email email : emails) {
// parse email and insert records
}
}
// TRANSACTION END

You want:

@Schedule(...)
public void processPendingEmail() {

ListEmail emails = fetchEmail();
for (Email email : emails) {
// TRANSACTION START
// parse email and insert records
// TRANSACTION END
}
}

This can be as easy as marking the `processPendingEmail` as
`@TransactionAttribute(NEVER)` then moving the insert logic to
another
EJB
method. Just make sure that if you move that logic to another method
of
the same EJB that you don't invoke that method with a `this`
reference
or
you won't get container-managed transactions on that method call.
(see
this
reference for why):

http://stackoverflow.com/questions/3381002/how-is-the-return-value-of-sessioncontext-getbusinessobject-different-from-th/3384128#3384128

Hope this helps!

-David

On Dec 11, 2012, at 12:21 PM, Howard W. Smith, Jr.
smithh032...@gmail.com wrote:

Prior to adding this @Stateless EJB (that's invoked via @Schedule
every
10
minutes), my JSF web app has been running 100% okay without any
deadlocks
or anything else like that.

So, is it okay if I just add optimistic locking, em.lock(entity,
LockType.Optimistic) before any/all em.persist(...) that I have in
the
@Stateless EJB?

JSF web app is working okay, because the JSF components call
multiple
@Stateless EJB that usually do any of the following, SELECT,
UPDATE,
DELETE, on one table at time.

On Tue, Dec 11, 2012 at 2:35 PM, Howard W. Smith, Jr.
smithh032...@gmail.com wrote:

You're absolutely right, David. Thanks for the response.

Re: Why are Tomcat's threads more costly than background threads?

Wow, i'm reading this now, because I just experienced an issue on my
production server that is TomEE 1.5.1 (Tomcat 7.0.34), and the whole server
locked up all because I had a @Stateless EJB inserting data into multiple
tables in the database, because @Schedule timed event triggered the EJB to
check email server for incoming (customer) requests, and it literally took
down the server. I was on it as well as few other endusers, and then one
enduser captured a LOCK error and the screen capture (photo/pic) had an
error message that showed a long SQL query with datatable table and column
names t0..., t0...

What I primarily saw was the word 'lock' at the top of that, and we
definitely experienced a lockup. I'm about to check server logs and read
this article.

The @Stateless EJB had one transaction (entitymanager / persistence
context) that made database updates to multiple tables in the database. I
am only using entitymanager.persist(), flush(), and few lookups/queries
during that process.

But other endusers (including myself) could not do simple queries against
the database at all. Most of my queries contain query hints (readonly,
statement caching).

Also, I never seen this behavior at all, but this is first time I added
@Stateless EJB along with @Schedule that does database updates 'during
business hours'. I thought this would be a no-brainer, but I guess it's
not. Again, the server is TomEE 1.5.1 (tomcat 7.0.34).

Any advise, then please let me know. Onto reading this post now. Thanks. :)



On Tue, Dec 11, 2012 at 10:49 AM, Julien Martin bal...@gmail.com wrote:

 Thank you very much for this exhaustive reply Christopher.

 2012/12/11 Christopher Schultz ch...@christopherschultz.net

  -BEGIN PGP SIGNED MESSAGE-
  Hash: SHA256
 
  Julien,
 
  Warning: this is long. Like, André-or-Mark-Eggers long.
 
  On 12/11/12 7:30 AM, Julien Martin wrote:
   I am in reference to the following blog entry: Blog
   entry
 
 http://blog.springsource.org/2012/05/06/spring-mvc-3-2-preview-introducing-servlet-3-async-support
  
  
  
  about Spring MVC 3.2 asynchronous support.
  
   I understand Tomcat uses a thread pool in order to serve http/web
   requests. Furthermore, the above article seems to indicate that
   Spring MVC asynchronous support relieves Tomcat's thread pool and
   allows for better concurrency in the webapp by using background
   threads for heavy-lift operations.
 
  I believe you are misinterpreting what that post has to say. It's not
  that a background thread itself is more efficient, it's that
  processing that does not need to communicate with the client can be
  de-coupled from the request-processing thread-pool that exists for
  that purpose.
 
  An example - right from the blog post - will make much more sense than
  what I wrote above. Let's take the example of sending an email
  message. First, some assumptions:
 
  1. Sending an email takes a long time (say, 5 seconds)
  2. The client does not need confirmation that the email has been sent
 
  If your were to write a classic servlet, it would look something
  like this:
 
  doPost() {
validateOrder();
 
queueOrder();
 
sendOrderConfirmation(); // This is the email
 
response.sendRedirect(/order_complete.jsp);
  }
 
  Let's say that validation takes 500ms, queuing takes 800ms, and
  emailing (as above) takes 5000ms. That means that the request, from
  the client perspective, takes 6300ms (6.3 sec). That's a noticeable
 delay.
 
  Also, during that whole time, a single request-processing thread (from
  Tomcat's thread pool) is tied-up, meaning that no other requests can
  be processed by that same thread.
 
  If you have a thread pool of size=1 (foolish, yet instructive), it
  means you can only process a single transaction every 6.3 seconds.
 
  Lets re-write the servlet with a background thread -- no
  asynchronous stuff from the servlet API, but just with a simple
  background thread:
 
  doPost() {
validateOrder();
 
queueOrder();
 
(new Thread() {
  public void run() {
sendOrderConfirmation();
  }
}).start();
 
response.sendRedirect(order_complete.jsp);
  }
 
  So, now the email is being sent by a background thread: the response
  returns to the client after 1.3 seconds which is a significant
  improvement. Now, we can handle a request once every 1.3 seconds with
  a request-processing thread-pool of size=1.
 
  Note that a better implementation of the above would be to use a
  thread pool for this sort of thing instead of creating a new thread
  for every request. This is what Spring provides. It's not that Spring
  can do a better job of thread management, it's that Tomcat's thread
  pool is special: it's the only one that can actually dispatch client
  requests. Off-loading onto another thread pool for background
  processing means more client requests can be handled with a smaller
  (or same-sized) pool.
 
  Looking above, you might notice that the validateOrder() and

Re: Why are Tomcat's threads more costly than background threads?

2012-12-11 Thread David Blevins

First a tip. Next time the server locks up, immediately do a thread dump.

- http://docs.oracle.com/javase/1.5.0/docs/tooldocs/share/jstack.html

Second, it sounds like this locking issue isn't a threading issue, but a
transaction issue. Accessing all your databases in the same JTA transaction is
not advised unless you're doing something like withdrawing $100 from database A
and putting it in database B. This kind of work is fine to do, but you want it
to be as focused and razor sharp as possible -- only do it with the
understanding you're locking the referenced tables/rows on both databases while
that transaction is open. This can be tragic if pessimistic locking is used,
less of a problem with more optimistic locking options. These options are all
database specific. Again, this kind of thing is fine if the task requires it
-- there's no other reliable way to solve the transfer data from one database
to the next problem other then to enroll them both in a JTA transaction -- but
such a thing must be done with full understanding of the cost.

What kind of work are you attempting to do with this job that accesses all your
databases?

-David

On Dec 11, 2012, at 8:40 AM, Howard W. Smith, Jr. smithh032...@gmail.com
wrote:

Okay, TomEE committers/users, I need your help/advice on this one. Please
read what I wrote in the email below.

I am considering to use the @Asynchronous on one of the methods on
@Stateless EJB to complete this task, but I think I still may need to do
something on the Persistence end to avoid locking up the entire app,
database, etc...

I guess this my first time experiencing a DEADLOCK, but I could be wrong. :)

On Tue, Dec 11, 2012 at 11:01 AM, Howard W. Smith, Jr.
smithh032...@gmail.com wrote:

Wow, i'm reading this now, because I just experienced an issue on my
production server that is TomEE 1.5.1 (Tomcat 7.0.34), and the whole server
locked up all because I had a @Stateless EJB inserting data into multiple
tables in the database, because @Schedule timed event triggered the EJB to
check email server for incoming (customer) requests, and it literally took
down the server. I was on it as well as few other endusers, and then one
enduser captured a LOCK error and the screen capture (photo/pic) had an
error message that showed a long SQL query with datatable table and column
names t0..., t0...

What I primarily saw was the word 'lock' at the top of that, and we
definitely experienced a lockup. I'm about to check server logs and read
this article.

The @Stateless EJB had one transaction (entitymanager / persistence
context) that made database updates to multiple tables in the database. I
am only using entitymanager.persist(), flush(), and few lookups/queries
during that process.

But other endusers (including myself) could not do simple queries against
the database at all. Most of my queries contain query hints (readonly,
statement caching).

Also, I never seen this behavior at all, but this is first time I added
@Stateless EJB along with @Schedule that does database updates 'during
business hours'. I thought this would be a no-brainer, but I guess it's
not. Again, the server is TomEE 1.5.1 (tomcat 7.0.34).

Any advise, then please let me know. Onto reading this post now. Thanks. :)

On Tue, Dec 11, 2012 at 10:49 AM, Julien Martin bal...@gmail.com wrote:

Thank you very much for this exhaustive reply Christopher.

2012/12/11 Christopher Schultz ch...@christopherschultz.net

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Julien,

Warning: this is long. Like, André-or-Mark-Eggers long.

On 12/11/12 7:30 AM, Julien Martin wrote:
I am in reference to the following blog entry: Blog
entry

http://blog.springsource.org/2012/05/06/spring-mvc-3-2-preview-introducing-servlet-3-async-support

about Spring MVC 3.2 asynchronous support.

I understand Tomcat uses a thread pool in order to serve http/web
requests. Furthermore, the above article seems to indicate that
Spring MVC asynchronous support relieves Tomcat's thread pool and
allows for better concurrency in the webapp by using background
threads for heavy-lift operations.

I believe you are misinterpreting what that post has to say. It's not
that a background thread itself is more efficient, it's that
processing that does not need to communicate with the client can be
de-coupled from the request-processing thread-pool that exists for
that purpose.

An example - right from the blog post - will make much more sense than
what I wrote above. Let's take the example of sending an email
message. First, some assumptions:

1. Sending an email takes a long time (say, 5 seconds)
2. The client does not need confirmation that the email has been sent

If your were to write a classic servlet, it would look something
like this:

doPost() {
validateOrder();

queueOrder();

sendOrderConfirmation(); // This is the email

Re: Why are Tomcat's threads more costly than background threads?

You're absolutely right, David. Thanks for the response.

It is a transaction issue and the @Stateless EJB is holding a transaction
open and writing into 6 to 10 tables in a single database, which is also
the database accessed by JSF components (by endusers). The @Stateless EJB
is on a timer ( @Schedule), checks email server for incoming requests from
customers, and takes that data and inserts it into database, so endusers
don't have to manually enter the data (that they receive from these
formmail results).

Below is the exception that I saw maybe an hour or two ago, and I confirmed
that it's a locking/transaction issue, and from the stacktrace, it seems as
though the JSF components cannot access the table, that is evidently locked
by @Stateless EJB. The @Stateless @EJB updated all tables on my
faster/dev/test server on average of 1 to 2 seconds, and/so in my opinion,
the data is not that much at all, maybe a few kbytes of data being inserted
into the database. Anyway, I've been reading up on pessismistic and
optimistic locking, and that's where I was in my reading before I noticed
your email. I really don't know what the database or JPA is defaulting to,
because honestly, I have no code that specifies any locks. Recently, I only
added query hint (readonly) to JPA queries. Personally, I feel as though I
can use 'no locking' for these inserts, since this is brand new data, but
the JSF components may be locking the entites/tables during simple queries
(SELECT statements).  Feel free to change this email/topic to a more
suitable topic/subject, if necessary. :)


enCore(Unknown Source)
at org.apache.derby.impl.sql.execute.BasicNoPutResultSetImpl.open(Unknown
Source)
at org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(Unknown
Source)
at org.apache.derby.impl.sql.GenericPreparedStatement.execute(Unknown
Source)
... 126 more
javax.ejb.EJBException: The bean encountered a non-application exception;
nested exception is:
Exception [EclipseLink-4002] (Eclipse Persistence Services -
2.3.2.v2025-r10461):
org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: java.sql.SQLTransactionRollbackException: A lock could
not be obtained within the time requested
Error Code: 3
Call: SELECT t0.ORDER_ID, t0.BALANCE, t0.CANCELLED, t0.CANCELLED_DATE,
t0.CHANGED_DT, t0.CONFIRMED, t0.CONFIRMED_DATE, t0.CONTRACT_SENT,
t0.CONTRACT_SENT_DATE, t0.CREATED_BY, t0.CREATED_DT, t0.DEPOSIT,
t0.DEPOSIT_PAID, t0.DEPOSIT_PAID_DATE, t0.DISCOUNT, t0.DISCOUNT_RATE,
t0.INTRASTATE, t0.INVOICE_SENT, t0.INVOICE_SENT_DATE, t0.PAID_IN_FULL,
t0.PAID_IN_FULL_DATE, t0.PASSENGERS, t0.QUOTE_SENT, t0.QUOTE_SENT_DATE,
t0.REPORT_DATE_TIME, t0.RETURN_DATE_TIME, t0.TOTAL_COST, t0.TRIP_DATE_TIME,
t0.BANK_FOR_BALANCE_ID, t0.BANK_FOR_DEPOSIT_ID, t0.CUSTOMER_ID,
t0.DETAILS_ID, t0.METHOD_OF_PAYMENT_FOR_BALANCE_ID,
t0.METHOD_OF_PAYMENT_FOR_DEPOSIT_ID, t0.NOTES_ID, t0.ROUTES_ID,
t0.TEMPLATE_ID FROM ORDERS t0 WHERE (t0.TRIP_DATE_TIME BETWEEN {ts
'2012-12-06 00:00:00.0'} AND {ts '2012-12-06 23:59:59.0'}) OR
(t0.RETURN_DATE_TIME BETWEEN {ts '2012-12-06 00:00:00.0'} AND {ts
'2012-12-06 23:59:59.0'})) OR ({ts '2012-12-06 00:00:00.0'} BETWEEN
t0.TRIP_DATE_TIME AND t0.RETURN_DATE_TIME)) OR ({ts '2012-12-06
23:59:59.0'} BETWEEN t0.TRIP_DATE_TIME AND t0.RETURN_DATE_TIME)) AND NOT
EXISTS (SELECT 1 FROM ORDER_TEMPLATE t2, ORDER_TEMPLATE t1 WHERE
((t1.TEMPLATE_ID = t2.TEMPLATE_ID) AND (t2.TEMPLATE_ID = t0.TEMPLATE_ID)))
) ORDER BY t0.REPORT_DATE_TIME ASC, t0.TRIP_DATE_TIME ASC
Query: ReadAllQuery(name=Orders.findAllBookedTrips referenceClass=Orders
sql=SELECT t0.ORDER_ID, t0.BALANCE, t0.CANCELLED, t0.CANCELLED_DATE,
t0.CHANGED_DT, t0.CONFIRMED, t0.CONFIRMED_DATE, t0.CONTRACT_SENT,
t0.CONTRACT_SENT_DATE, t0.CREATED_BY, t0.CREATED_DT, t0.DEPOSIT,
t0.DEPOSIT_PAID, t0.DEPOSIT_PAID_DATE, t0.DISCOUNT, t0.DISCOUNT_RATE,
t0.INTRASTATE, t0.INVOICE_SENT, t0.INVOICE_SENT_DATE, t0.PAID_IN_FULL,
t0.PAID_IN_FULL_DATE, t0.PASSENGERS, t0.QUOTE_SENT, t0.QUOTE_SENT_DATE,
t0.REPORT_DATE_TIME, t0.RETURN_DATE_TIME, t0.TOTAL_COST, t0.TRIP_DATE_TIME,
t0.BANK_FOR_BALANCE_ID, t0.BANK_FOR_DEPOSIT_ID, t0.CUSTOMER_ID,
t0.DETAILS_ID, t0.METHOD_OF_PAYMENT_FOR_BALANCE_ID,
t0.METHOD_OF_PAYMENT_FOR_DEPOSIT_ID, t0.NOTES_ID, t0.ROUTES_ID,
t0.TEMPLATE_ID FROM ORDERS t0 WHERE (t0.TRIP_DATE_TIME BETWEEN ? AND ?)
OR (t0.RETURN_DATE_TIME BETWEEN ? AND ?)) OR (? BETWEEN t0.TRIP_DATE_TIME
AND t0.RETURN_DATE_TIME)) OR (? BETWEEN t0.TRIP_DATE_TIME AND
t0.RETURN_DATE_TIME)) AND NOT EXISTS (SELECT ? FROM ORDER_TEMPLATE t2,
ORDER_TEMPLATE t1 WHERE ((t1.TEMPLATE_ID = t2.TEMPLATE_ID) AND
(t2.TEMPLATE_ID = t0.TEMPLATE_ID))) ) ORDER BY t0.REPORT_DATE_TIME ASC,
t0.TRIP_DATE_TIME ASC)
at
org.apache.openejb.core.ivm.BaseEjbProxyHandler.convertException(BaseEjbProxyHandler.java:363)
at
org.apache.openejb.core.ivm.BaseEjbProxyHandler.invoke(BaseEjbProxyHandler.java:283)
at
jpa.session.OrdersFacade$LocalBeanProxy.filterBy(jpa/session/OrdersFacade.java)
at

Re: Why are Tomcat's threads more costly than background threads?

I think I'm going to avoid the single transaction (via single @Stateless
EJB) approach, and use other @Stateless EJBs to make the database table
updates. This approach works everywhere else throughout the web app.

I'll just keep the @Stateless EJB for the purpose of being invoked by
@Schedule, and it will be the 'driver', and call the
already-defined-and-already-used other @Stateless EJBs that contain
Persistent context and entity manager to update tables, so this clearly
will be a multiple transaction commit, which should 'not' lock any tables.
:)



On Tue, Dec 11, 2012 at 3:21 PM, Howard W. Smith, Jr. 
smithh032...@gmail.com wrote:

 Prior to adding this @Stateless EJB (that's invoked via @Schedule every 10
 minutes), my JSF web app has been running 100% okay without any deadlocks
 or anything else like that.

 So, is it okay if I just add optimistic locking, em.lock(entity,
 LockType.Optimistic) before any/all em.persist(...) that I have in the
 @Stateless EJB?

 JSF web app is working okay, because the JSF components call multiple
 @Stateless EJB that usually do any of the following, SELECT, UPDATE,
 DELETE,  on one table at time.



 On Tue, Dec 11, 2012 at 2:35 PM, Howard W. Smith, Jr. 
 smithh032...@gmail.com wrote:

 You're absolutely right, David. Thanks for the response.

 It is a transaction issue and the @Stateless EJB is holding a transaction
 open and writing into 6 to 10 tables in a single database, which is also
 the database accessed by JSF components (by endusers). The @Stateless EJB
 is on a timer ( @Schedule), checks email server for incoming requests from
 customers, and takes that data and inserts it into database, so endusers
 don't have to manually enter the data (that they receive from these
 formmail results).

 Below is the exception that I saw maybe an hour or two ago, and I
 confirmed that it's a locking/transaction issue, and from the stacktrace,
 it seems as though the JSF components cannot access the table, that is
 evidently locked by @Stateless EJB. The @Stateless @EJB updated all tables
 on my faster/dev/test server on average of 1 to 2 seconds, and/so in my
 opinion, the data is not that much at all, maybe a few kbytes of data being
 inserted into the database. Anyway, I've been reading up on pessismistic
 and optimistic locking, and that's where I was in my reading before I
 noticed your email. I really don't know what the database or JPA is
 defaulting to, because honestly, I have no code that specifies any locks.
 Recently, I only added query hint (readonly) to JPA queries. Personally, I
 feel as though I can use 'no locking' for these inserts, since this is
 brand new data, but the JSF components may be locking the entites/tables
 during simple queries (SELECT statements).  Feel free to change this
 email/topic to a more suitable topic/subject, if necessary. :)

Re: Why are Tomcat's threads more costly than background threads?

2012-12-11 Thread David Blevins

Setting the LockModeType is good.  Understand, though, that all the values in 
LockModeType are not guaranteed to be supported in all databases.

It doesn't address the core problem in that the transaction scope is likely too 
broad.  As a general rule *always* ensure your transactions are as *small* as 
possible.  In order to be very conscious about it it can help to actually 
document the code and justify why two or more operations need to be in the same 
transaction.  It'll force you to think about transactions more deliberately and 
ask yourself other questions like where are my transactions starting and 
stopping in the first place.

In this particular situation, with default settings, you'll get one transaction 
started for you when the @Schedule method fires.  So essentially, you're batch 
processing unrelated data in one transaction.  If Joe and Jane both send email 
with data, you're processing Jane's data in Joe's transaction.  As as well your 
email fetching time is getting included in your transaction time.  If your 
email server hangs, your transaction hangs.

So instead of:

// TRANSACTION START
@Schedule(...)
public void processPendingEmail() {

ListEmail emails = fetchEmail();  
for (Email email : emails) {
   // parse email and insert records
}
}
// TRANSACTION END

You want:

@Schedule(...)
public void processPendingEmail() {

ListEmail emails = fetchEmail();  
for (Email email : emails) {
   // TRANSACTION START
   // parse email and insert records
   // TRANSACTION END
}
}

This can be as easy as marking the `processPendingEmail` as 
`@TransactionAttribute(NEVER)` then moving the insert logic to another EJB 
method.  Just make sure that if you move that logic to another method of the 
same EJB that you don't invoke that method with a `this` reference or you won't 
get container-managed transactions on that method call. (see this reference for 
why):


http://stackoverflow.com/questions/3381002/how-is-the-return-value-of-sessioncontext-getbusinessobject-different-from-th/3384128#3384128

Hope this helps!


-David

On Dec 11, 2012, at 12:21 PM, Howard W. Smith, Jr. smithh032...@gmail.com 
wrote:

 Prior to adding this @Stateless EJB (that's invoked via @Schedule every 10
 minutes), my JSF web app has been running 100% okay without any deadlocks
 or anything else like that.
 
 So, is it okay if I just add optimistic locking, em.lock(entity,
 LockType.Optimistic) before any/all em.persist(...) that I have in the
 @Stateless EJB?
 
 JSF web app is working okay, because the JSF components call multiple
 @Stateless EJB that usually do any of the following, SELECT, UPDATE,
 DELETE,  on one table at time.
 
 
 
 On Tue, Dec 11, 2012 at 2:35 PM, Howard W. Smith, Jr. 
 smithh032...@gmail.com wrote:
 
 You're absolutely right, David. Thanks for the response.
 
 It is a transaction issue and the @Stateless EJB is holding a transaction
 open and writing into 6 to 10 tables in a single database, which is also
 the database accessed by JSF components (by endusers). The @Stateless EJB
 is on a timer ( @Schedule), checks email server for incoming requests from
 customers, and takes that data and inserts it into database, so endusers
 don't have to manually enter the data (that they receive from these
 formmail results).
 
 Below is the exception that I saw maybe an hour or two ago, and I
 confirmed that it's a locking/transaction issue, and from the stacktrace,
 it seems as though the JSF components cannot access the table, that is
 evidently locked by @Stateless EJB. The @Stateless @EJB updated all tables
 on my faster/dev/test server on average of 1 to 2 seconds, and/so in my
 opinion, the data is not that much at all, maybe a few kbytes of data being
 inserted into the database. Anyway, I've been reading up on pessismistic
 and optimistic locking, and that's where I was in my reading before I
 noticed your email. I really don't know what the database or JPA is
 defaulting to, because honestly, I have no code that specifies any locks.
 Recently, I only added query hint (readonly) to JPA queries. Personally, I
 feel as though I can use 'no locking' for these inserts, since this is
 brand new data, but the JSF components may be locking the entites/tables
 during simple queries (SELECT statements).  Feel free to change this
 email/topic to a more suitable topic/subject, if necessary. :)

Re: Why are Tomcat's threads more costly than background threads?

2012-12-11 Thread David Blevins

Funny, I just sent that suggestion :)  But it took me too long to type it -- 
didn't see your response came in before I finished :)

-David

On Dec 11, 2012, at 12:53 PM, Howard W. Smith, Jr. smithh032...@gmail.com 
wrote:

 I think I'm going to avoid the single transaction (via single @Stateless
 EJB) approach, and use other @Stateless EJBs to make the database table
 updates. This approach works everywhere else throughout the web app.
 
 I'll just keep the @Stateless EJB for the purpose of being invoked by
 @Schedule, and it will be the 'driver', and call the
 already-defined-and-already-used other @Stateless EJBs that contain
 Persistent context and entity manager to update tables, so this clearly
 will be a multiple transaction commit, which should 'not' lock any tables.
 :)
 
 
 
 On Tue, Dec 11, 2012 at 3:21 PM, Howard W. Smith, Jr. 
 smithh032...@gmail.com wrote:
 
 Prior to adding this @Stateless EJB (that's invoked via @Schedule every 10
 minutes), my JSF web app has been running 100% okay without any deadlocks
 or anything else like that.
 
 So, is it okay if I just add optimistic locking, em.lock(entity,
 LockType.Optimistic) before any/all em.persist(...) that I have in the
 @Stateless EJB?
 
 JSF web app is working okay, because the JSF components call multiple
 @Stateless EJB that usually do any of the following, SELECT, UPDATE,
 DELETE,  on one table at time.
 
 
 
 On Tue, Dec 11, 2012 at 2:35 PM, Howard W. Smith, Jr. 
 smithh032...@gmail.com wrote:
 
 You're absolutely right, David. Thanks for the response.
 
 It is a transaction issue and the @Stateless EJB is holding a transaction
 open and writing into 6 to 10 tables in a single database, which is also
 the database accessed by JSF components (by endusers). The @Stateless EJB
 is on a timer ( @Schedule), checks email server for incoming requests from
 customers, and takes that data and inserts it into database, so endusers
 don't have to manually enter the data (that they receive from these
 formmail results).
 
 Below is the exception that I saw maybe an hour or two ago, and I
 confirmed that it's a locking/transaction issue, and from the stacktrace,
 it seems as though the JSF components cannot access the table, that is
 evidently locked by @Stateless EJB. The @Stateless @EJB updated all tables
 on my faster/dev/test server on average of 1 to 2 seconds, and/so in my
 opinion, the data is not that much at all, maybe a few kbytes of data being
 inserted into the database. Anyway, I've been reading up on pessismistic
 and optimistic locking, and that's where I was in my reading before I
 noticed your email. I really don't know what the database or JPA is
 defaulting to, because honestly, I have no code that specifies any locks.
 Recently, I only added query hint (readonly) to JPA queries. Personally, I
 feel as though I can use 'no locking' for these inserts, since this is
 brand new data, but the JSF components may be locking the entites/tables
 during simple queries (SELECT statements).  Feel free to change this
 email/topic to a more suitable topic/subject, if necessary. :)

Re: Why are Tomcat's threads more costly than background threads?

Done. testing now and watching JVM. I need to read that article you shared,
so i can learn a bit about heap dump. I see the heap dump button there, but
never used it 'yet'. :)

On Tue, Dec 11, 2012 at 3:53 PM, Howard W. Smith, Jr. 
smithh032...@gmail.com wrote:

 I think I'm going to avoid the single transaction (via single @Stateless
 EJB) approach, and use other @Stateless EJBs to make the database table
 updates. This approach works everywhere else throughout the web app.

 I'll just keep the @Stateless EJB for the purpose of being invoked by
 @Schedule, and it will be the 'driver', and call the
 already-defined-and-already-used other @Stateless EJBs that contain
 Persistent context and entity manager to update tables, so this clearly
 will be a multiple transaction commit, which should 'not' lock any tables.
 :)



 On Tue, Dec 11, 2012 at 3:21 PM, Howard W. Smith, Jr. 
 smithh032...@gmail.com wrote:

 Prior to adding this @Stateless EJB (that's invoked via @Schedule every
 10 minutes), my JSF web app has been running 100% okay without any
 deadlocks or anything else like that.

 So, is it okay if I just add optimistic locking, em.lock(entity,
 LockType.Optimistic) before any/all em.persist(...) that I have in the
 @Stateless EJB?

 JSF web app is working okay, because the JSF components call multiple
 @Stateless EJB that usually do any of the following, SELECT, UPDATE,
 DELETE,  on one table at time.



 On Tue, Dec 11, 2012 at 2:35 PM, Howard W. Smith, Jr. 
 smithh032...@gmail.com wrote:

 You're absolutely right, David. Thanks for the response.

 It is a transaction issue and the @Stateless EJB is holding a
 transaction open and writing into 6 to 10 tables in a single database,
 which is also the database accessed by JSF components (by endusers). The
 @Stateless EJB is on a timer ( @Schedule), checks email server for incoming
 requests from customers, and takes that data and inserts it into database,
 so endusers don't have to manually enter the data (that they receive from
 these formmail results).

 Below is the exception that I saw maybe an hour or two ago, and I
 confirmed that it's a locking/transaction issue, and from the stacktrace,
 it seems as though the JSF components cannot access the table, that is
 evidently locked by @Stateless EJB. The @Stateless @EJB updated all tables
 on my faster/dev/test server on average of 1 to 2 seconds, and/so in my
 opinion, the data is not that much at all, maybe a few kbytes of data being
 inserted into the database. Anyway, I've been reading up on pessismistic
 and optimistic locking, and that's where I was in my reading before I
 noticed your email. I really don't know what the database or JPA is
 defaulting to, because honestly, I have no code that specifies any locks.
 Recently, I only added query hint (readonly) to JPA queries. Personally, I
 feel as though I can use 'no locking' for these inserts, since this is
 brand new data, but the JSF components may be locking the entites/tables
 during simple queries (SELECT statements).  Feel free to change this
 email/topic to a more suitable topic/subject, if necessary. :)

Re: Why are Tomcat's threads more costly than background threads?

Shaking my head... test results were not good at all.

1. @StatelessEJB EmailStatelessBean has @Schedule getEmails()
2. @EmailStatelessBean has multiple @EJB references to @Stateless
(sessionfacade) classes that are the DAO classes
3. EmailStatelessBean gets invoked when triggered by @Schedule
4. EmailStatelessBean execution locks the tables, still, and locks up the
entire app, and seems to take longer...since endusers (myself) are making
requests against database (and web app).
5. Last but not least, EmailStatelessBean failed to commit the changes;
transaction rolled back for EmailStatelessBean as well as enduser requests.

So, my workaround options include the following:

1. EmailStatelessBean, when invoked by @Schedule method, will check the CDI
@ApplicationScoped bean (ApplicationScopeBean), and see if any endusers are
logged in; i developed a sessionInfo class that is updated everytime
enduser does a HTTP request; if any endusers are logged in, then I can use
Atmosphere (PrimeFaces Push) to push a message to the endusers, notifying
them to 'please logout, as there are customer requests pending that need to
be retrieved and inserted into the web app's database', and
EmailStatelessBean will 'return' (or exit) and not perform the operation.
This way, everytime @Schedule invokes the bean method, it will continue to
check for an available time to perform the
get-emails-and-insert-into-database operation. Also, I'll set when
EmailStatelessBean begins the operation, then I'll set flag on
@ApplicationScoped bean (insertingCustomerRequestsIntoDatabase = true);
this flag will be checked when endusers attempt to login web app, and
FacesMessage will be displayed, Please login a few minutes later as system
is inserting customer requests into database.

2. Another option would be to send a message to all clients via Atmosphere
(PrimeFaces Push), and the UI will be locked immediately, and user will be
required to wait.

Honestly, I don't like either of these approaches, but I think the endusers
will accept the 1st option (above). If I go with 1st approach, then I may
revert back to single transaction, which will complete the entire operation
much faster than multiple transactions (via multiple EJB DAO classes).

As you can see, I don't have enough experience locking database for such a
batch operation as this. I know the endusers want the customer requests to
show up into the database 'immediately', and they want to know when it
happens. Right now, this is my only option until I redesign the business
website to interface directly with the web app. right now, the web app is
'only' used by 'personnel' (the owners of the business...which is my
family). hahaha :)

They love the app (honestly, they liked the speed and reliability of the
web app when it was on Glassfish), but I'm doing all i can to win them over
with TomEE. :)

Your thoughts, please.

Thanks,
Howard



On Tue, Dec 11, 2012 at 9:23 PM, Howard W. Smith, Jr. 
smithh032...@gmail.com wrote:

 Done. testing now and watching JVM. I need to read that article you
 shared, so i can learn a bit about heap dump. I see the heap dump button
 there, but never used it 'yet'. :)


 On Tue, Dec 11, 2012 at 3:53 PM, Howard W. Smith, Jr. 
 smithh032...@gmail.com wrote:

 I think I'm going to avoid the single transaction (via single @Stateless
 EJB) approach, and use other @Stateless EJBs to make the database table
 updates. This approach works everywhere else throughout the web app.

 I'll just keep the @Stateless EJB for the purpose of being invoked by
 @Schedule, and it will be the 'driver', and call the
 already-defined-and-already-used other @Stateless EJBs that contain
 Persistent context and entity manager to update tables, so this clearly
 will be a multiple transaction commit, which should 'not' lock any tables.
 :)



 On Tue, Dec 11, 2012 at 3:21 PM, Howard W. Smith, Jr. 
 smithh032...@gmail.com wrote:

 Prior to adding this @Stateless EJB (that's invoked via @Schedule every
 10 minutes), my JSF web app has been running 100% okay without any
 deadlocks or anything else like that.

 So, is it okay if I just add optimistic locking, em.lock(entity,
 LockType.Optimistic) before any/all em.persist(...) that I have in the
 @Stateless EJB?

 JSF web app is working okay, because the JSF components call multiple
 @Stateless EJB that usually do any of the following, SELECT, UPDATE,
 DELETE,  on one table at time.



 On Tue, Dec 11, 2012 at 2:35 PM, Howard W. Smith, Jr. 
 smithh032...@gmail.com wrote:

 You're absolutely right, David. Thanks for the response.

 It is a transaction issue and the @Stateless EJB is holding a
 transaction open and writing into 6 to 10 tables in a single database,
 which is also the database accessed by JSF components (by endusers). The
 @Stateless EJB is on a timer ( @Schedule), checks email server for incoming
 requests from customers, and takes that data and inserts it into database,
 so endusers don't have to manually enter the data (that

Re: Why are Tomcat's threads more costly than background threads?