Martin,
I do just create pages and run them with Hudson. This makes batch jobs work
just like anything else. I have one thread per job, there is a request etc.
Hudson also keeps track of what's working, what's not and has a history of
everything. Best of all you don't need to write a single line of code to
get all this. Just drop the war file into your container!
Hudson is very good at running jobs. You can control the number of
concurrent tasks, it has cron like syntax for timing etc. The only real
drawback is it has 1 minute timing and your job may not run precisely when
you schedule it, For batch type processing I don't think either of these
would be a problem. It's also easy to restart if something fails since you
can run jobs manually.
This also makes writing jobs easy since you can just create a page and go
to it during development. The page can output what's going on making it
easy to debug. Hudson will also save the output so you can tell why things
are broken. You'll want to put something in the output so Hudson can figure
out if the job worked or not. I just put an attribute in the outer element
called cmpcod. A zero means the job worked. For testing you can also add a
meta refresh tag to the page and the browser will reload the page.
I separate all my batch jobs into one war file so they are easier to keep
track of. While I do have a permission system for jobs I also don't allow
external access to that war file. Many of my jobs communicate with external
servers and they are impossible to test so when I deploy (via hudson) I
create a backup so I can revert to the old process by just changing the
hudson job. This has proved useful on many occasions.
If you make each page load do one unit of work then it's easy to restart
and @CommitAfter works just fine. I also have jobs that run many small
transactions. In that case I have
@CommitAfter
void run(Transaction transaction) {
…
}
Either way when something breaks the next run just picks up where things
left off.
I run a lot of jobs and this method has proven both flexible and robust. I
came into work the other day and I had a few emails saying a job had
failed. I looked at the Hudson history and a tablespace ran out of disk
space. The dbas fixed it and Hudson had cleaned up the mess for me. I did
not get a call at 2:00am.
Finally if your application becomes so popular you need multiple servers
you can just add them because you don't end up with multiple copies of the
scheduler running. It's easy to either load balance your jobs or just run
them on one machine.
On Fri, Oct 25, 2013 at 6:18 PM, Martin Kersten <martin.kersten...@gmail.com
> wrote:
> Well I inject methods but I use a special working thread or at least you
> can do. There is this PerThreadManager service (lance, barry and thiago
> knows better) and the Job API of Tapestry. There are plenty of ways to get
> this done. Barry is writing pages for every batch job and those pages just
> return OK (or whatever) to indicate if the job was done or they contain
> error messages including the cause (Am I right on this Barry?). He triggers
> the pages from within Hudson and hudson does the collecting errors, notify
> him by emails about any errors etc. Its a very creative utilization of a
> Continous Integration Server if you ask me.
>
> I for myself prefer to use a transactional meaning over CommitAfter -
> therefore controlling the transaction myself. You can use
> session.getTransaction().isActive to know if you are inside an transaction
> and with session.getTransaction().commit/rollback yourself. It makes more
> sense to do so in my opinion.
>
> And yes of cause at the end of your batch when you are finally finished
> with it you might want to ensure the transaction is committed (test for
> active and commit). If you want to use the hibernate session manager of
> tapestry you should set the session to auto-clear (clear on
> commit/rollback) the way I mentioned. Therefore you do not worry about
> clear. It is the way sessions should behave according the JPA
> specification.
>
> So yes I try to avoid commit after. I introduced my own Transactional
> behaviour and pushed logic to the session to ensure I never be able to call
> save or update or whatever outside of an transaction and also allow using
> read only transactions (which is best in the clustering environment we use
> (multi master - slave configurations)).
>
> I replaced the Hibernate???Advisor and the Hibernate ??? Session Source in
> my project to be able to reset the test-database after each test to avoid
> restarting tapestry (and an embedded jetty) all the time) during the test
> (tapestry needs to have 400ms each start, and about 4secs on first start,
> if you use jetty you are likely to get even more startup time). Since I use
> a in memory database during testing and developing it is quite fast
> reducing the startup time < 1ms to <50ms per test. I always start with the
> same test scenario but having multiple users in multiple state to test all
> white box test cases.
>
> So in the end you are completely right. Just go ahead. But remember you do
> not need flush before commit and rollback (in case of an rollback a flush
> is also wasted afford since the flushed information are scraped on rollback
> anyways) and you need to clean after the commit/rollback not before since
> you will only clean some information that are already gone since you
> flushed those information to the database. What you try to do is reduce the
> number of managed entities and hibernate needs to have those entities being
> committed.
>
> Well the last part is not exactly true. You can detach those entities by
> hand using evict. But this is troublesome. So if you need your batch job to
> be done inside one big transaction you can still speed everything up by
> detaching (evict) all those entities you are sure (!) you dont use again
> during the very same session (well they get auto attached if you use them
> but this might lead to two entity object representing the same database row
> (entity) which will cause Hibernate to fail with exception leaving the
> session in an unpredictable state you should consider to be broken.
>
> Well that was quite some amount of text. Hopefully you get some additional
> use out of it.
>
>
> Cheers,
>
> Martin (Kersten)
>
>
>
> 2013/10/25 George Christman <gchrist...@cardaddy.com>
>
> > Thanks Martin, if I understood you correctly, you are still Injecting the
> > Session. Will this cause any issues where I'm still using @CommitAfter?
> > Also in the batch job, do you typically put an additional commit outside
> > the loop to pickup up the remaining values when they don't reach the
> > count, say % 250 == 0 mark or do you just put an additional condition in
> > there to commit when you hit the last record?
> >
> > so something like this,
> >
> > if(count % 250 == 0 || count == size)
> >
> >
> > On Fri, Oct 25, 2013 at 10:03 AM, Martin Kersten <
> > martin.kersten...@gmail.com> wrote:
> >
> > > well you are decorating the session not the session source of cause...
> .
> > >
> > >
> > > 2013/10/25 Martin Kersten <martin.kersten...@gmail.com>
> > >
> > > > >> Perhaps you want a @MaybeCommitAfter ;)
> > > >
> > > > What about simply having @WriteTransaction (und @ReadTransaction,
> since
> > > > lacking support for read transactions is the biggest performance
> issue
> > > when
> > > > dealing with databases). If you are inside a method annotated with
> > > > @WriteTransaction it is ignored. If nesting write within a read
> > > transaction
> > > > fails (there is something wrong with your service methods /
> thinking).
> > > And
> > > > also wrapping the session source (decorate) and checking on any
> > > > save+/update/delete + query if a transaction was set makes it
> crystal
> > > > clear if you messed up the transactional thingy. Also using
> > @CommitAfter
> > > > inside a @XxxxTransaction fails, since you can not commit inside it
> > > without
> > > > breaking the meaning.
> > > >
> > > > This would make it possible to have a good implementation which is
> self
> > > > explaining, would only cost two hours to build and test (at least I
> > > havent
> > > > spent an hour in total I think). And it wont break the backward
> > > > compatibility with @CommitAfter.
> > > >
> > > >
> > > > 2013/10/25 Martin Kersten <martin.kersten...@gmail.com>
> > > >
> > > >> And George, read this one:
> > > >>
> > >
> >
> http://stackoverflow.com/questions/10143880/hibernate-queries-much-slower-with-flushmode-auto-until-clear-is-called
> > > .
> > > >> It explains why you experience the slow downs and what is it causing
> > it.
> > > >>
> > > >>
> > > >> 2013/10/25 Martin Kersten <martin.kersten...@gmail.com>
> > > >>
> > > >>> You might also want to use such kind of a utility that allows you
> to
> > > run
> > > >>> in transaction:
> > > >>>
> > > >>> public interface InTransaction<T> {
> > > >>> public T run(Session session);
> > > >>> }
> > > >>>
> > > >>> public static <T> T DatabaseUtil.process(InTransaction<T>
> > > >>> transactionalRunnable) {
> > > >>> try {
> > > >>> Session session = currentSession(); //or newSession()
> > > >>> if(!session.getTransaction().isActive())
> > > >>> session.beginTransaction();
> > > >>> T result = transactionalRunnable.run(session);
> > > >>> if(!session.getTransaction.isActive()) {
> > > >>> session.getTransaction().commit();
> > > >>> session.clear();
> > > >>> }
> > > >>> }
> > > >>> catch(RuntimeException e) {
> > > >>> if(session.getTransaction().isActive()) {
> > > >>> session.getTransaction().rollback();
> > > >>> session.clear(),
> > > >>> }
> > > >>> }
> > > >>> finally {
> > > >>> //session.close(); //if you created a new one
> > > >>> }
> > > >>> }
> > > >>>
> > > >>>
> > > >>> This way you can control the whole transactional process while only
> > > >>> doing:
> > > >>>
> > > >>> List<Product> product = DatabaseUtil.process(new
> > > >>> InTransaction<List<Product>>() {
> > > >>> List<Product> run(Session session) {
> > > >>> //getUser etc are all static methods fetching from a
> > > >>> ThreadLocal RequestContext object so you dont need
> > > >>> //to inject / pass along parts of your model in every
> > > >>> component/page
> > > >>> UserInfo user = getUser();
> > > >>> userStats.countUserRequest(user); //changes the
> database;
> > > >>> return queryProductsForUser(user, session); //returning
> a
> > > list
> > > >>> }
> > > >>> }
> > > >>>
> > > >>> This example is just made up to demonstrate that you can return a
> > > >>> List<Product> or Integer or if you have nothing to return
> > > >>> a Void instance.
> > > >>>
> > > >>> The DatabaseUtil class I use for everything that happends outside
> of
> > a
> > > >>> request therefore when I have to control the sessions
> > > >>> being opened, closed and manage the transaction. This way you even
> > not
> > > >>> run into any difficult with nesting (you can not
> > > >>> leave the transaction (since it is an enclosed try catch block),
> > > >>>
> > > >>> I have set my session to automatically clear the persistence
> context
> > on
> > > >>> commit.
> > > >>> You can do this by ((SessionImpl)session).setAutoClear(true).
> > > >>>
> > > >>> And of cause checking if a transaction is active can be extracted
> to
> > a
> > > >>> private method to value the DRY principle.
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>> 2013/10/25 Martin Kersten <martin.kersten...@gmail.com>
> > > >>>
> > > >>>> > session.flush();
> > > >>>> > session.clear();
> > > >>>> > hibernateSessionManager.commit();
> > > >>>>
> > > >>>> This is wrong.
> > > >>>> First on commit you will do the flush automatically (flush means
> all
> > > >>>> changes are written to the database (performing outstanding
> updates,
> > > >>>> inserts, deletes))
> > > >>>> Clear clears the persistence context of all entities not taking
> part
> > > on
> > > >>>> any outstanding flush event (as far as I remember) therefore
> > > Hibernate does
> > > >>>> a deep
> > > >>>> inspection of the active entities and removes all entities that
> were
> > > >>>> not encountered during that process.
> > > >>>> Commit commits the entities.
> > > >>>>
> > > >>>> So the correct usage is:
> > > >>>> session.getTransaction().commit();
> > > >>>> session.clear();
> > > >>>> session.beginTransaction();
> > > >>>>
> > > >>>> (without the clear its what HibernateSessionManager is doing with
> > the
> > > >>>> session bound to the current thread).
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> 2013/10/25 Martin Kersten <martin.kersten...@gmail.com>
> > > >>>>
> > > >>>>> Use:
> > > >>>>>
> > > >>>>> @Inject
> > > >>>>> Session session; //current session bound to the current thread
> > > >>>>>
> > > >>>>>
> > > >>>>> or
> > > >>>>>
> > > >>>>> @Inject
> > > >>>>> HibernateSessionSource source; + source.create() for a really new
> > > >>>>> session (CommitAfter would not work with newly created one);
> > > >>>>>
> > > >>>>> Using the Manager does give you only the session associated with
> > the
> > > >>>>> current thread as would @Inject Session session; would do.
> > > >>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>> 2013/10/25 George Christman <gchrist...@cardaddy.com>
> > > >>>>>
> > > >>>>>> So I guess I'm still a little confused as to what is the best to
> > do
> > > >>>>>> it.
> > > >>>>>> @CommitAfter seems to work fine for individual transactions but
> > does
> > > >>>>>> not
> > > >>>>>> work well with batch jobs do to it holding on to the object in
> > > memory.
> > > >>>>>> Anyhow, I could not figure out how to get Martins
> > > >>>>>> session.getTransaction()
> > > >>>>>> to work, however I did end up getting the following code to
> work.
> > > >>>>>> Could
> > > >>>>>> someone tell me if I'm doing this correctly? Also should I be
> > > closing
> > > >>>>>> and
> > > >>>>>> rolling back the transaction?
> > > >>>>>>
> > > >>>>>> //Mock scenario
> > > >>>>>>
> > > >>>>>> @Inject
> > > >>>>>> private HibernateSessionManager hibernateSessionManager;
> > > >>>>>>
> > > >>>>>> public void onActionFromTest() {
> > > >>>>>> Session session = hibernateSessionManager.getSession();
> > > >>>>>>
> > > >>>>>> for (int i = 0; i < 100000; i++) {
> > > >>>>>> employee = new Employee("George " + i);
> > > >>>>>>
> > > >>>>>> session.save(employee);
> > > >>>>>>
> > > >>>>>> if (i % 250 == 0) {
> > > >>>>>> session.flush();
> > > >>>>>> session.clear();
> > > >>>>>> hibernateSessionManager.commit();
> > > >>>>>> }
> > > >>>>>> }
> > > >>>>>>
> > > >>>>>> session.flush();
> > > >>>>>> session.clear();
> > > >>>>>> hibernateSessionManager.commit();
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> On Fri, Oct 25, 2013 at 9:07 AM, Thiago H. de Paula Figueiredo <
> > > >>>>>> thiag...@gmail.com> wrote:
> > > >>>>>>
> > > >>>>>> > On Fri, Oct 25, 2013 at 10:53 AM, Barry Books <
> trs...@gmail.com
> > >
> > > >>>>>> wrote:
> > > >>>>>> >
> > > >>>>>> > While it's true you can run into problems by nesting
> > @CommitAfter
> > > >>>>>> the same
> > > >>>>>> > > can be said about nesting any commits. The Tapestry database
> > > >>>>>> model is
> > > >>>>>> > > simple. There is one connection per request and when you
> call
> > > >>>>>> commit it
> > > >>>>>> > > does a commit.
> > > >>>>>> > >
> > > >>>>>> >
> > > >>>>>> > <pedantic>
> > > >>>>>> > Tapestry itself doesn't have any database model. It's a web
> > > >>>>>> framework and
> > > >>>>>> > nothing else. You can use it with any database, including
> none.
> > > >>>>>> > Tapestry-Hibernate is a package that provides *simple* support
> > for
> > > >>>>>> > Hibernate and should be used in *simple* scenarios. If you
> need
> > > >>>>>> something
> > > >>>>>> > that's not simple, like any transaction handling not supported
> > by
> > > >>>>>> > @CommitAfter, use Tapestry and some transaction handler (EJB,
> > > >>>>>> Spring-TX,
> > > >>>>>> > etc) but not Tapestry-Hibernate.
> > > >>>>>> > </pedantic>
> > > >>>>>> >
> > > >>>>>> >
> > > >>>>>> >
> > > >>>>>> > >
> > > >>>>>> > >
> > > >>>>>> > > On Fri, Oct 25, 2013 at 7:31 AM, Lance Java <
> > > >>>>>> lance.j...@googlemail.com
> > > >>>>>> > > >wrote:
> > > >>>>>> > >
> > > >>>>>> > > > I'm assuming a fork is broken too because it's no good for
> > > >>>>>> eating soup?
> > > >>>>>> > > > Sounds like you need a spoon, it's easy to write your own
> > > >>>>>> annotation...
> > > >>>>>> > > > Perhaps you want a @MaybeCommitAfter ;)
> > > >>>>>> > > >
> > > >>>>>> > >
> > > >>>>>> >
> > > >>>>>> >
> > > >>>>>> >
> > > >>>>>> > --
> > > >>>>>> > Thiago
> > > >>>>>> >
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> --
> > > >>>>>> George Christman
> > > >>>>>> www.CarDaddy.com
> > > >>>>>> P.O. Box 735
> > > >>>>>> Johnstown, New York
> > > >>>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > > >
> > >
> >
> >
> >
> > --
> > George Christman
> > www.CarDaddy.com
> > P.O. Box 735
> > Johnstown, New York
> >
>
