Transaction & Batch-processing in Grails

February 1, 2009

Transaction handling is one of the more complex areas of web development. Anytime a user takes any action in the interface which demands a couple of database actions in the backend, then usually you end up having do it as transaction. For a user, everything is either a success or failure. Partial success may be either harmful to the system or doesn’t mean anything to the user. Grails, since it is built on top of Spring and Hibernate, uses their underlying mechanism to deal with transactions. While it may seem confusing in the beginning, Grails actually makes it even more easier.

Lets see it in code.

Suppose we are designing an Offers System, like the one that your bank sends notifying you of some offers and promotions on services. A simplified object structure may be like the following:

class Offer {
     String title
     String content
     Date expirationDate
     Date createdDate = new Date()

     static hasMany = [recipients: Recipient]
}
class Recipient {
     String email

     static belongsTo = [offer: Offer]
}

The relationship is fairly simple. Each Offer can have many Recipients, each Recipient belongs to an Offer.

belongsTo just means the offer Recipient will be deleted whenever the Offer is deleted, which makes sense coz we don’t want to keep the junk if the Offer itself is deleted.

hasMany lets you access to the Recipients from the Offer object. In other words, if you have the Offer object, you can do offer.recipients and get the list of all the recipients for the Offer. Cool.

Now, here is how it will work. We want to add an Offer and some Recipients. Everything should either succeed or fail. Even if only one Recipient out of thousands fail, everything should fail.

There are two ways to do transactions in Grails:

1. Method-level Transaction: By default, each Service class in grails has a transactional property set to true. So if any public method in a Service class throws a RuntimeException or any Error, the transaction will be rolled back. If you do not need this default transaction property, you can explicitly set transactional to false.

In the code below, the OfferService throws a Runtime Exception anytime it isn’t able to add the Fffer or the Recipient, and the transaction will be rolled back.
You wouldn’t get the same transactional behavior if the method throws any Checked Exception or if the transactional property is set to false.

class OfferService {

     boolean transactional = true

     def save(offer, recipients) {
         if(!offer.validate()) {
             throw new RuntimeException("Invalid offer.")
         } else {
             def result = offer.save()
             recipients.each {
            	if(!it.validate()) {
            		throw new RuntimeException("Invalid recipient")
                } else {
                  	offer.addToRecipients(it)
                }
             }
         }
     }
}

You can then catch the exception in the Controller and render a more user-friendly error message.

class OfferController {
     def save = {
    	 def offer = getOffer(params)
    	 def recipients = getRecipientList(params)
   	 try {
   		offerService.save(offer, recipients)
		flash.message = "Successfully added the offer."
   	 } catch (Exception e) {
   		log.error "Exception: " + e
 		flash.message = "An error occured, and the offer was not saved."
   	 }
     }

}

So all you need to do is make sure your Service is transactional, and that it throws a RuntimeException and not a Java CheckedException.

2. Block-level Transaction: The first approach treats everything inside the method as a single transaction. But sometimes your need more control than that. You might want to perform some actions within a transaction and do other operations regardless, or within another block of transaction within a single method.

Each domain class in grails has a static withTransaction method that takes a closure as an argument. Lets see the same thing in code using withTransaction:

class OfferService {

    boolean transactional = false

    def save(offer, recipients) {

    	Offer.withTransaction { status ->
             if(!offer.validate()) {
                  status.setRollbackOnly()
             } else {
         	  def result = offer.save()
                  recipients.each {
           	       if(!it.validate()) {
          	          status.setRollbackOnly()
                       } else {
                  	  offer.addToRecipients(it)
                       }
                  }
              }
         }

    }
}

The status object that we are calling the setRollbackOnly on is actually an instance of org.springframework.transaction.TransactionStatus class. The funny thing is it doesn’t really matter which Domain object you call the withTransaction method on. It just to mark the start and end of a transaction.

As you can guess, with this approach, you don’t need to be in a Service class to use a transaction. You can use it from inside a controller too.

In both of the process above, grails isn’t actually doing a database rollback. It is just saving the changes into the Hibernate session until the transaction is successful, at which point it commits the change to the database. So it is holding onto a lot of objects in session cache.

I have been fighting a situation where I have to batch insert a thousand of objects within a transaction. My common sense tells me I need something like nested transaction, where I could divide a large parent transaction into smaller chunks of transactions, and if one of them fails, it would trigger the parent to fail. But looking at the way Hibernate session works, I was worried that I might have to stick to plain JDBC to do it. But not anymore, coz I found this awesome doc on Hibernate Batch-processing.

Batch Processing: Each action in a Grails Controller is executed within a Hibernate Session. The session is started right before the action starts and is closed once it returns. That explains the reason for the memory issues because all the objects are being cached to the Hibernate session. Hibernate saves it to the database when the session is flushed, thus freeing the memory. The same code as above with the session cleared more frequently:

class OfferService {

    boolean transactional = false
    def sessionFactory

    def save(offer, recipients) {

    	Offer.withTransaction { status ->;
             if(!offer.validate()) {
                  status.setRollbackOnly()
             } else {
         	  def result = offer.save()
                  int i = 0
                  recipients.each {
           	       if(!it.validate()) {
          	          status.setRollbackOnly()
                       } else {
                          i++
                  	  offer.addToRecipients(it)
                          if(i % 100 == 0) {
                               sessionFactory.getCurrentSession().clear();
                          }
                       }
                  }
              }
         }

    }
}

Note: You can get the sessionFactory in a Controller similarly.
This way the Hibernate session, which is the first-level cache, is being cleared every once in a while.
This worked like a charm for me, solving all the memory issues I was having, and also speeding things up significantly.

23 Comments on Transaction & Batch-processing in Grails

Respond | Trackback

  1. Chris says:

    Finished this one! =)

  2. Chris says:

    This helped with a problem at work today. Thanks for this Suda!

  3. Hi,

    Just a minor point, but shouldn’t the transactional property actually be declared as static?

    I think that Grails may currently support static and non-static, but I believe the officially documented approach specifies static only.


    Stephen Souness

  4. sacharya says:

    Stephen, you’re right. I should have explicitly declared the ‘transactional’ property to be ’static’.

  5. [...] This post was Twitted by groovytweets [...]

  6. Achim says:

    Good examples!

    A question for the Method-level Transaction:

    If the offer.save() in line 9 is unsuccessful although the validate succeeded, will grails throw a RunTimeException by itself ?

    Or more general: Is the explicit validate done just for better messages or is it really necessary because the save does not throw an Exception?

  7. sacharya says:

    If the offer.validate() is successful, then offer.save() will be successful.
    The save() method does call the validate() method by itself,but unless you use flush true, it will not do the persistence instantly. And hence we are explicitly doing validate, and then letting it do the save whenever it wants, and knowing that the save will be successful.

  8. Paul says:

    When I run this I get the following error:

    ERROR errors.GrailsExceptionResolver – No signature of method: OfferController.getOffer() is applicable for argument types: (org.codehaus.groovy.grails.web.servlet.mvc.GrailsParameterMap) values: [[action:save, controller:offer]]
    groovy.lang.MissingMethodException: No signature of method: OfferController.getOffer() is applicable for argument types: (org.codehaus.groovy.grails.web.servlet.mvc.GrailsParameterMap) values: [[action:save, controller:offer]]
    at OfferController.invokeMethod(OfferController.groovy)

  9. admin says:

    You need to implement the getOffer(params) to build a Offer object from the request params.

    Sorry, the code isn’t meant to just take it and run it as it is. I was only talking about concepts and might have avoided some util method due to lack of space.

  10. scaler says:

    I would like to know who is the fucking idiot that implements grails persistence actions… No way to understand why a domain class is not valid when you call the method save(), no exception thrown !

    When I see how easy and fast it is to build J2E applications with maven, jsf, spring, hibernate and so on… I really dont see why grails should be a revolution.
    The first wish of a developer is to have pertinent and understanding exceptions thrown when he makes some mistakes…

  11. admin says:

    I know it can be confusing in the beginning, but it as easy as calling the the validate() method, if you need to validate that your domain will persist:

    def user =  new User()
    if(user.validate()) {
        user.save()
    } else {
        user.errors.allErrors.each {
            println it
        }
    }
    
  12. [...] Transaction & Batch-processing in Grails – Sudarshan Acharya [...]

  13. erandi says:

    Hi,

    Ur post was really helped me while coding. Now I am having a problem and my problem is regarding the Block-level Transaction.

    In my Offer domain class i used beforeUpdate event. Since in block level transaction keeps the object in session until the transaction is committed, the beforeUpdate event called as a loop and finally result in stachOverflow exception.

    How to avoid this situation?. Help me.

    Erandi

    • Mohan says:

      I have submitted a to the Grails JIRA that will iculnde the chainArgs in the ControllerUnitTest case, so maybe soon it will be available in the same way as renderArgs and redirectArgs. I’m glad you posted this, because I never knew it existed before. I was doing workarounds doing metaClass manipulation (yuck!).

  14. Enrique Medina Montenegro says:

    I wonder how GC gets impacted by massive creating of objects in Eden segment. Has anyone run into long GC process that blocks application for some seconds?

    If so, can you please share GC configuration? Parallel? Old parallel? New concurrent mark sweep? Size for Eden and Mature segments?

    Thanks.

  15. anand says:

    graet… it worked wonder for me/… thanx 4 such a great artical… i m lovin it…

  16. Matt Wersch says:

    We use a similar batch style save, but have run into a slight snag. Clearing the current session may cause side-effects if you do work in your controller AFTER calling the batch save and that work relies on lazy loading.

    For example:

    def action = {
    List referenceObjects = Service.findRefObjects()
    List objects = []
    populateObjects(objects, referenceObjects)
    Service.batchSave(objects) // internally clears the current session with .clear()
    referenceObjects.each { ref ->
    ref.children.each {child -> doSomeWork(child)} // LazyInitializationException unable to initialize Session
    }

    }

    Any one know a way of doing the batch save that either doesn’t need to clear or can utilize a session that operates only within the confines of the Service.batchSave() method leaving the request’s session intact?

    Cheers,
    -Matt

  17. gurp says:

    I am using grails 1.3.7 and facing an issue with the transactions. Service layer is not getting rolled back if any exception is thrown. I should be by default set to true and no need to give static def transactional =true at service layer but still if i give transactional = true it dosent work

    Only thing which works for me is Domain.withTransaction{}

    Any Idea why by deafult transaction rollback is not working.

    thanks for help in advance

  18. Matt Wersch says:

    In response to my previous post, performing a session.evict(obj) within the Service.batchSave() method after each individual save resolved my issues.

  19. Carlos says:

    Another way to do batch processing, it would be this:

    recipients.eachWithIndex {domain, index ->

    // do work

    index % 100 == 0 ? sessionFactory.getCurrentSession (). flush (). clear () : offer.addToRecipients(domain);
    }

  20. ew says:

    Hi,

    I tried everything I could to process a large process in one transaction but it always ran out of memory, i tried
    a) flushing and clearing session every so often via session.currentSession.clear()/flush()
    b) clearing DomainClassGrailsPlugin.PROPERTY_INSTANCE_MAP.get().clear()
    c) using save(insert:true, validate:false)
    d) also tried switching hibernate session cacheMode to GET and IGNORE
    but still no use. Only way I could do it was by setting static transactional = false and breaking the process into smaller transactions using Domain.withTransaction{} but this is not so good as if anything goes wrong then some of my database table is now only partially processed which is not good

    any suggestions ?

    • Pascal DeMilly says:

      I have the same problem. I am running a grails script that use a service to import few hundred thousand rows into a mongodb. I use grails to do the validation of my domain but right now I am running into a wall. It seems that grails is doing some caching for each find reqiest and finally running of GC Heap.

      I tried to clear the session but my sessionFactory is null. (not injected). Everything else is. So that is a mystery.

      Anyway wanted to know if you found a solution you could share

      Thanks

Respond

Comments

Comments