Category: Grails

Facebook Connect with JSecurity on Grails

April 18, 2010

Say you have a Grails application using JSecurity (now called Apache Shiro) for authentication. How do you provide an alternate mechanism to authenticate users using Facebook Connect?

Good thing! Grails has the Facebook Connect plugin so you can authenticate users without registering them to your system. But what you will want is to integrate the Facebook Connect Plugin with the JSecurity Plugin. That way, even though a user is actually authenticated using Facebook Connect, your JSecurity knows about the user and can assign roles and permissions to the user.

I assume you have both the Apache Shiro Plugin installed and authentication working properly. Now go ahead and install the and Facebook Connect Plugin by following the documentation.

In addition to the regular JSecurity Login form, you will have the Facebook Login button as follows:

login.gsp

<script type="text/javascript">
    <!--
        function delayer() {
            window.location = "${createLink(controller:'auth', action: 'signin')}"
        }
    //-->
</script>
<div id="login_section">
    <h4>Login using Facebook Connect</h4> 
    <div class="login_form">
        <fb:login-button autologoutlink="false" onlogin="setTimeout('delayer()',100)">
        </fb:login-button>
        <br/>
	<fb:name uid="loggedinuser" useyou="false"></fb:name>
	<fb:profile-pic uid="loggedinuser" size="normal" />			
	<g:facebookConnectJavascript  />
    </div>
</div> <!--Login Form Ends-->  

The above code snippet will display the Facebook login button. When the user clicks on the button, Facebook Connect’s login dialog is displayed. Once he user authenticates himself to Facebook, the delayer() javascript method is called which then redirects to the auth/signin grails action.

Now lets insert the following snippet of code into the signIn action.

AuthController.groovy

def facebookConnectService
def signin = {
    if(!params?.username && !params?.password) {
        if(facebookConnectService.isLoggedIn(request)) {
            try {
                facebookConnectService.getFacebookClient().users_getLoggedInUser()
                params.rememberMe = false
                params.username = "facebook"
                params.password = "randompassword"
            } catch (Exception e) {
                flash.error ="We are sorry. Please try again in a while."
                redirect(controller: 'home', action: 'index') 
                return
            }
        }
    }
    // Rest of the Code continues
}

where, facebookConnectService is the service class that is provided by the the Grails Facebook Connect plugin. We are just checking if the user is logged in to Facebook, and then setting values for username and password as per JSecurity’s needs.

The actual authentication of the user is done in grails-app/realms/ShiroDbRealm.groovy.

Now lets add a few lines to our ShiroDbRealm.groovy to handle our Facebook user:

ShiroDbRealm.groovy

def facebookConnectService

def authenticate(authToken) {
        log.info "Attempting to authenticate ${authToken.username} in DB realm..."
        def username = authToken.username

        // Null username is invalid
        if (username == null) {
            throw new AccountException('Null usernames are not allowed by this realm.')
        }

        def user = null
        def account = null
        if (username == "facebook") {
            def facebookUser = null
            try {
                facebookUser = getFacebookUser()
            } catch (Exception e) {
                e.printStackTrace()
            }
       	    try {
       	        user = User.findByUsername(facebookUser.username)
       	    } catch (Exception e) {
       	        e.printStackTrace()
       	    }

           if (!user) {
                facebookUser.passwordHash = new Sha1Hash("randompassword").toHex()
                if (facebookUser.validate()) {
	                user = facebookUser.save(flush: true)
                } else {
                        facebookUser.errors.allErrors.each {
                            log.error it
                        }
                }

                account = new SimpleAccount(user.username, user.passwordHash, "ShiroDbRealm")
                return account;

            } else {
                account = new SimpleAccount(user.username, user.passwordHash, "ShiroDbRealm")
                return account;
            }
      }
      // Rest of the Code continues
}

def getFacebookUser () {
        String userId = facebookConnectService.getFacebookClient().users_getLoggedInUser()
        java.lang.Iterable<java.lang.Long> userIds = new ArrayList<java.lang.Long>()
        userIds.add(Long.parseLong(userId))
		
        Set<ProfileField> c = new HashSet<ProfileField>()
        c.add(ProfileField.FIRST_NAME)
        c.add(ProfileField.LAST_NAME)
        c.add(ProfileField.NAME)

        def myresults = facebookConnectService.getFacebookClient().users_getInfo(userIds, c)
        def useObj = myresults.getJSONObject(0)
        User user = new User()
    	user.username = userId
    	user.firstName = useObj.getString("first_name")
    	user.lastName = useObj.getString("last_name")
    	user.name = useObj.getString("name")  	
    	user.displayName = user.name
    	
        Date date = new java.util.Date()
    	user.dateCreated = date
    	user.lastUpdated = date
    	user.lastVisited = date
        return user
}

Basically, if the user uses Facebook Connect, we are setting his username to ‘”facebook” and password to “randompassword” from the signin action. In the authenticate method, if the username is “facebook”, we are getting all the info of the user from facebook. If the user with that facebook id is already in our database, it is an existing user and is authenticated. If the user doesnt exist in our database, the new user is added to the databse with a username and password. The username and password is only for integrating Facebook Connect with JSecurity, and the user has no idea about it. Now even though the user is actually authenticated by Facebook, it still is a JSecurity user in our database, and will be treated just like any other user.

Please note that I have added a few properties like firstName, lastName, displayName etc to the User.groovy which is created by JSecurity. Feel free to add new properties to the User object if you want to capture more user info from Facebook for that user.

Now you are good to go. Deploy and test the app.

Soon, you will notice that this will work in development, but there is a bug in the Current revision of Grails Facebook Connect Plugin in production environment, due to which it it cannot find the FacebookConnectConfig.groovy in prod. Go and download the source for Facebook Connect Grails plugin and modify the afterPropertiesSet method under services/FacebookConnectService.groovy in the grails-facebook-connect-0.1 project.

FacebookConnectService.groovy

void afterPropertiesSet() {
    // check if there is a compiled class for the Facebook connect config groovy script
    // this will be the case when an application is bundled in a WAR
    def config = null
    try {
        config = Class.forName("FacebookConnectConfig").newInstance()
    } catch(Exception  e) {
        e.printStackTrace()
    }
    if (config != null) {
        // compiled config class found, we must be running as a WAR 
        facebookConnectConfig = new ConfigSlurper().parse(config.getClass())
    } else {
        // no compiled class exists for the config, we must be running the Grails built-in web server
        GroovyClassLoader loader = new GroovyClassLoader(getClass().getClassLoader())
        Class clazz = loader.parseClass(new File("grails-app/conf/FacebookConnectConfig.groovy"))
        facebookConnectConfig = new ConfigSlurper().parse(clazz)
    }
}

Now use the patched plugin in your app and that should work!

Shoud you use Facebook Connect?:
Based on my few months of experience, users are pretty hesitant to login to a site using Facebook Connect. In my case, less than 10% of the users used Facebook Connect on my site, and it certainly wasn’t worth the effort I put into making it work and mantaining it. Its kind of clunky at times too. I don’t think I am never gonna use it again just for the sake of authentication. So, make sure it really adds some value to your application and users, before you decide to integrate it in your app coz you thought its cool.

Grails, DBCP & Stale Connections

July 13, 2009

Does your application break after every few hours of inactivity even though you have enough database connections in the pool? It is a common problem with database connection pooling (and idle sockets connections in general).

I have been running a few Grails app with PostgreSQL database using Apache Commons DBCP connection pooling. Most of these apps are pretty busy, and are working quite well so far. But I have one critical app that doesn’t get used as much. Recently after a bug report, I was watching through the logs and I realized that this app was complaining about Socket Connection Exception after every hour of idle time. Try it again, and it would work. So why was it rejecting the DB connection the first time? Creepy!

I checked out the other apps I have, and all of them were suffering from the same problem – depending on how idle they were. I couldn’t ignore any longer.

I started off with the basics.

Basic configuration:

I have my datasource defined in the DataSource.groovy file under grails-app/conf. I have enabled connection pooling and I am using the appropriate PostgreSQL-JDBC driver. Grails comes with Apache DBCP connection pooling by default. So it just works.

environments {
     production {
          dataSource {
               pooled = true
               driverClassName = "org.postgresql.Driver"
               dbCreate = "update"
               url = "jdbc:postgresql://sacharya.com:5432/sacharya_prod_db"
               username = "myuser"
               password = "mypassword"
               logSql=true
          }
     }
}

This is what my production configuration looks like – running whatever the default behavior that Grails comes with. I printed the datasource object to see what DBCP configurations my app is using by default.

dataSource.properties.each { println it }
minIdle=0
initialSize=0
maxIdle=8
numIdle=0
maxActive=8
numActive=0

As the result shows, it is using the default values as defined in Apache DBCP

Problem:

Using netstat, I started watching if the application has any connections to the database open:

$ netstat -na | grep 5432
tcp4       0      0  24.25.26.27.50095     98.129.169.246.5432     ESTABLISHED

So DBCP says it has no connections open (active or idle) to the database yet in the pool. But netstat shows there is a TCP connection established to the database. Where does this connection come from – loading & initializing the Driver?

Now lets use the application for a while. Once you start asking the pool for connections, existing connections idle in the pool are at your service, or if there no connections in the pool, new connections are created and given to you, which get returned back to the pool after you complete the query. So at any given time, there will be minIdle to maxIdle ( 0 to 8 in our case) connections in the pool.

$ netstat -na | grep 5432
tcp4       0      0  24.25.26.27.51308     98.129.169.246.5432     ESTABLISHED
tcp4       0      0  24.25.26.27.50095     98.129.169.246.5432     ESTABLISHED

Now with some connections in the pool, I left the app idle for an hour and then I tried to access the app (netstat still shows 2 TCP connection). The first query got the following exception:

org.springframework.dao.DataAccessResourceFailureException: could not execute query;
nested exception is org.hibernate.exception.JDBCConnectionException: could not execute query

Caused by: org.hibernate.exception.JDBCConnectionException: could not execute query
at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:97)

Caused by: org.postgresql.util.PSQLException: An I/O error occured while sending to the backend.
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:217)

Caused by: org.hibernate.exception.JDBCConnectionException: could not execute query
at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:97)

Caused by: org.postgresql.util.PSQLException: An I/O error occured while sending to the backend.
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:217)

Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)

The subsequent query succeeds. There was one connection (ignore the other TCP connection used during Driver initialization) in the pool, but the database responded by saying that the connection was not valid.

Reasoning:

At this point it was clear that the TCP connection that was sitting idle was already broken, but our app still assumed it to be open. By idle connections, I mean connections in the pool that aren’t in active use at the moment by the application. After some search, I came to the conclusion that the network firewall between my app and the database is dropping the idle/stale connections after 1 hour. It seemed to be a common problem that many people have faced.

By default, DBCP holds the pooled connections open for infinite time. But a database connection is essentially a socket connection, and it doesn’t come for free. The host OS, database host, and firewall have to allocate a certain amount of memory and other resources for each socket connection. It makes sense to those devices not to hold onto idle connections for ever. So the idea is to make sure that you don’t have stale connections in your pool that would otherwise be silently dropped by OS or firewall. The system has no way of knowing if the connection is broken unless is sends a packet and waits for an acknowledgement. So even when the connection is timed out or closed by one side, the other side may still think the connection is open.

While there may not be a firewall between your server and database, even the OS has a timeout on TCP connections. You could probably increase the TCP keepalive of the OS itself, but that will affect the whole system, and yet you are only postponing the problem.

Solution:
Now lets try to modify some of the DBCP settings for the dataSource.

1. Validating Connections: DBCP allows you do define a validation query and do a sanity check of the connection before you actually use it in your application.
By default,

validationQuery=null
testOnBorrow=false
testOnReturn=false
testWhileIdle=false

The validation query must return at least one row, and using the query you can have DBCP test the connection for you while its idle, before you borrow and before you return it.
So lets change it to:

validationQuery="SELECT 1"
testOnBorrow=true
testOnReturn=true
testWhileIdle=true

If any of the connection object fails to validate, it will be dropped from the pool. There might be some performance implications of running these three SQLs (which I am not worried at the momet), and hence you might just want to try testOnBorrow.

2. Evicting Idle Connections: DBCP can run an idle object evictor at a regular interval and evict any connections that are older than some threshold. By default this behavior is turned off since timeBetweenEvictionRunsMillis is set to -1.

timeBetweenEvictionRunsMillis=-1
numTestsPerEvictionRun=3
minEvictableIdleTimeMillis=1000 * 60 * 30

Now lets run the evictor every 30 minutes and evict any connections older than 30 minutes.

timeBetweenEvictionRunsMillis=1000 * 60 * 30
numTestsPerEvictionRun=3
minEvictableIdleTimeMillis=1000 * 60 * 30

It turns out that you cannot change the DBCP settings from the DataSource.groovy file. The datasource object injected in the DataSource.groovy file is an instance of the javax.sql.DataSource. I can however do so by overriding the default DataSource from the BootStrap.groovy, which sets up the settings during start up.

import org.codehaus.groovy.grails.commons.ApplicationAttributes
class BootStrap {

     def init = { servletContext ->

          def ctx=servletContext.getAttribute(
                       ApplicationAttributes.APPLICATION_CONTEXT)
          def dataSource = ctx.dataSource

          dataSource.setMinEvictableIdleTimeMillis(1000 * 60 * 30)
          dataSource.setTimeBetweenEvictionRunsMillis(1000 * 60 * 30)
          dataSource.setNumTestsPerEvictionRun(3)

          dataSource.setTestOnBorrow(true)
          dataSource.setTestWhileIdle(false)
          dataSource.setTestOnReturn(false)
          dataSource.setValidationQuery("SELECT 1")

          dataSource.properties.each { println it }
     }
}

You can do the same from grails-app/conf/spring/Resource.groovy:

import org.apache.commons.dbcp.BasicDataSource
beans = {
     dataSource(BasicDataSource) {       
          minEvictableIdleTimeMillis=1800000
          timeBetweenEvictionRunsMillis=1800000
          numTestsPerEvictionRun=3

          testOnBorrow=true
          testWhileIdle=true
          testOnReturn=true
          validationQuery="SELECT 1"
     }
}

This seems to have solved the problem for me. Since my firewall was dropping the socket connections at 60 minutes, all I did was proactively run the idle object evictor every half 30 minutes, flush connections that are idle for more than 30 minutes and regenerate new connections in the pool. I also did sanity check over the connections in the pool.

Transaction & Batch-processing in Grails

February 1, 2009

Transaction handling is one of the more complex areas of web development. Anytime a user takes any action in the interface which demands a couple of database actions in the backend, then usually you end up having do it as transaction. For a user, everything is either a success or failure. Partial success may be either harmful to the system or doesn’t mean anything to the user. Grails, since it is built on top of Spring and Hibernate, uses their underlying mechanism to deal with transactions. While it may seem confusing in the beginning, Grails actually makes it even more easier.

Lets see it in code.

Suppose we are designing an Offers System, like the one that your bank sends notifying you of some offers and promotions on services. A simplified object structure may be like the following:

class Offer {
     String title
     String content
     Date expirationDate
     Date createdDate = new Date()

     static hasMany = [recipients: Recipient]
}
class Recipient {
     String email

     static belongsTo = [offer: Offer]
}

The relationship is fairly simple. Each Offer can have many Recipients, each Recipient belongs to an Offer.

belongsTo just means the offer Recipient will be deleted whenever the Offer is deleted, which makes sense coz we don’t want to keep the junk if the Offer itself is deleted.

hasMany lets you access to the Recipients from the Offer object. In other words, if you have the Offer object, you can do offer.recipients and get the list of all the recipients for the Offer. Cool.

Now, here is how it will work. We want to add an Offer and some Recipients. Everything should either succeed or fail. Even if only one Recipient out of thousands fail, everything should fail.

There are two ways to do transactions in Grails:

1. Method-level Transaction: By default, each Service class in grails has a transactional property set to true. So if any public method in a Service class throws a RuntimeException or any Error, the transaction will be rolled back. If you do not need this default transaction property, you can explicitly set transactional to false.

In the code below, the OfferService throws a Runtime Exception anytime it isn’t able to add the Fffer or the Recipient, and the transaction will be rolled back.
You wouldn’t get the same transactional behavior if the method throws any Checked Exception or if the transactional property is set to false.

class OfferService {

     boolean transactional = true

     def save(offer, recipients) {
         if(!offer.validate()) {
             throw new RuntimeException("Invalid offer.")
         } else {
             def result = offer.save()
             recipients.each {
            	if(!it.validate()) {
            		throw new RuntimeException("Invalid recipient")
                } else {
                  	offer.addToRecipients(it)
                }
             }
         }
     }
}

You can then catch the exception in the Controller and render a more user-friendly error message.

class OfferController {
     def save = {
    	 def offer = getOffer(params)
    	 def recipients = getRecipientList(params)
   	 try {
   		offerService.save(offer, recipients)
		flash.message = "Successfully added the offer."
   	 } catch (Exception e) {
   		log.error "Exception: " + e
 		flash.message = "An error occured, and the offer was not saved."
   	 }
     }

}

So all you need to do is make sure your Service is transactional, and that it throws a RuntimeException and not a Java CheckedException.

2. Block-level Transaction: The first approach treats everything inside the method as a single transaction. But sometimes your need more control than that. You might want to perform some actions within a transaction and do other operations regardless, or within another block of transaction within a single method.

Each domain class in grails has a static withTransaction method that takes a closure as an argument. Lets see the same thing in code using withTransaction:

class OfferService {

    boolean transactional = false

    def save(offer, recipients) {

    	Offer.withTransaction { status ->
             if(!offer.validate()) {
                  status.setRollbackOnly()
             } else {
         	  def result = offer.save()
                  recipients.each {
           	       if(!it.validate()) {
          	          status.setRollbackOnly()
                       } else {
                  	  offer.addToRecipients(it)
                       }
                  }
              }
         }

    }
}

The status object that we are calling the setRollbackOnly on is actually an instance of org.springframework.transaction.TransactionStatus class. The funny thing is it doesn’t really matter which Domain object you call the withTransaction method on. It just to mark the start and end of a transaction.

As you can guess, with this approach, you don’t need to be in a Service class to use a transaction. You can use it from inside a controller too.

In both of the process above, grails isn’t actually doing a database rollback. It is just saving the changes into the Hibernate session until the transaction is successful, at which point it commits the change to the database. So it is holding onto a lot of objects in session cache.

I have been fighting a situation where I have to batch insert a thousand of objects within a transaction. My common sense tells me I need something like nested transaction, where I could divide a large parent transaction into smaller chunks of transactions, and if one of them fails, it would trigger the parent to fail. But looking at the way Hibernate session works, I was worried that I might have to stick to plain JDBC to do it. But not anymore, coz I found this awesome doc on Hibernate Batch-processing.

Batch Processing: Each action in a Grails Controller is executed within a Hibernate Session. The session is started right before the action starts and is closed once it returns. That explains the reason for the memory issues because all the objects are being cached to the Hibernate session. Hibernate saves it to the database when the session is flushed, thus freeing the memory. The same code as above with the session cleared more frequently:

class OfferService {

    boolean transactional = false
    def sessionFactory

    def save(offer, recipients) {

    	Offer.withTransaction { status ->;
             if(!offer.validate()) {
                  status.setRollbackOnly()
             } else {
         	  def result = offer.save()
                  int i = 0
                  recipients.each {
           	       if(!it.validate()) {
          	          status.setRollbackOnly()
                       } else {
                          i++
                  	  offer.addToRecipients(it)
                          if(i % 100 == 0) {
                               sessionFactory.getCurrentSession().clear();
                          }
                       }
                  }
              }
         }

    }
}

Note: You can get the sessionFactory in a Controller similarly.
This way the Hibernate session, which is the first-level cache, is being cleared every once in a while.
This worked like a charm for me, solving all the memory issues I was having, and also speeding things up significantly.