Bash Script to Backup MySQL Databases

February 24, 2013

mysqldump is a program to do a dump of a MySQL database. It creates a .sql file, which you can then use to restore the database.

Back up a MySQL database:

mysqldump -u mysql_user -h ip -pmysql_password database_name > database_name.sql

To restore a database from the database_name.sql file:

mysql -u mysql_user -h ip -pmysql_password database_name < database_name.sql

Backup all databases on the server:
Interestingly, you can backup all databases on the server:

mysqldump -u mysql_user -h ip -pmysql_password -A > all_databases.sql

To restore all databases:

mysql  -u mysql_user -h ip -pmysql_password < all_databases.sql

Backup a table on a MySQL database:
You can also do mysqlduml at the table level:

mysqldump -u mysql_user -h ip -pmysql_password database_name table_name > table_name.sql

To restore the table to the database:

mysql  -u mysql_user -h ip -pmysql_password database_name table_name < table_name.sql

I have a bunch of MySQL databases hosted on a bunch of servers. I have been pretty lazy to back them up regularly. So I wrote a quick bash script to back up the MySQL databases, creating a separate backup file for each database.

Here’s what the script does in short:
1. You provide it a list of ip address, username and password.
2. It will mysqldump all the databases on each host server that the user has access to.
3. It will store all the dumps in the backup_dir and compress each dump using gunzip.

Bash doesn’t really support multi dimensional arrays. So I had to store the ip, username and password as a comma separated string and split it up in each iteration. Ugly, but gets the job done for now.

#!/bin/bash
# Script to do back of all mysql databases on different hosts.

servers=(
        "10.10.10.10,mysql_user0,mysql_password0"
        "10.10.10.11,mysql_user1,mysql_password1"
        "10.10.10.12,mysql_user2,mysql_password2"
        )

function mysql_dump() {
        local ip="$1"
        local mysql_user="$2"
        local mysql_password="$3"
        mysql_databases=`mysql -u ${mysql_user} -p${mysql_password} -h ${ip} -e "show databases"| sed /^Database$/d`
        for database in $mysql_databases
        do
                if [ "${database}" == "information_schema" ]; then
                        echo "Skipping $database"
                else
                        echo "Backing up ${database}"
                        mysqldump -u ${mysql_user} -p${mysql_password} -h ${ip} ${database} | gzip > "${backup_dir}/${database}.gz"
                fi
        done
}

parent_dir="/home/sacharya/database_backup"
backup_date=`date +%Y_%m_%d_%H_%M`
backup_dir=${parent_dir}/${backup_date}
mkdir -p "${backup_dir}"

saveIFS=$IFS
for server in ${servers[@]}
do
        IFS=","
        cols=($server)
        IFS=$saveIFS
        mysql_dump ${cols[0]} ${cols[1]} ${cols[2]}
done

Error Messages in Java

December 1, 2011

Every time something goes wrong within the application or you want to notify the user of something, you display messages to the user. These messages could be simple validation errors, or severe errors in the backend. Such events can happen anywhere within the system and we want to be able to tell the user what happened in a user-friendly way. So its important that there messages are well-written and consistent and they better be stored in a central place for maintenance.

Here’s a simple and clean way to do it that I have used over the years. Its pretty simple, really. I am using Enums for all error messages and using message format so I can pass in extra variables. While the messages itself are still in code, at least they are in the same file that anyone can use and maintain.
Errors.java

package errors;
import java.text.MessageFormat;

public enum Errors {

     USERNAME_NOT_FOUND("User not found"),
     USERNAME_EXISTS("Username {0} already exists."),
     USERNAME_CONTAINS_INVALID_CHARS("Username {0} contains invalid characters {1}.");

     private final String message;

     ErrorMessages(String message) {
          this.message = message;
     }

     @Override
     public String toString() {
          return message;
     }

     public String getMessage(Object... args) {
          return MessageFormat.format(message, args);
     }

     public static void main(String args[]) {
          System.out.println(ErrorMessages.USERNAME_NOT_FOUND);
          System.out.println(ErrorMessages.USERNAME_EXISTS.getMessage("sacharya"));
          System.out.println(ErrorMessages.USERNAME_CONTAINS_INVALID_CHARS.getMessage("s%acharya", "%"));
     }
}

Sometimes the above solution is not enough. You want to do internationalization on the messages and display different messages to the user based on the locale. In such a case, you can externalize the messages to a resource.properties file, and have translation of the messages in each locale file. You can then use ResourceBundle and currentLocale to get the right properties file and construct the right message for the given key using MessageFormat.

Messages.java

package errors;

import java.text.MessageFormat;
import java.util.Locale;
import java.util.ResourceBundle;

public class Messages {

     public static String getMessage(String key, Object... args) {
          // Decide which locale to use
          Locale currentLocale = new Locale("en", "US");
          ResourceBundle messages = ResourceBundle.getBundle("resource", currentLocale);

          MessageFormat formatter = new MessageFormat("");
          formatter.setLocale(currentLocale);

          formatter.applyPattern(messages.getString(key));
          String output = formatter.format(args);
          return output;
     }

     public static void main(String args[]) {
          System.out.println(Messages.getMessage("USERNAME_NOT_FOUND"));
          System.out.println(Messages.getMessage("USERNAME_EXISTS", "sacharya"));
          System.out.println(Messages.getMessage("USERNAME_CONTAINS_INVALID_CHARS", "s%acharya", "%"));
     }

}

There following resource.properties files should be available in the classpath so the class can see it.

resource.properties

USERNAME_NOT_FOUND=User not found.
USERNAME_EXISTS=Username {0} already exists."
USERNAME_CONTAINS_INVALID_CHARS=Username {0} contains invalid characters {1}.

resource.properties_en_US

USERNAME_NOT_FOUND=User not found.
USERNAME_EXISTS=Username {0} already exists."
USERNAME_CONTAINS_INVALID_CHARS=Username {0} contains invalid characters {1}.

Sample usages have been demonstrated in the main method above. Choose whatever way serves your purpose.

Nginx Proxy to Jetty for Java Apps

March 4, 2011

Traditionally, I used to go with Apache, Mod Jk and Tomcat to host any Java web apps. But this time I was working on a small hobby project written in Groovy on Grails and had to deploy it to a VPS with a very limited resources. So I had to make the most of the server configuration that I had. So I went with a combination of Nginx and Jetty.

If you’ve never heard of Nginx, it is a very simple HTTP server that is known for its high-performance, low and predictable resource consumption and low memory footprint under load. It uses an asynchronous even-driven model to handle requests which enables it to efficiently handle a large no of requests concurrently.

Similarly, Jetty provides a very good Java Servlet Container. Jetty can be used either as a Standalone application server or can be embedded into an application or framework as a HTTP Component or a servlet engine. It servers as a direct alternative to Tomcat in many cases. Because of its use of advanced NIO and small memory footprint, it provides very good scalability.

Below, I will jot down the steps I went through to configure Nginx as a frontend to Jetty on my VPS running Ubuntu Hardy.

Install Java:

$sudo apt-get install openjdk-6-jdk
$ java -version
java version "1.6.0_0"
OpenJDK  Runtime Environment (build 1.6.0_0-b11)
OpenJDK 64-Bit Server VM (build 1.6.0_0-b11, mixed mode)

$ which java
/usr/bin/java

Install Jetty:

Download the latest version of Jetty, and upload the tar file to your directory of chose on your server.

$ scp jetty-6.1.22.tar user@sacharya.com:/user/java

Now login to your server, go to the directory where you uploaded Jetty above.

$ cd /user/java
$ tar xvf  jetty-6.1.22.tar

Now you can start or stop the Jetty server using the following commands:

$ cd /user/java/jetty-6.1.22/bin
$./jetty.sh start
ps aux | grep java
root     21766  1.2 72.4 1085176 387196 ?    Sl   Mar27   1:12 /usr/lib/jvm/java-6-openjdk/
/bin/java -Djetty.home=/user/java/jetty-6.1.22 -Djava.io.tmpdir=/tmp -jar
/user/java/jetty-6.1.22/start.jar /user/java/jetty-6.1.22/etc/jetty-logging.xml
/user/java/jetty-6.1.22/etc/jetty.xml

The jetty logs are under jetty-6.1.22/logs if you are interested.

Open up your bash profile and set the following paths:

$ vi ~/.bash_profile
JAVA_HOME=/usr/lib/jvm/java-6-openjdk/
JETTY_HOME=/user/java/jetty-6.1.22/

PATH=$JETTY_HOME/bin:$PATH

export JAVA_HOME JETTY_HOME

Now that Jetty is running, you can go the its default port 8080 and verify that everything is working as expected.

Now that you have Jetty, its time to deploy your app to the Jetty container.

$ scp myapp.war  root@sacharya.com:/user/java/jetty-6.1.22/webapps
$ tar -xvf myapp.war

$ vi /user/java/jetty-6.1.22/contexts/myapp.xml

<?xml version="1.0"?>
<!DOCTYPE Configure PUBLIC "-//Mort Bay Consulting//DTD Configure//EN" "http://jetty.mortbay.org/configure.dtd">
<Configure class="org.mortbay.jetty.webapp.WebAppContext">
<Set name="configurationClasses">
<Array type="java.lang.String">
<Item>org.mortbay.jetty.webapp.WebInfConfiguration</Item>
<Item>org.mortbay.jetty.plus.webapp.EnvConfiguration</Item>
<Item>org.mortbay.jetty.plus.webapp.Configuration</Item>
<Item>org.mortbay.jetty.webapp.JettyWebXmlConfiguration</Item>
<Item>org.mortbay.jetty.webapp.TagLibConfiguration</Item>
</Array>
</Set>
<Set name="contextPath">/</Set>
<Set name="resourceBase"><SystemProperty name="jetty.home" default="."/>/webapps/myapp</Set>
</Configure>

Restart jetty and go to http://ipAddress:8080/myapp, and you should be getting your app.

Install Nginx:

$ sudo aptitude install nginx

This will install Nginx under /etc/nginx

You can start, stop or restart the Nginx server using the commands:

$ sudo /etc/init.d/nginx start
$ sudo /etc/init.d/nginx stop
$ sudo /etc/init.d/nginx restart

Go to your server ip address (or locahost of local) in your browser, and you should be able to see the default welcome page.

Nginx Proxy to Jetty:
Now, lets point configure Nginx as a proxy to our Jetty Server:

$ cd /etc/nginx/sites-available
$ vi default
Point your proxy_pass to:

location / {
proxy_pass         http://127.0.0.1:8080;
}

Basically, nginx listens on port 80 and forwards it to port 8080. Jetty sets anything on / to /webapps/myapp which means any request to http://127.0.0.1 from nginx is served from http://127.0.0.1:8080/myapp.

Now if you type your IP address or domain name in the browser, content will be served from your application in Jetty. Right now, you are serving everything through Jetty including the scatic files like images, javascript and css. But you can easily serve the static files directly through Nginx: Just add a couple of locations in there:

location /images {
root /user/java/jetty-6.1.22/webapps/myapp;
}
location /css {
root /user/java/jetty-6.1.22/webapps/myapp;
}
location /js {
root /user/java/jetty-6.1.22/webapps/myapp;
}

My final configuration is:

server {
listen   80;
server_name sacharya.com;

access_log  /var/log/nginx/localhost.access.log;

location / {
proxy_pass http://127.0.0.1:8080;
}
location /images {
root /user/java/jetty-6.1.22/webapps/myapp;
}
location /css {
root /user/java/jetty-6.1.22/webapps/myapp;
}
location /js {
root /user/java/jetty-6.1.22/webapps/myapp;
}

# redirect server error pages to the static page /50x.html
#
error_page   500 502 503 504  /50x.html;
location = /50x.html {
root   /var/www/nginx-default;
}
}

Find the Jar File Given a Class Name

March 3, 2011

Often times, while working in Java, you get a ClassNotFoundException or a ClassCastException and you are trying to find find out what Jar the class belongs to and where is it located in the classpath. Your application is either not finding the class or finding the wrong class with the same Class name in the classpath. So you wanna know what Jar is your class coming from at Runtime and whether that is the right class.

Grep to find all Jars with the Class name:

You could write a little bash script to do a find for the class file within your fileSystem, but that doesn’t tell you whats loaded in the classpath. So it will give you shit load of crap that have the class name:

$ cd ~/.groovy
$ $ find . -name "*.jar" -exec sh -c 'jar -tf {} | grep -H --label {} org.apache.commons.httpclient.HttpClient.class' \;
./lib/commons-httpclient-3.1.jar
./lib/commons-httpclient-3.1_patched.jar

This above command will search in the current directory and all sub directories for any jars. Then for each jar file, it will view the contents of the jar file using jar tf and look for the java class org.apache.commons.httpclient.HttpClient.class. The output will be different depending on where I am running the script from.

Java Class to find Jars with the given Class in Classpath:

But you only want to find the Jar File loaded into the Java Classpath. Here’s a simple Java Class that does the same from within an main class:

import java.net.URL;

import org.apache.commons.httpclient.HttpClient;

public class MainApp {

    public static void main(String[] args) {
        System.out.println(findPathJar(HttpClient.class));
    }

    public static String findPathJar(Class<?> context) throws IllegalStateException {
        URL location = context.getResource('/' + context.getName().replace(".", "/")
                            + ".class");
        String jarPath = location.getPath();
        return jarPath.substring("file:".length(), jarPath.lastIndexOf("!"));
    }
}

This will print the jar file in the classpath that contains the class HttpClient.class:

/Users/sacharya/Documents/MyLibs/commons-httpclient-3.1.jar

The output will be same no matter where I am running the class from, since it is looking at the classpath and not the current directory.

Handy Groovy Script to find Jar with the given Class in Classpath:

#!/usr/bin/env groovy

def klass
System.in.withReader {
   println "Enter the Class name you want to find the jar for:"
   klass = it.readLine()
}
def context = Class.forName(klass)
def absolutePath = context.getResource('/' + context.name.replace(".", "/")
           + ".class").getPath()
println absolutePath.substring("file:".length(), absolutePath.lastIndexOf("!"))

Running this as a script, I get:

$ ./getJarFile.groovy
Enter the Class name you want to find the jar for:
org.apache.commons.httpclient.HttpClient
/Users/sacharya/.groovy/lib/commons-httpclient-3.1.jar

Again, the output will be same no matter where I am running the script from.

Last of all, I find the site http://jarfinder.com/ very helpful too to find Jar files for a given class.

Facebook Connect with JSecurity on Grails

April 18, 2010

Say you have a Grails application using JSecurity (now called Apache Shiro) for authentication. How do you provide an alternate mechanism to authenticate users using Facebook Connect?

Good thing! Grails has the Facebook Connect plugin so you can authenticate users without registering them to your system. But what you will want is to integrate the Facebook Connect Plugin with the JSecurity Plugin. That way, even though a user is actually authenticated using Facebook Connect, your JSecurity knows about the user and can assign roles and permissions to the user.

I assume you have both the Apache Shiro Plugin installed and authentication working properly. Now go ahead and install the and Facebook Connect Plugin by following the documentation.

In addition to the regular JSecurity Login form, you will have the Facebook Login button as follows:

login.gsp

<script type="text/javascript">
    <!--
        function delayer() {
            window.location = "${createLink(controller:'auth', action: 'signin')}"
        }
    //-->
</script>
<div id="login_section">
    <h4>Login using Facebook Connect</h4>
    <div class="login_form">
        <fb:login-button autologoutlink="false" onlogin="setTimeout('delayer()',100)">
        </fb:login-button>
        <br/>
	<fb:name uid="loggedinuser" useyou="false"></fb:name>
	<fb:profile-pic uid="loggedinuser" size="normal" />
	<g:facebookConnectJavascript  />
    </div>
</div> <!--Login Form Ends-->

The above code snippet will display the Facebook login button. When the user clicks on the button, Facebook Connect’s login dialog is displayed. Once he user authenticates himself to Facebook, the delayer() javascript method is called which then redirects to the auth/signin grails action.

Now lets insert the following snippet of code into the signIn action.

AuthController.groovy

def facebookConnectService
def signin = {
    if(!params?.username && !params?.password) {
        if(facebookConnectService.isLoggedIn(request)) {
            try {
                facebookConnectService.getFacebookClient().users_getLoggedInUser()
                params.rememberMe = false
                params.username = "facebook"
                params.password = "randompassword"
            } catch (Exception e) {
                flash.error ="We are sorry. Please try again in a while."
                redirect(controller: 'home', action: 'index')
                return
            }
        }
    }
    // Rest of the Code continues
}

where, facebookConnectService is the service class that is provided by the the Grails Facebook Connect plugin. We are just checking if the user is logged in to Facebook, and then setting values for username and password as per JSecurity’s needs.

The actual authentication of the user is done in grails-app/realms/ShiroDbRealm.groovy.

Now lets add a few lines to our ShiroDbRealm.groovy to handle our Facebook user:

ShiroDbRealm.groovy

def facebookConnectService

def authenticate(authToken) {
        log.info "Attempting to authenticate ${authToken.username} in DB realm..."
        def username = authToken.username

        // Null username is invalid
        if (username == null) {
            throw new AccountException('Null usernames are not allowed by this realm.')
        }

        def user = null
        def account = null
        if (username == "facebook") {
            def facebookUser = null
            try {
                facebookUser = getFacebookUser()
            } catch (Exception e) {
                e.printStackTrace()
            }
       	    try {
       	        user = User.findByUsername(facebookUser.username)
       	    } catch (Exception e) {
       	        e.printStackTrace()
       	    }

           if (!user) {
                facebookUser.passwordHash = new Sha1Hash("randompassword").toHex()
                if (facebookUser.validate()) {
	                user = facebookUser.save(flush: true)
                } else {
                        facebookUser.errors.allErrors.each {
                            log.error it
                        }
                }

                account = new SimpleAccount(user.username, user.passwordHash, "ShiroDbRealm")
                return account;

            } else {
                account = new SimpleAccount(user.username, user.passwordHash, "ShiroDbRealm")
                return account;
            }
      }
      // Rest of the Code continues
}

def getFacebookUser () {
        String userId = facebookConnectService.getFacebookClient().users_getLoggedInUser()
        java.lang.Iterable<java.lang.Long> userIds = new ArrayList<java.lang.Long>()
        userIds.add(Long.parseLong(userId))

        Set<ProfileField> c = new HashSet<ProfileField>()
        c.add(ProfileField.FIRST_NAME)
        c.add(ProfileField.LAST_NAME)
        c.add(ProfileField.NAME)

        def myresults = facebookConnectService.getFacebookClient().users_getInfo(userIds, c)
        def useObj = myresults.getJSONObject(0)
        User user = new User()
    	user.username = userId
    	user.firstName = useObj.getString("first_name")
    	user.lastName = useObj.getString("last_name")
    	user.name = useObj.getString("name")
    	user.displayName = user.name

        Date date = new java.util.Date()
    	user.dateCreated = date
    	user.lastUpdated = date
    	user.lastVisited = date
        return user
}

Basically, if the user uses Facebook Connect, we are setting his username to ‘”facebook” and password to “randompassword” from the signin action. In the authenticate method, if the username is “facebook”, we are getting all the info of the user from facebook. If the user with that facebook id is already in our database, it is an existing user and is authenticated. If the user doesnt exist in our database, the new user is added to the databse with a username and password. The username and password is only for integrating Facebook Connect with JSecurity, and the user has no idea about it. Now even though the user is actually authenticated by Facebook, it still is a JSecurity user in our database, and will be treated just like any other user.

Please note that I have added a few properties like firstName, lastName, displayName etc to the User.groovy which is created by JSecurity. Feel free to add new properties to the User object if you want to capture more user info from Facebook for that user.

Now you are good to go. Deploy and test the app.

Soon, you will notice that this will work in development, but there is a bug in the Current revision of Grails Facebook Connect Plugin in production environment, due to which it it cannot find the FacebookConnectConfig.groovy in prod. Go and download the source for Facebook Connect Grails plugin and modify the afterPropertiesSet method under services/FacebookConnectService.groovy in the grails-facebook-connect-0.1 project.

FacebookConnectService.groovy

void afterPropertiesSet() {
    // check if there is a compiled class for the Facebook connect config groovy script
    // this will be the case when an application is bundled in a WAR
    def config = null
    try {
        config = Class.forName("FacebookConnectConfig").newInstance()
    } catch(Exception  e) {
        e.printStackTrace()
    }
    if (config != null) {
        // compiled config class found, we must be running as a WAR
        facebookConnectConfig = new ConfigSlurper().parse(config.getClass())
    } else {
        // no compiled class exists for the config, we must be running the Grails built-in web server
        GroovyClassLoader loader = new GroovyClassLoader(getClass().getClassLoader())
        Class clazz = loader.parseClass(new File("grails-app/conf/FacebookConnectConfig.groovy"))
        facebookConnectConfig = new ConfigSlurper().parse(clazz)
    }
}

Now use the patched plugin in your app and that should work!

Shoud you use Facebook Connect?:
Based on my few months of experience, users are pretty hesitant to login to a site using Facebook Connect. In my case, less than 10% of the users used Facebook Connect on my site, and it certainly wasn’t worth the effort I put into making it work and mantaining it. Its kind of clunky at times too. I don’t think I am never gonna use it again just for the sake of authentication. So, make sure it really adds some value to your application and users, before you decide to integrate it in your app coz you thought its cool.