Category: Erlang

Writing Ejabberd Modules

June 10, 2009

Ejabberd is an open-source XMPP server written in Erlang. Although XMPP has been known for building instant messaging applications for a long time now, over the last few years, people have used it for building very interesting realtime applications. In fact both XMPP and Erlang are being boosted by a new found enthusiasm in the last couple of years. Having played with Erlang for a couple of years now, Ejabberd is an obvious choice for me as an XMPP server. Here I am just documenting things for myself as I learn to write simple modules – which is a powerful way of extending and plugging into the basic Ejabberd server. 

I assume you already have Erlang installed. So lets install Ejabberd from the source.

$ svn co ejabberd
$ cd ejabberd/src
$ ./configure
$ make
$ sudo make install

Upon installation you will notice the following important directories and files created, whose meanings are quite obvious from their names:

      -> ejabberd.cfg
      -> ejabberdctl.cfg

You can then use the following commands to start and stop the server.

$sudo /sbin/ejabberdctl start
$sudo /sbin/ejabberdctl status
The node ejabberd@localhost is started with status: started
ejabberd 2.1.0-alpha is running in that node
$sudo /sbin/ejabberdctl stop

Writing an Inernal Module:
All internal modules in Ejabberd start with the name ‘mod_’ implement the gen_mod behavior through the two methods:

start(Host, Opts) -> ok
stop(Host) -> ok

where Host is the name of the virtual host running the module, and Opts is the set of options.
So lets just write a basic module that will print something when it starts.

$ cd /ERLANG_LIB/ejabberd/src
$ vi mod_hello.erl


start(_Host, _Opt) ->
        ?INFO_MSG("Loading module 'mod_hello' ", []).

stop(_Host) ->

Compile the module, move the beam file to /lib/ejabberd/ebin:

$ erlc mod_hello.erl
$ sudo mv  mod_hello.beam /lib/ejabberd/ebin

Now we need to configure our Ejabberd Configuration File to load the mod_hello module. So lets go to the sections where modules are enabled in the ejabberd.cfg, and add the following line to the file.

$ sudo vi /etc/ejabberd/ejabberd.cfg
  {mod_first_module, []},

Start the server and verify that the message is printed in the log.

$ sudo /sbin/ejabberdctl start
$ less /var/log/ejabberd.log


Writing an HTTP module:
Building an HTTP module is similar to building internal modules but the nice thing about HTTP modules is you can get requests from URLs, process them in the module and then send the response back. So if you need any extra interactivity or information exposed via the URL, HTTP module is the way do it.
Each HTTP module implements the gen_mod behavior. That means, it has start/2 and stop/1 functions. It also has a request handler process/2 function, that handles the actual request.

Now lets write a HTTP module mod_available_user.erl. Given a URL like http://localhost:5280/users/sacharya, lets build an HTTP module that will send a response back telling whether the username ‘sacharya’ is already registered or not. So let me grab the sample template from Ejabberd Documentation, and modify it.

$ vi mod_available_user.erl





start(_Host, _Opts) ->

stop(_Host) ->

process(Path, _Request) ->
    {xmlelement, "html", [{"xmlns", ""}],
     [{xmlelement, "head", [],
       [{xmlelement, "title", [], []}]},
      {xmlelement, "body", [],
       [{xmlelement, "p", [], [{xmlcdata, is_user_exists(Path)}]}]}]}.

is_user_exists(User) ->
        Result = ejabberd_auth:is_user_exists(User, "localhost"),
        case Result of
                true -> "The username " ++ User ++ " is already taken.";
                false ->"The username " ++ User ++ " is available."

How the process works is pretty clear if you have some familiarity with Erlang. All we are doing is using the function is_user_exists/1 provided in the the ejabberd_auth module and deciding what message to display in the response.

Compile the file and move the beam to ebin.

$ erlc -I /lib/ejabberd/include/ mod_available_user.erl
$ sudo mv mod_available_user.beam /lib/ejabberd/ebin

The flags ‘I’ (Include) in the erlc command is just used to reference to the directory where the included .hrl files are located.
Now lets add the module into the configuration file so that the request handler will dispatch any requests to /users to our module.

$ vi /etc/ejabber/ejabberd.cfg
{5280, ejabberd_http, [
                        {request_handlers, [{["users"], mod_available_user}]}

Create some Users and Test:
To test the above module, we need some users. So lets create an admin user in the domain, login as admin and create a couple of users.

$ sudo ejabberdctl start
$ sudo ejabberdctl register admin localhost password
User admin@localhost succesfully registered

Now go to /lib/erlanged/ejabberd.cfg and add this user:

{acl, admin, {user, "admin", "localhost"}}

Now bring localhost:5280/admin in the browser and login using, username: admin@localhost and password: password

Once you are logged in, go to http://localhost:5280/admin/server/localhost/users/, and add a few users eg. sudarshan@localhost, acharya@localhost.

Now that we have the users created and the HTTP module to check the users ready, lets test some URLs:

Request: http://localhost:5280/users/sudarshan
Response: The username sudarshan is already taken.

Request: http://localhost:5280/users/joeuser
HTTP Response: The username sacharya is available.

So in a very easy way, you can build applications that interact with your core Ejabberd services through the URL.

Download Youtube video

January 11, 2009

How to download Youtube videos has been done by a number of bloggers in a number of languages. I just wanted to try it out myself in Erlang, but I had no idea where to start.

Now that I’ve decided to start, let me grab an Eddie Vedder song from Youtube:

And lets sniff the HTTP traffic using Wireshark to see what actually happens when we watch a Youtube video. The browser made a conection to Youtube and before it started streaming the video, the input URL got redirected to a different Youtube URL (Depending on whether Youtube is caching the video or not, it could in turn probably be redirected to a different Cache server or a different IP):

No.     Time        Source                Destination           Protocol Info
     96 3.295105    xx.xx.xx.xx        HTTP     GET /get_video?video_id=gct6BB6ijcw&t=OEgsToPDskJ6n06uQXzbbyp7xAnxK6pN&el=detailpage&ps= HTTP/1.1

No.     Time        Source                Destination           Protocol Info
    115 3.879454    xx.xx.xx.xx         HTTP     GET /get_video? HTTP/1.1

Upon playing with the new URL in the browser, I realized that the URL that Youtube gets the video from is

The param “video_id” is the same as the param “v” in the original URL. So we only need to find the value for the param “t”.
Lets look at the HTML source for

and see if it contains the value for the parameter “t”. Luckily, grepping for “&t=”, I found this in the source:


The value of “t” in the HTML source and the value of “t” in the redirect URL is different, but I found that both values of “t” seemed to work when appended to the URL. Oh well, I tried it again just to make sure and realized that the value of the parameter “t” changes for every request, but all values seem to work(Probably it is timestamp dependent!).

So we have a plan now:
1. Get a Youtube video URL and make a HTTP request to it.
2. Get the body of the reponse and find the value of “t” using regex pattern matching.
3. Generate the proper redirect URL using the two parameters “video_id” and “t”.
4. Make a http request to the new URL and stream the bytes and write to a file with “.flv” extension.

Here is the full source code in Erlang: 


download(URL) ->
    {ok, {_Status, _Header, Body}} = http:request(URL),
    Video_URL = get_video_download_url(URL, Body),

stream_video(Video_URL) ->
    io:format("Downloading video from ~p~n", [Video_URL]),
    {ok, {_Status, _Header, Body}} = http:request(Video_URL),
    file:write_file("myvideo.flv", Body),
    io:format("Download complete!").

get_video_download_url(URL, Body) ->
    Matcher = "&t=[A-Za-z0-9-_]*",
    {match, Start, Length} = regexp:first_match(Body, Matcher),
    T = string:substr(Body, Start, Length),
    {ok, New, _No} = regexp:sub(URL, "watch\?v=", "get_video?video_id="),
    New ++ T.

Lets run it from the Erlang shell:

1> c(video_downloader.erl).
2> inets:start().
3> video_downloader:download("").
Downloading video from "
Download complete!ok

Go to your current directory and use any Flv Viewer to see if the downloaded file is a working video or convert it to format of your choice and watch it offline.

I would love to give an Erlang twist to it by spawning a few concurrent processes to download videos, but this is not quite a good example to do it from by localbox – too much of Disk IO, Network IO and slow Internet connection.

Erlang – overhyped or underestimated?

November 14, 2008

Erlang is like an exotic beautiful woman with no dressing sense.

I came across Erlang about a year ago. It is a language of a kind – precise, crafted and powerful. Its been there for about two decades now, and people have been using it for serious real-time applications. But the hype Erlang is getting in the last couple of years is mostly for wrong reasons.

Having said that, its a language every programmer should look into at least once – its refreshing. And it might just be the perfect language that fits your needs.

What doesn’t Erlang have? It has its own Virtual Machine that it runs on. It is ridiculously simple to write distributed applications, with its message-passing style concurrency. It has its own database. It supports hot code swapping without even restarting the servers, and its already been proven in some telecom applications that need very high uptime. Hot code swapping is still a nightmare even in a supposedly mature language like Java which is targeted at building web applications – that have high availability and scalability.

However there are certain things that either suck or are not as convincing about Erlang.

1. Today’s mainstream developers who are used to C or Java like syntax wont find its Prolog-like syntax too friendly. Unless you started programming since the 80′s & 90′s and are used to languages of similar syntax, it will take quite some time before you get comfortable with Erlang’s weird syntax. I never felt too comfortable with the syntax.

2. While the core language itself is small and easy to learn, the libraries within the language are inconsistent, incomplete and poorly documented. I posted a couple of questions in the forums regarding how to use a library, and usually the answers would be “Don’t use that library, use the other one”. (Oh yeah, kind of like the Java JDK Logging. Please use Commons Logging.)

3. Only a few people have written production level codes and you rarely get to hear from them. All you hear from is Erlang enthusiasts, who are hyping it as the next big thing, but haven’t done more than a few labs from Armstrong book.

4. I can’t imagine how you can organize large code-bases in Erlang or even work as team, and this doesn’t feel right to any OO programmer.

5. Most of the performance matrices are one-sided, and are performed by people who have an interest in Erlang. I would love to see some independent analysis.

6. Its support for web-development is very primitive. With web frameworks like rails and grails, there is a lot of serious work for Erlang if it ever intends to go to that market.

7. Did I talk about Strings in Erlang? IO speed?

I know weaknesses aren’t as important as the strengths of a language. Erlang has it own expertise, its syntax structure, and its own audience. But the flaws of Erlang might just turn away a new programmer, even before he gets to its beauty.

If you are writing a web crawler, Erlang may very well be your choice. If you want to write a client-server, where the client makes a large no of requests, and you want to spawn concurrent processes to process the requests, Erlang could be your choice. If you want to write a Distributed Hash Table, Erlang could be your choice. Or if you are writing a video streaming server or doing system integration or writing any system utility. But a regular developer (building a CRUD application on top of a database, right? ) doesn’t have much to do with Erlang as yet. Secondly, even if you are working on those highly scalable, reliable and concrurrent systems, people have a hard time accepting Erlang along with its flaws.

The industry has a definite space for Erlang, currently and more so in future as we deal with more and more users, more data, and more forms of distribution. If not for Erlang exactly, then for an improved version of Erlang. It isn’t here to be the next Java, but to solve out the problems that Java couldn’t do smoothly in over a decade (despite having such a great community).

Erlang is going from an underestimated to an overhyped language. I wish it can convert the hype and raw interest in Erlang into something meaningful. How about a modern variation of Erlang on the Erlang’s virtual machine. Is it too late?

MD5 in Erlang

September 10, 2008

An MD5 hash for any given message is 16 bytes (128 bits) in size and is represented by a unique 32 digit Hexadecimal number.

Erlang has a built-in-function to calculate the MD5, which returns the hash in the form of a binary data-structure.

$ erlang:md5("hello").

But what we usually need is a string representation of the Hexadecimal value. Hence, we need to convert the Erlang Binary to a Hex-string, for which I didn’t find any BIF.

So first of all, lets convert the Binary to a list (of integers)- so that we can process it easily. Erlang has a function for converting binary to list:

$ binary_to_list(<93,65,64,42,188,75,42,118,185,113,157,145,16,23,197,146>).

Now, we have to convert each of the integers in the list into its hex equivalent. How do you convert an integer in Decimal system to a Hexadecimal system?

Eg. Take an integer 230. Divide it by 16.
230 div 16 = 14 In Hex, 14 is E
230 rem 16 = 6 In Hex, 6 is 6
So 230 in Hex is E6.

Now we will have to do the same for every integer in the list. I did it using the lists:map function and the applying the int_to_hex conversion to every integer:

$ lists:map(fun(X) ->
int_to_hex(X) end, L).

This will actually return a list of integers representing the (hex) string, which is how a string is represented in Erlang – a list of Integers.

The complete code is below:


md5_hex(S) ->
       Md5_bin =  erlang:md5(S),
       Md5_list = binary_to_list(Md5_bin),

list_to_hex(L) ->
       lists:map(fun(X) -> int_to_hex(X) end, L).

int_to_hex(N) when N < 256 ->
       [hex(N div 16), hex(N rem 16)].

hex(N) when N < 10 ->
hex(N) when N >= 10, N < 16 ->
       $a + (N-10).


$ md5:md5_hex("hello").

Erlang mode on Emacs

August 12, 2008

Although there are a few other editors for Erlang, I prefer to use Emacs for Erlang and its the only major reason I use Emacs for. Erlang now has an eclipse plugin too, called Erlide.

Erlang comes with the emacs mode as part of its standard distribution, so you only need to customize your emacs settings to use the erlang mode. Once you have Emacs installed (I have Carbon Emacs on my Mac OSX), create a .emacs file in your home directory (or use the one that you already have).

$ vi ~/.emacs

Then insert the following lines of Lisp code into your .emacs file.

;Erlang Mode
(setq load-path (cons  "/usr/local/lib/erlang/lib/tools-2.6.1/emacs" load-path))
(setq erlang-root-dir "/usr/local/lib/erlang")
(setq exec-path (cons "/usr/local/lib/erlang/bin" exec-path))
(require 'erlang-start)

/usr/local/lib is where my Erlang is installed, and /usr/local/lib/erlang/lib/tools-2.6.1/emacs is the location where erlang.el and erlang-start.el files are, which actually define and initialize the Emacs Erlang-mode.

Update the path and version of the tools in the above code as per your installation, and enjoy the amazing features – including the Erlang shell right from Emacs.

Erlang on Mac OSX

April 28, 2008

On a fresh install of Leopard, the following is what I did in order to set up Erlang working on my Mackbook Pro:

1. Download Xcode by going to Apple ADC. Xcode is the Apple’s developer tool set, and its available for free. At the time of my install, the latest is Xcode 3.0 and its little over 1 GB. Double-click the downloaded file (named xcode_3.0.dmg), and follow the graphic package installer. This will install Xcode under /Developer by default. This step is required for Erlang because it will install GCC compiler required to build Erlang from source.

2. Get the latest Erlang source, which will be a file named otp_src_R12B-2.tar. Unarchive the file and follow the instructions:

$ cd otp_src_R12B-2
$. /configure
$ make
$ sudo make install

Now bring up a terminal and issue the command:

$ erl

Erlang (BEAM) emulator version 5.6.2  [smp:2] [async-threads:0] [kernel-poll:false]
Eshell V5.6.2  (abort with ^G)

I am all set to go…