erlang


Dear reader,

This blog is moving to some other hosting service (and other blog platform).

This blog URL will change and the RSS feed will change too; sorry about that.

If you want to follow this blog, please consider browsing http://blog.khigia.net and changing your RSS subscription to http://feeds2.feedburner.com/khigia/blog (or http://feeds2.feedburner.com/khigia/blog/erlang to filter only erlang related posts).

Thanks WordPress.com for both the hosting and the platform you’re providing.

New example in ocamerl lib of erlocaml project: a tiny map-reduce-like (1) word-count program (word-count is the simple example often used to illustrate map-reduce principle).

First part of this example is an ocaml distributed erlang node [l52]. It is composed of one permanent registered (‘wc’) mbox [l40] which upon request create other mboxes implementing the mappers [l19] (on request, a mapper read a text file [l8] and send it word by word to reducer [l29]).

Second part of this map-reduce example is an erlang module implementing two functionalities. One of them is the reduce part implementation [l34] done by a process which accept messages from mappers and update a dictionary accordingly.
Second responsibility of this module is orchestration of the map-reduce flow [l90], consisting of running all mappers [l66], waiting end of all those executions [l79], and collecting result of all reducers [l95].

Assuming EPMD is running, this script run an ocaml node and an erlang node, and run a simple map-reduce to count words in two test files.

As usual, not sure it’s a useful example, but it was sure fun to write ๐Ÿ™‚

(1) this is not a map-reduce lib … there is no failure case taken care of, processes pipe is mostly done through messages over network (while map-reduce is mainly meant to use an efficient filesystem based communication), etc!

Little intro step-by-step to the ocaml-erlang message exchange mechanism offered by ocamerl (a lib of erlocaml).

Subject of the following code is a wonderfull application … named “echo server”!

  1. In the interpretor shell of ocaml, we create a hidden erlang node and an activity to echo all received message (both on console as well as in respond to the caller).
  2. In the erlang interpretor shell, we send few message and compare them to the received ones.

The following assumes code is running on machine “comp1”.

Ocaml shell:

(* getting Ocamerl available in toplevel *)
ocaml> #use "topfind";; (* ocamerl used findlib for install *)
ocaml> #thread;; (* ocamerl is multithreaded app *)
ocaml> #require "ocamerl";;

(* creating/declaring a node, assuming epmd is running *)
ocaml> let o1 = Ocamerl.Enode.run "o1" ~cookie:"cookie";;
val o1 : Ocamerl.Enode.t = <abstr>

(* creating/declaring a mbox, equivalent of erlang process *)
ocaml> let m = Ocamerl.Enode.create_mbox o1;;
val m : Ocamerl.Enode.Mbox.t = <abstr>
ocaml> Ocamerl.Enode.register_mbox o1 m "echoer";; (* give it a name *)
- : unit = ()

ocaml> Ocamerl.Enode.Mbox.create_activity m (fun msg -> match msg with
    | Ocamerl.Eterm.ET_tuple [|pid; any;|] ->
        Printf.eprintf "MSG:%s\n%!" (Ocamerl.Eterm.to_string msg);
        Ocamerl.Enode.send o1 pid any
    | _ ->
        () (* drop unexpected msg *)
);;
- : unit = ()

Erlang shell:

# starting erlang node with same cookie
erl -sname e1 -setcookie cookie

% check connection
erl> pong = net_adm:ping(o1@comp1).
pong

% utility to print whatever is in message queue
erl> F = fun() -> receive W -> io:format("got back: ~w~n", [W]) after 1 -> error end end.
#Fun<erl_eval.20.67289768>

% some tests ... send data, received it back
erl> {echo1, o1@comp1} ! {self(), {1,2,"test"}}.
{<0.37.0>,{1,2,"test"}}
erl> F().
got back: {1,2,[116,101,115,116]}
ok
% in the mean time, ocaml shell also display the data

That’s it! A wonderfull echo server ๐Ÿ™‚

Amongst things on the to-do list:

  • Should not have to create a mbox and set its activity separately (need some wrapper)
  • Could have an onode ocaml toplevel which run one node by default and offer direct interface (e.g. “send”).

Yet another useless example for erlocaml, but this time it is not completely silly … it does something ๐Ÿ™‚

In fact the project eocarve uses:

Idea is simple: an ocaml node run and provide a API for Erlang node to call the seamcarving library. See eocarve wiki for details/example.

Aim of this project was mainly to demonstrate use of ocamerl lib … however it may be usefull for an Erlang web app which would need seamcarving (heavy weight for CPU!). Had fun to integrate those lib in same app.

Minor updates for erlocaml (more exactly on ocamerl) … mostly code cleaning and minor refactoring!

Only one new feature: ability for Erlang processes to send message to unregistered ocaml processes. An example of that is “ex_node_mult” which generalize “ex_node_double” by dynamically creating ocaml process to perform a multiplication … yep I know, not very useful

% erlang code using the ocaml node
{byn, OcamlNode} ! {self(), 2},
By2 = receive M -> M after 1 -> error,
{byn, OcamlNode} ! {self(), 3},
By3 = receive M -> M after 1 -> error,
P1 = make_ref(),
P2 = make_ref(),
By2 ! {self(), P1, 21},
By3 ! {self(), P1, 21},
ok = receive {P1, 42} -> ok after 1 -> error end,
ok = receive {P2, 63} -> ok after 1 -> error end

With this feature, ocamerl begin to be usable … and that’s exactly what I will be doing next: experiment with more useful (at least less silly) examples! Some ideas are:

  • tiny mapreduce example with ocaml workers (erlang doing the network part);
  • using the ocaml image processing lib from erlang (not best example as lot of data need to be exchange … this is task to be solved for future erlocaml development);
  • others???

Quick news about ocamerl:

  • I changed the build system to ocamlbuild (need ocaml 3.10).
  • … and that’s all! can’t find time to clean all the mess.

So anyway, here is the current state of ocamerl:

  • ocaml can register a hidden erlang node (epmd).
  • ocaml node can respond to ping and keep connection up with erlang node (net_kernel process)
  • erlang can send (some) terms to named ocaml process
  • ocaml can send (some) terms to erlang process if it received the pid in a previous message

That’s all for now! Argh!

Amongst things I’d like to change:

  • simplify terms manipulation in ocaml
  • add features (most important initial message from ocaml to erlang)
  • complete redesign of concurrency (using events or JoCaml maybe!)
  • add a minimum of documentation

I need motivation! In fact I have no project which would need ocaml + erlang … for now.

How could an Erlang process communicate with an ICE-based application? The mix seems a bit strange (why would you need ICE when you already have Erlang ๐Ÿ˜‰ ) but without entering in details of why we would like to do so, let’s consider how can we plug an Erlang sub-system in a ICE application.

One way to go would be to write a slice2erl compiler! That’s presumably lot of work. This kind of work is being done for OCaml in the Hydro project … and is on my list of next things to play with.

Another way to get Erlang talk to ICE application would be via a custom intermediate translator between the Erlang app and the ICE system (using some intermediate language having a complete interface with both Erlang and ICE, like C, Java, or Python); yes, this surely add complexity and slow down the full system, but efficiency of distributed application is more often bound by design than by runtime ;).

Btw, as far as ease-of-development is concern, I have the same kind of feeling when comparing C++ to Python or ICE to Erlang ๐Ÿ™‚

Managed to have some time to play with ocamerl: ocaml node is now able to receive data from erlang node, and also send data to pid on connected nodes.

I begin to understand what I’m doing, so that I will be able to clean the code and refactor some parts very soon.

As example, an ocaml process which multiply integer by 2! wow! What’ an interesting example!

bash> cd trunk/lib/ocamerl
bash> ./ex/node_double >/dev/null 2>&1 &
bash> erl -setcookie cookie  -sname erl
erl@devhost 1> net_adm:ping(ocaml@devhost).
pong
erl@devhost 2> {bytwo, ocaml@devhost} ! {self(), 8}.
{<0.37.0>,8}
erl@devhost 3>  receive I -> io:format("Got: ~w~n", [I]) after 0 -> ko end.
Got: 16
ok

Note: the ping was optional and the answer seems correct: 8 * 2 = 16!

It’s kind of surprising how people can like such a simple game as pong. Isn’t it even more odd to find programmer which enjoy their program to play?

All that to say: ocaml node (using the ocamerl lib of erlocaml) can now reply pong to a ping request from erlang node ๐Ÿ˜‰ … and that’s all it can do!

Next step: receive data ๐Ÿ™‚

Code in ocamerl (part of erlocaml) pass the node handshake. Hopefully nobody will read this code before I clean it up a bit … because right now it is a furious mess!!!

I spent some time (arg, that’s a lot in fact) on a stupid (as the bug’s author) bug. The (simplified) handshake is a follow:

  • the 2 nodes exchange a challenge which is 4 random bytes interpreted as a Int32;
  • each node computes the MD5 of (their own cookie + challenge);
  • each node compares its result with the digest of the other node.

My bug was to concatenate the cookie (string) with the raw 4 bytes of the challenge … where I had to concatenate the cookie with the string representation of the 32 bits long unsigned integer!

(in fact the handshake is done with 2 challenges: each node send a challenge to the other)

Anyway, the handshake pass correctly now, and ocaml node can receive the ping control message … but do not reply yet! after it reply, sure I clean the code!!!

As spotted in this blog post using Erlang in robotics seems a pretty good idea to implement robust systems and concurrent behaviours.

However I do not consider the subsumption architecture (Brooks, 1986) as the best example of action selection architecture for concurrent behaviours. As far as I remember, the subsumption architecture assumes predefined priorities on behaviours (hierarchy), and has a fix set of rules on how some behaviours can subsume others.

As far as reactive architectures are concerned, IMHO Erlang style of concurrency could be more effectively used on architecture using activation networks (Maes, 1989), or a voting mechanism a la DAMN (Rosenblatt, 1995), or a pertinence-based selection like Creature (Blumberg, 1997), or whatever architecture which enable a distribution of control.

But yes, Erlang seems quite a good match to implement autonomous entities (robots or software agents (MAS)).

Christmas is coming, and I found a new toy to play with: ocamerl. This is a very simple solution to exchange data between erlang and ocaml. In fact, this is mainly an ocaml implementation of the erlang external binary format (based on this discussion/code). I just added few tools to make a TCP connection to exchange those data.

On erlang side, I use 2 processes: the receiver process listen on a TCP port,read data and send it back to any erlang process meant to handle incoming data; the second process is busy transfering all data in its message queue to some TCP server (the ocaml part). On ocaml side, there are a couple a function to transform erlang forms in ocaml entities and vice versa.

I only wrote a very very simple exemple: an ocaml echo server, which multiply by 2 all integers sent in message (deep search in tuple).

Erlang side init:

1>Receiver = spawn(oe, print_mailbox, []).
2>Sender = oe:messenger(12345, 54321, Receiver).

Ocaml side init:

./ex/double -recv 12345 -send 54321

Then, any term sent to from erlang is sent back by ocaml.

3> Sender ! {true, 12, {1,2}}.
{true,12,{1,2}}
Sending data: <<131,104,3,100,0,4,116,114,117,101,97,12,104,2,97,1,97,2>>
Received binary: <<131,104,3,100,0,4,116,114,117,101,97,24,104,2,97,2,97,4>>
Converted to term: {true,24,{2,4}}
Received: {true,24,{2,4}}

Funny, isn’t it ? ๐Ÿ™‚ This is an early stage of course. Code is part of erlocaml project (in lib/ocamerl). My goal is to write a hidden erlang node in ocaml. Not sure it is useful, but quite certain it is interesting ๐Ÿ˜‰

Read some introductions to theory of categories (a very good introduction to the theory, or this one more programmer oriented). As I also stumble upon this thread, I discover a new concept: “hylomorphism is a composite of an anamorphism (unfold) and an catamorphism (fold/inject) [1], [2]“, along with a ruby code to implement it in some way (this the googlegroup thread above).

Just for fun (and be sure to understand it), I translated the code in Erlang. I’ve used the property list module to implement some kind of default value, but it’s not really elegant. Anyway, code is here.

-module(hylo).
-export([
    % hylomorphism API
    new/1,
    % examples/toys
    evens/1,
    fact/1,
    to_bin/1,
    expand/1
]).

% internal declarations

-record(hylo, {
    do,
    till,
    collecting,
    injecting
}).

% API

new(PL) when is_list(PL) -&gt;
    H = #hylo{
        do = proplists:get_value(
            do,
            PL,
            fun(X) -&gt; X + 1 end
        ),
        till = proplists:get_value(
            till,
            PL,
            fun(X) -&gt; X =:= undefined end
        ),
        collecting = proplists:get_value(
            collecting,
            PL,
            fun(X) -&gt; X end
        ),
        injecting = proplists:get_value(
            injecting,
            PL,
            {   [],
                fun(A,E) -&gt; [E|A] end,
                fun lists:reverse/1
            }
        )
    },
    fun(S) -&gt; eval(H, S, element(1,H#hylo.injecting)) end.

% internal implementation

eval(H, S1, R) -&gt;
    {_, InjF, InjR} = H#hylo.injecting,
    case (H#hylo.till)(S1) of
        false -&gt;
            V = (H#hylo.collecting)(S1),
            S2 = (H#hylo.do)(S1),
            eval(H, S2, InjF(R, V));
        _ -&gt;
            InjR(R)
    end.

% examples

evens(N) -&gt;
    H = new([
        {do, fun(X) -&gt; X + 2 end},
        {till, fun(X) -&gt; X &gt;= N end}
    ]),
    H(0).

fact(N) when N &gt; 0 -&gt;
    H = new([
        {do, fun(X) -&gt; X - 1 end},
        {till, fun(X) -&gt; X =&lt; 1 end},
        {injecting, {1, fun(A,E) -&gt; A * E end, fun(A) -&gt; A end}}
    ]),
    H(N).

to_bin(N) when N &gt; 0 -&gt;
    H = new([
        {do, fun(X) -&gt; X div 2 end},
        {till, fun(X) -&gt; X =&lt; 0 end},
        {collecting, fun(X) -&gt; (X rem 2) end},
        {injecting, {[], fun(A,E) -&gt; [E|A] end, fun(A) -&gt; A end}}
    ]),
    H(N).

expand(L) -&gt;
    H = new([
        {do, fun ([_|T]) -&gt; T; ([]) -&gt; [] end},
        {till, fun ([]) -&gt; true; (_) -&gt; false end},
        {collecting, fun([{C,N}|_]) -&gt; lists:duplicate(N, C) end}
    ]),
    H(L).

-ifdef(EUNIT).
-include_lib("eunit/include/eunit.hrl").

evens_test() -&gt;
    ?assert(evens(10) =:= [0,2,4,6,8]).

fact_test() -&gt;
    ?assert(fact(5) =:= 120).

to_bin_test() -&gt;
    ?assert(to_bin(10) =:= [1,0,1,0]).

expand_test() -&gt;
    ?assert(expand([{$a,2},{$b,3},{$c,4}]) =:= ["aa", "bbb", "cccc"]).

-endif.

Warning: long post ahead … but mostly code ๐Ÿ™‚

This post contains absolutely no idea nor thought: it is just a recap of my attempt to read an AVI file format (or RIFF file format, as I do not parse AVI data but only document structure). Let’s go directly in code with this simple module header!


-module(avir).
-compile([export_all]).
-include_lib("kernel/include/file.hrl").

dbg(Level, Template, Args) ->
    Indent = lists:flatten(lists:duplicate(Level, "  ")),
    io:format(Indent ++ Template, Args).

go() ->
    go("test.avi").

go(Filename) ->
    {ok, #file_info{size=Size}} = file:read_file_info(Filename),
    {ok, IODev} = file:open(Filename, [read, binary]),
    {ok, Parts} = walk_data(0, [], IODev, 0, Size).

So, dbg is a crap function to print debug message … yeah, the old fashion way, it’s so simple for just a post! go is the main entry point and call the ‘real’ code: the approach is to call the walk_data function which will build and return a list of AVI structures (first parameter will be level of nesting, used for printing comment with a meaningful indentation, and second one is an accumulator for recursion to come).

I mainly use this short document: AVI is a (nested) sequence of two kind of structure, either LIST or CHUNK. More precisely, first come a mandatory RIFF-AVI LIST then multiple (and optional) RIFF-AVIX kind of LIST. Let’s walk those structures:


walk_data(Level, Parts, File, From, To) when From < To ->
    case chunk_or_list(File, From) of
        avichunk ->
            {ok, Part, NextPos} = walk_chunk(Level, File, From, To),
            walk_data(Level, [Part|Parts], File, NextPos, To);
        avilist ->
            {ok, Part, NextPos} = walk_list(Level, File, From, To),
            walk_data(Level, [Part|Parts], File, NextPos, To);
        Error ->
            {error, "maybe unexpected EOF", Error}
    end;
walk_data(_Level, Parts, _File, _From, _To) ->
    {ok, lists:reverse(Parts)}.

chunk_or_list(File, Pos) ->
    case file:pread(File, Pos, 4) of
        {ok, <<"RIFF">>} ->
            avilist;
        {ok, <<"LIST">>} ->
            avilist;
        {ok, _FourCC} ->
            avichunk;
        eof ->
            eof
    end.

Walk is straightforward, from position From to To, accumulating result in reverse order (I love this [head|tail] list notation … was Prolog the first to use it?). chunk_or_list read few bytes (the FourCC header) to guess the kind of the next structure (CHUNK or LIST) in file; this structure is loaded, and walk continue.


walk_list(Level, File, From, To) ->
    case read_list_header(File, From) of
        {ok, AviList={avilist, List, FourCC, DataPos, DataSize}, NextPos} ->
            dbg(Level, "read list header (pos=~p, next=~p): List=~p FourCC=~p~n", [From, NextPos, List, FourCC]),
            {ok, SubPart} = case FourCC of
                <<"movi">> ->
                    dbg(Level, "... skipping list FourCC=~p...~n", [FourCC]),
                    {ok, []};
                _ ->
                    walk_data(Level + 1, [], File, DataPos, DataPos + DataSize)
            end,
            {ok, {AviList, SubPart}, NextPos};
        eof ->
            dbg(Level, "end of file~n", []),
            eof
    end.

read_list_header(File, Pos) ->
    case file:pread(File, [{Pos, 4}, {Pos + 4, 4}, {Pos + 8, 4}]) of
        {ok, [List, <<Size:4/little-unsigned-integer-unit:8>>, FourCC]} ->
            {ok, {avilist, List, FourCC, Pos + 12, Size - 4}, Pos + 8 + Size};
        {ok, [eof, eof, eof]} ->
            eof;
        _ ->
            {error, "no list header to read, but not empty data~n"}
    end.

To walk a LIST, read the header (remember that the FourCC field length is part of the data size …), read the nested data (this re-use the walk_data), and return the LIST representation: a 2-tuple with first the header (could be a record) and then a list of sub parts. There is a useless test to not walk the real data because my test file is kind of big. Walking the CHUNK is quite the same.


walk_chunk(Level, File, From, To) ->
    case read_chunk_header(File, From) of
        {ok, AviChunk={avichunk, FourCC, DataPos, DataSize}, NextPos} ->
            %FourCC = <<_StreamNumber:2/binary, _DataType:2/binary>>},
            dbg(Level,  "read chunk header (pos=~p, next=~p): FourCC=~p DataSize=~p~n", [From, NextPos, FourCC, DataSize]),
            chunk_spy(FourCC, File, DataPos, DataSize),
            {ok, AviChunk, NextPos};
        eof ->
            dbg(Level, "end of file~n", []),
            eof
    end.

read_chunk_header(File, Pos) ->
    case file:pread(File, [{Pos, 4}, {Pos + 4, 4}]) of
        {ok, [FourCC, <<Size:4/little-unsigned-integer-unit:8>>]} ->
            NextPos = Pos + 8 + Size,
            PaddedNextPos = NextPos + (NextPos rem 2),
            {ok, {avichunk, FourCC, Pos + 8, Size}, PaddedNextPos};
        {ok, [eof, eof]} ->
            eof;
        _ ->
            {error, "no chunk header to read, but not empty data~n"}
    end.

Similar to LIST, without nested data. Also, this went wrong at the first attempt: I found in this page that CHUNK data is padded to word boundary (grr).

But that’s all it take to read a well formated RIFF file. And for those wondering about the chunk_spy function, continue to read this blog :).

I’m probably missing something in Erlang/OTP installation from sources … my goal is to install multiple version of OTP in user space.

Here is one way to do it:

  1. remove any symlink if needed
    rm /home/khigia/local/lib/erlang
  2. apply the “normal” quick install steps for the new OTP version, let say otp-R11B5
    cd /home/khigia/apps/otp-src-R11B5; ./configure --prefix=/home/khigia/local ; make; make install
  3. modify the install to be version specific
    mv /home/khigia/local/lib/erlang /home/khigia/local/lib/otp-R11B5
    ln -s /home/khigia/local/lib/otp-R11B5 /home/khigia/local/lib/erlang

Changing the symlink enable to switch Erlang/OTP version. There must be an easiest way … how?

Next Page »