系统技术非业余研究

以下是会议议程安排。更多信息访问： http://ecug.org/
2009-11-6

* 讲师入住酒店
* 会务人员&讲师小聚

2009-11-7

08:30 ~ 09:00 签到
09:00 ~ 09:15 开场
09:15 ~ 10:15 成立涛 – Erlang开发实践
10:15 ~ 10:30 提问&交流
10:30 ~ 11:30 余锋 – Erlang系统调优
11:30 ~ 11:45 提问&交流
11:45 ~ 12:00 集体拍照留念

12:00 ~ 13:00 午餐
13:00 ~ 13:30 休息

13:30 ~ 14:30 范赟、周琦 – 恶狼战役及社区养成
14:30 ~ 14:45 提问&交流
14:45 ~ 15:45 Stewart Mackenzie – An Erlang Implementation of Restms
15:45 ~ 16:00 提问&交流
16:00 ~ 16:15 茶歇（水果）
16:15 ~ 17:15 金尹 – CUDA Programming，一次异域之旅
17:15 ~ 17:30 提问&交流

17:30 ~ 18:30 晚餐
18:30 ~ 19:00 休息
19:00 ~ 20:00 李杰、崔博 – ac_actor: Erlang进程模型在C++中的实践
20:00 ~ 21:00 许式伟 – CERL: 谈谈“Boost.ASIO、Erlang与服务器编程”

21:00 全天会议结束

2009-11-8

08:30 ~ 09:30 方焜平 – 利用已有单机程序组建分布式模型的分析与实例
09:30 ~ 09:45 提问&交流
09:45 ~ 10:45 侯明园 – 基于Erlang实现的MMO服务器连接管理服务
10:45 ~ 11:00 提问&交流（上水果）
11:00 ~ 12:00 沙龙。开放式讨论（社区建设、技术走向等等）。

12:00 ~ 13:00 午餐
13:00 ~ 13:30 休息

13:30 ~ 14:30 周爱民 – 谈谈erlang网络环境下的几种数据流转形式
14:30 ~ 14:45 提问&交流
14:45 ~ 15:45 吴峥涛 – XEngine介绍
15:45 ~ 16:00 提问&交流

16:00 会议结束

erlang的代码运行有2种方式 1. 编译成opcode 2. eval. 这2种形式都需要把erl代码翻译成abstract code, 然后在不同的途径分别开.

通常情况下我们在eshell 里面输入的东西都是动态eval的. erl还支持命令行动态解释. erl -eval

看下init.erl

start_it({eval,Bin}) ->
    Str = binary_to_list(Bin),
    {ok,Ts,_} = erl_scan:string(Str),
    Ts1 = case reverse(Ts) of
              [{dot,_}|_] -> Ts;
              TsR -> reverse([{dot,1} | TsR])
          end,
    {ok,Expr} = erl_parse:parse_exprs(Ts1),
    erl_eval:exprs(Expr, []),
    ok;

我们看到express先文法扫描, 确认以dot结尾, 然后进行语法分析,形成abstract code, 最后 erl_eval解释abstact code.
我们注意到 在erl_parse的时候 表达式是这样的 f()->expr. 这就解释了为什么在shell里面无法定义函数,无法定义module什么的.

io模块和shell模块都是以上面的形式来动态eval的. 这个特性很好用, 配合parse_transform, 我们可以很容易自己实现个DSL语言.

公司培训用的 凑合看吧 主要讲erlang系统的特点,分布集群以及mnesia的使用, 从比较高的角度来看erlang, 让你有了大体观.

Erlang高级原理和应用

原文地址：http://hyperstruct.net/2007/6/26/literal-xml-in-erlang-with-parse-transform-2

One of the things I dislike about Erlang is that it severely impairs bragging opportunities. Yesterday I wrote a module that allows writing literal XML in the source and have it parsed into Erlang structures at compile time—sort of like E4X minus the manipulation goodies at runtime (at least for now).

You write:

Doc = '<greeting>Hello!</greeting>',
io:format("~p~n", [Doc]).

And it prints…

{xmlElement,greeting,
            greeting,
            [],
            {xmlNamespace,[],[]},
            [],
            1,
            [],
            [{xmlText,[{greeting,1}],1,[],"Hello!",text}],
            [],
            "/tmp",
            undeclared}

In most languages I’m familiar with, this would have granted the author instant Yacc-demigod status. With Erlang… it was less than 40 LOC. Hardly something you’d wear at a party.

Anyway, this code owes everything to Philip’s writings. It also uses parse_transform/2, and “programmers are strongly advised not to engage in parse transformations and no support is offered for problems encountered”. So unless you, like me, are still at the kid-in-a-candy-shop stage of Erlang experience, think twice before using this in production, ok?

The code is here http://repo.hyperstruct.net/inline_xml/

前些天在erlang的源码里面闲逛的时候发现了 bin目录下的cerl，一看原来是个调试的高级货。

我之前写过一篇文章http://mryufeng.javaeye.com/blog/113852 如何调试erlang 但是这是土八路的方法， cerl才是现代化工业。

# This is a script to start Erlang/OTP for debugging. PATH is set to
# include this script so if slave nodes are started they will use this
# script as well.
#
# usage: cerl [ OPTIONS ] [ ARGS ]
#
# The OPTIONS are
#
# -rootdir $MYROOTDIR
# Run an installed emulator built from this source
# -debug Run debug compiled emulator
# -gdb Run the debug compiled emulator in emacs and gdb.
# You have to start beam in gdb using “run”.
# -break F Run the debug compiled emulator in emacs and gdb and set break.
# The session is started, i.e. “run” is already don for you.
# -xxgdb FIXME currently disabled
# -purify Run emulator compiled for purify
# -quantify Run emulator compiled for quantify
# -purecov Run emulator compiled for purecov
# -gcov Run emulator compiled for gcov
# -valgrind Run emulator compiled for valgrind
# -lcnt Run emulator compiled for lock counting
# -nox Unset the DISPLAY variable to disable us of X Windows
#

要使用cerl 我们最好准备个调试版本的erlang。R13B 修正了些编译错误,可以编译出debug版本的系统：./configure && make TYPE＝debug && make

这样就生成了个beam.debug的运行期。

我们要调试的时候 就可以在otp的binx目录下运行 cerl -debug -gdb -break main

这时候cerl自动会加载 emacs 启动 gud, 整个过程都是自动的。但是这个脚本有点小问题， gud模型下没有把源码和当前对应的调试对应起来。可以通过以下方式修正：

yu-fengdemacbook-2:bin yufeng$ diff cerl cerl2
284c284
<     exec $EMACS --eval "(progn (gdb \"gdb $EMU\") $gdbcmd)"
---
>     exec $EMACS --eval "(progn (gdb \"gdb --annotate=3  $EMU\") $gdbcmd)"

具体的操作和界面可以参照这篇文章：
http://www.nabble.com/printing-of-Eterm%27s-from-gdb-td19240493.html

在调试的时候 我们会希望查看 eterm的值，但是由于eterm的格式非常复杂， gdb的print什么的无法满足我们的需求。 otp开发团队于是开发出了一套方法来减轻我们的负担：

1. erts/etc/unix/etp-commands 这是gdb的脚本 包含了几十个etp方法，而且文档非常详细。

2. 源码里面的 pp， ps等函数， 这些函数是专门为gdb调试开发的。 可以用gdb的 call pp(xxx)来调用。

有了这些工具 调试和研究erts变成了一件很快乐的事情！

erlang的binary在这个网络程序里面占着非常重要的地位，所以otp团队采用了非常多的优化手段包括：

1. binary操作对应着 opcode
2. 根据生命期和作用，有4种类型的binary
3. hipe优化，把bs_操作直接翻译成asm指令
4. 编译器层面消除无必须的操作。

下面的例子就是演示4的特性：

yu-fengdemacbook-2:~ yufeng$ cat bin.erl

-module(bin).
-export([start/1]).

start(A)->
    B1= <<12>>,
    B2 = <<B1/binary, 5.0/float>>,
    B3= <<B2/binary,  "yes">>,
% B3的值是预先可以知道的， 无需一步步的构造

    <<"abcd", 3:32,B3:128/binary,_/binary>> = <<"abcd1234",A/binary,2:32, 8773:64, "a", 5.0/float>>.

％ 2:32, 8773:64, "a", 5.0/float 这些都是预先知道的 直接翻译成二进制流。

yu-fengdemacbook-2:~ yufeng$ erlc +"'S'" bin.erl
yu-fengdemacbook-2:~ yufeng$ cat bin.S

{module, bin}.  %% version = 0

{exports, [{module_info,0},{module_info,1},{start,1}]}.

{attributes, []}.

{labels, 8}.


{function, start, 1, 2}.
  {label,1}.
    {func_info,{atom,bin},{atom,start},1}.
  {label,2}.
    {move,{integer,0},{x,1}}.
    {gc_bif,byte_size,{f,0},2,[{x,0}],{x,2}}.
    {bs_add,{f,0},[{x,1},{x,2},1],{x,1}}.
    {bs_add,{f,0},[{x,1},{integer,29},1],{x,1}}.
    {bs_init2,{f,0},{x,1},0,1,{field_flags,[]},{x,1}}.
    {bs_put_string,8,{string,"abcd1234"}}.
    {bs_put_binary,{f,0},{atom,all},8,{field_flags,[unsigned,big]},{x,0}}.

％％ 一步到位
    {bs_put_string,21,
                   {string,[0,0,0,2,0,0,0,0,0,0,34,69,97,64,20,0,0,0,0,0,0]}}.

    {test,bs_start_match2,{f,3},[{x,1},2,0,{x,0}]}.
    {test,bs_match_string,{f,3},[{x,0},64,{string,[97,98,99,100,0,0,0,3]}]}.
％％  一步到位

    {test,bs_get_binary2,
          {f,3},
          [{x,0},
           2,
           {integer,128},
           8,
           {field_flags,[{anno,[8,{file,"./bin.erl"}]},unsigned,big]},
           {x,2}]}.
    {test,bs_skip_bits2,
          {f,3},
          [{x,0},
           {atom,all},
           8,
           {field_flags,[{anno,[8,{file,"./bin.erl"}]},unsigned,big]}]}.
    {test,is_eq_exact,
          {f,3},
          [{x,2},{literal,<<12,64,20,0,0,0,0,0,0,121,101,115>>}]}.
％％  一步到位

    {move,{x,1},{x,0}}.
    return.
  {label,3}.
    {badmatch,{x,1}}.


{function, module_info, 0, 5}.
  {label,4}.
    {func_info,{atom,bin},{atom,module_info},0}.
  {label,5}.
    {move,{atom,bin},{x,0}}.
    {call_ext_only,1,{extfunc,erlang,get_module_info,1}}.


{function, module_info, 1, 7}.
  {label,6}.
    {func_info,{atom,bin},{atom,module_info},1}.
  {label,7}.
    {move,{x,0},{x,1}}.
    {move,{atom,bin},{x,0}}.
    {call_ext_only,2,{extfunc,erlang,get_module_info,2}}.

所以我们在使用binary的时候， 尽可能的利用这个特性。

我们先看下cover模块的功能：

The module cover provides a set of functions for coverage analysis of Erlang programs, counting how many times each executable line of code is executed when a program is run.

那它是如何做到的呢？

它是这样实现的： cover一个模块的时候要先编译， 这个过程中， 根据模块的abstract code 里面的行号，在每个有效的语句前面插入一个 ets:update_couter() 语句，这样编译出来的模块运行的时候，我们就可以收集到每个有效行的运行信息。

我hack了下cover.erl：

yu-fengdemacbook-2:src yufeng$ diff cover.erl cover_orig.erl 
1242,1243c1242
<     {ok, Module, Binary} = compile:forms(Forms, [debug_info]),
<     io:format("abstract code: ~n~p~n", [get_abstract_code(Module, Binary)]),
---
>     {ok, Module, Binary} = compile:forms(Forms, []),

yu-fengdemacbook-2:~ yufeng$ cat float.erl

-module(float).

-export([new/1,update/3]).

new(N) ->
       hipe_bifs:bytearray(N*8,0).

update(Arr,N,Float) ->
       <<A1,A2,A3,A4,A5,A6,A7,A8>> = <<Float/float>>,
       Start=N*8,
       hipe_bifs:bytearray_update(Arr,Start,A1),
       hipe_bifs:bytearray_update(Arr,Start+1,A2),
       hipe_bifs:bytearray_update(Arr,Start+2,A3),
       hipe_bifs:bytearray_update(Arr,Start+3,A4),
       hipe_bifs:bytearray_update(Arr,Start+4,A5),
       hipe_bifs:bytearray_update(Arr,Start+5,A6),
       hipe_bifs:bytearray_update(Arr,Start+6,A7),
       hipe_bifs:bytearray_update(Arr,Start+7,A8).

yu-fengdemacbook-2:~ yufeng$ otp_src_R13B02-1/bin/erl
Erlang R13B02 (erts-5.7.3) [source][/source][/source] [smp:2:2] [rq:2] [async-threads:0] [kernel-poll:false]

Eshell V5.7.3  (abort with ^G)
1> cover:start().
{ok,<0.33.0>}
2> cover:compile(float).
abstract code:
{raw_abstract_v1,
    [{attribute,1,file,{"/Users/yufeng/float.erl",1}},
     {attribute,1,module,float},
     {attribute,3,export,[{new,1},{update,3}]},
     {function,5,new,1,
         [{clause,5,
              [{var,5,'N'}],
              [],
              [{call,0,
                   {remote,0,{atom,0,ets},{atom,0,update_counter}},
                   [{atom,0,cover_internal_data_table},
                    {tuple,0,
                        [{atom,0,bump},
                         {atom,0,float},
                         {atom,0,new},
                         {integer,0,1},
                         {integer,0,1},
                         {integer,0,6}]},
                    {integer,0,1}]},

%% 我们清楚的看到 ets:update_counter()的调用

              {call,6,
                   {remote,6,{atom,6,hipe_bifs},{atom,6,bytearray}},
                   [{op,6,'*',{var,6,'N'},{integer,6,8}},{integer,6,0}]}]}]},
     {function,8,update,3,
         [{clause,8,
              [{var,8,'Arr'},{var,8,'N'},{var,8,'Float'}],
              [],
              [{call,0,
                   {remote,0,{atom,0,ets},{atom,0,update_counter}},
                   [{atom,0,cover_internal_data_table},
                    {tuple,0,
                        [{atom,0,bump},
                         {atom,0,float},
                         {atom,0,update},
                         {integer,0,3},
                         {integer,0,1},
                         {integer,0,9}]},
                    {integer,0,1}]},
               {match,9,
                   {bin,9,
                       [{bin_element,9,{var,9,'A1'},default,default},
                        {bin_element,9,{var,9,'A2'},default,default},
                        {bin_element,9,{var,9,'A3'},default,default},
                        {bin_element,9,{var,9,'A4'},default,default},
                        {bin_element,9,{var,9,'A5'},default,default},
                        {bin_element,9,{var,9,'A6'},default,default},
                        {bin_element,9,{var,9,'A7'},default,default},
                        {bin_element,9,{var,9,'A8'},default,default}]},
                   {bin,9,[{bin_element,9,{var,9,'Float'},default,[float]}]}},
               {call,0,
                   {remote,0,{atom,0,ets},{atom,0,update_counter}},
                   [{atom,0,cover_internal_data_table},
                    {tuple,0,
                        [{atom,0,bump},
                         {atom,0,float},
                         {atom,0,update},
                         {integer,0,3},
                         {integer,0,1},
                         {integer,0,10}]},
                    {integer,0,1}]},
               {match,10,
                   {var,10,'Start'},
                   {op,10,'*',{var,10,'N'},{integer,10,8}}},
               {call,0,
                   {remote,0,{atom,0,ets},{atom,0,update_counter}},
                   [{atom,0,cover_internal_data_table},
                    {tuple,0,
                        [{atom,0,bump},
                         {atom,0,float},
                         {atom,0,update},
                         {integer,0,3},
                         {integer,0,1},
                         {integer,0,11}]},
                    {integer,0,1}]},
               {call,11,
                   {remote,11,{atom,11,hipe_bifs},{atom,11,bytearray_update}},
                   [{var,11,'Arr'},{var,11,'Start'},{var,11,'A1'}]},
               {call,0,
                   {remote,0,{atom,0,ets},{atom,0,update_counter}},
                   [{atom,0,cover_internal_data_table},
                    {tuple,0,
                        [{atom,0,bump},
                         {atom,0,float},
                         {atom,0,update},
                         {integer,0,3},
                         {integer,0,1},
                         {integer,0,12}]},
                    {integer,0,1}]},
               {call,12,
                   {remote,12,{atom,12,hipe_bifs},{atom,12,bytearray_update}},
                   [{var,12,'Arr'},
                    {op,12,'+',{var,12,'Start'},{integer,12,1}},
                    {var,12,'A2'}]},
               {call,0,
                   {remote,0,{atom,0,ets},{atom,0,update_counter}},
                   [{atom,0,cover_internal_data_table},
                    {tuple,0,
                        [{atom,0,bump},
                         {atom,0,float},
                         {atom,0,update},
                         {integer,0,3},
                         {integer,0,1},
                         {integer,0,13}]},
                    {integer,0,1}]},
               {call,13,
                   {remote,13,{atom,13,hipe_bifs},{atom,13,bytearray_update}},
                   [{var,13,'Arr'},
                    {op,13,'+',{var,13,'Start'},{integer,13,2}},
                    {var,13,'A3'}]},
               {call,0,
                   {remote,0,{atom,0,ets},{atom,0,update_counter}},
                   [{atom,0,cover_internal_data_table},
                    {tuple,0,
                        [{atom,0,bump},
                         {atom,0,float},
                         {atom,0,update},
                         {integer,0,3},
                         {integer,0,1},
                         {integer,0,14}]},
                    {integer,0,1}]},
               {call,14,
                   {remote,14,{atom,14,hipe_bifs},{atom,14,bytearray_update}},
                   [{var,14,'Arr'},
                    {op,14,'+',{var,14,'Start'},{integer,14,3}},
                    {var,14,'A4'}]},
               {call,0,
                   {remote,0,{atom,0,ets},{atom,0,update_counter}},
                   [{atom,0,cover_internal_data_table},
                    {tuple,0,
                        [{atom,0,bump},
                         {atom,0,float},
                         {atom,0,update},
                         {integer,0,3},
                         {integer,0,1},
                         {integer,0,15}]},
                    {integer,0,1}]},
               {call,15,
                   {remote,15,{atom,15,hipe_bifs},{atom,15,bytearray_update}},
                   [{var,15,'Arr'},
                    {op,15,'+',{var,15,'Start'},{integer,15,4}},
                    {var,15,'A5'}]},
               {call,0,
                   {remote,0,{atom,0,ets},{atom,0,update_counter}},
                   [{atom,0,cover_internal_data_table},
                    {tuple,0,
                        [{atom,0,bump},
                         {atom,0,float},
                         {atom,0,update},
                         {integer,0,3},
                         {integer,0,1},
                         {integer,0,16}]},
                    {integer,0,1}]},
               {call,16,
                   {remote,16,{atom,16,hipe_bifs},{atom,16,bytearray_update}},
                   [{var,16,'Arr'},
                    {op,16,'+',{var,16,'Start'},{integer,16,5}},
                    {var,16,'A6'}]},
               {call,0,
                   {remote,0,{atom,0,ets},{atom,0,update_counter}},
                   [{atom,0,cover_internal_data_table},
                    {tuple,0,
                        [{atom,0,bump},
                         {atom,0,float},
                         {atom,0,update},
                         {integer,0,3},
                         {integer,0,1},
                         {integer,0,17}]},
                    {integer,0,1}]},
               {call,17,
                   {remote,17,{atom,17,hipe_bifs},{atom,17,bytearray_update}},
                   [{var,17,'Arr'},
                    {op,17,'+',{var,17,'Start'},{integer,17,6}},
                    {var,17,'A7'}]},
               {call,0,
                   {remote,0,{atom,0,ets},{atom,0,update_counter}},
                   [{atom,0,cover_internal_data_table},
                    {tuple,0,
                        [{atom,0,bump},
                         {atom,0,float},
                         {atom,0,update},
                         {integer,0,3},
                         {integer,0,1},
                         {integer,0,18}]},
                    {integer,0,1}]},
               {call,18,
                   {remote,18,{atom,18,hipe_bifs},{atom,18,bytearray_update}},
                   [{var,18,'Arr'},
                    {op,18,'+',{var,18,'Start'},{integer,18,7}},
                    {var,18,'A8'}]}]}]},
     {eof,19}]}
{ok,float}

这个故事告诉我们对 erlang系统的跟踪除了trace机制以为， 我们还可以用parse transform在编译的时候加入我们想要的代码，达到跟踪， 了解系统的目的。具体的可以参考 compiler模块 parse_transform的文档。