Erlang非业余研究

July 05, 2010
ProcessOne has just released OneCached, a Memcached server and client implementation written in Erlang.
OneCached is released under the Erlang Public License (EPL),.

OneCached supports the set, add, replace, get, incr, decr, delete, flush_all and quit commands. It doesn’t handle expiration time.

You call pull the source code from the public repository at: https://git.process-one.net/onecached

git clone git://git.process-one.net/onecached/mainline.git

To compile, just run make, and to start, just type:
bin/onecachedctl start

这个项目是非常简单的一个memcached的模仿， 亮点在于底层用ets, dest, mnesia这样的erlang特有的存储。这个项目的意义在于演示了如何编写一个严肃的Erlang工程，以及需要掌握的技能。对于初学者来讲，这是很好的一个切入点。 你可以在上面修修改改， 打造出你自己的版本。

玩的开心！

准备为测试部门的同学作个Tsung的讲座, 点击下载Tsung

需要进一步了解Tsung的同学 点这里看官方的文档!
想了解如何使用的还可以到Erlang china这里看看!

公司培训用的文档, 对于Erlang的VM会有个大体的认识, 方便设计和使用Erlang.
点解下载pdf格式的文档

我们在编码的时候, 通常会好奇, 这时候需要观察erl源码生成的VM opcode. Erlang的VM是register based的VM, 生产的opcode很容易理解.
生成汇编格式有2种方式:
1. 从源码生成抽象码. erlc +”‘S’” mod.erl, 生成mod.S
2. 从beam生成Opcode. 未公开的功能. erts_debug:df 参数M或者 M, F, 生成mod.dis

来吧,实践下:

root@ubuntu:~/exam# ls
eg.erl
root@ubuntu:~/exam# erlc +"'S'" eg.erl
root@ubuntu:~/exam# erlc eg.erl
root@ubuntu:~/exam# erl
Erlang R14A (erts-5.8) [source] [smp:2:2] [rq:2] [async-threads:0] [hipe] [kernel-poll:false] [lock-counting]

Eshell V5.8  (abort with ^G)
1>  erts_debug:df(eg).
ok
2>
User switch command
 --> q
root@ubuntu:~/exam# ls eg*
eg.beam  eg.dis  eg.erl  eg.S

我们得到了eg.S, eg.dis这2个反汇编的结果. 我们再来参观下.
先看源码:
root@ubuntu:~/exam# cat eg.erl

-module(eg).
-import(lists).
-import(lists,[sum/1]).
-compile(export_all).

kilo_byte() ->
    kilo_byte(10, [42]).
kilo_byte(0, Acc) ->
    Acc;
kilo_byte(N, Acc) ->
    kilo_byte(N-1, [Acc|Acc]).

loop()->
    sum(lists:seq(1,100)),
    loop().

中间汇编码, 供transform进行处理和编译器进一步生成opcode.
root@ubuntu:~/exam# cat eg.S

{module, eg}.  %% version = 0

{exports, [{kilo_byte,0},
           {kilo_byte,2},
           {loop,0},
           {module_info,0},
           {module_info,1}]}.

{attributes, []}.

{labels, 12}.

{function, kilo_byte, 0, 2}.
  {label,1}.
    {func_info,{atom,eg},{atom,kilo_byte},0}.
  {label,2}.
    {move,{literal,"*"},{x,1}}.
    {move,{integer,10},{x,0}}.
    {call_only,2,{f,4}}.

{function, kilo_byte, 2, 4}.
  {label,3}.
    {func_info,{atom,eg},{atom,kilo_byte},2}.
  {label,4}.
    {test,is_eq_exact,{f,5},[{x,0},{integer,0}]}.
    {move,{x,1},{x,0}}.
    return.
  {label,5}.
    {gc_bif,'-',{f,0},2,[{x,0},{integer,1}],{x,0}}.
    {test_heap,2,2}.
    {put_list,{x,1},{x,1},{x,1}}.
    {call_only,2,{f,4}}.

{function, loop, 0, 7}.
  {label,6}.
    {func_info,{atom,eg},{atom,loop},0}.
  {label,7}.
    {allocate,0,0}.
    {move,{integer,100},{x,1}}.
    {move,{integer,1},{x,0}}.
    {call_ext,2,{extfunc,lists,seq,2}}.
    {call_ext,1,{extfunc,lists,sum,1}}.
    {call_last,0,{f,7},0}.

{function, module_info, 0, 9}.
  {label,8}.
    {func_info,{atom,eg},{atom,module_info},0}.
  {label,9}.
    {move,{atom,eg},{x,0}}.
    {call_ext_only,1,{extfunc,erlang,get_module_info,1}}.

{function, module_info, 1, 11}.
  {label,10}.
    {func_info,{atom,eg},{atom,module_info},1}.
  {label,11}.
    {move,{x,0},{x,1}}.
    {move,{atom,eg},{x,0}}.
    {call_ext_only,2,{extfunc,erlang,get_module_info,2}}.

VM opcode形式, VM就是来解释运行这些code的

root@ubuntu:~/exam# cat eg.dis
B5146074: i_func_info_IaaI 0 eg kilo_byte 0
B5146088: move_cx "*" x(1)
B5146094: i_move_call_only_fcr eg:kilo_byte/2 10 x(0) 

B51460A0: i_func_info_IaaI 0 eg kilo_byte 2
B51460B4: i_is_eq_immed_frc f(B51460C8) x(0) 0
B51460C0: move_return_xr x(1) x(0)
B51460C8: i_fetch_rc x(0) 1
B51460D0: i_minus_jId j(00000000) 2 x(0)
B51460E0: test_heap_II 2 2
B51460EC: put_list_xxx x(1) x(1) x(1)
B51460F4: i_call_only_f eg:kilo_byte/2 

B51460FC: i_func_info_IaaI 0 eg loop 0
B5146110: allocate_tt 0 0
B5146118: move_cx 100 x(1)
B5146124: i_move_call_ext_cre 1 x(0) lists:seq/2
B5146130: i_call_ext_e lists:sum/1
B5146138: i_call_last_fP eg:loop/0 0 

B5146144: i_func_info_IaaI 0 eg module_info 0
B5146158: move_cr eg x(0)
B5146160: allocate_tt 0 1
B5146168: call_bif1_e erlang:get_module_info/1
B5146170: deallocate_return_P 0 

B5146178: i_func_info_IaaI 0 eg module_info 1
B514618C: move_rx x(0) x(1)
B5146194: move_cr eg x(0)
B514619C: allocate_tt 0 2
B51461A4: call_bif2_e erlang:get_module_info/2
B51461AC: deallocate_return_P 0

收工!

我们通常在使用port的时候, 需要把他同其他的上下文关联起来, 以便在port给我们发生数据的时候, 我们能根据绑定的上下文, 知道如何处理数据.
有2种办法:
1. 用ets来保存{Port, Ctx},这个比较慢, 每次都要查表.
2. 用Port本身的空间来保存Ctx. erlang:port_set_data 和erlang:port_get_data就是干这类事情的, 一步到位, 多核free.

不啰嗦上代码:

root@ubuntu:~# echo test >> test.dat
root@ubuntu:~# erl
Erlang R14A (erts-5.8) [source] [smp:2:2] [rq:2] [async-threads:0] [hipe] [kernel-poll:false] [lock-counting]

Eshell V5.8  (abort with ^G)
1>  {ok,{_,_,{Port,_}}} = file:open("test.dat", [read,raw]).
{ok,{file_descriptor,prim_file,{#Port<0.498>,7}}}
2> erlang:port_info(Port).
[{name,"efile"},
 {links,[<0.31.0>]},
 {id,498},
 {connected,<0.31.0>},
 {input,9},
 {output,11}]
3> erlang:port_set_data(Port, abcdefg).
true
4>  erlang:port_get_data(Port).
abcdefg

注意: gen_tcp和gen_udp等的port_data已经被使用了.

收工!

参考Talk http://www.erlang-factory.com/upload/presentations/238/ErlangFView postactorySFBay2010-KennethLundin.pdf
摘抄我感兴趣的:

R14要实现的:

1. June 16: R14A, a beta release
2. Multi-core performance improvements

• – optimized rwlocks
• – delayed deallocation
• – ”lock-free” process table

3. NIF improvements (Native Implemented Function)

• – sending messages from a NIF
• – crypto application as NIFs, now using a driver.

4. search in binaries (as of EEP-31)

• – new module called binary with functions: match, matches, split, replace, longest_common_prefix, … part, at, copy, first,last

5. new SSL ready to replace old SSL
6. improved Reltool

• – for easy creation of minimal target systems and standalone deployments from an installed development system.

7. Tentative

• –Parameterized modules officially supported and with more efficient implementation.

Longer term plans:
1. More multi core performance improvements

• – lock free pre-allocators (thread specific pre allocated buffers)
• – Scheduler specific mseg_alloc reducing lock contention and necessary for future NUMA optimizations. Intel Nehalem, AMD Opteron

2. Clustered shared heap or other solution to allow parallell computing on large sets of data avoiding copying.
3 . New XML-schema/dtd validator complementing the XML SAX parser we already have.
4. SMP optimizations in existing applications (Mnesia, ASN.1 …)
5. Improved eprof profiler

抄书:otp_doc_html_R13B04/lib/tools-2.6.5.1/doc/html /lcnt_chapter.html#id2252207

lcnt – The Lock Profiler

Internally in the Erlang runtime system locks are used to protect resources from being updated from multiple threads in a fatal way. Locks are necessary to ensure that the runtime system works properly but it also introduces a couple of limitations. Lock contention and locking overhead.

With lock contention we mean when one thread locks a resource and another thread, or threads, tries to acquire the same resource at the same time. The lock will deny the other thread access to the resource and the thread will be blocked from continuing its execution. The second thread has to wait until the first thread has completed its access to the resource and unlocked it. The lcnt tool measures these lock conflicts.

Locks has an inherent cost in execution time and memory space. It takes time initialize, destroy, aquiring or releasing locks. To decrease lock contention it some times necessary to use finer grained locking strategies. This will usually also increase the locking overhead and hence there is a tradeoff between lock contention and overhead. In general, lock contention increases with the number of threads running concurrently. The lcnt tool does not measure locking overhead.

5.1 Enabling lock-counting

For investigation of locks in the emulator we use an internal tool called lcnt (short for lock-count). The VM needs to be compiled with this option enabled. To enable this, use:

cd $ERL_TOP
./configure --enable-lock-counter

Another way to enable this alongside a normal VM is to compile it at emulator directory level, much like a debug build. To compile it this way do the following,

cd $ERL_TOP/erts/emulator
make lcnt FLAVOR=smp

and then starting Erlang with,

$ERL_TOP/bin/cerl -lcnt

To verify that you lock-counting enabled check that [lock-counting] appears in the status text when the VM is started.

Erlang R13B03 (erts-5.7.4) [/source] [64-bit] [smp:8:8] [rq:8] [async-threads:0] [hipe]
[kernel-poll:false] [lock-counting]

5.2 Getting started

Once you have a lock counting enabled VM the module lcnt can be used. The module is intended to be used from the current running nodes shell. To access remote nodes use lcnt:clear(Node) and lcnt:collect(Node).

All locks are continuously monitored and its statistics updated. Use lcnt:clear/0 to initially clear all counters before running any specific tests. This command will also reset the duration timer internally.

To retrieve lock statistics information use, lcnt:collect/0,1. The collect operation will start a lcnt server if it not already started. All collected data will be built into an erlang term and uploaded to the server and a duration time will also be uploaded. This duration is the time between lcnt:clear/0,1 and lcnt:collect/0,1.

Once the data is collected to the server it can be filtered, sorted and printed in many different ways.

See the reference manual for a description of each function.