系统技术非业余研究

July 05, 2010
ProcessOne has just released OneCached, a Memcached server and client implementation written in Erlang.
OneCached is released under the Erlang Public License (EPL),.

OneCached supports the set, add, replace, get, incr, decr, delete, flush_all and quit commands. It doesn’t handle expiration time.

You call pull the source code from the public repository at: https://git.process-one.net/onecached

git clone git://git.process-one.net/onecached/mainline.git

To compile, just run make, and to start, just type:
bin/onecachedctl start

这个项目是非常简单的一个memcached的模仿， 亮点在于底层用ets, dest, mnesia这样的erlang特有的存储。这个项目的意义在于演示了如何编写一个严肃的Erlang工程，以及需要掌握的技能。对于初学者来讲，这是很好的一个切入点。 你可以在上面修修改改， 打造出你自己的版本。

玩的开心！

在即将发布的R14B版本中，包括了64位机器下用的Halfword版本。 我们知道在64位Linux下，指针和size_t都变成64位的， 而Erlang的基础数据结构Eterm都是用的和体系相同的数据结构，那么在64位机器下，虽然CPU运算的速度增加了，但是内存的访问速度不会随着CPU的加快而加快。反而是在64位系统下，比如Erlang的List，访问的内存加倍了，总体的速度下降了百分几十。

为了解决这个问题，Erlang的团队推出了halfword-emulator, 通过修改Etrem指针的长度， 来达到节省内存，加速的目的。
我们可以在编译的时候通过 configure –enable-halfword-emulator 来启用这个VM.

half word 64-bit Erlang VM有以下特性
– 4 Gbytes process heaps (in total)
–max size of Erlang term 4 Gbytes
– ets tables and binaries in separate space can utilize the full 64 bit address space

但是需要注意的是，这个patch推出的时间不长，可能存在风险。 在提高速度的同时，也有不稳定的风险。 用户需要自己去评估。

准备为测试部门的同学作个Tsung的讲座, 点击下载Tsung

需要进一步了解Tsung的同学 点这里看官方的文档!
想了解如何使用的还可以到Erlang china这里看看!

点击这里下载R14A
以下是这次发布的亮点，没有太大的性能改进， 主要是修理了很多BUG！

— HIGHLIGHTS ———————————————————-

OTP-8217 == erts stdlib compiler ==

The module binary from EEP31 (and EEP9) is implemented.

OTP-8485 == common_test ==

It is now possible for the user to provide specific callback
modules that handle test configuration data, so that data on
arbitray form can be accessed (e.g. by reading files or by
communicating with a configuration server process). Two
default callback modules have been introduced in Common Test:
ct_config_plain and ct_config_xml. The former is used to
handle the traditional Common Test configuration files (with
terms on key-value tuple form) and the latter to handle
configuration data on XML representation.

OTP-8555 == erts ==

New NIF features:

Send messages from a NIF, or from thread created by NIF, to
any local process (enif_send)

Store terms between NIF calls (enif_alloc_env,
enif_make_copy)

Create binary terms with user defined memory management
(enif_make_resource_binary)

And some incompatible changes made to the API. For more
information see the warning text in erl_nif(3).

OTP-8623 == compiler erts hipe stdlib ==

Receive statements that can only read out a newly created
reference are now specially optimized so that it will execute
in constant time regardless of the number of messages in the
receive queue for the process. That optimization will benefit
calls to gen_server:call(). (See gen:do_call/4 for an example
of a receive statement that will be optimized.)

OTP-8650 == common_test ==

The run_test script has been replaced by a program (with the
same name) which can be executed without explicit
installation. The start flags are the same as for the legacy
start script.

OTP-8706 == erts hipe tools ==

eprof has been reimplemented with support in the Erlang
virtual machine and is now both faster (i.e. slows down the
code being measured less) and scales much better. In
measurements we saw speed-ups compared to the old eprof
ranging from 6 times (for sequential code that only uses one
scheduler/core) up to 84 times (for parallel code that uses 8
cores).

Note: The API for the eprof has been cleaned up and extended.
See the documentation.

期待R14B有大的动作！

Erlang的binary数据结构非常强大，而且偏向底层，在作网络程序的时候，很方便的能够和二进制协议对应起来。但是由于这个数据结构加入erlang语言的时间不是很长，相关的配套模块不是很多。 在binary的匹配，替换，修改就显的非常麻烦。 于是有了EEP31 。 R14A昨天已经实现了这个功能， 在stdlib下添加了个binary模块。 这个模块大部分功能是由BIF实现的， 同时充分考虑了CPU使用的公平性，源码大部分在erl_bif_binary.c下。 还添加了个gurad函数： binary_part进一步方便我们写匹配条件。

我们在源码里面发现了以下注释：
/*
* The native implementation functions for the module binary.
* Searching is implemented using aither Boyer-More or Aho-Corasick
* depending on number of searchstrings (BM if one, AC if more than one).
* Native implementation is mostly for efficiency, nothing
* (except binary:referenced_byte_size) really *needs* to be implemented
* in native code.
*/

这个模块兼顾了效率和方便性，使用起来就大大简化了代码的复杂度，有福气了。

Erlang的惯用法之一就是在消息匹配的时候，如果需要唯一性，通常会通过make_ref搞个唯一的Ref来作为消息的一部分来匹配。这个惯用法用在gen_server:call或者demonitor这样的使用频度很高的函数里面。由于erlang的消息匹配是再消息队列里面挨个遍历来匹配，特别是消息队列特别长的时候，会有很大的性能瓶颈。于是新的优化出现了，以下是编译器beam_receive.erl里面的解释，写的很清楚：

%%%                                                                                                                                                                    
%%% In code such as:                                                                                                                                                   
%%%                                                                                                                                                                    
%%%    Ref = make_ref(),        %Or erlang:monitor(process, Pid)                                                                                                       
%%%      .                                                                                                                                                             
%%%      .                                                                                                                                                             
%%%      .                                                                                                                                                             
%%%    receive                                                                                                                                                         
%%%       {Ref,Reply} -> Reply                                                                                                                                         
%%%    end.                                                                                                                                                            
%%%                                                                                                                                                                    
%%% we know that none of the messages that exist in the message queue                                                                                                  
%%% before the call to make_ref/0 can be matched out in the receive                                                                                                    
%%% statement. Therefore we can avoid going through the entire message                                                                                                 
%%% queue if we introduce two new instructions (here written as                                                                                                        
%%% BIFs in pseudo-Erlang):                                                                                                                                            
%%%                                                                                                                                                                    
%%%    recv_mark(SomeUniqInteger),                                                                                                                                     
%%%    Ref = make_ref(),                                                                                                                                               
%%%      .                                                                                                                                                             
%%%      .                                                                                                                                                             
%%%      .                                                                                                                                                             
%%%    recv_set(SomeUniqInteger),                                                                                                                                      
%%%    receive                                                                                                                                                         
%%%       {Ref,Reply} -> Reply                                                                                                                                         
%%%    end.                                                                                                                                                            
%%%                                                                                                                                                                    
%%% The recv_mark/1 instruction will save the current position and                                                                                                     
%%% SomeUniqInteger in the process context. The recv_set                                                                                                               
%%% instruction will verify that SomeUniqInteger is still stored                                                                                                       
%%% in the process context. If it is, it will set the current pointer                                                                                                  
%%% for the message queue (the next message to be read out) to the                                                                                                     
%%% position that was saved by recv_mark/1.                                                                                                                            
%%%    
%%% The remove_message instruction must be modified toinvalidate                                                                                                      
%%% the information stored by the previous recv_mark/1, in case there                                                                                                  
%%% is another receive executed between the calls to recv_mark/1 and                                                                                                   
%%% recv_set/1.                                                                                                                                                        
%%%                                                                                                                                                                    
%%% We use a reference to a label (i.e. a position in the loaded code)                                                                                                 
%%% as the SomeUniqInteger.                                                                                                                                            
%%%                                                   

结论： 点滴优化成就高性能。

昨天听zhuzhaoyuan说的一个connect细节. 通常我们connect的时候都是非阻塞的, 在connect调用后把句柄挂到poll去, 等poll通知可写的时候, 我们就认为connect成功了. 但是在linux平台下实际上不一定成功, 具体的要用socket get_opt来检查下出错码来决定.

以下是从man 2 connnect摘抄的:

EINPROGRESS
The socket(2,7,n) is non-blocking and the connection cannot be com-
pleted immediately. It is possible to select(2,7,2 select_tut)(2) or poll(2) for
completion by selecting the socket(2,7,n) for writing. After select(2,7,2 select_tut)
indicates writability, use getsockopt(2) to read(2,n,1 builtins) the SO_ERROR
option at level SOL_SOCKET to determine whether connect com-
pleted successfully (SO_ERROR is zero) or unsuccessfully
(SO_ERROR is one of the usual error(8,n) codes listed here, explain-
ing the reason for the failure).

我们看下erlang的inet_drv是如何处理的.
./erts/emulator/drivers/common/inet_drv.c:

        {
            int error = 0;      /* Has to be initiated, we check it */
            unsigned int sz = sizeof(error); /* even if we get -1 */
            int code = sock_getopt(desc->inet.s, SOL_SOCKET, SO_ERROR,
                                   (void *)&error, &sz);

            if ((code < 0) || error) {
                desc->inet.state = TCP_STATE_BOUND;  /* restore state */
                ret = async_error(INETP(desc), error);
                goto done;
            }
        }

结论: 细节决定品质.