系统技术非业余研究

Erlang实现的网络服务器性能是非常高的，一个典型的服务器比如proxy我们可以处理40万个包的进出，链接数在万级别的。当然这么高的网络能力和底层的epoll实现有很大关系。那么通常我们的gen_tcp收到内核协议栈过来完整的封包的时候，有三种方式可以通知到我们，参见inet:setopts文档

{active, true | false | once}
If the value is true, which is the default, everything received from the socket will be sent as messages to the receiving process. If the value is false (passive mode), the process must explicitly receive incoming data by calling gen_tcp:recv/2,3 or gen_udp:recv/2,3 (depending on the type of socket).

If the value is once ({active, once}), one data message from the socket will be sent to the process. To receive one more message, setopts/2 must be called again with the {active, once} option.

When using {active, once}, the socket changes behaviour automatically when data is received. This can sometimes be confusing in combination with connection oriented sockets (i.e. gen_tcp) as a socket with {active, false} behaviour reports closing differently than a socket with {active, true} behaviour. To make programming easier, a socket where the peer closed and this was detected while in {active, false} mode, will still generate the message {tcp_closed,Socket} when set to {active, once} or {active, true} mode. It is therefore safe to assume that the message {tcp_closed,Socket}, possibly followed by socket port termination (depending on the exit_on_close option) will eventually appear when a socket changes back and forth between {active, true} and {active, false} mode. However, when peer closing is detected is all up to the underlying TCP/IP stack and protocol.

Note that {active,true} mode provides no flow control; a fast sender could easily overflow the receiver with incoming messages. Use active mode only if your high-level protocol provides its own flow control (for instance, acknowledging received messages) or the amount of data exchanged is small. {active,false} mode or use of the {active, once} mode provides flow control; the other side will not be able send faster than the receiver can read.

效率最高的当然是{active, true}方式，因为这种实现一个链接只一次epoll_ctl把socket的读事件挂上去，但是这种方式有致命的缺点。因为收到的包是通过消息的方式来通知我们的，完全是异步的。在正常情况下，没啥问题，但是如果我们的服务面对互联网就有很大的风险，如果遭受攻击的时候，对端发送大量的数据包的时候，我们的系统就会异步收到大量的消息，可能会超过我们的进程处理能力。最要命的是，我们无法让包停止下来，最后的结局就是我们的服务器因为缺少内存crash了。所以在实践中，我们都会用{active,once}方式来控制包的接收频率，这样避免了安全的问题，但是带来了性能的问题。每次设定{active,once}都意味着调用一次epoll_ctl。 如果strace我们的程序会发现有大量的epoll_ctl调用，基本上每秒达到QPS的数量。还有个问题也加剧了这个性能退化：erlang只有一个线程会收割epoll_wait事件，如果大量的ctl时间阻塞了事件的收割，网络处理的能力会大大下降。未来的版本官方计划会支持多个线程收割，但是现在还不行。

所以现在问题就来了，性能和安全如何平衡。Erlang出手拯救我们了，见这里：

inet driver add {active,N} socket option for TCP, UDP, and SCTP

这个功能在版本R16b03可用。

解决问题的思路很简单：
{active, true}有安全问题， {active, once}太慢， {active,N}我们一次设定来收N个消息包，摊薄epoll_ctl的代价，这样就可以大大缓解性能的压力。
Read more…

gen_tcp:unrecv是个未公开的函数，作用是往tcp的接收缓冲区里面填入指定的数据。别看这小小的函数，用起来很舒服的。
我们先看下它的代码实现，Erlang代码部分：

%%gen_tcp.erl:L299
unrecv(S, Data) when is_port(S) ->
    case inet_db:lookup_socket(S) of
        {ok, Mod} ->
            Mod:unrecv(S, Data);
        Error ->
            Error
    end.
%%inet_tcp.erl:L58 
unrecv(Socket, Data) -> prim_inet:unrecv(Socket, Data).

%%prim_inet.erl:L983
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%                                              
%%                                                                                                                          
%% UNRECV(insock(), data) -> ok | {error, Reason}                                                                           
%%                                                                                                                          
%%                                                                                                                          
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%                                              
unrecv(S, Data) ->
    case ctl_cmd(S, ?TCP_REQ_UNRECV, Data) of
        {ok, _} -> ok;
        Error  -> Error
    end.

运行期c代码部分:
Read more…

前天有同学在玩erlang gen_tcp的时候碰到了点小麻烦，描述如下：

比如说连接到baidu.com,发个http请求，然后马上接收数据，发现接收出错，wireshark抓包发现数据都有往返发送，比较郁闷。

我把问题演示下：

 
$ erl
Erlang R14B03 (erts-5.8.4) [source] [64-bit] [smp:16:16] [rq:16] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.8.4  (abort with ^G)
1> {ok,Sock} = gen_tcp:connect("baidu.com", 80, []).
{ok,#Port<0.582>}
2> gen_tcp:send(Sock, "GET / HTTP/1.1\r\n\r\n").
ok
3> gen_tcp:recv(Sock,0).
{error,einval}

这个问题的根源在于gen_tcp默认的{active,true},也就是说当gen_tcp收到网络包的时候，默认是把报文发送给它的宿主进程。而gen_tcp:recv是用户主动去拉数据，这二个模式是互斥的。

我们来看下代码otp/erts/emulator/drivers/common/inet_drv.c:7462

case TCP_REQ_RECV: {
..
        if (desc->inet.active || (len != 8))
            return ctl_error(EINVAL, rbuf, rsize);
..

那就解释为什么 gen_tcp:recv(Sock,0)返回错误码{error,einval}。
同时我们来验证下，报文是以消息的方式发送的。

 
4> flush().
Shell got {tcp,#Port<0.582>,
               "HTTP/1.1 400 Bad Request\r\nDate: Fri, 01 Jul 2011 03:51:25 GMT\r\nServer: Apache\r\nConnection: Keep-Alive\r\nTransfer-Encoding: chunked\r\nContent-Type: text/html; charset=iso-8859-1\r\n\r\n127\r\n<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<HTML><HEAD>\n<TITLE>400 Bad Request</TITLE>\n</HEAD><BODY>\n<H1>Bad Request</H1>\nYour browser sent a request that this server could not understand.<P>\nclient sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /<P>\n</BODY></HTML>\n\r\n0\r\n\r\n"}
ok
5>

搞清楚了问题，那解决方案很简单，connect的时候把active模式设成{active,false}.
再来演示下：

$ erl
Erlang R14B03 (erts-5.8.4) [source] [64-bit] [smp:16:16] [rq:16] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.8.4  (abort with ^G)
1> {ok,Sock} = gen_tcp:connect("baidu.com", 80, [{active,false}]).
{ok,#Port<0.582>}
2> gen_tcp:send(Sock, "GET / HTTP/1.1\r\n\r\n").
ok
3>  gen_tcp:recv(Sock,0).
{ok,"HTTP/1.1 400 Bad Request\r\nDate: Fri, 01 Jul 2011 05:25:15 GMT\r\nServer: Apache\r\nConnection: Keep-Alive\r\nTransfer-Encoding: chunked\r\nContent-Type: text/html; charset=iso-8859-1\r\n\r\n127\r\n<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<HTML><HEAD>\n<TITLE>400 Bad Request</TITLE>\n</HEAD><BODY>\n<H1>Bad Request</H1>\nYour browser sent a request that this server could not understand.<P>\nclient sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /<P>\n</BODY></HTML>\n\r\n0\r\n\r\n"}
4> 

搞定！

玩得开心！