Archive

Posts Tagged ‘gen_tcp’

inet驱动新增加{active,N} socket选项

November 3rd, 2013 5 comments

原创文章,转载请注明: 转载自系统技术非业余研究

本文链接地址: inet驱动新增加{active,N} socket选项

Erlang实现的网络服务器性能是非常高的,一个典型的服务器比如proxy我们可以处理40万个包的进出,链接数在万级别的。当然这么高的网络能力和底层的epoll实现有很大关系。那么通常我们的gen_tcp收到内核协议栈过来完整的封包的时候,有三种方式可以通知到我们,参见inet:setopts文档

{active, true | false | once}
If the value is true, which is the default, everything received from the socket will be sent as messages to the receiving process. If the value is false (passive mode), the process must explicitly receive incoming data by calling gen_tcp:recv/2,3 or gen_udp:recv/2,3 (depending on the type of socket).

If the value is once ({active, once}), one data message from the socket will be sent to the process. To receive one more message, setopts/2 must be called again with the {active, once} option.

When using {active, once}, the socket changes behaviour automatically when data is received. This can sometimes be confusing in combination with connection oriented sockets (i.e. gen_tcp) as a socket with {active, false} behaviour reports closing differently than a socket with {active, true} behaviour. To make programming easier, a socket where the peer closed and this was detected while in {active, false} mode, will still generate the message {tcp_closed,Socket} when set to {active, once} or {active, true} mode. It is therefore safe to assume that the message {tcp_closed,Socket}, possibly followed by socket port termination (depending on the exit_on_close option) will eventually appear when a socket changes back and forth between {active, true} and {active, false} mode. However, when peer closing is detected is all up to the underlying TCP/IP stack and protocol.

Note that {active,true} mode provides no flow control; a fast sender could easily overflow the receiver with incoming messages. Use active mode only if your high-level protocol provides its own flow control (for instance, acknowledging received messages) or the amount of data exchanged is small. {active,false} mode or use of the {active, once} mode provides flow control; the other side will not be able send faster than the receiver can read.

效率最高的当然是{active, true}方式,因为这种实现一个链接只一次epoll_ctl把socket的读事件挂上去,但是这种方式有致命的缺点。因为收到的包是通过消息的方式来通知我们的,完全是异步的。在正常情况下,没啥问题,但是如果我们的服务面对互联网就有很大的风险,如果遭受攻击的时候,对端发送大量的数据包的时候,我们的系统就会异步收到大量的消息,可能会超过我们的进程处理能力。最要命的是,我们无法让包停止下来,最后的结局就是我们的服务器因为缺少内存crash了。所以在实践中,我们都会用{active,once}方式来控制包的接收频率,这样避免了安全的问题,但是带来了性能的问题。每次设定{active,once}都意味着调用一次epoll_ctl。 如果strace我们的程序会发现有大量的epoll_ctl调用,基本上每秒达到QPS的数量。还有个问题也加剧了这个性能退化:erlang只有一个线程会收割epoll_wait事件,如果大量的ctl时间阻塞了事件的收割,网络处理的能力会大大下降。未来的版本官方计划会支持多个线程收割,但是现在还不行。

所以现在问题就来了,性能和安全如何平衡。Erlang出手拯救我们了,见这里

inet driver add {active,N} socket option for TCP, UDP, and SCTP

这个功能在版本R16b03可用。

解决问题的思路很简单:
{active, true}有安全问题, {active, once}太慢, {active,N}我们一次设定来收N个消息包,摊薄epoll_ctl的代价,这样就可以大大缓解性能的压力。
Read more…

Post Footer automatically generated by wp-posturl plugin for wordpress.

gen_tcp发送缓冲区以及水位线问题分析

May 15th, 2013 7 comments

原创文章,转载请注明: 转载自系统技术非业余研究

本文链接地址: gen_tcp发送缓冲区以及水位线问题分析

前段时间有同学在线上问了个问题:

服务器端我是这样设的:gen_tcp:listen(8000, [{active, false}, {recbuf,1}, {buffer,1}]).
客户端是这样设的:gen_tcp:connect(“localhost”, 8000, [{active, false}, {high_watermark,2}, {low_watermark,1}, {sndbuf,1}, {buffer,1}]).
我客户端每次gen_tcp:send()发送一个字节,前6个字节返回ok,第7个字节阻塞
服务端每次gen_tcp:recv(_,0)接收一个字节,接收三个字节后,客户端的第7次发送返回。
按我的理解的话:应该是 服务器端可以接收2个字节+sndbuf里的一个字节,第4个字节客户端就该阻塞的,可事实不时这样,求分析

这个问题确实还是比较复杂,涉及到gen_tcp的发送缓冲区和接收缓冲区,水位线等问题,其中接收缓冲区的问题在这篇 以及这篇 博文里面讲的比较清楚了,今天我们重点来分析下发送缓冲区和水位线的问题。

在开始分析前,我们需要熟悉几个gen_tcp的选项, 更多参见 这里
Read more…

Post Footer automatically generated by wp-posturl plugin for wordpress.

gen_tcp连接半关闭问题

May 14th, 2013 No comments

原创文章,转载请注明: 转载自系统技术非业余研究

本文链接地址: gen_tcp连接半关闭问题

很久之前我发在javaeye论坛上,预防丢了抄过来:

原文:http://erlang.group.iteye.com/group/wiki/1422-gen_tcp-half-closed

当tcp对端调用shutdown(RD/WR) 时候, 宿主进程默认将收到{tcp_closed, Socket}消息, 如果这个行为不是你想要的,那么请看:

shutdown(Socket, How) -> ok | {error, Reason}
Types:
Socket = socket()
How = read | write | read_write
Reason = posix()

Immediately close a socket in one or two directions.
How == write means closing the socket for writing, reading from it is still possible.
To be able to handle that the peer has done a shutdown on the write side, the {exit_on_close, false} option is useful.

简单的设置inets:setopts(Socket, [{exit_on_close, false}]). 这样就不会被强制退出了。

祝玩得开心!

Post Footer automatically generated by wp-posturl plugin for wordpress.

Erlang gen_tcp相关问题汇编索引

May 14th, 2013 2 comments
Categories: Erlang探索, 源码分析 Tags: , ,

gen_tcp如何限制封包大小

May 14th, 2013 4 comments

原创文章,转载请注明: 转载自系统技术非业余研究

本文链接地址: gen_tcp如何限制封包大小

我们在做tcp服务器的时候,通常会从安全考虑,限制封包的大小,预防被无端攻击或者避免极端的请求对业务造成损害。
我们的tcp服务器通常是erlang做的,那么就涉及到gen_tcp如何限制封包的大小.

gen_tcp对封包的获取有2种方式:
1. {active, false} 封包透过gen_tcp:recv(Socket, Length) -> {ok, Packet} | {error, Reason} 来接收。
2. {active, true} 封包以消息方式投递。

对于第一种方式:gen_tcp:recv(Socket, Length) 我们开看下代码实现:
Read more…

Post Footer automatically generated by wp-posturl plugin for wordpress.

gen_tcp接收缓冲区易混淆概念纠正

May 14th, 2013 1 comment

原创文章,转载请注明: 转载自系统技术非业余研究

本文链接地址: gen_tcp接收缓冲区易混淆概念纠正

Erlang的每个TCP网络链接是由相应的gen_tcp对象来表示的,说白了就是个port, 实现Erlang网络相关的逻辑,其实现代码位于erts/emulator/drivers/common/inet_drv.c

参照inet:setopts文档,它有三个buffer相关的选项,非常让人费解:

{buffer, Size}
Determines the size of the user-level software buffer used by the driver. Not to be confused with sndbuf and recbuf options which correspond to the kernel socket buffers. It is recommended to have val(buffer) >= max(val(sndbuf),val(recbuf)). In fact, the val(buffer) is automatically set to the above maximum when sndbuf or recbuf values are set.

{recbuf, Size}
Gives the size of the receive buffer to use for the socket.

{sndbuf, Size}
Gives the size of the send buffer to use for the socket.

其中sndbuf, recbuf选项比较好理解, 就是设置gen_tcp所拥有的socket句柄的内核的发送和接收缓冲区,从代码可以验证:

/* inet_drv.c */
#define INET_OPT_SNDBUF     6   /* set send buffer size */
#define INET_OPT_RCVBUF     7   /* set receive buffer size */
static int inet_set_opts(inet_descriptor* desc, char* ptr, int len)
{
...
        case INET_OPT_SNDBUF:    type = SO_SNDBUF;
            DEBUGF(("inet_set_opts(%ld): s=%d, SO_SNDBUF=%d\r\n",
                    (long)desc->port, desc->s, ival));
            break;
        case INET_OPT_RCVBUF:    type = SO_RCVBUF;
            DEBUGF(("inet_set_opts(%ld): s=%d, SO_RCVBUF=%d\r\n",
                    (long)desc->port, desc->s, ival));
            break;
...
        res = sock_setopt           (desc->s, proto, type, arg_ptr, arg_sz);
...
}

那buffer是什么呢,他们三者之间的关系? 从文档的描述来看:
It is recommended to have val(buffer) >= max(val(sndbuf),val(recbuf)). In fact, the val(buffer) is automatically set to the above maximum when sndbuf or recbuf values are set.
Read more…

Post Footer automatically generated by wp-posturl plugin for wordpress.

未公开的gen_tcp:unrecv以及接收缓冲区行为分析

October 1st, 2011 2 comments

原创文章,转载请注明: 转载自系统技术非业余研究

本文链接地址: 未公开的gen_tcp:unrecv以及接收缓冲区行为分析

gen_tcp:unrecv是个未公开的函数,作用是往tcp的接收缓冲区里面填入指定的数据。别看这小小的函数,用起来很舒服的。
我们先看下它的代码实现,Erlang代码部分:

%%gen_tcp.erl:L299
unrecv(S, Data) when is_port(S) ->
    case inet_db:lookup_socket(S) of
        {ok, Mod} ->
            Mod:unrecv(S, Data);
        Error ->
            Error
    end.
%%inet_tcp.erl:L58 
unrecv(Socket, Data) -> prim_inet:unrecv(Socket, Data).

%%prim_inet.erl:L983
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%                                              
%%                                                                                                                          
%% UNRECV(insock(), data) -> ok | {error, Reason}                                                                           
%%                                                                                                                          
%%                                                                                                                          
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%                                              
unrecv(S, Data) ->
    case ctl_cmd(S, ?TCP_REQ_UNRECV, Data) of
        {ok, _} -> ok;
        Error  -> Error
    end.

运行期c代码部分:
Read more…

Post Footer automatically generated by wp-posturl plugin for wordpress.

Categories: Erlang探索 Tags: , ,