系统技术非业余研究

具体的可以参考这里

他的最大特点是快, 当你的系统有上万个tcp链接要了解的时候的时候, 你就知道我说什么了. netstat等常规工具变成废铁了, 这时候他的作用就非常明显了.

/proc interface is inadequate, unfortunately. When amount of sockets is enough large, netstat or even plain cat /proc/net/tcp/ cause nothing but pains and curses. In linux-2.4 the desease became worse: even if amount of sockets is small reading /proc/net/tcp/ is slow enough.

This utility presents a new approach, which is supposed to scale well. I am not going to describe technical details here and will concentrate on description of the command. The only important thing to say is that it is not so bad idea to load module tcp_diag, which can be found in directory Modules of iproute2. If you do not make this ss will work, but it falls back to /proc and becomes slow like netstat, well, a bit faster yet (see section “Some numbers”).

从技术上讲, 它主要是通过/proc来获取各种各样的统计信息. 特别是当处理tcp的时候用到了, tcp_diag功能. tcp_diag是tcp协议栈用于诊断和统计的一个模块, 用netfilter来获取第一手的信息的.  ss就用到了这个技术, 保证了信息的快捷获取.

这个软件是隶属于iproute包的,  如果你的系统没有的话可以这样安装:

apt-get  -y install iproute

用它可以了解下协议栈占用的内存, 每个链接的队列使用情况, 每个链接属于哪个进程.
而之前唯一能用的就是:

watch -n  1 'cat /proc/net/sockstat'

下面简单介绍下他的使用:

root@ubuntu:/usr/src/iproute-20090324/misc# ss --help
Usage: ss [ OPTIONS ]
ss [ OPTIONS ] [ FILTER ]
-h, --help           this message
-V, --version        output version information
-n, --numeric        don't resolve service names
-r, --resolve       resolve host names
-a, --all            display all sockets
-l, --listening      display listening sockets
-o, --options       show timer information
-e, --extended      show detailed socket information
-m, --memory        show socket memory usage
-p, --processes      show process using socket
-i, --info           show internal TCP information
-s, --summary        show socket usage summary

-4, --ipv4          display only IP version 4 sockets
-6, --ipv6          display only IP version 6 sockets
-0, --packet display PACKET sockets
-t, --tcp            display only TCP sockets
-u, --udp            display only UDP sockets
-d, --dccp           display only DCCP sockets
-w, --raw            display only RAW sockets
-x, --unix           display only Unix domain sockets
-f, --family=FAMILY display sockets of type FAMILY

-A, --query=QUERY
QUERY := {all|inet|tcp|udp|raw|unix|packet|netlink}[,QUERY]

-F, --filter=FILE   read filter information from FILE
FILTER := [ state TCP-STATE ] [ EXPRESSION ]

root@ubuntu:/usr/src/iproute-20090324/misc# ss -s
Total: 750 (kernel 761)
TCP:   21 (estab 7, closed 0, orphaned 0, synrecv 0, timewait 0/0), ports 0

Transport Total     IP        IPv6
*         761       -         -
RAW       0         0         0
UDP       3         3         0
TCP       21        18        3
INET      24        21        3
FRAG      0         0         0

root@ubuntu:/usr/src/iproute-20090324/misc# ss -t -m
State       Recv-Q Send-Q                                                                     Local Address:Port                                                                         Peer Address:Port
ESTAB       0      0                                                                        192.168.235.147:ssh                                                                       192.168.235.155:4251
mem:(r0,w0,f4096,t0)
CLOSE-WAIT  38     0                                                                        192.168.235.147:13910                                                                        174.36.30.67:https
mem:(r672,w0,f3424,t0)
ESTAB       0      0                                                                        192.168.235.147:ssh                                                                       192.168.235.155:4635
mem:(r0,w0,f4096,t0)
ESTAB       0      0                                                                              127.0.0.1:4369                                                                            127.0.0.1:54620
mem:(r0,w0,f0,t0)
ESTAB       0      0                                                                        192.168.235.147:ssh                                                                       192.168.235.155:1095
mem:(r0,w0,f4096,t0)
ESTAB       0      0                                                                              127.0.0.1:4369                                                                            127.0.0.1:44234
mem:(r0,w0,f0,t0)
ESTAB       0      0                                                                              127.0.0.1:54620                                                                           127.0.0.1:4369
mem:(r0,w0,f0,t0)
ESTAB       0      0                                                                              127.0.0.1:44234                                                                           127.0.0.1:4369
mem:(r0,w0,f0,t0)

root@ubuntu:/usr/src/iproute-20090324/misc# ss -p
State       Recv-Q Send-Q                                                                     Local Address:Port                                                                         Peer Address:Port
ESTAB       0      0                                                                        192.168.235.147:ssh                                                                       192.168.235.155:4251     users:(("sshd",8373,3))
CLOSE-WAIT  38     0                                                                        192.168.235.147:13910                                                                        174.36.30.67:https    users:(("dropbox",2674,13))
ESTAB       0      0                                                                        192.168.235.147:ssh                                                                       192.168.235.155:4635     users:(("sshd",350,3))
ESTAB       0      0                                                                              127.0.0.1:4369                                                                            127.0.0.1:54620    users:(("epmd",28338,5))
ESTAB       0      0                                                                        192.168.235.147:ssh                                                                       192.168.235.155:1095     users:(("sshd",29044,3))
ESTAB       0      0                                                                              127.0.0.1:4369                                                                            127.0.0.1:44234    users:(("epmd",28338,4))
ESTAB       0      0                                                                              127.0.0.1:54620                                                                           127.0.0.1:4369     users:(("beam.smp",25947,11))
ESTAB       0      0                                                                              127.0.0.1:44234                                                                           127.0.0.1:4369     users:(("inet_gethost",23783,8),("inet_gethost",23784,8))
root@ubuntu:/usr/src/iproute-20090324/misc# ss -i
State       Recv-Q Send-Q                                                                     Local Address:Port                                                                         Peer Address:Port
ESTAB       0      0                                                                        192.168.235.147:ssh                                                                       192.168.235.155:4251
cubic rto:396 rtt:124.5/66 ato:40 cwnd:4 send 375.3Kbps rcv_rtt:296456 rcv_space:66472
CLOSE-WAIT  38     0                                                                        192.168.235.147:13910                                                                        174.36.30.67:https
cubic wscale:7,9 rto:612 rtt:256/52 ato:40 cwnd:5 send 213.8Kbps rcv_rtt:259 rcv_space:5840
ESTAB       0      0                                                                        192.168.235.147:ssh                                                                       192.168.235.155:4635
cubic rto:356 rtt:144/33 ato:40 cwnd:4 send 324.4Kbps rcv_rtt:511258 rcv_space:46760
ESTAB       0      0                                                                              127.0.0.1:4369                                                                            127.0.0.1:54620
cubic wscale:9,9 rto:204 rtt:4/2 ato:40 cwnd:3 send 98.3Mbps rcv_space:32768
ESTAB       0      0                                                                        192.168.235.147:ssh                                                                       192.168.235.155:1095
cubic rto:232 rtt:21/18 ato:40 cwnd:5 ssthresh:4 send 2.8Mbps rcv_rtt:173516 rcv_space:118904
ESTAB       0      0                                                                              127.0.0.1:4369                                                                            127.0.0.1:44234
cubic wscale:9,9 rto:204 rtt:4/2 ato:40 cwnd:3 send 98.3Mbps rcv_space:32768
ESTAB       0      0                                                                              127.0.0.1:54620                                                                           127.0.0.1:4369
cubic wscale:9,9 rto:204 rtt:4/2 ato:40 cwnd:3 send 98.3Mbps rcv_space:32792
ESTAB       0      0                                                                              127.0.0.1:44234                                                                           127.0.0.1:4369
cubic wscale:9,9 rto:204 rtt:4/2 ato:40 cwnd:3 send 98.3Mbps rcv_space:32792

root@ubuntu:/usr/src/iproute-20090324/misc# ss -e
State       Recv-Q Send-Q                                                                     Local Address:Port                                                                         Peer Address:Port
ESTAB       0      0                                                                        192.168.235.147:ssh                                                                       192.168.235.155:4251     timer:(keepalive,50min,0) ino:1843366 sk:c2ea2600
CLOSE-WAIT  38     0                                                                        192.168.235.147:13910                                                                        174.36.30.67:https    ino:12200 sk:c2ea2140
ESTAB       0      0                                                                        192.168.235.147:ssh                                                                       192.168.235.155:4635     timer:(keepalive,2min59sec,0) ino:1101514 sk:c2ea4740
ESTAB       0      0                                                                              127.0.0.1:4369                                                                            127.0.0.1:54620    ino:2309430 sk:c2ea2ac0
ESTAB       0      0                                                                        192.168.235.147:ssh                                                                       192.168.235.155:1095     timer:(keepalive,6min59sec,0) ino:379842 sk:c2f4af80
ESTAB       0      0                                                                              127.0.0.1:4369                                                                            127.0.0.1:44234    ino:2044745 sk:c2ea3440
ESTAB       0      0                                                                              127.0.0.1:54620                                                                           127.0.0.1:4369     ino:2309429 sk:c2ea7200
ESTAB       0      0                                                                              127.0.0.1:44234                                                                           127.0.0.1:4369     ino:2044744 sk:c2ea1300

root@ubuntu:/usr/src/iproute-20090324/misc# ss -o
State       Recv-Q Send-Q                                                                     Local Address:Port                                                                         Peer Address:Port
ESTAB       0      0                                                                        192.168.235.147:ssh                                                                       192.168.235.155:4251     timer:(keepalive,50min,0)
CLOSE-WAIT  38     0                                                                        192.168.235.147:13910                                                                        174.36.30.67:https
ESTAB       0      0                                                                        192.168.235.147:ssh                                                                       192.168.235.155:4635     timer:(keepalive,2min50sec,0)
ESTAB       0      0                                                                              127.0.0.1:4369                                                                            127.0.0.1:54620
ESTAB       0      0                                                                        192.168.235.147:ssh                                                                       192.168.235.155:1095     timer:(keepalive,6min50sec,0)
ESTAB       0      0                                                                              127.0.0.1:4369                                                                            127.0.0.1:44234
ESTAB       0      0                                                                              127.0.0.1:54620                                                                           127.0.0.1:4369
ESTAB       0      0                                                                              127.0.0.1:44234                                                                           127.0.0.1:4369

Have fun!!!

gen*都需要callback模块提供一个init模块, 在这个模块里面做相应的初始化操作.

我们以gen_server为例子. 当我们调用gen_server:start_link的时候,  底层会委托proc_lib来启动新的进程同时调用我们的init函数进行初始化.  同时gen系列会同步等待新进程初始化完毕, 才接着往下执行.

我们看下文档:

gen_server:start_link(ServerName, Module, Args, Options) -> Result
The gen_server process calls Module:init/1 to initialize. To ensure a synchronized start-up procedure, start_link/3,4 does not return until Module:init/1 has returned.

我们会经常误解这个行为, 以为start是马上返回的, otp团队之前也修正httpc的一个这类问题的bug.

所以我们在init的时候, 尽可能的做些简单的工作. 如果需要做费时的操作, 那么可以先让一条消息, 放到消息队列去, 让init马上返回. 这时候兵分二路, 调用进程可以继续往下执行, 费时操作也在稍后被调用.

而且gen*系列一旦启动起来, 如果该进程不幸发生意外崩溃,  proc_lib会负责记录当时的现场和原因,  非常有助于诊断.

我们演示下这个问题:

root@ubuntu:~# cat tg.erl

-module(tg).

-behaviour(gen_server).

%% API
-export([start/0]).

%% gen_server callbacks
-export([init/1, handle_call/3, handle_cast/2, handle_info/2,
terminate/2, code_change/3]).

-record(state, {}).
-define(SERVER, ?MODULE).

start() ->
gen_server:start_link({local, ?SERVER}, ?MODULE, [], []).

init([]) ->

io:format("init pid ~p~n", [self()]),

%% sleep
receive
after 3000 ->
cont
end,

erlang:send_after(0, self(), crash_me),

{ok, #state{}}.

handle_call(_Request, _From, State) ->
Reply = ok,
{reply, Reply, State}.

handle_cast(_Msg, State) ->
{noreply, State}.

handle_info(crash_me, State) ->
% do some hard work

% crash me
exit(crash_me),
{noreply, State};

handle_info(_Info, State) ->
{noreply, State}.

terminate(_Reason, _State) ->
ok.

code_change(_OldVsn, State, _Extra) ->
{ok, State}.

root@ubuntu:~# erl
Erlang R13B04 (erts-5.7.5) [source] [smp:2:2] [rq:2] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.7.5  (abort with ^G)
1> self().
<0.36.0>
2> timer:tc(tg, start, []).
init pid <0.34.0>
{3003583,{ok,<0.34.0>}}
2>
=ERROR REPORT==== 8-Apr-2010::16:20:09 ===
** Generic server tg terminating
** Last message in was crash_me
** When Server state == {state}
** Reason for termination ==
** crash_me
** exception error: crash_me
2>

1. 我们个tg:start启动了3003毫秒, 证明这是同步调用.

2. init是在新进程里面执行的
3. 我们的费时间操作用erlang:send_after发起
4. 一旦发生crash我们有记录在案.

收工!

很多同学在使用erlang的过程中, 碰到了很奇怪的问题, 后来查明都是文件句柄不够用了, 因为系统默认的是每个进程1024. 所以我们有必要在程序运行的时候, 了解这些信息, 以便诊断和预警.
下面的这个程序就演示了这个如何查看节点的可用句柄数目和已用句柄数的功能.

首先确保你已经安装了lsof, 我的系统是ubuntu可以这样安装.

root@ubuntu:~# apt-get -y install lsof   

root@ubuntu:~# cat fd.erl

-module(fd).
-export([start/0]).

get_total_fd_ulimit() ->
    {MaxFds, _} = string:to_integer(os:cmd("ulimit -n")),
    MaxFds.

get_total_fd() -> get_total_fd(os:type()).

get_total_fd({unix, Os})
  when Os =:= linux orelse
       Os =:= darwin orelse
       Os =:= freebsd orelse Os =:= sunos ->
    get_total_fd_ulimit();
get_total_fd(_) -> unknown.

get_used_fd_lsof() ->
    Lsof = os:cmd("lsof -d \"0-9999999\" -lna -p " ++
                  os:getpid()),
    string:words(Lsof, $\n).

get_used_fd() -> get_used_fd(os:type()).

get_used_fd({unix, Os})
  when Os =:= linux orelse
       Os =:= darwin orelse Os =:= freebsd ->
    get_used_fd_lsof();
get_used_fd(_) -> unknown.

start()->
    io:format("total fd: ~p~n"
              "used fd: ~p~n", [get_total_fd(), get_used_fd()]),
    halt(0).

root@ubuntu:~# erlc fd.erl
root@ubuntu:~# ulimit -n 1024
root@ubuntu:~# erl -noshell -s fd      
total fd: 1024
used fd: 10
root@ubuntu:~# ulimit -n 10240   
root@ubuntu:~# erl -noshell -s fd
total fd: 10240
used fd: 10
root@ubuntu:~# 

收工!

erts_debug:get_internal_state是用来获取Erlang运行期内部信息的主要手段. 但是这个功能是用来给开发人员或者说需要了解系统内部细节的场合, 比如说系统调优.
在R13B04可以使用的选项有:
1. reds_left
2. node_and_dist_references
3. monitoring_nodes
4. next_pid
5. next_port
6. ‘DbTable_words’
7. check_io_debug
8. process_info_args
9. processes
10. processes_bif_info
11. max_atom_out_cache_index
12. nbalance
13. available_internal_state
14. force_heap_frags
15. {process_status, Pid}
16. {link_list, Pid} 或者 {link_list, Port} 或者 {link_list, Nodename}
17. {monitor_list, Pid} 或者 {monitor_list, Nodename}
18. {channel_number, Sysname}
19. {have_pending_exit, Pid}
20. {binary_info, Binary}
21. {dist_port, Sysname}
22. {atom_out_cache_index, Atom}
23. {fake_scheduler_bindings, How} How=spread | processor_spread | thread_spread | default_bind | no_node_processor_spread | no_node_thread_spread | no_spread | unbound

使用此功能的前提是先用erts_debug:set_internal_state(available_internal_state, true). 否者调用get_internal_state会提示失败.
我们演示下:

root@ubuntu:~/otp/test# erl -sname x
Erlang R14A (erts-5.8) [source] [smp:2:2] [rq:2] [async-threads:0] [kernel-poll:false]

Eshell V5.8  (abort with ^G)
(x@ubuntu)1> erts_debug:get_internal_state(check_io_debug). 
** exception error: undefined function erts_debug:get_internal_state/1
(x@ubuntu)2> erts_debug:set_internal_state(available_internal_state, true).
false
=ERROR REPORT==== 7-Apr-2010::14:40:41 ===
Process <0.38.0> on node 'x@ubuntu' enabled access to the emulator internal state.
NOTE: This is an erts internal test feature and should *only* be used by OTP test-suites.
(x@ubuntu)3> erts_debug:get_internal_state(check_io_debug). 
--- fds in pollset --------------------------------------
fd=0 type=chr driver_select ev=IN  inport=#Port<0.294> inname=tty_sl -c -e indrv=tty_sl 
fd=3 type=fifo internal ep_ev=IN
fd=4 type=fifo internal 
fd=7 type=sock driver_select ev=IN  inport=#Port<0.62> inname=tcp_inet indrv=tcp_inet 
fd=8 type=sock driver_select ev=IN  inport=#Port<0.65> inname=tcp_inet indrv=tcp_inet 
fd=9 type=fifo driver_select ev=IN  inport=#Port<0.486> inname=inet_gethost 4  indrv=spawn 
fd=10 type=sock driver_select ev=IN  inport=#Port<0.492> inname=tcp_inet indrv=tcp_inet 
fd=11 type=sock driver_select ev=IN  inport=#Port<0.495> inname=tcp_inet indrv=tcp_inet 

used fds=6
internal fds=2
---------------------------------------------------------
0
(x@ubuntu)4>  erts_debug:get_internal_state({process_status, self()}).
running

List Comprehensions
List comprehensions are a feature of many modern functional programming languages. Subject to certain rules, they provide a succinct notation for generating elements in a list.
List comprehensions are analogous to set comprehensions in Zermelo-Frankel set theory and are called ZF expressions in Miranda. They are analogous to the setof and findall predicates in Prolog.

List comprehensions are written with the following syntax:

[Expr || Qualifier1,…,QualifierN]

Expr is an arbitrary expression, and each Qualifier is either a generator or a filter.

* A generator is written as:
Pattern <- ListExpr. ListExpr must be an expression which evaluates to a list of terms. * A bit string generator is written as: BitstringPattern <= BitStringExpr. BitStringExpr must be an expression which evaluates to a bitstring. * A filter is an expression which evaluates to true or false.

但是从R13B以后lc是可以不要generator的, 只要filter也可以.

root@ubuntu:~# erl
Erlang R13B04 (erts-5.7.5) [source] [smp:2:2] [rq:2] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.7.5  (abort with ^G)
1> Nums = [1,2,3], Nouns = [shoe,box,tape], Verbs = [].
[]
2> [Nums || Nums =/= []] ++ [Nouns || Nouns =/= []] ++ [Verbs || Verbs
=/= []].
[[1,2,3],[shoe,box,tape]] 

这样的话你就可以写类似这样的代码:

Cond = 2,
[io:format("hello~n", []) || Cond =/=1].

免得用if 或者case语句写, 整个代码显得更清爽.

vanilla_driver是Erlang内置的驱动,用于高效读取文件句柄或者文件名,官方的文档没记载.

我来挖掘下:

root@ubuntu:~# cat >>hello.txt
hello
world
CTRL+D

root@ubuntu:~# erl
Erlang R13B04 (erts-5.7.5) [source] [smp:2:2] [rq:2] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.7.5  (abort with ^G)
1>  erlang:open_port("hello.txt", [stream,line, in,eof]).
#Port<0.498>
2>  flush(). 
Shell got {#Port<0.498>,{data,{eol,"hello"}}}
Shell got {#Port<0.498>,{data,{eol,"world"}}}
Shell got {#Port<0.498>,eof}
ok
3> 

高效无毒!

我们经常需要在erlang shell下显示变量的值. 为了节省版面, 变量的输出是会被截断的 以 …]来作为提示. 有时候这样很不方便, 不能看到全部的值, 比如processes().
这时候shell内置命令rp用于看格式化的数据来救助了, 比如我们可以这rp(processes()).

演示下

root@ubuntu:~# erl
Erlang R13B04 (erts-5.7.5) [source] [smp:2:2] [rq:2] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.7.5  (abort with ^G)
1> lists:duplicate(100, 'a').
[a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a|...]  %%被截断
2> rp(lists:duplicate(100, 'a')).
[a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,
 a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,
 a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,a,
 a,a,a,a,a,a,a,a,a,a,a]
ok
3>