原创文章，转载请注明： 转载自Erlang非业余研究

本文链接地址: MYSQL数据库网卡软中断不平衡问题及解决方案

最近公司在MySQL的数据库上由于采用了高速的如PCIe卡以及大内存，去年在压力测试的时候突然发现数据库的流量可以把一个千M网卡压满了。随着数据库的优化，现在流量可以达到150M，所以我们采用了双网卡，在交换机上绑定，做LB的方式，提高系统的吞吐量。

但是在最近压测试的一个数据库中，mpstat发现其中一个核的CPU被软中断耗尽：

Mysql QPS 2W左右

——– —–load-avg—- —cpu-usage— —swap— -QPS- -TPS- -Hit%-
time | 1m 5m 15m |usr sys idl iow| si so| ins upd del sel iud| lor hit|
13:43:46| 0.00 0.00 0.00| 67 27 3 3| 0 0| 0 0 0 0 0| 0 100.00|
13:43:47| 0.00 0.00 0.00| 30 10 60 0| 0 0| 0 0 0 19281 0| 326839 100.00|
13:43:48| 0.00 0.00 0.00| 28 10 63 0| 0 0| 0 0 0 19083 0| 323377 100.00|
13:43:49| 0.00 0.00 0.00| 28 10 63 0| 0 0| 0 0 0 19482 0| 330185 100.00|
13:43:50| 0.00 0.00 0.00| 26 9 65 0| 0 0| 0 0 0 19379 0| 328575 100.00|
13:43:51| 0.00 0.00 0.00| 27 9 64 0| 0 0| 0 0 0 19723 0| 334378 100.00|

mpstat -P ALL 1说：

针对这个问题，我们利用工具，特别是systemtap, 一步步来调查和解决问题。

首先我们来确认下网卡的设置：

$uname -r
2.6.32-131.21.1.tb399.el6.x86_64

$ lspci -vvvv
01:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
        Subsystem: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
        Latency: 0, Cache Line Size: 256 bytes
        Interrupt: pin A routed to IRQ 114
        Region 0: Memory at f6000000 (64-bit, non-prefetchable) [size=32M]
        Capabilities: <access denied>

01:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
        Subsystem: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
        Latency: 0, Cache Line Size: 256 bytes
        Interrupt: pin B routed to IRQ 122
        Region 0: Memory at f8000000 (64-bit, non-prefetchable) [size=32M]
        Capabilities: <access denied>

$cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: em1
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: em1
MII Status: up
Link Failure Count: 0
Permanent HW addr: 78:2b:cb:1f:eb:c9
Slave queue ID: 0

Slave Interface: em2
MII Status: up
Link Failure Count: 0
Permanent HW addr: 78:2b:cb:1f:eb:ca
Slave queue ID: 0

从上面的信息我们可以确认二块 Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)网卡在做bonding。

我们的系统内核组维护的是RHEL 6.1, 很容易可以从/proc/interrupts和/proc/softirqs得到中断和软中断的信息的信息。
我们特别留意下softirq, 由于CPU太多，信息太乱，我只列出7个核心的情况：

$cat /proc/softirqs|tr -s ' ' '\t'|cut -f 1-8
        CPU0    CPU1    CPU2    CPU3    CPU4    CPU5    CPU6
        HI:     0       0       0       0       0       0
        TIMER:  401626149       366513734       274660062       286091775       252287943       258932438
        NET_TX: 136905  10428   17269   25080   16613   17876
        NET_RX: 1898437808      2857018450      580117978       26443   11820   15545
        BLOCK:  716495491       805780859       113853932       132589667       106297189       104629321
BLOCK_IOPOLL:   0       0       0       0       0       0       0
        TASKLET:        190643874       775861235       0       0       1       0
        SCHED:  61726009        66994763        102590355       83277433        144588168       154635009
        HRTIMER:        1883420 1837160 2316722 2369920 1721755 1666867
        RCU:    391610041       365150626       275741153       287074106       253401636       260389306

从上面我们粗粗可以看出网卡的软中断接收和发送都不平衡。
单单这些信息还不够，还是无法区别为什么一个核心被压垮了，因为我们的机器上还有个中断的大户:fusionIO PCIe卡，在过去的测试中该卡也会吃掉大量的CPU，所以目前无法判断就是网卡引起的，因而我们用stap来double check下：

$cat i.stp
global hard, soft, wq

probe irq_handler.entry {
hard[irq, dev_name]++;
}

probe timer.s(1) {
println("==irq number:dev_name")
foreach( [irq, dev_name] in hard- limit 5) {
printf("%d,%s->%d\n", irq, kernel_string(dev_name), hard[irq, dev_name]);
}

println("==softirq cpu:h:vec:action")
foreach( 1 in soft- limit 5) {
printf("%d:%x:%x:%s->%d\n", c, h, vec, symdata(action), soft1);
}

println("==workqueue wq_thread:work_func")
foreach( [wq_thread,work_func] in wq- limit 5) {
printf("%x:%x->%d\n", wq_thread, work_func, wq[wq_thread, work_func]);
}

println("\n")
delete hard
delete soft
delete wq
}

probe softirq.entry {
soft[cpu(), h,vec,action]++;
}

probe workqueue.execute {
wq[wq_thread, work_func]++
}

probe begin {
println("~")
}

$sudo stap i.stp
==irq number:dev_name
73,em1-6->7150
50,iodrive-fct0->7015
71,em1-4->6985
74,em1-7->6680
69,em1-2->6557
==softirq cpu:h:vec:action
1:ffffffff81a23098:ffffffff81a23080:0xffffffff81411110->36627
1:ffffffff81a230b0:ffffffff81a23080:0xffffffff8106f950->2169
1:ffffffff81a230a0:ffffffff81a23080:0xffffffff81237100->1736
0:ffffffff81a230a0:ffffffff81a23080:0xffffffff81237100->1308
1:ffffffff81a23088:ffffffff81a23080:0xffffffff81079ee0->941
==workqueue wq_thread:work_func
ffff880c14268a80:ffffffffa026b390->51
ffff880c1422e0c0:ffffffffa026b390->30
ffff880c1425f580:ffffffffa026b390->25
ffff880c1422f540:ffffffffa026b390->24
ffff880c14268040:ffffffffa026b390->23

#上面软中断的action的符号信息：
$addr2line -e /usr/lib/debug/lib/modules/2.6.32-131.21.1.tb411.el6.x86_64/vmlinux ffffffff81411110
/home/ads/build22_6u0_x64/workspace/kernel-el6/origin/taobao-kernel-build/kernel-2.6.32-131.21.1.el6/linux-2.6.32-131.21.1.el6.x86_64/net/core/ethtool.c:653

$addr2line -e /usr/lib/debug/lib/modules/2.6.32-131.21.1.tb411.el6.x86_64/vmlinux ffffffff810dc3a0
/home/ads/build22_6u0_x64/workspace/kernel-el6/origin/taobao-kernel-build/kernel-2.6.32-131.21.1.el6/linux-2.6.32-131.21.1.el6.x86_64/kernel/relay.c:466

$addr2line -e /usr/lib/debug/lib/modules/2.6.32-131.21.1.tb411.el6.x86_64/vmlinux ffffffff81079ee0
/home/ads/build22_6u0_x64/workspace/kernel-el6/origin/taobao-kernel-build/kernel-2.6.32-131.21.1.el6/linux-2.6.32-131.21.1.el6.x86_64/include/trace/events/timer.h:118

$addr2line -e /usr/lib/debug/lib/modules/2.6.32-131.21.1.tb411.el6.x86_64/vmlinux ffffffff8105d120
/home/ads/build22_6u0_x64/workspace/kernel-el6/origin/taobao-kernel-build/kernel-2.6.32-131.21.1.el6/linux-2.6.32-131.21.1.el6.x86_64/kernel/sched.c:2460

这次我们可以轻松的定位到硬中断基本上是平衡的，软中断都基本压在了1号核心上，再根据符号查找确认是网卡的问题。

好了，现在定位到了，问题解决起来就容易了：
1. 采用多队列万M网卡。
2. 用google的RPS patch来解决软中断平衡的问题, 把软中断分散到不同的核心去，参见这里.

我们还是用穷人的方案，写了个shell脚本来做这个事情：

$cat em.sh
#! /bin/bash                                                                                                                                                           

for i in `seq 0 7`
do
  echo f|sudo tee /sys/class/net/em1/queues/rx-$i/rps_cpus >/dev/null
  echo f|sudo tee /sys/class/net/em2/queues/rx-$i/rps_cpus >/dev/null
done

$sudo ./em.sh

$mpstat -P ALL 1

就可以看到我们的成果：

网卡的软中断成功分到二个核心上了，不再把一个核心拖死。

小结：多观察系统是好事。
后记：

有同学留言说:

根据我们的测试，BCM5709应该是支持多队列的

从中断来看确实是平衡的，也就是说多队列在工作，但是为什么软中断不平衡呢，还有CPU1上还压着什么任务呢？继续调查！
待续～

祝玩得开心！

原创文章，转载请注明： 转载自Erlang非业余研究

本文链接地址: gen_tcp接受链接时enfile的问题分析及解决

最近我们为了安全方面的原因，在RDS服务器上做了个代理程序把普通的MYSQL TCP连接变成了SSL链接，在测试的时候，皓庭同学发现Tsung发起了几千个TCP链接后Erlang做的SSL PROXY老是报告gen_tcp:accept返回{error, enfile}错误。针对这个问题，我展开了如下的调查：

首先man accept手册，确定enfile的原因，因为gen_tcp肯定是调用accept系统调用的:

EMFILE The per-process limit of open file descriptors has been reached.
ENFILE The system limit on the total number of open files has been reached.

从文档来看是由于系统的文件句柄数用完了，我们顺着来调查下：

$ uname -r
2.6.18-164.el5
$ cat /proc/sys/fs/file-nr
2040    0       2417338
$ ulimit -n
65535

由于我们微调了系统的文件句柄，具体参考这里 老生常谈: ulimit问题及其影响， 这些参数看起来非常的正常。
先看下net/socket.c代码：

static int sock_alloc_fd(struct file **filep)
{
        int fd;

        fd = get_unused_fd();
        if (likely(fd >= 0)) {
                struct file *file = get_empty_filp();

                *filep = file;
                if (unlikely(!file)) {
                        put_unused_fd(fd);
                        return -ENFILE;
                }
        } else
                *filep = NULL;
        return fd;
}

static int __sock_create(int family, int type, int protocol, struct socket **res, int kern)
{
...
/*
 *      Allocate the socket and allow the family to set things up. if
 *      the protocol is 0, the family is instructed to select an appropriate
 *      default.
 */

        if (!(sock = sock_alloc())) {
                if (net_ratelimit())
                        printk(KERN_WARNING "socket: no more sockets\n");
                err = -ENFILE;          /* Not exactly a match, but its the
                                           closest posix thing */
                goto out;
        }
...
}

asmlinkage long sys_accept(int fd, struct sockaddr __user *upeer_sockaddr, int __user *upeer_addrlen)
{
        struct socket *sock, *newsock;
        struct file *newfile;
        int err, len, newfd, fput_needed;
        char address[MAX_SOCK_ADDR];

        sock = sockfd_lookup_light(fd, &err, &fput_needed);
        if (!sock)
                goto out;

        err = -ENFILE;
        if (!(newsock = sock_alloc()))
                goto out_put;
...
}

从代码来看，会返回ENFILE都是由于socket句柄分配不出来了，我们还是本着怀疑的态度来写个stap脚本来再次验证下：

$ cat enfile.stp
probe kernel.function("kmem_cache_alloc").return,
          kernel.function("get_empty_filp").return{
  if($return == 0) { print_backtrace();exit();}
}
probe kernel.function("sock_alloc_fd").return {
  if($return < 0) { print_backtrace(); exit();}
}
probe syscall.accept.return {
  if($return == -23) {print_backtrace(); exit();}
}
probe begin {
println(":~");
}
$ sudo stap enfile.stp
:~

gen_tcp:accept报告{error, enfile}的时候，也没看到stap报异常，基本上可以排除操作系统的原因了，那么我们现在回到gen_tcp的实现来看。
gen_tcp是个port, 具体实现在erts/emulator/drivers/common/inet_drv.c，我们来看下有ENFILE的地方：

/* Copy a descriptor, by creating a new port with same settings
 * as the descriptor desc.
 * return NULL on error (ENFILE no ports avail)
 */
static tcp_descriptor* tcp_inet_copy(tcp_descriptor* desc,SOCKET s,
                                     ErlDrvTermData owner, int* err)
{
...
    /* The new port will be linked and connected to the original caller */
    port = driver_create_port(port, owner, "tcp_inet", (ErlDrvData) copy_desc);
    if ((long)port == -1) {
        *err = ENFILE;
        FREE(copy_desc);
        return NULL;
    }
...
}

当 driver_create_port 失败的时候，gen_tcp返回ENFILE,看起来这次找对地方了。我们继续看下 driver_create_port的实现：
看下erts/emulator/beam/io.c：

/*
 * Driver function to create new instances of a driver
 * Historical reason: to be used with inet_drv for creating
 * accept sockets inorder to avoid a global table.
 */
ErlDrvPort
driver_create_port(ErlDrvPort creator_port_ix, /* Creating port */
                   ErlDrvTermData pid,    /* Owner/Caller */
                   char* name,            /* Driver name */
                   ErlDrvData drv_data)   /* Driver data */
{
...
    rp = erts_pid2proc(NULL, 0, pid, ERTS_PROC_LOCK_LINK);
    if (!rp) {
        erts_smp_mtx_unlock(&erts_driver_list_lock);
        return (ErlDrvTermData) -1;   /* pid does not exist */
    }
    if ((port_num = get_free_port()) < 0) {
        errno = ENFILE;
        erts_smp_proc_unlock(rp, ERTS_PROC_LOCK_LINK);
        erts_smp_mtx_unlock(&erts_driver_list_lock);
        return (ErlDrvTermData) -1;
    }

    port_id = make_internal_port(port_num);
    port = &erts_port[port_num & erts_port_tab_index_mask];
...
}

get_free_port()<0的时候就返回ENFILE错误。
那我们看下port总的数目是如何设定的：

/* initialize the port array */
void init_io(void)
{
…
 if (erts_sys_getenv("ERL_MAX_PORTS", maxports, &maxportssize) == 0)
        erts_max_ports = atoi(maxports);
    else
        erts_max_ports = sys_max_files();

    if (erts_max_ports > ERTS_MAX_PORTS)
        erts_max_ports = ERTS_MAX_PORTS;
    if (erts_max_ports < 1024)
        erts_max_ports = 1024;

    if (erts_use_r9_pids_ports) {
        ports_bits = ERTS_R9_PORTS_BITS;
        if (erts_max_ports > ERTS_MAX_R9_PORTS)
            erts_max_ports = ERTS_MAX_R9_PORTS;
    }

    port_extra_shift = erts_fit_in_bits(erts_max_ports – 1);
    port_num_mask = (1 << ports_bits) – 1;
…
}

第一步：如果设定了ERL_MAX_PORTS环境变量，那么就按照用户设定的，否则就和ulimit -n 一样大。
第二部：这个值不能大于ERTS_MAX_PORTS或者小于1024.

好了，我们基本上明白这个问题的原因了： erts_max_ports设定的太小.

我们再来验证下：
gdb attach到我们的进程下
(gdb) p erts_max_ports
$1 = 4096
原来是port设置有问题，导致上面的现象，看起来很绕的，Erlang的设计者认为PORT资源（相当于操作系统的IO资源）短缺如同操作系统的文件句柄短缺一样，达到system_limit就应该出ENFILE错误！

解决方案是: erl -env ERTS_MAX_PORTS NNNN 搞大点就好。

顺便再来强调下Erlang服务器几个关键的参数，来源：http://www.ejabberd.im/tuning，对服务器的设置很有帮助。

This page lists several tricks to tune your ejabberd and Erlang installation for maximum performance gains. Remark that some of the described options are experimental.

Erlang Ports Limit: ERL_MAX_PORTS
Erlang consumes one port for every connection, either from a client or from another Jabber server. The option ERL_MAX_PORTS limits the number of concurrent connections and can be specified when starting ejabberd:

erl -s ejabberd -env ERL_MAX_PORTS 5000 ...

Maximum Number of Erlang Processes: +P
Erlang consumes a lot of lightweight processes. If there is a lot of activity on ejabberd so that the maximum number of proccesses is reached, people will experiment greater latency times. As these processes are implemented in Erlang, and therefore not related to the operating system processes, you do not have to worry about allowing a huge number of them.

erl -s ejabberd +P 250000 ...

ERL_FULLSWEEP_AFTER: Maximum number of collections before a forced fullsweep
The ERL_FULLSWEEP_AFTER option shrinks the size of the Erlang process after RAM intensive events. Note that this option may downgrade performance. Hence this option is only interesting on machines that host other services (webserver, mail) on which ejabberd does not receive constant load.

erl -s ejabberd -env ERL_FULLSWEEP_AFTER 0 ...

Kernel Polling: +K true

The kernel polling option requires that you have support for it in your kernel. By default, Erlang currently supports kernel polling under FreeBSD, Mac OS X, and Solaris. If you use Linux, check this newspost. Additionaly, you need to enable this feature while compiling Erlang.

From Erlang documentation -> Basic Applications -> erts -> erl -> System Flags:

+K true|false

Enables or disables the kernel poll functionality if the emulator has kernel poll support. By default the kernel poll; functionality is disabled. If the emulator doesn't have kernel poll support and the +K flag is passed to the emulator, a warning is issued at startup.

If you meet all requirements, you can enable it in this way:

erl -s ejabberd +K true ...

Mnesia Tables to Disk
By default, ejabberd uses Mnesia as its database. In Mnesia you can configure each table in the database to be stored on RAM, on RAM and on disk, or only on disk. You can configure this in the web interface: Nodes -> 'mynode' -> DB Management. Modification of this option will consume some memory and CPU time.
Number of Concurrent ETS and Mnesia Tables: ERL_MAX_ETS_TABLES
The number of concurrent ETS and Mnesia tables is limited. When the limit is reached, errors will appear in the logs:

** Too many db tables **

You can safely increase this limit when starting ejabberd. It impacts memory consumption but the difference will be quite small.

erl -s ejabberd -env ERL_MAX_ETS_TABLES 20000 ...

小结：很多问题是很绕的，要多方面考虑验证。

祝玩得开心！

原创文章，转载请注明： 转载自Erlang非业余研究

本文链接地址: Linux下pipe使用注意事项

Linux下的pipe使用非常广泛, shell本身就大量用pipe来粘合生产者和消费者的. 我们的服务器程序通常会用pipe来做线程间的ipc通讯. 由于unix下的任何东西都是文件,只要是文件,在读取的时候,,就会设置last access time, 所以pipe也不例外., 但是这个时间对我们没有意义 如果pipe使用的非常频繁的时候会碰到由于设置访问时间导致的性能问题. 这个开销远比pipe读写的本身开销大. 相比文件读写的开销, atime微不足道,但是对pipe来讲就不同了.
这个事情是上次和多隆同学在把玩他的网络框架的时候,无意发现的.

我们来分析下pipe的这部分代码:

//pipe.c:L349
static ssize_t
pipe_read(struct kiocb *iocb, const struct iovec *_iov,
               unsigned long nr_segs, loff_t pos)
{
...
   if (ret > 0)
        file_accessed(filp);
    return ret;
}

我们可以看到在pipe读的时候要设置 file_accessed时间的,接着:

//fs.h:L1761
extern void touch_atime(struct vfsmount *mnt, struct dentry *dentry);
static inline void file_accessed(struct file *file)
{
        if (!(file->f_flags & O_NOATIME))
                touch_atime(file->f_path.mnt, file->f_path.dentry);
}

如果文件没设置 O_NOATIME就真正动手设置atime,接着:

//inode.c:L1493
void touch_atime(struct vfsmount *mnt, struct dentry *dentry)
{
        struct inode *inode = dentry->d_inode;
        struct timespec now;

        if (inode->i_flags & S_NOATIME)
                return;
        if (IS_NOATIME(inode))
                return;
        if ((inode->i_sb->s_flags & MS_NODIRATIME) && S_ISDIR(inode->i_mode))
                return;

        if (mnt->mnt_flags & MNT_NOATIME)
                return;
        if ((mnt->mnt_flags & MNT_NODIRATIME) && S_ISDIR(inode->i_mode))
                return;

        now = current_fs_time(inode->i_sb);

        if (!relatime_need_update(mnt, inode, now))
                return;

        if (timespec_equal(&inode->i_atime, &now))
                return;

        if (mnt_want_write(mnt))
                return;

        inode->i_atime = now;
        mark_inode_dirty_sync(inode);
        mnt_drop_write(mnt);
}

我们可以看出上面的流程还是比较复杂的,开销也很大.
我们来演示下:

$ cat > pipe_test.c
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <assert.h>
#include <pthread.h>
#include <fcntl.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <linux/unistd.h>

static int fds[2];
static pthread_t rp;

static void *rp_entry(void *arg) {
  char c[1];
  while (1 == read(fds[0], c, 1)) {
    if (*c == 'Q') break;
  }
  fprintf(stderr, "pipe read ok\n");
  return NULL;
}

int main(int argc, char *argv[]) {
  long i, n;
  int rc;
  if (argc < 2) {
    fprintf(stderr, "usage: pipe_test NNNNNN\n");
    return -1;
  }
  n = atol(argv[1]);
  pipe(fds);
  //fcntl(fds[0], F_SETFL, O_NOATIME);
  pthread_create(&rp, NULL, rp_entry, NULL);
  fprintf(stderr, "pipe write %ld...", n);
  for (i = 0; i < n; i++) {
    write(fds[1], "A", 1);
  }
  write(fds[1], "Q", 1);
  fprintf(stderr, "ok\n");
  pthread_join(rp, NULL);
  close(fds[0]);
  close(fds[1]);
  return 0;
}
CTRL+D
$ gcc -D_GNU_SOURCE pipe_test.c -lpthread
$ sudo opcontrol --setup --vmlinux=/usr/lib/debug/lib/modules/2.6.18-164.el5/vmlinux
$ sudo opcontrol --init && sudo opcontrol --reset && sudo opcontrol --start
$ ./a.out 10000000
pipe write 10000000...ok
pipe read ok
$ sudo opcontrol --shutdown
$ opreport -l|less
samples  %        app name                 symbol name
378654   92.7742  vmlinux                  .text.acpi_processor_idle
12978     3.1797  vmlinux                  current_fs_time
2530      0.6199  vmlinux                  thread_return
2345      0.5745  vmlinux                  touch_atime
2253      0.5520  vmlinux                  .text.acpi_safe_halt
1597      0.3913  vmlinux                  timespec_trunc
1368      0.3352  vmlinux                  file_update_time
1253      0.3070  vmlinux                  __mark_inode_dirty
901       0.2208  vmlinux                  pipe_writev
768       0.1882  vmlinux                  __mutex_lock_slowpath
763       0.1869  vmlinux                  try_to_wake_up
270       0.0662  vmlinux                  copy_user_generic_unrolled
254       0.0622  vmlinux                  acpi_set_register
254       0.0622  vmlinux                  system_call
233       0.0571  vmlinux                  pipe_readv
188       0.0461  vmlinux                  dnotify_parent
167       0.0409  vmlinux                  mutex_unlock
...

我们可以看到touch_atime的开销很大,远比pipe的读写大.
这次把这行注释去掉: fcntl(fds[0], F_SETFL, O_NOATIME); 指示pipe在读的时候不更新atime,看下效果:

$ opreport -l|less
samples  %        app name                 symbol name
599018   95.2466  vmlinux                  .text.acpi_processor_idle
4140      0.6583  vmlinux                  .text.acpi_safe_halt
3281      0.5217  vmlinux                  thread_return
2812      0.4471  vmlinux                  current_fs_time
2615      0.4158  vmlinux                  file_update_time
1790      0.2846  vmlinux                  __mutex_lock_slowpath
1657      0.2635  vmlinux                  timespec_trunc
1341      0.2132  vmlinux                  try_to_wake_up
1281      0.2037  vmlinux                  mutex_unlock
1080      0.1717  vmlinux                  mutex_lock
1001      0.1592  vmlinux                  pipe_readv
925       0.1471  vmlinux                  pipe_writev

这下看不到touch_atime了,开销省了,对于高性能服务器是很重要的.
小结: 细节很重要,记得开文件open的时候设置O_NOATIME或者用fcntl搞定它.
祝玩得开心!

原创文章，转载请注明： 转载自Erlang非业余研究

本文链接地址: Erlang版TCP服务器对抗攻击解决方案

互联网上的TCP服务器面对的环境情况比企业私有的服务器要复杂很多。常见的针对tcp服务器的攻击有以下几种：
1. 伪造协议，导致服务器crash. 比如说某条命令的字段长度，协议最大规定是1024，伪造个4096的。
2. 伪造大的报文。比如说一个包有1024M这么大。
3. 消耗服务器资源。开大量的连接， 以龟速发送报文，比如说每分钟一个字节。
4. DDOS攻击，从不同的IP发起大量的连接。
5. 攻击Erlang集群的授权体系。
6. 报文发送顺序逻辑错误，导致服务器crash. 比如说逻辑上应该先发A，再发B, 攻击者调了顺序。
7. 不停的连接断开，消耗服务器对资源的申请和释放，这个通常很耗。
等等，有很多方法。

那我们的Erlang版的TCP服务器如何应对这些情况呢？

刚才不小心按了发布，待续。。。

原创文章，转载请注明： 转载自Erlang非业余研究

本文链接地址: TCP链接主动关闭不发fin包奇怪行为分析

问题描述：
多隆同学在做网络框架的时候，发现一条tcp链接在close的时候，对端会收到econnrest,而不是正常的fin包. 通过抓包发现close系统调用的时候，我端发出rst报文, 而不是正常的fin。这个问题比较有意思，我们来演示下：

$ erl
Erlang R14B03 (erts-5.8.4) 1 [64-bit] [smp:16:16] [rq:16] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.8.4  (abort with ^G)
1> {ok,Sock} = gen_tcp:connect("baidu.com", 80, [{active,false}]).
{ok,#Port<0.582>}
2> gen_tcp:send(Sock, "GET / HTTP/1.1\r\n\r\n").
ok
3> gen_tcp:close(Sock).
ok

我们往baidu的首页发了个http请求，百度会给我们回应报文的,我们send完立即调用close.

然后我们在另外一个终端开tcpdump抓包确认：

$ sudo tcpdump port 80 -i bond0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on bond0, link-type EN10MB (Ethernet), capture size 96 bytes
17:22:38.246507 IP my031089.sqa.cm4.tbsite.net.19500 > 220.181.111.86.http: S 2228211568:2228211568(0) win 5840 <mss 1460,sackOK,timestamp 2607833238 0,nop,wscale 7>
17:22:38.284602 IP 220.181.111.86.http > my031089.sqa.cm4.tbsite.net.19500: S 3250338304:3250338304(0) ack 2228211569 win 8190 <mss 1436>
17:22:38.284624 IP my031089.sqa.cm4.tbsite.net.19500 > 220.181.111.86.http: . ack 1 win 5840
17:22:52.748468 IP my031089.sqa.cm4.tbsite.net.19500 > 220.181.111.86.http: P 1:19(18) ack 1 win 5840
17:22:52.786855 IP 220.181.111.86.http > my031089.sqa.cm4.tbsite.net.19500: . ack 19 win 5840
17:22:52.787194 IP 220.181.111.86.http > my031089.sqa.cm4.tbsite.net.19500: P 1:179(178) ack 19 win 5840
17:22:52.787203 IP my031089.sqa.cm4.tbsite.net.19500 > 220.181.111.86.http: . ack 179 win 6432
17:22:52.787209 IP 220.181.111.86.http > my031089.sqa.cm4.tbsite.net.19500: P 179:486(307) ack 19 win 5840
17:22:52.787214 IP my031089.sqa.cm4.tbsite.net.19500 > 220.181.111.86.http: . ack 486 win 7504
17:23:01.564358 IP my031089.sqa.cm4.tbsite.net.19500 > 220.181.111.86.http: R 19:19(0) ack 486 win 7504
...

我们可以清楚的看到 R 19:19(0) ack 486 win 7504,发了个rst包，通过strace系统调用也确认erlang确实调用了close系统调用。
那为什么呢？ @淘宝雕梁，tcp协议栈专家回答了这个问题：

在net/ipv4/tcp.c:1900附近

...
 /* As outlined in RFC 2525, section 2.17, we send a RST here because
    * data was lost. To witness the awful effects of the old behavior of
    * always doing a FIN, run an older 2.1.x kernel or 2.0.x, start a bulk
    * GET in an FTP client, suspend the process, wait for the client to
    * advertise a zero window, then kill -9 the FTP client, wheee...
    * Note: timeout is always zero in such a case.
    */
   if (data_was_unread) {
      /* Unread data was tossed, zap the connection. */
      NET_INC_STATS_USER(sock_net(sk), LINUX_MIB_TCPABORTONCLOSE);
      tcp_set_state(sk, TCP_CLOSE);
      tcp_send_active_reset(sk, sk->sk_allocation);
..

代码里面写的很清楚，如果你的接收缓冲去还有数据，协议栈就会发rst代替fin.
我们再来验证一下：

$ erl
Erlang R14B03 (erts-5.8.4) 1 [64-bit] [smp:16:16] [rq:16] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.8.4  (abort with ^G)
1> {ok,Sock} = gen_tcp:connect("baidu.com", 80, [{active,false}]).
{ok,#Port<0.582>}
2> gen_tcp:send(Sock, "GET / HTTP/1.1\r\n\r\n").
ok
3> gen_tcp:recv(Sock,0).
{ok,"HTTP/1.1 400 Bad Request\r\nDate: Fri, 01 Jul 2011 09:24:37 GMT\r\nServer: Apache\r\nConnection: Keep-Alive\r\nTransfer-Encoding: chunked\r\nContent-Type: text/html; charset=iso-8859-1\r\n\r\n127\r\n<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<HTML><HEAD>\n<TITLE>400 Bad Request</TITLE>\n</HEAD><BODY>\n<H1>Bad Request</H1>\nYour browser sent a request that this server could not understand.<P>\nclient sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /<P>\n</BODY></HTML>\n\r\n0\r\n\r\n"}
4> gen_tcp:close(Sock).
ok
5>

这次我们把接收缓冲区里的东西拉干净了。

再看下tcpdump：

...
17:36:07.236627 IP my031089.sqa.cm4.tbsite.net.9405 > 123.125.114.144.http: S 3086473299:3086473299(0) win 5840 <mss 1460,sackOK,timestamp 2608642228 0,nop,wscale 7>
17:36:07.274661 IP 123.125.114.144.http > my031089.sqa.cm4.tbsite.net.9405: S 738551248:738551248(0) ack 3086473300 win 8190 <mss 1436>
17:36:07.274685 IP my031089.sqa.cm4.tbsite.net.9405 > 123.125.114.144.http: . ack 1 win 5840
17:36:10.295795 IP my031089.sqa.cm4.tbsite.net.9405 > 123.125.114.144.http: P 1:19(18) ack 1 win 5840
17:36:10.334280 IP 123.125.114.144.http > my031089.sqa.cm4.tbsite.net.9405: . ack 19 win 5840
17:36:10.334547 IP 123.125.114.144.http > my031089.sqa.cm4.tbsite.net.9405: P 1:179(178) ack 19 win 5840
17:36:10.334554 IP my031089.sqa.cm4.tbsite.net.9405 > 123.125.114.144.http: . ack 179 win 6432
17:36:10.334563 IP 123.125.114.144.http > my031089.sqa.cm4.tbsite.net.9405: P 179:486(307) ack 19 win 5840
17:36:10.334566 IP my031089.sqa.cm4.tbsite.net.9405 > 123.125.114.144.http: . ack 486 win 7504
17:36:19.671374 IP my031089.sqa.cm4.tbsite.net.9405 > 123.125.114.144.http: F 19:19(0) ack 486 win 7504
17:36:19.709619 IP 123.125.114.144.http > my031089.sqa.cm4.tbsite.net.9405: . ack 20 win 5840
17:36:19.709643 IP 123.125.114.144.http > my031089.sqa.cm4.tbsite.net.9405: F 486:486(0) ack 20 win 5840
17:36:19.709652 IP my031089.sqa.cm4.tbsite.net.9405 > 123.125.114.144.http: . ack 487 win 7504
...

这次是发fin包了。

多隆同学再进一步，找出来之前squid client代码中不能理解的一句话：
client_side.c

...
 /* prevent those nasty RST packets */
    {
        char buf[SQUID_TCP_SO_RCVBUF];
        while (FD_READ_METHOD(fd, buf, SQUID_TCP_SO_RCVBUF) > 0);
    }
...

总算明白了这句话的意思了！

小结：认真学习协议栈太重要了。

玩得开心！

原创文章，转载请注明： 转载自Erlang非业余研究

本文链接地址: gen_tcp容易误用的一点解释

前天有同学在玩erlang gen_tcp的时候碰到了点小麻烦，描述如下：

比如说连接到baidu.com,发个http请求，然后马上接收数据，发现接收出错，wireshark抓包发现数据都有往返发送，比较郁闷。

我把问题演示下：

$ erl
Erlang R14B03 (erts-5.8.4) 1 [64-bit] [smp:16:16] [rq:16] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.8.4  (abort with ^G)
1> {ok,Sock} = gen_tcp:connect("baidu.com", 80, []).
{ok,#Port<0.582>}
2> gen_tcp:send(Sock, "GET / HTTP/1.1\r\n\r\n").
ok
3> gen_tcp:recv(Sock,0).
{error,einval}

这个问题的根源在于gen_tcp默认的{active,true},也就是说当gen_tcp收到网络包的时候，默认是把报文发送给它的宿主进程。而gen_tcp:recv是用户主动去拉数据，这二个模式是互斥的。

我们来看下代码otp/erts/emulator/drivers/common/inet_drv.c:7462

case TCP_REQ_RECV: {
..
        if (desc->inet.active || (len != 8))
            return ctl_error(EINVAL, rbuf, rsize);
..

那就解释为什么 gen_tcp:recv(Sock,0)返回错误码{error,einval}。
同时我们来验证下，报文是以消息的方式发送的。

4> flush().
Shell got {tcp,#Port<0.582>,
               "HTTP/1.1 400 Bad Request\r\nDate: Fri, 01 Jul 2011 03:51:25 GMT\r\nServer: Apache\r\nConnection: Keep-Alive\r\nTransfer-Encoding: chunked\r\nContent-Type: text/html; charset=iso-8859-1\r\n\r\n127\r\n<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<HTML><HEAD>\n<TITLE>400 Bad Request</TITLE>\n</HEAD><BODY>\n<H1>Bad Request</H1>\nYour browser sent a request that this server could not understand.<P>\nclient sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /<P>\n</BODY></HTML>\n\r\n0\r\n\r\n"}
ok
5>

搞清楚了问题，那解决方案很简单，connect的时候把active模式设成{active,false}.
再来演示下：

$ erl
Erlang R14B03 (erts-5.8.4) 1 [64-bit] [smp:16:16] [rq:16] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.8.4  (abort with ^G)
1> {ok,Sock} = gen_tcp:connect("baidu.com", 80, [{active,false}]).
{ok,#Port<0.582>}
2> gen_tcp:send(Sock, "GET / HTTP/1.1\r\n\r\n").
ok
3>  gen_tcp:recv(Sock,0).
{ok,"HTTP/1.1 400 Bad Request\r\nDate: Fri, 01 Jul 2011 05:25:15 GMT\r\nServer: Apache\r\nConnection: Keep-Alive\r\nTransfer-Encoding: chunked\r\nContent-Type: text/html; charset=iso-8859-1\r\n\r\n127\r\n<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<HTML><HEAD>\n<TITLE>400 Bad Request</TITLE>\n</HEAD><BODY>\n<H1>Bad Request</H1>\nYour browser sent a request that this server could not understand.<P>\nclient sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /<P>\n</BODY></HTML>\n\r\n0\r\n\r\n"}
4>

搞定！

玩得开心！

原创文章，转载请注明： 转载自Erlang非业余研究

本文链接地址: Linux下方便的socket读写查看器（socktop）

晚上 雕梁 说要找个工具来调查下unix域套接字的发送和接受情况，比如说A程序是否送出，B程序是否接收到，他找了tcpdump ,wireshark什么的，貌似都不支持。

这时候还是伟大的systemtap来救助了。 因为所有的socket通讯都是通过socket接口来的，任何family的通讯包括unix域套接都要走的，所以只要截获了socket 读写的几个syscall 就搞定了.

systemtap发行版本提供了个工具socktop， 位于 /usr/share/doc/systemtap/examples/network/socktop, 是个非常方便的工具, 干这个事情最合适了。

socktop源码里面的版权和简单的功能介绍：

# Socktop systemtap script
# Copyright (C) 2006 IBM Corp.
#
# This file is part of systemtap, and is free software. You can
# redistribute it and/or modify it under the terms of the GNU General
# Public License (GPL); either version 2, or (at your option) any
# later version.

###
### socktop – Combination shell/systemtap script to track reads and writes
### on sockets by process. Can be filtered by process IDs and
### names, protocols, protocol families, users and socket type.
###

$ uname -r
2.6.18-164.el5

$ rpm -i kernel-debuginfo-common-2.6.18-164.el5.x86_64.rpm
$ rpm -i kernel-debuginfo-2.6.18-164.el5.x86_64.rpm  

#使用帮助
$ /usr/share/doc/systemtap/examples/network/socktop -h
USAGE: socktop [-d] [-i interval] [-N num] [-P protocol]... [-f family]...
               [-t stype]... [-n pname]... [-p pid]... [-u username]... [-h]
    -d           # print network device traffic (default: off)
    -i interval  # interval in seconds between printing (default: 5)
    -N num       # number of top processes and devices to print (default: 10)
    -f family    # this protocol family only (default: all)
    -P protocol  # this protocol only (default: all)
    -t stype     # this socket type only (default: all)
    -n pname     # this process name only (default: all)
    -p pid       # this process ID only (default: all)
    -u username  # this user only (default: all)
    -c count     # number of iteration
    -m mod_name  # generate instrumentation (but do not run)
    -h           # print this help text

Protocol Families:
    LOCAL, INET, INET6, IPX, NETLINK, X25, AX25, ATMPVC, APPLETALK, PACKET

Protocols:
    TCP, UDP, SCTP, IP, FC, ... (see /etc/protocols for complete list)

Socket Types:
    STREAM, DGRAM, RAW, RDM, SEQPACKET, DCCP, PACKET

上面的使用写的很明白了，我们要过滤的是unix套接字， 每5秒报告下情况， 还顺手把网络设备的流量打出来。

$sudo /usr/share/doc/systemtap/examples/network/socktop -f LOCAL -i 5 -d
======================= Thu Mar 31 21:23:03 2011 ========================
------------------------------- PROCESSES -------------------------------
PID   UID     #SEND   #RECV SEND_KB RECV_KB PROT FAMILY   COMMAND
24821 50453       1       0       0       0 IP   LOCAL    crond
3840  0           0       2       0       0 IP   LOCAL    syslog-ng      

-------------------------------- DEVICES --------------------------------
DEV             #XMIT         #RECV         XMIT_KB         RECV_KB
eth0              457           250             102              38
bond0             457             0             102               0
lo                 24            24               2               2
eth1                0            10               0               0
=========================================================================

我们很清楚的看到了，crond在发，syslog-ng在收。

如果你想知道报文的内容的话，可以改改脚本把报文也dump出来。

玩得开心!

原创文章，转载请注明： 转载自Erlang非业余研究

本文链接地址: Systemtap辅助设置tcp_init_cwnd,免对操作系统打Patch

前段时间google的工程师提出对tcp的拥塞窗口的初始值进行增大可以显著的提高http的性能，这个主要是针对tcp的slow start的优化.
具体参考这里, 这里. 谢谢叔度同学从美国带回第一手信息!

由于低版本的linux内核的问题，这个参数的正确设置需要对os打patch,这个过程对线上机器来讲非常麻烦。 底下我用systemtap给出了个解决方案，免除这个麻烦. 我们在RHEL 5U4上作这个试验:

首先需要在开发机器上:

#安装符号信息
$uname -r
2.6.18-164.el5
$ sudo rpm -i kernel-debuginfo-common-2.6.18-164.el5.x86_64.rpm
$ sudo rpm -i kernel-debuginfo-2.6.18-164.el5.x86_64.rpm 

$ sudo yum install systemtap

$ cat > tcp_init_cwnd.stp
probe kernel.function("tcp_init_cwnd").return
{
$return = $1
}
CTRL+D

#设成7个mss
$ sudo stap -p4 -g -m initcwnd tcp_init_cwnd.stp 7
initcwnd.ko

好了,现在我们需要的模块有了, 可以直接拷贝到目标机器去的.
#目标机器需要安装systemtap,用于运行我们的模块.

$uname -r
2.6.18-164.el5
$ sudo yum install systemtap

#在系统的启动脚本里面运行以下命令:
$ sudo staprun -o initcwnd.out -D initcwnd.ko
$ 19122
#19122是initcwnd模块加载器在后台运行的进程号，需要的时候可以把模块停掉。

#当然如果你的模块没有输出的话也可以直接, 这样更简单
$ sudo insmod initcwnd.ko

大家碰到类似的问题,可以参考下这个思路.

玩得开心!

原创文章，转载请注明： 转载自Erlang非业余研究

本文链接地址: 来自RHEL系统调优手册的几张经典图

调优手册在这里下载: http://www.redbooks.ibm.com/redpapers/pdfs/redp4285.pdf
看图不说话：）

IO架构图：

内存管理图：

很容易误解的socket buffer：

原创文章，转载请注明： 转载自Erlang非业余研究

本文链接地址: 非阻塞connect的一个细节

昨天听zhuzhaoyuan说的一个connect细节. 通常我们connect的时候都是非阻塞的, 在connect调用后把句柄挂到poll去, 等poll通知可写的时候, 我们就认为connect成功了. 但是在linux平台下实际上不一定成功, 具体的要用socket get_opt来检查下出错码来决定.

以下是从man 2 connnect摘抄的:

EINPROGRESS
The socket(2,7,n) is non-blocking and the connection cannot be com-
pleted immediately. It is possible to select(2,7,2 select_tut)(2) or poll(2) for
completion by selecting the socket(2,7,n) for writing. After select(2,7,2 select_tut)
indicates writability, use getsockopt(2) to read(2,n,1 builtins) the SO_ERROR
option at level SOL_SOCKET to determine whether connect com-
pleted successfully (SO_ERROR is zero) or unsuccessfully
(SO_ERROR is one of the usual error(8,n) codes listed here, explain-
ing the reason for the failure).

我们看下erlang的inet_drv是如何处理的.
./erts/emulator/drivers/common/inet_drv.c:

        {
            int error = 0;      /* Has to be initiated, we check it */
            unsigned int sz = sizeof(error); /* even if we get -1 */
            int code = sock_getopt(desc->inet.s, SOL_SOCKET, SO_ERROR,
                                   (void *)&error, &sz);

            if ((code < 0) || error) {
                desc->inet.state = TCP_STATE_BOUND;  /* restore state */
                ret = async_error(INETP(desc), error);
                goto done;
            }
        }

结论: 细节决定品质.