系统技术非业余研究

Erlang其实是个操作系统，从下面的图可以看出它的架构：

erlang系统设计的时候是完全按照unix的理念来平行移植的，它的引导进程叫otp_ring0，第一个进程叫init,有没有感觉很熟悉？
既然是一个操作系统，那么这个系统就是一个非常复杂的系统。erlang vm运行的时候是以unix的进程方式体现的，然后这个进程本身是个小世界，这个世界会跑着很多application,每个application包含配置，数据，模块代码等，多个applcaiton之间相互协作，完成指定的业务目标。

当我们的业务系统出现不按预期执行的时候，那么问题出在哪里呢？我们如何调查呢？
首先erlang当然提供了非常多的调查类的函数，如erlang:system_info,erlang:memory,os:getenv等提供各种各样的信息。但是问题是这些信息是散落在各地的，很难汇集起来提供全面的诊断信息。

官方在最近也认识到了这个问题，在最新的R16版本添加了system_information模块来解决这个问题，具体的patch参考 这里

这个patch总的来讲做了三件比较大的事情：

1. Add system information aggregate
2. Add erts app-file
3. erts: Add cflags, ldflags and config.h into executable

除聚合信息以外，还有一个就是把编译beam的时候的配置， cflags, ldflags 什么的都聚合进去。因为beam vm是依靠操作系统提供各种服务的，如锁，原子，epoll，网络等服务，这些服务每个系统都是不太一样的，出了问题很有必要先调查这些结合处。

新增加的system_information模块位于runtime_tools应用中，它的注释里面也写的清楚：

%% The main purpose of system_information is to aggregate all information
%% deemed useful for investigation, i.e. system_information:report/0.

简单的分析下代码，它聚合的信息主要有以下几块：

report() -> [
        {init_arguments,    init:get_arguments()},
        {code_paths,        code:get_path()},
        {code,              code()},
        {system_info,       erlang_system_info()},
        {erts_compile_info, erlang:system_info(compile_info)},
        {beam_dynamic_libraries, get_dynamic_libraries()},
        {environment_erts,  os_getenv_erts_specific()},
        {environment,       [split_env(Env) || Env <- os:getenv()]}
    ].

1. vm的启动参数
2. 代码和应用的情况，版本,md5等等
3. vm和操作系统相关的情况
4. vm编译参数
5. vm依赖的动态库情况
6. 运行期的环境情况
这些参数对于一个beam vm的运行至关重要，任何一个地方不对，都可能影响系统的运行。

好，我们来演示下这个功能.我是用 kerl 来编译我们的这个版本的:

$ erl
Erlang R17A (erts-5.11) [source-b7fbc28] [64-bit] [smp:16:16] [async-threads:10] [hipe] [kernel-poll:false]

Eshell V5.11  (abort with ^G)
1> system_information:to_file("si.txt").
ok

这个si.txt就包括了我们所需要的信息，我们摘抄部分出来：

{system_information_version, “1.0”}.
{system_information, [{init_arguments,
[{root,[“/home/chuba/.kerl/builds/r16m/release_git”]},
{progname,[“erl”]},
{home,[“/home/chuba”]}]},
{code_paths,
[“.”,
“/home/chuba/.kerl/builds/r16m/release_git/lib/kernel-2.17/ebin”,
…
]},
{code,
[{code,[{path,”.”},{modules,[]}]},
{application,
{kernel,
[{description,”ERTS CXC 138 10″},
{vsn,”2.17″},
{path,
“/home/chuba/.kerl/builds/r16m/release_git/lib/kernel-2.17/ebin”},
{modules,
[{application,
[{loaded,true},
{native,false},
{compiler,”4.9.1″},
{md5,”69c3102d717e7258701a536ddae1eb89″}]},
…
{application,
{compiler,
[{description,”ERTS CXC 138 10″},
{vsn,”4.9.2″},
{path,
“/home/chuba/.kerl/builds/r16m/release_git/lib/compiler-4.9.2/ebin”},
{modules,
[{beam_a,
[{loaded,false},
{native,false},
{compiler,”4.9.1″},
{md5,”a338f3a7878d0c9a407950fcb5a2e7ed”}]},
…
{system_info,
[{allocator,
{glibc,
[2,12],
[sys_alloc,temp_alloc,sl_alloc,std_alloc,ll_alloc,
eheap_alloc,ets_alloc,fix_alloc,binary_alloc,
driver_alloc,mseg_alloc],
[{sys_alloc,[{e,true},{m,libc},{tt,131072},{tp,0}]},
{temp_alloc,
[{e,true},
{t,true},
…
{check_io,
[{name,erts_poll},
{primary,poll},
{fallback,false},
{kernel_poll,false},
{memory_size,49808},
{total_poll_set_size,2},
{lazy_updates,true},
{pending_updates,0},
{batch_updates,false},
{concurrent_updates,false},
{max_fds,131072}]},
{otp_release,”R17A”},
{port_limit,131072},
{process_limit,262144},
{smp_support,true},
{system_version,
“Erlang R17A (erts-5.11) [source-b7fbc28] [64-bit] [smp:16:16] [async-threads:10] [hipe] [kernel\
-poll:false]\n”},
{system_architecture,”x86_64-unknown-linux-gnu”},
{threads,true},
{thread_pool_size,10},
{{wordsize,internal},8},
{{wordsize,external},8},
{{cpu_topology,defined},undefined},
{{cpu_topology,detected},
{compat_rel,17},
{schedulers_state,{16,16,16}},
{build_type,opt},
{logical_processors,16},
{logical_processors_online,16},
{logical_processors_available,16},
{driver_version,”2.1″},
{emu_args,
[“/home/chuba/.kerl/builds/r16m/release_git/erts-5.11/bin/beam.smp”,
“–“,”-root”,
“/home/chuba/.kerl/builds/r16m/release_git”,
“-progname”,”erl”,”–“,”-home”,”/home/chuba”,”–“]},
{ethread_info,
[{“32-bit native atomics”,”ethread”,
[“ethr_native_atomic32_dec_mb()”,
“ethr_native_atomic32_inc_mb()”,
“ethr_native_atomic32_add_mb()”,
“ethr_native_atomic32_read()”,
“ethr_native_atomic32_add_return_mb()”,
“ethr_native_atomic32_set_mb()”,
“ethr_native_atomic32_set_relb()”,
“ethr_native_atomic32_set()”,
“ethr_native_atomic32_xchg_mb()”,
“ethr_native_atomic32_cmpxchg_mb()”]},

{erts_compile_info,
[{ldflags,[]},
{cflags,
“-Werror=return-type -g -O2 -I/home/chuba/.kerl/builds/r16m/otp_src_git/erts/x86_64-unknown-lin\
ux-gnu -fno-tree-copyrename -D_GNU_SOURCE -DERTS_SMP -DHAVE_CONFIG_H -Wall -Wstrict-prototypes -Wmissing-prototypes -W\
declaration-after-statement -DUSE_THREADS -D_THREAD_SAFE -D_REENTRANT -DPOSIX_THREADS -D_POSIX_THREAD_SAFE_FUNCTIONS -DE\
RLANG_GIT_VERSION=\”b7fbc28\””},
{config_h,
“/* x86_64-unknown-linux-gnu/config.h. Generated from config.h.in by configure. */\n/* config.\
h.in. Generated from configure.in by autoheader. */\n\n\n#define GHBN_R_SOLARIS 2\n#define GHBN_R_AIX 3\n#define \
GHBN_R_GLIBC 4\n\n\n/* Define if building universal (internal helper macro) */\n/* #undef AC_APPLE_UNIVERSAL_BUILD */\\
n\n/* Define the brk() argument type. */\n
…
{beam_dynamic_libraries,
“\tlinux-vdso.so.1 => (0x00007fff6326f000)\n\tlibutil.so.1 => /lib64/libutil.so.1 (0x000000323060\
0000)\n\tlibdl.so.2 => /lib64/libdl.so.2 (0x000000363e000000)\n\tlibm.so.6 => /lib64/libm.so.6 (0x000000322b600000)\n\tli\
bncurses.so.5 => /lib64/libncurses.so.5 (0x0000003d93000000)\n\tlibpthread.so.0 => /lib64/libpthread.so.0 (0x000000322ae0\
0000)\n\tlibrt.so.1 => /lib64/librt.so.1 (0x0000003d92c00000)\n\tlibc.so.6 => /lib64/libc.so.6 (0x000000322aa00000)\n\tli\
btinfo.so.5 => /lib64/libtinfo.so.5 (0x000000322ce00000)\n\t/lib64/ld-linux-x86-64.so.2 (0x000000322a200000)\n”},
{environment_erts,
[{“BINDIR”,
“/home/chuba/.kerl/builds/r16m/release_git/erts-5.11/bin”},
{“DIALYZER_EMULATOR”,false},
{“CERL_DETACHED_PROG”,false},
{“EMU”,”beam”},
{“ERL_CONSOLE_MODE”,false},
{“ERL_CRASH_DUMP”,false},
{“ERL_CRASH_DUMP_NICE”,false},
{“ERL_CRASH_DUMP_SECONDS”,false},
{“ERL_EPMD_PORT”,false},
{“ERL_EMULATOR_DLL”,false},
{“ERL_FULLSWEEP_AFTER”,false},
{“ERL_LIBS”,false},
{“ERL_MALLOC_LIB”,false},
{“ERL_MAX_PORTS”,false},
…
{environment,
[{“G_BROKEN_FILENAMES”,”1″},
{“LESSOPEN”,”|/usr/bin/lesspipe.sh %s”},
{“SSH_CONNECTION”,
“10.232.2.221 55860 10.232.64.76 22”},
{“CVS_RSH”,”ssh”},
{“LOGNAME”,”chuba”},
{“SHLVL”,”1″},
{“HOME”,”/home/chuba”},
{“HISTCONTROL”,”ignoredups”},
{“PROGNAME”,”erl”},
{“LANG”,”en_US.UTF-8″},
{“PWD”,”/home/chuba/.kerl/builds/r16m/release_git”},
{“MAIL”,”/var/spool/mail/chuba”},
{“PATH”,
{“ERL_MAX_ETS_TABLES”,false},
{“ERL_NO_VFORK”,false},
{“ERL_NO_KERNEL_POLL”,false},
{“ERL_THREAD_POOL_SIZE”,false},
{“ERLC_EMULATOR”,false},
{“ESCRIPT_EMULATOR”,false},
{“HOME”,”/home/chuba”},
{“HOMEDRIVE”,false},
{“HOMEPATH”,false},
{“LANG”,”en_US.UTF-8″},
{“LC_ALL”,false},
{“LC_CTYPE”,false},
{“PATH”,
“/home/chuba/.kerl/builds/r16m/release_git/erts-5.11/bin:/home/chuba/.kerl/builds/r16m/release_g\
it/bin:/home/y/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/chuba/bin”},
{“PROGNAME”,”erl”},
{“RELDIR”,false},
{“ROOTDIR”,
“/home/chuba/.kerl/builds/r16m/release_git”},
{“TERM”,”linux”},
{“COMSPEC”,false},
{“HEART_COMMAND”,false},
{“RUN_ERL_LOG_ALIVE_MINUTES”,false},
{“RUN_ERL_LOG_ACTIVITY_MINUTES”,false},
{“RUN_ERL_LOG_ALIVE_FORMAT”,false},
{“HISTSIZE”,”2000″},
{“TERM”,”linux”},
{“SHELL”,”/bin/bash”},
{“HOSTNAME”,”rds064076.sqa.cm4″},
{“EMU”,”beam”}]}]}.

信息非常的完整，系统异常的时候，很容易报告给别人或者配合crashdump等工具定位到问题。

小结：成熟的系统需要成熟的运维工具。

祝玩得开心！