linux – Ubuntu HTTP延迟陷入奇怪的分位数

我有一个Ubuntu 10.10服务器,有足够的RAM,带宽和CPU.在从Apache和nginx提供静态文件时,我在延迟分布中看到了一种奇怪的,可重复的模式.因为这两个http服务器都存在这个问题,所以我想知道我是否配置错误或调整了Ubuntu的网络或缓存参数.

ab -n 1000 -c 4 http：//apache-host/static-file.jpg：

Percentage of the requests served within a certain time (ms)
  50%      5
  66%   3007
  75%   3009
  80%   3011
  90%   9021
  95%   9032
  98%  21068
  99%  45105
 100%  45105 (longest request)

ab -n 1000 -c 4 http：//nginx-host/static-file.jpg：

Percentage of the requests served within a certain time (ms)
  50%     19
  66%     19
  75%   3011
  80%   3017
  90%   9021
  95%  12026
  98%  12028
  99%  18063
 100%  18063 (longest request)

结果始终遵循这种模式 – 50％或更多的请求按预期服务,然后其余部分落入离散频段,最慢的几个数量级更慢.

Apache是​​2.x并且安装了mod_php. nginx是1.0.x并且安装了Passenger(但是app服务器都不应该位于静态文件的关键路径中).每次测试运行时,平均负载大约为1(服务器有12个物理内核). 5GB免费RAM,7GB缓存交换.测试从localhost运行.

以下是我从Ubuntu服务器10.10默认设置所做的配置更改：

/etc/sysctl.conf:
    net.core.rmem_default = 65536
    net.core.wmem_default = 65536
    net.core.rmem_max = 16777216
    net.core.wmem_max = 16777216
    net.ipv4.tcp_rmem = 4096 87380 16777216
    net.ipv4.tcp_wmem = 4096 65536 16777216
    net.ipv4.tcp_mem = 16777216 16777216 16777216
    net.ipv4.tcp_window_scaling = 1
    net.ipv4.route.flush = 1
    net.ipv4.tcp_no_metrics_save = 1
    net.ipv4.tcp_moderate_rcvbuf = 1
    net.core.somaxconn = 8192 

/etc/security/limits.conf:
    * hard nofile 65535
    * soft nofile 65535
    root hard nofile 65535
    root soft nofile 65535

other config:
    ifconfig eth0 txqueuelen 1000

如果这种问题响起,或者如果有关配置的更多信息会有所帮助,请告诉我.谢谢你的时间.

更新：这是我在增加net.netfilter.nf_conntrack_max之后看到的内容,如下所示：

Percentage of the requests served within a certain time (ms)
  50%      2
  66%      2
  75%      2
  80%      2
  90%      3
  95%      3
  98%      3
  99%      3
 100%      5 (longest request)

解决方法

关闭你的评论,这是nf_conntrack完全问题,你可以增加conntrak表：

sysctl -w net.netfilter.nf_conntrack_max = 131072

或者,如果您已经位于防火墙后面,则可以从连接跟踪中免除HTTP流量：

# iptables -L -t raw
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
NOTRACK    tcp  --  anywhere             anywhere            tcp dpt:www 
NOTRACK    tcp  --  anywhere             anywhere            tcp spt:www 

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
NOTRACK    tcp  --  anywhere             anywhere            tcp spt:www 
NOTRACK    tcp  --  anywhere             anywhere            tcp dpt:www