服务器：Poweredge r620
操作系统：RHEL 6.4
内核：2.6.32-358.18.1.el6.x86_64
操作系统：RHEL 6.4
内核：2.6.32-358.18.1.el6.x86_64
我在生产环境中遇到应用程序警报.关键的cpu饥饿进程正在缺乏资源并导致处理积压.问题出现在最近部署的集群中的所有第12代戴尔服务器(r620)上.据我所知,这种情况的实例与cpu峰值的使用率相匹配,并伴随着dmesg中的大量“功率限制通知”垃圾邮件.其中一个事件的摘录：
Nov 7 10:15:15 someserver [.crit] cpu12: Core power limit notification (total events = 14) Nov 7 10:15:15 someserver [.crit] cpu0: Core power limit notification (total events = 14) Nov 7 10:15:15 someserver [.crit] cpu6: Core power limit notification (total events = 14) Nov 7 10:15:15 someserver [.crit] cpu14: Core power limit notification (total events = 14) Nov 7 10:15:15 someserver [.crit] cpu18: Core power limit notification (total events = 14) Nov 7 10:15:15 someserver [.crit] cpu2: Core power limit notification (total events = 14) Nov 7 10:15:15 someserver [.crit] cpu4: Core power limit notification (total events = 14) Nov 7 10:15:15 someserver [.crit] cpu16: Core power limit notification (total events = 14) Nov 7 10:15:15 someserver [.crit] cpu0: Package power limit notification (total events = 11) Nov 7 10:15:15 someserver [.crit] cpu6: Package power limit notification (total events = 13) Nov 7 10:15:15 someserver [.crit] cpu14: Package power limit notification (total events = 14) Nov 7 10:15:15 someserver [.crit] cpu18: Package power limit notification (total events = 14) Nov 7 10:15:15 someserver [.crit] cpu20: Core power limit notification (total events = 14) Nov 7 10:15:15 someserver [.crit] cpu8: Core power limit notification (total events = 14) Nov 7 10:15:15 someserver [.crit] cpu2: Package power limit notification (total events = 12) Nov 7 10:15:15 someserver [.crit] cpu10: Core power limit notification (total events = 14) Nov 7 10:15:15 someserver [.crit] cpu22: Core power limit notification (total events = 14) Nov 7 10:15:15 someserver [.crit] cpu4: Package power limit notification (total events = 14) Nov 7 10:15:15 someserver [.crit] cpu16: Package power limit notification (total events = 13) Nov 7 10:15:15 someserver [.crit] cpu20: Package power limit notification (total events = 14) Nov 7 10:15:15 someserver [.crit] cpu8: Package power limit notification (total events = 14) Nov 7 10:15:15 someserver [.crit] cpu10: Package power limit notification (total events = 14) Nov 7 10:15:15 someserver [.crit] cpu22: Package power limit notification (total events = 14) Nov 7 10:15:15 someserver [.crit] cpu15: Core power limit notification (total events = 369) Nov 7 10:15:15 someserver [.crit] cpu3: Core power limit notification (total events = 369) Nov 7 10:15:15 someserver [.crit] cpu1: Core power limit notification (total events = 369) Nov 7 10:15:15 someserver [.crit] cpu5: Core power limit notification (total events = 369) Nov 7 10:15:15 someserver [.crit] cpu17: Core power limit notification (total events = 369) Nov 7 10:15:15 someserver [.crit] cpu13: Core power limit notification (total events = 369) Nov 7 10:15:15 someserver [.crit] cpu15: Package power limit notification (total events = 375) Nov 7 10:15:15 someserver [.crit] cpu3: Package power limit notification (total events = 374) Nov 7 10:15:15 someserver [.crit] cpu1: Package power limit notification (total events = 376) Nov 7 10:15:15 someserver [.crit] cpu5: Package power limit notification (total events = 376) Nov 7 10:15:15 someserver [.crit] cpu7: Core power limit notification (total events = 369) Nov 7 10:15:15 someserver [.crit] cpu19: Core power limit notification (total events = 369) Nov 7 10:15:15 someserver [.crit] cpu17: Package power limit notification (total events = 377) Nov 7 10:15:15 someserver [.crit] cpu9: Core power limit notification (total events = 369) Nov 7 10:15:15 someserver [.crit] cpu21: Core power limit notification (total events = 369) Nov 7 10:15:15 someserver [.crit] cpu23: Core power limit notification (total events = 369) Nov 7 10:15:15 someserver [.crit] cpu11: Core power limit notification (total events = 369) Nov 7 10:15:15 someserver [.crit] cpu13: Package power limit notification (total events = 376) Nov 7 10:15:15 someserver [.crit] cpu7: Package power limit notification (total events = 375) Nov 7 10:15:15 someserver [.crit] cpu19: Package power limit notification (total events = 375) Nov 7 10:15:15 someserver [.crit] cpu9: Package power limit notification (total events = 374) Nov 7 10:15:15 someserver [.crit] cpu21: Package power limit notification (total events = 375) Nov 7 10:15:15 someserver [.crit] cpu23: Package power limit notification (total events = 374)
一个小小的谷福显示,这通常与cpu运行热,或电压调节有关.我不认为这是发生了什么.群集中所有服务器的温度传感器运行正常,iDRAC中禁用了功率上限策略,并且我的系统配置文件在所有这些服务器上都设置为“性能”：
# omreport chassis biossetup | grep -A10 'System Profile' System Profile Settings ------------------------------------------ System Profile : Performance cpu Power Management : Maximum Performance Memory Frequency : Maximum Performance Turbo Boost : Enabled C1E : disabled C States : disabled Monitor/Mwait : Enabled Memory Patrol Scrub : Standard Memory Refresh Rate : 1x Memory Operating Voltage : Auto Collaborative cpu Performance Control : disabled
> A Dell mailing list post几乎完美地描述了症状.戴尔建议作者尝试使用性能配置文件,但这没有帮助.他最终在Dell’s guide for configuring a server for low latency environments中应用了一些设置,其中一个设置(或其组合)似乎解决了问题.
> Kernel.org bug #36182注意到默认情况下启用了功率限制中断调试,这会导致cpu电压调节进入的情况下性能下降.
> A RHN KB article(需要RHN登录)提到影响PE r620和r720服务器未运行性能配置文件的问题,并建议更新两周前发布的内核. …除了我们正在运行性能配置文件…
我在网上找到的所有东西都在这里让我圈起来.到底发生了什么事？
解决方法
这不是导致性能问题的电压调节,而是由它触发的调试内核中断.
尽管有一些关于Redhat的错误信息,但所有链接页面都指的是同一现象.电压调节在有或没有性能配置文件的情况下发生,可能是由于启用了Turbo Boost功能.无论原因如何,这些电压波动与内核2.6.32-358.18.1.el6.x86_64中默认启用的功率限制内核中断的交互性很差.
确认的解决方法：
>升级到最近发布的Redhat内核(2.6.32-358.23.2.el6)会禁用此调试并消除性能问题.
>将以下内核参数添加到grub.conf将禁用PLN：clearcpuid = 229
片状解决方法：
>设置“性能”的系统配置文件.这本身不足以在我们的服务器上禁用PLN.你的旅费可能会改变.
不好的解决方法：
>将ACPI相关模块列入黑名单.我在一些论坛帖子中看到了这一点.不明智,所以don’t.
