低延迟测试方案.docx
- 文档编号:5282820
- 上传时间:2022-12-14
- 格式:DOCX
- 页数:29
- 大小:78.98KB
低延迟测试方案.docx
《低延迟测试方案.docx》由会员分享,可在线阅读,更多相关《低延迟测试方案.docx(29页珍藏版)》请在冰豆网上搜索。
低延迟测试方案
低延迟测试方案
初稿
<徐彬彬>
修订
<徐彬彬>
日期
2013年06月
文档控制
修改记录
日期
作者
版本
修改记录
2013-06-28
徐彬彬
1.0
创建文档
2013-07-26
徐彬彬
1.1
升级驱动固件后的内容更新,优化内容描述补充
2013-08-08
徐彬彬
1.2
确定netperf和sockperf为测试工具,确定测试方法
审阅
分发
拷贝No.
姓名
单位
一:
整体系统环境介绍
目前硬件:
HPDL380PGen8*2
操作系统
Rhel5.8X64
RBSU
12/14/2012
网卡固件
5719-v1.31n\NCSIv1.1.15.0
网卡驱动
tg33.119
cciss版本
3.6.28
CPU数量
2
内核数量
每个CPU8core总计16cores
更新后
操作系统
Rhel5.8X64
RBSU
12/14/2012
网卡固件
5719-v1.31n\NCSIv1.1.15.0
网卡驱动
tg33.125g
cciss版本
3.6.28-22
CPU数量
2
内核数量
每个CPU8core总计16cores
测试将会采用直连以及交换机互联两种模式进行。
二.服务操作系统安装/运行状态
目前服务器安装使用ks脚本ks-hp-ctp_rhel5.8.cfg
安装软件包列表为
@admin-tools
@base
@chinese-support
@core
@development-libs
@development-tools
@editors
@gnome-desktop
@graphical-internet
@legacy-software-development
@system-tools
@text-internet
@base-x
kexec-tools
iscsi-initiator-utils
fipscheck
device-mapper-multipath
sgpio
python-dmidecode
imake
emacs
audit
xorg-x11-utils
xorg-x11-server-Xnest
xorg-x11-server-Xvfb
-nspluginwrapper
分区状况为
clearpart--all--drives=cciss/c0d0
part/boot--fstypeext3--size=100--ondisk=cciss/c0d0
partpv.100000--size=100--grow--ondisk=cciss/c0d0
volgroupvg00--pesize=32768pv.100000
logvolswap--fstypeswap--name=lv_swap--vgname=vg00--size=32768
logvol/--fstypeext3--name=lvol0--vgname=vg00--size=51200
logvol/var--fstypeext3--name=lvol1--vgname=vg00--size=10240
logvol/usr--fstypeext3--name=lvol2--vgname=vg00--size=5120
logvol/tmp--fstypeext3--name=lvol3--vgname=vg00--size=5120
已禁用服务列表
chkconfiganacronoff
chkconfigarptables_jfoff
chkconfigatdoff
chkconfigautofsoff
chkconfigcupsoff
chkconfigcups-config-daemonoff
chkconfigfirstbootoff
chkconfighaldaemonoff
chkconfigisdnoff
chkconfigkudzuoff
chkconfignetfsoff
chkconfignfslockoff
chkconfigpcmciaoff
chkconfigportmapoff
chkconfigrpcgssdoff
chkconfigrpcidmapdoff
chkconfigsendmailoff
chkconfigsmartdoff
chkconfigsnmpdoff
chkconfigavahi-daemonoff
参数为默认值
default=0
timeout=5
splashimage=(hd0,0)/grub/.gz
hiddenmenu
titleRedHatEnterpriseLinuxServer(2.6.5)
root(hd0,0)
kernel/vmlinuz-2.6.5roroot=/dev/vg00/lvol0rhgbquiet
initrd/initrd-2.6.5.img
titleOther
rootnoverify(hd1,0)
chainloader+1
为默认未修改
#KernelsysctlconfigurationfileforRedHatLinux
#
#Forbinaryvalues,0isdisabled,1isenabled.Seesysctl(8)and
#(5)formoredetails.
#ControlsIPpacketforwarding
4.ip_forward=0
#Controlssourcerouteverification
4.lt.rp_filter=1
#Donotacceptsourcerouting
4.lt.accept_source_route=0
#ControlstheSystemRequestdebuggingfunctionalityofthekernel
=0
#ControlswhethercoredumpswillappendthePIDtothecorefilename
#Usefulfordebuggingmulti-threadedapplications
_uses_pid=1
#ControlstheuseofTCPsyncookies
4.tcp_syncookies=1
#Controlsthemaximumsizeofamessage,inbytes
b=65536
#Controlsthedefaultmaxmimumsizeofamesagequeue
x=65536
#Controlsthemaximumsharedsegmentsize,inbytes
x=
#Controlsthemaximumnumberofsharedmemorysegments,inpages
l=
三.优化内容
RBSU优化
当前RBSU设定为
参数
当前值
建议值
IntelVirtualizationTechnology
Enable
Disabled
HT
Disable
Disabled
Turbomode
Enable
Disabled
VT-d
Disable
Disabled
Thermalconfiguration
Maxcooling
Maxcooling
HPPowerprofile
Maximumperformance
Maximumperformance
HPPowerRegulator
StaticHighperformancemode
StaticHighperformancemode
QPILinkpowermgmt.
Disable
Disable
MinimumProcessorIdlePowerCoreState
Noc-state
Noc-state
MinimumProcessorIdlePowerPackageState
Nop-state
Nop-state
Energy/PerformanceBias
MaxPerformance
MaxPerformance
CollaborativePowerControl
Enable
Disabled
DIMMVoltagePreference
Optimizedforperformance
Optimizedforperformance
DynamicPowerCappingFunctionality
Enable
Disabled
MemoryPowerSavingsMode
Balanced
MaximumPerformance
ACPISLITPerferences
Disable
Enabled
ProcessorPowerandUtilMonitoring
Enable
Disabled
MemoryPre-failureNotification
Enable
Disabled
MemoryRefreshrate
2x
1x
关闭服务建议
按照ConfigurationandTuningHPProliantServersforLow-LatencyApplicationsWhitePaper中的建议
目前开启的服务中有如下服务可以关闭
acpid0:
off1:
off2:
on3:
on4:
on5:
on6:
off
autofs0:
off1:
off2:
off3:
on4:
on5:
on6:
off
avahi-daemon0:
off1:
off2:
off3:
on4:
on5:
on6:
off
bluetooth0:
off1:
off2:
on3:
on4:
on5:
on6:
off
cpuspeed0:
off1:
on2:
on3:
on4:
on5:
on6:
off
crond0:
off1:
off2:
on3:
on4:
on5:
on6:
off
cups0:
off1:
off2:
on3:
on4:
on5:
on6:
off
firstboot0:
off1:
off2:
off3:
on4:
off5:
on6:
off
hidd0:
off1:
off2:
on3:
on4:
on5:
on6:
off
ip6tables0:
off1:
off2:
on3:
on4:
on5:
on6:
off
iscsi0:
off1:
off2:
off3:
on4:
on5:
on6:
off
iscsid0:
off1:
off2:
off3:
on4:
on5:
on6:
off
kudzu0:
off1:
off2:
off3:
on4:
on5:
on6:
off
lvm2-monitor0:
off1:
on2:
on3:
on4:
on5:
on6:
off
mcstrans0:
off1:
off2:
on3:
on4:
on5:
on6:
off
mdmonitor0:
off1:
off2:
on3:
on4:
on5:
on6:
off
messagebus0:
off1:
off2:
off3:
on4:
on5:
on6:
off
netfs0:
off1:
off2:
off3:
on4:
on5:
on6:
off
pcscd0:
off1:
off2:
on3:
on4:
on5:
on6:
off
readahead_early0:
off1:
off2:
on3:
on4:
on5:
on6:
off
readahead_later0:
off1:
off2:
off3:
off4:
off5:
on6:
off
restorecond0:
off1:
off2:
on3:
on4:
on5:
on6:
off
rhnsd0:
off1:
off2:
off3:
on4:
on5:
on6:
off
rpcgssd0:
off1:
off2:
off3:
on4:
on5:
on6:
off
rpcidmapd0:
off1:
off2:
off3:
on4:
on5:
on6:
off
sendmail0:
off1:
off2:
on3:
on4:
on5:
on6:
off
smartd0:
off1:
off2:
on3:
on4:
on5:
on6:
off
sysstat0:
off1:
off2:
on3:
on4:
off5:
on6:
off
xfs0:
off1:
off2:
on3:
on4:
on5:
on6:
off
yum-updatesd0:
off1:
off2:
on3:
on4:
on5:
on6:
off
内核参数调整
4.tcp_timestamps=1改为0
描述:
关闭TCP时间戳,减少包头的11字节。
作用:
每个tcp包都会有一个发送和回包的时间戳,用以防止tcp包在回包延迟过大时导致包乱序。
在速度较慢的广域网中该功能应该开启。
在响应速度较好的局域网内,可以关闭该功能。
4.tcp_sack=1改为0
描述:
selectiveacks,ack包选择性确认,使TCP只重新发送丢失的包,不用发送后续所有的包
作用:
A节点向B节点发送10000个数据包,由于网络原因第3000~5000的2000个包传输失败。
Sack会检测到这2000个包传送失败,并请求A节点重发第3000~5000个包。
该功能关闭的状态下则会重发10000个包。
4.tcp_low_latency=1
描述:
TCP协议栈的决策,更喜欢更低的延迟而不是更高的吞吐量
作用:
rhel在运行时更倾向于降低延迟或者更高吞吐。
0为更高吞吐,1为更低延迟。
增加TCP最大和默认的buffer
.rmem_max=
.wmem_max=
.rmem_default=
.wmem_default=
.optmem_max=
增加TCPbuffer最小,默认和最大值限制
4.tcp_wmem="409665536"
4.tcp_rmem="409687380"
关闭nagle算法,没有系统层面开关,需要在建立连接时定义TCP_NODELAY(默认没有TCP_NODELAY)
调整网卡
使用ping–c
得到这样的输出
说明MTU过大,需要逐步减少-s后的包大小
直到找到合适的大小能正常发包后
普通网络MTU值在1500,测试得到恰当值后用ifconfigethXmtu
或者在/etc/sysconfig/network-script/ifcfg-ethx中增加MTU=
网卡驱动参数调整:
tg3在3.119中没有参数,在3.125g中有参数可供调整。
parm:
tg3_disable_eee:
DisableEnergyEfficientEthernet(EEE)support(int)
在/etc/中增加一行optionstg3tg3_disable_eee=1
作用:
禁用网卡节能,在进入和退出节能状态时网卡传输效率会降低,所以关闭EEE支持。
Grub修改
在/boot/grub/中的kernel行增加nosoftlockupintel__cstate=0mce=ignore_ce
nosoftlockup防止RHEL记录一个高优先级的线程在一个核上执行时间超过softlockup阈值
intel_idle_max_cstate=0防止操作系统override在RBSU中设定的nocstate
mce=ignore_ce忽略每五分钟MachineCheck对于可修正错误的初始化,这可能会导致网络延迟的毛刺
四.测试内容
NETPERF
在两个服务器上安装nerperf
从,最新版本为2.60
Compilation
./configure--enable-intervals--enable-histogram--enable-burst
make
makeinstall
两台服务器一台作为Server,另一台作为Client
1.Server端执行netserver-D–f
客户端执行:
TCPlatency测试:
netperf-n16-H
TCP吞吐量测试:
netperf-n16-H
UDPlatency测试:
netperf-n16-H
UDP吞吐量测试:
netperf-n16-H1.1.1.1-c-C-tUDP_STREAM-l120-T2,2-v2---m64-kP50_LATENCY,P90_LATENCY,P99_LATENCY,MEAN_LATENCY,STDDEV_LATENCY,MAX_LATENCY
可用参数:
-c声明在测试过程中包括连接的establishment和tear-down过程
-d
可用值有:
2netperfsendtonetserver
4netserversendtonetperf
6request/responsetest
-DTCP_NODELAY关闭nagle算法
-k选取keyvalue作为输出
例子中的值释义为
MIN_LATENCY
Theminimum“latency”oroperationtime(send,receiveorrequest/responseexchangedependingonthetest)asmeasuredonthenetperfsidewhentheglobal-joptionwasspecified.Units:
Microseconds.
MAX_LATENCY
Themaximum“latency”oroperationtime(send,receiveorrequest/responseexchangedependingonthetest)asmeasuredonthenetperfsidewhentheglobal-joptionwasspecified.Units:
Microseconds.
P50_LATENCY
The50thpercentilevalueof“latency”oroperationtime(send,receiveorrequest/responseexchangedependingonthetest)asmeasuredonthenetperfsidewhentheglobal-joptionwasspecified.Units:
Microseconds.
P90_LATENCY
The90thpercentilevalueof“latency”oroperationtime(send,receiveorrequest/responseexchangedependingonthetest)asmeasuredonthenetperfsidewhentheglobal-joptionwasspecified.Units:
Microseconds.
P99_LATENCY
The99thpercentilevalueof“latency”oroperationtime(send,receiveorrequest/responseexchangedependingonthetest)asmeasuredonthenetperfsidewhentheglobal-joptionwasspecified.Units:
Microseconds.
MEAN_LATENCY
Theaverage“latency”oroperationtime(send,receiveorrequest/responseexchangedependingonthetest)asmeasuredonthenetperfsidewhentheglobal-joptionwasspecified.Units:
Microseconds.
STDDEV_LATENCY
Thestandarddeviationof“latency”oroperationtime(send,receiveorrequest/responseexchangedependingonthetest)asmeasuredonthenetperfsidewhentheglobal-joptionwasspecified.Units:
Microseconds.
-t指定socket类型为stream(数据流)或dgram(数据报),stream基于TCP,dgram基于
- 配套讲稿:
如PPT文件的首页显示word图标,表示该PPT已包含配套word讲稿。双击word图标可打开word文档。
- 特殊限制:
部分文档作品中含有的国旗、国徽等图片,仅作为作品整体效果示例展示,禁止商用。设计者仅对作品中独创性部分享有著作权。
- 关 键 词:
- 延迟 测试 方案