DDN Exascaler文件系统安装手册.docx

DDN Exascaler文件系统安装手册.docx

《DDN Exascaler文件系统安装手册.docx》由会员分享，可在线阅读，更多相关《DDN Exascaler文件系统安装手册.docx（14页珍藏版）》请在冰豆网上搜索。

DDNExascaler文件系统安装手册

DDNExascaler文件系统安装报告

1.DDN存储系统架构

硬件设备构成：

DDNSFA7700存储盘阵：

✓1台DDNSFA7700主盘柜，每个主盘柜带一个SFA7000副盘柜并配置2个DDNSFA8460副盘柜；

✓DDNSFA7700主盘柜配置有2个控制器，每个控制器有2个56Gb/sFQRInfiniband接口与两台IO节点直连；

✓168块4TB7,200RPM6Gb/sSAS硬盘；

✓1台EF3015存储，用作文件系统元数据。

IO节点基本配置：

✓共计2台IO节点；

✓每台IO节点2个IntelCPU（Intel（R）Xeon（R）CPUE5-26200@2.00GHz）

✓每台IO节点64GBMemory

✓每台IO节点通过一个40Gb万兆卡与交换机相连，

✓每台IO节点通过一块双端口HBA与EF3015存储相连。

2.EXASCALER部署前准备:

存储系统：

数据存储盘阵和元数据存储盘阵必须配置完成，保障数据存储系统配置性能最优化，如关闭DDN存储系统Read-ahead或预取功能；每个MDS/OSS到存储系统链路冗余以保障存储系统的高可靠性；每个OSS建议挂载想同数量的OST以达到存储系统的负载均衡目的。

服务器端：

-服务器端建议关闭防火墙，SELinux选项；

-IO服务器之间配置SSH无密码访问；

-IO服务器端需通过NTP服务与Clustre之间进行时间同步；

-IO节点配置用户组与Cluster集群间进行同步；

-服务器配置IPMI管理。

3.EXASCALER部署安装:

Server端安装配置

EXASCALER是DDN集成在IOSERVER操作系统中，所有安装部署都通过脚本及配置文件实现。

中石化项目配置文件文件如下：

#ExaScalerclusterconfigurationfile.

#CopyrightDDN,2010,2011

#Fornewinstallspleaseconsiderusingaexampleconfigfilemore

#suitedtotheindividualinstall,thereare3inthisdirectorynamed:

#exascaler-traditional.conf

#exascaler-10ke.conf

#exascaler-10ke-external-mds.conf

#Therearemoredetailedcommentsandsettingsshowninthisfilefor

#reference.

[global]

#Youcanprovideashadowconfigurationfile,inordertostoreallyourpasswords.

#Thees_showallcommanduploadsalltheneededfilestoDDN'sFTPserver,whichalso

#containstheexascaler.conffile.Forsecuritypurposeyoucanmentionallyourpasswordsinto

#theshadowconfigurationfile.Shadowfileis/etc/ddn/exascaler-shadow.confand

#itshouldhavereadandwritepermissionsonlyfortheuser.Exampleoftheshadowfilecanbe

#foundat/opt/ddn/example/exascaler-shadow.conf.Toenablethisbehavior,enablethissetting

;shadow_conf:

no

#Listoffilesystemsexportedfromthiscluster（spacedelimited）

fs_list:

ddnfs

#Listofemailstoreceivelogging（commadelimited）

#;email_list:

TestUser

#;email_domain:

#;email_relay:

#Extrahostswhichareknownaboutbutaren'tusedtoservelustre.Hosts

#listeherearemanagedbythesametools,arepartoftheheartbeatcluster

#andareincludedinpdshhostlists.'start'hostsarehostswho'sIP'scome

#beforethefshosts,'end'hostsarehostswho'sip'scomeafter

;extra_hosts_start:

head

;extra_hosts_end:

client0client1

#ListofS2Acontrollersthatarepartofthecluster

#ThisisusedtoretrievelogsfromeachS2Acontroller

#Thelistshouldbeaspace-separatedlistofhostnamesorIPaddresses,with

#thecontrollerslistedaspairs;thefirstcontrollerfollowedbythesecond

#;s2a_list:

s2a1-c1s2a1-c2s2a2-c1s2a2-c2

#;s2a_user:

admin

#;s2a_pass:

password

#ListofSFAcontrollersthatarepartofthecluster.

#Formatisass2a_list,controllersareassumedtobelistedinpairs.

sfa_list:

192.168.40.186192.168.40.187

#PasswordforSFA"user"

#defaultsto:

user

sfa_password:

user

#Thedirectorywherelogswillbesavedto.

#Defaultsto/scratch/logandisalwaysappendedbythefilesystemname.

#e2fsck/lfsckdatabasesandlogswillbesavedinthesubdirectory${fs_name}/fsck

#Largefilesarecreatedinthisdirectorysocareshouldbetakenchosingit.

#defaultsto:

/scratch/log

log_dir:

/scratch/log

#Ifpingdisenabled,itwilldetectnetworkfailuresandstopresourceson

#thehostwithnetworkissues.MonitorednetworksarethoseusedbytheMDS.

#Givenbylustreinternalconstraints,allOSSareforcedtosupportthe

#networksoftheMDS.However,inrarecasesanOSSalsomighthave

#additionalnetworktoclients,butnottotheMDS（e.g.tcponMDS,

#buttcp+o2ibonOSS）.Inthatcaseonlythenetworksavailable

#ontheMDSwillbemonitored.Untilfurthertested,pingddefaultsto'yes'.

pingd:

yes

#FrequencyofautomaticMDTbackups

#Canbedaily,weekly,ornone

;mdt_backup:

none

#Pathwherekdumpwillcreatethecrashdump

#Thisisoptionalparameteranddefaultsto/scratch/crash

;kdump_path:

/scratch/crash

#Bydefaultallthevolumegroupsareactivatedatnodereboot.Butifyouareusing

#HAthevolumegroupsarebeingmanagedbyHAandsoweshouldexcludethemfrom

#theautoactivationlistforvolumegroups.Settingthisto'auto'will

#automaticallydetectwhichvolumegroupstoautoactivate.Youcanalsoexplicitlyspecify

#thelistofvolumegroupslike'vg_activation_list:

VolGrp0000VolGrp0001'.

#Settingthisto'none'willnotaddanythingtoautoactivationlistwhichmeansall

#thevolumegroupswillbeactivatedonreboot.Thedefaultvalueis'auto'.

#Use'none'optiontosticktotheoldbehavior.

;vg_activation_list:

auto

[HA]

#WhattypeofHAweareusingonthissystem.

#Validoptionsare:

#none.NoHAconfigured

#heartbeat.rhel5systemsusingheartbeat

#corosync.rhel6systemsusingcorosync

#chroma.rhel6systemsusingcorosync,stonithonly.

#Fornewsystemsismostlikelyselectionhereis'corosync'althoughtosupport

#legacysystemsthedefaultifnotsetis'heartbeat'

type:

corosync

#Whichtransporttypetouse（onlyforcorosync）.

#Validoptionsare:

#multicast.Usemulticasttocommunicate

#unicast.Useunicast.

#Thedefaultismulticast.

;transport:

multicast

#Allowdisablingofpacemakerquorumpolicy.Bydefaultitshouldbeenabled

#andsetto'stop',butitMUSTbedisabledonsystemswith2hostsonly

#orontheSFA10000Eifonlyasinglecoupletisused,as

#thenonlyn/2systemswillbeavailableonacontrollerfailure.Butit

#shouldbeenabledagain,ifafurthersystemisadded（forexample

#externalMDS）oranotherSFA10000Ecoupletisadded.Defaultis'stop'.

#Setto'ignore'onaSFA10000Eorsystemwith2hostsonly.

#Theonlyallowedvaluesare'stop'and'ignore'

no_quorum_policy:

ignore

#Deadtime

#ThedeadtimedirectiveisusedtospecifyhowquicklyHeartbeatshould

#decidethatanodeinaclusterisdead.Settingthisvaluetoolowwill

#causethesystemtofalselydeclareitselfdead.Settingittoohigh

#willdelaytakeoverafterthefailureofanodeinthecluster.

#Thedefaultvalueis360,thisshouldbeusedforrealhardware,forthe

#10keavalueof60isappropriate

#Note:

thisisonlyusedfor'heartbeat'

;deadtime:

360

#Howoftentomonitorheartbeatresources

;monitor_interval:

600

#Multicastport.

#Ifrunningmultiplecorosyncclustersonthesamesubnetyou'llneedtochange

#thistoavoidconflicts.

;mcastport:

5405

#Corosyncnics

#Thenicstoruncorosyncringson.Corosynccurrentlyonlysupportstworings

#Ifundefined,thefirsttwonicsinnic_listwillbeused

;corosync_nics:

eth0eth4

#Startonboot

#ControliftheHAsoftwarestartsautomaticallyonnodeboot.Withthisenabled

#HAwillbothfail-overandfail-backresourcesasrequired.Toallowfail-over

#butpreventautomaticfail-backthissettingcanbesettofalsewhichwill

#startonboot

#ControliftheHAsoftwarestartsautomaticallyonnodeboot.Withthisenabled

##HAwillbothfail-overandfail-backre

;start_on_boot:

true

#HAGroups

#Corosyncandpacemakerhaveahardtimedealingwithlargeclusters.With

#theseparameters,youcansplitupaclusterintomultiplegroups.Failover

#peersshouldbeinthesamegrouporelsetheywon'tbeabletofailoverto

#eachother.

;ha_group_count:

1#changebylu

;ha_group0:

ionode01ionode02

#;ha_group1:

oss4oss5oss6oss7oss8oss9

[host_defaults]

#Pacemakerstonithtypes,validvaluesaressh,ipmi,ipmi-slow,ilo,sfa10ke

#ipmi-slowshouldbeusedfortheDRAC-Express（sharedport）IPMIcontrollers

#sshshouldonlybeusedinnon-productionenvironments

stonith_type:

ipmi

#IPMIusernameandpassword.

#Forusewith'ipmi','ipmi-slow'or'ilo'stonith_type.

stonith_user:

root

stonith_pass:

ddn

#Emulatedserialspeedforserial-over-lan.settononeifnoserialexists

;serial_speed:

115200

#Porttobeusedforserial-over-lan.Defaultis'ttyS1'

;serial_port:

ttyS1

#Hostnetworkinterfacedevices（spacedelimited）

#Thiscanbeoverriddeninhost-specificsectionsbelow

nic_list:

eth4

#Forchannel-bonding,whatmodetouse

;bonding_mode:

4

#Thefollowingoptionscanbeusedtoauto-generateIPaddresses

#foreachhostnodeinthecluster.Hostnodesaredeterminedbythe

#mds_listandoss_listoptionsineachfilesystemspecification.

#TheIPaddressesareincrementedby1,startingwithmdshostsandthen

#movingtoosshosts,foreachfilesystem.

#

#NOTE:

Donotuseastartingbaseaddressonaninterfaceifanyhost

#intheclusterwillbeassignedanaddressinthehost-specificsections

#ipmi_ip_base:

192.168.40.184

#eth0_ip_base:

198.168.40.186

eth4_netmask:

255.255.255.0

#Thisisoptionalfornowbutmaybeessentialinfuture.

ipmi_netmask:

255.255.255.0

#Usethistoaddanyadditionalnetworkoptionstoaninterface

eth4_gateway:

192.168.39.254

#eth1_netmask:

255.255.255.0

#ib0_netmask:

255.255.255.0

#Anyadditionalmodprobesettings,oneperline.LNETnetworkswillbeadded

#automatically,howeverotherLNEToptionsmaybeaddedhere.

;modprobe_cfg:

;optionsksocklndenable_irq_affinity=0

;optionslnetroutes="o2ib4711198.51.100.[231-232]@o2ib1;o2ib5130.242.72.[227-228]@o2ib1"\

;check_routers_before_use=1router_ping_timeout=250dead_router_check_interval=300live_router_check_interval=251

#Thedefaultnetworksusedbythefilesystem（spacedelimited）

#Entriesshouldbeoftheform'name（interface）'where'name'isthename

#thatwillbeassignedtothenetworkand'interface'isthedevicethe

#networkwillrunon.Ifonlyaninterfaceisspecified,thenaname

#willbeautomaticallygenerated.

lnets:

tcp（eth4）

#FileSystemOptions

#Thisisalistofoptions,onepe