Restoring Vote disk due to ASM disk failures1.docx

Restoring Vote disk due to ASM disk failures1.docx

《Restoring Vote disk due to ASM disk failures1.docx》由会员分享，可在线阅读，更多相关《Restoring Vote disk due to ASM disk failures1.docx（13页珍藏版）》请在冰豆网上搜索。

RestoringVotediskduetoASMdiskfailures1

RestoringVotediskduetoASMdiskfailures-1

RestoringOCRduetoASMdiskfailures wasbloggedpreviously.ThisbloglooksatrestoringvotedisksduetoASMdiskfailures.

SomeimportantpointersfromClusterwareAdminGuide."OCRandvotingdiskscanbestoredinOracleAutomaticStorageManagement（OracleASM）.TheOracleASMpartnershipandstatustable（PST）isreplicatedonmultipledisksandisextendedtostoreOCR.Consequently,OCRcantoleratethelossofthesamenumberofdisksasareintheunderlyingdiskgroupandberelocatedinresponsetodiskfailures.

OracleASMreservesseveralblocksatafixedlocationoneveryOracleASMdiskforstoringthevotingdisk.Shouldthediskholdingthevotingdiskfail,OracleASMselectsanotherdiskonwhichtostorethisdata.StoringOCRandthevotingdiskonOracleASMeliminatestheneedforthird-partyclustervolumemanagersandeliminatesthecomplexityofmanagingdiskpartitionsforOCRandvotingdisksinOracleClusterwareinstallations.

TheddcommandsusedtobackupandrecovervotingdisksinpreviousversionsofOracleClusterwarearenotsupportedinOracleClusterware11grelease2（11.2）.

VotingdiskmanagementrequiresavalidandworkingOCR.Beforeyouadd,delete,replace,orrestorevotingdisks,runtheocrcheckcommandasroot.IfOCRisnotavailableoritiscorrupt,thenyoumustrestoreOCR.

ThevotingdiskdataisautomaticallybackedupinOCRaspartofanyconfigurationchangeandisautomaticallyrestoredtoanyvotingdiskadded.

Ifallofthevotingdisksarecorrupted,thenRestoreOCR.ThisstepisnecessaryonlyifOCRisalsocorruptedorotherwiseunavailable,suchasifOCRisonOracleASMandthediskgroupisnolongeravailable.

ThenumberofvotingfilesyoucanstoreinaparticularOracleASMdiskgroupdependsupontheredundancyofthediskgroup.

Externalredundancy:

Adiskgroupwithexternalredundancycanstoreonlyonevotingdisk

Normalredundancy:

Adiskgroupwithnormalredundancystoresthreevotingdisks

Highredundancy:

Adiskgroupwithhighredundancystoresfivevotingdisks

Bydefault,OracleASMputseachvotingdiskinitsownfailuregroupwithinthediskgroup.AnormalredundancydiskgroupmustcontainatleasttwofailuregroupsbutifyouarestoringyourvotingdisksonOracleASM,thenanormalredundancydiskgroupmustcontainatleastthreefailuregroups."（OracleClusterwareAdminGuide）

Followingerrorwillbethrownifadiskgroupwithtwofailuregroupswasusedtostorethevotedisks

crsctlreplacevotedisk+CLUSTERDG

FailedtocreatevotingfilesondiskgroupCLUSTERDG.

Changetoconfigurationfailed,butwassuccessfullyrolledback.

CRS-4000:

CommandReplacefailed,orcompletedwitherrors.

OntheASMlog

ThuAug1915:

34:

292010

NOTE:

updatedgpnpprofileASMdiskstring:

ORCL:

*

ThuAug1915:

34:

292010

NOTE:

CreatingvotingfilesindiskgroupCLUSTERDG

ThuAug1915:

34:

292010

NOTE:

VotingFilerefreshpendingforgroup1/0x23d6dd4（CLUSTERDG）

NOTE:

AttemptingvotingfilecreationindiskgroupCLUSTERDG

NOTE:

votingfileallocationongrp1diskCLUS2

NOTE:

votingfileallocationongrp1diskCLUS3

ERROR:

VotingfileallocationfailedforgroupCLUSTERDG

Errorsinfile/opt/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_20421.trc:

ORA-15273:

Couldnotcreatetherequirednumberofvotingfiles.

NOTE:

VotingfilerelocationisrequiredindiskgroupCLUSTERDG

NOTE:

AttemptingvotingfilerelocationondiskgroupCLUSTERDG

NOTE:

votingfiledeletionongrp1diskCLUS2

NOTE:

votingfiledeletionongrp1diskCLUS3

"Ahighredundancydiskgroupmustcontainatleastthreefailuregroups.However,Oraclerecommendsusingseveralfailuregroups.Asmallnumberoffailuregroups,orfailuregroupsofunevencapacity,cancreateallocationproblemsthatpreventfulluseofalloftheavailablestorage.

Youmustspecifyenoughfailuregroupsineachdiskgrouptosupporttheredundancytypeforthatdiskgroup.

NeithershouldyouaddavotingdisktoaclusterfilesysteminadditiontothevotingdisksstoredinanOracleASMdiskgroup.OracledoesnotsupporthavingvotingdisksinOracleASManddirectlyonaclusterfilesystemforthesameclusteratthesametime."（OracleClusterwareAdminGuide）

Scenario1.

1.OnlyVotedisksareinASMdiskgroup

2.ASMdiskgrouphasnormalredundancywithonlythreefailuregroups

3.Onlyonefailuregroupisaffected

4.OCRislocatedinaseparatelocation（inanotherdiskgrouporblockdevicelocation-notsupportedbyOracleonlyvalidduringmigrationcouldbemovedtoafterinstallationfortestingpurposes）

AllthatisrequiredinthisscenarioistodroptheaffecteddiskfromtheASMdiskgroup,repairitandadditbacktothediskgroup.

1.ItisassumedthatvotedisksarealreadyinASMdiskgroup,ifnot movetoASMdiskgroup.Currentvotediskconfigurationisasfollows

crsctlquerycssvotedisk

##STATEFileUniversalIdFileNameDiskgroup

------------------------------------------

1.ONLINE393aebdd039e4fdabf242bf461c45136（ORCL:

CLUS1）[CLUSTERDG]

2.ONLINE04daf60741d34f9abf22e76fe22913b2（ORCL:

CLUS2）[CLUSTERDG]

3.ONLINE2f38bb8318d34f1bbf49d28f6e400f60（ORCL:

CLUS3）[CLUSTERDG]

Located3votingdisk（s）.

2.Corruptoneofthediskstosimulatediskfailure

/etc/init.d/oracleasmquerydisk-pCLUS1

Disk"CLUS1"isavalidASMdisk

/dev/sdc2:

LABEL="CLUS1"TYPE="oracleasm"

ddif=/dev/zeroof=/dev/sdc2count=20480bs=8192

20480+0recordsin

20480+0recordsout

167772160bytes（168MB）copied,0.162996seconds,1.0GB/s

3.ThishasnoeffectontheOCR.InthiscaseOCRisstoredinablockdevice（notsupportedbyOracle）

ocrcheck

StatusofOracleClusterRegistryisasfollows:

Version:

3

Totalspace（kbytes）:

148348

Usedspace（kbytes）:

4580

Availablespace（kbytes）:

143768

ID:

552644455

Device/FileName:

/dev/sdc5

Device/Fileintegritychecksucceeded

Device/Filenotconfigured

Device/Filenotconfigured

Device/Filenotconfigured

Device/Filenotconfigured

Clusterregistryintegritychecksucceeded

Logicalcorruptionchecksucceeded

4.GridInfrastructurealertlog（$CRS_HOME/log/`hostname-s`/alert`hostname-s`.log）showsthefollowingentryaftersometime

2010-08-1915:

51:

56.537

[cssd（25119）]CRS-1605:

CSSDvotingfileisonline:

ORCL:

CLUS1;detailsin/opt/app/11.2.0/grid/log/hpc1/cssd/ocssd.log.

5.OCSSDloglistthefollowing

2010-08-1915:

51:

44.790:

[CSSD][1230346560]clssnmSendingThread:

sendingstatusmsgtoallnodes

2010-08-1915:

51:

44.790:

[CSSD][1230346560]clssnmSendingThread:

sent5statusmsgstoallnodes

....

2010-08-1915:

51:

49.790:

[CSSD][1230346560]clssnmSendingThread:

sendingstatusmsgtoallnodes

2010-08-1915:

51:

49.790:

[CSSD][1230346560]clssnmSendingThread:

sent5statusmsgstoallnodes

2010-08-1915:

51:

53.513:

[CSSD][1303775552]clssnmvDiskKillCheck:

votingdiskcorrupted（0x00000000,0x00000000）（ORCL:

CLUS1）

2010-08-1915:

51:

53.513:

[CSSD][1303775552]clssnmvDiskAvailabilityChange:

votingfileORCL:

CLUS1nowoffline

2010-08-1915:

51:

53.816:

[CLSF][1366714688]Closinghandle:

0x1caa6570

2010-08-1915:

51:

53.816:

[SKGFD][1366714688]Lib:

ASM:

/opt/oracle/extapi/64/asm/orcl/1/libasm.so:

closinghandle0x1ccf45c0for

disk:

ORCL:

CLUS1:

2010-08-1915:

51:

54.515:

[CLSF][1303775552]Closinghandle:

0x1cd496d0

2010-08-1915:

51:

54.515:

[SKGFD][1303775552]Lib:

ASM:

/opt/oracle/extapi/64/asm/orcl/1/libasm.so:

closinghandle0x1cd48e80for

disk:

ORCL:

CLUS1:

2010-08-1915:

51:

54.516:

[CLSF][1324755264]Closinghandle:

0x1c958930

2010-08-1915:

51:

54.516:

[SKGFD][1324755264]Lib:

ASM:

/opt/oracle/extapi/64/asm/orcl/1/libasm.so:

closinghandle0x1cac17c0for

disk:

ORCL:

CLUS1:

2010-08-1915:

51:

54.790:

[CSSD][1230346560]clssnmSendingThread:

sendingstatusmsgtoallnodes

2010-08-1915:

51:

54.790:

[CSSD][1230346560]clssnmSendingThread:

sent5statusmsgstoallnodes

2010-08-1915:

51:

54.799:

[CSSD][1146427712]clssnmvSchedDiskThreads:

DiskPingThreadforvotingfileORCL:

CLUS1scheddelay2300>margin1500cur_ms527387044lastalive527384744

2010-08-1915:

51:

56.523:

[CSSD][1324755264]clssnmvDiskOpen:

OpeningORCL:

CLUS1

2010-08-1915:

51:

56.523:

[SKGFD][1324755264]Handle0x1cac17c0fromlib:

ASM:

/opt/oracle/extapi/64/asm/orcl/1/libasm.so:

fordisk:

ORCL:

CLUS1:

2010-08-1915:

51:

56.523:

[CLSF][1324755264]Openedhdl:

0x1ca4d320fordev:

ORCL:

CLUS1:

2010-08-1915:

51:

56.533:

[CSSD][1324755264]clssnmvStatusBlkInit:

myinfonodenamehpc1,uniqueness1281715015

2010-08-1915:

51:

56.533:

[CSSD][1324755264]clssnmvDiskAvailabilityChange:

votingfileORCL:

CLUS1nowonline

2010-08-1915:

51:

56.533:

[SKGFD][1366714688]Handle0x1cd48e80fromlib:

ASM:

/opt/oracle/extapi/64/asm/orcl/1/libasm.so:

fordisk:

ORCL:

CLUS1:

2010-08-1915:

51:

56.533:

[CLSF][1366714688]Openedhdl:

0x1cba4ac0fordev:

ORCL:

CLUS1:

2010-08-1915:

51:

57.524:

[SKGFD][1303775552]Handle0x1ccf45c0fromlib:

ASM:

/opt/oracle/extapi/64/asm/orcl/1/libasm.so:

fordisk:

ORCL:

CLUS1:

2010-08-1915:

51:

57.524:

[CLSF][1303775552]Openedhdl:

0x1cd496d0fordev:

ORCL:

CLUS1:

2010-08-1915:

51:

59.790:

[CSSD][1230346560]clssnmSendingThread:

sendingstatusmsgtoallnodes

2010-08-1915:

51:

59.790:

[CSSD][1230346560]clssnmSendingThread:

sent5statusmsgstoallnodes

2010-08-1915:

52:

03.403:

[CSSD][1146427712]clssscSelect:

cookieacceptrequest0x1c75c298

2010-08-1915:

52:

03.403:

[CSSD][1146427712]clssgmAllocProc:

（0x2aaab421d380）allocated

2010-08-1915:

52:

03.403:

[CSSD][1146427712]clssgmClientConnectMsg:

propertiesofcmProc0x2aaab421d380-1,2,3,4

2010-08-1915:

52:

03.403:

[CSSD][1146427712]clssgmClientConnectMsg:

Connectfromcon（0x41ee9f）proc（0x2aaab421d380）pid（21407）

version11:

2:

1:

4,properties:

1,2,3,4

2010-08-1915:

52:

03.403:

[CS