【博客文章2024】RAC 19c磁盘硬件严重故障处置系列一：集群元数据和数据的备份以及磁盘硬件故障模拟
Author： Bo Tang
1. 场景描述和系统备份描述：
1.1 ASM磁盘的LUN绑定情况：
[root@station11 ~]# for i in $(oracleasm listdisks); do ls -l /dev/oracleasm/disks/$i ; done
brw-rw---- 1 grid asmadmin 8, 1 Apr 4 03:05 /dev/oracleasm/disks/RACDISK01
brw-rw---- 1 grid asmadmin 8, 17 Apr 4 03:05 /dev/oracleasm/disks/RACDISK02
brw-rw---- 1 grid asmadmin 8, 33 Apr 4 03:05 /dev/oracleasm/disks/RACDISK03
brw-rw---- 1 grid asmadmin 8, 49 Apr 4 03:05 /dev/oracleasm/disks/RACDISK04
brw-rw---- 1 grid asmadmin 8, 65 Apr 4 03:05 /dev/oracleasm/disks/RACDISK05
brw-rw---- 1 grid asmadmin 8, 81 Apr 4 03:05 /dev/oracleasm/disks/RACDISK06
brw-rw---- 1 grid asmadmin 8, 97 Apr 4 03:05 /dev/oracleasm/disks/RACDISK07
brw-rw---- 1 grid asmadmin 8, 113 Apr 4 03:05 /dev/oracleasm/disks/RACDISK08
brw-rw---- 1 grid asmadmin 8, 129 Apr 4 03:05 /dev/oracleasm/disks/RACDISK09
brw-rw---- 1 grid asmadmin 8, 145 Apr 4 03:05 /dev/oracleasm/disks/RACDISK10
brw-rw---- 1 grid asmadmin 8, 161 Apr 4 03:04 /dev/oracleasm/disks/RACDISK11
|
1.2 ASM磁盘组元数据的备份： +data磁盘组的元数据备份：
[root@station11 ~]# su - grid
Last login: Thu Apr 4 03:08:54 CST 2024 on pts/0
[grid@station11 ~]$ asmcmd
ASMCMD> md_backup /home/grid/data.txt -G data
Disk group metadata to be backed up: DATA
Current alias directory path: RACORCL/ONLINELOG
Current alias directory path: _MGMTDB/PARAMETERFILE
Current alias directory path: ASM
Current alias directory path: RACORCL/CONTROLFILE
Current alias directory path: _MGMTDB/12D2E2161B7F1288E0630BFA19ACB8F7/DATAFILE
Current alias directory path: cluster11/ASMPARAMETERFILE
Current alias directory path: _MGMTDB/12D2C496AC8DE99CE0630BFA19ACCB81/TEMPFILE
Current alias directory path: cluster11/OCRFILE
Current alias directory path: RACORCL
Current alias directory path: RACORCL/PASSWORD
Current alias directory path: cluster11/OCRBACKUP
Current alias directory path: _MGMTDB/PASSWORD
Current alias directory path: _MGMTDB/TEMPFILE
Current alias directory path: RACORCL/DATAFILE
Current alias directory path: _MGMTDB/86B638145FF37A6FE053F706E80A81F1/DATAFILE
Current alias directory path: _MGMTDB/CONTROLFILE
Current alias directory path: ASM/PASSWORD
Current alias directory path: _MGMTDB/86B638145FF37A6FE053F706E80A81F1
Current alias directory path: _MGMTDB/12D2E2161B7F1288E0630BFA19ACB8F7
Current alias directory path: cluster11
Current alias directory path: _MGMTDB/12D2E2161B7F1288E0630BFA19ACB8F7/TEMPFILE
Current alias directory path: _MGMTDB
Current alias directory path: RACORCL/TEMPFILE
Current alias directory path: _MGMTDB/12D2C496AC8DE99CE0630BFA19ACCB81
Current alias directory path: _MGMTDB/DATAFILE
Current alias directory path: _MGMTDB/ONLINELOG
Current alias directory path: RACORCL/PARAMETERFILE
|
查看/home/grid/data.txt：
[root@station11 ~]# cat /home/grid/data.txt
@diskgroup_set = (
{
'TEMPLATEINFO' => {
'21' => {
'STRIPE' => 'COARSE',
'DGNAME' => 'DATA',
'SYSTEM' => 'Y',
'TEMPNAME' => 'VOTINGFILE',
'REDUNDANCY' => 'MIRROR'
},
'13' => {
'DGNAME' => 'DATA',
'STRIPE' => 'COARSE',
'REDUNDANCY' => 'MIRROR',
'TEMPNAME' => 'BACKUPSET',
'SYSTEM' => 'Y'
},
'6' => {
'TEMPNAME' => 'KEY_STORE',
'REDUNDANCY' => 'MIRROR',
'SYSTEM' => 'Y',
'STRIPE' => 'COARSE',
'DGNAME' => 'DATA'
},
'20' => {
'TEMPNAME' => 'DUMPSET',
'REDUNDANCY' => 'MIRROR',
'SYSTEM' => 'Y',
'STRIPE' => 'COARSE',
'DGNAME' => 'DATA'
},
'12' => {
'DGNAME' => 'DATA',
'STRIPE' => 'COARSE',
'SYSTEM' => 'Y',
'TEMPNAME' => 'XTRANSPORT BACKUPSET',
'REDUNDANCY' => 'MIRROR'
},
'10' => {
'DGNAME' => 'DATA',
'STRIPE' => 'COARSE',
'SYSTEM' => 'Y',
'TEMPNAME' => 'AUTOBACKUP',
'REDUNDANCY' => 'MIRROR'
},
'9' => {
'SYSTEM' => 'Y',
'TEMPNAME' => 'XTRANSPORT',
'REDUNDANCY' => 'MIRROR',
'DGNAME' => 'DATA',
'STRIPE' => 'COARSE'
},
'7' => {
'SYSTEM' => 'Y',
'TEMPNAME' => 'FLASHBACK',
'REDUNDANCY' => 'MIRROR',
'DGNAME' => 'DATA',
'STRIPE' => 'COARSE'
},
'15' => {
'STRIPE' => 'COARSE',
'DGNAME' => 'DATA',
'SYSTEM' => 'Y',
'REDUNDANCY' => 'MIRROR',
'TEMPNAME' => 'DATAFILE'
},
'4' => {
'STRIPE' => 'COARSE',
'DGNAME' => 'DATA',
'SYSTEM' => 'Y',
'TEMPNAME' => 'AUDIT_SPILLFILES',
'REDUNDANCY' => 'MIRROR'
},
'19' => {
'STRIPE' => 'FINE',
'DGNAME' => 'DATA',
'TEMPNAME' => 'CONTROLFILE',
'REDUNDANCY' => 'HIGH',
'SYSTEM' => 'Y'
},
'8' => {
'TEMPNAME' => 'CHANGETRACKING',
'REDUNDANCY' => 'MIRROR',
'SYSTEM' => 'Y',
'STRIPE' => 'COARSE',
'DGNAME' => 'DATA'
},
'11' => {
'STRIPE' => 'COARSE',
'DGNAME' => 'DATA',
'REDUNDANCY' => 'MIRROR',
'TEMPNAME' => 'INCR XTRANSPORT BACKUPSET',
'SYSTEM' => 'Y'
},
'3' => {
'TEMPNAME' => 'DATAGUARDCONFIG',
'REDUNDANCY' => 'MIRROR',
'SYSTEM' => 'Y',
'DGNAME' => 'DATA',
'STRIPE' => 'COARSE'
},
'16' => {
'DGNAME' => 'DATA',
'STRIPE' => 'COARSE',
'SYSTEM' => 'Y',
'REDUNDANCY' => 'MIRROR',
'TEMPNAME' => 'ONLINELOG'
},
'17' => {
'SYSTEM' => 'Y',
'TEMPNAME' => 'ARCHIVELOG',
'REDUNDANCY' => 'MIRROR',
'STRIPE' => 'COARSE',
'DGNAME' => 'DATA'
},
'5' => {
'SYSTEM' => 'Y',
'REDUNDANCY' => 'MIRROR',
'TEMPNAME' => 'AUTOLOGIN_KEY_STORE',
'DGNAME' => 'DATA',
'STRIPE' => 'COARSE'
},
'1' => {
'TEMPNAME' => 'ASMPARAMETERFILE',
'REDUNDANCY' => 'MIRROR',
'SYSTEM' => 'Y',
'DGNAME' => 'DATA',
'STRIPE' => 'COARSE'
},
'0' => {
'DGNAME' => 'DATA',
'STRIPE' => 'COARSE',
'SYSTEM' => 'Y',
'REDUNDANCY' => 'MIRROR',
'TEMPNAME' => 'PARAMETERFILE'
},
'2' => {
'STRIPE' => 'COARSE',
'DGNAME' => 'DATA',
'SYSTEM' => 'Y',
'TEMPNAME' => 'OCRFILE',
'REDUNDANCY' => 'MIRROR'
},
'14' => {
'DGNAME' => 'DATA',
'STRIPE' => 'COARSE',
'SYSTEM' => 'Y',
'TEMPNAME' => 'TEMPFILE',
'REDUNDANCY' => 'MIRROR'
},
'18' => {
'STRIPE' => 'COARSE',
'DGNAME' => 'DATA',
'SYSTEM' => 'Y',
'TEMPNAME' => 'FLASHFILE',
'REDUNDANCY' => 'MIRROR'
}
},
'ATTRINFO' => {
'AU_SIZE' => '4194304',
'CELL.SPARSE_DG' => 'allnonsparse',
'COMPATIBLE.RDBMS' => '12.2.0.0.0',
'_._DIRVERSION' => '19.0.0.0.0',
'_USD.LIST_SIZE' => '4',
'COMPATIBLE.ASM' => '19.0.0.0.0'
},
'ALIASINFO' => {
'7' => {
'REFERENCE_INDEX' => '16778117',
'SYSTEM_CREATED' => 'Y',
'DGNAME' => 'DATA',
'LEVEL' => 1,
'ALIASNAME' => '_MGMTDB/12D2E2161B7F1288E0630BFA19ACB8F7'
},
'4' => {
'LEVEL' => 2,
'ALIASNAME' => '_MGMTDB/86B638145FF37A6FE053F706E80A81F1/DATAFILE',
'SYSTEM_CREATED' => 'Y',
'REFERENCE_INDEX' => '16777905',
'DGNAME' => 'DATA'
},
'2' => {
'SYSTEM_CREATED' => 'N',
'REFERENCE_INDEX' => '16777534',
'DGNAME' => 'DATA',
'LEVEL' => 0,
'ALIASNAME' => '_MGMTDB'
},
'3' => {
'SYSTEM_CREATED' => 'Y',
'REFERENCE_INDEX' => '16777852',
'DGNAME' => 'DATA',
'LEVEL' => 1,
'ALIASNAME' => '_MGMTDB/86B638145FF37A6FE053F706E80A81F1'
},
'9' => {
'SYSTEM_CREATED' => 'Y',
'REFERENCE_INDEX' => '16778223',
'DGNAME' => 'DATA',
'LEVEL' => 2,
'ALIASNAME' => '_MGMTDB/12D2E2161B7F1288E0630BFA19ACB8F7/TEMPFILE'
},
'10' => {
'LEVEL' => 0,
'ALIASNAME' => 'RACORCL',
'SYSTEM_CREATED' => 'N',
'REFERENCE_INDEX' => '16778276',
'DGNAME' => 'DATA'
},
'5' => {
'REFERENCE_INDEX' => '16777958',
'SYSTEM_CREATED' => 'Y',
'DGNAME' => 'DATA',
'LEVEL' => 1,
'ALIASNAME' => '_MGMTDB/12D2C496AC8DE99CE0630BFA19ACCB81'
},
'1' => {
'ALIASNAME' => 'ASM',
'LEVEL' => 0,
'DGNAME' => 'DATA',
'REFERENCE_INDEX' => '16777375',
'SYSTEM_CREATED' => 'Y'
},
'8' => {
'DGNAME' => 'DATA',
'REFERENCE_INDEX' => '16778170',
'SYSTEM_CREATED' => 'Y',
'ALIASNAME' => '_MGMTDB/12D2E2161B7F1288E0630BFA19ACB8F7/DATAFILE',
'LEVEL' => 2
},
'0' => {
'DGNAME' => 'DATA',
'REFERENCE_INDEX' => '16777269',
'SYSTEM_CREATED' => 'Y',
'ALIASNAME' => 'cluster11',
'LEVEL' => 0
},
'6' => {
'REFERENCE_INDEX' => '16778011',
'SYSTEM_CREATED' => 'Y',
'DGNAME' => 'DATA',
'LEVEL' => 2,
'ALIASNAME' => '_MGMTDB/12D2C496AC8DE99CE0630BFA19ACCB81/TEMPFILE'
}
},
'DGINFO' => {
'DGTORESTORE' => 0,
'DGNAME' => 'DATA',
'DGDBCOMPAT' => '12.2.0.0.0',
'DGTYPE' => 'FLEX',
'DGCOMPAT' => '19.0.0.0.0',
'DGAUSZ' => '4194304'
},
'DISKSINFO' => {
'FG1' => {
'DATA_0000' => {
'TOTAL_MB' => '16324',
'FAILGROUP' => 'FG1',
'NAME' => 'DATA_0000',
'PATH' => '/dev/oracleasm/disks/RACDISK01',
'DGNAME' => 'DATA',
'QUORUM' => 'REGULAR'
},
'DATA_0001' => {
'QUORUM' => 'REGULAR',
'PATH' => '/dev/oracleasm/disks/RACDISK02',
'DGNAME' => 'DATA',
'NAME' => 'DATA_0001',
'TOTAL_MB' => '16324',
'FAILGROUP' => 'FG1'
},
'DATA_0002' => {
'DGNAME' => 'DATA',
'PATH' => '/dev/oracleasm/disks/RACDISK03',
'QUORUM' => 'REGULAR',
'FAILGROUP' => 'FG1',
'TOTAL_MB' => '16324',
'NAME' => 'DATA_0002'
}
},
'FG3' => {
'DATA_0007' => {
'NAME' => 'DATA_0007',
'FAILGROUP' => 'FG3',
'TOTAL_MB' => '16324',
'QUORUM' => 'REGULAR',
'DGNAME' => 'DATA',
'PATH' => '/dev/oracleasm/disks/RACDISK08'
},
'DATA_0008' => {
'TOTAL_MB' => '16324',
'FAILGROUP' => 'FG3',
'NAME' => 'DATA_0008',
'PATH' => '/dev/oracleasm/disks/RACDISK09',
'DGNAME' => 'DATA',
'QUORUM' => 'REGULAR'
},
'DATA_0006' => {
'TOTAL_MB' => '16324',
'FAILGROUP' => 'FG3',
'NAME' => 'DATA_0006',
'PATH' => '/dev/oracleasm/disks/RACDISK07',
'DGNAME' => 'DATA',
'QUORUM' => 'REGULAR'
}
},
'FG2' => {
'DATA_0004' => {
'NAME' => 'DATA_0004',
'FAILGROUP' => 'FG2',
'TOTAL_MB' => '16324',
'QUORUM' => 'REGULAR',
'DGNAME' => 'DATA',
'PATH' => '/dev/oracleasm/disks/RACDISK05'
},
'DATA_0005' => {
'PATH' => '/dev/oracleasm/disks/RACDISK06',
'DGNAME' => 'DATA',
'QUORUM' => 'REGULAR',
'TOTAL_MB' => '16324',
'FAILGROUP' => 'FG2',
'NAME' => 'DATA_0005'
},
'DATA_0003' => {
'DGNAME' => 'DATA',
'PATH' => '/dev/oracleasm/disks/RACDISK04',
'QUORUM' => 'REGULAR',
'FAILGROUP' => 'FG2',
'TOTAL_MB' => '16324',
'NAME' => 'DATA_0003'
}
}
}
}
);
|
+fra磁盘组的元数据备份：
ASMCMD> md_backup /home/grid/fra.txt -G fra
Disk group metadata to be backed up: FRA
Current alias directory path: RACORCL/CONTROLFILE
Current alias directory path: RACORCL/AUTOBACKUP/2024_04_04
Current alias directory path: RACORCL/ONLINELOG
Current alias directory path: RACORCL/AUTOBACKUP/2024_03_07
Current alias directory path: RACORCL/BACKUPSET/2024_03_07
Current alias directory path: RACORCL/BACKUPSET
Current alias directory path: RACORCL/BACKUPSET/2024_04_04
Current alias directory path: RACORCL
Current alias directory path: RACORCL/ARCHIVELOG/2024_04_04
Current alias directory path: RACORCL/AUTOBACKUP
Current alias directory path: RACORCL/ARCHIVELOG |
查看/home/grid/fra.txt：
[grid@station11 ~]$ cat fra.txt @diskgroup_set = (
{
'DISKSINFO' => {
'FRA_0000' => {
'FRA_0000' => {
'QUORUM' => 'REGULAR',
'TOTAL_MB' => '16324',
'DGNAME' => 'FRA',
'FAILGROUP' => 'FRA_0000',
'NAME' => 'FRA_0000',
'PATH' => '/dev/oracleasm/disks/RACDISK10'
}
},
'FRA_0001' => {
'FRA_0001' => {
'QUORUM' => 'REGULAR',
'TOTAL_MB' => '16324',
'DGNAME' => 'FRA',
'FAILGROUP' => 'FRA_0001',
'NAME' => 'FRA_0001',
'PATH' => '/dev/oracleasm/disks/RACDISK11'
}
}
},
'TEMPLATEINFO' => {
'7' => {
'SYSTEM' => 'Y',
'REDUNDANCY' => 'UNPROT',
'TEMPNAME' => 'XTRANSPORT',
'STRIPE' => 'COARSE',
'DGNAME' => 'FRA'
},
'18' => {
'SYSTEM' => 'Y',
'REDUNDANCY' => 'UNPROT',
'TEMPNAME' => 'DUMPSET',
'STRIPE' => 'COARSE',
'DGNAME' => 'FRA'
},
'8' => {
'STRIPE' => 'COARSE',
'TEMPNAME' => 'AUTOBACKUP',
'DGNAME' => 'FRA',
'SYSTEM' => 'Y',
'REDUNDANCY' => 'UNPROT'
},
'17' => {
'DGNAME' => 'FRA',
'TEMPNAME' => 'CONTROLFILE',
'STRIPE' => 'FINE',
'REDUNDANCY' => 'UNPROT',
'SYSTEM' => 'Y'
},
'13' => {
'REDUNDANCY' => 'UNPROT',
'SYSTEM' => 'Y',
'DGNAME' => 'FRA',
'STRIPE' => 'COARSE',
'TEMPNAME' => 'DATAFILE'
},
'12' => {
'SYSTEM' => 'Y',
'REDUNDANCY' => 'UNPROT',
'STRIPE' => 'COARSE',
'TEMPNAME' => 'TEMPFILE',
'DGNAME' => 'FRA'
},
'3' => {
'STRIPE' => 'COARSE',
'TEMPNAME' => 'AUTOLOGIN_KEY_STORE',
'DGNAME' => 'FRA',
'SYSTEM' => 'Y',
'REDUNDANCY' => 'UNPROT'
},
'9' => {
'TEMPNAME' => 'INCR XTRANSPORT BACKUPSET',
'STRIPE' => 'COARSE',
'DGNAME' => 'FRA',
'SYSTEM' => 'Y',
'REDUNDANCY' => 'UNPROT'
},
'16' => {
'TEMPNAME' => 'FLASHFILE',
'STRIPE' => 'COARSE',
'DGNAME' => 'FRA',
'SYSTEM' => 'Y',
'REDUNDANCY' => 'UNPROT'
},
'2' => {
'DGNAME' => 'FRA',
'TEMPNAME' => 'AUDIT_SPILLFILES',
'STRIPE' => 'COARSE',
'REDUNDANCY' => 'UNPROT',
'SYSTEM' => 'Y'
},
'20' => {
'REDUNDANCY' => 'UNPROT',
'SYSTEM' => 'Y',
'DGNAME' => 'FRA',
'STRIPE' => 'COARSE',
'TEMPNAME' => 'ASMPARAMETERFILE'
},
'6' => {
'STRIPE' => 'COARSE',
'TEMPNAME' => 'CHANGETRACKING',
'DGNAME' => 'FRA',
'SYSTEM' => 'Y',
'REDUNDANCY' => 'UNPROT'
},
'21' => {
'DGNAME' => 'FRA',
'TEMPNAME' => 'DATAGUARDCONFIG',
'STRIPE' => 'COARSE',
'REDUNDANCY' => 'UNPROT',
'SYSTEM' => 'Y'
},
'15' => {
'REDUNDANCY' => 'UNPROT',
'SYSTEM' => 'Y',
'DGNAME' => 'FRA',
'TEMPNAME' => 'ARCHIVELOG',
'STRIPE' => 'COARSE'
},
'5' => {
'TEMPNAME' => 'FLASHBACK',
'STRIPE' => 'COARSE',
'DGNAME' => 'FRA',
'SYSTEM' => 'Y',
'REDUNDANCY' => 'UNPROT'
},
'14' => {
'SYSTEM' => 'Y',
'REDUNDANCY' => 'UNPROT',
'TEMPNAME' => 'ONLINELOG',
'STRIPE' => 'COARSE',
'DGNAME' => 'FRA'
},
'4' => {
'STRIPE' => 'COARSE',
'TEMPNAME' => 'KEY_STORE',
'DGNAME' => 'FRA',
'SYSTEM' => 'Y',
'REDUNDANCY' => 'UNPROT'
},
'19' => {
'TEMPNAME' => 'VOTINGFILE',
'STRIPE' => 'COARSE',
'DGNAME' => 'FRA',
'SYSTEM' => 'Y',
'REDUNDANCY' => 'UNPROT'
},
'0' => {
'DGNAME' => 'FRA',
'STRIPE' => 'COARSE',
'TEMPNAME' => 'PARAMETERFILE',
'REDUNDANCY' => 'UNPROT',
'SYSTEM' => 'Y'
},
'11' => {
'SYSTEM' => 'Y',
'REDUNDANCY' => 'UNPROT',
'STRIPE' => 'COARSE',
'TEMPNAME' => 'BACKUPSET',
'DGNAME' => 'FRA'
},
'1' => {
'STRIPE' => 'COARSE',
'TEMPNAME' => 'OCRFILE',
'DGNAME' => 'FRA',
'SYSTEM' => 'Y',
'REDUNDANCY' => 'UNPROT'
},
'10' => {
'REDUNDANCY' => 'UNPROT',
'SYSTEM' => 'Y',
'DGNAME' => 'FRA',
'TEMPNAME' => 'XTRANSPORT BACKUPSET',
'STRIPE' => 'COARSE'
}
},
'ALIASINFO' => {
'4' => {
'SYSTEM_CREATED' => 'Y',
'DGNAME' => 'FRA',
'LEVEL' => 2,
'REFERENCE_INDEX' => '33554962',
'ALIASNAME' => 'RACORCL/BACKUPSET/2024_03_07'
},
'3' => {
'SYSTEM_CREATED' => 'Y',
'DGNAME' => 'FRA',
'LEVEL' => 2,
'REFERENCE_INDEX' => '33555015',
'ALIASNAME' => 'RACORCL/BACKUPSET/2024_04_04'
},
'1' => {
'SYSTEM_CREATED' => 'Y',
'ALIASNAME' => 'RACORCL',
'DGNAME' => 'FRA',
'REFERENCE_INDEX' => '33554485',
'LEVEL' => 0
},
'0' => {
'ALIASNAME' => 'RACORCL/AUTOBACKUP/2024_03_07',
'LEVEL' => 2,
'REFERENCE_INDEX' => '33554803',
'DGNAME' => 'FRA',
'SYSTEM_CREATED' => 'Y'
},
'2' => {
'ALIASNAME' => 'RACORCL/ARCHIVELOG/2024_04_04',
'LEVEL' => 2,
'REFERENCE_INDEX' => '33554697',
'DGNAME' => 'FRA',
'SYSTEM_CREATED' => 'Y'
},
'5' => {
'SYSTEM_CREATED' => 'Y',
'REFERENCE_INDEX' => '33554538',
'LEVEL' => 2,
'DGNAME' => 'FRA',
'ALIASNAME' => 'RACORCL/AUTOBACKUP/2024_04_04'
}
},
'ATTRINFO' => {
'COMPATIBLE.ASM' => '19.0.0.0.0',
'COMPATIBLE.RDBMS' => '10.1.0.0.0',
'AU_SIZE' => '4194304',
'CELL.SPARSE_DG' => 'allnonsparse',
'_._DIRVERSION' => '19.0.0.0.0',
'_USD.LIST_SIZE' => '4'
},
'DGINFO' => {
'DGTYPE' => 'EXTERN',
'DGTORESTORE' => 0,
'DGNAME' => 'FRA',
'DGCOMPAT' => '19.0.0.0.0',
'DGDBCOMPAT' => '10.1.0.0.0',
'DGAUSZ' => '4194304'
}
}
);
|
1.3 OCR和OCR的备份： RAC 19c的OCR：
[root@station11 ~]# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 4
Total space (kbytes) : 491684
Used space (kbytes) : 84484
Available space (kbytes) : 407200
ID : 598828247
Device/File Name : +DATA
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
|
在RAC 19c的clusterware运行过程中，每4个小时OCR会自动备份。系统会保留最后3个副本，以及前一天、前一周的最后一个备份副本：
[root@station11 ~]# ocrconfig -showbackup
station11 2024/03/07 09:12:04 +DATA:/cluster11/OCRBACKUP/backup00.ocr.299.1162977119 724960844
station11 2024/03/07 05:11:58 +DATA:/cluster11/OCRBACKUP/backup01.ocr.296.1162962711 724960844
station11 2024/03/07 01:11:49 +DATA:/cluster11/OCRBACKUP/backup02.ocr.302.1162948301 724960844
station11 2024/03/06 00:48:26 +DATA:/cluster11/OCRBACKUP/day.ocr.297.1162860507 724960844
station11 2024/03/04 20:47:29 +DATA:/cluster11/OCRBACKUP/week.ocr.298.1162759651 724960844
PROT-25: Manual backups for the Oracle Cluster Registry are not available
|
为了防止ASM磁盘组出现硬件存储系统的故障，建议一定在文件系统上保留至少一份OCR备份。如果有条件的话，甚至应该在异机上保留这些备份。再次提醒：千万要在文件系统上留下OCR备份，下面进行这样的备份：
[root@station11 ~]# ocrconfig -manualbackup
station11 2024/04/04 03:08:00 +DATA:/cluster11/OCRBACKUP/backup_20240404_030800.ocr.286.1165374481 724960844
[root@station11 ~]# ocrconfig -showbackup
station11 2024/03/07 09:12:04 +DATA:/cluster11/OCRBACKUP/backup00.ocr.299.1162977119 724960844
station11 2024/03/07 05:11:58 +DATA:/cluster11/OCRBACKUP/backup01.ocr.296.1162962711 724960844
station11 2024/03/07 01:11:49 +DATA:/cluster11/OCRBACKUP/backup02.ocr.302.1162948301 724960844
station11 2024/03/06 00:48:26 +DATA:/cluster11/OCRBACKUP/day.ocr.297.1162860507 724960844
station11 2024/03/04 20:47:29 +DATA:/cluster11/OCRBACKUP/week.ocr.298.1162759651 724960844
station11 2024/04/04 03:08:00 +DATA:/cluster11/OCRBACKUP/backup_20240404_030800.ocr.286.1165374481 724960844
[root@station11 ~]# su - grid
Last login: Thu Apr 4 03:04:18 CST 2024
[grid@station11 ~]$ asmcmd
ASMCMD> cp +data/cluster11/OCRBACKUP/backup_20240404_030800.ocr.286.1165374481 /home/grid/
copying +data/cluster11/OCRBACKUP/backup_20240404_030800.ocr.286.1165374481 -> /home/grid//backup_20240404_030800.ocr.286.1165374481
|
为了万无一失，我们也在文件系统上保留一份OCR的逻辑备份（可能不一定会使用到它：ocrconfig -import /home/oracle/ocr.exp）：
[root@station11 ~]# ocrconfig -export /home/oracle/ocr.exp
PROT-58: successfully exported the Oracle Cluster Registry contents to file '/home/oracle/ocr.exp' |
我们也做一份orcdump：
[root@station11 ~]# ocrdump
[root@station11 ~]# ls
anaconda-ks.cfg Documents Music original-ks.cfg Pictures rac19cbr_install-V5-root.sh tmp
Desktop Downloads OCRDUMPFILE physical-block.sh Public Templates Videos |
ocrdum前40行：
[root@station11 ~]# head -n 40 OCRDUMPFILE
04/04/2024 03:11:36
/u01/app/19.3.0/grid/bin/ocrdump.bin
[SYSTEM]
UNDEF :
SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_WRITE, OTHER_PERMISSION : PROCR_READ, USER_NAME : grid, GROUP_NAME : asmdba}
[SYSTEM.version]
UB4 (10) : 19
SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_READ, OTHER_PERMISSION : PROCR_READ, USER_NAME : root, GROUP_NAME : root}
[SYSTEM.version.activeversion]
ORATEXT : 19.0.0.0.0
SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_READ, OTHER_PERMISSION : PROCR_READ, USER_NAME : root, GROUP_NAME : root}
[SYSTEM.version.activeversion.state]
UB4 (10) : 0
SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_READ, OTHER_PERMISSION : PROCR_READ, USER_NAME : root, GROUP_NAME : root}
[SYSTEM.version.activeversion.state.toversion]
ORATEXT : 19.0.0.0.0
SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_READ, OTHER_PERMISSION : PROCR_READ, USER_NAME : root, GROUP_NAME : root}
[SYSTEM.version.activeversion.patchlevel]
UB4 (10) : 724960844
SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_READ, OTHER_PERMISSION : PROCR_READ, USER_NAME : root, GROUP_NAME : root}
[SYSTEM.version.hostnames]
UNDEF :
SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_READ, OTHER_PERMISSION : PROCR_READ, USER_NAME : root, GROUP_NAME : root}
[SYSTEM.version.hostnames.station11]
ORATEXT : 19.0.0.0.0
SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_READ, OTHER_PERMISSION : PROCR_READ, USER_NAME : root, GROUP_NAME : root}
[SYSTEM.version.hostnames.station11.patchlevel]
UB4 (10) : 724960844
SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_READ, OTHER_PERMISSION : PROCR_READ, USER_NAME : root, GROUP_NAME : root}
[SYSTEM.version.hostnames.station11.site]
|
ocrdum后40行：
[root@station11 ~]# tail -n 40 OCRDUMPFILE
[DATABASE]
UNDEF :
SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_CREATE_SUB_KEY, OTHER_PERMISSION : PROCR_CREATE_SUB_KEY, USER_NAME : root, GROUP_NAME : root}
[DATABASE.NODEAPPS]
UNDEF :
SECURITY : {USER_PERMISSION : PROCR_CREATE_SUB_KEY, GROUP_PERMISSION : PROCR_CREATE_SUB_KEY, OTHER_PERMISSION : PROCR_CREATE_SUB_KEY, USER_NAME : grid, GROUP_NAME : oinstall}
[DATABASE.VIP_RANGE]
UNDEF :
SECURITY : {USER_PERMISSION : PROCR_CREATE_SUB_KEY, GROUP_PERMISSION : PROCR_CREATE_SUB_KEY, OTHER_PERMISSION : PROCR_CREATE_SUB_KEY, USER_NAME : grid, GROUP_NAME : oinstall}
[DATABASE.LOG]
UNDEF :
SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_ALL_ACCESS, OTHER_PERMISSION : PROCR_ALL_ACCESS, USER_NAME : grid, GROUP_NAME : oinstall}
[DATABASE.ASM]
UNDEF :
SECURITY : {USER_PERMISSION : PROCR_CREATE_SUB_KEY, GROUP_PERMISSION : PROCR_CREATE_SUB_KEY, OTHER_PERMISSION : PROCR_READ, USER_NAME : grid, GROUP_NAME : oinstall}
[DATABASE.DATABASES]
UNDEF :
SECURITY : {USER_PERMISSION : PROCR_CREATE_SUB_KEY, GROUP_PERMISSION : PROCR_CREATE_SUB_KEY, OTHER_PERMISSION : PROCR_CREATE_SUB_KEY, USER_NAME : grid, GROUP_NAME : oinstall}
[CRS]
UNDEF :
SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_READ, OTHER_PERMISSION : PROCR_READ, USER_NAME : root, GROUP_NAME : root}
[CRS.CUR]
UNDEF :
SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_NONE, OTHER_PERMISSION : PROCR_NONE, USER_NAME : root, GROUP_NAME : root}
[CRS.HIS]
UNDEF :
SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_NONE, OTHER_PERMISSION : PROCR_NONE, USER_NAME : root, GROUP_NAME : root}
[CRS.SEC]
UNDEF :
SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_NONE, OTHER_PERMISSION : PROCR_NONE, USER_NAME : root, GROUP_NAME : root}
[root@station11 ~]# tail -n 40 OCRDUMPFILE
[DATABASE]
UNDEF :
SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_CREATE_SUB_KEY, OTHER_PERMISSION : PROCR_CREATE_SUB_KEY, USER_NAME : root, GROUP_NAME : root}
[DATABASE.NODEAPPS]
UNDEF :
SECURITY : {USER_PERMISSION : PROCR_CREATE_SUB_KEY, GROUP_PERMISSION : PROCR_CREATE_SUB_KEY, OTHER_PERMISSION : PROCR_CREATE_SUB_KEY, USER_NAME : grid, GROUP_NAME : oinstall}
[DATABASE.VIP_RANGE]
UNDEF :
SECURITY : {USER_PERMISSION : PROCR_CREATE_SUB_KEY, GROUP_PERMISSION : PROCR_CREATE_SUB_KEY, OTHER_PERMISSION : PROCR_CREATE_SUB_KEY, USER_NAME : grid, GROUP_NAME : oinstall}
[DATABASE.LOG]
UNDEF :
SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_ALL_ACCESS, OTHER_PERMISSION : PROCR_ALL_ACCESS, USER_NAME : grid, GROUP_NAME : oinstall}
[DATABASE.ASM]
UNDEF :
SECURITY : {USER_PERMISSION : PROCR_CREATE_SUB_KEY, GROUP_PERMISSION : PROCR_CREATE_SUB_KEY, OTHER_PERMISSION : PROCR_READ, USER_NAME : grid, GROUP_NAME : oinstall}
[DATABASE.DATABASES]
UNDEF :
SECURITY : {USER_PERMISSION : PROCR_CREATE_SUB_KEY, GROUP_PERMISSION : PROCR_CREATE_SUB_KEY, OTHER_PERMISSION : PROCR_CREATE_SUB_KEY, USER_NAME : grid, GROUP_NAME : oinstall}
[CRS]
UNDEF :
SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_READ, OTHER_PERMISSION : PROCR_READ, USER_NAME : root, GROUP_NAME : root}
[CRS.CUR]
UNDEF :
SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_NONE, OTHER_PERMISSION : PROCR_NONE, USER_NAME : root, GROUP_NAME : root}
[CRS.HIS]
UNDEF :
SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_NONE, OTHER_PERMISSION : PROCR_NONE, USER_NAME : root, GROUP_NAME : root}
[CRS.SEC]
UNDEF :
SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_NONE, OTHER_PERMISSION : PROCR_NONE, USER_NAME : root, GROUP_NAME : root}
|
1.4 VotingDisk： VotingDisk不需要备份，我们只需要确认它们的存在：
[root@station11 ~]# crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 79e85ab2cd054f2cbfa9e65951cba2b4 (/dev/oracleasm/disks/RACDISK01) [DATA]
2. ONLINE fd34651875404fa3bfcb38f8bd066e8c (/dev/oracleasm/disks/RACDISK06) [DATA]
3. ONLINE aabcf28776e34f2bbfb63cffc6dd5911 (/dev/oracleasm/disks/RACDISK08) [DATA]
Located 3 voting disk(s).
|
1.5 RAC数据库的备份恢复配置和备份： 在文件系统上，建议保留至少一份数据库的全备份和归档日志的备份：
[oracle@station11 ~]$ sqlplus /nolog
SQL*Plus: Release 19.0.0.0.0 - Production on Fri Apr 5 07:14:28 2024 Version 19.3.0.0.0
Copyright (c) 1982, 2019, Oracle. All rights reserved.
SQL> conn / as sysdba Connected. SQL> alter system set log_archive_dest_1='location=use_db_recovery_file_dest';
System altered. SQL> alter system set log_archive_dest_2='location=/home/oracle/archivelog';
System altered. |
连接Recovery Catalog来备份数据库，备份保存在文件系统上：
[oracle@station11 ~]$ rman target / catalog urcat/oracle_4U@rcat
Recovery Manager: Release 19.0.0.0.0 - Production on Fri Apr 5 08:57:34 2024 Version 19.3.0.0.0
Copyright (c) 1982, 2019, Oracle and/or its affiliates. All rights reserved.
connected to target database: RACORCL (DBID=2908490689) connected to recovery catalog database
RMAN> run { 2> allocate channel c1 device type disk format '/home/oracle/backup/%U'; 3> allocate channel c2 device type disk format '/home/oracle/backup/%U'; 4> allocate channel c3 device type disk format '/home/oracle/backup/%U'; 5> backup tag '1D_WHOLE_INCR0' incremental level 0 database; 6> }
allocated channel: c1 channel c1: SID=861 instance=racorcl1 device type=DISK
allocated channel: c2 channel c2: SID=983 instance=racorcl1 device type=DISK
allocated channel: c3 channel c3: SID=1109 instance=racorcl1 device type=DISK
Starting backup at 05-APR-24 channel c1: starting incremental level 0 datafile backup set channel c1: specifying datafile(s) in backup set input datafile file number=00001 name=+DATA/RACORCL/DATAFILE/system.283.1162980303 channel c1: starting piece 1 at 05-APR-24 channel c2: starting incremental level 0 datafile backup set channel c2: specifying datafile(s) in backup set input datafile file number=00003 name=+DATA/RACORCL/DATAFILE/sysaux.295.1162980337 input datafile file number=00007 name=+DATA/RACORCL/DATAFILE/users.303.1162980353 channel c2: starting piece 1 at 05-APR-24 channel c3: starting incremental level 0 datafile backup set channel c3: specifying datafile(s) in backup set input datafile file number=00004 name=+DATA/RACORCL/DATAFILE/undotbs1.288.1162980353 input datafile file number=00005 name=+DATA/RACORCL/DATAFILE/undotbs2.301.1162980561 channel c3: starting piece 1 at 05-APR-24 channel c3: finished piece 1 at 05-APR-24 piece handle=/home/oracle/backup/0u2nfltc_1_1 tag=1D_WHOLE_INCR0 comment=NONE channel c3: backup set complete, elapsed time: 00:00:01 channel c1: finished piece 1 at 05-APR-24 piece handle=/home/oracle/backup/0s2nfltc_1_1 tag=1D_WHOLE_INCR0 comment=NONE channel c1: backup set complete, elapsed time: 00:00:03 channel c2: finished piece 1 at 05-APR-24 piece handle=/home/oracle/backup/0t2nfltc_1_1 tag=1D_WHOLE_INCR0 comment=NONE channel c2: backup set complete, elapsed time: 00:00:03 Finished backup at 05-APR-24
Starting Control File and SPFILE Autobackup at 05-APR-24 piece handle=+FRA/RACORCL/AUTOBACKUP/2024_04_05/s_1165481904.285.1165481905 comment=NONE Finished Control File and SPFILE Autobackup at 05-APR-24 released channel: c1 released channel: c2 released channel: c3 |
连接Recovery Catalog来备份归档日志，备份保存在文件系统上：
[oracle@station11 ~]$ rman target / catalog urcat/oracle_4U@rcat
Recovery Manager: Release 19.0.0.0.0 - Production on Fri Apr 5 09:03:45 2024 Version 19.3.0.0.0
Copyright (c) 1982, 2019, Oracle and/or its affiliates. All rights reserved.
connected to target database: RACORCL (DBID=2908490689) connected to recovery catalog database
RMAN> run{ 2> allocate channel c1 device type disk format '/home/oracle/backup/%U'; 3> allocate channel c2 device type disk format '/home/oracle/backup/%U'; 4> allocate channel c3 device type disk format '/home/oracle/backup/%U'; 5> backup tag '2D_ARCHIVELOG' archivelog all not backed up; 6> }
allocated channel: c1 channel c1: SID=624 instance=racorcl1 device type=DISK
allocated channel: c2 channel c2: SID=743 instance=racorcl1 device type=DISK
allocated channel: c3 channel c3: SID=865 instance=racorcl1 device type=DISK
Starting backup at 05-APR-24 current log archived archived log /home/oracle/archivelog2_10_1162980420.dbf not found or out of sync with catalog trying alternate file for archived log of thread 2 with sequence 10 channel c1: starting archived log backup set channel c1: specifying archived log(s) in backup set input archived log thread=2 sequence=9 RECID=30 STAMP=1165482116 channel c1: starting piece 1 at 05-APR-24 channel c2: starting archived log backup set channel c2: specifying archived log(s) in backup set input archived log thread=1 sequence=15 RECID=24 STAMP=1165481707 input archived log thread=1 sequence=16 RECID=28 STAMP=1165482116 channel c2: starting piece 1 at 05-APR-24 channel c3: starting archived log backup set channel c3: specifying archived log(s) in backup set input archived log thread=1 sequence=13 RECID=20 STAMP=1165481582 input archived log thread=2 sequence=8 RECID=26 STAMP=1165481710 input archived log thread=1 sequence=14 RECID=22 STAMP=1165481591 channel c3: starting piece 1 at 05-APR-24 channel c1: finished piece 1 at 05-APR-24 piece handle=/home/oracle/backup/102nfm93_1_1 tag=2D_ARCHIVELOG comment=NONE channel c1: backup set complete, elapsed time: 00:00:01 channel c1: starting archived log backup set channel c1: specifying archived log(s) in backup set input archived log thread=1 sequence=17 RECID=32 STAMP=1165482270 input archived log thread=2 sequence=10 RECID=34 STAMP=1165482273 channel c1: starting piece 1 at 05-APR-24 channel c2: finished piece 1 at 05-APR-24 piece handle=/home/oracle/backup/112nfm93_1_1 tag=2D_ARCHIVELOG comment=NONE channel c2: backup set complete, elapsed time: 00:00:01 channel c3: finished piece 1 at 05-APR-24 piece handle=/home/oracle/backup/122nfm93_1_1 tag=2D_ARCHIVELOG comment=NONE channel c3: backup set complete, elapsed time: 00:00:01 channel c1: finished piece 1 at 05-APR-24 piece handle=/home/oracle/backup/132nfm94_1_1 tag=2D_ARCHIVELOG comment=NONE channel c1: backup set complete, elapsed time: 00:00:01 Finished backup at 05-APR-24
Starting Control File and SPFILE Autobackup at 05-APR-24 piece handle=+FRA/RACORCL/AUTOBACKUP/2024_04_05/s_1165482278.276.1165482279 comment=NONE Finished Control File and SPFILE Autobackup at 05-APR-24 released channel: c1 released channel: c2 released channel: c3 |
2. 磁盘硬件严重故障模拟：
[root@station11 archivelog]# for i in $(oracleasm listdisks); do dd if=/dev/zero of=/dev/oracleasm/disks/$i bs=1G count=1 ; done
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.33088 s, 461 MB/s
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.45912 s, 736 MB/s
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.43771 s, 747 MB/s
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.48258 s, 724 MB/s
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.42861 s, 752 MB/s
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.73923 s, 617 MB/s
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.51108 s, 711 MB/s
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.67056 s, 643 MB/s
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.52674 s, 703 MB/s
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.17484 s, 914 MB/s
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.2929 s, 830 MB/s
|
ASM磁盘已经彻底不见了：
[root@station11 ~]# oracleasm-discover
Using ASMLib from /opt/oracle/extapi/64/asm/orcl/1/libasm.so
[ASM Library - Generic Linux, version 2.0.17 (KABI_V2)]
[root@station11 ~]# ssh station12
root@station12's password:
Activate the web console with: systemctl enable --now cockpit.socket
Last login: Fri Apr 5 06:45:11 2024 from 172.25.250.11
[root@station12 ~]# oracleasm-discover
Using ASMLib from /opt/oracle/extapi/64/asm/orcl/1/libasm.so
[ASM Library - Generic Linux, version 2.0.17 (KABI_V2)]
|
节点都自动重启，集群已经奔溃：
[root@station11 ~]# last
root pts/0 172.25.250.92 Fri Apr 5 10:29 still logged in
reboot system boot 5.4.17-2011.7.4. Fri Apr 5 10:27 still running
root pts/2 172.25.250.92 Fri Apr 5 09:57 - crash (00:30)
root pts/0 172.25.250.92 Fri Apr 5 07:55 - crash (02:31)
root pts/1 172.25.250.92 Fri Apr 5 07:14 - crash (03:13)
root pts/0 172.25.250.92 Fri Apr 5 06:51 - 07:55 (01:04)
root pts/0 172.25.250.92 Fri Apr 5 06:38 - 06:46 (00:07)
......
|
[root@station12 ~]# last
reboot system boot 5.4.17-2011.7.4. Fri Apr 5 10:27 still running
root pts/0 172.25.250.11 Fri Apr 5 06:45 - 06:45 (00:00)
root pts/0 172.25.250.11 Thu Apr 4 14:41 - 06:44 (16:02)
......
|
OCR当然也不存在了：
[root@station11 ~]# ocrcheck
PROT-602: Failed to retrieve data from the cluster registry
PROC-26: Error while accessing the physical storage Storage layer error [Insufficient quorum to open OCR devices] [0]
[root@station11 ~]# crsctl status res -t
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.
|
[root@station11 ~]# crsctl status res -t
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.
|
[root@station12 ~]# crsctl status res -t
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.
|
将通过之后的三篇博客来进行集群的彻底恢复。
|
