cannot mount ASM disk

[Sayid Osman]

Hello Erman,

I am having problem when i try to mount ASM disk. I got following error when i try alter diskgroup DATA mount.  I rebooted machine but i got same error.


ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15040: diskgroup is incomplete
ORA-15042: ASM disk "2" is missing from group number "2"
ORA-15042: ASM disk "1" is missing from group number "2"

I can see my disks with following commands.

[oracle@erppreprod ~]$ oracleasm listdisks
DATAERP
FRA
 
[oracle@erppreprod ~]$ oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
 
[oracle@erppreprod /]$ ll /dev/oracleasm/disks/
total 0
brw-rw---- 1 oracle oinstall 8, 33 Sep 19 11:40 DATAERP
brw-rw---- 1 oracle oinstall 8, 49 Sep 19 11:40 FRA

[ErmanArslansOracleBlog]

Did you do any modifications on these disk? Any OS operations maybe?

1)What does "kfed read" report?

kfed read <disk_name>

for ex: kfed read ASMRECO03

Ref: KFED Reports “KFBTYP_INVALID” & OS Metadata [EFI PART] In ASMLIB Disk /ASM disk Member (ASM Disk Overlapping : Scenario #1). (Doc ID 1673750.1)

2)What does  ASM logs show?

[Sayid Osman]

Only ip adrress has been changed. I can mount another diskgroup FRA. You can see kfed read DATAERP
output and ASM alert.log below


kfbh.endian:                          1 ; 0x000: 0x01
kfbh.hard:                          130 ; 0x001: 0x82
kfbh.type:                            1 ; 0x002: KFBTYP_DISKHEAD
kfbh.datfmt:                          1 ; 0x003: 0x01
kfbh.block.blk:                       0 ; 0x004: blk=0
kfbh.block.obj:              2147483648 ; 0x008: disk=0
kfbh.check:                  2802812966 ; 0x00c: 0xa70f8826
kfbh.fcn.base:                   793806 ; 0x010: 0x000c1cce
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000
kfdhdb.driver.provstr:  ORCLDISKDATAERP ; 0x000: length=15
kfdhdb.driver.reserved[0]:   1096040772 ; 0x008: 0x41544144
kfdhdb.driver.reserved[1]:      5263941 ; 0x00c: 0x00505245
kfdhdb.driver.reserved[2]:            0 ; 0x010: 0x00000000
kfdhdb.driver.reserved[3]:            0 ; 0x014: 0x00000000
kfdhdb.driver.reserved[4]:            0 ; 0x018: 0x00000000
kfdhdb.driver.reserved[5]:            0 ; 0x01c: 0x00000000
kfdhdb.compat:                186646528 ; 0x020: 0x0b200000
kfdhdb.dsknum:                        0 ; 0x024: 0x0000
kfdhdb.grptyp:                        1 ; 0x026: KFDGTP_EXTERNAL
kfdhdb.hdrsts:                        3 ; 0x027: KFDHDR_MEMBER
kfdhdb.dskname:                 DATAERP ; 0x028: length=7
kfdhdb.grpname:                 DATAERP ; 0x048: length=7
kfdhdb.fgname:                  DATAERP ; 0x068: length=7
kfdhdb.capname:                         ; 0x088: length=0
kfdhdb.crestmp.hi:             33004590 ; 0x0a8: HOUR=0xe DAYS=0x1 MNTH=0x7 YEAR=0x7de
kfdhdb.crestmp.lo:           2138901504 ; 0x0ac: USEC=0x0 MSEC=0x343 SECS=0x37 MINS=0x1f
kfdhdb.mntstmp.hi:             33055053 ; 0x0b0: HOUR=0xd DAYS=0xa MNTH=0x8 YEAR=0x7e1
kfdhdb.mntstmp.lo:           3344378880 ; 0x0b4: USEC=0x0 MSEC=0x1cb SECS=0x35 MINS=0x31
kfdhdb.secsize:                     512 ; 0x0b8: 0x0200
kfdhdb.blksize:                    4096 ; 0x0ba: 0x1000
kfdhdb.ausize:                  1048576 ; 0x0bc: 0x00100000
kfdhdb.mfact:                    113792 ; 0x0c0: 0x0001bc80
kfdhdb.dsksize:                  204797 ; 0x0c4: 0x00031ffd
kfdhdb.pmcnt:                         3 ; 0x0c8: 0x00000003
kfdhdb.fstlocn:                       1 ; 0x0cc: 0x00000001
kfdhdb.altlocn:                       2 ; 0x0d0: 0x00000002
kfdhdb.f1b1locn:                      2 ; 0x0d4: 0x00000002
kfdhdb.redomirrors[0]:                0 ; 0x0d8: 0x0000
kfdhdb.redomirrors[1]:            65535 ; 0x0da: 0xffff
kfdhdb.redomirrors[2]:            65535 ; 0x0dc: 0xffff
kfdhdb.redomirrors[3]:            65535 ; 0x0de: 0xffff
kfdhdb.dbcompat:              168820736 ; 0x0e0: 0x0a100000
kfdhdb.grpstmp.hi:             33004590 ; 0x0e4: HOUR=0xe DAYS=0x1 MNTH=0x7 YEAR=0x7de
kfdhdb.grpstmp.lo:           2138827776 ; 0x0e8: USEC=0x0 MSEC=0x2fb SECS=0x37 MINS=0x1f
kfdhdb.vfstart:                       0 ; 0x0ec: 0x00000000
kfdhdb.vfend:                         0 ; 0x0f0: 0x00000000
kfdhdb.spfile:                        0 ; 0x0f4: 0x00000000
kfdhdb.spfflg:                        0 ; 0x0f8: 0x00000000
kfdhdb.ub4spare[0]:                   0 ; 0x0fc: 0x00000000
kfdhdb.ub4spare[1]:                   0 ; 0x100: 0x00000000
kfdhdb.ub4spare[2]:                   0 ; 0x104: 0x00000000
kfdhdb.ub4spare[3]:                   0 ; 0x108: 0x00000000
kfdhdb.ub4spare[4]:                   0 ; 0x10c: 0x00000000
kfdhdb.ub4spare[5]:                   0 ; 0x110: 0x00000000
kfdhdb.ub4spare[6]:                   0 ; 0x114: 0x00000000
kfdhdb.ub4spare[7]:                   0 ; 0x118: 0x00000000
kfdhdb.ub4spare[8]:                   0 ; 0x11c: 0x00000000
kfdhdb.ub4spare[9]:                   0 ; 0x120: 0x00000000
kfdhdb.ub4spare[10]:                  0 ; 0x124: 0x00000000
kfdhdb.ub4spare[11]:                  0 ; 0x128: 0x00000000
kfdhdb.ub4spare[12]:                  0 ; 0x12c: 0x00000000
kfdhdb.ub4spare[13]:                  0 ; 0x130: 0x00000000
kfdhdb.ub4spare[14]:                  0 ; 0x134: 0x00000000
kfdhdb.ub4spare[15]:                  0 ; 0x138: 0x00000000
kfdhdb.ub4spare[16]:                  0 ; 0x13c: 0x00000000
kfdhdb.ub4spare[17]:                  0 ; 0x140: 0x00000000
kfdhdb.ub4spare[18]:                  0 ; 0x144: 0x00000000
kfdhdb.ub4spare[19]:                  0 ; 0x148: 0x00000000
kfdhdb.ub4spare[20]:                  0 ; 0x14c: 0x00000000
kfdhdb.ub4spare[21]:                  0 ; 0x150: 0x00000000
kfdhdb.ub4spare[22]:                  0 ; 0x154: 0x00000000
kfdhdb.ub4spare[23]:                  0 ; 0x158: 0x00000000
kfdhdb.ub4spare[24]:                  0 ; 0x15c: 0x00000000
kfdhdb.ub4spare[25]:                  0 ; 0x160: 0x00000000
kfdhdb.ub4spare[26]:                  0 ; 0x164: 0x00000000
kfdhdb.ub4spare[27]:                  0 ; 0x168: 0x00000000
kfdhdb.ub4spare[28]:                  0 ; 0x16c: 0x00000000
kfdhdb.ub4spare[29]:                  0 ; 0x170: 0x00000000
kfdhdb.ub4spare[30]:                  0 ; 0x174: 0x00000000
kfdhdb.ub4spare[31]:                  0 ; 0x178: 0x00000000
kfdhdb.ub4spare[32]:                  0 ; 0x17c: 0x00000000
kfdhdb.ub4spare[33]:                  0 ; 0x180: 0x00000000
kfdhdb.ub4spare[34]:                  0 ; 0x184: 0x00000000
kfdhdb.ub4spare[35]:                  0 ; 0x188: 0x00000000
kfdhdb.ub4spare[36]:                  0 ; 0x18c: 0x00000000
kfdhdb.ub4spare[37]:                  0 ; 0x190: 0x00000000
kfdhdb.ub4spare[38]:                  0 ; 0x194: 0x00000000
kfdhdb.ub4spare[39]:                  0 ; 0x198: 0x00000000
kfdhdb.ub4spare[40]:                  0 ; 0x19c: 0x00000000
kfdhdb.ub4spare[41]:                  0 ; 0x1a0: 0x00000000
kfdhdb.ub4spare[42]:                  0 ; 0x1a4: 0x00000000
kfdhdb.ub4spare[43]:                  0 ; 0x1a8: 0x00000000
kfdhdb.ub4spare[44]:                  0 ; 0x1ac: 0x00000000
kfdhdb.ub4spare[45]:                  0 ; 0x1b0: 0x00000000
kfdhdb.ub4spare[46]:                  0 ; 0x1b4: 0x00000000
kfdhdb.ub4spare[47]:                  0 ; 0x1b8: 0x00000000
kfdhdb.ub4spare[48]:                  0 ; 0x1bc: 0x00000000
kfdhdb.ub4spare[49]:                  0 ; 0x1c0: 0x00000000
kfdhdb.ub4spare[50]:                  0 ; 0x1c4: 0x00000000
kfdhdb.ub4spare[51]:                  0 ; 0x1c8: 0x00000000
kfdhdb.ub4spare[52]:                  0 ; 0x1cc: 0x00000000
kfdhdb.ub4spare[53]:                  0 ; 0x1d0: 0x00000000
kfdhdb.acdb.aba.seq:                  0 ; 0x1d4: 0x00000000
kfdhdb.acdb.aba.blk:                  0 ; 0x1d8: 0x00000000
kfdhdb.acdb.ents:                     0 ; 0x1dc: 0x0000
kfdhdb.acdb.ub2spare:                 0 ; 0x1de: 0x0000


ASM alert log;

Fri Sep 22 13:47:41 2017
NOTE: GMON heartbeating for grp 1
GMON querying group 1 at 8 for pid 17, osid 24889
NOTE: Assigning number (1,1) to disk ()
NOTE: Assigning number (1,2) to disk ()
GMON querying group 1 at 9 for pid 17, osid 24889
NOTE: cache dismounting (clean) group 1/0x82CCD2E9 (DATAERP)
NOTE: messaging CKPT to quiesce pins Unix process pid: 24889, image: oracle@erppreprod (TNS V1-V3)
NOTE: dbwr not being msg'd to dismount
NOTE: lgwr not being msg'd to dismount
NOTE: cache dismounted group 1/0x82CCD2E9 (DATAERP)
NOTE: cache ending mount (fail) of group DATAERP number=1 incarn=0x82ccd2e9
NOTE: cache deleting context for group DATAERP 1/0x82ccd2e9
GMON dismounting group 1 at 10 for pid 17, osid 24889
NOTE: Disk  in mode 0x8 marked for de-assignment
NOTE: Disk  in mode 0x8 marked for de-assignment
NOTE: Disk  in mode 0x8 marked for de-assignment
ERROR: diskgroup DATAERP was not mounted
ORA-15032: not all alterations performed
ORA-15040: diskgroup is incomplete
ORA-15042: ASM disk "2" is missing from group number "1"
ORA-15042: ASM disk "1" is missing from group number "1"
ERROR: alter diskgroup DATAERP mount
Fri Sep 22 13:47:41 2017
ASM Health Checker found 1 new failures
Fri Sep 22 13:47:41 2017
ASM Health Checker found 1 new failures

[ErmanArslansOracleBlog]

You may be having a disk issue on OS level. A missing disk or a duplicate disk.

Follow this note: KFED.PL for diagnosing - ORA-15063 ORA-15042 ORA-15020 ORA-15033 (Doc ID 1346190.1)

Check kfed.out if there is any missing disks from the diskgroup in question and work with System administrator team to find them.
Check kfed.out if there is any duplicated disks in terms of disk number from the diskgroup in question, see note - 1299866.1.
(You can also upload it to me , so I can check)

[ErmanArslansOracleBlog]

Also, ensure that: all the disks of the problematic diskgroup is mapped to the server.
Your ASM disks should be available in order to be able to mount the related diskgroup.

[ErmanArslansOracleBlog]

In addition,

ensure that the permissions of the disks are okay and udev rules are present  and the asm_diskstring is set properly.

Here is an example:

Udev rule:

 cat 99-oracle-asmdevices.rules
KERNEL=="sde", OWNER="oracle", GROUP="oinstall", MODE="0660"
KERNEL=="sdf", OWNER="oracle", GROUP="oinstall", MODE="0660"
KERNEL=="sde1", OWNER="oracle", GROUP="oinstall", MODE="0660"
KERNEL=="sdf1", OWNER="oracle", GROUP="oinstall", MODE="0660"

Udev restart

udevadm control --reload-rules && udevadm trigger

SQL> alter system set asm_diskstring='ORCL:*','/dev/sd*' scope=memory;

Mount:

SQL>  alter diskgroup ERMAN mount;

Diskgroup altered.

Sometimes, when you don't use asmlib/oracleasm for adding ASM devices, these udev rules may be missing. Also when you don't use oracleasm/asmlib, ASM needs to see the /dev device name directly (/dev/sd devices), so you should configure your asm_diskstring properly.