Cisco FI Recovery

 

Pre-troubleshooting checks:

1. Check Cluster State

Central-BLRQA-Mini-10-B# show cluster state

Cluster Id: 0xb1125e5643ad11e0-0x921574a02f402f50

B: UP, SUBORDINATE

A: UNRESPONSIVE, PRIMARY

HA NOT READY

2. Check f/w versions on active node(if any)

Central-BLRQA-Mini-10-B# sh version

System version: 3.1(0.297c)

Central-BLRQA-Mini-10-B# show fabric-interconnect a detail

Central-BLRQA-Mini-10-B# show fabric-interconnect b detail

3. Bootability of node which is not active:

UCSM prompt or Loader prompt

Troubleshooting:

1. Go to the linux prompt of both the FIs

a. For the FI which is bootable, connect console port and login using  adminbackup

b. For the FI which is not in proper state, connect console port and power cycle the switch.

While the switch is rebooting , press ctlr+L to come to Loader prompt.

loader> dir

(displays the list of images including the kickstart image)

loader> boot  <kickstart image name>

switch(boot)# start

bash-3.2#

c. If kickstart image is not available, goto https://wiki.cisco.com/display/391974/FI+bringup+procedure+from+Loader+prompt

2. At the linux/bash prompt on both the FIs, issue the following command to erase the DB. This is    required to stop the auto-install process(if running)

bash-3.2# rm /opt/db/nvram/dme.db /opt/db/flash/dme.db

3. At the linux/bash prompt on both the FIs, perform the following:

a.  Verify the existing packages in the system to revert back to

bash-3.2# cd /bootflash/installables/switch/

bash-3.2# ls –l

total 5614168

-rwxr-xr-x 1 root root  38409728 Mar 19 06:19 ucs-6100-k9-kickstart.5.2.3.N2.2.21.179.gbin

-rwxr-xr-x 1 root root  38422016 Jul 22 06:23 ucs-6100-k9-kickstart.5.2.3.N2.2.21.188.gbin

-rwxr-xr-x 1 root root  38370304 Aug  7 12:29 ucs-6100-k9-kickstart.5.2.3.N2.2.21.198.bin

-rwxr-xr-x 1 root root  38376960 Aug  7 09:59 ucs-6100-k9-kickstart.5.2.3.N2.2.21.205.gbin

-rwxr-xr-x 1 root root 272711285 Mar 19 06:40 ucs-6100-k9-system.5.2.3.N2.2.21.179.gbin

-rwxr-xr-x 1 root root 273244129 Jul 22 06:23 ucs-6100-k9-system.5.2.3.N2.2.21.188.gbin

-rwxr-xr-x 1 root root 271342749 Aug  7 12:29 ucs-6100-k9-system.5.2.3.N2.2.21.198.bin

-rwxr-xr-x 1 root root 273317725 Aug  7 09:59 ucs-6100-k9-system.5.2.3.N2.2.21.205.gbin

-rwxr-xr-x 1 root root 178303518 Jun 10 12:50 ucs-manager-k9.2.2.2.135.gbin

-rwxr-xr-x 1 root root 181035134 Jul 22 06:23 ucs-manager-k9.2.2.2.188.gbin

-rwxr-xr-x 1 root root 183592165 Aug  7 12:29 ucs-manager-k9.2.2.2.198.bin

--rwxr-xr-x 1 root root  93776495 Aug  7 09:57 ucs-mgmtext.2.2.2.205.gbin

b. Edit the following file with proper images which were working fine before upgrade

bash-3.2# cd /mnt/cfg/1/boot/grub/

bash-3.2#  vi menu.lst.local

bash-3.2#  cat  menu.lst.local

# 697741597

#

# General configuration

#

disable certificate

# Menu entry for the available image

title bootflash:/installables/switch/ucs-6100-k9-kickstart.5.2.3.N2.2.21.188.gbin

boot bootflash:/installables/switch/ucs-6100-k9-kickstart.5.2.3.N2.2.21.188.gbin bootflash:/installables/switch/ucs-6100-k9-system.5.2.3.N2.2.21.188.gbin

c. update the UCSM image too with the same version

bash-3.2# cd /bootflash/

bash-3.2# ln  –sf  /bootflash/installables/switch/ucs-manager-k9.2.2.2.188.gbin   nuova-sim-mgmt-nsg.0.1.0.001.bin

d. Reboot the system

4. The system should come back to the older version and cluster should be up and running.

SAM-QA-20-B# show cluster state

Cluster Id: 0xf76a633e5f9511e2-0x826d547fee5682a4

 

B: UP, PRIMARY

A: UP, SUBORDINATE

 

HA NOT READY

SAM-QA-20-B#