XSAN: No visible Luns

vinnih's picture

Hello I have XSAN with 4 MacPro's. Two of them are MDC and two clients. Suddenly the volumes disappeared from one of the clients. I checked on one of the mdc and the volumes were visible but in Xsan admin there was a message No visible luns. I turned off all computers and turned on just one of the mdc. Immediately i get a message that the system see some volumes that are not readable and offer me to initiliaze them. Xsan admin no does not see any luns and no volumes are mounted. The LUN are visible in disk utility.

Can you help? What is going on? Can I get my volumes back?

lfrancois's picture

Did you plug a windows machine on the FC switch? I am pretty sure you wiped out the XSAN label from your LUNs.

Do you have a copy of your cvlabel somewhere? You just need to apply the XSAN label to your LUNs again.

Could you give us an output of cvlabel -l ?

Best Regards,
Ludo

vinnih's picture

Hello LFrancois,

thank you for oyur responce. There is a windows machine in the LAN network, but it is not connected to the FC switch.

Here is the output from cvlabel -l

erver:~ server$ sudo cvlabel -l
Password:
/dev/rdisk0 [ itachi HDS72107 K8OA70] EFI Sectors: 1465130703. Sector Size: 512.
/dev/rdisk1 [ itachi HDS72107 K8OA70] EFI Sectors: 1465130703. Sector Size: 512.
/dev/rdisk3 [easyRAIDERP16R-8FR3 0001] MBR Sectors: 30701535. Sector Size: 512.
/dev/rdisk5 [G-TECH G-SPEED -VOL#00 R001] MBR Sectors: 19507167. Sector Size: 512.
/dev/rdisk4 [easyRAIDERP16R-8FR3 0001] MBR Sectors: 54662619103. Sector Size: 512.
/dev/rdisk6 [G-TECH G-SPEED -VOL#01 R001] MBR Sectors: 27324196831. Sector Size: 512.

The last 4 lines are the LUNs of the two RAID systems.

vinnih's picture

Hello LFrancois,

the guys from the studio just admited that they rebooted one of the MacPro's under windows yesterday evening and in the morning the problem occur. Can you please help what to do next?

Thank you

abstractrude's picture

I have been able to bring back SANS like this from the dead. If they didn't actually make changes to the disk in windows you should be ok. You have to relabel the luns using cvlabel. Sounds like your not really following best practices though, no dedicated controllers, users managing the machines.

Note, they probably converted the LUN to windows after they booted up.

You essentially need to get out old cvlabel output, which is saved in logs. then reapply to your luns. This may be over hard if your new to Xsan sorry.

-Trevor Carlson
THUMBWAR

vinnih's picture

Hello Trevor,

thank you for your reply. I hope that we will be able to bring back to life that SAN without loosing info. We have dedicated MDC and a secondary one in stand by, but indeed we left the clients stations to the users, something we will change in the future.

Can you help how can I recover the LUN labels. I checked the cvlabels man. It seems that I have to create a cvlabel file. I checked the logs but not sure what to look for and the current cvlabel file /Library/Preferences/Xsan/ which consists of two lines:
XsanLUN3 /dev/rdisk7 - EFI
XsanLUN4 /dev/rdisk3 - EFI

Kind regards

abstractrude's picture

You actually need the old cvlabel output from when you created the volume. Do you have any backups of the old cvlabel file? can you run a cvgather on the metadata controller and post it to a dropbox so I can take a look.

Thanks

-Trevor Carlson
THUMBWAR

vinnih's picture

Hello Trevor,

unfortunatelly I do not have a backup of the cvlabel file. I have send you a link to a dropbox folder with the result of the cvgather command.

vinnih's picture

Hello Trevor,

and thank you again for taking the time and helping me.

Here is the output of cvlabel -L

server:~ server$ sudo cvlabel -L
Password:
/dev/rdisk1 [ itachi HDS72107 K8OA70] EFI Controller 'default', Serial 'unknown', Sector Size 512, Sectors Max 1465130703 (698.6GB)
/dev/rdisk7 [Apple Disk Image 0.8v34] EFI Controller '06ab95313990c405629aad229310a1c4', Serial '06ab95313990c405629aad229310a1c4', Sector Size 512, Sectors Max 2078687 (1015.0MB)
/dev/rdisk0 [ itachi HDS72107 K8OA70] EFI Controller 'default', Serial 'unknown', Sector Size 512, Sectors Max 1465130703 (698.6GB)
/dev/rdisk3 [easyRAIDERP16R-8FR3 0001] MBR Controller 'default', Serial '2465000612666464', Sector Size 512, Sectors Max 30701535 (14.6GB)
/dev/rdisk5 [G-TECH G-SPEED -VOL#00 R001] MBR Controller 'default', Serial '001B4D280000E37A', Sector Size 512, Sectors Max 19507167 (9.3GB)
/dev/rdisk4 [easyRAIDERP16R-8FR3 0001] MBR Controller 'default', Serial '2264000612336438', Sector Size 512, Sectors Max 54662619103 (25.5TB)
/dev/rdisk6 [G-TECH G-SPEED -VOL#01 R001] MBR Controller 'default', Serial '001B4D280100E37A', Sector Size 512, Sectors Max 27324196831 (12.7TB)

This “MBR” is not good I guess?

Unfortunately the output from cvlabel -D is

server:~ server$ sudo cvlabel -D GTech
cvlabel: couldn't dump label for disk "GTech"
server:~ server$ sudo cvlabel -D SIF309
cvlabel: couldn't dump label for disk "SIF309"

vinnih's picture

Hi Trevor,

here what I come up with:

from the cvgather command I think I found the correct LUN labels - XsanLUN2, XsanLUN4, XsanLUN3, XsanLUN1

[1121 11:02:31] 0x7fff7c7bc180 NOTICE PortMapper: CVFS Volume XsanLUN2 on device: /dev/rdisk3 (blk 0x1000009 raw 0x1000009) con: 1 lun: 0 state: 0x74 inquiry [easyRAIDERP16R-8FR3 0001] controller # 'default' serial # '2465000612666464' Size: 30701535 Sector Size: 512
[1121 11:02:31] 0x7fff7c7bc180 NOTICE PortMapper: CVFS Volume XsanLUN4 on device: /dev/rdisk5 (blk 0x100000b raw 0x100000b) con: 1 lun: 0 state: 0xf4 inquiry [G-TECH G-SPEED -VOL#00 R001] controller # 'default' serial # '001B4D280000E37A' Size: 19507167 Sector Size: 512
[1121 11:02:31] 0x7fff7c7bc180 NOTICE PortMapper: CVFS Volume XsanLUN1 on device: /dev/rdisk4 (blk 0x100000a raw 0x100000a) con: 1 lun: 1 state: 0x74 inquiry [easyRAIDERP16R-8FR3 0001] controller # 'default' serial # '2264000612336438' Size: 54662619103 Sector Size: 512
[1121 11:02:31] 0x7fff7c7bc180 NOTICE PortMapper: CVFS Volume XsanLUN3 on device: /dev/rdisk6 (blk 0x100000c raw 0x100000c) con: 1 lun: 1 state: 0xf4 inquiry [G-TECH G-SPEED -VOL#01 R001] controller # 'default' serial # '001B4D280100E37A' Size: 27324196831 Sector Size: 512

I have created a cvlabel file which now looks like this:

CvfsDisk_UNKNOWN /dev/rdisk1 # host 0 lun 0 sectors 1465130703 sector_size 512 inquiry [ itachi HDS72107 K8OA70] serial unknown
CvfsDisk_UNKNOWN /dev/rdisk0 # host 0 lun 0 sectors 1465130703 sector_size 512 inquiry [ itachi HDS72107 K8OA70] serial unknown
CvfsDisk_UNKNOWN /dev/rdisk3 # host 1 lun 0 sectors 30701535 sector_size 512 inquiry [easyRAIDERP16R-8FR3 0001] serial 2465000612666464
CvfsDisk_UNKNOWN /dev/rdisk5 # host 1 lun 0 sectors 19507167 sector_size 512 inquiry [G-TECH G-SPEED -VOL#00 R001] serial 001B4D280000E37A
CvfsDisk_UNKNOWN /dev/rdisk4 # host 1 lun 1 sectors 54662619103 sector_size 512 inquiry [easyRAIDERP16R-8FR3 0001] serial 2264000612336438
CvfsDisk_UNKNOWN /dev/rdisk6 # host 1 lun 1 sectors 27324196831 sector_size 512 inquiry [G-TECH G-SPEED -VOL#01 R001] serial 001B4D280100E37A

and modify it to look like this:

CvfsDisk_UNKNOWN /dev/rdisk1 # host 0 lun 0 sectors 1465130703 sector_size 512 inquiry [ itachi HDS72107 K8OA70] serial unknown
CvfsDisk_UNKNOWN /dev/rdisk0 # host 0 lun 0 sectors 1465130703 sector_size 512 inquiry [ itachi HDS72107 K8OA70] serial unknown
CvfsDisk_XsanLUN2 /dev/rdisk3 # host 1 lun 0 sectors 30701535 sector_size 512 inquiry [easyRAIDERP16R-8FR3 0001] serial 2465000612666464
CvfsDisk_XsanLUN4 /dev/rdisk5 # host 1 lun 0 sectors 19507167 sector_size 512 inquiry [G-TECH G-SPEED -VOL#00 R001] serial 001B4D280000E37A
CvfsDisk_XsanLUN1 /dev/rdisk4 # host 1 lun 1 sectors 54662619103 sector_size 512 inquiry [easyRAIDERP16R-8FR3 0001] serial 2264000612336438
CvfsDisk_XsanLUN3 /dev/rdisk6 # host 1 lun 1 sectors 27324196831 sector_size 512 inquiry [G-TECH G-SPEED -VOL#01 R001] serial 001B4D280100E37A

Do you think I am on the right path here?

Thank you

vinnih's picture

Hello Trevor,

here is what I come up with:

From the cvgateher command I found the original LUN labels (i think)

[0918 10:31:04] 0x7fff7baba180 NOTICE PortMapper: CVFS Volume XsanLUN2 on device: /dev/rdisk3 (blk 0x1000009 raw 0x1000009) con: 1 lun: 0 state: 0x74 inquiry [easyRAIDERP16R-8FR3 0001] controller # 'default' serial # '2465000612666464' Size: 30701535 Sector Size: 512
[0918 10:31:04] 0x7fff7baba180 NOTICE PortMapper: CVFS Volume XsanLUN4 on device: /dev/rdisk5 (blk 0x100000b raw 0x100000b) con: 1 lun: 0 state: 0xf4 inquiry [G-TECH G-SPEED -VOL#00 R001] controller # 'default' serial # '001B4D280000E37A' Size: 19507167 Sector Size: 512
[0918 10:31:04] 0x7fff7baba180 NOTICE PortMapper: CVFS Volume XsanLUN1 on device: /dev/rdisk4 (blk 0x100000a raw 0x100000a) con: 1 lun: 1 state: 0x74 inquiry [easyRAIDERP16R-8FR3 0001] controller # 'default' serial # '2264000612336438' Size: 54662619103 Sector Size: 512
[0918 10:31:04] 0x7fff7baba180 NOTICE PortMapper: CVFS Volume XsanLUN3 on device: /dev/rdisk6 (blk 0x100000c raw 0x100000c) con: 1 lun: 1 state: 0xf4 inquiry [G-TECH G-SPEED -VOL#01 R001] controller # 'default' serial # '001B4D280100E37A' Size: 27324196831 Sector Size: 512

then I have created a cvlabel file which looks like that

CvfsDisk_UNKNOWN /dev/rdisk1 # host 0 lun 0 sectors 1465130703 sector_size 512 inquiry [ itachi HDS72107 K8OA70] serial unknown
CvfsDisk_UNKNOWN /dev/rdisk0 # host 0 lun 0 sectors 1465130703 sector_size 512 inquiry [ itachi HDS72107 K8OA70] serial unknown
CvfsDisk_UNKNOWN /dev/rdisk3 # host 1 lun 0 sectors 30701535 sector_size 512 inquiry [easyRAIDERP16R-8FR3 0001] serial 2465000612666464
CvfsDisk_UNKNOWN /dev/rdisk5 # host 1 lun 0 sectors 19507167 sector_size 512 inquiry [G-TECH G-SPEED -VOL#00 R001] serial 001B4D280000E37A
CvfsDisk_UNKNOWN /dev/rdisk4 # host 1 lun 1 sectors 54662619103 sector_size 512 inquiry [easyRAIDERP16R-8FR3 0001] serial 2264000612336438
CvfsDisk_UNKNOWN /dev/rdisk6 # host 1 lun 1 sectors 27324196831 sector_size 512 inquiry [G-TECH G-SPEED -VOL#01 R001] serial 001B4D280100E37A

and I edited it to look like that replacing UNKNOWN with the LUN labels

CvfsDisk_UNKNOWN /dev/rdisk1 # host 0 lun 0 sectors 1465130703 sector_size 512 inquiry [ itachi HDS72107 K8OA70] serial unknown
CvfsDisk_UNKNOWN /dev/rdisk0 # host 0 lun 0 sectors 1465130703 sector_size 512 inquiry [ itachi HDS72107 K8OA70] serial unknown
CvfsDisk_XsanLUN2 /dev/rdisk3 # host 1 lun 0 sectors 30701535 sector_size 512 inquiry [easyRAIDERP16R-8FR3 0001] serial 2465000612666464
CvfsDisk_XsanLUN4 /dev/rdisk5 # host 1 lun 0 sectors 19507167 sector_size 512 inquiry [G-TECH G-SPEED -VOL#00 R001] serial 001B4D280000E37A
CvfsDisk_XsanLUN1 /dev/rdisk4 # host 1 lun 1 sectors 54662619103 sector_size 512 inquiry [easyRAIDERP16R-8FR3 0001] serial 2264000612336438
CvfsDisk_XsanLUN3 /dev/rdisk6 # host 1 lun 1 sectors 27324196831 sector_size 512 inquiry [G-TECH G-SPEED -VOL#01 R001] serial 001B4D280100E37A

Do you think I am on the right path? Where should I put the new cvlabel.txt file?

Kind regards

abstractrude's picture

We built a new label file
XsanLUN2 /dev/rdisk3 # host 1 lun 0 sectors 30701535 sector_size 512 inquiry [easyRAIDERP16R-8FR3 0001] serial 2465000612666464
XsanLUN4 /dev/rdisk5 # host 1 lun 0 sectors 19507167 sector_size 512 inquiry [G-TECH G-SPEED -VOL#00 R001] serial 001B4D280000E37A
XsanLUN1 /dev/rdisk4 # host 1 lun 1 sectors 54662619103 sector_size 512 inquiry [easyRAIDERP16R-8FR3 0001] serial 2264000612336438
XsanLUN3 /dev/rdisk6 # host 1 lun 1 sectors 27324196831 sector_size 512 inquiry [G-TECH G-SPEED -VOL#01 R001] serial 001B4D280100E37A

cvlabel filename

-Trevor Carlson
THUMBWAR

vinnih's picture

Hello Trevor,

it worked. After I run the cvlabel command pointing to the file with the correct labels, the two volumes immediately showed up. I rebooted and then turn on all the other computers in the xsan and the volumes mounted on all of them.

Thank you Trevor for your help. You are great. It is very difficult to get support arround here and we can only count on people like you in difficult situations like that.

Again thank you for your time, efforts and paintiense.

Kind regards

abstractrude's picture

Good to hear. Happy xsaning.

-Trevor Carlson
THUMBWAR