Corrupted Xsan4 setup on OS X Server app on primary and secondary MDC

Tags: 

Primary and Secondary metadata controllers GUIs stopped showing all the XSAN volumes in the Server App.

Primary and secondary controllers were showing missing OD replica as well.

Used cvadmin to stop and unmount the volumes
Removed secondary controller from Xsan
Backup Xsan Config - sudo cp -r /Library/Preferences/Xsan /Library/Preferences/Xsan.old
Destroy the OD MASTER in Server App GUI
Move Server App to the Trash
Destroy any remnants of OD - sudo slapconfig -destroyldapserver
Remove Certs - sudo rm -R /var/root/Library/Application\ Support/Certificate\ Authority
Remove Kerberos - sudo rm -R /var/db/krb5kdc
Remove Xsan Config - sudo rm -r /Library/Preferences/Xsan
Move Server app back to Applications
In Server App Created New OD MASTER
In Server App turned on Xsan (For SAN NAME used the exact same name [SAN_NAME])
Close Server app

Opened up the new Xsan config file (config.plist) at /Library/Preferences/Xsan and copied and pasted the following into the original config file from our old Xsan configuration:

certSetRevision [ID]
sanRevision [ID]
sanUUID [ID]

Copy the original Xsan config files and fsmlist I made a backup of into /Library/Preferences/Xsan
Copy those config files into LDAP - sudo xsanctl pushConfigUpdate

Got this error after executing command
2015-10-15 05:03:05.570 xsanctl[36615:130261] buildSanConfig started
2015-10-15 05:03:05.570 xsanctl[36615:130261] buildSanConfig about to check LDAP
2015-10-15 05:03:05.589 xsanctl[36615:130261] buildSanConfig step 4 with {
globals = {
certSetRevision = "632228D9-C182-4E6F-A1B0-B729BCCB640A";
controllers = {
"30CDC770-A2A1-5F0D-970F-22FBBE04C72B" = {
IPAddress = "10.0.0.11";
hostName = "md.prestonwood.org";
};
};
fsnameservers = (
{
addr = "10.0.0.11";
uuid = "30CDC770-A2A1-5F0D-970F-22FBBE04C72B";
}
);
notifications = {
FreeSpaceThreshold = 20;
};
revision = "6D28EF16-9BCE-4672-A7A6-86582921257B";
sanAuthMethod = "auth_secret";
sanConfigURLs = (
"ldaps://md.prestonwood.org:389"
);
sanName = pbcxsan;
sanState = active;
sanUUID = "D6CB40CB-7AA4-4DF4-AF84-B751AB436C79";
sharedSecret = "********";
};
volumes = {
};
}
2015-10-15 05:03:05.593 xsanctl[36615:130261] *** Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '*** -[__NSPlaceholderDictionary initWithObjects:forKeys:count:]: attempt to insert nil object from objects[1]'

      • First throw call stack:

(
0 CoreFoundation 0x00007fff8f02a03c __exceptionPreprocess + 172
1 libobjc.A.dylib 0x00007fff91fc476e objc_exception_throw + 43
2 CoreFoundation 0x00007fff8ef15e3f -[__NSPlaceholderDictionary initWithObjects:forKeys:count:] + 383
3 CoreFoundation 0x00007fff8ef2babb +[NSDictionary dictionaryWithObjects:forKeys:count:] + 59
4 acfs 0x000000010f6d3bb9 read_fsnameservers + 358
5 acfs 0x000000010f6cf5d9 buildSanConfig + 3012
6 acfs 0x000000010f6ce98e buildSanConfigC + 155
7 xsanctl 0x000000010f66de21 command_pushConfigUpdate + 61
8 xsanctl 0x000000010f66d623 main + 247
9 libdyld.dylib 0x00007fff984bf5c9 start + 1
)
libc++abi.dylib: terminating with uncaught exception of type NSException
Open server app and volumes do not appear in the Server App

Volumes do mount on Primary MDC via xsanctl mount volume_name

I even reinstalled OS X and attempted the steps above and the result was the same. I could mount the SAN volumes and access the data on the SAN volumes only on the MDC; however the MDC would not show the volumes in the GUI. The only way I was able to get it back up and running was that I had a disk image backed up of the MDC and restored it back to the HD. That obviously brought everything back. For my knowledge, I would like to avoid restoring the disk image in case this happens again.

Is there anything else that I could have done?