Open Directory not starting on XSAN metadata controller
Help! Everything with our metadata controllers have been running fun. I have two MDC and am on XSAN 4.1/El Cap. I have added a client to the XSAN since the upgrade so I'm pretty sure nothing bad happened during the upgrade.
One of my MDC's has been acting funky - I think it's an issue with the 8 Promise Pegasus2 arrays I have connected it it via Thunderbolt - and I've had to do a few hard reboots over the last couple of weeks.
Well, yesterday I went to add a new client to the XSAN (we had to wipe and reinstall the OS on a previous client). It failed and the error led me to find that my MDC that's been giving me trouble doesn't have Open Directory running. OD won't start and from other searching it seems that the database has become corrupted. Running sudo /usr/libexec/slapd -Tt gives the error:
57880743 bdb(cn=authdata): /var/db/openldap/authdata/id2entry.bdb: unexpected file type or format
57880743 bdb_db_open: database "cn=authdata": db_open(/var/db/openldap/authdata/id2entry.bdb) failed: Invalid argument (22).
57880743 backend_startup_one (type=bdb, suffix="cn=authdata"): bi_db_open failed! (22)
I've tried recovering it using various methods posted on different forums but am having no luck at all. I did not know to be archiving my OD Master - so I do not have an archive.
The XSAN environment seems to be working fine otherwise - all the clients are happy. I just can't add the new client.
I'm only using OD for the XSAN environment - I don't use it for users or groups (the XSAN clients just have a local user).
Environment: 2 XSAN MDCs; 7 XSAN clients; all running 10.11.5.
Here's some questions:
1) is there anyway to add a client with the XSAN in this state?
2) Does Time Machine help me at all (I have had it running)?
3) How do I get this fixed long term?
Thanks for any help!