El Capitan - Automount

ljungholms's picture

ljungholms's picture

Does anyone know where the auto mount settings have moved for 10.11? The plist file isn't bring created any more in the normal location. Thanks

brianwells's picture

ljungholms's picture

Thanks, I tried that, modified the configprofile and no luck. Spoke with apple and quantum and still have no luck. Any more ideas?

csanchez's picture

Mount settings are saved in /etc/fstab. For example, to always mount a volume as read-only:

LABEL=VolumeName none acfs ro

See the fstab man page for basic info. You can specify any valid comma-separated list of mount options as per the mount_acfs man page.

When the mount point is "none", OS X mounts the volume at /Volumes/VolumeName. You can specify a different mount point as long as it's a directory that exists before the mount attempt.

rowie's picture

I've been having trouble with auto mount on a 10.11.2 client mac client (the only El Capitan client I've tested with Xsan MDCs) - and dug fairly deep, unsuccessfully. The problem is only getting the volume to mount on boot, everything else is working fine. Here is what I've tried and found:

1. There is a profile loaded with both xsan and preference payloads, and onlyMount key set to auto mount one volume (out of several). When I install the profile, the expected volume mounts immediately, no others do, and everything is perfect. This indicates there is nothing wrong with the profile, the volume, zoning, etc and the auto mount feature "works" if the system is already booted and I load the profile at that point.

2. Looking at the nssdbg log after booting and the volume not mounting shows nothing obviously relevant. At first I thought maybe the LUNs weren't being scanned by the time the mount attempt happens, triggering the failure (a long standing issue with macs, except they usually retry until successful), and that there may be something needed in order to get it to retry after all LUNs are found. I thought that because in the nssdbg.out the disk scan successfully finding the LUNs always happens just after the "new mount registered" entry (sometimes within the same second, which may mean its just a display output that is run after the mount attempt happens). After some troubleshooting, I now doubt this is the issue, or the *only* issue I should say. As a test we zoned out all other storage controllers except the one chassis that contained the MDJ, and 2 LUNs that make up the one volume. These 3 LUNs should be able to be scanned plenty fast enough to register the LUNs before the mount attempt unless there is an order of operations issue in the code.

3. I went a step further and used xsanctl to add several options to fstab (in order to see if I could get more info and potentially make the client continue attempting to mount once the LUNs are found) but I could never get those options to be respected or show an effect. For example: I ran:

xsanctl mount volname --mnt_retrans=10 --mnt_recon=hard --mnt_type=bg --verbose=yes --debug=yes"

which mounted the volume successfully and automatically altered the fstab line to look like this:

LABEL=volname none acfs rw,mnt_retrans=10,mnt_recon=hard,mnt_type=bg,verbose=yes,debug=yes

Next i used the "mount" command to see if the options would be displayed for this mount, but they weren't, only "acfs" and "local" were there. I also watched the system and nssdbg logs, and did not notice any uptick in verbosity or debug info. I then rebooted as I did during every other test, never did the volume mount on boot, and no increase in log verbosity. I believe the options would only show in mount output and have an effect if the volume was mounted via fstab (as it would during boot), so....

4. I took my fstab experiment one step further and this is where I hit a wall. I issued:

xsanctl mount --at=/Volumes/volname

to explicitly set the mount point. The command successfully mounted the volume, and again automatically altered the fstab to include the specified option, in this case mount point, and removed all other options. I then unmounted the volume, created the /Volumes/volname directory manually and issued "mount -a" in order to have the mount routine use the options listed in fstab - in this case, only the mount path being specified, all else just standard (rw and acfs). I got a strange output pasted below which seems to indicate there is a potential issue with fstab, which may also explain the auto mount on boot not working:

bash-3.2# mount -a

======================================

mount_acfs: argc = 5
mount_acfs: argv[0] =
mount_acfs: argv[1] = <-o>
mount_acfs: argv[2] =
mount_acfs: argv[3] =
mount_acfs: argv[4] =
Thu Dec 17 18:40:03 MST 2015
find_IORegistryBSDName: searching for
IOServiceMatching dictionary
{type = mutable dict, count = 1,
entries =>
0 : {contents = "IOProviderClass"} = IOMedia
}
get_IORegistry_info: find_IORegistryBSDName for returned 2
Filesystem NOT mounted on /Volumes/volname

======================================

Does anyone know what this means?

Has anyone successfully managed to get 10.11.2 to auto mount an Xsan volume with Xsan MDCs?

Any thoughts on what might be the issue?

Thanks for any feedback or help.

typical log messages after boot:


Dec 17 17:54:01 machinename fsmpm[205]: XsanPostVolumeStateChange: sending request for volume 'volname'
volume = volname;
Dec 17 17:54:30 machinename kernel[0]: CVFS 'volname': Buffer Cache blksize 4096, #blocks 0 dirty low/high 0/0
Dec 17 17:54:30 machinename kernel[0]: CVFS 'volname': request reserved space 0x11a00000
Dec 17 17:54:30 machinename kernel[0]: Not all drives available on stripe group 1 for filesystem 'volname'
Dec 17 17:54:30 machinename kernel[0]: Could not mount filesystem volname, cvfs error 'No such device' (36)

sgljungholm's picture

I still have no luck with this. I have been playing with different settings.

I edited my XSAN profile with the following

"<key>onlyMount</key>
<array>
<string>san1</string>
<string>san2</string>
<string>san3</string>
</array>"
When I load the profile the drives all appear. So far so good. The FSTAB file is not modified and remains empty. I tried the xsanctl mount and still no modification to the fstab file. I then edited the file and added the lines with no luck.

Hopefully we can get this worked out soon.

abeazy's picture

We have been having this issue at a few different client locations. Here is what I have tested/learned.

All systems running El Capitan that are trying to connect to and mount an Xsan volume using an external Promise SANLink2 are having automount issues where the volume does not automount when the computer is rebooted.

If the system is running El Capitan and using a PCIe fibre card (ATTO or LSI tested) the automount works as expected.

Test performed

First thing we tried, after noticing it was longer there in El Capitan, is to simply add the automount.plist back in the /Library/Preferences/Xsan folder to see if it would be that easy. Of course, that didn’t work.

I searched for the issue and found the initial post by LJUNGHOMS as well as the Apple KB article with the Xsan Preferences payload instructions.

Trying to get the onlyMount defaults write to work was my first attempt. Using the following:

sudo defaults write /Library/Preferences/com.apple.xsan.plist onlyMount ‘VOLUME_NAME'

No luck.

We had two systems to test with. Luckily one was auto mounting correctly and the other was not, both running El Capitan. In Terminal, I compared all of the files in the Xsan folder, plist files, anything I could think of that pertained to Xsan. Everything was identical.

I tried editing the /System/Library/LaunchDaemons/com.apple.xsan.plist to see if I could change the LaunchOnlyOnce key to see if maybe for some reason in El Capitan, the machine was only requesting the auto mount once instead of continuously. Because of SIP, I wasn't able to. I copied file to the /Library/LaunchDaemons folder and was able to edit it there. I then unloaded:

sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.xsan.plist

and reloaded from the /Library folder:

sudo launchctl load -w /Library/LaunchDaemons/com.apple.xsan.plist

The volume mounted as expected but after a reboot, sadly, did not automount.

After jumping around in Terminal trying to edit config and plist files to get something to work, I started to look at hardware and do some further testing focusing on hardware differences.

What I uncovered:

When a machine with a PCIe fibre card (MacPro 2009 or Xserve) is rebooted running El Capitan, the Xsan volume automounts without issue.

When a machine with a Promise SANLink is rebooted running El Capitan, the Xsan Volume does not automount but can be mounted running:

sudo xsanctl mount ‘VOLUME_NAME’

After a reboot, however, the volume does not automount.

When a machine in a Sonnet xMac Thunderbolt enclosure with a PCIe fibre card is rebooted, the Xsan Volume mounts without issue.

The issue seems to stem from connectivity through a Promise SANLink2. I have submitted a ticket with Promise and am working with them to confirm the issue and hopefully they will release a driver update to address the issue.

Can everyone that is experiencing the issue confirm that they are in fact using SANLink’s as your method of connectivity for fibre? Also, does anyone else have any instances of non-SANLink systems that could be upgraded to verify that automount still works with those systems?

If there is an instance where someone is using something other than a Promise SANLink and experiencing the issue in El Capitan, then I'm heading down the wrong path. But, if everyone can confirm they are ONLY seeing the issue using SANLinks, I would advise submitting a ticket to Promise to confirm that you are experiencing the issue as well.

sgljungholm's picture

I thought the same thing but sadly this doesn't seem to be the case. I am testing more this morning.

I have seen the issue with both the sanlink2 and the Sonnet Echo external box. I am thinking this has more to do with the machine not seeing the cards during boot as they are on a thunderbolt cable.

I use JAMF to control all the machines and am experimenting with login scripts to run a terminal command to mount the volumes.

Not an ideal fix.

abeazy's picture

Promise has confirmed the issue is related to the SANLink driver. They have also confirmed they are working on a new driver for El Capitan that should fix the problem.

As far as the Sonnet Echo external box, we do not have one to test with but have tested with the Sonnet xMac enclosure (which is also Thunderbolt) and the volume auto mounts without issue.

abeazy's picture

We received a beta driver from Promise and applied it to three of our systems that were not automounting after reboot. After the install of the driver and a reboot, all three systems correctly automount the Xsan volume.

If there is still an issue with the Sonnet Echo external box, look to see if there is a newer driver available or reach out to Sonnet for assistance.

rowie's picture

Confirming we were using a Promise Sanlink2, and the beta driver appears to have resolved the issue. Thanks for the tip abeazy!

What driver was provided to you? Was it 1.2.4b2?

- Philaneous

rowie's picture

Yes, 1.2.4b2

How many LUNs are in your set up?

- Philaneous

abeazy's picture

No problem, Rowie!

One other thing to point out, we have found that this does not fix the issue on workstations using an original SANLink. I should say workstation, we only have one to test with. I have requested an updated driver for this as well but not sure if one will be issued.

For those that the driver fixed the issue, what are the numbers of LUNs that are zoned to each initiator?

- Philaneous

Gerard's picture

Same issue.

Running XSAN v10.11.2
Using a Sanlink 2 (8Gb model)

The XSAN mount doesn't appear after a restart/shutdown unless I;
Delete and reinstall the profile file that I receive from the MDCs.
Mount the XSAN via terminal (sudo xsanctl mount "name of volume")

Just got the v1.2.4b from Promise. Deleted the old driver from /Library/Extensions, per Promise Support, installed the beta then restarted.

My XSAN volume appeared then again after two restarts to verify things work.
Restarted again, but no mount comes up. So, it's a hit/miss for me right now.

Gerard's picture

Just to confirm, this is the beta file that I received:
F_MacDrv_V1.2.4_109_1010.pkg

Is that the same that so of you also received?

I'm seeing the same issue with the released version of the SANlink2 driver (F_MacDrv_V1.2.5_109_Above.zip)

It looks to me like some sort of race condition, where the filesystem is attempted to be mounted before the FibreChannel system is fully online and available to the OS, and then after failing it OS doesn't try and mount the volume again.

brianwells's picture

You could try adding a delay like had to be done at various points in the past: https://discussions.apple.com/thread/4434393?tstart=0

The following is untested but is likely to work. Another alternative is to contact AppleCare Enterprise for assistance.

A quick way to do this could be to write out this script to /usr/local/bin/wait4sixty

#!/usr/bin/ruby
sleep 60
exec(*ARGV)

Make sure the script is executable with:

chmod +x /usr/local/bin/wait4sixty

Then copy /System/Library/LaunchDaemons/com.apple.xsan.plist to /Library/LaunchDaemons/com.apple.xsan.plist and change the Program and ProgramArguments parameters like this:

<key>Program</key>
<string>/usr/local/bin/wait4sixty</string>
<key>ProgramArguments</key>
<array>
<string>wait4sixty</string>
<string>/System/Library/Filesystems/acfs.fs/Contents/bin/xsand</string>
</array>

Finally, unload the system xsand.plist and load the one with the delay:

launchctl unload -w /System/Library/LaunchDaemons/com.apple.xsan.plist
launchctl load -w /Library/LaunchDaemons/com.apple.xsan.plist

Gerard's picture

I've been working with Apple Enterprise support with using a delayed Xsan mounting. So far, no luck with the script that was created for my environment.

JSamuel's picture

Seems less than ideal (and more painful than a driver-based remedy or JAMF deviceman work...) but you could use fibreconfig (various queries and greps) to guage link status and visibility, prior to having it query the xsan .plist files / LDAP for mount config (should it mount or not) and then issue the mount command?

Something in-addition-to as opposed to in-the-middle-of feels best when manipulating system/Xsan files, that will no doubt be wiped during any kind of upgrade and/or SIP.

As said, driver-based (or most other things) would be best of course :-)

We have 10.10.x clients with Sonnet chassis, we'll take this as a warning when considering their upgrade.

Joel Samuel.
/thirtytwo - Consultancy & Direction
Proud sponsor of Xsanity.com

All contributions are my own personal opinions - not those of any entity I represent.

rowie's picture

Looks like something changed on the OS level in 10.11 (up to 10.11.2 so far) where the mount retries that used to happen until the LUNs were finished scanning, no longer happens. So if the LUNs aren't ready by the time the mount attempt happens, it fails (which was typical at that point in the process), but crucially, it now does not retry as it used to. This may be considered a "feature" or "fix" by Apple since it's likely intended to stop system hangs and increase stability and boot speed. Hopefully if that's the case, or even if it's just a bug, they'll find a way to put the retries back in 10.11.3 or .4, perhaps just for systems that have Xsan enabled.

The beta Promise driver essentially just speeds up the LUN scanning process to try to get more LUNs scanned in time for the mount attempt, but any volumes with LUNs that haven't been scanned in time will still fail to automount. Looks like systems with around 12 or 13 LUNs can be scanned quick enough now, but more than that may not work, and depending on other speed impacting factors, YMMV. If you have priority volumes with a small enough set of LUNs that need automounting, you may want to zone only those LUNs, and perhaps minimize the amount of zoned fibre paths, and see if that helps as a workaround.

@Gerard: If you have a borderline amount of LUNs, small variations in the timing may be the reason behind the inconsistent mounting. And the full filename of the beta driver I received is: F_MacDrv_V1.2.4b2_109_1010.pkg

Gerard's picture

Rowie,

After talking to a XSAN engineer from Apple, we were able to get a mount script going. I am able to restart the 10.11/Promise Sanlink workstation and still have the volume up upon login.

Rowie, you are correct about O/S level changes in 10.11. and speed of the LUN scans.

rowie's picture

Gerard, glad you were able to come away with a working solution. Would you mind posting it here to help the community work around the issue until Apple more permanently fixes it?

abstractrude's picture

Yes please share!

-Trevor Carlson
THUMBWAR

Gerard's picture

Here is the script. Once downloaded, change the extension from .txt to .tgz

Run this command:
sudo tar xvzpf (path to .tgz) -C /

You should see something like this next:
x ./
x ./._Library
x ./Library/: Can't set user=0/group=0 for Library
x ./usr/: Can't set user=0/group=0 for usrCan't update time for usr
x ./usr/local/
x ./usr/local/libexec/
x ./usr/local/libexec/xsandelay.py
x ./Library/LaunchDaemons/
x ./Library/LaunchDaemons/com.apple.support.ht205706.xsandelay.plist
tar: copyfile unpack (./Library) failed: Operation not permitted

Note:
When extracting the files with tar, if you see errors specifically about /Library or /usr, you can ignore them.

The first file is the script
The second is the launchd job that kicks off at startup.
Now, the script needs to be activated. To do so, run:
sudo launchctl load -w /Library/LaunchDaemons/com.apple.support.ht205706.xsandelay.plist

Restart, login and wait a few seconds for the Xsan volume to appear.

Rather than simply waiting for a specific amount of time, the script delays Xsan start until the number of targets and LUNs stops changing.

Depending on your network infrastructure, the mount might take a little bit to load, but it does mount and does reappear after a reboot/shutdown.

Hope this helps.

attachment: 
JSamuel's picture

Gerard wrote:

Rather than simply waiting for a specific amount of time, the script delays Xsan start until the number of targets and LUNs stops changing.

/quote

My Python is a little rusty, but, that is not what it is doing.

Gerard wrote:

Depending on your network infrastructure, the mount might take a little bit to load, but it does mount and does reappear after a reboot/shutdown.

/quote

The loop is one second, so it will enforce the DELAY_SECONDS. If Xsan tries to run before then (which it will, as that LaunchDaemon plist is still there) all it does is stop it and loop until DELAY_SECONDS has lapsed.

Joel Samuel.
/thirtytwo - Consultancy & Direction
Proud sponsor of Xsanity.com

All contributions are my own personal opinions - not those of any entity I represent.

JSamuel's picture

^ I broke the quote tags... :-/

Joel Samuel.
/thirtytwo - Consultancy & Direction
Proud sponsor of Xsanity.com

All contributions are my own personal opinions - not those of any entity I represent.

MattG's picture

...however most folks will probably need to increase the DELAY_SECONDS to something greater than 60.

BlackF1re's picture

Did someone tried the 10.11.3 to check if they fixed the issue without the need of a script?

Personal Website: http://gabriele-zanon.branded.me
Twitter: @GabrieleZanon83
Linkedin: https://it.linkedin.com/in/neverimpossible

MattG's picture

The script is still necessary.

JSamuel's picture

I realise that Xsanity can attract some less seasoned colleagues looking for help, so a word of warning about deploying any code (in general) you do not understand and so on.

It's 11pm my time, so I might be missing something, but that HT is also not published.

We'll schedule some time to give this a whirl (same code as provided, and/or our own code that does similar) and if reasonable to do, upload the source for a flat pkg, so those looking to Casper/Munki can do so. Unsure when.

Joel Samuel.
/thirtytwo - Consultancy & Direction
Proud sponsor of Xsanity.com

All contributions are my own personal opinions - not those of any entity I represent.

csanchez's picture