Advertisements

Ubuntu 12.04: NFS Mounts vs. Upstart

Back in the old days, the Unix startup sequence was rigidly fixed. For a variety of reasons, that’s no longer the case; Ubuntu (and, presumably, other distros) now use upstart, which turns the startup sequence into a lightly documented Pachinko machine. This parallel processing presumably works great for most of Ubuntu’s use cases and falls flat on its face for me: I’m apparently the only person who expects NFS mounts to be in place before signing in.

Well, maybe other folks expect that, but the entire startup mechanism is apparently broken as designed.

The only solution seems to be stalling the user sign-on screen by jamming the display manager until the NFS client hauls itself to its feet. This takes up to a minute, for reasons I do not understand, but it’s better to let it run to completion rather than signing on and expecting one’s files to be in the right places. Email clients, in particular, have difficulty coping with missing files.

The fix involves adding a line to /etc/init/lightdm.conf, as mentioned there (albeit with incorrect syntax):

start on ((filesystem
           and runlevel [!06]
           and started dbus
           and (drm-device-added card0 PRIMARY_DEVICE_FOR_DISPLAY=1
                or stopped udev-fallback-graphics)
           and mounted MOUNTPOINT=/mnt/bulkdata)
          or runlevel PREVLEVEL=S)

I tried to check for another filesystem that should also be mounted, but, as I understand neither the syntax nor the semantics of the language, what you see is what finally worked. As it turns out, upstart's syntax error messages aren’t particularly helpful; a single line (helpfully relating, perhaps, that the parser expected a token on line 16) appears on VT 7, but if you don’t know to switch from VT 1, you’ll never get even that minimal assistance. No, such errors don’t appear in the /var/log/upstart/* logs.

For unknown reasons, waiting for the remote-filesystems event didn’t delay the startup at all. Evidently, mountall emits that event almost immediately, long before the NFS mounts happen. Perhaps the event occurs even when the mount fails, contrary to what the doc suggests?

Most of the debugging occurred through an ssh session across the room. Edit the file, try a new version, reboot, watch for the filesystems to come up, watch for the sign-in screen to appear. Or not, as the case may be.

Grumpy though I may seem, the great thing about Open Source / Free Software is that when it breaks, you have access to all the pieces and can actually fix the problem. That makes up for nearly everything, I’d say.

No, I didn’t update any of those bug reports or start another one. It’s obvious this isn’t getting any attention, so what’s the point? If you’re also having the problem, you’ll eventually wind up here…

FWIW, I knew the NFS mounts weren’t working because I always set the screen background to an image on the file server: no mount = no picture = fix-the-problem-now. This image seemed appropriate:

XB-70A Cockpit

XB-70A Cockpit

Back then, transistors were countable resources…

[Update: The previous picture link, now broken, was to http://www.nasaimages.org/luna/servlet/detail/nasaNAS~2~2~2995~104520. The revised link points to a description on archive.org, with versions of the picture available for download.]

Advertisements

  1. #1 by AR on 2013-01-10 - 14:56

    Found this while trying to delay startup of mysql until nfs share is mounted; seems your case:
    http://serverfault.com/questions/157430/how-to-ensure-nfs-home-directory-is-mounted-before-login
    Initially I did not realize the file mentioned was /etc/init/… instead of /etc/init.d as I thought.

  2. #2 by AR on 2013-01-10 - 15:09

    Additionally if not using autofs, you can use the keyword “mounted” within the start on section of the /etc/init/xx.. script, i.e.

    start on ( mounted MOUNTPOINT=/media/theNFSShare and runlevel [2345] )

    • #3 by Ed on 2013-01-10 - 20:16

      That’s the ticket! I think my troubles with the other filesystem were due to not getting the Boolean operators quite right, but in practice it seems that when one NFS volume mounts after the long delay, all the others happen at the same time.

      Thanks for the update…

  3. #4 by Patrick on 2013-02-27 - 12:04

    I had the same problem with NFS. And was also getting slightly frustrated with the design of upstart.

    We eventually figured out the following solution:

    – specifying ‘noauto’ for the NFS mounts in /etc/fstab
    – add mount /the_nfs_share to /etc/rc.local

    This effectively avoids the delay in NFS mounting (for my machine at least)!

    The reason for the long delay of the NFS mounts is apparently that the network is not quite there, when
    mountall tries to connect to the NFS server. Since rc.local is the last job started for the runlevel, the network is there
    already and it mounts without any delay.

    -Patrick

    • #5 by Ed on 2013-02-27 - 12:16

      specifying ‘noauto’ for the NFS mounts

      Now that would never have occurred to me!

      Since rc.local is the last job started for the runlevel

      That appears to be one of the assumptions that upstart upends: at least on Ubuntu 12.x, rc.local starts asynchronously with regard to everything.

      I did manage to create a job that starts after eth0 comes up, so I can mount the NFS shares there and it ought to Just Work. I don’t know what interaction that will have with starting lightdm, but maybe I should change that to trigger on the same event.

      Thanks for the suggestion…

  4. #6 by Izio on 2013-05-04 - 18:12

    I’m also facing the same problem, and I found a very elegant solution here:
    http://jurjenbokma.com/ApprenticesNotes/job_ordering_in_upstart.html

    Basically, you just need to drop a .conf file like the following in you /etc/init directory, without the need to modify a packaged init job.


    # /etc/init/wait-for-nfs-mount.conf
    # Wait for remote filesystem mount

    description “Delay kdm until remote filesystems are mounted”

    start on (starting kdm and mounted MOUNTPOINT=/mnt/whatever)

    task

    pre-start script
    echo “Waiting for NFS mount…”
    end script

    script
    echo “NFS filesystem should be mounted now”
    end script

    Hope this helps…

    – Izio

    • #7 by Ed on 2013-05-04 - 19:43

      The solution turned out to be even simpler: don’t mount the NFS shares in fstab, then do them “by hand” in a script that runs when the network comes up. There’s no hard interlock, but the timing has worked out perfectly…

      Opinion: That we must all grope around figuring this solution out for what should be a common task says that the overall system design wasn’t quite fully baked.

      • #8 by keber on 2013-07-09 - 18:52

        Hi,

        This thread helped me a lot. I’m trying something similar to populate a tmpfs NFS share from a hardcopy on startup, It works now thanks to your work around, but I was wondering if someone figures out why the mountall over /etc/fstab doesn’t trigger a mounted event.

        • #9 by Ed on 2013-07-09 - 22:00

          mountall over /etc/fstab doesn’t trigger a mounted event

          The only thing I can think of is that the fstab mounts fail when NFS isn’t quite ready to go. The failure either doesn’t generate the proper signals or generates a different signal that everybody ignores. How to confirm any of that, without deep-diving the source, remains unknown…

  1. Ubuntu 12.04: GRUB2 Tweaks « The Smell of Molten Projects in the Morning
  2. Ubuntu 12.04: Random Crashes « The Smell of Molten Projects in the Morning
  3. Upstart vs. rc.local « The Smell of Molten Projects in the Morning
  4. Xubuntu 12.04: Some Steps Forward, Some Steps Back « The Smell of Molten Projects in the Morning
  5. Poking Around in Upstart | The Smell of Molten Projects in the Morning
  6. Upstart vs. NFS Mounts vs. LightDM: Success! | The Smell of Molten Projects in the Morning
  7. Upstart vs. NFS Mounts vs. Display Manager: Resolved! | The Smell of Molten Projects in the Morning