How To Fix “rpcbind dead but pid file exists” Error in RHEL 6


My job as a Linux Admin revolves a lot around NFS. My current set-up has 3 dedicated NFS servers & 29 NFS clients. So, the performance of NFS is always a top priority. But of late, I had been facing challenges with a few clients where NFS services wouldn’t start 😦 Threw below error. And because of this, client machines would experience ridiculously long hangs 😦

[root@serverlog]# service nfs status
rpc.svcgssd is stopped
rpc.mountd dead but subsys locked
nfsd dead but subsys locked

And when I tried to start it, it got hung.

[root@server log]# service nfs start
Starting NFS services: [ OK ]
Starting NFS mountd: [FAILED]
Starting NFS daemon:

Error log showed this : –

[root@server ~]# dmesg | grep nfs
[ 26.527173] FS-Cache: Netfs 'nfs' registered for caching
[ 27.574444] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[ 448.380948] svc: failed to register nfsdv2 RPC service (errno 110).
[ 508.498842] svc: failed to register nfsaclv2 RPC service (errno 110).
[ 809.057252] nfsd: last server has exited, flushing export cache
[ 1169.683243] svc: failed to register nfsdv2 RPC service (errno 97).
[ 1169.712310] svc: failed to register nfsaclv2 RPC service (errno 97).
[ 1470.200560] nfsd: last server has exited, flushing export cache

Below are the steps how I fixed this issue.

Since NFS service depends on rpcbind service, first step was to check rpcbind status. Got this when checked.

[root@server ~]# service rpcbind status
rpcbind dead but pid file exists

I then tried it bring it up. No luck 😦 It does show OK, but it wouldn’t start!

[root@server ~]# service rpcbind start
Starting rpcbind: [ OK ]

Also, saw this : –

[root@server ~]# rpcinfo -p
rpcinfo: can't contact portmapper: RPC: Remote system error

So, drilling deep into the logs, I found that server couldn’t get UID of rpc.

[root@server ~]# cat /var/log/messages | grep -i rpcbind
Feb 18 17:03:09 server kernel: [602777.346862] xs_local_setup_socket: unhandled error (111) connecting to /var/run/rpcbind.sock
Feb 18 17:08:21 server rpcbind: cannot get uid of 'rpc': Success

But rpc ID was present on the system.

[root@server ~]# id -a rpc
uid=32(rpc) gid=32 groups=32

Then, why on earth would it show above error log about missing UID? Going further deep, I found that rpc ID was not present in the password file.

[root@server ~]# cat /etc/passwd | grep -i rpc
[root@server ~]# 

So, that was the catch 😉 I edited the passwd file with below contents.

rpc:x:32:32:Rpcbind Daemon:/var/cache/rpcbind:/sbin/nologin
rpcuser:x:29:29:RPC Service User:/var/lib/nfs:/sbin/nologin

Now both rpcbind service & NFS service are up 😀 🙂 😉

[root@server ~]# vi /etc/passwd
[root@server ~]# service rpcbind start
Starting rpcbind: [ OK ]
[root@server ~]# service rpcbind status
rpcbind (pid 64975) is running...
[root@server ~]# service nfs status
rpc.svcgssd is stopped
rpc.mountd dead but subsys locked
nfsd dead but subsys locked
[root@server ~]# service nfs start
Starting NFS services: [ OK ]
Starting NFS mountd: [ OK ]
Starting NFS daemon: [ OK ]
Starting RPC idmapd: [ OK ]
Advertisements

2 thoughts on “How To Fix “rpcbind dead but pid file exists” Error in RHEL 6

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s