Unclean shutdown detected. for mongod process in Arbiter server

Following messages were received in “/var/log/mongo/mongod.log” of primary server  and other servers of replicatset

Sat Jul 30 02:48:08.631 [rsHealthPoll] couldn't connect to xxx.local:27017: couldn't connect to server xxx.local:27017

Same message was appearing on other arbiter server

when checked on arbiter server no mongod process was running.

[root@xxx ~]# ps -ef| grep mongo

root     10581 10533  0 09:43 pts/0    00:00:00 grep mongo

Starting Mongod process:

When starting mongod process with command “service mongod start”, it encountered following error:

# service mongod start

Starting mongod: about to fork child process, waiting until server is ready for connections.

forked process: 10345

all output going to: /var/log/mongodb/mongod.log

ERROR: child process failed, exited with error number 100

When checking /var/log/mongodb/mongod.log, following was logged.

Fri Oct 14 09:29:41.540 [initandlisten] MongoDB starting : pid=10345 port=27017 dbpath=/data/arb 64-bit host=controller-02

Fri Oct 14 09:29:41.540 [initandlisten] db version v2.4.14

Fri Oct 14 09:29:41.540 [initandlisten] git version: 05bebf9ab15511a71bfbded684bb226014c0a553

Fri Oct 14 09:29:41.540 [initandlisten] build info: Linux ip-10-154-253-119 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49

Fri Oct 14 09:29:41.540 [initandlisten] allocator: tcmalloc

Fri Oct 14 09:29:41.540 [initandlisten] options: { auth: "true", config: "/etc/mongod.conf", dbpath: "/data/arb", fork: "true", keyFile: "/opt/mongo/key/keyfile1", logappend: "true", logpath: "/var/log/mongodb/mongod.log", nojournal: "true", pidfilepath: "/var/run/mongodb/mongod.pid", port: 27017, replSet: "sitMongoSet", smallfiles: "true" }

**************

Unclean shutdown detected.

Please visit http://dochub.mongodb.org/core/repair for recovery instructions.

*************

Fri Oct 14 09:29:41.695 [initandlisten] exception in initAndListen: 12596 old lock file, terminating

Fri Oct 14 09:29:41.695 dbexit:

Fri Oct 14 09:29:41.695 [initandlisten] shutdown: going to close listening sockets...

Fri Oct 14 09:29:41.695 [initandlisten] shutdown: going to flush diaglog...

Fri Oct 14 09:29:41.695 [initandlisten] shutdown: going to close sockets...

Fri Oct 14 09:29:41.695 [initandlisten] shutdown: waiting for fs preallocator...

Fri Oct 14 09:29:41.695 [initandlisten] shutdown: closing all files...

Fri Oct 14 09:29:41.696 [initandlisten] closeAllFiles() finished

Fri Oct 14 09:29:41.696 dbexit: really exiting now

It appears mongod process was not cleanly shutdown or server might have rebooted leaving mongodb in unclean state.

Journal was disabled on this Arbiter, so I ran following command to recover the state.

mongod --dbpath /data/arb –repair

After running above command, I again received following errors:

Fri Oct 14 10:09:59.059 [initandlisten] MongoDB starting : pid=11079 port=27017 dbpath=/data/arb 64-bit host=controller-02

Fri Oct 14 10:09:59.060 [initandlisten] db version v2.4.14

Fri Oct 14 10:09:59.060 [initandlisten] git version: 05bebf9ab15511a71bfbded684bb226014c0a553

Fri Oct 14 10:09:59.060 [initandlisten] build info: Linux ip-10-154-253-119 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49

Fri Oct 14 10:09:59.060 [initandlisten] allocator: tcmalloc

Fri Oct 14 10:09:59.060 [initandlisten] options: { auth: "true", config: "/etc/mongod.conf", dbpath: "/data/arb", fork: "true", keyFile: "/opt/mongo/key/keyfile1", logappend: "true", logpath: "/var/log/mongodb/mongod.log", nojournal: "true", pidfilepath: "/var/lib/mongo/mongod.pid", port: 27017, replSet: "sitMongoSet", smallfiles: "true" }

Fri Oct 14 10:09:59.108 [initandlisten] couldn't open /data/arb/local.ns errno:13 Permission denied

Fri Oct 14 10:09:59.108 [initandlisten] error couldn't open file /data/arb/local.ns terminating

Fri Oct 14 10:09:59.108 dbexit:

Fri Oct 14 10:09:59.108 [initandlisten] shutdown: going to close listening sockets...

Fri Oct 14 10:09:59.108 [initandlisten] shutdown: going to flush diaglog...

Fri Oct 14 10:09:59.108 [initandlisten] shutdown: going to close sockets...

Fri Oct 14 10:09:59.108 [initandlisten] shutdown: waiting for fs preallocator...

Fri Oct 14 10:09:59.108 [initandlisten] shutdown: closing all files...

Fri Oct 14 10:09:59.108 [initandlisten] closeAllFiles() finished

Fri Oct 14 10:09:59.109 [initandlisten] shutdown: removing fs lock...

Fri Oct 14 10:09:59.109 dbexit: really exiting now

It was identified “/data/arb/local.ns” permission was not correct. So I ran following command to correct it (root was the owner)

chown mongod:mongod /data/arb/local.ns

Once above was fixed, mongod process started running fine.