It was Friday around 3PM and most people were already gone for the day. One of the few employees left came by my desk and asked if something was up with the Exchange server. I checked Outlook – it said it was connected. Shift + F9 to force a reconnection and it failed out. Error 0x80004005. Last email received was 30 minutes ago. Great.
So error 0x80004005 says this:
The client operation failed. Microsoft Exchange Information Store.
Log onto the Exchange server and verify the services are all running. They all were running.
Check the event viewer log files – way too many errors to go through all of them while a production server was down, but this one stood out:
Event viewer showed this error:
A transient failure has occurred. The problem may resolve itself in a while. The service will retry in 56 seconds. Diagnostic information:
Cannot open mailbox /o=blah/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Configuration/cn=Servers/cn=blah/cn=Microsoft System Attendant.
Microsoft.Exchange.Data.Storage.ConnectionFailedTransientException: Cannot open mailbox /o=blah/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Configuration/cn=Servers/cn=blah/cn=Microsoft System Attendant. —> Microsoft.Mapi.MapiExceptionLogonFailed: MapiExceptionLogonFailed: Unable to open message store. (hr=0x80040111)
Restart the Information Store service.
Now I get this error in the event viewer:
MapiExceptionCallFailed: Unable to mount database. (hr=0x80004005, ec=1276)
Crap. I verify that we’re running Exchange enterprise and not standard (database store limits). I check the management console and see that the database is offline and refuses to mount. Since the database was already offline, I decided to run a repair on the DB.
Open a command prompt:
eseutil /p "C:\program files\microsoft\exchange server\mailbox\first storage group\mailbox database.edb"
This took roughly 30 minutes on a 44GB database.
Try to mount the database again. It fails but gives me a different error this time:
Exchange is unable to mount the database that you specified. Specified database: Exchange07\Storage Group 1\Mailstore 1; Error code: MapiExceptionCallFailed: Unable to mount database. (hr=0x80004005, ec=-515)
This means that I’m missing a required transaction log and that the DB won’t start without it. Great.
So I navigate to the first storage group log files (C:\program files\microsoft\exchange server\mailbox\first storage group)
I moved all of the files (excluding Mailbox Databse.edb and CatalogData-* folder) to another directory. Then I attempted to mount the DB again. Success!
Total downtime: 75 minutes
Active working time: 45 minutes
So what happened?
McAfee Antivirus grabbed a file from the transaction log before it could be written to the database. Since the transaction log was altered while it was trying to write to the database, the mailstore became slightly corrupt and required a repair. And because the transaction logs and the database were at different parts, even after the repair, the log files needed to be “destroyed” in order for the DB to be mounted again. Oh, the logs were recreated after successfully mounting the database.
Thanks for that, ive been looking for this for about 20 minutes now! Cheers.