vCenter Appliance fsck failed

While I have been working with a customer on upgrading their infrastructure to vSphere 5.5, they had a power outage during the night which got their full virtual infrastructure down including storage (Not sure what the story with their UPS/Power generator). After they have brought their virtual infrastructure backup up both their vCenter Appliance and VDP appliance and few other Linux VMs has failed the fsck check while it was booting and it was stuck at the following error:

=========================================

fsck failed.  Please repair manually and reboot.  The root
file system is currently mounted read-only.  To remount it
read-write do:

bash# mount -n -o remount,rw /

Atention:  Only CONTROL-D will reboot the system in this
maintanance mode.  shutdow or reboot will not work.

Give root password for maintenance
(or type Control-D to continue):

==========================================

Below is an actual screenshot of the error for you to compare it with what you got:

VMware vCSA fsck failed

Below is the steps I have followed to fix this on both vCSA and VDP Advanced:

  1. Stay Calm & Don’t Panic!
  2. Take a backup/snapshot in case the below steps does not work for you and you need to revert it back to the current state when you call support or call for help. (Hopefully you won’t need this, but it is like your car insurance policy you pay for it but you hope you will never need to use it).
  3. Enter your Root Password.
  4. PRESS ENTER
  5. Type: mount -n -o remount,rw
  6. PRESS ENTER
  7. You will get some complain and that you should check with: e2fsck  (Check the below screenshot to see what it looks like)
  8. Type: e2fsck -y /dev/sda3
  9. PRESS ENTER
  10. It will check for error and ask you if you want to fix errors (Make sure you answer with yes to fix each of the errors).
  11. press: CTRL+D to exit
  12. press: CTRL+D to  restart the system.
  13. This should get your vSphere Data Protection Appliance or vCenter Appliance into an operational state and should get you back on track.

The below screenshots show how I executed the above procedure and the expected output:

VMware vCSA running e2fsck

 e2fsck disk repair completed

As I mentioned earlier I was able to use the same procedure above to fix:

  • vCenter Appliance (vCSA) fsck failed error
  • vSphere Data Protection (VDP) fsck failed error
  • Other Linux appliance and VMs that had the same fsck failed error.

While at my customer this error was caused by a power outage, this same error can happen after any type of any non proper shutdown of your Linux VMs. Hope this help get you back on track. If this fix your problem please share your result in the comment. If it does not and you had to do things differently, please share it in the comments below so every one can benefit.

Comments

  1. Just what I needed! Thanks for the article, very helpful. Cheers Benjamin

  2. Thank you for the great write up. It was an exact keyword match with my issue. But unfortunately, the client does not have the root password for the VDP appliance. I attempted to reset the VDP password using this link:
    http://www.vladan.fr/how-to-reset-root-password-in-vdp/

    But that failed as the file system came up in read only mode. I tried the default “changeme” password but unfortunately – that didn’t work either. So the implementation engineer must have changed the root password -but it was not properly documented.

    Amir

  3. Hi,

    I get exactly this error when deploying a brand new vMA from OVF. I can’t enter a root password as I have never configured one. Any hints?

    Thanks in advance

  4. If the VA gives you error you may check the downloaded files are not corrupted – you may calculate MD5/SHA1 sum especially on the VMDK files. Or download them again.

    If the root password is not set – maybe there is default password? Did you check the doc? Or you can use this guide to boot in maintenance mode when the password is forgotten. This is a nice guide with screenshots http://adminramble.com/reset-lost-root-password-suse-linux-enterprise-server/ .

  5. > 5. Type: mount -n -o remount,rw
    This should give you an error – no mount point parameter. And also you should not do re-mount in R/W if you want to make fsck – this can be dangerous depending which file system should be fixed.

  6. Jonathan says:

    Very nice. This helped tremendously today. Now life is good again with vCenter.

  7. Just what I needed. You’re the man!

  8. THanks for this!

    Just as a note, my vcenter did not boot properly after fixinf sda3, needed to check the rest too 🙂

  9. Thanks, this works for me.

  10. Thanks a lot. It really saved my day.

  11. Thanks – this fixed my issue. I’m currently running a lab environment and every time I make progress on the course I’m following, I cleanly shut down the appliance to take a snapshot. Without fail, every time I do this I get this error on boot up again – weird. This has saved me a a massive headache. Thanks again.

  12. Worked like a champ for my home lab following a power outage. This blog was at the top of my Bing search (yes, Bing) and I’ve shared with my co-workers who have also occasionally experienced a failed vCenter Server Appliance fsck failure.

    Thank you for the great write up!

    Dave

  13. Glad you found it useful Dave.

  14. Thank you, this fixed my problem. Very helpful.

  15. Thanks man! I own you one!

  16. Just needed to use this with a vCenter 6 installation that experienced a power outage. The “shell.set –enabled true” command wasn’t working and I had no SSH access so I followed the second part of KB2069041 @ http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2069041 in order to get booted directly into bash. Forwhat ever reason I needed to go through this process twice until e2fsck said that the volume was clean. Thanks for your help.

  17. many thanks

  18. James Lenaburg says:

    Found this today after running into same problem on my VCSA6 VM, thanks for the write up.

  19. Hi Guys

    the cmds above didn’t work for me, however it pointed me in the right direction. Un mounted the drive and then used e2fsck command with following switches -y -v -f

    worked a treat.
    use following info http://www.linuxspy.info/606/when-and-how-to-runexecute-e2fsck/

  20. sorry had to add my correct email

  21. You rock Brother. Had a SAN Storage disconnect that poot me in the fsck failed loop.
    Thanks for awsome posts!

  22. Hot dam, worked like a charm for me breh!

  23. THANKS!!! This fixed the problem I was having.

  24. My root password isn’t working. I reinstalled vcenter 6 few times and it get stuck here and then my root password for maintenance don’t work.

  25. You just saved my arse with a powersupply that killed my host. When I replaced it, I got this exact error. Your steps years later did the trick and my lab environment is coming back online.

    Thank you sir,

    Ron

  26. My Pleasure Ron :).

  27. This article saved my lab! My NAS/SAN died and after I brought it back up, I was getting this error. I had shell disabled for my VCSA instance, so I had to do a workaround of getting into shell through modifying the boot process and then run these commands. It ended up working in the end and saved my bacon. For instructions on how to get into shell from the boot menu, I used an article on resetting the root password for when it gets locked out (but I didn’t reset the root password.)

  28. i got FSCK error when i powered on VCSA… i unable to enter the root password… i just typed it but says login failed.. any one help me in that

  29. Thanks, I got faced this issue today with 5.5 old environment 🙂

Trackbacks

  1. […] and see if its of any help. I was able to use it to recover few VMs with similar error to yours: http://www.virtualizationteam.com/se…ck-failed.html Good luck, […]

  2. […] e2fsck might be what you need Try to use the following command: e2fsck -y /dev/sda3 Replace the /dev/sda3 with the disk you are having a problem with. For more details, you can check the following post: http://www.virtualizationteam.com/se…ck-failed.html […]

Speak Your Mind

*