Troubleshoot your VPS with Bootable ISOs
Introduction
Like physical servers, virtual servers sometimes need rescue. System problems can occur for many reasons. The system may run out of space, Misbehaving applications can corrupt the disk, or failed updates may leave the system unbootable. If the system isn't accessible, use a bootable rescue ISO to resolve the issue.
Make a Backup
Before attempting any rescue, back up your VPS. Vultr offers manual snapshots and automated backups for VPS instances. If you do not have automatic backups enabled, make a manual snapshot. Attempting to rescue a server without a backup is reckless. There is no substitute for a sound backup strategy. You can create a server snapshot in a few clicks in the Vultr Console.
You are responsible to back up all your mission-critical data, following your local laws and organizational policies. Vultr does not maintain additional backups outside your Vultr Console.
Mount a Bootable ISO
There are several bootable ISOs in the Vultr ISO library suitable for system rescue work.
- Finnix
- SystemRescue x64
You may choose any ISO in the Vultr ISO library, or upload one. To attach an ISO to your server, go to the Server Information page in the Vultr Console. Select Settings > Custom ISO. Select your ISO, then click Attach ISO and Reboot.
Your server will reboot from the ISO. Connect to your server with the web console to perform the rescue steps.
Note: The Finnix ISO will time out and revert to the VPS operating system if you wait more than 60 seconds. If this happens, click the Send CtrlAltDel button at the top-right of the console to reboot the server to the ISO boot screen again. The SystemRescue x64 ISO does not revert back to the VPS operating system after booting.
Connect via SSH
The Vultr web console is limited by the capabilities of noVNC, which is HTML-based. For easier access to the server, enable SSH in your rescue environment.
Finnix is lightweight and useful for low-memory environments, but many of its tools are outdated. For more about Finnix, see the Vultr Doc, "Using Finnix Rescue CD to Rescue, Repair, or Backup Your Linux System".
If possible, we recommend using SystemRescue x64 for most tasks. The majority of examples in this guide use SystemRescue x64.
To enable SSH in SystemRescue x64:
Boot the SystemRescue x64 ISO.
Select Boot SystemRescue using default options
Set the root password with passwd.
Until the next reboot, SystemRescue x64 uses the new password. Setting the root password for SystemRescue x64 does not permanently change the root password of your server.
- SystemRescue x64 automatically starts the SSH server. You can stop it with
systemctl sshd stopor restart it withsystemctl sshd restart.
Connect to the server as root with an SSH client, or use SFTP to access mounted disks. Locate the server IP address on the server information page. If you have previously connected to the server over SSH, be aware that the server's fingerprint will be different. You may need to remove the old entry from ~/.ssh/known_hosts before SSH connects.
Common Linux Rescue Scenarios
Locate the name of the VPS disk device
Use lsblk to locate your VPS disk and partition device names.
This example shows vda1 is a 55GB partition. Inspect further with blkid.
/dev/vda1 is the only ext4 partition on the server, confirming that /dev/vda1 is the VPS data device. Your server configuration may be different, make sure you clearly understand your data devices and names before proceeding.
Check the filesystem
Use fsck to check and correct filesystem corruption.
Use df -h to see the mounted disks.
Verify the VPS disk (vda in this example) is not in the list of mounted devices. Never run fsck on a mounted disk. Do not continue until you are sure the disk is unmounted.
Check the filesystem with fsck.
If no problems are detected:
If errors are found, fsck will request permission to fix the issue.
Remove the ISO and reboot your server to verify the problem is resolved once fsck completes.
Mount VPS filesystem
Mount /dev/vda1 to a folder if you'd like to access the files directly. This mounts the vda1 partition to /mnt.
Browse /mnt to see the entire root filesystem. Use SCP or SFTP to transfer files from the system.
Change Root
Changing root (chroot) changes the apparent disk root directory, which is useful for system maintenance, such as reinstalling the bootloader or resetting passwords. This topic is complex. We recommend reviewing "What's the proper way to prepare chroot to recover a broken Linux installation?" to set up chroot properly. Once chroot is enabled, you can change user passwords, remove or install packages, and other tasks.
Check VPS system logs
Check the log files for clues about server problems. For example:
Verify the logfiles exist and have current dates.
View the last 500 lines of syslog.
Check VPS disk space / filesystem inodes
If the VPS disk runs out of either blocks or inodes, your server will not function properly.
Check free blocks on device /dev/vda1 with
df -h.Check the inodes on device /dev/vda1 with the
-iswitch.
If the server is out of either blocks or inodes, delete some files or move them to another device.
Repair the Linux Bootloader
Bootloader problems vary by distribution, and specific instructions are beyond this short troubleshooting guide. This summary of GRUB and GRUB2 repair is useful if you are using SystemRescue x64. Here are some guides for popular distributions.
If your distribution isn't listed, we recommend a web search for "how to repair boot loader [distro name]".
Windows Password Reset
The Vultr Doc "Reset Windows Server Administrator Password" explains how to reset a lost Windows administrator password.
0 Comments
Be the first to comment and share your perspective with the community.