Recovering Production Services After Failed Linux and Database Updates
Software updates are usually routine maintenance. But when an update is interrupted, performed outside the expected package-management workflow, or runs into limitations in the original server layout, the result can be anything from a single broken service to a server that no longer boots. In several customer environments, the Unihost support team had to recover production services after unsuccessful operating system, kernel, and database updates. Where possible, the objective was to restore the existing environment rather than reinstall the operating system, while preserving customer data and minimizing changes to the working configuration. Two cases illustrate the most common types of problems we encountered.
Case 1: Linux Kernel Update Failed Because /boot Ran Out of Space
A Linux server stopped booting correctly after a kernel update. The problem was not related to the new kernel itself. The server had been deployed using a standard data-center operating system template where the /boot partition was only 1 GB. Over time, several kernel versions and their associated initramfs files accumulated in /boot. During the next kernel upgrade, there was no longer enough free space to complete the update correctly.
Technical challenge
Kernel updates write several files to /boot, including kernel images, initramfs files, and bootloader-related data. If the partition becomes full during an upgrade, the package manager may leave the system in an inconsistent state.
Possible consequences include:
- an incomplete kernel installation;
- a missing or damaged initramfs;
- incorrect bootloader configuration;
- the system booting only with an older kernel;
- in the worst case, the server fails to boot normally.
The immediate task was to restore bootability without affecting the customer’s application data.
What the Unihost team did
The recovery process focused first on determining which kernel versions were installed and which files in /boot were still required.
The support team:
- reviewed the existing /boot contents;
- identified obsolete kernel files and packages;
- freed enough space to complete the interrupted update;
- repaired the affected kernel package state where required;
- regenerated the necessary boot files;
- verified the boot configuration;
- confirmed that the operating system could start normally again.
The customer’s application data and existing services did not need to be rebuilt.
Preventing the same problem
The incident also highlighted a weakness in the original server partitioning. Some automated operating system templates still allocate approximately 1 GB to /boot. This may be sufficient for an initial installation but leaves little margin for years of kernel updates. For systems we prepare or repartition, we normally allocate at least 2 GB for /boot. The additional capacity provides significantly more headroom for kernel packages and reduces the likelihood that a routine update will fail simply because the boot partition is full.
Result
The existing operating system installation was preserved and the server returned to normal operation without reinstalling the system or restoring application data from backup. The partitioning issue was also identified so it could be avoided during future deployments.
Case 2: Recovering MySQL and Web Services After an Unsupported Database Migration
A more complex incident occurred on a server running a hosting control panel. The customer’s developer had attempted to migrate the database environment manually from MySQL to MariaDB. Instead of performing the migration through the hosting control panel and its supported package-management process, database packages were replaced manually.
This created inconsistencies between:
- the installed database packages;
- the hosting control panel;
- package dependencies;
- the database configuration;
- the web-server environment.
After the changes, both the database and web services stopped operating correctly.
Additional complication: an obsolete operating system
Under normal circumstances, the first recovery options would include rolling back the affected packages or reinstalling the correct versions from the operating system repositories. In this case, the server was running an outdated operating system release. Its original official repositories were no longer available through the standard package sources. This meant that a simple package rollback could not be performed immediately. Reinstalling the entire operating system was one possible solution, but it would have required rebuilding the environment and migrating the customer’s applications and data. The goal was therefore to recover the existing installation first.
What the Unihost team did
The support team approached the incident as a package and configuration recovery rather than immediately rebuilding the server.
- Investigated the package state
The installed MySQL and MariaDB components were reviewed together with the package dependencies expected by the hosting control panel. This made it possible to identify which manually introduced changes were preventing the panel from managing the database stack correctly.
- Restored access to package sources
Because the operating system had reached end of life, the normal repositories were no longer available. The team located working archive mirrors containing the required packages for that operating system version and temporarily restored access to the packages needed for recovery.
- Rolled back the unsupported changes
Manually installed database components were removed or rolled back where necessary. The objective was to return the server to a package state that the hosting control panel could manage again.
- Installed MariaDB through the supported control-panel workflow
Once the package environment was restored, MariaDB was installed through the hosting control panel rather than directly through the operating system package manager.
This ensured that the database version matched the panel’s supported configuration, package dependencies were handled correctly, service configuration remained consistent with the hosting environment, future database updates could again be managed through the panel.
- Corrected the control-panel configuration
The manual migration had also left configuration inconsistencies inside the hosting control panel. The support team adjusted the affected configuration so that the panel correctly recognized and managed the restored MariaDB service.
- Verified dependent services
After the database was restored, the team checked the web-server environment and the services depending on the database. The server was returned to normal operation without requiring a complete operating system reinstall.
Why the recovery approach mattered
The fastest theoretical response to a badly damaged package environment is often: reinstall the operating system, restore data, rebuild services. But this is not always the least disruptive option. When the existing system can be recovered safely, repairing the package state and configuration can avoid a complete server rebuild, application migration, DNS or service changes, additional downtime and unnecessary data transfer. In this case, the combination of an unsupported database migration and an obsolete operating system made the repair significantly more complicated than a normal package rollback. The critical part was reconstructing a package state that both the operating system and the hosting control panel could manage again.
Engineering takeaway
Both incidents originated from different technical problems, but they had something in common: a routine infrastructure change became a production incident because the underlying environment had little tolerance for failure. The first case was caused by an undersized /boot partition. The second was caused by changing a panel-managed database stack outside the supported update mechanism. For production systems, update preparation should therefore include more than simply running the package manager.
Please check before major operating system, kernel, database, or control-panel updates:
- available disk space, especially /boot;
- current backup status;
- operating system support status;
- repository availability;
- control-panel compatibility;
- database upgrade path;
- rollback options;
- dependencies between services.
For environments managed by a hosting control panel, database and web-stack upgrades should normally be performed through the mechanisms supported by that panel unless there is a specific migration plan.
Unihost support work included:
- failed update diagnostics;
- Linux package recovery;
- kernel and boot troubleshooting;
- database recovery;
- MariaDB installation and configuration;
- hosting control panel troubleshooting;
- repository recovery for legacy operating systems;
- web-service restoration;
- configuration correction;
- post-recovery validation.
Result
Both environments were restored without requiring an immediate full operating system reinstall. The affected services were returned to operation while preserving the existing customer data and application environment. The incidents also resulted in configuration improvements designed to reduce the risk of similar failures during future updates.
Results at a glance: production services restored, existing data preserved, failed package changes rolled back, database environment recovered, boot configuration repaired, future update risks reduced.
