Case study · Rescue
Two communities, rebuilt from a drive that came back empty.
A root-level server compromise destroyed the live databases behind WhiteBlaze and HammockForums. What initially appeared to be a total loss became two functioning, searchable communities with decades of history preserved.
The situation
WhiteBlaze and HammockForums are long-running outdoor communities. Between them they hold decades of posts: trail reports, gear arguments, photographs, and the accumulated knowledge of people who have been posting to the same place for many years.
An attacker gained root on the server. The live databases were destroyed. When we came to look at the source drive, it presented as empty.
For a community, that is not the loss of a website. It is the loss of the reason anybody visits.
The constraints
- Two separate communities, each with its own database, users and uploaded content, both affected.
- No intact copy of the live data to restore from. Recovery had to start from what could be pulled off the disk itself.
- The archive was the asset. A working forum with the history missing would have been a failure dressed as a success.
- Whatever came back had to be trustworthy. Partially reconstructed data that silently loses posts or misattributes them is worse than an outage, because nobody notices until much later.
What we did
We took forensic images of the drive and worked from the images, not the original, so that nothing we did could destroy evidence or reduce what was still recoverable.
Recovery then came from four directions at once:
- Raw data carving. Locating database and file content in unallocated space by its structure, where the filesystem no longer had a reference to it.
- Filesystem metadata. Using what the filesystem still recorded about deleted content to place and identify the fragments that carving recovered.
- Historical backups. Older copies that were incomplete or out of date, but which gave a known-good skeleton to reconcile fragments against.
- Custom reconstruction tooling. Written for this job, to reassemble records from fragments, resolve conflicts between sources, and rebuild the relationships between them.
The tooling was most of the work. Recovering a table is one problem. Rebuilding a forum is another: a post has to belong to the right thread, the thread to the right forum, the attachment to the right post, and every one of those to the right member. Get any of it wrong and the result looks restored while being false.
Millions of records were recovered and reassembled in this way: users, threads, posts, attachments, galleries, blogs, subscriptions, and the data belonging to custom applications the boards had accumulated over the years.
The rebuilt sites were then validated before going anywhere near production, put behind Cloudflare, and given a new backup system writing to multiple locations, so that a single compromised host could not produce this situation twice.
The result
Two functioning, searchable communities, with decades of history preserved and their members' contributions still attributed to them.
Both are online. You can go and read them.
What this cost, and what it says
This was an expensive way to arrive at a working forum. Forensic recovery is slow, and it is charged by the hour because nobody can estimate it honestly in advance.
Almost every recovery we are called into follows a period when nobody was watching the system. That is not a criticism of the people who own these boards, most of whom are running them for love rather than money. It is the reason Infrastructure Care exists, and the reason it is the cheaper half of what we do.
Lost something?
If your board is down, or your host has told you the data is gone, tell us before anybody writes anything else to that disk.