Recovering from a corrupted git repo

I do a lot of work on the go. Offline. Sometimes it takes a long time to push changes to a remote repository. As always, Murphy’s law applies, and the one repo that explodes into my face is the one with ten days worth of work in it.

While working, suddenly my laptop hang. Music looping. No mouse movement. Nothing. The only possible solution was to do a cold-reboot. I was not worried. Everything was saved, and I only changed a few lines and can easily recover if something went awry. So I rebooted.

Once back in the system, I immediately wanted to do a git status and git diff. Git spat back the following error message:

My Laptop had shown some strange and erratic behaviour over the last few days already. I already left a memtest running for about 24 hours earlier this week without errors. The only possible explanation left was the hard-disk.

Fun times ahead! 10 days of work at risk… 10 days of important changes! Sweat building up my forehead. Bloody sweat!

I trust my tools to keep my code safe. I trust git. I trust vim. I do microscopic commits, and I knew my current uncommitted changes only involved a few lines. So maybe only the last commit got corrupted? Let’s see…

First I dug around my existing code base. Fired up vim to look at the source and noticed some files have become empty. The 0 bytes kind of empty! Luckily vim keeps swap files and I recovered from there. The code I recovered with vim was completely correct. It even had the last line I was working on. Good. So I have my code in a good state.

But this leaves my git history. I did not want to lose all my commits, and I did not want to re-clone from the remote repo and make one huge clunky commit.

As my code was safe now, I decided to dig into git.

My understanding at that point: git saves everything into the objects folder. Commits, tree-ishes, … everything. Every commit has a pointer to it’s parent. So if I find the last valid commit, I can reach my history again. With that knowledge, hop in.

So the plan of action was simple: Sort objects by date, and from there, find the latest commit. And somehow get rid of the corrupted objects (as it turned out, the corrupted objects were also 0-byte files).

To find the latest commit, I decided to write a quick-and-dirty (… very dirty) Python script to order the objects by date. Let me quickly add some comments for readability’s sake:

So far so good. Running this gives me a list of objects, sorted by date. Example output:

Would you look at that: 9bd41c2f96f295924af92a9da175cb3686f13359 is the last saved object. The same as reported by the git status I ran in the beginning!

For the next part, I ran git show starting from the end, writing down each commit which was corrupted. This gave me the list of the following objects:

More than I expected … But still… let’s continue:

It turned out, they were all afflicted by the same illness. All had zero bytes. So lost… Forever! To get git working again, I simply deleted them, and re-ran git status:

Hmmm… the corrupted object is gone… but now git cannot find HEAD? But that’s only a simple file in .git pointing to something in .git/refs/heads… surely it hasn’t…

Well fuck! Well, essentially, not too bad. refs/heads/backend is just a file containing an object name (hash). So all I would need to do is to find the proper hash and put it into the file. But let’s first dig into the git docs. I vaguely remember a git fsck command. And sure enough. It exists, with a --lost-found option. Sounds about right. Let’s run it:

According to the docs, this should give me something in .git/lost-found:

Interesting. 5 commits, and two file blobs (as it turns out). This is looking good! I was working on the “backend” branch. And only on that branch, so the error above with refs/heads/backend is correct.

In theory, all that is left to do is look at the commits in lost-found, take the last one (by date), make refs/heads/backend point to that one, and make HEAD point to the proper ref. Finding the last one is easy, run git show on each object, and note the dates.

So, 24b1869bbbb63f2e6fb831bd55c0be6953443b39 is apparently the most recent commit!

Let’s recover HEAD, which still contains the correct pointer. But the pointer target (refs/heads/backend) is missing:

And tadaa:

This is already a lot better. There are still some errors. But the important thing is that the history between the local backend and origin/backend is back:

This is all I wanted. I now pushed this history back to the server, and to make sure everything is clean again, I will simple re-clone from the server, apply the pending local changes and I am back to a clean workspace.

All in all, in retrospect, the whole operation was very easy, and I am glad that I did not lose my history.

I will need to get myself a new HDD for my laptop. That much is certain! But I am also playing with the idea to set up a git hook to automatically sync to a connected USB stick. That way, I would have my commits on two different file systems, while offline. Should not be too difficult to set up… We’ll see.