ZODB: repair PosKeyErrors in Plone/Zope

Some times you have a broken ZODB database because of harddisk problems and sadly no suitable backup exist. Then we can try to repair the ZODB database. Below you can find the needed steps to get a running ZODB database and a script for repairing PosKeyErrors in ZODB.

First of all we need to recover the ZODB with fsrecover.py script which comes with ZODB.

./bin/zopepy ../buildout-cache/eggs/ZODB3-3.10.5-py2.7-linux-x86_64.egg/ZODB/fsrecover.py -P 0 var/filestorage/broken-Data.fs var/filestorage/Data.fs

Then we need to analyze the ZODB to get a list of known PosKeyErrors. To do this we use fsrefs.py script which comes with ZODB.

./bin/zopepy ../buildout-cache/eggs/ZODB3-3.10.5-py2.7-linux-x86_64.egg/ZODB/scripts/fsrefs.py -v var/filestorage/Data.fs > fsrefs-output.txt

Then we run the following script as follows to fix the broken references with fake objects. After that, we should have a working database and can see what data is still alive and can be saved.

Script: fix-broken-references.py

Save the following code into a script called fix-broken-references.py under your buildout directory and call it as followed.

./bin/instance run fix-broken-references.py

import re
import os
import transaction as zt
from ZODB.utils import p64
from persistent import Persistent


refs_file = open('fsrefs-output.txt', 'r')
oids = []
for line in refs_file:
    rematch = re.search(r'oid\s+(?P<oid>[\w\W]+?) .*', line)
    if not rematch:
        continue
    oids.append(rematch.group('oid'))

i = 0
for oid in set(oids):
    zt.begin()
    i += 1
    print("create fake obj [%s] for: %s" % (i, str(oid)))
    a = Persistent()
    try:
        oid_int = int(oid, 16)
    except ValueError:
        print("ValueEror on %s, skip!" % oid)
        zt.abort()
        continue
    a._p_oid = p64(oid_int)
    a._p_jar = app._p_jar
    app._p_jar._register(a)
    app._p_jar._added[a._p_oid] = a
    zt.commit()
print("finished!")