ZODB: repair PosKeyErrors in Plone/Zope
First of all we need to recover the ZODB with fsrecover.py script which comes with ZODB.
./bin/zopepy ../buildout-cache/eggs/ZODB3-3.10.5-py2.7-linux-x86_64.egg/ZODB/fsrecover.py -P 0 var/filestorage/broken-Data.fs var/filestorage/Data.fs
Then we need to analyze the ZODB to get a list of known PosKeyErrors. To do this we use fsrefs.py script which comes with ZODB.
./bin/zopepy ../buildout-cache/eggs/ZODB3-3.10.5-py2.7-linux-x86_64.egg/ZODB/scripts/fsrefs.py -v var/filestorage/Data.fs > fsrefs-output.txt
Then we run the following script as follows to fix the broken references with fake objects. After that, we should have a working database and can see what data is still alive and can be saved.
Save the following code into a script called fix-broken-references.py under your buildout directory and call it as followed.
./bin/instance run fix-broken-references.py
import re import os import transaction as zt from ZODB.utils import p64 from persistent import Persistent refs_file = open('fsrefs-output.txt', 'r') oids =  for line in refs_file: rematch = re.search(r'oid\s+(?P<oid>[\w\W]+?) .*', line) if not rematch: continue oids.append(rematch.group('oid')) i = 0 for oid in set(oids): zt.begin() i += 1 print("create fake obj [%s] for: %s" % (i, str(oid))) a = Persistent() try: oid_int = int(oid, 16) except ValueError: print("ValueEror on %s, skip!" % oid) zt.abort() continue a._p_oid = p64(oid_int) a._p_jar = app._p_jar app._p_jar._register(a) app._p_jar._added[a._p_oid] = a zt.commit() print("finished!")