Skip to main content

Serialization

msgpack and JSON with exact type tags, registering your own types, and why pickle is off by default.

A cache has a quiet second job: it’s a program that reads bytes it didn’t write, left on disk by whatever process got there first. That makes the serializer the most security-relevant piece of the whole library, and it has three jobs. What goes into the cache should come out as exactly what you put in. Encoding and decoding should be fast, because they happen on every call. And reading the cache should never run code that someone else left there. diskcache’s default, pickle, does the first two well and fails the third by design: unpickling runs whatever the bytes tell it to. diskcache-oxide gets all three, and pickle becomes something you opt into, one cache at a time.

Two formats

Each cache picks a structured format with serializer=:

cache = Cache('/var/cache/my-app')                       # msgpack, the default
readable = Cache('/var/cache/my-app-json', serializer='json')
  • msgpack (the default) is compact and fast. On a 450 KB page-cache entry it’s about 25× faster than tagged JSON, and it stores nested bytes without base64.
  • json is human-readable: open diskcache-oxide.sqlite in any SQLite browser and read your values. It uses serde_json with exact float round-tripping.

Plain str and bytes values skip both and are stored raw. Every row records how it was encoded, so if you change a cache’s serializer, existing entries stay readable.

Exact types

JSON turns tuples into lists and can’t store sets, datetimes, NaN, very large ints, or dicts with non-str keys. msgpack has some of the same gaps. The usual answer is to coerce quietly and let the bug show up later.

diskcache-oxide’s encoder, written in Rust, notices anything its format would change and tags it instead, so it decodes back to the same type:

Python value Comes back as
tuple a tuple, not a list
set a set
frozenset a frozenset
datetime.datetime a datetime, with its timezone and microseconds
datetime.date a date
decimal.Decimal a Decimal, exact
int wider than 64 bits the same int
float('nan'), inf, -inf the same float
bytes nested in a container bytes
dict with int (or other non-str) keys the same keys 
import datetime

cache.set('record', {
    'created': datetime.datetime.now(datetime.UTC),
    'point': (37.77, -122.42),
    'ids': {101, 102},
    'big': 2**128,
    'by_id': {1: 'one', 2: 'two'},
})

record = cache.get('record')
assert isinstance(record['point'], tuple)
assert record['by_id'][1] == 'one'

User data can’t be mistaken for a tag: a dict that happens to look like one is escaped on the way in and decoded as an ordinary dict on the way out.

What isn’t stored

A value that neither format can represent exactly, such as an instance of your own class, raises TypeError at set():

cache.set('service', MyService())
# TypeError: values of type MyService can't be stored without pickle (only exact built-in types round-trip)

Only exact built-in types count. A subclass of dict or str isn’t stored as its base type, because it wouldn’t come back as the class you stored. Failing at set() time means you find out where the value was created, not three requests later when something reads it. For a type of your own, or one from a library, register it.

Your own types

Most of what an app stores that isn’t a built-in is a type it doesn’t own: a bson ObjectId inside a session dict, a namedtuple a library hands back, the str subclass a Markdown renderer returns. Converting those by hand at every call site works, but it’s the kind of code that drifts. Instead, tell the codec once how to store the type:

from bson import ObjectId
import diskcache_oxide

diskcache_oxide.register_type(ObjectId, name='bson.ObjectId', encode=str, decode=ObjectId)

cache.set('session', {'id': ObjectId(), 'user': ObjectId()})
cache.get('session')['id']   # an ObjectId again

encode turns an instance into a value the codec can already store, built-ins or other registered types, nested as deep as you like. decode turns that value back into an instance. The registration is process-wide, so make it at import time, in every process that opens the directory.

A dataclass needs only a name. Its init=True fields are read straight off the instance, with no intermediate dict, and on the way out it’s rebuilt with cls(**fields), so defaults and __post_init__ still run:

@dataclasses.dataclass
class Episode:
    id: int
    title: str
    published: datetime.datetime | None = None

diskcache_oxide.register_type(Episode, name='tp.Episode')
cache.set('episode:491', Episode(491, 'Rust and Python'))

The write path skips dataclasses.asdict(), which deep-copies: on a 20-field dataclass a set() goes from about 23 µs to 18. A read still builds one keyword dict and calls the constructor, so it saves almost nothing (4.0 µs against 3.8), and that’s fine: a cache is mostly reads, and what you gain is the call sites.

What the registry guarantees:

  • Nothing is imported by name. A registered value is stored as {"$t:bson.ObjectId": "..."}, and a reader looks that name up in its own registry. A process that hasn’t registered it gets ForeignValueError, and the row stays for one that has. That is the whole difference from pickle.
  • The name is yours. Rename or move the class and the stored rows still read, because the tag is the name you chose, not the class’s path.
  • Lookup is by exact type. A subclass of a registered class is its own registration. The codec’s own types (str, dict, datetime and the rest) can’t be registered, since their wire form is a contract across versions; their subclasses can, which is how a namedtuple or a str subclass gets in.
  • A registered type is never pickled, even on a cache with pickle=True. Once the encoder has used a registration for any part of a value, that value is stored through it or set() raises: if its encode raises, or returns something the codec can’t store. You registered it to stay off pickle, and a silent fallback would hide the bug.
  • Last registration wins, by class and by name, so a module that a dev server reloads can register its new class object again.
  • A decode function’s errors are classified as a pickle hook’s would be. A Timeout or OSError raises with the row kept (a miss under durable=True). An error about the row, such as a TypeError from a dataclass row written before a field was removed, becomes CorruptValueError with yours as __cause__, and on_decode_error='miss' deletes that row so the writer replaces it. Decode runs while the cache holds a connection, as unpickling does, so a slow decode delays forks and checkpoints for as long as it runs; one that re-enters the cache is fine.

A cache that needs a different encoding of a type than the rest of the process gets its own registry, consulted before the global one:

reg = diskcache_oxide.TypeRegistry()
reg.register(ObjectId, name='bson.ObjectId', encode=lambda o: o.binary, decode=ObjectId)
cache = Cache('/var/cache/my-app-binary-ids', types=reg)

Two processes with different registrations on one directory write rows the other can’t read, the same way two processes with different serializer= settings would. Register at import time, the same way everywhere.

Why pickle is off by default

  • Unpickling runs code. Anyone who can write to the cache directory, including a compromised neighbor process or a shared volume, can make every reader run arbitrary Python.
  • Pickle hides mistakes. Database sessions, open files and other objects that were never meant to be cached get pickled without complaint and fail later in confusing ways.

Opting in to pickle

When you really do need to cache arbitrary objects, turn pickle on for that cache. It’s used only for values the structured format can’t represent; everything else is still stored as msgpack or JSON.

objects = Cache('/var/cache/my-app-objects', pickle=True)
objects.set('service', MyService())

A good habit is to give pickled values their own cache directory, and keep everything else in a cache without pickle.

Reading safely: UnsafeValueError

Several processes can open one directory with different settings. A handle opened without pickle=True never unpickles. If it reads a value that a pickle=True handle stored, it raises diskcache_oxide.UnsafeValueError (a TypeError subclass) and leaves the row in place, so the pickle=True handle can still read it.

A value whose stored bytes can’t be decoded at all raises CorruptValueError (a ValueError subclass).