PickleCache

PickleCache.py

PickleCache provides tools for keeping fast-loading cached versions of files so that subsequent loads are faster. This is similar to how Python silently caches .pyc files next to .py files.

The typical scenario is that you have a type of text file that gets “translated” to Pythonic data (dictionaries, tuples, instances, ints, etc.). By caching the Python data on disk in pickle format, you can avoid the expensive translation on subsequent reads of the file.

Two real life cases are MiscUtils.DataTable, which loads and represents comma-separated files, and the separate MiddleKit plug-in which has an object model file. So for examples on using this module, load up the following files and search for “Pickle”:

MiscUtils/DataTable.py
MiddleKit/Core/Model.py

The cached file is named the same as the original file with ‘.pickle.cache’ suffixed. The utility of ‘.pickle’ is to denote the file format and the utility of ‘.cache’ is to provide *.cache as a simple pattern that can be removed, ignored by backup scripts, etc.

The treatment of the cached file is silent and friendly just like Python’s approach to .pyc files. If it cannot be read or written for various reasons (cache is out of date, permissions are bad, wrong python version, etc.), then it will be silently ignored.

GRANULARITY

In constructing the test suite, I discovered that if the source file is newly written less than 1 second after the cached file, then the fact that the source file is newer will not be detected and the cache will still be used. I believe this is a limitation of the granularity of os.path.getmtime(). If anyone knows of a more granular solution, please let me know.

This would only be a problem in programmatic situations where the source file was rapidly being written and read. I think that’s fairly rare.

SEE ALSO

https://docs.python.org/3/library/pickle.html

class MiscUtils.PickleCache.PickleCache

Bases: object

Abstract base class for PickleCacheReader and PickleCacheWriter.

picklePath(filename)

class MiscUtils.PickleCache.PickleCacheReader

Bases: PickleCache

picklePath(filename)

read(filename, pickleProtocol=None, source=None, verbose=None)

Read data from pickle cache.

Returns the data from the pickle cache version of the filename, if it can read. Otherwise returns None, which also indicates that writePickleCache() should be subsequently called after the original file is read.

class MiscUtils.PickleCache.PickleCacheWriter

Bases: PickleCache

picklePath(filename)

write(data, filename, pickleProtocol=None, source=None, verbose=None)

MiscUtils.PickleCache.readPickleCache(filename, pickleProtocol=None, source=None, verbose=None)

Read data from pickle cache.

Returns the data from the pickle cache version of the filename, if it can read. Otherwise returns None, which also indicates that writePickleCache() should be subsequently called after the original file is read.

MiscUtils.PickleCache.writePickleCache(data, filename, pickleProtocol=None, source=None, verbose=None)