The shelve module

Notice that to create a pickled object the entire object must be in memory; similarly, when you unpickle an object, you must read the entire object into memory. When you're dealing with very large amounts of data, the shelve module may provide a better alternative. (It should be noted that interfacing to a relational database may be a more useful solution in some cases. There are a number of Python modules available to access many different databases; see http://www.python.org/topics/database/modules.html for more information.) The shelve module creates a persistent, file-based version of an object very similar to a Python dictionary, but data is only read from or written to the file when necessary, for example when you need to access a value stored in the file, or add a value to the file. One limitation of shelve objects is that the keys to the objects must be strings, but the values stored in a shelve object can be any Python object, as long as it can be written with the pickle module.

To create a shelve object, use the open function from the shelve module, providing the name of a file to be used to hold the shelve object. The shelve module may call another program which will create more than one file, but the open method will always find your shelved object when you refer to the filename that was used when you first open the shelve object. If the file you specify already exists, but is not a shelve object created in a previous program, you'll get a anydbm.error exception; otherwise the open function will recognize a previously shelved object and open it. Since information is written to the shelved object only when necessary, it's important to invoke the close method on any shelved objects you use to insure that changes that you make during your program are properly stored.

Here's the employee example revisited, using a shelve object. While there's not much advantage in shelving over pickling for such a small data set, it will illustrate the basic ideas of using a shelve object.

>>> import shelve
>>> employees = {
... 'smith':{'firstname':'fred','office':201,'id':'0001','phone':'x232'},
... 'jones':{'firstname':'sue','office':207,'id':'0003','phone':'x225'},
... 'williams':{'firstname':'bill','office':215,'id':'0004',
... 'phone':'x219'}}
>>> try:
...     emp = shelve.open('employees.dat')
... except IOError:
...     print >> sys.stderr, 'Error opening employees.dat'
...     sys.exit(1)
... 
>>> for k in employees:
...     emp[k] = employees[k]
... 
>>> emp.close()

When we need to access the shelved data, we simply open the appropriate file, and the data will be available:

>>> import shelve
>>> employees = shelve.open('employees.dat')
>>> employees['jones']['office']
207
>>> employees['smith']['firstname']
'fred'
>>>