A Guide For Filesystem Implementers

PyFilesystem objects are designed to be as generic as possible and still expose the full filesystem functionality. With a little care, you can write a wrapper for your filesystem that allows it to work interchangeably with any of the built-in FS classes and tools.

To create a working PyFilesystem interface, derive a class from FS and implement the 9 Essential Methods. The base class uses these essential methods as a starting point for providing a lot of extra functionality, but in some cases the default implementation may not be the most efficient. For example, most filesystems have an atomic way of moving a file from one location to another without having to copy data, whereas the default implementation of move() method must copy all the bytes in the source file to the destination file. Any of the Non - Essential Methods may be overridden, but efficient custom versions of the following methods will have the greatest impact on performance:

For network based filesystems (i.e. where the physical data is pulled over a network), there are a few methods which can reduce the number of round trips to the server, if an efficient implementation is provided:

The generator methods (beginning with i) are intended for use with filesystems that contain a lot of files, where reading the directory in one go may be expensive.

Other methods in the Filesystem Interface are unlikely to require a non-default implementation, but there is nothing preventing you from implementing them – just be careful to use the same signature and replicate expected functionality.

Filesystem Errors

With the exception of the constructor, FS methods should throw FSError exceptions in preference to any implementation-specific exception classes, so that generic exception handling can be written. The constructor may throw a non-FSError exception, if no appropriate FSError exists. The rationale for this is that creating an FS interface may require specific knowledge, but this shouldn’t prevent it from working with more generic code.

If specific exceptions need to be translated in to an equivalent FSError, pass the original exception class to the FSError constructor with the ‘details’ keyword argument.

For example, the following translates some fictitious exception in to an FSError exception, and passes the original exception as an argument.:

try:
    someapi.open(path, mode)
except someapi.UnableToOpen, e:
    raise errors.ResourceNotFoundError(path=path, details=e)

Any code written to catch the generic error, can also retrieve the original exception if it contains additional information.

Thread Safety

All PyFilesystem methods, other than the constructor, should be thread-safe where-ever possible. One way to do this is to pass threads_synchronize=True to the base constructor and use the synchronize() decorator to lock the FS object when a method is called.

If the implementation cannot be made thread-safe for technical reasons, ensure that getmeta("thread_safe") returns False.

Meta Values

The getmeta() method is designed to return implementation specific information. PyFilesystem implementations should return as much of the standard set of meta values as possible.

Implementations are also free to reserve a dotted namespace notation for themselves, to provide an interface to highly specific information. If you do this, please avoid generic terms as they may conflict with existing or future implementations. For example "bobs_ftpfs.author", rather than "ftpfs.author".

If your meta values are static, i.e. they never change, then create a dictionary class attribute called _meta in your implementation that contains all the meta keys and values. The default getmeta implementation will pull the meta values from this dictionary.

Essential Methods

The following methods are required for a minimal Filesystem interface:

  • open() Opens a file for read/writing
  • isfile() Check whether the path exists and is a file
  • isdir() Check whether a path exists and is a directory
  • listdir() List the contents of a directory
  • makedir() Create a new directory
  • remove() Remove an existing file
  • removedir() Remove an existing directory
  • rename() Atomically rename a file or directory
  • getinfo() Return information about the path e.g. size, mtime

Non - Essential Methods

The following methods have default implementations in FS and aren’t required for a functional FS interface. They may be overridden if an alternative implementation can be supplied:

  • copy() Copy a file to a new location
  • copydir() Recursively copy a directory to a new location
  • desc() Return a short descriptive text regarding a path
  • exists() Check whether a path exists as file or directory
  • listdirinfo() Get a directory listing along with the info dict for each entry
  • ilistdir() Generator version of the listdir method
  • ilistdirinfo() Generator version of the listdirinfo method
  • getpathurl() Get an external URL at which the given file can be accessed, if possible
  • getsyspath() Get a file’s name in the local filesystem, if possible
  • getmeta() Get the value of a filesystem meta value, if it exists
  • getmmap() Gets an mmap object for the given resource, if supported
  • hassyspath() Check if a path maps to a system path (recognized by the OS)
  • haspathurl() Check if a path maps to an external URL
  • hasmeta() Check if a filesystem meta value exists
  • move() Move a file to a new location
  • movedir() Recursively move a directory to a new location
  • settimes() Sets the accessed and modified times of a path