fs.s3fs

Currently only avaiable on Python2 due to boto not being available for Python3

FS subclass accessing files in Amazon S3

This module provides the class ‘S3FS’, which implements the FS filesystem interface for objects stored in Amazon Simple Storage Service (S3).

class fs.s3fs.S3FS(bucket, prefix='', aws_access_key=None, aws_secret_key=None, separator='/', thread_synchronize=True, key_sync_timeout=1, **conn_kwargs)

A filesystem stored in Amazon S3.

This class provides the FS interface for files stored in Amazon’s Simple Storage Service (S3). It should be instantiated with the name of the S3 bucket to use, and optionally a prefix under which the files should be stored.

Local temporary files are used when opening files from this filesystem, and any changes are only pushed back into S3 when the files are closed or flushed.

Constructor for S3FS objects.

S3FS objects require the name of the S3 bucket in which to store files, and can optionally be given a prefix under which the files should be stored. AWS connection arguments (e.g. aws_access_key_id, aws_secret_access_key, etc) may be specified as additional keyword arguments to be passed to boto.s3.connection.S3Connection; if they are not specified boto will try to read them from various locations ( see http://boto.cloudhackers.com/en/latest/boto_config_tut.html). Note that legacy arguments ‘aws_access_key’ and ‘aws_secret_key’ will be passed to boto as ‘aws_access_key_id’ and ‘aws_secret_access_key’.

The keyword argument ‘key_sync_timeout’ specifies the maximum time in seconds that the filesystem will spend trying to confirm that a newly-uploaded S3 key is available for reading. For no timeout set it to zero. To disable these checks entirely (and thus reduce the filesystem’s consistency guarantees to those of S3’s “eventual consistency” model) set it to None.

By default the path separator is “/”, but this can be overridden by specifying the keyword ‘separator’ in the constructor.

copy(src, dst, overwrite=False, chunk_size=16384)

Copy a file from ‘src’ to ‘dst’.

src – The source path dst – The destination path overwrite – If True, then the destination may be overwritten (if a file exists at that location). If False then an exception will be thrown if the destination exists chunk_size – Size of chunks to use in copy (ignored by S3)

exists(path)

Check whether a path exists.

getpathurl(path, allow_none=False, expires=3600)

Returns a url that corresponds to the given path.

ilistdir(path='./', wildcard=None, full=False, absolute=False, dirs_only=False, files_only=False)

List contents of a directory.

isdir(path)

Check whether a path exists and is a directory.

isfile(path)

Check whether a path exists and is a regular file.

listdir(path='./', wildcard=None, full=False, absolute=False, dirs_only=False, files_only=False)

List contents of a directory.

makedir(path, recursive=False, allow_recreate=False)

Create a directory at the given path.

The ‘mode’ argument is accepted for compatibility with the standard FS interface, but is currently ignored.

makepublic(path)

Mark given path as publicly accessible using HTTP(S)

move(src, dst, overwrite=False, chunk_size=16384)

Move a file from one location to another.

open(path, mode=u'rt', buffering=-1, encoding=None, errors=None, newline=None, line_buffering=False, **kwargs)

Open the named file in the given mode.

This method downloads the file contents into a local temporary file so that it can be worked on efficiently. Any changes made to the file are only sent back to S3 when the file is flushed or closed.

remove(path)

Remove the file at the given path.

removedir(path, recursive=False, force=False)

Remove the directory at the given path.

rename(src, dst)

Rename the file at ‘src’ to ‘dst’.