Skip to main content
Version: 0.5.4 (latest)

common.storages.fsspec_filesystem

FileItem Objects

class FileItem(TypedDict)

[view_source]

A DataItem representing a file

fsspec_filesystem

def fsspec_filesystem(
protocol: str,
credentials: FileSystemCredentials = None,
kwargs: Optional[DictStrAny] = None,
client_kwargs: Optional[DictStrAny] = None
) -> Tuple[AbstractFileSystem, str]

[view_source]

Instantiates an authenticated fsspec FileSystem for a given protocol and credentials.

Please supply credentials instance corresponding to the protocol. The protocol is just the code name of the filesystem i.e.:

  • s3
  • az, abfs, abfss, adl, azure
  • gcs, gs

also see filesystem_from_config

prepare_fsspec_args

def prepare_fsspec_args(config: FilesystemConfiguration) -> DictStrAny

[view_source]

Prepare arguments for fsspec filesystem constructor.

Arguments:

  • config FilesystemConfiguration - The filesystem configuration.

Returns:

  • DictStrAny - The arguments for the fsspec filesystem constructor.

fsspec_from_config

def fsspec_from_config(
config: FilesystemConfiguration) -> Tuple[AbstractFileSystem, str]

[view_source]

Instantiates an authenticated fsspec FileSystem from config argument.

Authenticates following filesystems:

  • s3
  • az, abfs, abfss, adl, azure
  • gcs, gs

All other filesystems are not authenticated

Returns: (fsspec filesystem, normalized url)

FileItemDict Objects

class FileItemDict(DictStrAny)

[view_source]

A FileItem dictionary with additional methods to get fsspec filesystem, open and read files.

__init__

def __init__(mapping: FileItem,
credentials: Optional[Union[FileSystemCredentials,
AbstractFileSystem]] = None)

[view_source]

Create a dictionary with the filesystem client.

Arguments:

  • mapping FileItem - The file item TypedDict.
  • credentials Optional[FileSystemCredentials], optional - The credentials to the filesystem. Defaults to None.

fsspec

@property
def fsspec() -> AbstractFileSystem

[view_source]

The filesystem client is based on the given credentials.

Returns:

  • AbstractFileSystem - The fsspec client.

local_file_path

@property
def local_file_path() -> str

[view_source]

Gets a valid local filesystem path from file:// scheme. Supports POSIX/Windows/UNC paths

Returns:

  • str - local filesystem path

open

def open(mode: str = "rb",
compression: Literal["auto", "disable", "enable"] = "auto",
**kwargs: Any) -> IO[Any]

[view_source]

Open the file as a fsspec file.

This method opens the file represented by this dictionary as a file-like object using the fsspec library.

Arguments:

  • mode Optional[str] - Open mode.
  • compression Optional[str] - A flag to enable/disable compression. Can have one of three values: "disable" - no compression applied, "enable" - gzip compression applied, "auto" (default) - compression applied only for files compressed with gzip.
  • **kwargs Any - The arguments to pass to the fsspec open function.

Returns:

  • IOBase - The fsspec file.

read_bytes

def read_bytes() -> bytes

[view_source]

Read the file content.

Returns:

  • bytes - The file content.

glob_files

def glob_files(fs_client: AbstractFileSystem,
bucket_url: str,
file_glob: str = "**") -> Iterator[FileItem]

[view_source]

Get the files from the filesystem client.

Arguments:

  • fs_client AbstractFileSystem - The filesystem client.
  • bucket_url str - The url to the bucket.
  • file_glob str - A glob for the filename filter.

Returns:

  • Iterable[FileItem] - The list of files.

This demo works on codespaces. Codespaces is a development environment available for free to anyone with a Github account. You'll be asked to fork the demo repository and from there the README guides you with further steps.
The demo uses the Continue VSCode extension.

Off to codespaces!

DHelp

Ask a question

Welcome to "Codex Central", your next-gen help center, driven by OpenAI's GPT-4 model. It's more than just a forum or a FAQ hub – it's a dynamic knowledge base where coders can find AI-assisted solutions to their pressing problems. With GPT-4's powerful comprehension and predictive abilities, Codex Central provides instantaneous issue resolution, insightful debugging, and personalized guidance. Get your code running smoothly with the unparalleled support at Codex Central - coding help reimagined with AI prowess.