FileAutomation

Automation-first Python library for local file / directory / zip operations, HTTP downloads, and remote storage (Google Drive, S3, Azure Blob, Dropbox, SFTP). Actions are defined as JSON and dispatched through a central registry so they can be executed in-process, from disk, over a TCP socket, or over HTTP.

Architecture

Layered architecture with Facade + Registry + Command + Strategy patterns:

automation_file/
├── __init__.py          # Public API facade (__all__); launch_ui is loaded lazily via __getattr__
├── __main__.py          # CLI entry: subcommands plus the legacy -e/-d/-c/--execute_str flags
├── exceptions.py        # FileAutomationException hierarchy
├── logging_config.py    # file_automation_logger (file + stderr handlers)
├── core/                # Engine: action_registry (ActionRegistry, build_default_registry), action_executor
│                        # (shared `executor`), callback_executor, package_loader, plugins, dag_executor,
│                        # action_queue, json_store, substitution; cross-cutting helpers: retry, quota,
│                        # rate_limit, circuit_breaker, file_lock, sqlite_lock, checksum, manifest, crypto,
│                        # secrets, config, config_watcher, audit, metrics, tracing, progress, fim, content_store
├── local/               # Strategy modules: file/dir/zip/tar/archive ops, sync, diff, text/JSON/data edits,
│                        # templates, versioning, trash, shell_ops (argv-only subprocess), conditional;
│                        # safe_paths.py guards against path traversal
├── remote/              # url_validator (SSRF guard), http_download, cross_backend, fsspec_bridge, and one
│                        # subpackage per backend: google_drive, s3, azure_blob, dropbox_api, sftp, ftp,
│                        # onedrive, box (client.py + *_ops.py + register_<backend>_ops); smb and webdav
│                        # have a client only
├── server/              # tcp_server, http_server, mcp_server (MCP over stdio), web_ui, metrics_server,
│                        # action_acl (ActionACL), network_guards (ensure_loopback)
├── client/              # HTTPActionClient for the HTTP action server
├── trigger/, scheduler/, notify/   # watchdog file triggers, cron scheduler, notification sinks;
│                                   # each registers its own FA_* ops
├── project/             # ProjectBuilder, create_project_dir
├── ui/                  # PySide6 GUI: launcher.launch_ui, main_window.MainWindow, worker.ActionWorker,
│                        # log_widget.LogPanel, tabs/ (home, local, http, JSON editor, servers, scheduler,
│                        # trigger, progress; the cloud backends are panels grouped under transfer_tab)
└── utils/               # file discovery, fast find, grep, duplicate finder, backup rotation

architecture.md §2 carries the same map with one row per directory; keep the two in step.

Key design patterns in use:

• Facade: automation_file/__init__.py re-exports every supported name (execute_action, driver_instance, start_autocontrol_socket_server, …).
• Registry + Command: ActionRegistry maps action name → callable. JSON action lists are command objects ([name, kwargs] / [name, [args]] / [name]) dispatched through the registry.
• Template Method: ActionExecutor._execute_event defines the single-action lifecycle (resolve → call → wrap result); execute_action is the outer iteration template.
• Strategy: Each local/*_ops.py and remote/<backend>/*_ops.py module is an independent strategy that plugs into the registry.
• Singleton (module-level): driver_instance, executor, callback_executor, package_manager are shared instances wired in __init__.py so callback_executor.registry is executor.registry.
• Builder: ProjectBuilder assembles the keyword/ + executor/ skeleton.

Key types

• ActionRegistry — mutable name → callable mapping. register, register_many, resolve, unregister, event_dict (live view for legacy callers).
• ActionExecutor — holds a registry and runs JSON action lists. execute_action(list|dict, validate_first=False, dry_run=False), execute_action_parallel(list, max_workers=None), validate(list) -> list[str], execute_files(paths), add_command_to_executor(mapping).
• CallbackExecutor — runs a registered trigger, then a user callback, sharing the executor’s registry.
• PackageLoader — imports a package by name and registers its top-level functions / classes / builtins as <package>_<member>.
• GoogleDriveClient — wraps OAuth2 credential loading; exposes service lazily. later_init(token_path, credentials_path) bootstraps; require_service() raises if not initialised.
• S3Client / AzureBlobClient / DropboxClient / SFTPClient — singleton wrappers around the required SDKs. Each exposes later_init(...) plus close() where relevant. Their ops are auto-registered by build_default_registry(); register_<backend>_ops(registry) is still exported so callers can populate custom registries.
• MainWindow — PySide6 tabbed control surface (ui/main_window.py). Nine tabs — Local, HTTP, Google Drive, S3, Azure Blob, Dropbox, SFTP, JSON actions, Servers — share a LogPanel and dispatch work through ActionWorker(QRunnable) on the global QThreadPool.
• launch_ui(argv=None) — boots / reuses a QApplication, shows MainWindow, and returns the exec code. Exposed lazily on the facade via __getattr__ so the Qt runtime isn’t paid for by non-UI importers.
• TCPActionServer — threaded TCP server that deserialises a JSON action list per connection. Defaults to loopback; optional shared_secret enforces AUTH <secret>\n prefix.
• HTTPActionServer — ThreadingHTTPServer exposing POST /actions plus GET /healthz, /readyz, /openapi.json and /progress. Defaults to loopback; optional shared_secret enforces Authorization: Bearer <secret>.
• Quota — frozen dataclass capping bytes and wall-clock seconds per action or block (check_size, time_budget context manager, wraps decorator). 0 disables each cap.
• retry_on_transient(max_attempts, backoff_base, backoff_cap, retriable) — decorator that retries with capped exponential back-off and raises RetryExhaustedException chained to the last error.
• safe_join(root, user_path) / is_within(root, path) — path traversal guard; safe_join raises PathTraversalException when the resolved path escapes root.

Branching & CI

• main branch: stable releases, publishes automation_file to PyPI (version in stable.toml).
• dev branch: development, publishes automation_file_dev to PyPI (version in dev.toml).
• Keep dependencies and [project.optional-dependencies] (dev) in sync across both TOMLs. Backends (boto3, azure-storage-blob, dropbox, paramiko) and PySide6 are first-class runtime deps — do not move them back under extras.
• Version bumping is automatic. A dedicated publish workflow bumps the patch in both stable.toml and dev.toml, builds, uploads to PyPI, then commits the bump back to main tagged as vX.Y.Z. Do not hand-bump before merging to main. The next publish run is skipped via a commit-message guard (chore: bump version), so the bump itself never re-triggers publishing.
• CI: GitHub Actions — a lint job on Ubuntu (Python 3.12), then pytest on Windows across Python 3.10 / 3.11 / 3.12 / 3.13 / 3.14. One workflow per branch: .github/workflows/ci-dev.yml, .github/workflows/ci-stable.yml.
• CI steps: lint (ruff check + ruff format –check + mypy) → pytest with coverage → uploads coverage.xml as an artifact.
• Publishing lives in a separate workflow (.github/workflows/publish.yml) that runs on push to main: bumps both TOMLs, copies stable.toml to pyproject.toml, builds the sdist + wheel, twine upload via PYPI_API_TOKEN, then commits + tags + pushes and creates gh release create v<version> --generate-notes.
• pre-commit is configured (.pre-commit-config.yaml): trailing-whitespace, eof-fixer, check-yaml, check-toml, check-added-large-files, ruff, ruff-format, mypy. Install with pre-commit install after cloning.

Development

python -m pip install -r dev_requirements.txt pytest pytest-cov
python -m pip install -e ".[dev]"       # ruff, mypy, pre-commit
python -m pytest tests/ -v --tb=short
ruff check automation_file/ tests/
ruff format --check automation_file/ tests/
mypy automation_file/
python -m automation_file --help

Testing:

• Unit tests live under tests/ (pytest). Fixtures in tests/conftest.py (sample_file, sample_dir).
• Tests cover every module in core/, local/, remote/url_validator, project/, server/, utils/, plus a facade smoke test, retry/quota/safe_paths, HTTP+TCP auth, and optional-backend registration.
• Google Drive / HTTP-download / S3 / Azure / Dropbox / SFTP code paths that require real credentials or network access are not exercised in CI — only their URL-validation, auth, and guard-clause behaviour are.
• Run all tests before submitting changes: python -m pytest tests/ -v.

Conventions

• Python 3.10+ — use X | Y union syntax, not Union[X, Y].
• Use from __future__ import annotations at the top of every module for deferred type evaluation.
• Exception hierarchy: all custom exceptions inherit from FileAutomationException; never raise Exception(...) directly.
• Logging: use file_automation_logger from automation_file.logging_config. Never print() for diagnostics.
• Action-list shape: [name], [name, {kwargs}], or [name, [args]] — nothing else.
• Delete all unused code — no dead imports, commented-out blocks, unreachable branches, or _old_-prefixed names. Git history is the archive.
• Prefer updating the registry over extending the executor class. Plugins register via add_command_to_executor({name: callable}).

Security

All code must follow secure-by-default principles. Review every change against the checklist below.

General rules

• Never use eval(), exec(), or pickle.loads() on untrusted data.
• Never use subprocess.Popen(..., shell=True) — always pass argument lists.
• Never log or display secrets, tokens, passwords, or API keys. OAuth2 tokens handled by GoogleDriveClient are kept on disk only at the caller-supplied token_path.
• Use json.loads() / json.dumps() for serialisation — never pickle.
• Validate all user input at system boundaries (CLI args, URL inputs, TCP payloads).

Network requests (SSRF prevention)

• All outbound HTTP requests to user-specified URLs must validate the target first via automation_file.remote.url_validator.validate_http_url:
  1. Only http:// and https:// schemes — rejects file://, ftp://, data:, gopher://.
  2. Resolve the hostname and reject IPs in private / loopback / link-local / reserved / multicast / unspecified ranges.
• http_download.download_file calls the validator, uses allow_redirects=False, enforces a default 20 MB response cap and 15 s connection timeout, and never downgrades TLS verification.
• Never pass user-supplied URLs directly to urlopen() / requests.* without the validator.

Network requests (TLS)

• All HTTPS requests must use default TLS verification — never set verify=False.
• No bespoke SSH logic in this project; if added, match PyBreeze’s InteractiveHostKeyPolicy pattern.

Subprocess execution

• This library does not spawn subprocesses on the hot path. If you add one, pass argument lists (never shell=True), set an explicit timeout, and never interpolate user input into a command string.

TCP server

• TCPActionServer binds to localhost by default. start_autocontrol_socket_server(host=…) raises ValueError if the resolved address is not loopback unless allow_non_loopback=True is passed explicitly.
• Do not remove the loopback guard to “make it easier to test remotely”. The server dispatches arbitrary registry commands; exposing it to the network is equivalent to exposing a Python REPL.
• The server accepts a single JSON payload per connection (recv(8192)). Do not raise that limit without also adding a length-framed protocol.
• quit_server triggers an orderly shutdown; do not add an administrative bypass that skips the loopback check.
• Optional shared_secret= enforces an AUTH <secret>\n prefix; the comparison uses hmac.compare_digest (constant time). Never log the secret or the raw payload.

HTTP server

• HTTPActionServer / start_http_action_server mirror the TCP server’s posture: loopback-only by default, allow_non_loopback=True required to bind elsewhere, optional shared_secret enforced as Authorization: Bearer <secret> using hmac.compare_digest.
• POST /actions is the only endpoint that runs anything; the GET routes (/healthz, /readyz, /openapi.json, /progress) only report. Request body capped at 1 MB — do not raise without also switching to a streaming parser.
• Responses are JSON. Auth failures return 401; malformed JSON returns 400; unknown paths return 404.

Path traversal

• Any caller resolving a user-supplied path against a trusted root must go through automation_file.local.safe_paths.safe_join (raises PathTraversalException) or the is_within check. Never concatenate + Path.resolve() yourself and skip the containment check — symlinks and .. segments bypass naive string checks.

SFTP host verification

• SFTPClient uses paramiko.RejectPolicy() — unknown hosts are rejected, never auto-added. Callers pass known_hosts= explicitly or rely on ~/.ssh/known_hosts. Do not swap in AutoAddPolicy for convenience.

Reliability (retry / quota)

• retry_on_transient only retries the exception types passed via retriable=(…). Never widen to bare Exception — masks logic bugs as transient failures. Always exhausts to RetryExhaustedException chained with raise ... from err.
• Quota(max_bytes=…, max_seconds=…) — prefer Quota.wraps(...) over inline checks when guarding a whole operation. 0 disables each cap.

Google Drive

• Credentials are stored at the caller-supplied token_path with encoding="utf-8". Never log or print the token contents.
• GoogleDriveClient.require_service() raises rather than silently operating with a None service — do not paper over it by catching RuntimeError at the call site.

File I/O

• Always use pathlib.Path for path manipulation; never string-concatenate paths with user input.
• Use with open(...) as f: for every file operation; close via context manager.
• Always pass encoding="utf-8" when reading or writing text.
• Never follow symlinks from untrusted sources — resolve and re-check the parent.
• JSON writes go through automation_file.core.json_store.write_action_json which holds a module-level lock.

Plugin / package loading

• PackageLoader.add_package_to_executor(package) registers every function / class / builtin of a package under <package>_<member>. Treat it as eval-grade power: never expose it to arbitrary clients (e.g. via the TCP server). If you add a remote plugin-load command, gate it behind an explicit admin flag and authenticated transport.

Secrets and credentials

• Google OAuth tokens live on disk at the user-supplied path; keep the path out of logs.
• API keys / credentials must come from env vars or caller-supplied paths; never hardcode.

Dependency security

• Pin dependencies in requirements.txt / dev_requirements.txt.
• Do not add new dependencies without reviewing their security posture.
• Avoid transitive bloat — prefer stdlib when the alternative is a single-function dependency.

Code quality (SonarQube / Codacy compliance)

All code must satisfy common static-analysis rules. Review every change against the checklist below.

Complexity & size

• Cyclomatic complexity per function: ≤ 15 (hard cap 20). Break large branches into helpers.
• Cognitive complexity per function: ≤ 15. Flatten nested if/for/try chains with early returns.
• Function length: ≤ 75 lines of code (excluding docstring / blank lines). Extract helpers past that.
• Parameter count: ≤ 7 per function/method. Use a dataclass when more are needed.
• Nesting depth: ≤ 4 levels. Refactor with early returns instead of pyramids.
• File length: ≤ 1000 lines.

Exception handling

• Never use bare except: — always specify exception types.
• Avoid catching Exception / BaseException unless immediately logging and re-raising, or running at a top-level dispatcher boundary (the ActionExecutor.execute_action loop is one of these — it intentionally records per-action failures without aborting the batch).
• Never pass silently inside except — log via file_automation_logger at minimum.
• Do not return / break / continue inside a finally block — it swallows exceptions.
• Custom exceptions must inherit from FileAutomationException.
• Use raise ... from err (or raise ... from None) when re-raising to preserve / suppress the chain explicitly.

Pythonic correctness

• Compare with None using is / is not, never == / !=.
• Type checks use isinstance(obj, T), never type(obj) == T.
• Never use mutable default arguments — use None and initialise inside.
• Prefer f-strings over % formatting or str.format() (except inside lazy log calls: logger.info("x=%s", x)).
• Use context managers for every file / socket / lock.
• Use enumerate() instead of range(len(...)) when the index is needed.
• Use dict.get(key, default) over key in dict and dict[key].

Naming & style (PEP 8)

• snake_case for functions, methods, variables, module names.
• PascalCase for classes.
• UPPER_SNAKE_CASE for module-level constants.
• _leading_underscore for protected / internal members.
• Do not shadow built-ins (id, type, list, dict, input, file, open, etc.).

Duplication & dead code

• String literal used 3+ times in the same module → extract a module-level constant.
• Identical 6+ line blocks in 2+ places → extract a helper.
• Remove unused imports, unused parameters, unused local variables, unreachable code after return / raise.
• No commented-out code blocks — delete them.
• No TODO / FIXME / XXX without an issue reference (# TODO(#123): …).

Logging, printing, assertions

• Never use print() for diagnostics in library code — use file_automation_logger.
• Use lazy logging (logger.debug("x=%s", x)) to avoid eager f-string formatting on hot paths.
• Never use assert for runtime validation; assert is for tests only.

Hardcoded values & secrets

• No hardcoded passwords, tokens, API keys, or secrets.
• No hardcoded IPs / hostnames outside of documented localhost / loopback defaults.
• Magic numbers (except 0, 1, -1) should be named constants when repeated or non-obvious.

Boolean & return hygiene

• return bool(cond) or return cond, not if cond: return True else: return False.
• if x / if not x, not if x == True / if x == False.
• A function should have a consistent return type.

Imports

• One import per line; grouped from x import a, b is fine.
• Order: stdlib → third-party → first-party (automation_file.*) — separated by blank lines.
• No wildcard imports outside __init__.py re-exports.
• Max one level of relative import.

Running the linter

• Before committing any non-trivial change, run ruff check automation_file/ tests/ locally.
• When adding a # noqa: RULE, justify it in the comment — never blanket-disable.

Documentation

• README parity. This repository ships README.md (English) alongside the translated README.zh-CN.md and README.zh-TW.md. All three must stay current with the code.
• When a change alters anything user-facing — features, commands, CLI flags, install/setup steps, configuration, or requirements — update README.md, both translated READMEs (README.zh-CN.md and README.zh-TW.md), and the affected docs/ sources, all in the same commit, keeping their section structure and content aligned across languages. The translations must reflect the English content, not merely match its headings.
• Never update one language, or README.md alone, and leave the other READMEs or the docs stale. There is no automated README-parity guard, so verify this by hand before committing.

Stage commits, progress.md, docs/updates/ and architecture.md

Workspace rule shared by every repository under D:\Codes (full text: D:\Codes\CLAUDE.md).

• Commit at every stage. A stage is the smallest piece of work that leaves the repository consistent and passes this project’s checks (definition of done, tests, lint): one finished progress.md item, or one self-contained step of a larger one. Commit it before starting the next stage, before switching to another repository, and before the session ends. Do not leave work uncommitted across sessions; if a stage cannot be finished, commit the consistent part and record the rest in progress.md.
  • Stage only the files that stage touched (git add <path>, never git add -A), follow this file’s commit-message rules, and never add AI attribution.
  • Committing is not pushing: push or open a PR only as this project’s branch flow says or when asked.
  • Commit and push frequently. After each big feature — a self-contained stage that passes this project’s checks — commit and push to the remote; do not pile up a large batch of work before committing or pushing. Smaller batches collide less with other sessions, let CI catch problems earlier, and are easier to revert. Follow this project’s normal branch flow (usually dev).
• progress.md (repository root, tracked) holds outstanding work only: no finished items, no history, no rules.
• docs/updates/ records finished work: one batch file per month (YYYY-MM.md), one entry per piece of work headed ## U-YYYYMMDD-NN · date · title · #tags, and an index with query commands in docs/updates/README.md. When a progress.md item is done, delete it and add a #done entry plus its index row in the same commit.
• architecture.md (repository root) is the short architecture overview: layers, entry points, main flows, extension points, cross-project boundaries. Update it in the same commit whenever a change alters any of those.
• Cross-project contracts are listed in architecture.md §6: what other repositories rely on here (CLI flags, import paths, constructor arguments, file layouts) and what this repository relies on elsewhere. No test here protects them, so never rename or remove one without changing its consumers in the same round, and update §6 whenever a contract is added or changes.

Commit & PR rules

• Commit messages: short imperative sentence (e.g., “Fix rename_file overwrite bug”, “Update stable version”).
• Do not mention any AI tools, assistants, or co-authors in commit messages or PR descriptions.
• Do not add Co-Authored-By headers referencing any AI.
• PR target: dev for development work, main for stable releases.

This site is open source. Improve this page.