Removes tools from grouping.

This commit is contained in:
2026-02-22 21:31:21 -08:00
parent d43c49d219
commit a5e9835d55
77 changed files with 260 additions and 2497 deletions

View File

@@ -33,13 +33,9 @@ uv run my-service # starts on auto-assigned port
castle service enable my-service # register with systemd
castle gateway reload # update reverse proxy routes
# Standalone tool
# Tool
castle create my-tool --type tool --description "Does something"
cd my-tool && uv sync
# Category tool (adds to existing tools/<category>/ package)
castle create my-tool --type tool --category document --description "Does something"
cd tools/document && uv sync
```
The `castle create` command scaffolds the project, generates a CLAUDE.md,
@@ -59,8 +55,8 @@ castle lint [project] # Run linter (one or all)
castle sync # Update submodules + uv sync all
castle run <component> # Run component in foreground
castle logs <component> [-f] [-n 50] # View component logs
castle tool list # List tools grouped by category
castle tool info <name> # Show tool details + docs
castle tool list # List all tools
castle tool info <name> # Show tool details
castle gateway start|stop|reload|status # Manage Caddy reverse proxy
castle service enable|disable <name> # Manage individual systemd service
castle service status # Show all service statuses

13
TODO.md Normal file
View File

@@ -0,0 +1,13 @@
# TO DO
- Remove devbox-connect. Instead, make it easy to copy an ssh tunnel command to expose all ports of all services to a remote box: `ssh -L 9000:localhost:9000 payne@dev.payne.io, etc.
- tool.uv.sources should be in cli, not in castle-api
- Add a scripts dir?
- Maybe there's no real reason to have special handling for the `tools` dir (one md file per tool, put in categories, etc.). On the one hand, the categories help them share dependencies. On the other, there's not really a need to share dependencies because uv does just fine. Also, they prob don't need markdown files because their description can just be in the --help arg, and in the `castle.yaml` registration. Having them flat would allow us to just think of everything as a tool: a bash script tool, a python tool, a rust tool. They're all things just follow a std-in std-out pattern so they are unix-philosophy good. Daemons otoh, follow daemon patterns (env config, logging, long-running, port mapping, etc.)
- tools: use std-in/std-out unix philosophy
- daemons or web-services: expose ports
- workers: just keep doing work in a background--usually watching the filesystem or a queue
- jobs: scheduled tools or daemon requests
- frontend: files served up by caddy

View File

@@ -1,17 +1,16 @@
[project]
name = "castle-system"
name = "android-backup"
version = "0.1.0"
description = "Castle system administration tools"
description = "Backup Android device using ADB"
requires-python = ">=3.11"
dependencies = []
[project.scripts]
backup-collect = "system.backup_collect:main"
schedule = "system.schedule:main"
android-backup = "android_backup.main:main"
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[tool.hatch.build.targets.wheel]
packages = ["src/system"]
packages = ["src/android_backup"]

View File

@@ -0,0 +1,16 @@
[project]
name = "backup-collect"
version = "0.1.0"
description = "Collect files from various sources into backup directory"
requires-python = ">=3.11"
dependencies = []
[project.scripts]
backup-collect = "backup_collect.main:main"
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[tool.hatch.build.targets.wheel]
packages = ["src/backup_collect"]

View File

@@ -1,7 +1,7 @@
[project]
name = "castle-browser"
name = "browser"
version = "0.1.0"
description = "Castle browser automation tools"
description = "Browser automation tool powered by browser-use"
requires-python = ">=3.11"
dependencies = [
"python-dotenv>=1.0.0",
@@ -11,7 +11,7 @@ dependencies = [
]
[project.scripts]
browser = "browser.browser:main"
browser = "browser.main:main"
[build-system]
requires = ["hatchling"]

View File

@@ -79,13 +79,6 @@ class ToolSummary(BaseModel):
installed: bool = False
class ToolCategory(BaseModel):
"""Tools grouped by category."""
name: str
tools: list[ToolSummary]
class ToolDetail(ToolSummary):
"""Full detail for a single tool, including documentation."""

View File

@@ -11,7 +11,7 @@ from castle_cli.config import load_config
from castle_cli.manifest import ComponentManifest
from castle_api.config import settings
from castle_api.models import ToolCategory, ToolDetail, ToolSummary
from castle_api.models import ToolDetail, ToolSummary
router = APIRouter(tags=["tools"])
@@ -42,55 +42,16 @@ def _tool_summary(name: str, manifest: ComponentManifest, root: Path | None = No
)
def _find_md_for_tool(
root: Path,
source: str,
tool_name: str,
) -> Path | None:
"""Find the .md documentation file for a tool source path."""
source_path = root / source
if source_path.is_file():
md = source_path.with_suffix(".md")
if md.exists():
return md
elif source_path.is_dir():
py_name = tool_name.replace("-", "_")
pkg_name = source_path.name
md = source_path / "src" / pkg_name / f"{py_name}.md"
if md.exists():
return md
return None
def _strip_frontmatter(content: str) -> str:
"""Strip YAML frontmatter from markdown content."""
if content.startswith("---\n"):
end = content.find("\n---\n", 4)
if end != -1:
content = content[end + 5:]
return content.strip()
@router.get("/tools", response_model=list[ToolCategory])
def list_tools() -> list[ToolCategory]:
"""List tools grouped by source directory."""
@router.get("/tools", response_model=list[ToolSummary])
def list_tools() -> list[ToolSummary]:
"""List all registered tools."""
config = load_config(settings.castle_root)
tools = {k: v for k, v in config.components.items() if v.tool}
by_group: dict[str, list[ToolSummary]] = {}
for name, manifest in tools.items():
t = manifest.tool
assert t is not None
if t.source:
group = Path(t.source).name
else:
group = "standalone"
by_group.setdefault(group, []).append(_tool_summary(name, manifest, config.root))
return [
ToolCategory(name=group, tools=sorted(items, key=lambda t: t.id))
for group, items in sorted(by_group.items())
]
return sorted(
[_tool_summary(name, manifest, config.root) for name, manifest in tools.items()],
key=lambda t: t.id,
)
@router.get("/tools/{name}", response_model=ToolDetail)
@@ -112,16 +73,7 @@ def get_tool(name: str) -> ToolDetail:
)
summary = _tool_summary(name, manifest, config.root)
docs: str | None = None
t = manifest.tool
if t.source:
md_path = _find_md_for_tool(config.root, t.source, name)
if md_path and md_path.exists():
docs = _strip_frontmatter(md_path.read_text())
if not docs:
docs = None
return ToolDetail(**summary.model_dump(), docs=docs)
return ToolDetail(**summary.model_dump())
@router.post("/tools/{name}/install")

View File

@@ -38,14 +38,14 @@ def castle_root(tmp_path: Path) -> Generator[Path, None, None]:
"description": "Test tool",
"install": {"path": {"alias": "test-tool"}},
"tool": {
"source": "tools/document",
"source": "test-tool",
"system_dependencies": ["pandoc"],
},
},
"test-tool-2": {
"description": "Another test tool",
"tool": {
"source": "tools/utility",
"source": "test-tool-2",
"version": "2.0.0",
},
},

View File

@@ -8,56 +8,51 @@ from fastapi.testclient import TestClient
class TestToolsList:
"""GET /tools endpoint tests."""
def test_returns_grouped_tools(self, client: TestClient) -> None:
"""Returns tools grouped by source directory name."""
def test_returns_flat_list(self, client: TestClient) -> None:
"""Returns tools as a flat sorted list."""
response = client.get("/tools")
assert response.status_code == 200
data = response.json()
names = [cat["name"] for cat in data]
assert "document" in names
assert "utility" in names
assert isinstance(data, list)
ids = [t["id"] for t in data]
assert "test-tool" in ids
assert "test-tool-2" in ids
def test_groups_sorted(self, client: TestClient) -> None:
"""Groups are sorted alphabetically."""
def test_sorted_alphabetically(self, client: TestClient) -> None:
"""Tools are sorted alphabetically by id."""
response = client.get("/tools")
data = response.json()
names = [cat["name"] for cat in data]
assert names == sorted(names)
ids = [t["id"] for t in data]
assert ids == sorted(ids)
def test_tool_fields(self, client: TestClient) -> None:
"""Tool summary has expected fields."""
response = client.get("/tools")
data = response.json()
doc_group = next(c for c in data if c["name"] == "document")
tool = next(t for t in doc_group["tools"] if t["id"] == "test-tool")
tool = next(t for t in data if t["id"] == "test-tool")
assert tool["description"] == "Test tool"
assert tool["source"] == "tools/document"
assert tool["source"] == "test-tool"
assert tool["system_dependencies"] == ["pandoc"]
# tool_type and category should not be present
assert "tool_type" not in tool
assert "category" not in tool
def test_installed_flag(self, client: TestClient) -> None:
"""Tool with install.path is marked as installed."""
response = client.get("/tools")
data = response.json()
doc_group = next(c for c in data if c["name"] == "document")
tool = next(t for t in doc_group["tools"] if t["id"] == "test-tool")
tool = next(t for t in data if t["id"] == "test-tool")
assert tool["installed"] is True
def test_not_installed_flag(self, client: TestClient) -> None:
"""Tool without install.path is not marked as installed."""
response = client.get("/tools")
data = response.json()
util_group = next(c for c in data if c["name"] == "utility")
tool = next(t for t in util_group["tools"] if t["id"] == "test-tool-2")
tool = next(t for t in data if t["id"] == "test-tool-2")
assert tool["installed"] is False
def test_service_excluded(self, client: TestClient) -> None:
"""Services without tool spec are not listed."""
response = client.get("/tools")
data = response.json()
all_ids = [t["id"] for cat in data for t in cat["tools"]]
all_ids = [t["id"] for t in data]
assert "test-svc" not in all_ids
@@ -70,27 +65,11 @@ class TestToolDetail:
assert response.status_code == 200
data = response.json()
assert data["id"] == "test-tool"
assert data["source"] == "tools/document"
assert data["source"] == "test-tool"
assert data["system_dependencies"] == ["pandoc"]
def test_docs_from_file(self, client: TestClient, castle_root: Path) -> None:
"""Reads documentation from .md file."""
# Create doc file matching the source lookup (src/<dir_name>/<tool>.md)
doc_dir = castle_root / "tools" / "document" / "src" / "document"
doc_dir.mkdir(parents=True)
(doc_dir / "test_tool.md").write_text(
"---\ntitle: Test\n---\n\n# Test Tool\n\nUsage info here."
)
response = client.get("/tools/test-tool")
assert response.status_code == 200
data = response.json()
assert data["docs"] is not None
assert "# Test Tool" in data["docs"]
# Frontmatter should be stripped
assert "---" not in data["docs"]
def test_no_docs(self, client: TestClient) -> None:
"""Tool with no doc file returns null docs."""
"""Tool detail returns null docs (no .md files anymore)."""
response = client.get("/tools/test-tool")
assert response.status_code == 200
data = response.json()

View File

@@ -120,7 +120,7 @@ components:
manage:
systemd: {}
tool:
source: tools/system/
source: backup-collect/
system_dependencies:
- rsync
backup-data:
@@ -162,14 +162,14 @@ components:
path:
alias: mbox2eml
tool:
source: tools/document/
source: mbox2eml/
android-backup:
description: Backup Android device using ADB
install:
path:
alias: android-backup
tool:
source: tools/android/
source: android-backup/
system_dependencies:
- adb
browser:
@@ -178,14 +178,14 @@ components:
path:
alias: browser
tool:
source: tools/browser/
source: browser/
docx-extractor:
description: Extract content and metadata from Word .docx files
install:
path:
alias: docx-extractor
tool:
source: tools/search/
source: docx-extractor/
system_dependencies:
- pandoc
docx2md:
@@ -194,7 +194,7 @@ components:
path:
alias: docx2md
tool:
source: tools/document/
source: docx2md/
system_dependencies:
- pandoc
gpt:
@@ -203,21 +203,21 @@ components:
path:
alias: gpt
tool:
source: tools/gpt/
source: gpt/
html2text:
description: Convert HTML content to plain text
install:
path:
alias: html2text
tool:
source: tools/document/
source: html2text/
md2pdf:
description: Convert Markdown files to PDF
install:
path:
alias: md2pdf
tool:
source: tools/document/
source: md2pdf/
system_dependencies:
- pandoc
- texlive-latex-base
@@ -227,21 +227,21 @@ components:
path:
alias: mdscraper
tool:
source: tools/mdscraper/
source: mdscraper/
pdf-extractor:
description: Extract content and metadata from PDF files
install:
path:
alias: pdf-extractor
tool:
source: tools/search/
source: pdf-extractor/
pdf2md:
description: Convert PDF files to Markdown
install:
path:
alias: pdf2md
tool:
source: tools/document/
source: pdf2md/
system_dependencies:
- pandoc
- poppler-utils
@@ -251,18 +251,18 @@ components:
path:
alias: schedule
tool:
source: tools/system/
source: schedule/
search:
description: Manage self-contained searchable collections of files
install:
path:
alias: search
tool:
source: tools/search/
source: search/
text-extractor:
description: Extract content and metadata from text files
install:
path:
alias: text-extractor
tool:
source: tools/search/
source: text-extractor/

View File

@@ -19,7 +19,7 @@ from castle_cli.manifest import (
SystemdSpec,
ToolSpec,
)
from castle_cli.templates.scaffold import scaffold_category_tool, scaffold_project
from castle_cli.templates.scaffold import scaffold_project
def next_available_port(config: object) -> int:
@@ -42,16 +42,11 @@ def run_create(args: argparse.Namespace) -> int:
config = load_config()
name = args.name
proj_type = args.type
category = getattr(args, "category", None)
if name in config.components:
print(f"Error: component '{name}' already exists in castle.yaml")
return 1
# Category tool: add to existing category package
if proj_type == "tool" and category:
return _create_category_tool(config, name, args.description, category)
project_dir = config.root / name
if project_dir.exists():
print(f"Error: directory '{name}' already exists")
@@ -128,59 +123,3 @@ def run_create(args: argparse.Namespace) -> int:
return 0
def _create_category_tool(config: object, name: str, description: str | None, category: str) -> int:
"""Create a tool inside an existing category package."""
category_dir = config.root / "tools" / category
if not category_dir.exists():
print(f"Error: category directory 'tools/{category}/' does not exist")
print(" Create the category package first, then add tools to it.")
return 1
package_name = name.replace("-", "_")
desc = description or "A castle tool"
# Scaffold the .py and .md files into the category package
scaffold_category_tool(
category_dir=category_dir,
tool_name=name,
package_name=package_name,
category=category,
description=desc,
)
# Append entry point to existing pyproject.toml
pyproject_path = category_dir / "pyproject.toml"
if pyproject_path.exists():
content = pyproject_path.read_text()
entry_line = f'{name} = "{category}.{package_name}:main"'
if entry_line not in content:
# Find [project.scripts] section and append
if "[project.scripts]" in content:
content = content.replace(
"[project.scripts]",
f"[project.scripts]\n{entry_line}",
)
pyproject_path.write_text(content)
print(f" Added entry point to tools/{category}/pyproject.toml")
else:
print(" Warning: could not find [project.scripts] in pyproject.toml")
# Register in castle.yaml
manifest = ComponentManifest(
id=name,
description=desc,
tool=ToolSpec(source=f"tools/{category}/"),
install=InstallSpec(path=PathInstallSpec(alias=name)),
)
config.components[name] = manifest
save_config(config)
print(f"Created tool '{name}' in tools/{category}/")
print(f" Source: tools/{category}/src/{category}/{package_name}.py")
print(" Registered in castle.yaml")
print("\nNext steps:")
print(f" Edit tools/{category}/src/{category}/{package_name}.py")
print(f" cd tools/{category} && uv sync")
print(f" castle tool info {name}")
return 0

View File

@@ -3,7 +3,6 @@
from __future__ import annotations
import argparse
from pathlib import Path
from castle_cli.config import load_config
@@ -28,7 +27,7 @@ def run_tool(args: argparse.Namespace) -> int:
def _tool_list() -> int:
"""List tools grouped by category."""
"""List all registered tools."""
config = load_config()
tools = {k: v for k, v in config.components.items() if v.tool}
@@ -36,24 +35,14 @@ def _tool_list() -> int:
print("No tools registered.")
return 0
by_group: dict[str, list[tuple[str, object]]] = {}
for name, manifest in tools.items():
if manifest.tool and manifest.tool.source:
group = Path(manifest.tool.source).name
else:
group = "standalone"
by_group.setdefault(group, []).append((name, manifest))
for group in sorted(by_group):
items = by_group[group]
print(f"\n{BOLD}{CYAN}{group}{RESET}")
print(f"{CYAN}{'' * 40}{RESET}")
for name, manifest in sorted(items):
desc = manifest.description or ""
deps = ""
if manifest.tool.system_dependencies:
deps = f" {DIM}[{', '.join(manifest.tool.system_dependencies)}]{RESET}"
print(f" {BOLD}{name:<20}{RESET} {desc}{deps}")
print(f"\n{BOLD}{CYAN}Tools{RESET}")
print(f"{CYAN}{'' * 40}{RESET}")
for name, manifest in sorted(tools.items()):
desc = manifest.description or ""
deps = ""
if manifest.tool and manifest.tool.system_dependencies:
deps = f" {DIM}[{', '.join(manifest.tool.system_dependencies)}]{RESET}"
print(f" {BOLD}{name:<20}{RESET} {desc}{deps}")
print()
return 0
@@ -83,39 +72,5 @@ def _tool_info(name: str) -> int:
if t.system_dependencies:
print(f" {BOLD}requires{RESET}: {', '.join(t.system_dependencies)}")
# Read and display .md documentation if available
if t.source:
md_path = _find_md_for_tool(config.root, t.source, name)
if md_path and md_path.exists():
content = md_path.read_text()
# Strip YAML frontmatter
if content.startswith("---\n"):
end = content.find("\n---\n", 4)
if end != -1:
content = content[end + 5:]
content = content.strip()
if content:
print(f"\n{BOLD}{CYAN}Documentation{RESET}")
print(f"{CYAN}{'' * 40}{RESET}")
print(f"{DIM}{content}{RESET}")
print()
return 0
def _find_md_for_tool(
root: Path, source: str, tool_name: str,
) -> Path | None:
"""Find the .md documentation file for a tool source path."""
source_path = root / source
if source_path.is_file():
md = source_path.with_suffix(".md")
if md.exists():
return md
elif source_path.is_dir():
py_name = tool_name.replace("-", "_")
pkg_name = source_path.name
md = source_path / "src" / pkg_name / f"{py_name}.md"
if md.exists():
return md
return None

View File

@@ -46,11 +46,6 @@ def build_parser() -> argparse.ArgumentParser:
create_parser.add_argument(
"--port", type=int, help="Port number (services only)"
)
create_parser.add_argument(
"--category", default=None,
help="Tool category (e.g. document, search, system)",
)
# castle info
info_parser = subparsers.add_parser("info", help="Show component details")
info_parser.add_argument("project", help="Component name")

View File

@@ -5,99 +5,6 @@ from __future__ import annotations
from pathlib import Path
def scaffold_category_tool(
category_dir: Path,
tool_name: str,
package_name: str,
category: str,
description: str,
) -> None:
"""Scaffold a tool inside an existing category package.
Creates the .py and .md files only — the category package
(pyproject.toml, tests, CLAUDE.md) already exists.
"""
src_dir = category_dir / "src" / category
# tool .py file
_write(
src_dir / f"{package_name}.py",
f'''#!/usr/bin/env python3
"""{tool_name}: {description}
Usage:
{tool_name} [options] [input]
cat input.txt | {tool_name}
Examples:
{tool_name} input.txt
{tool_name} input.txt -o output.txt
cat input.txt | {tool_name} > output.txt
"""
import argparse
import sys
__all__ = ["main"]
def main() -> int:
"""Main entry point."""
parser = argparse.ArgumentParser(
description="{description}",
formatter_class=argparse.RawDescriptionHelpFormatter,
)
parser.add_argument("input", nargs="?", help="Input file (default: stdin)")
parser.add_argument(
"-o", "--output", default=None, help="Output file (default: stdout)"
)
args = parser.parse_args()
if args.input:
with open(args.input) as f:
data = f.read()
else:
data = sys.stdin.read()
# TODO: implement tool logic
result = data
if args.output:
with open(args.output, "w") as f:
f.write(result)
else:
print(result, end="")
return 0
if __name__ == "__main__":
sys.exit(main())
''',
)
# tool .md documentation
_write(
src_dir / f"{package_name}.md",
f"""---
name: {tool_name}
category: {category}
---
# {tool_name}
{description}
## Usage
```bash
{tool_name} [options] [input]
cat input.txt | {tool_name}
```
""",
)
def scaffold_project(
project_dir: Path,
name: str,

View File

@@ -95,7 +95,7 @@ class TestRoleDerivation:
"""Component with tool spec gets TOOL role."""
m = ComponentManifest(
id="docx2md",
tool=ToolSpec(source="tools/document/"),
tool=ToolSpec(source="docx2md/"),
)
assert Role.TOOL in m.roles

View File

@@ -132,7 +132,7 @@ Creates a shim so the tool is available system-wide after
```yaml
tool:
source: tools/document/ # Source directory
source: my-tool/ # Source directory
version: "1.0.0"
system_dependencies: [pandoc, poppler-utils]
```
@@ -143,8 +143,7 @@ It's separate from `install` (which handles PATH registration) and `run`
The install method (uv tool install vs symlink) is inferred from the source
directory: if `pyproject.toml` exists, it's a Python package; if the source
is a file, it's symlinked. Grouping for `castle tool list` and the dashboard
is derived from the source directory name (e.g., `tools/document/` → "document").
is a file, it's symlinked.
### `build` — How to build it
@@ -210,11 +209,8 @@ A component can have multiple roles. For example, `protonmail` is both a
# Service — scaffolds project, assigns port, registers in castle.yaml
castle create my-service --type service --description "Does something"
# Standalone tool — scaffolds at repo root
# Tool — scaffolds at repo root
castle create my-tool --type tool --description "Does something"
# Category tool — adds to existing tools/<category>/ package
castle create my-tool --type tool --category document --description "Does something"
```
### Manually

View File

@@ -22,11 +22,9 @@ How to build CLI tools following Unix philosophy.
| **Type checking** | pyright (shared `pyrightconfig.json` at repo root) |
| **Python** | 3.11+ minimum |
## Two kinds of tools
## Project layout
### Standalone tools
Independent projects at the repo root with their own `pyproject.toml`:
Each tool is an independent project at the repo root with its own `pyproject.toml`:
```
my-tool/
@@ -39,31 +37,10 @@ my-tool/
└── CLAUDE.md
```
Examples: `devbox-connect/`, `mboxer/`, `protonmail/`
### Category tools
Multiple tools sharing a single package under `tools/<category>/`:
```
tools/document/
├── src/document/
│ ├── __init__.py
│ ├── pdf2md.py # Each tool is a module
│ ├── docx2md.py
│ ├── html2text.py
│ └── md2pdf.py
├── tests/
├── pyproject.toml # One pyproject with multiple [project.scripts]
└── CLAUDE.md
```
Examples: `tools/document/`, `tools/search/`, `tools/system/`
Examples: `pdf2md/`, `gpt/`, `protonmail/`
## Creating a new tool
### Standalone
```bash
castle create my-tool --type tool --description "Does something"
cd my-tool && uv sync
@@ -71,20 +48,8 @@ cd my-tool && uv sync
This scaffolds the project and registers it in `castle.yaml`.
### Category tool
```bash
castle create my-tool --type tool --category document --description "Does something"
cd tools/document && uv sync
```
This adds a `.py` file to the existing category package and updates its
`pyproject.toml` entry points.
## pyproject.toml
### Standalone tool
```toml
[project]
name = "my-tool"
@@ -110,27 +75,7 @@ dev = ["pytest>=7.0.0"]
known-first-party = ["my_tool"]
```
### Category package
```toml
[project]
name = "castle-document"
version = "0.1.0"
description = "Castle document conversion tools"
requires-python = ">=3.11"
dependencies = []
[project.scripts]
docx2md = "document.docx2md:main"
pdf2md = "document.pdf2md:main"
html2text = "document.html2text:main"
md2pdf = "document.md2pdf:main"
[tool.hatch.build.targets.wheel]
packages = ["src/document"]
```
After `uv tool install --editable .`, all commands are in PATH.
After `uv tool install --editable .`, the command is in PATH.
## Tool implementation patterns
@@ -368,8 +313,6 @@ uv run ruff format . # Format
## Registering in castle.yaml
Standalone tools:
```yaml
components:
my-tool:
@@ -381,14 +324,14 @@ components:
alias: my-tool
```
Category tools get one entry per tool, all pointing to the same source:
Tools with system dependencies declare them in the manifest:
```yaml
components:
pdf2md:
description: Convert PDF files to Markdown
tool:
source: tools/document/
source: pdf2md/
system_dependencies: [pandoc, poppler-utils]
install:
path:

View File

@@ -0,0 +1,16 @@
[project]
name = "docx-extractor"
version = "0.1.0"
description = "Extract content and metadata from Word .docx files"
requires-python = ">=3.11"
dependencies = []
[project.scripts]
docx-extractor = "docx_extractor.main:main"
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[tool.hatch.build.targets.wheel]
packages = ["src/docx_extractor"]

View File

@@ -1,16 +1,16 @@
[project]
name = "castle-android"
name = "docx2md"
version = "0.1.0"
description = "Castle Android tools"
description = "Convert Word .docx files to Markdown"
requires-python = ">=3.11"
dependencies = []
[project.scripts]
android-backup = "android.android_backup:main"
docx2md = "docx2md.main:main"
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[tool.hatch.build.targets.wheel]
packages = ["src/android"]
packages = ["src/docx2md"]

View File

@@ -1,14 +1,14 @@
[project]
name = "castle-gpt"
name = "gpt"
version = "0.1.0"
description = "Castle GPT tools"
description = "OpenAI text generation utility"
requires-python = ">=3.11"
dependencies = [
"openai>=1.6.0",
]
[project.scripts]
gpt = "gpt.gpt:main"
gpt = "gpt.main:main"
[build-system]
requires = ["hatchling"]

16
html2text/pyproject.toml Normal file
View File

@@ -0,0 +1,16 @@
[project]
name = "html2text"
version = "0.1.0"
description = "Convert HTML content to plain text"
requires-python = ">=3.11"
dependencies = []
[project.scripts]
html2text = "html2text.main:main"
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[tool.hatch.build.targets.wheel]
packages = ["src/html2text"]

16
mbox2eml/pyproject.toml Normal file
View File

@@ -0,0 +1,16 @@
[project]
name = "mbox2eml"
version = "0.1.0"
description = "Convert MBOX mailbox files to individual .eml files"
requires-python = ">=3.11"
dependencies = []
[project.scripts]
mbox2eml = "mbox2eml.main:main"
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[tool.hatch.build.targets.wheel]
packages = ["src/mbox2eml"]

View File

16
md2pdf/pyproject.toml Normal file
View File

@@ -0,0 +1,16 @@
[project]
name = "md2pdf"
version = "0.1.0"
description = "Convert Markdown files to PDF"
requires-python = ">=3.11"
dependencies = []
[project.scripts]
md2pdf = "md2pdf.main:main"
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[tool.hatch.build.targets.wheel]
packages = ["src/md2pdf"]

View File

View File

@@ -1,12 +1,12 @@
[project]
name = "castle-mdscraper"
name = "mdscraper"
version = "0.1.0"
description = "Castle markdown scraping tools"
description = "Combine text files into a single markdown document"
requires-python = ">=3.11"
dependencies = []
[project.scripts]
mdscraper = "mdscraper.mdscraper:main"
mdscraper = "mdscraper.main:main"
[build-system]
requires = ["hatchling"]

View File

View File

@@ -0,0 +1,18 @@
[project]
name = "pdf-extractor"
version = "0.1.0"
description = "Extract content and metadata from PDF files"
requires-python = ">=3.11"
dependencies = [
"pymupdf>=1.24.11",
]
[project.scripts]
pdf-extractor = "pdf_extractor.main:main"
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[tool.hatch.build.targets.wheel]
packages = ["src/pdf_extractor"]

16
pdf2md/pyproject.toml Normal file
View File

@@ -0,0 +1,16 @@
[project]
name = "pdf2md"
version = "0.1.0"
description = "Convert PDF files to Markdown"
requires-python = ">=3.11"
dependencies = []
[project.scripts]
pdf2md = "pdf2md.main:main"
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[tool.hatch.build.targets.wheel]
packages = ["src/pdf2md"]

View File

16
schedule/pyproject.toml Normal file
View File

@@ -0,0 +1,16 @@
[project]
name = "schedule"
version = "0.1.0"
description = "Systemd timer and service management tool"
requires-python = ">=3.11"
dependencies = []
[project.scripts]
schedule = "schedule.main:main"
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[tool.hatch.build.targets.wheel]
packages = ["src/schedule"]

View File

18
search/pyproject.toml Normal file
View File

@@ -0,0 +1,18 @@
[project]
name = "search"
version = "0.1.0"
description = "Manage searchable collections of files"
requires-python = ">=3.11"
dependencies = [
"toml>=0.10.2",
]
[project.scripts]
search = "search.main:main"
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[tool.hatch.build.targets.wheel]
packages = ["src/search"]

View File

View File

@@ -0,0 +1,16 @@
[project]
name = "text-extractor"
version = "0.1.0"
description = "Extract content and metadata from text files"
requires-python = ">=3.11"
dependencies = []
[project.scripts]
text-extractor = "text_extractor.main:main"
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[tool.hatch.build.targets.wheel]
packages = ["src/text_extractor"]

View File

@@ -1,141 +0,0 @@
# Castle Tools
CLI utilities managed by the castle platform. Each tool follows Unix philosophy:
read from stdin or file arguments, write to stdout, compose via pipes.
## Installation
Each category is its own package, installed individually:
```bash
castle sync # installs all category packages
```
Or manually install a single category:
```bash
uv tool install --editable tools/document/
```
## Tools by Category
### android (`castle-android`)
| Tool | Description | Requires |
|------|-------------|----------|
| `android-backup` | Backup Android device using ADB | adb |
### browser (`castle-browser`)
| Tool | Description |
|------|-------------|
| `browser` | Browse the web using natural language via browser-use |
### document (`castle-document`)
| Tool | Description | Requires |
|------|-------------|----------|
| `docx2md` | Convert Word .docx files to Markdown | pandoc |
| `html2text` | Convert HTML content to plain text | |
| `md2pdf` | Convert Markdown files to PDF | pandoc, texlive-latex-base |
| `pdf2md` | Convert PDF files to Markdown | pandoc, poppler-utils |
### gpt (`castle-gpt`)
| Tool | Description |
|------|-------------|
| `gpt` | OpenAI text generation utility |
### mdscraper (`castle-mdscraper`)
| Tool | Description |
|------|-------------|
| `mdscraper` | Combine text files into a single markdown document |
### search (`castle-search`)
| Tool | Description | Requires |
|------|-------------|----------|
| `docx-extractor` | Extract content and metadata from Word .docx files | pandoc |
| `pdf-extractor` | Extract content and metadata from PDF files | |
| `search` | Manage self-contained searchable collections of files | |
| `text-extractor` | Extract content and metadata from text files | |
### system (`castle-system`)
| Tool | Description | Requires |
|------|-------------|----------|
| `backup-collect` | Collect files from various sources into backup directory | rsync |
| `schedule` | Manage systemd user timers and scheduled tasks | |
## Directory Structure
```
tools/
├── <category>/
│ ├── pyproject.toml # Per-category package
│ └── src/<category>/
│ ├── __init__.py
│ ├── <tool>.py # Implementation
│ └── <tool>.md # Documentation (YAML frontmatter + docs)
└── README.md
```
Each tool has two files:
- **`<tool>.py`** -- the implementation, with a `main()` entry point
- **`<tool>.md`** -- YAML frontmatter (command, description, version, category,
system_dependencies) followed by usage documentation
## Adding a New Tool
```bash
castle create my-tool --type tool --category document
```
Or manually:
1. Create `tools/<category>/src/<category>/my_tool.py` with an argparse `main()` function
2. Create `tools/<category>/src/<category>/my_tool.md` with YAML frontmatter
3. Add the entry point to `tools/<category>/pyproject.toml`:
```toml
my-tool = "<category>.my_tool:main"
```
4. Register in `castle.yaml`:
```yaml
my-tool:
description: What it does
tool:
tool_type: python_standalone
category: <category>
source: tools/<category>/
install:
path: { alias: my-tool }
```
5. Run `castle sync` to install
## Tool Types
Castle manages three types of tools:
| Type | Description | Installation |
|------|-------------|-------------|
| `python_standalone` | Own pyproject.toml | Individual `uv tool install` per category |
| `script` | Bash or binary | Symlinked to `~/.local/bin/` |
## Conventions
- Read from stdin or file argument, write to stdout
- Error messages and status to stderr
- Exit 0 on success, 1 on error
- `--help` for usage, `--version` where applicable
- Composable via Unix pipes: `pdf2md paper.pdf | gpt "summarize this"`
## Managing Tools
```bash
castle tool list # All tools grouped by category
castle tool info <name> # Tool details + documentation
castle list --role tool # Tools in the component listing
castle info <name> --json # Full manifest including tool metadata
```

View File

@@ -1,244 +0,0 @@
---
command: android-backup
script: android/android-backup.py
description: Backup Android device using ADB
version: 1.0.0
category: android
system_dependencies:
- adb
---
# android-backup
Backup Android device data using Android Debug Bridge (ADB). Supports backing up photos, videos, documents, app data, SMS, RCS, contacts, and call logs.
## Prerequisites
- **ADB (Android Debug Bridge)** must be installed and in your PATH
- **Android device** with USB debugging enabled
- **USB connection** between computer and Android device
### Enable USB Debugging
1. Go to Settings → About Phone
2. Tap "Build Number" 7 times to enable Developer Options
3. Go to Settings → Developer Options
4. Enable "USB Debugging"
5. Connect device via USB and authorize the computer
## Installation
```bash
# Install the toolkit
cd /data/repos/toolkit
make install
# Verify installation
which android-backup
# Verify ADB is working
adb devices
```
## Usage
### Basic Usage
```bash
# Backup everything using default configuration
android-backup
# Use custom configuration file
android-backup -c custom.json
# Specify device when multiple devices connected
android-backup --device ABC123DEF456
```
### Selective Backups
```bash
# Only backup photos
android-backup --photos-only
# Only backup SMS messages
android-backup --sms-only
# Only backup contacts
android-backup --contacts-only
# Only backup call logs
android-backup --call-logs-only
# Only backup specific app data
android-backup --app-data whatsapp
android-backup --app-data com.example.app
```
### Options
```bash
android-backup [options]
Options:
-c, --config FILE Use custom configuration file (default: android_backup_config.json)
--device SERIAL Specify device serial number when multiple devices connected
--photos-only Only backup photos
--videos-only Only backup videos
--documents-only Only backup documents
--sms-only Only backup SMS messages
--contacts-only Only backup contacts
--call-logs-only Only backup call logs
--app-data PACKAGE Only backup specified app data
-v, --verbose Enable verbose logging
-h, --help Show help message
```
## Configuration
The tool uses a JSON configuration file (default: `android_backup_config.json`) to specify what to backup and where to store it.
### Example Configuration
```json
{
"backup_root": "/data/backups/android",
"device_name": "MyPhone",
"backup_photos": true,
"backup_videos": true,
"backup_documents": true,
"backup_sms": true,
"backup_contacts": true,
"backup_call_logs": true,
"backup_app_data": ["com.whatsapp", "org.telegram.messenger"],
"photo_dirs": ["/sdcard/DCIM", "/sdcard/Pictures"],
"video_dirs": ["/sdcard/DCIM", "/sdcard/Movies"],
"document_dirs": ["/sdcard/Documents", "/sdcard/Download"]
}
```
### Configuration Options
- `backup_root` - Root directory for backups
- `device_name` - Friendly name for the device
- `backup_photos` - Enable photo backup (default: true)
- `backup_videos` - Enable video backup (default: true)
- `backup_documents` - Enable document backup (default: true)
- `backup_sms` - Enable SMS backup (default: true)
- `backup_contacts` - Enable contacts backup (default: true)
- `backup_call_logs` - Enable call logs backup (default: true)
- `backup_app_data` - List of app package names to backup
- `photo_dirs` - List of directories to search for photos
- `video_dirs` - List of directories to search for videos
- `document_dirs` - List of directories to search for documents
## Backup Structure
Backups are organized by date and type:
```
/data/backups/android/MyPhone/
├── 2024-07-17_14-30-00/
│ ├── photos/
│ ├── videos/
│ ├── documents/
│ ├── sms/
│ ├── contacts/
│ ├── call_logs/
│ └── app_data/
│ ├── com.whatsapp/
│ └── org.telegram.messenger/
└── 2024-07-18_10-15-00/
└── ...
```
## Features
- **Incremental backups** - Only copies new or changed files
- **Multiple device support** - Specify device when multiple connected
- **Selective backups** - Choose what to backup
- **App data backup** - Backup specific app data
- **Organized structure** - Date-stamped backup directories
- **Logging** - Detailed logs of backup operations
## Examples
### Full Backup
```bash
# Create full backup with default settings
android-backup
```
### Daily Automated Backup
```bash
# Set up daily backup at 2 AM using schedule tool
schedule add android-backup \
--command "android-backup" \
--schedule "daily" \
--on-calendar "*-*-* 02:00:00" \
--description "Daily Android backup"
```
### Backup Multiple Devices
```bash
# List connected devices
adb devices
# Backup specific device
android-backup --device ABC123DEF456
```
### Custom Configuration
```bash
# Create custom config
cat > my_backup_config.json <<EOF
{
"backup_root": "/mnt/nas/android_backups",
"device_name": "WorkPhone",
"backup_photos": true,
"backup_videos": false,
"backup_app_data": ["com.slack", "com.microsoft.teams"]
}
EOF
# Use custom config
android-backup -c my_backup_config.json
```
## Troubleshooting
### Device Not Found
```bash
# Check if device is connected and authorized
adb devices
# If "unauthorized", check phone for authorization dialog
# If no devices listed, check USB connection and debugging is enabled
```
### Permission Denied
Some files may require root access. The tool will log warnings for files it cannot access but will continue with the backup.
### Multiple Devices
```bash
# List all connected devices
adb devices
# Specify which device to backup
android-backup --device SERIAL_NUMBER
```
## Notes
- Requires USB debugging to be enabled on the Android device
- Some app data may require root access
- Large backups can take significant time
- Ensure sufficient disk space for backups
- First backup will be slower; subsequent backups are incremental

View File

@@ -1,166 +0,0 @@
---
command: browser
script: browser/browser.py
description: Browse the web using natural language via browser-use
version: 1.0.0
category: browser
---
# browser
Browser automation tool powered by browser-use. Execute browser tasks from the command line using natural language instructions.
## Prerequisites
- **LLM Provider** - Requires API key for OpenAI or Anthropic
- **Chromium/Chrome** - Used by the automation framework
## Installation
```bash
# Install the toolkit
cd /data/repos/toolkit
make install
# Verify installation
which browser
```
## Setup
### API Key Configuration
Set up your LLM provider API key:
```bash
# For OpenAI
export OPENAI_API_KEY=sk-...
# For Anthropic Claude
export ANTHROPIC_API_KEY=sk-ant-...
# Add to ~/.bashrc or ~/.profile to persist
```
## Usage
### Basic Usage
```bash
# Execute a browser task
browser "Go to google.com and search for python tutorials"
# Use specific LLM provider
browser --model gpt-4 "Fill out the contact form on example.com"
browser --model claude-3-5-sonnet "Take a screenshot of the homepage"
```
### Options
```bash
browser [options] "instruction"
Options:
instruction Natural language instruction for browser task (required)
--model MODEL LLM model to use (default: gpt-4o-mini)
--headless Run browser in headless mode (no visible window)
--timeout SECONDS Maximum time for task execution (default: 300)
-h, --help Show help message
```
## Supported Models
### OpenAI Models
- `gpt-4o` - Most capable, best for complex tasks
- `gpt-4o-mini` - Fast and affordable (default)
- `gpt-4-turbo`
- `gpt-4`
- `gpt-3.5-turbo`
### Anthropic Models
- `claude-3-5-sonnet-20241022` - Most capable
- `claude-3-opus-20240229`
- `claude-3-sonnet-20240229`
- `claude-3-haiku-20240307`
## Examples
### Web Search
```bash
browser "Search Google for 'best python libraries 2024' and summarize the top 3 results"
```
### Form Filling
```bash
browser "Go to example.com/contact and fill in the form with: Name: John Doe, Email: john@example.com, Message: Testing automation"
```
### Screenshots
```bash
browser "Navigate to github.com and take a screenshot"
```
### Data Extraction
```bash
browser "Go to news.ycombinator.com and list the top 5 post titles"
```
### Multi-step Tasks
```bash
browser "Go to amazon.com, search for 'wireless mouse', filter by 4+ stars, and show me the top 3 results with prices"
```
## Features
- **Natural language control** - Use plain English to automate browser tasks
- **AI-powered** - Uses LLMs to understand and execute complex instructions
- **Multi-step workflows** - Handle complex sequences of actions
- **Screenshot capture** - Can take screenshots as part of tasks
- **Form filling** - Automatically fill out forms
- **Data extraction** - Extract information from web pages
- **Headless mode** - Run without visible browser window
## Environment Variables
- `OPENAI_API_KEY` - API key for OpenAI models
- `ANTHROPIC_API_KEY` - API key for Anthropic Claude models
## Notes
- Tasks are executed using browser automation via browser-use
- Complex tasks may take longer to complete
- Headless mode is faster but you won't see the browser actions
- Some websites may block automation (check robots.txt)
- API costs apply based on model usage
- Default timeout is 5 minutes; adjust with `--timeout` for longer tasks
## Troubleshooting
### API Key Not Found
```bash
# Ensure API key is exported
echo $OPENAI_API_KEY
# or
echo $ANTHROPIC_API_KEY
# Add to shell config if missing
export OPENAI_API_KEY=sk-...
```
### Browser Launch Fails
Ensure Chromium or Chrome is installed on your system. The browser-use library will try to find it automatically.
### Timeout Errors
For complex tasks, increase the timeout:
```bash
browser --timeout 600 "complex long-running task"
```

View File

@@ -1,20 +0,0 @@
[project]
name = "castle-document"
version = "0.1.0"
description = "Castle document conversion tools"
requires-python = ">=3.11"
dependencies = []
[project.scripts]
docx2md = "document.docx2md:main"
pdf2md = "document.pdf2md:main"
html2text = "document.html2text:main"
md2pdf = "document.md2pdf:main"
mbox2eml = "document.mbox2eml:main"
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[tool.hatch.build.targets.wheel]
packages = ["src/document"]

View File

@@ -1,68 +0,0 @@
---
command: docx2md
script: document/docx2md.py
description: Convert Word .docx files to Markdown
version: 1.0.0
category: document
system_dependencies:
- pandoc
---
# docx2md
Convert Microsoft Word (.docx) documents to Markdown format.
## Installation
```bash
cd /data/repos/toolkit
make install
which docx2md
```
## Usage
```bash
# Convert single file
docx2md document.docx
# Output to specific file
docx2md document.docx -o output.md
# Convert from stdin
cat document.docx | docx2md > output.md
```
### Options
```bash
docx2md input.docx [-o output.md]
Options:
input.docx Input Word document
-o, --output FILE Output markdown file (default: stdout)
-h, --help Show help message
```
## Features
- Preserves document structure (headings, lists, tables)
- Converts formatting (bold, italic, links)
- Handles images (extracts and references)
- Supports tables
- Maintains document hierarchy
## Examples
```bash
# Convert and save
docx2md report.docx -o report.md
# Convert multiple files
for file in *.docx; do
docx2md "$file" -o "${file%.docx}.md"
done
# Pipe to other tools
docx2md document.docx | grep "important"
```

View File

@@ -1,69 +0,0 @@
---
command: html2text
script: document/html2text.py
description: Convert HTML content to plain text
version: 1.0.0
category: document
---
# html2text (document)
Convert HTML files to readable plain text, removing tags while preserving structure.
Note: This is the same tool as in the email category, available for document conversion tasks.
## Installation
```bash
cd /data/repos/toolkit
make install
which html2text
```
## Usage
```bash
# Convert HTML file
html2text page.html
# Convert from stdin
cat page.html | html2text
# Save to file
html2text page.html > page.txt
```
### Options
```bash
html2text [file]
Options:
file HTML file to convert (optional, reads from stdin if not provided)
-h, --help Show help message
```
## Features
- Removes HTML tags
- Preserves text structure
- Formats lists and paragraphs
- Converts links to readable format
- Terminal-friendly output
## Examples
```bash
# Convert web page
curl https://example.com | html2text
# Convert multiple files
for file in *.html; do
html2text "$file" > "${file%.html}.txt"
done
# Extract article text
html2text article.html | less
```
See also: [email/html2text](../email/html2text.md) for email-specific usage.

View File

@@ -1,25 +0,0 @@
# mbox2eml
Convert MBOX mailbox files to individual .eml files with clean, descriptive
filenames based on date, sender, and recipient.
## Usage
```bash
mbox2eml mailbox.mbox # Export to eml_output/
mbox2eml mailbox.mbox -o emails/ # Export to custom directory
mbox2eml mailbox.mbox --max-length 80 # Truncate filenames at 80 chars
```
Output filenames look like:
```
2025-03-23_15-09-42_from_alice_example_com_to_bob_example_com.eml
```
## What can you do with .eml files?
- Open in email clients (Thunderbird, Outlook, etc.)
- Parse and analyze with Python or CLI tools
- Extract metadata, body, or attachments
- Archive and search with grep, ripgrep, etc.

View File

@@ -1,76 +0,0 @@
---
command: md2pdf
script: document/md2pdf.py
description: Convert Markdown files to PDF
version: 1.0.0
category: document
system_dependencies:
- pandoc
- texlive-latex-base
---
# md2pdf
Convert Markdown documents to PDF format.
## Installation
```bash
cd /data/repos/toolkit
make install
which md2pdf
```
## Usage
```bash
# Convert single file
md2pdf document.md
# Output to specific file
md2pdf document.md -o output.pdf
# Convert from stdin
cat document.md | md2pdf > output.pdf
```
### Options
```bash
md2pdf input.md [-o output.pdf]
Options:
input.md Input Markdown document
-o, --output FILE Output PDF file (default: stdout)
-h, --help Show help message
```
## Features
- Converts Markdown to formatted PDF
- Preserves headings, lists, and formatting
- Supports code blocks
- Handles links
- Clean, readable PDF output
## Examples
```bash
# Convert and save
md2pdf README.md -o README.pdf
# Convert multiple files
for file in *.md; do
md2pdf "$file" -o "${file%.md}.pdf"
done
# Create PDF from concatenated markdown
cat intro.md body.md conclusion.md | md2pdf -o complete.pdf
```
## Notes
- Supports standard Markdown syntax
- Some advanced Markdown features may not render
- PDF styling is fixed (no custom themes)
- Images referenced in Markdown should be accessible

View File

@@ -1,76 +0,0 @@
---
command: pdf2md
script: document/pdf2md.py
description: Convert PDF files to Markdown
version: 1.0.0
category: document
system_dependencies:
- pandoc
- poppler-utils
---
# pdf2md
Convert PDF documents to Markdown format with text extraction and formatting preservation.
## Installation
```bash
cd /data/repos/toolkit
make install
which pdf2md
```
## Usage
```bash
# Convert single PDF
pdf2md document.pdf
# Output to specific file
pdf2md document.pdf -o output.md
# Convert from stdin
cat document.pdf | pdf2md > output.md
```
### Options
```bash
pdf2md input.pdf [-o output.md]
Options:
input.pdf Input PDF document
-o, --output FILE Output markdown file (default: stdout)
-h, --help Show help message
```
## Features
- Extracts text from PDFs
- Preserves basic formatting where possible
- Handles multi-page documents
- Works with text-based PDFs
- Pipe-friendly output
## Examples
```bash
# Convert and save
pdf2md report.pdf -o report.md
# Convert multiple PDFs
for file in *.pdf; do
pdf2md "$file" -o "${file%.pdf}.md"
done
# Extract and search
pdf2md document.pdf | grep "keyword"
```
## Notes
- Works best with text-based PDFs
- Scanned PDFs (images) require OCR
- Complex layouts may lose some formatting
- Tables are converted to plain text

View File

@@ -1,157 +0,0 @@
---
command: gpt
script: gpt/gpt.py
description: A simple OpenAI text generation utility
version: 1.0.0
category: gpt
---
# gpt
Interact with OpenAI's GPT models from the command line. Send prompts and receive AI-generated responses.
## Prerequisites
- OpenAI API key
## Installation
```bash
cd /data/repos/toolkit
make install
which gpt
```
## Setup
```bash
# Set API key
export OPENAI_API_KEY=sk-...
# Add to ~/.bashrc or ~/.profile to persist
```
## Usage
### Basic Usage
```bash
# Send a prompt
gpt "Explain quantum computing in simple terms"
# Use specific model
gpt --model gpt-4 "Write a Python function to sort a list"
# Read prompt from file
gpt < prompt.txt
# Interactive mode
echo "What is the capital of France?" | gpt
```
### Options
```bash
gpt [options] "prompt"
Options:
prompt Text prompt for GPT (required)
--model MODEL GPT model to use (default: gpt-4o-mini)
--temperature TEMP Creativity level 0.0-2.0 (default: 0.7)
--max-tokens NUM Maximum response length (default: 1000)
--system PROMPT System prompt to set behavior
-h, --help Show help message
```
## Supported Models
- `gpt-4o` - Most capable, best for complex tasks
- `gpt-4o-mini` - Fast and affordable (default)
- `gpt-4-turbo` - Large context window
- `gpt-4` - High capability
- `gpt-3.5-turbo` - Fast and economical
## Examples
### Simple Questions
```bash
gpt "What is the weather like today?"
gpt "Explain recursion"
```
### Code Generation
```bash
gpt "Write a Python function to calculate fibonacci numbers"
gpt --model gpt-4 "Create a React component for a todo list"
```
### Text Processing
```bash
# Summarize a file
cat article.txt | gpt "Summarize this article in 3 bullet points"
# Translate
echo "Hello world" | gpt "Translate to French"
# Code review
cat script.py | gpt "Review this code and suggest improvements"
```
### With System Prompts
```bash
gpt --system "You are a Python expert" "How do I use asyncio?"
gpt --system "Respond in haiku form" "Describe AI"
```
### Batch Processing
```bash
# Process multiple prompts
for question in "What is AI?" "What is ML?" "What is DL?"; do
gpt "$question" >> answers.txt
done
```
## Features
- **Multiple models** - Choose from various GPT models
- **Flexible input** - Prompt as argument or from stdin
- **Customizable** - Adjust temperature, tokens, system prompts
- **Pipe-friendly** - Works in Unix pipelines
- **Fast** - Direct API access
## Environment Variables
- `OPENAI_API_KEY` - Your OpenAI API key (required)
## Tips
- Use `gpt-4o-mini` for fast, cheap responses (default)
- Use `gpt-4` or `gpt-4o` for complex reasoning
- Lower temperature (0.1-0.3) for factual responses
- Higher temperature (0.8-1.5) for creative writing
- System prompts help control tone and expertise level
## Cost Considerations
- `gpt-4o-mini` is most economical for general use
- `gpt-4` costs more but provides higher quality
- Limit max-tokens to control costs
- Monitor usage in OpenAI dashboard
## Troubleshooting
### API Key Not Found
```bash
echo $OPENAI_API_KEY # Should show your key
export OPENAI_API_KEY=sk-...
```
### Rate Limits
If you hit rate limits, wait a moment and retry, or upgrade your OpenAI plan.

View File

@@ -1,142 +0,0 @@
---
command: mdscraper
script: mdscraper/mdscraper.py
description: Combine text files from a directory into a single markdown document with
.gitignore-style filtering
version: 1.0.0
category: mdscraper
---
# mdscraper
Convert web pages to clean Markdown format. Scrapes web content and converts to readable Markdown, removing ads and clutter.
## Installation
```bash
cd /data/repos/toolkit
make install
which mdscraper
```
## Usage
### Basic Usage
```bash
# Convert web page to Markdown
mdscraper https://example.com
# Save to file
mdscraper https://example.com > article.md
# Multiple URLs
mdscraper https://site1.com https://site2.com
```
### Options
```bash
mdscraper [options] URL [URL...]
Options:
URL Web page URL to scrape (required)
-o, --output FILE Output file (default: stdout)
--user-agent UA Custom user agent string
-h, --help Show help message
```
## Features
- **Clean conversion** - Removes ads, navigation, footers
- **Markdown output** - Well-formatted Markdown
- **Main content extraction** - Focuses on article content
- **Link preservation** - Maintains links in Markdown format
- **Image handling** - Includes images with Markdown syntax
- **Fast** - Efficient scraping and conversion
## Examples
### Save Article
```bash
mdscraper https://blog.example.com/article > article.md
```
### Scrape Multiple Pages
```bash
# Scrape all pages
for url in $(cat urls.txt); do
mdscraper "$url" > "$(echo $url | md5sum | cut -d' ' -f1).md"
done
```
### Convert for Reading
```bash
# Scrape and view
mdscraper https://news.site/article | less
# Scrape and convert to PDF
mdscraper https://article.com | md2pdf -o article.pdf
```
### With Custom User Agent
```bash
mdscraper --user-agent "MyBot/1.0" https://example.com
```
## Use Cases
- **Article archiving** - Save web articles as Markdown
- **Content migration** - Convert web content for static sites
- **Offline reading** - Download articles for later
- **Documentation** - Scrape docs to local Markdown
- **Research** - Collect and organize web content
## Output Format
Markdown output includes:
- Article title as H1
- Headings at appropriate levels
- Paragraphs with proper spacing
- Lists (ordered and unordered)
- Links in `[text](url)` format
- Images in `![alt](url)` format
- Code blocks when present
## Notes
- Respects robots.txt where possible
- Some sites may block scraping
- JavaScript-heavy sites may not work well
- Rate limit requests to avoid overwhelming servers
- Consider caching results for repeated access
## Troubleshooting
### Site Blocks Requests
Try using a custom user agent:
```bash
mdscraper --user-agent "Mozilla/5.0..." https://site.com
```
### JavaScript Content Missing
For JavaScript-heavy sites, consider using the `browser` tool instead:
```bash
browser "Go to URL and extract the article text"
```
### Rate Limiting
Add delays between requests:
```bash
for url in $(cat urls.txt); do
mdscraper "$url" > file.md
sleep 2
done
```

View File

@@ -1,22 +0,0 @@
[project]
name = "castle-search"
version = "0.1.0"
description = "Castle search and indexing tools"
requires-python = ">=3.11"
dependencies = [
"toml>=0.10.2",
"pymupdf>=1.24.11",
]
[project.scripts]
search = "search.search:main"
pdf-extractor = "search.pdf_extractor:main"
docx-extractor = "search.docx_extractor:main"
text-extractor = "search.text_extractor:main"
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[tool.hatch.build.targets.wheel]
packages = ["src/search"]

View File

@@ -1,73 +0,0 @@
---
command: docx-extractor
script: search/docx_extractor.py
description: Extract content and metadata from Word .docx files
version: 1.0.0
category: search
system_dependencies:
- pandoc
---
# docx-extractor
Extract text from Microsoft Word (.docx) documents.
## Installation
```bash
cd /data/repos/toolkit
make install
which docx-extractor
```
## Usage
```bash
# Extract from Word doc
docx-extractor document.docx
# Save to file
docx-extractor document.docx > text.txt
# Multiple documents
docx-extractor file1.docx file2.docx
```
### Options
```bash
docx-extractor [files...]
Options:
files DOCX files to extract text from
-h, --help Show help message
```
## Features
- Extract text from .docx files
- Handles headers, footers, and body text
- Preserves basic structure
- Fast extraction
## Examples
```bash
# Single document
docx-extractor report.docx
# Batch extraction
for file in *.docx; do
docx-extractor "$file" > "${file%.docx}.txt"
done
# Search extracted text
docx-extractor document.docx | grep "keyword"
```
## Notes
- Only supports .docx format (not .doc)
- Formatting is stripped
- Tables become plain text
- Images are not extracted

View File

@@ -1,70 +0,0 @@
---
command: pdf-extractor
script: search/pdf_extractor.py
description: Extract content and metadata from PDF files
version: 1.0.0
category: search
---
# pdf-extractor
Extract text from PDF documents.
## Installation
```bash
cd /data/repos/toolkit
make install
which pdf-extractor
```
## Usage
```bash
# Extract from PDF
pdf-extractor document.pdf
# Save to file
pdf-extractor document.pdf > text.txt
# Multiple PDFs
pdf-extractor file1.pdf file2.pdf
```
### Options
```bash
pdf-extractor [files...]
Options:
files PDF files to extract text from
-h, --help Show help message
```
## Features
- Extract text from PDFs
- Handles multi-page documents
- Preserves text layout where possible
- Fast extraction
## Examples
```bash
# Single PDF
pdf-extractor report.pdf
# Batch extraction
for file in *.pdf; do
pdf-extractor "$file" > "${file%.pdf}.txt"
done
# Search extracted text
pdf-extractor document.pdf | grep "keyword"
```
## Notes
- Works with text-based PDFs
- Scanned PDFs (images) require OCR
- Complex layouts may affect text order

View File

@@ -1,123 +0,0 @@
---
command: search
script: search/search.py
description: Manage self-contained searchable collections of files
version: 1.0.0
category: search
---
# search
Fast full-text search using Tantivy search index. Index and search documents with blazing speed.
## Installation
```bash
cd /data/repos/toolkit
make install
which search
```
## Usage
### Index Documents
```bash
# Index a directory
search index /path/to/documents
# Index specific files
search index file1.txt file2.pdf file3.docx
```
### Search
```bash
# Simple search
search query "search terms"
# Search with field
search query --field title "document title"
# Limit results
search query --limit 10 "search terms"
```
### Options
```bash
search index [paths...] Index files or directories
search query [options] QUERY Search indexed documents
Query options:
QUERY Search query (required)
--field FIELD Search specific field (title, body, path)
--limit N Maximum results to return (default: 20)
-h, --help Show help message
```
## Features
- **Fast searching** - Powered by Tantivy search engine
- **Multiple formats** - Supports PDF, DOCX, TXT, and more
- **Full-text search** - Search entire document contents
- **Field search** - Search specific fields
- **Relevance ranking** - Results sorted by relevance
## Examples
### Index and Search
```bash
# Index your documents
search index ~/Documents
# Search for terms
search query "important meeting notes"
search query "project proposal 2024"
```
### Field-Specific Search
```bash
# Search in title only
search query --field title "Annual Report"
# Search in body
search query --field body "quarterly results"
```
### Batch Operations
```bash
# Index multiple directories
search index ~/Documents ~/Downloads ~/Desktop
# Incremental indexing
search index ~/Documents/new_files
```
## Supported File Types
- **Text files** - .txt, .md, .log
- **PDF documents** - .pdf
- **Word documents** - .docx
- **Other formats** - via text extractors
## Index Location
Index is stored in `~/.local/share/search/index` by default.
## Tips
- Index documents once, search many times
- Re-index when documents change
- Use quotes for exact phrases
- Field searches are faster for specific needs
## Notes
- Initial indexing may take time for large collections
- Index size depends on document collection size
- Supports incremental indexing
- UTF-8 text encoding recommended

View File

@@ -1,71 +0,0 @@
---
command: text-extractor
script: search/text_extractor.py
description: Extract content and metadata from text files
version: 1.0.0
category: search
---
# text-extractor
Extract plain text from various document formats.
## Installation
```bash
cd /data/repos/toolkit
make install
which text-extractor
```
## Usage
```bash
# Extract from file
text-extractor document.txt
# Extract to file
text-extractor document.txt > output.txt
# Process multiple files
text-extractor file1.txt file2.txt file3.txt
```
### Options
```bash
text-extractor [files...]
Options:
files Files to extract text from
-h, --help Show help message
```
## Supported Formats
- Plain text (.txt)
- Markdown (.md)
- Log files (.log)
- Other text-based formats
## Features
- Fast text extraction
- Handles multiple files
- Preserves text encoding
- Pipe-friendly output
## Examples
```bash
# Extract single file
text-extractor document.txt
# Batch processing
for file in *.txt; do
text-extractor "$file" > "${file%.txt}_extracted.txt"
done
# Use in pipeline
text-extractor doc.txt | grep "keyword"
```

View File

@@ -1,137 +0,0 @@
---
command: backup-collect
script: system/backup-collect.py
description: Collect files from various sources into backup directory
version: 1.0.0
category: system
system_dependencies:
- rsync
---
# backup-collect
Collect system information and configuration files for backup purposes.
## Installation
```bash
cd /data/repos/toolkit
make install
which backup-collect
```
## Usage
```bash
# Collect system information
backup-collect
# Specify output directory
backup-collect -o /path/to/backup
# Collect specific categories
backup-collect --category config
backup-collect --category packages
```
### Options
```bash
backup-collect [options]
Options:
-o, --output DIR Output directory for collected data
--category CAT Collect specific category (config, packages, system)
-v, --verbose Enable verbose output
-h, --help Show help message
```
## What It Collects
### System Information
- OS version and details
- Kernel information
- Hardware details
- System uptime
### Configuration Files
- User dotfiles (.bashrc, .vimrc, etc.)
- Application configs
- SSH keys (with permissions preserved)
- Custom scripts
### Package Lists
- Installed system packages
- Python packages
- npm packages
- Other package managers
## Features
- **Comprehensive** - Collects essential system data
- **Organized** - Structured output directory
- **Safe** - Preserves permissions
- **Selective** - Choose what to collect
- **Portable** - Outputs in standard formats
## Examples
### Full Backup
```bash
backup-collect -o ~/backups/system-$(date +%Y%m%d)
```
### Scheduled Backup
```bash
# Weekly system backup
schedule add system-backup \
--command "backup-collect -o /mnt/backups/system-\$(date +\%Y\%m\%d)" \
--schedule "weekly" \
--on-calendar "Sun 02:00:00"
```
### Selective Collection
```bash
# Only collect configs
backup-collect --category config -o ~/configs
# Only package lists
backup-collect --category packages -o ~/packages
```
## Output Structure
```
backup-directory/
├── system/
│ ├── os-release
│ ├── kernel-version
│ └── hardware-info
├── config/
│ ├── dotfiles/
│ ├── ssh/
│ └── app-configs/
└── packages/
├── apt-packages.txt
├── pip-packages.txt
└── npm-packages.txt
```
## Use Cases
- **System migration** - Document current setup
- **Disaster recovery** - Quick system restoration
- **Configuration management** - Track system changes
- **Compliance** - Document installed software
- **Development** - Share development environment setup
## Notes
- Does not collect sensitive data by default
- SSH keys are collected but not passwords
- Large files are skipped
- Runs without root (collects what's accessible)
- Safe to run frequently

View File

@@ -1,388 +0,0 @@
# schedule
A tool to easily create, manage, and monitor systemd user timers and services following best practices.
## Overview
`schedule` simplifies the management of systemd user timers by providing a simple command-line interface to create timer/service pairs that execute bash commands on a schedule. It handles all the boilerplate configuration and follows systemd best practices for security and resource management.
## Installation
Install the toolkit:
```bash
cd /data/repos/toolkit
make install
```
The `schedule` command will be available in your PATH.
### Enable Linger (Optional but Recommended)
By default, systemd user units only run while you're logged in. To allow timers to run even when you're logged out, enable linger:
```bash
loginctl enable-linger $USER
```
This is essential for timers that should run 24/7, like automated sync jobs.
## Features
- **Easy timer creation** - Create timers with a single command
- **Multiple schedule types** - Support for interval-based and calendar-based schedules
- **Environment variables** - Support for environment files and inline variables
- **Pre-conditions** - Run commands only when conditions are met
- **Security hardening** - Automatic security settings (NoNewPrivileges, PrivateTmp)
- **Log management** - Easy access to service logs via journalctl
- **Status monitoring** - Check timer and service status
## Usage
### List timers
Show all active user timers:
```bash
schedule list
```
Show all timers including inactive ones:
```bash
schedule list --all
```
### Add a timer
Basic timer that runs every 5 minutes:
```bash
schedule add my-task \
--command "echo 'Hello, World!'" \
--schedule "5min"
```
Daily backup at 2 AM:
```bash
schedule add daily-backup \
--command "backup run" \
--schedule "daily" \
--on-calendar "*-*-* 02:00:00" \
--description "Daily backup job"
```
ProtonMail sync with environment file and condition:
```bash
schedule add protonmail-sync \
--command "protonmail sync" \
--schedule "5min" \
--description "Sync ProtonMail emails" \
--env-file ~/.config/protonmail/env \
--condition-command "pgrep -f protonmail-bridge"
```
With environment variables:
```bash
schedule add my-task \
--command "my-script.sh" \
--schedule "1hour" \
--environment "API_KEY=secret123" \
--environment "LOG_LEVEL=debug" \
--working-directory "/home/user/projects"
```
### Check status
View timer and service status:
```bash
schedule status protonmail-sync
```
This shows:
- Timer status (active/inactive)
- Service status (last run, next run)
- Next scheduled execution time
### View logs
Follow logs in real-time:
```bash
schedule logs protonmail-sync --follow
```
Show last 50 lines:
```bash
schedule logs protonmail-sync --lines 50
```
Show logs since today:
```bash
schedule logs protonmail-sync --since today
```
### Start/Stop timers
Start a timer:
```bash
schedule start protonmail-sync
```
Stop a timer:
```bash
schedule stop protonmail-sync
```
### Enable/Disable timers
Enable timer (start on boot):
```bash
schedule enable protonmail-sync
```
Disable timer (don't start on boot):
```bash
schedule disable protonmail-sync
```
### Remove a timer
Remove timer and service files:
```bash
schedule remove protonmail-sync
```
Remove but keep logs:
```bash
schedule remove protonmail-sync --keep-logs
```
## Schedule Formats
### Interval-based schedules
Use simple time units:
- `5min` - Every 5 minutes
- `1hour` or `1h` - Every hour
- `30min` - Every 30 minutes
- `1day` or `1d` - Every day
- `1week` or `1w` - Every week
- `hourly` - Every hour (alias for 1hour)
- `daily` - Every day (alias for 1day)
### Calendar-based schedules
Use systemd calendar syntax with `--on-calendar`:
- `*-*-* 02:00:00` - Every day at 2 AM
- `Mon *-*-* 09:00:00` - Every Monday at 9 AM
- `*-*-01 00:00:00` - First day of each month at midnight
- `*-*-* *:00:00` - Every hour on the hour
## Advanced Options
### Add command options
- `--name` - Name of the timer/service (required)
- `--command` - Bash command to execute (required)
- `--schedule` - Schedule format (required)
- `--description` - Human-readable description
- `--env-file` - Path to environment file to source
- `--environment` / `-e` - Set environment variable (KEY=VALUE)
- `--condition-command` - Command that must succeed before running
- `--working-directory` - Working directory for the command
- `--on-boot-sec` - Delay after boot (default: 1min)
- `--on-calendar` - Override with calendar-based schedule
- `--persistent` - Run missed timers if system was offline
- `--force` - Overwrite existing timer/service
- `--no-enable` - Don't enable the timer
- `--no-start` - Don't start the timer
## Examples
### ProtonMail Sync
```bash
schedule add protonmail-sync \
--command "protonmail sync" \
--schedule "5min" \
--description "Sync ProtonMail emails every 5 minutes" \
--env-file ~/.config/protonmail/env \
--condition-command "pgrep -f protonmail-bridge"
```
### Daily Backup
```bash
schedule add daily-backup \
--command "backup run --full" \
--schedule "daily" \
--on-calendar "*-*-* 02:00:00" \
--description "Daily backup at 2 AM" \
--environment "BACKUP_DIR=/mnt/backups"
```
### Hourly Data Sync
```bash
schedule add data-sync \
--command "rsync -av /source/ /dest/" \
--schedule "hourly" \
--description "Sync data every hour" \
--working-directory "/home/user/sync"
```
### Git Auto-Commit
```bash
schedule add git-autocommit \
--command "git add -A && git commit -m 'Auto-commit' && git push" \
--schedule "15min" \
--description "Auto-commit and push changes" \
--working-directory "/home/user/projects/myrepo"
```
## File Locations
Timer and service files are stored in:
```
~/.config/systemd/user/
├── <name>.service
└── <name>.timer
```
## Security Features
All services created by `schedule` include:
- `NoNewPrivileges=yes` - Cannot gain new privileges
- `PrivateTmp=yes` - Isolated /tmp directory
- Proper PATH configuration
- Journal logging for stdout/stderr
## Troubleshooting
### Timer not running
Check if the timer is enabled and active:
```bash
schedule status <name>
```
### Service failing
View the logs to see what went wrong:
```bash
schedule logs <name> --lines 50
```
### Command not found
Ensure the command is in the PATH or use an absolute path:
```bash
schedule add my-task \
--command "/usr/local/bin/my-script" \
--schedule "5min"
```
### Environment variables not working
Use an environment file:
```bash
# Create env file
echo "API_KEY=secret" > ~/.config/myapp/env
chmod 600 ~/.config/myapp/env
# Use it
schedule add my-task \
--command "my-script" \
--schedule "5min" \
--env-file ~/.config/myapp/env
```
## Comparison with Manual Setup
Instead of manually creating:
1. `.service` file with proper configuration
2. `.timer` file with scheduling
3. Running `systemctl --user daemon-reload`
4. Running `systemctl --user enable <name>.timer`
5. Running `systemctl --user start <name>.timer`
You can do it all with one command:
```bash
schedule add <name> --command "<cmd>" --schedule "<schedule>"
```
## See Also
- `man systemd.timer` - systemd timer documentation
- `man systemd.service` - systemd service documentation
- `man systemd.time` - systemd time specification
- `journalctl` - Query systemd journal
## Important Notes
### Environment Variables
**Critical**: Systemd user units do NOT inherit environment variables from your shell (e.g., ~/.bashrc). Variables set with `export` in your shell will not be available to services.
To make environment variables available to your scheduled commands, pass them inline when creating the timer:
```bash
schedule add my-task \
--command "my-script" \
--schedule "1hour" \
--environment "API_KEY=secret123" \
--environment "DATABASE_URL=postgres://localhost/mydb"
```
You can also use the `-e` shorthand:
```bash
schedule add my-task \
--command "my-script" \
--schedule "1hour" \
-e "API_KEY=secret123" \
-e "DATABASE_URL=postgres://localhost/mydb"
```
### User vs System Units
This tool creates **user** systemd units (`systemctl --user`), not system units. User units:
- Run as your user (no root required)
- Stored in `~/.config/systemd/user/`
- Only run while you're logged in (unless linger is enabled)
- Have access to your user's files and permissions
To enable timers to run 24/7 even when logged out:
```bash
loginctl enable-linger $USER
```
## License
Same license as the toolkit project.