Removes tools from grouping.
This commit is contained in:
10
CLAUDE.md
10
CLAUDE.md
@@ -33,13 +33,9 @@ uv run my-service # starts on auto-assigned port
|
||||
castle service enable my-service # register with systemd
|
||||
castle gateway reload # update reverse proxy routes
|
||||
|
||||
# Standalone tool
|
||||
# Tool
|
||||
castle create my-tool --type tool --description "Does something"
|
||||
cd my-tool && uv sync
|
||||
|
||||
# Category tool (adds to existing tools/<category>/ package)
|
||||
castle create my-tool --type tool --category document --description "Does something"
|
||||
cd tools/document && uv sync
|
||||
```
|
||||
|
||||
The `castle create` command scaffolds the project, generates a CLAUDE.md,
|
||||
@@ -59,8 +55,8 @@ castle lint [project] # Run linter (one or all)
|
||||
castle sync # Update submodules + uv sync all
|
||||
castle run <component> # Run component in foreground
|
||||
castle logs <component> [-f] [-n 50] # View component logs
|
||||
castle tool list # List tools grouped by category
|
||||
castle tool info <name> # Show tool details + docs
|
||||
castle tool list # List all tools
|
||||
castle tool info <name> # Show tool details
|
||||
castle gateway start|stop|reload|status # Manage Caddy reverse proxy
|
||||
castle service enable|disable <name> # Manage individual systemd service
|
||||
castle service status # Show all service statuses
|
||||
|
||||
13
TODO.md
Normal file
13
TODO.md
Normal file
@@ -0,0 +1,13 @@
|
||||
# TO DO
|
||||
|
||||
- Remove devbox-connect. Instead, make it easy to copy an ssh tunnel command to expose all ports of all services to a remote box: `ssh -L 9000:localhost:9000 payne@dev.payne.io, etc.
|
||||
- tool.uv.sources should be in cli, not in castle-api
|
||||
- Add a scripts dir?
|
||||
- Maybe there's no real reason to have special handling for the `tools` dir (one md file per tool, put in categories, etc.). On the one hand, the categories help them share dependencies. On the other, there's not really a need to share dependencies because uv does just fine. Also, they prob don't need markdown files because their description can just be in the --help arg, and in the `castle.yaml` registration. Having them flat would allow us to just think of everything as a tool: a bash script tool, a python tool, a rust tool. They're all things just follow a std-in std-out pattern so they are unix-philosophy good. Daemons otoh, follow daemon patterns (env config, logging, long-running, port mapping, etc.)
|
||||
|
||||
|
||||
- tools: use std-in/std-out unix philosophy
|
||||
- daemons or web-services: expose ports
|
||||
- workers: just keep doing work in a background--usually watching the filesystem or a queue
|
||||
- jobs: scheduled tools or daemon requests
|
||||
- frontend: files served up by caddy
|
||||
@@ -1,17 +1,16 @@
|
||||
[project]
|
||||
name = "castle-system"
|
||||
name = "android-backup"
|
||||
version = "0.1.0"
|
||||
description = "Castle system administration tools"
|
||||
description = "Backup Android device using ADB"
|
||||
requires-python = ">=3.11"
|
||||
dependencies = []
|
||||
|
||||
[project.scripts]
|
||||
backup-collect = "system.backup_collect:main"
|
||||
schedule = "system.schedule:main"
|
||||
android-backup = "android_backup.main:main"
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
[tool.hatch.build.targets.wheel]
|
||||
packages = ["src/system"]
|
||||
packages = ["src/android_backup"]
|
||||
16
backup-collect/pyproject.toml
Normal file
16
backup-collect/pyproject.toml
Normal file
@@ -0,0 +1,16 @@
|
||||
[project]
|
||||
name = "backup-collect"
|
||||
version = "0.1.0"
|
||||
description = "Collect files from various sources into backup directory"
|
||||
requires-python = ">=3.11"
|
||||
dependencies = []
|
||||
|
||||
[project.scripts]
|
||||
backup-collect = "backup_collect.main:main"
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
[tool.hatch.build.targets.wheel]
|
||||
packages = ["src/backup_collect"]
|
||||
@@ -1,7 +1,7 @@
|
||||
[project]
|
||||
name = "castle-browser"
|
||||
name = "browser"
|
||||
version = "0.1.0"
|
||||
description = "Castle browser automation tools"
|
||||
description = "Browser automation tool powered by browser-use"
|
||||
requires-python = ">=3.11"
|
||||
dependencies = [
|
||||
"python-dotenv>=1.0.0",
|
||||
@@ -11,7 +11,7 @@ dependencies = [
|
||||
]
|
||||
|
||||
[project.scripts]
|
||||
browser = "browser.browser:main"
|
||||
browser = "browser.main:main"
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
@@ -79,13 +79,6 @@ class ToolSummary(BaseModel):
|
||||
installed: bool = False
|
||||
|
||||
|
||||
class ToolCategory(BaseModel):
|
||||
"""Tools grouped by category."""
|
||||
|
||||
name: str
|
||||
tools: list[ToolSummary]
|
||||
|
||||
|
||||
class ToolDetail(ToolSummary):
|
||||
"""Full detail for a single tool, including documentation."""
|
||||
|
||||
|
||||
@@ -11,7 +11,7 @@ from castle_cli.config import load_config
|
||||
from castle_cli.manifest import ComponentManifest
|
||||
|
||||
from castle_api.config import settings
|
||||
from castle_api.models import ToolCategory, ToolDetail, ToolSummary
|
||||
from castle_api.models import ToolDetail, ToolSummary
|
||||
|
||||
router = APIRouter(tags=["tools"])
|
||||
|
||||
@@ -42,55 +42,16 @@ def _tool_summary(name: str, manifest: ComponentManifest, root: Path | None = No
|
||||
)
|
||||
|
||||
|
||||
def _find_md_for_tool(
|
||||
root: Path,
|
||||
source: str,
|
||||
tool_name: str,
|
||||
) -> Path | None:
|
||||
"""Find the .md documentation file for a tool source path."""
|
||||
source_path = root / source
|
||||
if source_path.is_file():
|
||||
md = source_path.with_suffix(".md")
|
||||
if md.exists():
|
||||
return md
|
||||
elif source_path.is_dir():
|
||||
py_name = tool_name.replace("-", "_")
|
||||
pkg_name = source_path.name
|
||||
md = source_path / "src" / pkg_name / f"{py_name}.md"
|
||||
if md.exists():
|
||||
return md
|
||||
return None
|
||||
|
||||
|
||||
def _strip_frontmatter(content: str) -> str:
|
||||
"""Strip YAML frontmatter from markdown content."""
|
||||
if content.startswith("---\n"):
|
||||
end = content.find("\n---\n", 4)
|
||||
if end != -1:
|
||||
content = content[end + 5:]
|
||||
return content.strip()
|
||||
|
||||
|
||||
@router.get("/tools", response_model=list[ToolCategory])
|
||||
def list_tools() -> list[ToolCategory]:
|
||||
"""List tools grouped by source directory."""
|
||||
@router.get("/tools", response_model=list[ToolSummary])
|
||||
def list_tools() -> list[ToolSummary]:
|
||||
"""List all registered tools."""
|
||||
config = load_config(settings.castle_root)
|
||||
tools = {k: v for k, v in config.components.items() if v.tool}
|
||||
|
||||
by_group: dict[str, list[ToolSummary]] = {}
|
||||
for name, manifest in tools.items():
|
||||
t = manifest.tool
|
||||
assert t is not None
|
||||
if t.source:
|
||||
group = Path(t.source).name
|
||||
else:
|
||||
group = "standalone"
|
||||
by_group.setdefault(group, []).append(_tool_summary(name, manifest, config.root))
|
||||
|
||||
return [
|
||||
ToolCategory(name=group, tools=sorted(items, key=lambda t: t.id))
|
||||
for group, items in sorted(by_group.items())
|
||||
]
|
||||
return sorted(
|
||||
[_tool_summary(name, manifest, config.root) for name, manifest in tools.items()],
|
||||
key=lambda t: t.id,
|
||||
)
|
||||
|
||||
|
||||
@router.get("/tools/{name}", response_model=ToolDetail)
|
||||
@@ -112,16 +73,7 @@ def get_tool(name: str) -> ToolDetail:
|
||||
)
|
||||
|
||||
summary = _tool_summary(name, manifest, config.root)
|
||||
docs: str | None = None
|
||||
t = manifest.tool
|
||||
if t.source:
|
||||
md_path = _find_md_for_tool(config.root, t.source, name)
|
||||
if md_path and md_path.exists():
|
||||
docs = _strip_frontmatter(md_path.read_text())
|
||||
if not docs:
|
||||
docs = None
|
||||
|
||||
return ToolDetail(**summary.model_dump(), docs=docs)
|
||||
return ToolDetail(**summary.model_dump())
|
||||
|
||||
|
||||
@router.post("/tools/{name}/install")
|
||||
|
||||
@@ -38,14 +38,14 @@ def castle_root(tmp_path: Path) -> Generator[Path, None, None]:
|
||||
"description": "Test tool",
|
||||
"install": {"path": {"alias": "test-tool"}},
|
||||
"tool": {
|
||||
"source": "tools/document",
|
||||
"source": "test-tool",
|
||||
"system_dependencies": ["pandoc"],
|
||||
},
|
||||
},
|
||||
"test-tool-2": {
|
||||
"description": "Another test tool",
|
||||
"tool": {
|
||||
"source": "tools/utility",
|
||||
"source": "test-tool-2",
|
||||
"version": "2.0.0",
|
||||
},
|
||||
},
|
||||
|
||||
@@ -8,56 +8,51 @@ from fastapi.testclient import TestClient
|
||||
class TestToolsList:
|
||||
"""GET /tools endpoint tests."""
|
||||
|
||||
def test_returns_grouped_tools(self, client: TestClient) -> None:
|
||||
"""Returns tools grouped by source directory name."""
|
||||
def test_returns_flat_list(self, client: TestClient) -> None:
|
||||
"""Returns tools as a flat sorted list."""
|
||||
response = client.get("/tools")
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
names = [cat["name"] for cat in data]
|
||||
assert "document" in names
|
||||
assert "utility" in names
|
||||
assert isinstance(data, list)
|
||||
ids = [t["id"] for t in data]
|
||||
assert "test-tool" in ids
|
||||
assert "test-tool-2" in ids
|
||||
|
||||
def test_groups_sorted(self, client: TestClient) -> None:
|
||||
"""Groups are sorted alphabetically."""
|
||||
def test_sorted_alphabetically(self, client: TestClient) -> None:
|
||||
"""Tools are sorted alphabetically by id."""
|
||||
response = client.get("/tools")
|
||||
data = response.json()
|
||||
names = [cat["name"] for cat in data]
|
||||
assert names == sorted(names)
|
||||
ids = [t["id"] for t in data]
|
||||
assert ids == sorted(ids)
|
||||
|
||||
def test_tool_fields(self, client: TestClient) -> None:
|
||||
"""Tool summary has expected fields."""
|
||||
response = client.get("/tools")
|
||||
data = response.json()
|
||||
doc_group = next(c for c in data if c["name"] == "document")
|
||||
tool = next(t for t in doc_group["tools"] if t["id"] == "test-tool")
|
||||
tool = next(t for t in data if t["id"] == "test-tool")
|
||||
assert tool["description"] == "Test tool"
|
||||
assert tool["source"] == "tools/document"
|
||||
assert tool["source"] == "test-tool"
|
||||
assert tool["system_dependencies"] == ["pandoc"]
|
||||
# tool_type and category should not be present
|
||||
assert "tool_type" not in tool
|
||||
assert "category" not in tool
|
||||
|
||||
def test_installed_flag(self, client: TestClient) -> None:
|
||||
"""Tool with install.path is marked as installed."""
|
||||
response = client.get("/tools")
|
||||
data = response.json()
|
||||
doc_group = next(c for c in data if c["name"] == "document")
|
||||
tool = next(t for t in doc_group["tools"] if t["id"] == "test-tool")
|
||||
tool = next(t for t in data if t["id"] == "test-tool")
|
||||
assert tool["installed"] is True
|
||||
|
||||
def test_not_installed_flag(self, client: TestClient) -> None:
|
||||
"""Tool without install.path is not marked as installed."""
|
||||
response = client.get("/tools")
|
||||
data = response.json()
|
||||
util_group = next(c for c in data if c["name"] == "utility")
|
||||
tool = next(t for t in util_group["tools"] if t["id"] == "test-tool-2")
|
||||
tool = next(t for t in data if t["id"] == "test-tool-2")
|
||||
assert tool["installed"] is False
|
||||
|
||||
def test_service_excluded(self, client: TestClient) -> None:
|
||||
"""Services without tool spec are not listed."""
|
||||
response = client.get("/tools")
|
||||
data = response.json()
|
||||
all_ids = [t["id"] for cat in data for t in cat["tools"]]
|
||||
all_ids = [t["id"] for t in data]
|
||||
assert "test-svc" not in all_ids
|
||||
|
||||
|
||||
@@ -70,27 +65,11 @@ class TestToolDetail:
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["id"] == "test-tool"
|
||||
assert data["source"] == "tools/document"
|
||||
assert data["source"] == "test-tool"
|
||||
assert data["system_dependencies"] == ["pandoc"]
|
||||
|
||||
def test_docs_from_file(self, client: TestClient, castle_root: Path) -> None:
|
||||
"""Reads documentation from .md file."""
|
||||
# Create doc file matching the source lookup (src/<dir_name>/<tool>.md)
|
||||
doc_dir = castle_root / "tools" / "document" / "src" / "document"
|
||||
doc_dir.mkdir(parents=True)
|
||||
(doc_dir / "test_tool.md").write_text(
|
||||
"---\ntitle: Test\n---\n\n# Test Tool\n\nUsage info here."
|
||||
)
|
||||
response = client.get("/tools/test-tool")
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["docs"] is not None
|
||||
assert "# Test Tool" in data["docs"]
|
||||
# Frontmatter should be stripped
|
||||
assert "---" not in data["docs"]
|
||||
|
||||
def test_no_docs(self, client: TestClient) -> None:
|
||||
"""Tool with no doc file returns null docs."""
|
||||
"""Tool detail returns null docs (no .md files anymore)."""
|
||||
response = client.get("/tools/test-tool")
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
|
||||
30
castle.yaml
30
castle.yaml
@@ -120,7 +120,7 @@ components:
|
||||
manage:
|
||||
systemd: {}
|
||||
tool:
|
||||
source: tools/system/
|
||||
source: backup-collect/
|
||||
system_dependencies:
|
||||
- rsync
|
||||
backup-data:
|
||||
@@ -162,14 +162,14 @@ components:
|
||||
path:
|
||||
alias: mbox2eml
|
||||
tool:
|
||||
source: tools/document/
|
||||
source: mbox2eml/
|
||||
android-backup:
|
||||
description: Backup Android device using ADB
|
||||
install:
|
||||
path:
|
||||
alias: android-backup
|
||||
tool:
|
||||
source: tools/android/
|
||||
source: android-backup/
|
||||
system_dependencies:
|
||||
- adb
|
||||
browser:
|
||||
@@ -178,14 +178,14 @@ components:
|
||||
path:
|
||||
alias: browser
|
||||
tool:
|
||||
source: tools/browser/
|
||||
source: browser/
|
||||
docx-extractor:
|
||||
description: Extract content and metadata from Word .docx files
|
||||
install:
|
||||
path:
|
||||
alias: docx-extractor
|
||||
tool:
|
||||
source: tools/search/
|
||||
source: docx-extractor/
|
||||
system_dependencies:
|
||||
- pandoc
|
||||
docx2md:
|
||||
@@ -194,7 +194,7 @@ components:
|
||||
path:
|
||||
alias: docx2md
|
||||
tool:
|
||||
source: tools/document/
|
||||
source: docx2md/
|
||||
system_dependencies:
|
||||
- pandoc
|
||||
gpt:
|
||||
@@ -203,21 +203,21 @@ components:
|
||||
path:
|
||||
alias: gpt
|
||||
tool:
|
||||
source: tools/gpt/
|
||||
source: gpt/
|
||||
html2text:
|
||||
description: Convert HTML content to plain text
|
||||
install:
|
||||
path:
|
||||
alias: html2text
|
||||
tool:
|
||||
source: tools/document/
|
||||
source: html2text/
|
||||
md2pdf:
|
||||
description: Convert Markdown files to PDF
|
||||
install:
|
||||
path:
|
||||
alias: md2pdf
|
||||
tool:
|
||||
source: tools/document/
|
||||
source: md2pdf/
|
||||
system_dependencies:
|
||||
- pandoc
|
||||
- texlive-latex-base
|
||||
@@ -227,21 +227,21 @@ components:
|
||||
path:
|
||||
alias: mdscraper
|
||||
tool:
|
||||
source: tools/mdscraper/
|
||||
source: mdscraper/
|
||||
pdf-extractor:
|
||||
description: Extract content and metadata from PDF files
|
||||
install:
|
||||
path:
|
||||
alias: pdf-extractor
|
||||
tool:
|
||||
source: tools/search/
|
||||
source: pdf-extractor/
|
||||
pdf2md:
|
||||
description: Convert PDF files to Markdown
|
||||
install:
|
||||
path:
|
||||
alias: pdf2md
|
||||
tool:
|
||||
source: tools/document/
|
||||
source: pdf2md/
|
||||
system_dependencies:
|
||||
- pandoc
|
||||
- poppler-utils
|
||||
@@ -251,18 +251,18 @@ components:
|
||||
path:
|
||||
alias: schedule
|
||||
tool:
|
||||
source: tools/system/
|
||||
source: schedule/
|
||||
search:
|
||||
description: Manage self-contained searchable collections of files
|
||||
install:
|
||||
path:
|
||||
alias: search
|
||||
tool:
|
||||
source: tools/search/
|
||||
source: search/
|
||||
text-extractor:
|
||||
description: Extract content and metadata from text files
|
||||
install:
|
||||
path:
|
||||
alias: text-extractor
|
||||
tool:
|
||||
source: tools/search/
|
||||
source: text-extractor/
|
||||
|
||||
@@ -19,7 +19,7 @@ from castle_cli.manifest import (
|
||||
SystemdSpec,
|
||||
ToolSpec,
|
||||
)
|
||||
from castle_cli.templates.scaffold import scaffold_category_tool, scaffold_project
|
||||
from castle_cli.templates.scaffold import scaffold_project
|
||||
|
||||
|
||||
def next_available_port(config: object) -> int:
|
||||
@@ -42,16 +42,11 @@ def run_create(args: argparse.Namespace) -> int:
|
||||
config = load_config()
|
||||
name = args.name
|
||||
proj_type = args.type
|
||||
category = getattr(args, "category", None)
|
||||
|
||||
if name in config.components:
|
||||
print(f"Error: component '{name}' already exists in castle.yaml")
|
||||
return 1
|
||||
|
||||
# Category tool: add to existing category package
|
||||
if proj_type == "tool" and category:
|
||||
return _create_category_tool(config, name, args.description, category)
|
||||
|
||||
project_dir = config.root / name
|
||||
if project_dir.exists():
|
||||
print(f"Error: directory '{name}' already exists")
|
||||
@@ -128,59 +123,3 @@ def run_create(args: argparse.Namespace) -> int:
|
||||
return 0
|
||||
|
||||
|
||||
def _create_category_tool(config: object, name: str, description: str | None, category: str) -> int:
|
||||
"""Create a tool inside an existing category package."""
|
||||
category_dir = config.root / "tools" / category
|
||||
if not category_dir.exists():
|
||||
print(f"Error: category directory 'tools/{category}/' does not exist")
|
||||
print(" Create the category package first, then add tools to it.")
|
||||
return 1
|
||||
|
||||
package_name = name.replace("-", "_")
|
||||
desc = description or "A castle tool"
|
||||
|
||||
# Scaffold the .py and .md files into the category package
|
||||
scaffold_category_tool(
|
||||
category_dir=category_dir,
|
||||
tool_name=name,
|
||||
package_name=package_name,
|
||||
category=category,
|
||||
description=desc,
|
||||
)
|
||||
|
||||
# Append entry point to existing pyproject.toml
|
||||
pyproject_path = category_dir / "pyproject.toml"
|
||||
if pyproject_path.exists():
|
||||
content = pyproject_path.read_text()
|
||||
entry_line = f'{name} = "{category}.{package_name}:main"'
|
||||
if entry_line not in content:
|
||||
# Find [project.scripts] section and append
|
||||
if "[project.scripts]" in content:
|
||||
content = content.replace(
|
||||
"[project.scripts]",
|
||||
f"[project.scripts]\n{entry_line}",
|
||||
)
|
||||
pyproject_path.write_text(content)
|
||||
print(f" Added entry point to tools/{category}/pyproject.toml")
|
||||
else:
|
||||
print(" Warning: could not find [project.scripts] in pyproject.toml")
|
||||
|
||||
# Register in castle.yaml
|
||||
manifest = ComponentManifest(
|
||||
id=name,
|
||||
description=desc,
|
||||
tool=ToolSpec(source=f"tools/{category}/"),
|
||||
install=InstallSpec(path=PathInstallSpec(alias=name)),
|
||||
)
|
||||
config.components[name] = manifest
|
||||
save_config(config)
|
||||
|
||||
print(f"Created tool '{name}' in tools/{category}/")
|
||||
print(f" Source: tools/{category}/src/{category}/{package_name}.py")
|
||||
print(" Registered in castle.yaml")
|
||||
print("\nNext steps:")
|
||||
print(f" Edit tools/{category}/src/{category}/{package_name}.py")
|
||||
print(f" cd tools/{category} && uv sync")
|
||||
print(f" castle tool info {name}")
|
||||
|
||||
return 0
|
||||
|
||||
@@ -3,7 +3,6 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
from pathlib import Path
|
||||
|
||||
from castle_cli.config import load_config
|
||||
|
||||
@@ -28,7 +27,7 @@ def run_tool(args: argparse.Namespace) -> int:
|
||||
|
||||
|
||||
def _tool_list() -> int:
|
||||
"""List tools grouped by category."""
|
||||
"""List all registered tools."""
|
||||
config = load_config()
|
||||
tools = {k: v for k, v in config.components.items() if v.tool}
|
||||
|
||||
@@ -36,24 +35,14 @@ def _tool_list() -> int:
|
||||
print("No tools registered.")
|
||||
return 0
|
||||
|
||||
by_group: dict[str, list[tuple[str, object]]] = {}
|
||||
for name, manifest in tools.items():
|
||||
if manifest.tool and manifest.tool.source:
|
||||
group = Path(manifest.tool.source).name
|
||||
else:
|
||||
group = "standalone"
|
||||
by_group.setdefault(group, []).append((name, manifest))
|
||||
|
||||
for group in sorted(by_group):
|
||||
items = by_group[group]
|
||||
print(f"\n{BOLD}{CYAN}{group}{RESET}")
|
||||
print(f"{CYAN}{'─' * 40}{RESET}")
|
||||
for name, manifest in sorted(items):
|
||||
desc = manifest.description or ""
|
||||
deps = ""
|
||||
if manifest.tool.system_dependencies:
|
||||
deps = f" {DIM}[{', '.join(manifest.tool.system_dependencies)}]{RESET}"
|
||||
print(f" {BOLD}{name:<20}{RESET} {desc}{deps}")
|
||||
print(f"\n{BOLD}{CYAN}Tools{RESET}")
|
||||
print(f"{CYAN}{'─' * 40}{RESET}")
|
||||
for name, manifest in sorted(tools.items()):
|
||||
desc = manifest.description or ""
|
||||
deps = ""
|
||||
if manifest.tool and manifest.tool.system_dependencies:
|
||||
deps = f" {DIM}[{', '.join(manifest.tool.system_dependencies)}]{RESET}"
|
||||
print(f" {BOLD}{name:<20}{RESET} {desc}{deps}")
|
||||
|
||||
print()
|
||||
return 0
|
||||
@@ -83,39 +72,5 @@ def _tool_info(name: str) -> int:
|
||||
if t.system_dependencies:
|
||||
print(f" {BOLD}requires{RESET}: {', '.join(t.system_dependencies)}")
|
||||
|
||||
# Read and display .md documentation if available
|
||||
if t.source:
|
||||
md_path = _find_md_for_tool(config.root, t.source, name)
|
||||
if md_path and md_path.exists():
|
||||
content = md_path.read_text()
|
||||
# Strip YAML frontmatter
|
||||
if content.startswith("---\n"):
|
||||
end = content.find("\n---\n", 4)
|
||||
if end != -1:
|
||||
content = content[end + 5:]
|
||||
content = content.strip()
|
||||
if content:
|
||||
print(f"\n{BOLD}{CYAN}Documentation{RESET}")
|
||||
print(f"{CYAN}{'─' * 40}{RESET}")
|
||||
print(f"{DIM}{content}{RESET}")
|
||||
|
||||
print()
|
||||
return 0
|
||||
|
||||
|
||||
def _find_md_for_tool(
|
||||
root: Path, source: str, tool_name: str,
|
||||
) -> Path | None:
|
||||
"""Find the .md documentation file for a tool source path."""
|
||||
source_path = root / source
|
||||
if source_path.is_file():
|
||||
md = source_path.with_suffix(".md")
|
||||
if md.exists():
|
||||
return md
|
||||
elif source_path.is_dir():
|
||||
py_name = tool_name.replace("-", "_")
|
||||
pkg_name = source_path.name
|
||||
md = source_path / "src" / pkg_name / f"{py_name}.md"
|
||||
if md.exists():
|
||||
return md
|
||||
return None
|
||||
|
||||
@@ -46,11 +46,6 @@ def build_parser() -> argparse.ArgumentParser:
|
||||
create_parser.add_argument(
|
||||
"--port", type=int, help="Port number (services only)"
|
||||
)
|
||||
create_parser.add_argument(
|
||||
"--category", default=None,
|
||||
help="Tool category (e.g. document, search, system)",
|
||||
)
|
||||
|
||||
# castle info
|
||||
info_parser = subparsers.add_parser("info", help="Show component details")
|
||||
info_parser.add_argument("project", help="Component name")
|
||||
|
||||
@@ -5,99 +5,6 @@ from __future__ import annotations
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def scaffold_category_tool(
|
||||
category_dir: Path,
|
||||
tool_name: str,
|
||||
package_name: str,
|
||||
category: str,
|
||||
description: str,
|
||||
) -> None:
|
||||
"""Scaffold a tool inside an existing category package.
|
||||
|
||||
Creates the .py and .md files only — the category package
|
||||
(pyproject.toml, tests, CLAUDE.md) already exists.
|
||||
"""
|
||||
src_dir = category_dir / "src" / category
|
||||
|
||||
# tool .py file
|
||||
_write(
|
||||
src_dir / f"{package_name}.py",
|
||||
f'''#!/usr/bin/env python3
|
||||
"""{tool_name}: {description}
|
||||
|
||||
Usage:
|
||||
{tool_name} [options] [input]
|
||||
cat input.txt | {tool_name}
|
||||
|
||||
Examples:
|
||||
{tool_name} input.txt
|
||||
{tool_name} input.txt -o output.txt
|
||||
cat input.txt | {tool_name} > output.txt
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
|
||||
__all__ = ["main"]
|
||||
|
||||
|
||||
def main() -> int:
|
||||
"""Main entry point."""
|
||||
parser = argparse.ArgumentParser(
|
||||
description="{description}",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
)
|
||||
parser.add_argument("input", nargs="?", help="Input file (default: stdin)")
|
||||
parser.add_argument(
|
||||
"-o", "--output", default=None, help="Output file (default: stdout)"
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.input:
|
||||
with open(args.input) as f:
|
||||
data = f.read()
|
||||
else:
|
||||
data = sys.stdin.read()
|
||||
|
||||
# TODO: implement tool logic
|
||||
result = data
|
||||
|
||||
if args.output:
|
||||
with open(args.output, "w") as f:
|
||||
f.write(result)
|
||||
else:
|
||||
print(result, end="")
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
''',
|
||||
)
|
||||
|
||||
# tool .md documentation
|
||||
_write(
|
||||
src_dir / f"{package_name}.md",
|
||||
f"""---
|
||||
name: {tool_name}
|
||||
category: {category}
|
||||
---
|
||||
|
||||
# {tool_name}
|
||||
|
||||
{description}
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
{tool_name} [options] [input]
|
||||
cat input.txt | {tool_name}
|
||||
```
|
||||
""",
|
||||
)
|
||||
|
||||
|
||||
def scaffold_project(
|
||||
project_dir: Path,
|
||||
name: str,
|
||||
|
||||
@@ -95,7 +95,7 @@ class TestRoleDerivation:
|
||||
"""Component with tool spec gets TOOL role."""
|
||||
m = ComponentManifest(
|
||||
id="docx2md",
|
||||
tool=ToolSpec(source="tools/document/"),
|
||||
tool=ToolSpec(source="docx2md/"),
|
||||
)
|
||||
assert Role.TOOL in m.roles
|
||||
|
||||
|
||||
@@ -132,7 +132,7 @@ Creates a shim so the tool is available system-wide after
|
||||
|
||||
```yaml
|
||||
tool:
|
||||
source: tools/document/ # Source directory
|
||||
source: my-tool/ # Source directory
|
||||
version: "1.0.0"
|
||||
system_dependencies: [pandoc, poppler-utils]
|
||||
```
|
||||
@@ -143,8 +143,7 @@ It's separate from `install` (which handles PATH registration) and `run`
|
||||
|
||||
The install method (uv tool install vs symlink) is inferred from the source
|
||||
directory: if `pyproject.toml` exists, it's a Python package; if the source
|
||||
is a file, it's symlinked. Grouping for `castle tool list` and the dashboard
|
||||
is derived from the source directory name (e.g., `tools/document/` → "document").
|
||||
is a file, it's symlinked.
|
||||
|
||||
### `build` — How to build it
|
||||
|
||||
@@ -210,11 +209,8 @@ A component can have multiple roles. For example, `protonmail` is both a
|
||||
# Service — scaffolds project, assigns port, registers in castle.yaml
|
||||
castle create my-service --type service --description "Does something"
|
||||
|
||||
# Standalone tool — scaffolds at repo root
|
||||
# Tool — scaffolds at repo root
|
||||
castle create my-tool --type tool --description "Does something"
|
||||
|
||||
# Category tool — adds to existing tools/<category>/ package
|
||||
castle create my-tool --type tool --category document --description "Does something"
|
||||
```
|
||||
|
||||
### Manually
|
||||
|
||||
@@ -22,11 +22,9 @@ How to build CLI tools following Unix philosophy.
|
||||
| **Type checking** | pyright (shared `pyrightconfig.json` at repo root) |
|
||||
| **Python** | 3.11+ minimum |
|
||||
|
||||
## Two kinds of tools
|
||||
## Project layout
|
||||
|
||||
### Standalone tools
|
||||
|
||||
Independent projects at the repo root with their own `pyproject.toml`:
|
||||
Each tool is an independent project at the repo root with its own `pyproject.toml`:
|
||||
|
||||
```
|
||||
my-tool/
|
||||
@@ -39,31 +37,10 @@ my-tool/
|
||||
└── CLAUDE.md
|
||||
```
|
||||
|
||||
Examples: `devbox-connect/`, `mboxer/`, `protonmail/`
|
||||
|
||||
### Category tools
|
||||
|
||||
Multiple tools sharing a single package under `tools/<category>/`:
|
||||
|
||||
```
|
||||
tools/document/
|
||||
├── src/document/
|
||||
│ ├── __init__.py
|
||||
│ ├── pdf2md.py # Each tool is a module
|
||||
│ ├── docx2md.py
|
||||
│ ├── html2text.py
|
||||
│ └── md2pdf.py
|
||||
├── tests/
|
||||
├── pyproject.toml # One pyproject with multiple [project.scripts]
|
||||
└── CLAUDE.md
|
||||
```
|
||||
|
||||
Examples: `tools/document/`, `tools/search/`, `tools/system/`
|
||||
Examples: `pdf2md/`, `gpt/`, `protonmail/`
|
||||
|
||||
## Creating a new tool
|
||||
|
||||
### Standalone
|
||||
|
||||
```bash
|
||||
castle create my-tool --type tool --description "Does something"
|
||||
cd my-tool && uv sync
|
||||
@@ -71,20 +48,8 @@ cd my-tool && uv sync
|
||||
|
||||
This scaffolds the project and registers it in `castle.yaml`.
|
||||
|
||||
### Category tool
|
||||
|
||||
```bash
|
||||
castle create my-tool --type tool --category document --description "Does something"
|
||||
cd tools/document && uv sync
|
||||
```
|
||||
|
||||
This adds a `.py` file to the existing category package and updates its
|
||||
`pyproject.toml` entry points.
|
||||
|
||||
## pyproject.toml
|
||||
|
||||
### Standalone tool
|
||||
|
||||
```toml
|
||||
[project]
|
||||
name = "my-tool"
|
||||
@@ -110,27 +75,7 @@ dev = ["pytest>=7.0.0"]
|
||||
known-first-party = ["my_tool"]
|
||||
```
|
||||
|
||||
### Category package
|
||||
|
||||
```toml
|
||||
[project]
|
||||
name = "castle-document"
|
||||
version = "0.1.0"
|
||||
description = "Castle document conversion tools"
|
||||
requires-python = ">=3.11"
|
||||
dependencies = []
|
||||
|
||||
[project.scripts]
|
||||
docx2md = "document.docx2md:main"
|
||||
pdf2md = "document.pdf2md:main"
|
||||
html2text = "document.html2text:main"
|
||||
md2pdf = "document.md2pdf:main"
|
||||
|
||||
[tool.hatch.build.targets.wheel]
|
||||
packages = ["src/document"]
|
||||
```
|
||||
|
||||
After `uv tool install --editable .`, all commands are in PATH.
|
||||
After `uv tool install --editable .`, the command is in PATH.
|
||||
|
||||
## Tool implementation patterns
|
||||
|
||||
@@ -368,8 +313,6 @@ uv run ruff format . # Format
|
||||
|
||||
## Registering in castle.yaml
|
||||
|
||||
Standalone tools:
|
||||
|
||||
```yaml
|
||||
components:
|
||||
my-tool:
|
||||
@@ -381,14 +324,14 @@ components:
|
||||
alias: my-tool
|
||||
```
|
||||
|
||||
Category tools get one entry per tool, all pointing to the same source:
|
||||
Tools with system dependencies declare them in the manifest:
|
||||
|
||||
```yaml
|
||||
components:
|
||||
pdf2md:
|
||||
description: Convert PDF files to Markdown
|
||||
tool:
|
||||
source: tools/document/
|
||||
source: pdf2md/
|
||||
system_dependencies: [pandoc, poppler-utils]
|
||||
install:
|
||||
path:
|
||||
|
||||
16
docx-extractor/pyproject.toml
Normal file
16
docx-extractor/pyproject.toml
Normal file
@@ -0,0 +1,16 @@
|
||||
[project]
|
||||
name = "docx-extractor"
|
||||
version = "0.1.0"
|
||||
description = "Extract content and metadata from Word .docx files"
|
||||
requires-python = ">=3.11"
|
||||
dependencies = []
|
||||
|
||||
[project.scripts]
|
||||
docx-extractor = "docx_extractor.main:main"
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
[tool.hatch.build.targets.wheel]
|
||||
packages = ["src/docx_extractor"]
|
||||
@@ -1,16 +1,16 @@
|
||||
[project]
|
||||
name = "castle-android"
|
||||
name = "docx2md"
|
||||
version = "0.1.0"
|
||||
description = "Castle Android tools"
|
||||
description = "Convert Word .docx files to Markdown"
|
||||
requires-python = ">=3.11"
|
||||
dependencies = []
|
||||
|
||||
[project.scripts]
|
||||
android-backup = "android.android_backup:main"
|
||||
docx2md = "docx2md.main:main"
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
[tool.hatch.build.targets.wheel]
|
||||
packages = ["src/android"]
|
||||
packages = ["src/docx2md"]
|
||||
@@ -1,14 +1,14 @@
|
||||
[project]
|
||||
name = "castle-gpt"
|
||||
name = "gpt"
|
||||
version = "0.1.0"
|
||||
description = "Castle GPT tools"
|
||||
description = "OpenAI text generation utility"
|
||||
requires-python = ">=3.11"
|
||||
dependencies = [
|
||||
"openai>=1.6.0",
|
||||
]
|
||||
|
||||
[project.scripts]
|
||||
gpt = "gpt.gpt:main"
|
||||
gpt = "gpt.main:main"
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
16
html2text/pyproject.toml
Normal file
16
html2text/pyproject.toml
Normal file
@@ -0,0 +1,16 @@
|
||||
[project]
|
||||
name = "html2text"
|
||||
version = "0.1.0"
|
||||
description = "Convert HTML content to plain text"
|
||||
requires-python = ">=3.11"
|
||||
dependencies = []
|
||||
|
||||
[project.scripts]
|
||||
html2text = "html2text.main:main"
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
[tool.hatch.build.targets.wheel]
|
||||
packages = ["src/html2text"]
|
||||
16
mbox2eml/pyproject.toml
Normal file
16
mbox2eml/pyproject.toml
Normal file
@@ -0,0 +1,16 @@
|
||||
[project]
|
||||
name = "mbox2eml"
|
||||
version = "0.1.0"
|
||||
description = "Convert MBOX mailbox files to individual .eml files"
|
||||
requires-python = ">=3.11"
|
||||
dependencies = []
|
||||
|
||||
[project.scripts]
|
||||
mbox2eml = "mbox2eml.main:main"
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
[tool.hatch.build.targets.wheel]
|
||||
packages = ["src/mbox2eml"]
|
||||
0
mbox2eml/src/mbox2eml/__init__.py
Normal file
0
mbox2eml/src/mbox2eml/__init__.py
Normal file
16
md2pdf/pyproject.toml
Normal file
16
md2pdf/pyproject.toml
Normal file
@@ -0,0 +1,16 @@
|
||||
[project]
|
||||
name = "md2pdf"
|
||||
version = "0.1.0"
|
||||
description = "Convert Markdown files to PDF"
|
||||
requires-python = ">=3.11"
|
||||
dependencies = []
|
||||
|
||||
[project.scripts]
|
||||
md2pdf = "md2pdf.main:main"
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
[tool.hatch.build.targets.wheel]
|
||||
packages = ["src/md2pdf"]
|
||||
0
md2pdf/src/md2pdf/__init__.py
Normal file
0
md2pdf/src/md2pdf/__init__.py
Normal file
@@ -1,12 +1,12 @@
|
||||
[project]
|
||||
name = "castle-mdscraper"
|
||||
name = "mdscraper"
|
||||
version = "0.1.0"
|
||||
description = "Castle markdown scraping tools"
|
||||
description = "Combine text files into a single markdown document"
|
||||
requires-python = ">=3.11"
|
||||
dependencies = []
|
||||
|
||||
[project.scripts]
|
||||
mdscraper = "mdscraper.mdscraper:main"
|
||||
mdscraper = "mdscraper.main:main"
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
0
mdscraper/src/mdscraper/__init__.py
Normal file
0
mdscraper/src/mdscraper/__init__.py
Normal file
18
pdf-extractor/pyproject.toml
Normal file
18
pdf-extractor/pyproject.toml
Normal file
@@ -0,0 +1,18 @@
|
||||
[project]
|
||||
name = "pdf-extractor"
|
||||
version = "0.1.0"
|
||||
description = "Extract content and metadata from PDF files"
|
||||
requires-python = ">=3.11"
|
||||
dependencies = [
|
||||
"pymupdf>=1.24.11",
|
||||
]
|
||||
|
||||
[project.scripts]
|
||||
pdf-extractor = "pdf_extractor.main:main"
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
[tool.hatch.build.targets.wheel]
|
||||
packages = ["src/pdf_extractor"]
|
||||
0
pdf-extractor/src/pdf_extractor/__init__.py
Normal file
0
pdf-extractor/src/pdf_extractor/__init__.py
Normal file
16
pdf2md/pyproject.toml
Normal file
16
pdf2md/pyproject.toml
Normal file
@@ -0,0 +1,16 @@
|
||||
[project]
|
||||
name = "pdf2md"
|
||||
version = "0.1.0"
|
||||
description = "Convert PDF files to Markdown"
|
||||
requires-python = ">=3.11"
|
||||
dependencies = []
|
||||
|
||||
[project.scripts]
|
||||
pdf2md = "pdf2md.main:main"
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
[tool.hatch.build.targets.wheel]
|
||||
packages = ["src/pdf2md"]
|
||||
0
pdf2md/src/pdf2md/__init__.py
Normal file
0
pdf2md/src/pdf2md/__init__.py
Normal file
16
schedule/pyproject.toml
Normal file
16
schedule/pyproject.toml
Normal file
@@ -0,0 +1,16 @@
|
||||
[project]
|
||||
name = "schedule"
|
||||
version = "0.1.0"
|
||||
description = "Systemd timer and service management tool"
|
||||
requires-python = ">=3.11"
|
||||
dependencies = []
|
||||
|
||||
[project.scripts]
|
||||
schedule = "schedule.main:main"
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
[tool.hatch.build.targets.wheel]
|
||||
packages = ["src/schedule"]
|
||||
0
schedule/src/schedule/__init__.py
Normal file
0
schedule/src/schedule/__init__.py
Normal file
18
search/pyproject.toml
Normal file
18
search/pyproject.toml
Normal file
@@ -0,0 +1,18 @@
|
||||
[project]
|
||||
name = "search"
|
||||
version = "0.1.0"
|
||||
description = "Manage searchable collections of files"
|
||||
requires-python = ">=3.11"
|
||||
dependencies = [
|
||||
"toml>=0.10.2",
|
||||
]
|
||||
|
||||
[project.scripts]
|
||||
search = "search.main:main"
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
[tool.hatch.build.targets.wheel]
|
||||
packages = ["src/search"]
|
||||
0
search/src/search/__init__.py
Normal file
0
search/src/search/__init__.py
Normal file
16
text-extractor/pyproject.toml
Normal file
16
text-extractor/pyproject.toml
Normal file
@@ -0,0 +1,16 @@
|
||||
[project]
|
||||
name = "text-extractor"
|
||||
version = "0.1.0"
|
||||
description = "Extract content and metadata from text files"
|
||||
requires-python = ">=3.11"
|
||||
dependencies = []
|
||||
|
||||
[project.scripts]
|
||||
text-extractor = "text_extractor.main:main"
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
[tool.hatch.build.targets.wheel]
|
||||
packages = ["src/text_extractor"]
|
||||
0
text-extractor/src/text_extractor/__init__.py
Normal file
0
text-extractor/src/text_extractor/__init__.py
Normal file
141
tools/README.md
141
tools/README.md
@@ -1,141 +0,0 @@
|
||||
# Castle Tools
|
||||
|
||||
CLI utilities managed by the castle platform. Each tool follows Unix philosophy:
|
||||
read from stdin or file arguments, write to stdout, compose via pipes.
|
||||
|
||||
## Installation
|
||||
|
||||
Each category is its own package, installed individually:
|
||||
|
||||
```bash
|
||||
castle sync # installs all category packages
|
||||
```
|
||||
|
||||
Or manually install a single category:
|
||||
|
||||
```bash
|
||||
uv tool install --editable tools/document/
|
||||
```
|
||||
|
||||
## Tools by Category
|
||||
|
||||
### android (`castle-android`)
|
||||
|
||||
| Tool | Description | Requires |
|
||||
|------|-------------|----------|
|
||||
| `android-backup` | Backup Android device using ADB | adb |
|
||||
|
||||
### browser (`castle-browser`)
|
||||
|
||||
| Tool | Description |
|
||||
|------|-------------|
|
||||
| `browser` | Browse the web using natural language via browser-use |
|
||||
|
||||
### document (`castle-document`)
|
||||
|
||||
| Tool | Description | Requires |
|
||||
|------|-------------|----------|
|
||||
| `docx2md` | Convert Word .docx files to Markdown | pandoc |
|
||||
| `html2text` | Convert HTML content to plain text | |
|
||||
| `md2pdf` | Convert Markdown files to PDF | pandoc, texlive-latex-base |
|
||||
| `pdf2md` | Convert PDF files to Markdown | pandoc, poppler-utils |
|
||||
|
||||
### gpt (`castle-gpt`)
|
||||
|
||||
| Tool | Description |
|
||||
|------|-------------|
|
||||
| `gpt` | OpenAI text generation utility |
|
||||
|
||||
### mdscraper (`castle-mdscraper`)
|
||||
|
||||
| Tool | Description |
|
||||
|------|-------------|
|
||||
| `mdscraper` | Combine text files into a single markdown document |
|
||||
|
||||
### search (`castle-search`)
|
||||
|
||||
| Tool | Description | Requires |
|
||||
|------|-------------|----------|
|
||||
| `docx-extractor` | Extract content and metadata from Word .docx files | pandoc |
|
||||
| `pdf-extractor` | Extract content and metadata from PDF files | |
|
||||
| `search` | Manage self-contained searchable collections of files | |
|
||||
| `text-extractor` | Extract content and metadata from text files | |
|
||||
|
||||
### system (`castle-system`)
|
||||
|
||||
| Tool | Description | Requires |
|
||||
|------|-------------|----------|
|
||||
| `backup-collect` | Collect files from various sources into backup directory | rsync |
|
||||
| `schedule` | Manage systemd user timers and scheduled tasks | |
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
tools/
|
||||
├── <category>/
|
||||
│ ├── pyproject.toml # Per-category package
|
||||
│ └── src/<category>/
|
||||
│ ├── __init__.py
|
||||
│ ├── <tool>.py # Implementation
|
||||
│ └── <tool>.md # Documentation (YAML frontmatter + docs)
|
||||
└── README.md
|
||||
```
|
||||
|
||||
Each tool has two files:
|
||||
|
||||
- **`<tool>.py`** -- the implementation, with a `main()` entry point
|
||||
- **`<tool>.md`** -- YAML frontmatter (command, description, version, category,
|
||||
system_dependencies) followed by usage documentation
|
||||
|
||||
## Adding a New Tool
|
||||
|
||||
```bash
|
||||
castle create my-tool --type tool --category document
|
||||
```
|
||||
|
||||
Or manually:
|
||||
|
||||
1. Create `tools/<category>/src/<category>/my_tool.py` with an argparse `main()` function
|
||||
2. Create `tools/<category>/src/<category>/my_tool.md` with YAML frontmatter
|
||||
3. Add the entry point to `tools/<category>/pyproject.toml`:
|
||||
```toml
|
||||
my-tool = "<category>.my_tool:main"
|
||||
```
|
||||
4. Register in `castle.yaml`:
|
||||
```yaml
|
||||
my-tool:
|
||||
description: What it does
|
||||
tool:
|
||||
tool_type: python_standalone
|
||||
category: <category>
|
||||
source: tools/<category>/
|
||||
install:
|
||||
path: { alias: my-tool }
|
||||
```
|
||||
5. Run `castle sync` to install
|
||||
|
||||
## Tool Types
|
||||
|
||||
Castle manages three types of tools:
|
||||
|
||||
| Type | Description | Installation |
|
||||
|------|-------------|-------------|
|
||||
| `python_standalone` | Own pyproject.toml | Individual `uv tool install` per category |
|
||||
| `script` | Bash or binary | Symlinked to `~/.local/bin/` |
|
||||
|
||||
## Conventions
|
||||
|
||||
- Read from stdin or file argument, write to stdout
|
||||
- Error messages and status to stderr
|
||||
- Exit 0 on success, 1 on error
|
||||
- `--help` for usage, `--version` where applicable
|
||||
- Composable via Unix pipes: `pdf2md paper.pdf | gpt "summarize this"`
|
||||
|
||||
## Managing Tools
|
||||
|
||||
```bash
|
||||
castle tool list # All tools grouped by category
|
||||
castle tool info <name> # Tool details + documentation
|
||||
castle list --role tool # Tools in the component listing
|
||||
castle info <name> --json # Full manifest including tool metadata
|
||||
```
|
||||
@@ -1,244 +0,0 @@
|
||||
---
|
||||
command: android-backup
|
||||
script: android/android-backup.py
|
||||
description: Backup Android device using ADB
|
||||
version: 1.0.0
|
||||
category: android
|
||||
system_dependencies:
|
||||
- adb
|
||||
---
|
||||
|
||||
# android-backup
|
||||
|
||||
Backup Android device data using Android Debug Bridge (ADB). Supports backing up photos, videos, documents, app data, SMS, RCS, contacts, and call logs.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **ADB (Android Debug Bridge)** must be installed and in your PATH
|
||||
- **Android device** with USB debugging enabled
|
||||
- **USB connection** between computer and Android device
|
||||
|
||||
### Enable USB Debugging
|
||||
|
||||
1. Go to Settings → About Phone
|
||||
2. Tap "Build Number" 7 times to enable Developer Options
|
||||
3. Go to Settings → Developer Options
|
||||
4. Enable "USB Debugging"
|
||||
5. Connect device via USB and authorize the computer
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
# Install the toolkit
|
||||
cd /data/repos/toolkit
|
||||
make install
|
||||
|
||||
# Verify installation
|
||||
which android-backup
|
||||
|
||||
# Verify ADB is working
|
||||
adb devices
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Usage
|
||||
|
||||
```bash
|
||||
# Backup everything using default configuration
|
||||
android-backup
|
||||
|
||||
# Use custom configuration file
|
||||
android-backup -c custom.json
|
||||
|
||||
# Specify device when multiple devices connected
|
||||
android-backup --device ABC123DEF456
|
||||
```
|
||||
|
||||
### Selective Backups
|
||||
|
||||
```bash
|
||||
# Only backup photos
|
||||
android-backup --photos-only
|
||||
|
||||
# Only backup SMS messages
|
||||
android-backup --sms-only
|
||||
|
||||
# Only backup contacts
|
||||
android-backup --contacts-only
|
||||
|
||||
# Only backup call logs
|
||||
android-backup --call-logs-only
|
||||
|
||||
# Only backup specific app data
|
||||
android-backup --app-data whatsapp
|
||||
android-backup --app-data com.example.app
|
||||
```
|
||||
|
||||
### Options
|
||||
|
||||
```bash
|
||||
android-backup [options]
|
||||
|
||||
Options:
|
||||
-c, --config FILE Use custom configuration file (default: android_backup_config.json)
|
||||
--device SERIAL Specify device serial number when multiple devices connected
|
||||
--photos-only Only backup photos
|
||||
--videos-only Only backup videos
|
||||
--documents-only Only backup documents
|
||||
--sms-only Only backup SMS messages
|
||||
--contacts-only Only backup contacts
|
||||
--call-logs-only Only backup call logs
|
||||
--app-data PACKAGE Only backup specified app data
|
||||
-v, --verbose Enable verbose logging
|
||||
-h, --help Show help message
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
The tool uses a JSON configuration file (default: `android_backup_config.json`) to specify what to backup and where to store it.
|
||||
|
||||
### Example Configuration
|
||||
|
||||
```json
|
||||
{
|
||||
"backup_root": "/data/backups/android",
|
||||
"device_name": "MyPhone",
|
||||
"backup_photos": true,
|
||||
"backup_videos": true,
|
||||
"backup_documents": true,
|
||||
"backup_sms": true,
|
||||
"backup_contacts": true,
|
||||
"backup_call_logs": true,
|
||||
"backup_app_data": ["com.whatsapp", "org.telegram.messenger"],
|
||||
"photo_dirs": ["/sdcard/DCIM", "/sdcard/Pictures"],
|
||||
"video_dirs": ["/sdcard/DCIM", "/sdcard/Movies"],
|
||||
"document_dirs": ["/sdcard/Documents", "/sdcard/Download"]
|
||||
}
|
||||
```
|
||||
|
||||
### Configuration Options
|
||||
|
||||
- `backup_root` - Root directory for backups
|
||||
- `device_name` - Friendly name for the device
|
||||
- `backup_photos` - Enable photo backup (default: true)
|
||||
- `backup_videos` - Enable video backup (default: true)
|
||||
- `backup_documents` - Enable document backup (default: true)
|
||||
- `backup_sms` - Enable SMS backup (default: true)
|
||||
- `backup_contacts` - Enable contacts backup (default: true)
|
||||
- `backup_call_logs` - Enable call logs backup (default: true)
|
||||
- `backup_app_data` - List of app package names to backup
|
||||
- `photo_dirs` - List of directories to search for photos
|
||||
- `video_dirs` - List of directories to search for videos
|
||||
- `document_dirs` - List of directories to search for documents
|
||||
|
||||
## Backup Structure
|
||||
|
||||
Backups are organized by date and type:
|
||||
|
||||
```
|
||||
/data/backups/android/MyPhone/
|
||||
├── 2024-07-17_14-30-00/
|
||||
│ ├── photos/
|
||||
│ ├── videos/
|
||||
│ ├── documents/
|
||||
│ ├── sms/
|
||||
│ ├── contacts/
|
||||
│ ├── call_logs/
|
||||
│ └── app_data/
|
||||
│ ├── com.whatsapp/
|
||||
│ └── org.telegram.messenger/
|
||||
└── 2024-07-18_10-15-00/
|
||||
└── ...
|
||||
```
|
||||
|
||||
## Features
|
||||
|
||||
- **Incremental backups** - Only copies new or changed files
|
||||
- **Multiple device support** - Specify device when multiple connected
|
||||
- **Selective backups** - Choose what to backup
|
||||
- **App data backup** - Backup specific app data
|
||||
- **Organized structure** - Date-stamped backup directories
|
||||
- **Logging** - Detailed logs of backup operations
|
||||
|
||||
## Examples
|
||||
|
||||
### Full Backup
|
||||
|
||||
```bash
|
||||
# Create full backup with default settings
|
||||
android-backup
|
||||
```
|
||||
|
||||
### Daily Automated Backup
|
||||
|
||||
```bash
|
||||
# Set up daily backup at 2 AM using schedule tool
|
||||
schedule add android-backup \
|
||||
--command "android-backup" \
|
||||
--schedule "daily" \
|
||||
--on-calendar "*-*-* 02:00:00" \
|
||||
--description "Daily Android backup"
|
||||
```
|
||||
|
||||
### Backup Multiple Devices
|
||||
|
||||
```bash
|
||||
# List connected devices
|
||||
adb devices
|
||||
|
||||
# Backup specific device
|
||||
android-backup --device ABC123DEF456
|
||||
```
|
||||
|
||||
### Custom Configuration
|
||||
|
||||
```bash
|
||||
# Create custom config
|
||||
cat > my_backup_config.json <<EOF
|
||||
{
|
||||
"backup_root": "/mnt/nas/android_backups",
|
||||
"device_name": "WorkPhone",
|
||||
"backup_photos": true,
|
||||
"backup_videos": false,
|
||||
"backup_app_data": ["com.slack", "com.microsoft.teams"]
|
||||
}
|
||||
EOF
|
||||
|
||||
# Use custom config
|
||||
android-backup -c my_backup_config.json
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Device Not Found
|
||||
|
||||
```bash
|
||||
# Check if device is connected and authorized
|
||||
adb devices
|
||||
|
||||
# If "unauthorized", check phone for authorization dialog
|
||||
# If no devices listed, check USB connection and debugging is enabled
|
||||
```
|
||||
|
||||
### Permission Denied
|
||||
|
||||
Some files may require root access. The tool will log warnings for files it cannot access but will continue with the backup.
|
||||
|
||||
### Multiple Devices
|
||||
|
||||
```bash
|
||||
# List all connected devices
|
||||
adb devices
|
||||
|
||||
# Specify which device to backup
|
||||
android-backup --device SERIAL_NUMBER
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- Requires USB debugging to be enabled on the Android device
|
||||
- Some app data may require root access
|
||||
- Large backups can take significant time
|
||||
- Ensure sufficient disk space for backups
|
||||
- First backup will be slower; subsequent backups are incremental
|
||||
@@ -1,166 +0,0 @@
|
||||
---
|
||||
command: browser
|
||||
script: browser/browser.py
|
||||
description: Browse the web using natural language via browser-use
|
||||
version: 1.0.0
|
||||
category: browser
|
||||
---
|
||||
|
||||
# browser
|
||||
|
||||
Browser automation tool powered by browser-use. Execute browser tasks from the command line using natural language instructions.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **LLM Provider** - Requires API key for OpenAI or Anthropic
|
||||
- **Chromium/Chrome** - Used by the automation framework
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
# Install the toolkit
|
||||
cd /data/repos/toolkit
|
||||
make install
|
||||
|
||||
# Verify installation
|
||||
which browser
|
||||
```
|
||||
|
||||
## Setup
|
||||
|
||||
### API Key Configuration
|
||||
|
||||
Set up your LLM provider API key:
|
||||
|
||||
```bash
|
||||
# For OpenAI
|
||||
export OPENAI_API_KEY=sk-...
|
||||
|
||||
# For Anthropic Claude
|
||||
export ANTHROPIC_API_KEY=sk-ant-...
|
||||
|
||||
# Add to ~/.bashrc or ~/.profile to persist
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Usage
|
||||
|
||||
```bash
|
||||
# Execute a browser task
|
||||
browser "Go to google.com and search for python tutorials"
|
||||
|
||||
# Use specific LLM provider
|
||||
browser --model gpt-4 "Fill out the contact form on example.com"
|
||||
browser --model claude-3-5-sonnet "Take a screenshot of the homepage"
|
||||
```
|
||||
|
||||
### Options
|
||||
|
||||
```bash
|
||||
browser [options] "instruction"
|
||||
|
||||
Options:
|
||||
instruction Natural language instruction for browser task (required)
|
||||
--model MODEL LLM model to use (default: gpt-4o-mini)
|
||||
--headless Run browser in headless mode (no visible window)
|
||||
--timeout SECONDS Maximum time for task execution (default: 300)
|
||||
-h, --help Show help message
|
||||
```
|
||||
|
||||
## Supported Models
|
||||
|
||||
### OpenAI Models
|
||||
- `gpt-4o` - Most capable, best for complex tasks
|
||||
- `gpt-4o-mini` - Fast and affordable (default)
|
||||
- `gpt-4-turbo`
|
||||
- `gpt-4`
|
||||
- `gpt-3.5-turbo`
|
||||
|
||||
### Anthropic Models
|
||||
- `claude-3-5-sonnet-20241022` - Most capable
|
||||
- `claude-3-opus-20240229`
|
||||
- `claude-3-sonnet-20240229`
|
||||
- `claude-3-haiku-20240307`
|
||||
|
||||
## Examples
|
||||
|
||||
### Web Search
|
||||
|
||||
```bash
|
||||
browser "Search Google for 'best python libraries 2024' and summarize the top 3 results"
|
||||
```
|
||||
|
||||
### Form Filling
|
||||
|
||||
```bash
|
||||
browser "Go to example.com/contact and fill in the form with: Name: John Doe, Email: john@example.com, Message: Testing automation"
|
||||
```
|
||||
|
||||
### Screenshots
|
||||
|
||||
```bash
|
||||
browser "Navigate to github.com and take a screenshot"
|
||||
```
|
||||
|
||||
### Data Extraction
|
||||
|
||||
```bash
|
||||
browser "Go to news.ycombinator.com and list the top 5 post titles"
|
||||
```
|
||||
|
||||
### Multi-step Tasks
|
||||
|
||||
```bash
|
||||
browser "Go to amazon.com, search for 'wireless mouse', filter by 4+ stars, and show me the top 3 results with prices"
|
||||
```
|
||||
|
||||
## Features
|
||||
|
||||
- **Natural language control** - Use plain English to automate browser tasks
|
||||
- **AI-powered** - Uses LLMs to understand and execute complex instructions
|
||||
- **Multi-step workflows** - Handle complex sequences of actions
|
||||
- **Screenshot capture** - Can take screenshots as part of tasks
|
||||
- **Form filling** - Automatically fill out forms
|
||||
- **Data extraction** - Extract information from web pages
|
||||
- **Headless mode** - Run without visible browser window
|
||||
|
||||
## Environment Variables
|
||||
|
||||
- `OPENAI_API_KEY` - API key for OpenAI models
|
||||
- `ANTHROPIC_API_KEY` - API key for Anthropic Claude models
|
||||
|
||||
## Notes
|
||||
|
||||
- Tasks are executed using browser automation via browser-use
|
||||
- Complex tasks may take longer to complete
|
||||
- Headless mode is faster but you won't see the browser actions
|
||||
- Some websites may block automation (check robots.txt)
|
||||
- API costs apply based on model usage
|
||||
- Default timeout is 5 minutes; adjust with `--timeout` for longer tasks
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### API Key Not Found
|
||||
|
||||
```bash
|
||||
# Ensure API key is exported
|
||||
echo $OPENAI_API_KEY
|
||||
# or
|
||||
echo $ANTHROPIC_API_KEY
|
||||
|
||||
# Add to shell config if missing
|
||||
export OPENAI_API_KEY=sk-...
|
||||
```
|
||||
|
||||
### Browser Launch Fails
|
||||
|
||||
Ensure Chromium or Chrome is installed on your system. The browser-use library will try to find it automatically.
|
||||
|
||||
### Timeout Errors
|
||||
|
||||
For complex tasks, increase the timeout:
|
||||
|
||||
```bash
|
||||
browser --timeout 600 "complex long-running task"
|
||||
```
|
||||
@@ -1,20 +0,0 @@
|
||||
[project]
|
||||
name = "castle-document"
|
||||
version = "0.1.0"
|
||||
description = "Castle document conversion tools"
|
||||
requires-python = ">=3.11"
|
||||
dependencies = []
|
||||
|
||||
[project.scripts]
|
||||
docx2md = "document.docx2md:main"
|
||||
pdf2md = "document.pdf2md:main"
|
||||
html2text = "document.html2text:main"
|
||||
md2pdf = "document.md2pdf:main"
|
||||
mbox2eml = "document.mbox2eml:main"
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
[tool.hatch.build.targets.wheel]
|
||||
packages = ["src/document"]
|
||||
@@ -1,68 +0,0 @@
|
||||
---
|
||||
command: docx2md
|
||||
script: document/docx2md.py
|
||||
description: Convert Word .docx files to Markdown
|
||||
version: 1.0.0
|
||||
category: document
|
||||
system_dependencies:
|
||||
- pandoc
|
||||
---
|
||||
|
||||
# docx2md
|
||||
|
||||
Convert Microsoft Word (.docx) documents to Markdown format.
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
cd /data/repos/toolkit
|
||||
make install
|
||||
which docx2md
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
# Convert single file
|
||||
docx2md document.docx
|
||||
|
||||
# Output to specific file
|
||||
docx2md document.docx -o output.md
|
||||
|
||||
# Convert from stdin
|
||||
cat document.docx | docx2md > output.md
|
||||
```
|
||||
|
||||
### Options
|
||||
|
||||
```bash
|
||||
docx2md input.docx [-o output.md]
|
||||
|
||||
Options:
|
||||
input.docx Input Word document
|
||||
-o, --output FILE Output markdown file (default: stdout)
|
||||
-h, --help Show help message
|
||||
```
|
||||
|
||||
## Features
|
||||
|
||||
- Preserves document structure (headings, lists, tables)
|
||||
- Converts formatting (bold, italic, links)
|
||||
- Handles images (extracts and references)
|
||||
- Supports tables
|
||||
- Maintains document hierarchy
|
||||
|
||||
## Examples
|
||||
|
||||
```bash
|
||||
# Convert and save
|
||||
docx2md report.docx -o report.md
|
||||
|
||||
# Convert multiple files
|
||||
for file in *.docx; do
|
||||
docx2md "$file" -o "${file%.docx}.md"
|
||||
done
|
||||
|
||||
# Pipe to other tools
|
||||
docx2md document.docx | grep "important"
|
||||
```
|
||||
@@ -1,69 +0,0 @@
|
||||
---
|
||||
command: html2text
|
||||
script: document/html2text.py
|
||||
description: Convert HTML content to plain text
|
||||
version: 1.0.0
|
||||
category: document
|
||||
---
|
||||
|
||||
# html2text (document)
|
||||
|
||||
Convert HTML files to readable plain text, removing tags while preserving structure.
|
||||
|
||||
Note: This is the same tool as in the email category, available for document conversion tasks.
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
cd /data/repos/toolkit
|
||||
make install
|
||||
which html2text
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
# Convert HTML file
|
||||
html2text page.html
|
||||
|
||||
# Convert from stdin
|
||||
cat page.html | html2text
|
||||
|
||||
# Save to file
|
||||
html2text page.html > page.txt
|
||||
```
|
||||
|
||||
### Options
|
||||
|
||||
```bash
|
||||
html2text [file]
|
||||
|
||||
Options:
|
||||
file HTML file to convert (optional, reads from stdin if not provided)
|
||||
-h, --help Show help message
|
||||
```
|
||||
|
||||
## Features
|
||||
|
||||
- Removes HTML tags
|
||||
- Preserves text structure
|
||||
- Formats lists and paragraphs
|
||||
- Converts links to readable format
|
||||
- Terminal-friendly output
|
||||
|
||||
## Examples
|
||||
|
||||
```bash
|
||||
# Convert web page
|
||||
curl https://example.com | html2text
|
||||
|
||||
# Convert multiple files
|
||||
for file in *.html; do
|
||||
html2text "$file" > "${file%.html}.txt"
|
||||
done
|
||||
|
||||
# Extract article text
|
||||
html2text article.html | less
|
||||
```
|
||||
|
||||
See also: [email/html2text](../email/html2text.md) for email-specific usage.
|
||||
@@ -1,25 +0,0 @@
|
||||
# mbox2eml
|
||||
|
||||
Convert MBOX mailbox files to individual .eml files with clean, descriptive
|
||||
filenames based on date, sender, and recipient.
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
mbox2eml mailbox.mbox # Export to eml_output/
|
||||
mbox2eml mailbox.mbox -o emails/ # Export to custom directory
|
||||
mbox2eml mailbox.mbox --max-length 80 # Truncate filenames at 80 chars
|
||||
```
|
||||
|
||||
Output filenames look like:
|
||||
|
||||
```
|
||||
2025-03-23_15-09-42_from_alice_example_com_to_bob_example_com.eml
|
||||
```
|
||||
|
||||
## What can you do with .eml files?
|
||||
|
||||
- Open in email clients (Thunderbird, Outlook, etc.)
|
||||
- Parse and analyze with Python or CLI tools
|
||||
- Extract metadata, body, or attachments
|
||||
- Archive and search with grep, ripgrep, etc.
|
||||
@@ -1,76 +0,0 @@
|
||||
---
|
||||
command: md2pdf
|
||||
script: document/md2pdf.py
|
||||
description: Convert Markdown files to PDF
|
||||
version: 1.0.0
|
||||
category: document
|
||||
system_dependencies:
|
||||
- pandoc
|
||||
- texlive-latex-base
|
||||
---
|
||||
|
||||
# md2pdf
|
||||
|
||||
Convert Markdown documents to PDF format.
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
cd /data/repos/toolkit
|
||||
make install
|
||||
which md2pdf
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
# Convert single file
|
||||
md2pdf document.md
|
||||
|
||||
# Output to specific file
|
||||
md2pdf document.md -o output.pdf
|
||||
|
||||
# Convert from stdin
|
||||
cat document.md | md2pdf > output.pdf
|
||||
```
|
||||
|
||||
### Options
|
||||
|
||||
```bash
|
||||
md2pdf input.md [-o output.pdf]
|
||||
|
||||
Options:
|
||||
input.md Input Markdown document
|
||||
-o, --output FILE Output PDF file (default: stdout)
|
||||
-h, --help Show help message
|
||||
```
|
||||
|
||||
## Features
|
||||
|
||||
- Converts Markdown to formatted PDF
|
||||
- Preserves headings, lists, and formatting
|
||||
- Supports code blocks
|
||||
- Handles links
|
||||
- Clean, readable PDF output
|
||||
|
||||
## Examples
|
||||
|
||||
```bash
|
||||
# Convert and save
|
||||
md2pdf README.md -o README.pdf
|
||||
|
||||
# Convert multiple files
|
||||
for file in *.md; do
|
||||
md2pdf "$file" -o "${file%.md}.pdf"
|
||||
done
|
||||
|
||||
# Create PDF from concatenated markdown
|
||||
cat intro.md body.md conclusion.md | md2pdf -o complete.pdf
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- Supports standard Markdown syntax
|
||||
- Some advanced Markdown features may not render
|
||||
- PDF styling is fixed (no custom themes)
|
||||
- Images referenced in Markdown should be accessible
|
||||
@@ -1,76 +0,0 @@
|
||||
---
|
||||
command: pdf2md
|
||||
script: document/pdf2md.py
|
||||
description: Convert PDF files to Markdown
|
||||
version: 1.0.0
|
||||
category: document
|
||||
system_dependencies:
|
||||
- pandoc
|
||||
- poppler-utils
|
||||
---
|
||||
|
||||
# pdf2md
|
||||
|
||||
Convert PDF documents to Markdown format with text extraction and formatting preservation.
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
cd /data/repos/toolkit
|
||||
make install
|
||||
which pdf2md
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
# Convert single PDF
|
||||
pdf2md document.pdf
|
||||
|
||||
# Output to specific file
|
||||
pdf2md document.pdf -o output.md
|
||||
|
||||
# Convert from stdin
|
||||
cat document.pdf | pdf2md > output.md
|
||||
```
|
||||
|
||||
### Options
|
||||
|
||||
```bash
|
||||
pdf2md input.pdf [-o output.md]
|
||||
|
||||
Options:
|
||||
input.pdf Input PDF document
|
||||
-o, --output FILE Output markdown file (default: stdout)
|
||||
-h, --help Show help message
|
||||
```
|
||||
|
||||
## Features
|
||||
|
||||
- Extracts text from PDFs
|
||||
- Preserves basic formatting where possible
|
||||
- Handles multi-page documents
|
||||
- Works with text-based PDFs
|
||||
- Pipe-friendly output
|
||||
|
||||
## Examples
|
||||
|
||||
```bash
|
||||
# Convert and save
|
||||
pdf2md report.pdf -o report.md
|
||||
|
||||
# Convert multiple PDFs
|
||||
for file in *.pdf; do
|
||||
pdf2md "$file" -o "${file%.pdf}.md"
|
||||
done
|
||||
|
||||
# Extract and search
|
||||
pdf2md document.pdf | grep "keyword"
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- Works best with text-based PDFs
|
||||
- Scanned PDFs (images) require OCR
|
||||
- Complex layouts may lose some formatting
|
||||
- Tables are converted to plain text
|
||||
@@ -1,157 +0,0 @@
|
||||
---
|
||||
command: gpt
|
||||
script: gpt/gpt.py
|
||||
description: A simple OpenAI text generation utility
|
||||
version: 1.0.0
|
||||
category: gpt
|
||||
---
|
||||
|
||||
# gpt
|
||||
|
||||
Interact with OpenAI's GPT models from the command line. Send prompts and receive AI-generated responses.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- OpenAI API key
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
cd /data/repos/toolkit
|
||||
make install
|
||||
which gpt
|
||||
```
|
||||
|
||||
## Setup
|
||||
|
||||
```bash
|
||||
# Set API key
|
||||
export OPENAI_API_KEY=sk-...
|
||||
|
||||
# Add to ~/.bashrc or ~/.profile to persist
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Usage
|
||||
|
||||
```bash
|
||||
# Send a prompt
|
||||
gpt "Explain quantum computing in simple terms"
|
||||
|
||||
# Use specific model
|
||||
gpt --model gpt-4 "Write a Python function to sort a list"
|
||||
|
||||
# Read prompt from file
|
||||
gpt < prompt.txt
|
||||
|
||||
# Interactive mode
|
||||
echo "What is the capital of France?" | gpt
|
||||
```
|
||||
|
||||
### Options
|
||||
|
||||
```bash
|
||||
gpt [options] "prompt"
|
||||
|
||||
Options:
|
||||
prompt Text prompt for GPT (required)
|
||||
--model MODEL GPT model to use (default: gpt-4o-mini)
|
||||
--temperature TEMP Creativity level 0.0-2.0 (default: 0.7)
|
||||
--max-tokens NUM Maximum response length (default: 1000)
|
||||
--system PROMPT System prompt to set behavior
|
||||
-h, --help Show help message
|
||||
```
|
||||
|
||||
## Supported Models
|
||||
|
||||
- `gpt-4o` - Most capable, best for complex tasks
|
||||
- `gpt-4o-mini` - Fast and affordable (default)
|
||||
- `gpt-4-turbo` - Large context window
|
||||
- `gpt-4` - High capability
|
||||
- `gpt-3.5-turbo` - Fast and economical
|
||||
|
||||
## Examples
|
||||
|
||||
### Simple Questions
|
||||
|
||||
```bash
|
||||
gpt "What is the weather like today?"
|
||||
gpt "Explain recursion"
|
||||
```
|
||||
|
||||
### Code Generation
|
||||
|
||||
```bash
|
||||
gpt "Write a Python function to calculate fibonacci numbers"
|
||||
gpt --model gpt-4 "Create a React component for a todo list"
|
||||
```
|
||||
|
||||
### Text Processing
|
||||
|
||||
```bash
|
||||
# Summarize a file
|
||||
cat article.txt | gpt "Summarize this article in 3 bullet points"
|
||||
|
||||
# Translate
|
||||
echo "Hello world" | gpt "Translate to French"
|
||||
|
||||
# Code review
|
||||
cat script.py | gpt "Review this code and suggest improvements"
|
||||
```
|
||||
|
||||
### With System Prompts
|
||||
|
||||
```bash
|
||||
gpt --system "You are a Python expert" "How do I use asyncio?"
|
||||
gpt --system "Respond in haiku form" "Describe AI"
|
||||
```
|
||||
|
||||
### Batch Processing
|
||||
|
||||
```bash
|
||||
# Process multiple prompts
|
||||
for question in "What is AI?" "What is ML?" "What is DL?"; do
|
||||
gpt "$question" >> answers.txt
|
||||
done
|
||||
```
|
||||
|
||||
## Features
|
||||
|
||||
- **Multiple models** - Choose from various GPT models
|
||||
- **Flexible input** - Prompt as argument or from stdin
|
||||
- **Customizable** - Adjust temperature, tokens, system prompts
|
||||
- **Pipe-friendly** - Works in Unix pipelines
|
||||
- **Fast** - Direct API access
|
||||
|
||||
## Environment Variables
|
||||
|
||||
- `OPENAI_API_KEY` - Your OpenAI API key (required)
|
||||
|
||||
## Tips
|
||||
|
||||
- Use `gpt-4o-mini` for fast, cheap responses (default)
|
||||
- Use `gpt-4` or `gpt-4o` for complex reasoning
|
||||
- Lower temperature (0.1-0.3) for factual responses
|
||||
- Higher temperature (0.8-1.5) for creative writing
|
||||
- System prompts help control tone and expertise level
|
||||
|
||||
## Cost Considerations
|
||||
|
||||
- `gpt-4o-mini` is most economical for general use
|
||||
- `gpt-4` costs more but provides higher quality
|
||||
- Limit max-tokens to control costs
|
||||
- Monitor usage in OpenAI dashboard
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### API Key Not Found
|
||||
|
||||
```bash
|
||||
echo $OPENAI_API_KEY # Should show your key
|
||||
export OPENAI_API_KEY=sk-...
|
||||
```
|
||||
|
||||
### Rate Limits
|
||||
|
||||
If you hit rate limits, wait a moment and retry, or upgrade your OpenAI plan.
|
||||
@@ -1,142 +0,0 @@
|
||||
---
|
||||
command: mdscraper
|
||||
script: mdscraper/mdscraper.py
|
||||
description: Combine text files from a directory into a single markdown document with
|
||||
.gitignore-style filtering
|
||||
version: 1.0.0
|
||||
category: mdscraper
|
||||
---
|
||||
|
||||
# mdscraper
|
||||
|
||||
Convert web pages to clean Markdown format. Scrapes web content and converts to readable Markdown, removing ads and clutter.
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
cd /data/repos/toolkit
|
||||
make install
|
||||
which mdscraper
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Usage
|
||||
|
||||
```bash
|
||||
# Convert web page to Markdown
|
||||
mdscraper https://example.com
|
||||
|
||||
# Save to file
|
||||
mdscraper https://example.com > article.md
|
||||
|
||||
# Multiple URLs
|
||||
mdscraper https://site1.com https://site2.com
|
||||
```
|
||||
|
||||
### Options
|
||||
|
||||
```bash
|
||||
mdscraper [options] URL [URL...]
|
||||
|
||||
Options:
|
||||
URL Web page URL to scrape (required)
|
||||
-o, --output FILE Output file (default: stdout)
|
||||
--user-agent UA Custom user agent string
|
||||
-h, --help Show help message
|
||||
```
|
||||
|
||||
## Features
|
||||
|
||||
- **Clean conversion** - Removes ads, navigation, footers
|
||||
- **Markdown output** - Well-formatted Markdown
|
||||
- **Main content extraction** - Focuses on article content
|
||||
- **Link preservation** - Maintains links in Markdown format
|
||||
- **Image handling** - Includes images with Markdown syntax
|
||||
- **Fast** - Efficient scraping and conversion
|
||||
|
||||
## Examples
|
||||
|
||||
### Save Article
|
||||
|
||||
```bash
|
||||
mdscraper https://blog.example.com/article > article.md
|
||||
```
|
||||
|
||||
### Scrape Multiple Pages
|
||||
|
||||
```bash
|
||||
# Scrape all pages
|
||||
for url in $(cat urls.txt); do
|
||||
mdscraper "$url" > "$(echo $url | md5sum | cut -d' ' -f1).md"
|
||||
done
|
||||
```
|
||||
|
||||
### Convert for Reading
|
||||
|
||||
```bash
|
||||
# Scrape and view
|
||||
mdscraper https://news.site/article | less
|
||||
|
||||
# Scrape and convert to PDF
|
||||
mdscraper https://article.com | md2pdf -o article.pdf
|
||||
```
|
||||
|
||||
### With Custom User Agent
|
||||
|
||||
```bash
|
||||
mdscraper --user-agent "MyBot/1.0" https://example.com
|
||||
```
|
||||
|
||||
## Use Cases
|
||||
|
||||
- **Article archiving** - Save web articles as Markdown
|
||||
- **Content migration** - Convert web content for static sites
|
||||
- **Offline reading** - Download articles for later
|
||||
- **Documentation** - Scrape docs to local Markdown
|
||||
- **Research** - Collect and organize web content
|
||||
|
||||
## Output Format
|
||||
|
||||
Markdown output includes:
|
||||
- Article title as H1
|
||||
- Headings at appropriate levels
|
||||
- Paragraphs with proper spacing
|
||||
- Lists (ordered and unordered)
|
||||
- Links in `[text](url)` format
|
||||
- Images in `` format
|
||||
- Code blocks when present
|
||||
|
||||
## Notes
|
||||
|
||||
- Respects robots.txt where possible
|
||||
- Some sites may block scraping
|
||||
- JavaScript-heavy sites may not work well
|
||||
- Rate limit requests to avoid overwhelming servers
|
||||
- Consider caching results for repeated access
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Site Blocks Requests
|
||||
|
||||
Try using a custom user agent:
|
||||
```bash
|
||||
mdscraper --user-agent "Mozilla/5.0..." https://site.com
|
||||
```
|
||||
|
||||
### JavaScript Content Missing
|
||||
|
||||
For JavaScript-heavy sites, consider using the `browser` tool instead:
|
||||
```bash
|
||||
browser "Go to URL and extract the article text"
|
||||
```
|
||||
|
||||
### Rate Limiting
|
||||
|
||||
Add delays between requests:
|
||||
```bash
|
||||
for url in $(cat urls.txt); do
|
||||
mdscraper "$url" > file.md
|
||||
sleep 2
|
||||
done
|
||||
```
|
||||
@@ -1,22 +0,0 @@
|
||||
[project]
|
||||
name = "castle-search"
|
||||
version = "0.1.0"
|
||||
description = "Castle search and indexing tools"
|
||||
requires-python = ">=3.11"
|
||||
dependencies = [
|
||||
"toml>=0.10.2",
|
||||
"pymupdf>=1.24.11",
|
||||
]
|
||||
|
||||
[project.scripts]
|
||||
search = "search.search:main"
|
||||
pdf-extractor = "search.pdf_extractor:main"
|
||||
docx-extractor = "search.docx_extractor:main"
|
||||
text-extractor = "search.text_extractor:main"
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
[tool.hatch.build.targets.wheel]
|
||||
packages = ["src/search"]
|
||||
@@ -1,73 +0,0 @@
|
||||
---
|
||||
command: docx-extractor
|
||||
script: search/docx_extractor.py
|
||||
description: Extract content and metadata from Word .docx files
|
||||
version: 1.0.0
|
||||
category: search
|
||||
system_dependencies:
|
||||
- pandoc
|
||||
---
|
||||
|
||||
# docx-extractor
|
||||
|
||||
Extract text from Microsoft Word (.docx) documents.
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
cd /data/repos/toolkit
|
||||
make install
|
||||
which docx-extractor
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
# Extract from Word doc
|
||||
docx-extractor document.docx
|
||||
|
||||
# Save to file
|
||||
docx-extractor document.docx > text.txt
|
||||
|
||||
# Multiple documents
|
||||
docx-extractor file1.docx file2.docx
|
||||
```
|
||||
|
||||
### Options
|
||||
|
||||
```bash
|
||||
docx-extractor [files...]
|
||||
|
||||
Options:
|
||||
files DOCX files to extract text from
|
||||
-h, --help Show help message
|
||||
```
|
||||
|
||||
## Features
|
||||
|
||||
- Extract text from .docx files
|
||||
- Handles headers, footers, and body text
|
||||
- Preserves basic structure
|
||||
- Fast extraction
|
||||
|
||||
## Examples
|
||||
|
||||
```bash
|
||||
# Single document
|
||||
docx-extractor report.docx
|
||||
|
||||
# Batch extraction
|
||||
for file in *.docx; do
|
||||
docx-extractor "$file" > "${file%.docx}.txt"
|
||||
done
|
||||
|
||||
# Search extracted text
|
||||
docx-extractor document.docx | grep "keyword"
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- Only supports .docx format (not .doc)
|
||||
- Formatting is stripped
|
||||
- Tables become plain text
|
||||
- Images are not extracted
|
||||
@@ -1,70 +0,0 @@
|
||||
---
|
||||
command: pdf-extractor
|
||||
script: search/pdf_extractor.py
|
||||
description: Extract content and metadata from PDF files
|
||||
version: 1.0.0
|
||||
category: search
|
||||
---
|
||||
|
||||
# pdf-extractor
|
||||
|
||||
Extract text from PDF documents.
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
cd /data/repos/toolkit
|
||||
make install
|
||||
which pdf-extractor
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
# Extract from PDF
|
||||
pdf-extractor document.pdf
|
||||
|
||||
# Save to file
|
||||
pdf-extractor document.pdf > text.txt
|
||||
|
||||
# Multiple PDFs
|
||||
pdf-extractor file1.pdf file2.pdf
|
||||
```
|
||||
|
||||
### Options
|
||||
|
||||
```bash
|
||||
pdf-extractor [files...]
|
||||
|
||||
Options:
|
||||
files PDF files to extract text from
|
||||
-h, --help Show help message
|
||||
```
|
||||
|
||||
## Features
|
||||
|
||||
- Extract text from PDFs
|
||||
- Handles multi-page documents
|
||||
- Preserves text layout where possible
|
||||
- Fast extraction
|
||||
|
||||
## Examples
|
||||
|
||||
```bash
|
||||
# Single PDF
|
||||
pdf-extractor report.pdf
|
||||
|
||||
# Batch extraction
|
||||
for file in *.pdf; do
|
||||
pdf-extractor "$file" > "${file%.pdf}.txt"
|
||||
done
|
||||
|
||||
# Search extracted text
|
||||
pdf-extractor document.pdf | grep "keyword"
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- Works with text-based PDFs
|
||||
- Scanned PDFs (images) require OCR
|
||||
- Complex layouts may affect text order
|
||||
@@ -1,123 +0,0 @@
|
||||
---
|
||||
command: search
|
||||
script: search/search.py
|
||||
description: Manage self-contained searchable collections of files
|
||||
version: 1.0.0
|
||||
category: search
|
||||
---
|
||||
|
||||
# search
|
||||
|
||||
Fast full-text search using Tantivy search index. Index and search documents with blazing speed.
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
cd /data/repos/toolkit
|
||||
make install
|
||||
which search
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### Index Documents
|
||||
|
||||
```bash
|
||||
# Index a directory
|
||||
search index /path/to/documents
|
||||
|
||||
# Index specific files
|
||||
search index file1.txt file2.pdf file3.docx
|
||||
```
|
||||
|
||||
### Search
|
||||
|
||||
```bash
|
||||
# Simple search
|
||||
search query "search terms"
|
||||
|
||||
# Search with field
|
||||
search query --field title "document title"
|
||||
|
||||
# Limit results
|
||||
search query --limit 10 "search terms"
|
||||
```
|
||||
|
||||
### Options
|
||||
|
||||
```bash
|
||||
search index [paths...] Index files or directories
|
||||
search query [options] QUERY Search indexed documents
|
||||
|
||||
Query options:
|
||||
QUERY Search query (required)
|
||||
--field FIELD Search specific field (title, body, path)
|
||||
--limit N Maximum results to return (default: 20)
|
||||
-h, --help Show help message
|
||||
```
|
||||
|
||||
## Features
|
||||
|
||||
- **Fast searching** - Powered by Tantivy search engine
|
||||
- **Multiple formats** - Supports PDF, DOCX, TXT, and more
|
||||
- **Full-text search** - Search entire document contents
|
||||
- **Field search** - Search specific fields
|
||||
- **Relevance ranking** - Results sorted by relevance
|
||||
|
||||
## Examples
|
||||
|
||||
### Index and Search
|
||||
|
||||
```bash
|
||||
# Index your documents
|
||||
search index ~/Documents
|
||||
|
||||
# Search for terms
|
||||
search query "important meeting notes"
|
||||
search query "project proposal 2024"
|
||||
```
|
||||
|
||||
### Field-Specific Search
|
||||
|
||||
```bash
|
||||
# Search in title only
|
||||
search query --field title "Annual Report"
|
||||
|
||||
# Search in body
|
||||
search query --field body "quarterly results"
|
||||
```
|
||||
|
||||
### Batch Operations
|
||||
|
||||
```bash
|
||||
# Index multiple directories
|
||||
search index ~/Documents ~/Downloads ~/Desktop
|
||||
|
||||
# Incremental indexing
|
||||
search index ~/Documents/new_files
|
||||
```
|
||||
|
||||
## Supported File Types
|
||||
|
||||
- **Text files** - .txt, .md, .log
|
||||
- **PDF documents** - .pdf
|
||||
- **Word documents** - .docx
|
||||
- **Other formats** - via text extractors
|
||||
|
||||
## Index Location
|
||||
|
||||
Index is stored in `~/.local/share/search/index` by default.
|
||||
|
||||
## Tips
|
||||
|
||||
- Index documents once, search many times
|
||||
- Re-index when documents change
|
||||
- Use quotes for exact phrases
|
||||
- Field searches are faster for specific needs
|
||||
|
||||
## Notes
|
||||
|
||||
- Initial indexing may take time for large collections
|
||||
- Index size depends on document collection size
|
||||
- Supports incremental indexing
|
||||
- UTF-8 text encoding recommended
|
||||
@@ -1,71 +0,0 @@
|
||||
---
|
||||
command: text-extractor
|
||||
script: search/text_extractor.py
|
||||
description: Extract content and metadata from text files
|
||||
version: 1.0.0
|
||||
category: search
|
||||
---
|
||||
|
||||
# text-extractor
|
||||
|
||||
Extract plain text from various document formats.
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
cd /data/repos/toolkit
|
||||
make install
|
||||
which text-extractor
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
# Extract from file
|
||||
text-extractor document.txt
|
||||
|
||||
# Extract to file
|
||||
text-extractor document.txt > output.txt
|
||||
|
||||
# Process multiple files
|
||||
text-extractor file1.txt file2.txt file3.txt
|
||||
```
|
||||
|
||||
### Options
|
||||
|
||||
```bash
|
||||
text-extractor [files...]
|
||||
|
||||
Options:
|
||||
files Files to extract text from
|
||||
-h, --help Show help message
|
||||
```
|
||||
|
||||
## Supported Formats
|
||||
|
||||
- Plain text (.txt)
|
||||
- Markdown (.md)
|
||||
- Log files (.log)
|
||||
- Other text-based formats
|
||||
|
||||
## Features
|
||||
|
||||
- Fast text extraction
|
||||
- Handles multiple files
|
||||
- Preserves text encoding
|
||||
- Pipe-friendly output
|
||||
|
||||
## Examples
|
||||
|
||||
```bash
|
||||
# Extract single file
|
||||
text-extractor document.txt
|
||||
|
||||
# Batch processing
|
||||
for file in *.txt; do
|
||||
text-extractor "$file" > "${file%.txt}_extracted.txt"
|
||||
done
|
||||
|
||||
# Use in pipeline
|
||||
text-extractor doc.txt | grep "keyword"
|
||||
```
|
||||
@@ -1,137 +0,0 @@
|
||||
---
|
||||
command: backup-collect
|
||||
script: system/backup-collect.py
|
||||
description: Collect files from various sources into backup directory
|
||||
version: 1.0.0
|
||||
category: system
|
||||
system_dependencies:
|
||||
- rsync
|
||||
---
|
||||
|
||||
# backup-collect
|
||||
|
||||
Collect system information and configuration files for backup purposes.
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
cd /data/repos/toolkit
|
||||
make install
|
||||
which backup-collect
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
# Collect system information
|
||||
backup-collect
|
||||
|
||||
# Specify output directory
|
||||
backup-collect -o /path/to/backup
|
||||
|
||||
# Collect specific categories
|
||||
backup-collect --category config
|
||||
backup-collect --category packages
|
||||
```
|
||||
|
||||
### Options
|
||||
|
||||
```bash
|
||||
backup-collect [options]
|
||||
|
||||
Options:
|
||||
-o, --output DIR Output directory for collected data
|
||||
--category CAT Collect specific category (config, packages, system)
|
||||
-v, --verbose Enable verbose output
|
||||
-h, --help Show help message
|
||||
```
|
||||
|
||||
## What It Collects
|
||||
|
||||
### System Information
|
||||
- OS version and details
|
||||
- Kernel information
|
||||
- Hardware details
|
||||
- System uptime
|
||||
|
||||
### Configuration Files
|
||||
- User dotfiles (.bashrc, .vimrc, etc.)
|
||||
- Application configs
|
||||
- SSH keys (with permissions preserved)
|
||||
- Custom scripts
|
||||
|
||||
### Package Lists
|
||||
- Installed system packages
|
||||
- Python packages
|
||||
- npm packages
|
||||
- Other package managers
|
||||
|
||||
## Features
|
||||
|
||||
- **Comprehensive** - Collects essential system data
|
||||
- **Organized** - Structured output directory
|
||||
- **Safe** - Preserves permissions
|
||||
- **Selective** - Choose what to collect
|
||||
- **Portable** - Outputs in standard formats
|
||||
|
||||
## Examples
|
||||
|
||||
### Full Backup
|
||||
|
||||
```bash
|
||||
backup-collect -o ~/backups/system-$(date +%Y%m%d)
|
||||
```
|
||||
|
||||
### Scheduled Backup
|
||||
|
||||
```bash
|
||||
# Weekly system backup
|
||||
schedule add system-backup \
|
||||
--command "backup-collect -o /mnt/backups/system-\$(date +\%Y\%m\%d)" \
|
||||
--schedule "weekly" \
|
||||
--on-calendar "Sun 02:00:00"
|
||||
```
|
||||
|
||||
### Selective Collection
|
||||
|
||||
```bash
|
||||
# Only collect configs
|
||||
backup-collect --category config -o ~/configs
|
||||
|
||||
# Only package lists
|
||||
backup-collect --category packages -o ~/packages
|
||||
```
|
||||
|
||||
## Output Structure
|
||||
|
||||
```
|
||||
backup-directory/
|
||||
├── system/
|
||||
│ ├── os-release
|
||||
│ ├── kernel-version
|
||||
│ └── hardware-info
|
||||
├── config/
|
||||
│ ├── dotfiles/
|
||||
│ ├── ssh/
|
||||
│ └── app-configs/
|
||||
└── packages/
|
||||
├── apt-packages.txt
|
||||
├── pip-packages.txt
|
||||
└── npm-packages.txt
|
||||
```
|
||||
|
||||
## Use Cases
|
||||
|
||||
- **System migration** - Document current setup
|
||||
- **Disaster recovery** - Quick system restoration
|
||||
- **Configuration management** - Track system changes
|
||||
- **Compliance** - Document installed software
|
||||
- **Development** - Share development environment setup
|
||||
|
||||
## Notes
|
||||
|
||||
- Does not collect sensitive data by default
|
||||
- SSH keys are collected but not passwords
|
||||
- Large files are skipped
|
||||
- Runs without root (collects what's accessible)
|
||||
- Safe to run frequently
|
||||
@@ -1,388 +0,0 @@
|
||||
# schedule
|
||||
|
||||
A tool to easily create, manage, and monitor systemd user timers and services following best practices.
|
||||
|
||||
## Overview
|
||||
|
||||
`schedule` simplifies the management of systemd user timers by providing a simple command-line interface to create timer/service pairs that execute bash commands on a schedule. It handles all the boilerplate configuration and follows systemd best practices for security and resource management.
|
||||
|
||||
## Installation
|
||||
|
||||
Install the toolkit:
|
||||
|
||||
```bash
|
||||
cd /data/repos/toolkit
|
||||
make install
|
||||
```
|
||||
|
||||
The `schedule` command will be available in your PATH.
|
||||
|
||||
### Enable Linger (Optional but Recommended)
|
||||
|
||||
By default, systemd user units only run while you're logged in. To allow timers to run even when you're logged out, enable linger:
|
||||
|
||||
```bash
|
||||
loginctl enable-linger $USER
|
||||
```
|
||||
|
||||
This is essential for timers that should run 24/7, like automated sync jobs.
|
||||
|
||||
## Features
|
||||
|
||||
- **Easy timer creation** - Create timers with a single command
|
||||
- **Multiple schedule types** - Support for interval-based and calendar-based schedules
|
||||
- **Environment variables** - Support for environment files and inline variables
|
||||
- **Pre-conditions** - Run commands only when conditions are met
|
||||
- **Security hardening** - Automatic security settings (NoNewPrivileges, PrivateTmp)
|
||||
- **Log management** - Easy access to service logs via journalctl
|
||||
- **Status monitoring** - Check timer and service status
|
||||
|
||||
## Usage
|
||||
|
||||
### List timers
|
||||
|
||||
Show all active user timers:
|
||||
|
||||
```bash
|
||||
schedule list
|
||||
```
|
||||
|
||||
Show all timers including inactive ones:
|
||||
|
||||
```bash
|
||||
schedule list --all
|
||||
```
|
||||
|
||||
### Add a timer
|
||||
|
||||
Basic timer that runs every 5 minutes:
|
||||
|
||||
```bash
|
||||
schedule add my-task \
|
||||
--command "echo 'Hello, World!'" \
|
||||
--schedule "5min"
|
||||
```
|
||||
|
||||
Daily backup at 2 AM:
|
||||
|
||||
```bash
|
||||
schedule add daily-backup \
|
||||
--command "backup run" \
|
||||
--schedule "daily" \
|
||||
--on-calendar "*-*-* 02:00:00" \
|
||||
--description "Daily backup job"
|
||||
```
|
||||
|
||||
ProtonMail sync with environment file and condition:
|
||||
|
||||
```bash
|
||||
schedule add protonmail-sync \
|
||||
--command "protonmail sync" \
|
||||
--schedule "5min" \
|
||||
--description "Sync ProtonMail emails" \
|
||||
--env-file ~/.config/protonmail/env \
|
||||
--condition-command "pgrep -f protonmail-bridge"
|
||||
```
|
||||
|
||||
With environment variables:
|
||||
|
||||
```bash
|
||||
schedule add my-task \
|
||||
--command "my-script.sh" \
|
||||
--schedule "1hour" \
|
||||
--environment "API_KEY=secret123" \
|
||||
--environment "LOG_LEVEL=debug" \
|
||||
--working-directory "/home/user/projects"
|
||||
```
|
||||
|
||||
### Check status
|
||||
|
||||
View timer and service status:
|
||||
|
||||
```bash
|
||||
schedule status protonmail-sync
|
||||
```
|
||||
|
||||
This shows:
|
||||
- Timer status (active/inactive)
|
||||
- Service status (last run, next run)
|
||||
- Next scheduled execution time
|
||||
|
||||
### View logs
|
||||
|
||||
Follow logs in real-time:
|
||||
|
||||
```bash
|
||||
schedule logs protonmail-sync --follow
|
||||
```
|
||||
|
||||
Show last 50 lines:
|
||||
|
||||
```bash
|
||||
schedule logs protonmail-sync --lines 50
|
||||
```
|
||||
|
||||
Show logs since today:
|
||||
|
||||
```bash
|
||||
schedule logs protonmail-sync --since today
|
||||
```
|
||||
|
||||
### Start/Stop timers
|
||||
|
||||
Start a timer:
|
||||
|
||||
```bash
|
||||
schedule start protonmail-sync
|
||||
```
|
||||
|
||||
Stop a timer:
|
||||
|
||||
```bash
|
||||
schedule stop protonmail-sync
|
||||
```
|
||||
|
||||
### Enable/Disable timers
|
||||
|
||||
Enable timer (start on boot):
|
||||
|
||||
```bash
|
||||
schedule enable protonmail-sync
|
||||
```
|
||||
|
||||
Disable timer (don't start on boot):
|
||||
|
||||
```bash
|
||||
schedule disable protonmail-sync
|
||||
```
|
||||
|
||||
### Remove a timer
|
||||
|
||||
Remove timer and service files:
|
||||
|
||||
```bash
|
||||
schedule remove protonmail-sync
|
||||
```
|
||||
|
||||
Remove but keep logs:
|
||||
|
||||
```bash
|
||||
schedule remove protonmail-sync --keep-logs
|
||||
```
|
||||
|
||||
## Schedule Formats
|
||||
|
||||
### Interval-based schedules
|
||||
|
||||
Use simple time units:
|
||||
|
||||
- `5min` - Every 5 minutes
|
||||
- `1hour` or `1h` - Every hour
|
||||
- `30min` - Every 30 minutes
|
||||
- `1day` or `1d` - Every day
|
||||
- `1week` or `1w` - Every week
|
||||
- `hourly` - Every hour (alias for 1hour)
|
||||
- `daily` - Every day (alias for 1day)
|
||||
|
||||
### Calendar-based schedules
|
||||
|
||||
Use systemd calendar syntax with `--on-calendar`:
|
||||
|
||||
- `*-*-* 02:00:00` - Every day at 2 AM
|
||||
- `Mon *-*-* 09:00:00` - Every Monday at 9 AM
|
||||
- `*-*-01 00:00:00` - First day of each month at midnight
|
||||
- `*-*-* *:00:00` - Every hour on the hour
|
||||
|
||||
## Advanced Options
|
||||
|
||||
### Add command options
|
||||
|
||||
- `--name` - Name of the timer/service (required)
|
||||
- `--command` - Bash command to execute (required)
|
||||
- `--schedule` - Schedule format (required)
|
||||
- `--description` - Human-readable description
|
||||
- `--env-file` - Path to environment file to source
|
||||
- `--environment` / `-e` - Set environment variable (KEY=VALUE)
|
||||
- `--condition-command` - Command that must succeed before running
|
||||
- `--working-directory` - Working directory for the command
|
||||
- `--on-boot-sec` - Delay after boot (default: 1min)
|
||||
- `--on-calendar` - Override with calendar-based schedule
|
||||
- `--persistent` - Run missed timers if system was offline
|
||||
- `--force` - Overwrite existing timer/service
|
||||
- `--no-enable` - Don't enable the timer
|
||||
- `--no-start` - Don't start the timer
|
||||
|
||||
## Examples
|
||||
|
||||
### ProtonMail Sync
|
||||
|
||||
```bash
|
||||
schedule add protonmail-sync \
|
||||
--command "protonmail sync" \
|
||||
--schedule "5min" \
|
||||
--description "Sync ProtonMail emails every 5 minutes" \
|
||||
--env-file ~/.config/protonmail/env \
|
||||
--condition-command "pgrep -f protonmail-bridge"
|
||||
```
|
||||
|
||||
### Daily Backup
|
||||
|
||||
```bash
|
||||
schedule add daily-backup \
|
||||
--command "backup run --full" \
|
||||
--schedule "daily" \
|
||||
--on-calendar "*-*-* 02:00:00" \
|
||||
--description "Daily backup at 2 AM" \
|
||||
--environment "BACKUP_DIR=/mnt/backups"
|
||||
```
|
||||
|
||||
### Hourly Data Sync
|
||||
|
||||
```bash
|
||||
schedule add data-sync \
|
||||
--command "rsync -av /source/ /dest/" \
|
||||
--schedule "hourly" \
|
||||
--description "Sync data every hour" \
|
||||
--working-directory "/home/user/sync"
|
||||
```
|
||||
|
||||
### Git Auto-Commit
|
||||
|
||||
```bash
|
||||
schedule add git-autocommit \
|
||||
--command "git add -A && git commit -m 'Auto-commit' && git push" \
|
||||
--schedule "15min" \
|
||||
--description "Auto-commit and push changes" \
|
||||
--working-directory "/home/user/projects/myrepo"
|
||||
```
|
||||
|
||||
## File Locations
|
||||
|
||||
Timer and service files are stored in:
|
||||
|
||||
```
|
||||
~/.config/systemd/user/
|
||||
├── <name>.service
|
||||
└── <name>.timer
|
||||
```
|
||||
|
||||
## Security Features
|
||||
|
||||
All services created by `schedule` include:
|
||||
|
||||
- `NoNewPrivileges=yes` - Cannot gain new privileges
|
||||
- `PrivateTmp=yes` - Isolated /tmp directory
|
||||
- Proper PATH configuration
|
||||
- Journal logging for stdout/stderr
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Timer not running
|
||||
|
||||
Check if the timer is enabled and active:
|
||||
|
||||
```bash
|
||||
schedule status <name>
|
||||
```
|
||||
|
||||
### Service failing
|
||||
|
||||
View the logs to see what went wrong:
|
||||
|
||||
```bash
|
||||
schedule logs <name> --lines 50
|
||||
```
|
||||
|
||||
### Command not found
|
||||
|
||||
Ensure the command is in the PATH or use an absolute path:
|
||||
|
||||
```bash
|
||||
schedule add my-task \
|
||||
--command "/usr/local/bin/my-script" \
|
||||
--schedule "5min"
|
||||
```
|
||||
|
||||
### Environment variables not working
|
||||
|
||||
Use an environment file:
|
||||
|
||||
```bash
|
||||
# Create env file
|
||||
echo "API_KEY=secret" > ~/.config/myapp/env
|
||||
chmod 600 ~/.config/myapp/env
|
||||
|
||||
# Use it
|
||||
schedule add my-task \
|
||||
--command "my-script" \
|
||||
--schedule "5min" \
|
||||
--env-file ~/.config/myapp/env
|
||||
```
|
||||
|
||||
## Comparison with Manual Setup
|
||||
|
||||
Instead of manually creating:
|
||||
|
||||
1. `.service` file with proper configuration
|
||||
2. `.timer` file with scheduling
|
||||
3. Running `systemctl --user daemon-reload`
|
||||
4. Running `systemctl --user enable <name>.timer`
|
||||
5. Running `systemctl --user start <name>.timer`
|
||||
|
||||
You can do it all with one command:
|
||||
|
||||
```bash
|
||||
schedule add <name> --command "<cmd>" --schedule "<schedule>"
|
||||
```
|
||||
|
||||
## See Also
|
||||
|
||||
- `man systemd.timer` - systemd timer documentation
|
||||
- `man systemd.service` - systemd service documentation
|
||||
- `man systemd.time` - systemd time specification
|
||||
- `journalctl` - Query systemd journal
|
||||
|
||||
## Important Notes
|
||||
|
||||
### Environment Variables
|
||||
|
||||
**Critical**: Systemd user units do NOT inherit environment variables from your shell (e.g., ~/.bashrc). Variables set with `export` in your shell will not be available to services.
|
||||
|
||||
To make environment variables available to your scheduled commands, pass them inline when creating the timer:
|
||||
|
||||
```bash
|
||||
schedule add my-task \
|
||||
--command "my-script" \
|
||||
--schedule "1hour" \
|
||||
--environment "API_KEY=secret123" \
|
||||
--environment "DATABASE_URL=postgres://localhost/mydb"
|
||||
```
|
||||
|
||||
You can also use the `-e` shorthand:
|
||||
|
||||
```bash
|
||||
schedule add my-task \
|
||||
--command "my-script" \
|
||||
--schedule "1hour" \
|
||||
-e "API_KEY=secret123" \
|
||||
-e "DATABASE_URL=postgres://localhost/mydb"
|
||||
```
|
||||
|
||||
### User vs System Units
|
||||
|
||||
This tool creates **user** systemd units (`systemctl --user`), not system units. User units:
|
||||
|
||||
- Run as your user (no root required)
|
||||
- Stored in `~/.config/systemd/user/`
|
||||
- Only run while you're logged in (unless linger is enabled)
|
||||
- Have access to your user's files and permissions
|
||||
|
||||
To enable timers to run 24/7 even when logged out:
|
||||
|
||||
```bash
|
||||
loginctl enable-linger $USER
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
Same license as the toolkit project.
|
||||
Reference in New Issue
Block a user