wild-cloud/wild-pc
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: Add new tools and configurations for Castle platform

Browse Source 
- Introduced `uv.lock` for dependency management with various packages including `pytest`, `colorama`, and `pluggy`.
- Added `pyrightconfig.json` for Python type checking configuration.
- Expanded `recommendations.md` with detailed scaling recommendations and project management strategies.
- Created shared `ruff.toml` for consistent linting across projects.
- Developed `Castle Tools` with various utilities including Android backup, browser automation, document conversion, and search tools.
- Implemented `backup-collect` and `schedule` tools for system administration tasks.
- Enhanced `search` functionality with indexing and querying capabilities using Tantivy.
- Added comprehensive documentation for each tool, including usage examples and installation instructions.
This commit is contained in:
Paul Payne
2026-02-20 16:41:19 -08:00
parent 0d35ac9ffd
commit f39a551aad
152 changed files with 21197 additions and 83 deletions
Download Patch File Download Diff File Expand all files Collapse all files

16
tools/mdscraper/pyproject.toml Normal file
View File

@@ -0,0 +1,16 @@
[project]
name = "castle-mdscraper"
version = "0.1.0"
description = "Castle markdown scraping tools"
requires-python = ">=3.11"
dependencies = []
[project.scripts]
mdscraper = "mdscraper.mdscraper:main"
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[tool.hatch.build.targets.wheel]
packages = ["src/mdscraper"]

0
tools/mdscraper/src/mdscraper/__init__.py Normal file
View File

142
tools/mdscraper/src/mdscraper/mdscraper.md Normal file
View File

@@ -0,0 +1,142 @@
---
command: mdscraper
script: mdscraper/mdscraper.py
description: Combine text files from a directory into a single markdown document with
.gitignore-style filtering
version: 1.0.0
category: mdscraper
---
# mdscraper
Convert web pages to clean Markdown format. Scrapes web content and converts to readable Markdown, removing ads and clutter.
## Installation
```bash
cd /data/repos/toolkit
make install
which mdscraper
```
## Usage
### Basic Usage
```bash
# Convert web page to Markdown
mdscraper https://example.com
# Save to file
mdscraper https://example.com > article.md
# Multiple URLs
mdscraper https://site1.com https://site2.com
```
### Options
```bash
mdscraper [options] URL [URL...]
Options:
URL Web page URL to scrape (required)
-o, --output FILE Output file (default: stdout)
--user-agent UA Custom user agent string
-h, --help Show help message
```
## Features
- **Clean conversion** - Removes ads, navigation, footers
- **Markdown output** - Well-formatted Markdown
- **Main content extraction** - Focuses on article content
- **Link preservation** - Maintains links in Markdown format
- **Image handling** - Includes images with Markdown syntax
- **Fast** - Efficient scraping and conversion
## Examples
### Save Article
```bash
mdscraper https://blog.example.com/article > article.md
```
### Scrape Multiple Pages
```bash
# Scrape all pages
for url in $(cat urls.txt); do
mdscraper "$url" > "$(echo $url | md5sum | cut -d' ' -f1).md"
done
```
### Convert for Reading
```bash
# Scrape and view
mdscraper https://news.site/article | less
# Scrape and convert to PDF
mdscraper https://article.com | md2pdf -o article.pdf
```
### With Custom User Agent
```bash
mdscraper --user-agent "MyBot/1.0" https://example.com
```
## Use Cases
- **Article archiving** - Save web articles as Markdown
- **Content migration** - Convert web content for static sites
- **Offline reading** - Download articles for later
- **Documentation** - Scrape docs to local Markdown
- **Research** - Collect and organize web content
## Output Format
Markdown output includes:
- Article title as H1
- Headings at appropriate levels
- Paragraphs with proper spacing
- Lists (ordered and unordered)
- Links in `[text](url)` format
- Images in `![alt](url)` format
- Code blocks when present
## Notes
- Respects robots.txt where possible
- Some sites may block scraping
- JavaScript-heavy sites may not work well
- Rate limit requests to avoid overwhelming servers
- Consider caching results for repeated access
## Troubleshooting
### Site Blocks Requests
Try using a custom user agent:
```bash
mdscraper --user-agent "Mozilla/5.0..." https://site.com
```
### JavaScript Content Missing
For JavaScript-heavy sites, consider using the `browser` tool instead:
```bash
browser "Go to URL and extract the article text"
```
### Rate Limiting
Add delays between requests:
```bash
for url in $(cat urls.txt); do
mdscraper "$url" > file.md
sleep 2
done
```

259
tools/mdscraper/src/mdscraper/mdscraper.py Executable file
View File

@@ -0,0 +1,259 @@
#!/usr/bin/env python3
"""
mdscraper: Combine text files into a single markdown document
Scans a directory for text files and combines them into a single markdown document.
Supports filtering through a .gitignore-style configuration file.
Usage: mdscraper [options] [directory]
mdscraper -o output.md [directory]
mdscraper --ignore-file .mdscraper [directory]
Options:
-o, --output FILE Write markdown to FILE instead of stdout
-i, --ignore-file FILE Use specified ignore file instead of .mdscraper
-r, --recursive Recursively process subdirectories
--include-path Include file paths as headers in the output
--no-headers Don't add file names as headers
--toc Add table of contents at the beginning
"""
import sys
import os
import argparse
import fnmatch
import re
def parse_ignore_file(ignore_file_path):
"""Parse a .gitignore style file into a list of patterns."""
if not os.path.exists(ignore_file_path):
return []
patterns = []
with open(ignore_file_path, "r") as f:
for line in f:
line = line.strip()
# Skip empty lines and comments
if not line or line.startswith("#"):
continue
patterns.append(line)
return patterns
def should_ignore(path, patterns):
"""Determine if a file should be ignored based on patterns."""
# Always ignore hidden files
if os.path.basename(path).startswith("."):
return True
# Check for directory patterns - if path contains a directory that should be ignored
if "/" in path:
path_parts = path.split("/")
for i in range(len(path_parts)):
partial_path = "/".join(path_parts[: i + 1]) + "/"
for pattern in patterns:
if (
not pattern.startswith("!")
and pattern.endswith("/")
and fnmatch.fnmatch(partial_path, pattern)
):
# Check if there's a negation pattern that overrides
negation_overrides = False
for neg_pattern in patterns:
if neg_pattern.startswith("!") and fnmatch.fnmatch(
path, neg_pattern[1:]
):
negation_overrides = True
break
if not negation_overrides:
return True
# Check if path matches any negation pattern first
for pattern in patterns:
if pattern.startswith("!"):
if fnmatch.fnmatch(path, pattern[1:]):
# If it matches a negation pattern, do not ignore
return False
# Then check if it matches any regular pattern
for pattern in patterns:
if not pattern.startswith("!") and not pattern.endswith("/"):
if fnmatch.fnmatch(path, pattern):
return True
# Default: don't ignore
return False
def scan_directory(directory, patterns, recursive=False):
"""Scan a directory for text files, filtering by ignore patterns."""
files = []
for item in os.listdir(directory):
path = os.path.join(directory, item)
rel_path = os.path.relpath(path, directory)
if should_ignore(rel_path, patterns):
continue
if os.path.isfile(path):
# Only include text files
try:
with open(path, "r", encoding="utf-8") as f:
f.read(1024) # Try to read a bit to check if it's text
files.append(path)
except UnicodeDecodeError:
# Skip binary files
continue
elif os.path.isdir(path) and recursive:
# For directories, check if the directory itself should be ignored
dir_path = rel_path + "/"
if any(
pattern.endswith("/") and fnmatch.fnmatch(dir_path, pattern)
for pattern in patterns
if not pattern.startswith("!")
):
# Directory matches an ignore pattern, skip it
continue
# Process subdirectories recursively if requested
subdir_files = scan_directory(path, patterns, recursive)
files.extend(subdir_files)
# Sort files for consistent output
return sorted(files)
def extract_file_content(file_path):
"""Extract content from a file."""
try:
with open(file_path, "r", encoding="utf-8") as f:
return f.read()
except UnicodeDecodeError:
return f"ERROR: Could not decode {file_path} as UTF-8"
def create_toc(files, base_dir):
"""Create a table of contents from the list of files."""
toc = ["# Table of Contents\n"]
for file_path in files:
rel_path = os.path.relpath(file_path, base_dir)
file_name = os.path.basename(file_path)
# Create a GitHub-style anchor link
anchor = file_name.lower().replace(" ", "-")
anchor = re.sub(r"[^\w-]", "", anchor)
toc.append(f"- [{rel_path}](#{anchor})")
return "\n".join(toc) + "\n\n"
def main():
parser = argparse.ArgumentParser(
description="Combine text files into a single markdown document"
)
parser.add_argument(
"directory",
nargs="?",
default=".",
help="Directory to process (default: current directory)",
)
parser.add_argument("-o", "--output", help="Output markdown file (default: stdout)")
parser.add_argument(
"-i",
"--ignore-file",
default=".mdscraper",
help="Gitignore-style file with patterns to ignore (default: .mdscraper)",
)
parser.add_argument(
"-r",
"--recursive",
action="store_true",
help="Recursively process subdirectories",
)
parser.add_argument(
"--include-path", action="store_true", help="Include file paths as headers"
)
parser.add_argument(
"--no-headers", action="store_true", help="Don't add file names as headers"
)
parser.add_argument("--toc", action="store_true", help="Add table of contents")
parser.add_argument("--version", action="version", version="mdscraper 1.0.0")
args = parser.parse_args()
# Resolve the directory path
directory = os.path.abspath(args.directory)
if not os.path.isdir(directory):
print(f"Error: {directory} is not a directory", file=sys.stderr)
return 1
# Load ignore patterns
ignore_file = args.ignore_file
if not os.path.isabs(ignore_file):
# First check if ignore file exists in specified directory
dir_ignore_file = os.path.join(directory, ignore_file)
if os.path.exists(dir_ignore_file):
ignore_file = dir_ignore_file
ignore_patterns = parse_ignore_file(ignore_file)
# Always ignore the output file if specified
if args.output:
output_basename = os.path.basename(args.output)
ignore_patterns.append(output_basename)
# Scan for files
files = scan_directory(directory, ignore_patterns, args.recursive)
if not files:
print(f"Warning: No text files found in {directory}", file=sys.stderr)
return 0
# Build the markdown content
content = []
# Add table of contents if requested
if args.toc:
content.append(create_toc(files, directory))
# Process each file
for file_path in files:
rel_path = os.path.relpath(file_path, directory)
file_name = os.path.basename(file_path)
# Add header unless disabled
if not args.no_headers:
if args.include_path:
content.append(f"# {rel_path}\n")
else:
content.append(f"# {file_name}\n")
# Add file content
file_content = extract_file_content(file_path)
content.append(file_content)
# Add separator between files
content.append("\n---\n")
# Remove the last separator if it exists
if content and content[-1] == "\n---\n":
content.pop()
# Join all content
markdown = "\n".join(content)
# Output the result
if args.output:
with open(args.output, "w", encoding="utf-8") as f:
f.write(markdown)
print(f"Created '{args.output}'", file=sys.stderr)
else:
print(markdown)
return 0
if __name__ == "__main__":
sys.exit(main())