html2rss is a Ruby gem that generates RSS 2.0 feeds from websites by scraping HTML or JSON content with CSS selectors or auto-detection.

This gem is the core of the html2rss-web application.

Most people looking for a first working feed should start with html2rss-web, run it with Docker, and open one of the included feeds from their own instance before moving to custom configs or the gem APIs.

Documentation

Detailed usage guides, reference docs, and the feed directory live on the project website:

• Ruby gem documentation
• Request strategies (auto = faraday → botasaurus; or pin concrete strategies)
• Selectors & pagination
• Web application
• Feed directory
• Contributing guide
• GitHub Discussions
• Sponsor on GitHub

Cloud development: Open in GitHub Codespaces (also covered in the installation guide).

Architecture

1. Config — loads and validates configuration (YAML/hash); schema via html2rss schema / schema/html2rss-config.schema.json
2. RequestService — fetches pages (faraday, botasaurus, or local_file)
3. Selectors — extracts content via CSS selectors with extractors/post-processors
4. AutoSource — auto-detects content (Schema.org, JSON state, semantic HTML, structural patterns)
5. FeedBuilder — assembles Article objects and renders feeds (RSS 2.0 / JSON Feed 1.1)

Config -> Request -> Extraction -> Processing -> Building -> Output

Capture API

The Html2rss.capture method analyzes any URL and produces a reusable feed config hash with derived CSS selectors. Use it to speed up writing feed configuration files.

config = Html2rss.capture('https://example.com/articles')
File.write('my-feed.yml', YAML.dump(Html2rss::HashUtil.deep_stringify_keys(config)))

The CLI alias html2rss capture prints the generated config as YAML to stdout. See docs/capture.md for detailed documentation.

MCP Server

html2rss ships with an MCP server that exposes gem capabilities as AI-consumable tools, resources, and prompts:

# Start with stdio transport (default; for Cursor/Claude Desktop)
html2rss mcp

# Start with HTTP transport (binds 127.0.0.1 only — local use)
html2rss mcp --transport http --port 8080

HTTP transport needs rack, rackup, and webrick (declared gem dependencies). It listens on 127.0.0.1 only; do not expose it on a public interface without your own auth and Host/Origin controls.

Strategy note: MCP tool strategy: "auto" collapses to faraday (no FeedPipeline botasaurus fallback). If results are empty or JS-gated, retry with strategy: "botasaurus" and BOTASAURUS_SCRAPER_URL set.

Tools

Name	When to use
scrape_url	One-shot articles now (no saved config)
inspect_url	Diagnose weak scrape/capture (scrapers/SST/segments)
capture_config	Derive a durable feed config (+ quality _meta)
validate_config	Schema-check a config before apply (isError on failure)
apply_config	Run a validated config → RSS XML

Resources

URI	Description
html2rss://schema	Full JSON Schema for feed configurations
html2rss://extractors	Registered extractor names (options live in schema $defs)
html2rss://strategies	Registered request strategy names

Prompts

Name	Description
scrape-webpage	Guided scrape → inspect/retry with botasaurus if needed
capture-feed-config	Guided capture → validate → optional apply

The MCP module (Html2rss::MCP) lazy-loads the mcp gem — no cost when the server is not running.

Botasaurus scrape API (Docker)

Start the Botasaurus scrape API for JavaScript-rendered pages (this compose file is not the MCP server):

docker compose -f docker-compose.botasaurus.yml up -d

Set BOTASAURUS_SCRAPER_URL to http://127.0.0.1:4010 and use strategy botasaurus in MCP tools, Capture, or the CLI.

Request Strategies

Strategy	Description
auto	Tries faraday, falls back to botasaurus (default in gem/CLI FeedPipeline)
faraday	Plain HTTP requests via Faraday
botasaurus	Puppeteer-backed scraping for JavaScript pages

MCP tools intentionally collapse auto → faraday (see MCP section above). Elsewhere, strategy can be set via CLI (--strategy), gem API keyword argument, or feed config request.strategy. See the request strategies docs for more details.

License

This project is licensed under the MIT License — see the LICENSE file for details.