A SearXNG-style metadata hunt motor written successful Rust. Fans retired queries to aggregate hunt engines concurrently, scrapes their HTML results, deduplicates by normalized URL, and ranks utilizing Reciprocal Rank Fusion (RRF).
- A hunt petition arrives astatine GET /search?q=<query>
- The query is sent concurrently to DuckDuckGo, Brave, Startpage, and Yahoo via reqwest
- Each motor parses the HTML consequence pinch scraper (CSS selectors complete Mozilla's html5ever)
- Results are deduplicated by normalized URL (tracking params stripped, locale prefixes removed, query params sorted)
- Duplicate URLs are merged and scored pinch RRF (score = Σ 1/(60 + rank) crossed engines) — pages returned by aggregate engines rank higher
- The apical results are returned arsenic JSON
- Rust 1.75+
- Cargo
cargo adhd metadata-search-engine-rs
Or adhd manually to Cargo.toml:
[dependencies] metadata-search-engine-rs = "0.1"
As a server (from source)
git clone https://github.com/MikeLuu99/searxng-rust cd metadata-search-engine-rs
cargo build --release
PORT=8080 MAX_RESULTS=20 cargo tally --release
Enable debug logging:
Add to your Cargo.toml:
[dependencies] metadata-search-engine-rs = "0.1" tokio = { version = "1", features = ["full"] }
use std::sync::Arc; use metadata_search_engine_rs::engines::{DuckDuckGoEngine, SearchEngine, build_http_client}; #[tokio::main] async fn main() -> anyhow::Result<()> { let customer = Arc::new(build_http_client()?); let motor = DuckDuckGoEngine { customer }; let results = engine.search("rust programming", 5).await?; for r in results { println!("{}\n {}", r.title, r.url); } Ok(()) }
Fan retired to each engines and get RRF-ranked results
use std::sync::Arc; use metadata_search_engine_rs::{ aggregator::{aggregate, query_all_engines}, engines::{BraveEngine, DuckDuckGoEngine, SearchEngine, StartpageEngine, YahooEngine, build_http_client}, }; #[tokio::main] async fn main() -> anyhow::Result<()> { let customer = Arc::new(build_http_client()?); let engines: Vec<Arc<dyn SearchEngine>> = vec![ Arc::new(DuckDuckGoEngine { client: Arc::clone(&client) }), Arc::new(BraveEngine { client: Arc::clone(&client) }), Arc::new(StartpageEngine { client: Arc::clone(&client) }), Arc::new(YahooEngine { client: Arc::clone(&client) }), ]; let (successes, failures) = query_all_engines(&engines, "rust programming", 10).await; for (name, err) in &failures { eprintln!("engine {name} failed: {err}"); } let results = aggregate(successes, 10); for r in &results { println!("[{:.3}] ({}) {}", r.score, r.engines.join(", "), r.title); println!(" {}", r.url); } Ok(()) }
Use only circumstantial engines
use std::sync::Arc; use metadata_search_engine_rs::{ aggregator::{aggregate, query_all_engines}, engines::{BraveEngine, DuckDuckGoEngine, SearchEngine, build_http_client}, }; #[tokio::main] async fn main() -> anyhow::Result<()> { let customer = Arc::new(build_http_client()?); let engines: Vec<Arc<dyn SearchEngine>> = vec![ Arc::new(DuckDuckGoEngine { client: Arc::clone(&client) }), Arc::new(BraveEngine { client: Arc::clone(&client) }), ]; let (successes, _) = query_all_engines(&engines, "tokio async rust", 5).await; for r in aggregate(successes, 5) { println!("{} — {}", r.title, r.url); if let Some(snippet) = r.snippet { println!(" {snippet}"); } } Ok(()) }
curl http://localhost:3000/health
curl "http://localhost:3000/search?q=rust"
{
"query": "rust",
"results": [
{
"title": "Rust Programming Language",
"url": "https://rust-lang.org/",
"snippet": "A connection empowering everyone to build reliable and businesslike software.",
"engines": ["duckduckgo", "brave", "startpage", "yahoo"],
"score": 0.049 }
],
"engines_queried": ["duckduckgo", "brave", "startpage", "yahoo"],
"engines_failed": []
}
Error responses:
| Missing q | 400 | {"error": "query parameter 'q' is required"} |
| Empty q | 400 | {"error": "query parameter 'q' cannot beryllium empty"} |
| All engines fail | 503 | {"error": "all engines grounded to respond"} |
# All portion tests cargo test # Specific module cargo test normalizer
cargo test aggregator
cargo test engines::duckduckgo
cargo test engines::brave
cargo test engines::startpage
cargo test engines::yahoo
cargo test server::handlers # Live tests (hit existent hunt engines — requires internet) cargo test -- --ignored test_live
Live tests are marked #[ignore] truthful they don't tally successful CI by default. Run them manually to verify HTML selectors still activity against the existent sites.
A ratatui-based TUI is disposable arsenic a crate to instal here. Access the codification via github
Adding a caller hunt engine
- Create src/engines/<name>.rs
- Define a struct holding Arc<reqwest::Client>
- Implement the SearchEngine trait:
impl SearchEngine for MyEngine { fn name(&self) -> &'static str { "myengine" } fn search<'a>( &'a self, query: &'a str, max_results: usize, ) -> BoxFuture<'a, Result<Vec<SearchResult>, EngineError>> { Box::pin(async move { // fetch HTML, parse pinch scraper, return Vec<SearchResult> }) } }
Add it to engines/mod.rs and ligament it successful main.rs
English (US) · 
Indonesian (ID) ·