Implement broker-side web search tools (Serper / Exa) for all providers #154

New issue

Open

opened 2026-06-15 10:54:07 +00:00 by nabil_salah · 1 comment

nabil_salah commented

2026-06-15 10:54:07 +00:00

Member

Problem

The broker supports many providers: OpenRouter, Kimi (direct), Groq, SambaNova, OpenAI, and Alibaba. Only OpenRouter has built-in server tools like openrouter:web_search and openrouter:web_fetch. When a request routes to any direct provider, clients cannot use web search or fetch.

Goal

Add broker-side tool execution so that web search and web fetch work uniformly with every configured chat provider.

Backend fallback order for tool execution:

Exa
Serper
Direct HTTP fetch

Scope

Implement a tool executor inside hero_aibroker_server.
Supported tools:
- web_search
- web_fetch
Supported execution backends:
- Exa Search / Contents API
- Serper.dev Google Search API
- Direct HTTP fetch
Tool execution must be provider-agnostic: only OpenAI-compatible function-tool schemas are sent upstream.
Support non-streaming and streaming chat requests.

Proposed design

1. Configuration

Add new config keys / secrets:

HERO_AIBROKER_EXA_API_KEYS
HERO_AIBROKER_SERPER_API_KEYS
Optional enablement/priority config.

Update crates/hero_aibroker_server/src/config/mod.rs.

2. Tool definitions

Add crates/hero_aibroker_server/src/tools/:

ToolBackend enum: Exa, Serper, DirectFetch
ToolExecutor trait
Implementations for each backend
Convert each tool to an OpenAI function definition for upstream injection.

3. Request lifecycle changes

In crates/hero_aibroker_server/src/service/router.rs:

Before sending upstream, if the model has tool_calling capability and broker tools are enabled:
- Append broker tool definitions (__hero_web_search, __hero_web_fetch) to request.tools.
On response, detect tool calls targeting broker tools:
- Select backend in priority order: Exa → Serper → Direct fetch.
- Execute the tool.
- Append a tool / function result message.
- Re-send the conversation upstream.
- Loop until final answer or max iterations reached.
For streaming: buffer to detect tool calls, then either continue or fall back to non-streaming.

4. Direct HTTP fetch fallback

If Exa and Serper are not configured:

web_fetch: fetch the URL directly via HTTP and return extracted text/Markdown.
web_search: this may not be possible with direct HTTP fetch; consider returning an error or requiring at least one search backend.

5. Observability

Log each tool call, selected backend, latency, and result size.
Attribute tool cost to the parent request.

Non-goals

No MCP support for this feature.
No provider-specific tool schemas beyond OpenAI-compatible functions.

Acceptance criteria

web_fetch works with kimi-latest, kimi-k2, groq, sambanova, openai, and openrouter.
web_search works with the same providers when Exa or Serper is configured.
Backend fallback order is Exa → Serper → Direct fetch.
Requests without broker tools are unchanged.
Unit tests for backend selection and tool execution.
Integration tests with mocked Exa, Serper, and HTTP endpoints.

## Problem The broker supports many providers: OpenRouter, Kimi (direct), Groq, SambaNova, OpenAI, and Alibaba. Only OpenRouter has built-in server tools like `openrouter:web_search` and `openrouter:web_fetch`. When a request routes to any direct provider, clients cannot use web search or fetch. ## Goal Add broker-side tool execution so that web search and web fetch work uniformly with **every** configured chat provider. Backend fallback order for tool execution: 1. Exa 2. Serper 3. Direct HTTP fetch ## Scope - Implement a tool executor inside `hero_aibroker_server`. - Supported tools: - `web_search` - `web_fetch` - Supported execution backends: - Exa Search / Contents API - Serper.dev Google Search API - Direct HTTP fetch - Tool execution must be provider-agnostic: only OpenAI-compatible function-tool schemas are sent upstream. - Support non-streaming and streaming chat requests. ## Proposed design ### 1. Configuration Add new config keys / secrets: - `HERO_AIBROKER_EXA_API_KEYS` - `HERO_AIBROKER_SERPER_API_KEYS` - Optional enablement/priority config. Update `crates/hero_aibroker_server/src/config/mod.rs`. ### 2. Tool definitions Add `crates/hero_aibroker_server/src/tools/`: - `ToolBackend` enum: `Exa`, `Serper`, `DirectFetch` - `ToolExecutor` trait - Implementations for each backend - Convert each tool to an OpenAI `function` definition for upstream injection. ### 3. Request lifecycle changes In `crates/hero_aibroker_server/src/service/router.rs`: - Before sending upstream, if the model has `tool_calling` capability and broker tools are enabled: - Append broker tool definitions (`__hero_web_search`, `__hero_web_fetch`) to `request.tools`. - On response, detect tool calls targeting broker tools: - Select backend in priority order: Exa → Serper → Direct fetch. - Execute the tool. - Append a `tool` / `function` result message. - Re-send the conversation upstream. - Loop until final answer or max iterations reached. - For streaming: buffer to detect tool calls, then either continue or fall back to non-streaming. ### 4. Direct HTTP fetch fallback If Exa and Serper are not configured: - `web_fetch`: fetch the URL directly via HTTP and return extracted text/Markdown. - `web_search`: this may not be possible with direct HTTP fetch; consider returning an error or requiring at least one search backend. ### 5. Observability - Log each tool call, selected backend, latency, and result size. - Attribute tool cost to the parent request. ## Non-goals - No MCP support for this feature. - No provider-specific tool schemas beyond OpenAI-compatible functions. ## Acceptance criteria - [ ] `web_fetch` works with `kimi-latest`, `kimi-k2`, `groq`, `sambanova`, `openai`, and `openrouter`. - [ ] `web_search` works with the same providers when Exa or Serper is configured. - [ ] Backend fallback order is Exa → Serper → Direct fetch. - [ ] Requests without broker tools are unchanged. - [ ] Unit tests for backend selection and tool execution. - [ ] Integration tests with mocked Exa, Serper, and HTTP endpoints.

nabil_salah self-assigned this

2026-06-15 10:54:10 +00:00

nabil_salah commented

2026-06-15 13:49:31 +00:00

Author

Member

Implemented broker-side web search/fetch tools. Summary:

Added crates/hero_aibroker_server/src/tools/ with:
- ToolRegistry, ToolExecutor trait, ToolError
- ExaExecutor (__hero_web_search, __hero_web_fetch)
- SerperExecutor (__hero_web_search)
- DirectFetchExecutor using reqwest + html_to_markdown
Wired tool injection and execution into service/router.rs:
- Injects __hero_web_search / __hero_web_fetch when a search backend is configured
- Runs tool-call loop up to 5 iterations
- Streaming requests with broker tools downgrade to a single SSE event after the loop
Added exa_api_keys / serper_api_keys to Config, loaded from EXA_API_KEYS / SERPER_API_KEYS secrets in hero_proc core
Selection order: Exa → Serper → Direct fetch; when no Exa/Serper key is configured, no tools are injected and requests pass through unchanged
Fixed admin reload path to rebuild ToolRegistry from live config, so keys can be added without restart
Cleaned modelsconfig.yml to keep only kimi-latest and kimi-k2

Verification:

cargo clippy -p hero_aibroker_server -- -D warnings — clean
cargo test -p hero_aibroker_server — 115 passed

Will open the PR and do live testing tomorrow.

Implemented broker-side web search/fetch tools. Summary: - Added `crates/hero_aibroker_server/src/tools/` with: - `ToolRegistry`, `ToolExecutor` trait, `ToolError` - `ExaExecutor` (`__hero_web_search`, `__hero_web_fetch`) - `SerperExecutor` (`__hero_web_search`) - `DirectFetchExecutor` using `reqwest` + `html_to_markdown` - Wired tool injection and execution into `service/router.rs`: - Injects `__hero_web_search` / `__hero_web_fetch` when a search backend is configured - Runs tool-call loop up to 5 iterations - Streaming requests with broker tools downgrade to a single SSE event after the loop - Added `exa_api_keys` / `serper_api_keys` to `Config`, loaded from `EXA_API_KEYS` / `SERPER_API_KEYS` secrets in hero_proc `core` - Selection order: Exa → Serper → Direct fetch; when no Exa/Serper key is configured, no tools are injected and requests pass through unchanged - Fixed admin reload path to rebuild `ToolRegistry` from live config, so keys can be added without restart - Cleaned `modelsconfig.yml` to keep only `kimi-latest` and `kimi-k2` Verification: - `cargo clippy -p hero_aibroker_server -- -D warnings` — clean - `cargo test -p hero_aibroker_server` — 115 passed Will open the PR and do live testing tomorrow.