A to Z Experience Verification #3

New issue

Open

opened 2026-06-15 10:44:13 +00:00 by fatmaebrahim · 1 comment

fatmaebrahim commented

2026-06-15 10:44:13 +00:00

Member

Tested API Methods

Health & Discovery: health, system.ping, status, heroservice.json, openrpc.json
Models: models.list (7 models)
Introspection: ort_info, mem_info
Embeddings REST: single, batch, all quality levels, OpenAI aliases, fallback, usage stats
Embeddings RPC: positional/named params, batch, int8/fp16 precision metadata
Rerank REST: string/object docs, top_n, usage stats, sorting
Rerank RPC: positional/named params, top_k, hit fields
CORS: wildcard origin verified

End-to-end scenarios

Quality levels — Q1–Q4 verified on both REST and RPC
OpenAI compatibility — aliases resolve correctly, unknown models fall back to Q1
Semantic correctness — related texts rank higher, reranker orders relevant docs first
Determinism — same input always produces same output
Batch processing — multiple inputs return correct count and indices
Three listeners — TCP, rpc.sock, rest.sock all work independently
INT8 metadata — RPC preserves quantization info, REST returns plain f32
Error handling — bad input, missing fields, invalid JSON, unknown methods all return correct error codes
Performance — embed ~234 items/s, rerank ~50 items/s
Admin dashboard — status, embed/rerank playgrounds, benchmarks, API docs, theme toggle all functional

Bugs Found

Wrong precision label
The FP32 models were mislabeled as "fp16" everywhere — enum, JSON responses, OpenRPC spec. The code actually returns 32-bit floats with no half-precision conversion. Fixed by renaming to "fp32" across the codebase.
Wrong model label in mem_info
The bge-small/model.onnx file was labeled "Q2/Q4" but Q4 is bge-base, not bge-small. Fixed to "Q2".

### Tested API Methods - Health & Discovery: health, system.ping, status, heroservice.json, openrpc.json - Models: models.list (7 models) - Introspection: ort_info, mem_info - Embeddings REST: single, batch, all quality levels, OpenAI aliases, fallback, usage stats - Embeddings RPC: positional/named params, batch, int8/fp16 precision metadata - Rerank REST: string/object docs, top_n, usage stats, sorting - Rerank RPC: positional/named params, top_k, hit fields - CORS: wildcard origin verified ### End-to-end scenarios - Quality levels — Q1–Q4 verified on both REST and RPC - OpenAI compatibility — aliases resolve correctly, unknown models fall back to Q1 - Semantic correctness — related texts rank higher, reranker orders relevant docs first - Determinism — same input always produces same output - Batch processing — multiple inputs return correct count and indices - Three listeners — TCP, rpc.sock, rest.sock all work independently - INT8 metadata — RPC preserves quantization info, REST returns plain f32 - Error handling — bad input, missing fields, invalid JSON, unknown methods all return correct error codes - Performance — embed ~234 items/s, rerank ~50 items/s - Admin dashboard — status, embed/rerank playgrounds, benchmarks, API docs, theme toggle all functional ### Bugs Found - Wrong precision label The FP32 models were mislabeled as "fp16" everywhere — enum, JSON responses, OpenRPC spec. The code actually returns 32-bit floats with no half-precision conversion. Fixed by renaming to "fp32" across the codebase. - Wrong model label in mem_info The bge-small/model.onnx file was labeled "Q2/Q4" but Q4 is bge-base, not bge-small. Fixed to "Q2".

fatmaebrahim commented

2026-06-15 12:45:35 +00:00

Author

Member

Deps References

All git dependencies pointed to development branch instead of main:

herolib_core
hero_lifecycle
hero_admin_lib
herolib_derive
herolib_openrpc
hero_proc_sdk

### Deps References All git dependencies pointed to development branch instead of main: - herolib_core - hero_lifecycle - hero_admin_lib - herolib_derive - herolib_openrpc - hero_proc_sdk

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

lhumina_code/hero_embedder_provider#3

No description provided.

Rows
Columns