lhumina_code/hero_books
20
0
Fork 0
Code Issues 8 Pull requests Projects Releases Packages Wiki Activity Actions

Issues sweep: 13 issues addressed (development-fixes-feb) #24

Closed
mik-tf wants to merge 0 commits from development-fixes-feb into development
pull from: development-fixes-feb
merge into: lhumina_code:development
Conversation 0 Commits - Files changed -
mik-tf commented 2026-02-08 01:39:51 +00:00
Owner
Copy link

Issues Sweep: 13 issues addressed

This branch tackles 13 of the 16 open issues in a single coordinated effort.

Phase 1: Quick Wins

  • #17 CORS — verified, headers on all endpoints including /rpc
  • #21 TOML in .book dir — scanner accepts TOML configs
  • #11 Indexing always INT Q1 — force quality=1 for all namespaces

Phase 2: Pipeline Restructure + Persistence

  • #22 Processing order fixed: collections Q&A → export → index
  • #23 KVS persistence via hero_embedder (book/page state survives restarts)

Phase 3: Incremental Processing

  • #20 Autocommit .ai/ metadata to git after collection processing

Phase 5: CLI & Demos

  • #19 Demos are now rhai scripts, not Rust binaries. Added run subcommand to CLI. Removed 5 demo_*.rs files.

Phase 6: Web Features

  • #16 Import form — full pipeline: clone → Q&A extraction → push .ai/ → index
  • #8 Library routing: /library/{name} shows books scoped to namespace
  • #9 Tags & filtering: auto-extracted from Q&A topics, displayed on book cards

Phase 7: Polish

  • #14 PDF caching with hash-based change detection via KVS
  • #13 UI rebrand: "AI Librarian" instead of "Search"

Additional improvements (not tied to specific issues)

  • The Knowledge World homepage redesign — library cards with Alexandria vibe
  • Auto-tag extraction from Q&A topics into book metadata
  • Import pipeline rewrite — end-to-end: clone → process → push .ai/ → index
  • VISION.md — Phase 2+ roadmap (oschema, Dioxus, MCP, skills)
  • .gitignore updated for .ai/ directories
  • README updated with env vars table

Skipped (3 issues)

  • #18 MCP over SSE — requires async server migration (axum), high risk
  • #15 K8s deploy — needs cluster access, not a code issue
  • #10 API-driven + widget — correctly scoped as next milestone

Dependencies

  • hero_embedder: namespace_create_with_quality() SDK method pushed to development (1 commit)

13 commits:

5e34cd1 Add VISION.md: Phase 2+ roadmap (oschema, Dioxus, MCP, skills)
36dcfd5 Auto-extract tags from Q&A topics, aggregate in book listings
093e24a UI: The Knowledge World — libraries front and center
dcbdad5 Import form: full pipeline (clone → Q&A → push → index)
24e60b1 gitignore: ignore .ai/ generated metadata directories
ffa9dd3 README: env vars table, reference env_secrets convention
06a8d1a Phase 7: #14 PDF cache, #13 UI rebrand
c98cf72 Phase 6: #16 import, #8 library routing, #9 tags
f8535bc Phase 5: #19 rhai demos, remove Rust demo binaries
0f35fa1 Phase 3: #20 autocommit .ai/ metadata
23ea189 Phase 2: #22 pipeline restructure + #23 KVS persistence
08591c5 Phase 1: #17 CORS, #21 TOML, #11 Q1 indexing, #23 KVS methods
## Issues Sweep: 13 issues addressed This branch tackles 13 of the 16 open issues in a single coordinated effort. ### Phase 1: Quick Wins - **#17** CORS — verified, headers on all endpoints including `/rpc` - **#21** TOML in .book dir — scanner accepts TOML configs - **#11** Indexing always INT Q1 — force quality=1 for all namespaces ### Phase 2: Pipeline Restructure + Persistence - **#22** Processing order fixed: collections Q&A → export → index - **#23** KVS persistence via hero_embedder (book/page state survives restarts) ### Phase 3: Incremental Processing - **#20** Autocommit .ai/ metadata to git after collection processing ### Phase 5: CLI & Demos - **#19** Demos are now rhai scripts, not Rust binaries. Added `run` subcommand to CLI. Removed 5 demo_*.rs files. ### Phase 6: Web Features - **#16** Import form — full pipeline: clone → Q&A extraction → push .ai/ → index - **#8** Library routing: `/library/{name}` shows books scoped to namespace - **#9** Tags & filtering: auto-extracted from Q&A topics, displayed on book cards ### Phase 7: Polish - **#14** PDF caching with hash-based change detection via KVS - **#13** UI rebrand: "AI Librarian" instead of "Search" ### Additional improvements (not tied to specific issues) - **The Knowledge World** homepage redesign — library cards with Alexandria vibe - **Auto-tag extraction** from Q&A topics into book metadata - **Import pipeline** rewrite — end-to-end: clone → process → push .ai/ → index - **VISION.md** — Phase 2+ roadmap (oschema, Dioxus, MCP, skills) - **.gitignore** updated for .ai/ directories - **README** updated with env vars table ### Skipped (3 issues) - **#18** MCP over SSE — requires async server migration (axum), high risk - **#15** K8s deploy — needs cluster access, not a code issue - **#10** API-driven + widget — correctly scoped as `next` milestone ### Dependencies - hero_embedder: `namespace_create_with_quality()` SDK method pushed to development (1 commit) ### 13 commits: ``` 5e34cd1 Add VISION.md: Phase 2+ roadmap (oschema, Dioxus, MCP, skills) 36dcfd5 Auto-extract tags from Q&A topics, aggregate in book listings 093e24a UI: The Knowledge World — libraries front and center dcbdad5 Import form: full pipeline (clone → Q&A → push → index) 24e60b1 gitignore: ignore .ai/ generated metadata directories ffa9dd3 README: env vars table, reference env_secrets convention 06a8d1a Phase 7: #14 PDF cache, #13 UI rebrand c98cf72 Phase 6: #16 import, #8 library routing, #9 tags f8535bc Phase 5: #19 rhai demos, remove Rust demo binaries 0f35fa1 Phase 3: #20 autocommit .ai/ metadata 23ea189 Phase 2: #22 pipeline restructure + #23 KVS persistence 08591c5 Phase 1: #17 CORS, #21 TOML, #11 Q1 indexing, #23 KVS methods ``` 
- CORS already covers /rpc endpoints (verified, no changes needed)
- Scanner now finds TOML files inside .book directories
- All namespace creation now uses quality=1 (INT8, fastest)
- Added namespace_create_with_quality to hero_embedder SDK
- Added KVS wrapper methods to Embedder (set/get/delete/list/clear)
- Added hero_client() accessor for direct SDK access

Issues: #11, #17, #21, #23 (partial)
Pipeline order fixed:
1. process_collections_for_qa() — extract Q&A in source collections first
2. export_books_for_serving() — now copies .ai/ metadata to export dir
3. index_books_for_search() — uses cached Q&A, avoids redundant AI calls
4. persist_books_state() — saves book metadata to hero_embedder KVS

Exporter now copies .ai/*.toml and .ai/*.embeddings from source collections.
Indexer checks both source collection and export dir for cached metadata.
Book state persisted in herobooks_state namespace via KVS for fast restart.

Issues: #22, #23
- Added autocommit field to ProcessorConfig
- After Q&A extraction, optionally git add .ai/ && git commit
- Checks if collection is in a git repo before attempting commit
- Added --autocommit flag to hero_books_client process subcommand
- Only commits if there are actual changes to .ai/ directory

Issue: #20
- Add 'run' subcommand to hero_books_client for executing .rhai scripts
- Register all 4 rhai modules (doctree, vectorsdk, ontology, book)
- Remove 5 Rust demo binaries (demo_basic/geomind/mycelium/ourworld/all.rs)
- Remove corresponding [[example]] entries from Cargo.toml
- Add Makefile targets: demo-basic, demo-geomind, demo-mycelium, demo-ourworld, demo-all
- Keep non-demo Rust examples (rust/, hero_to_toml.rs, process_herobooks.rs)

Closes #19
- Add GET/POST /import route with form for importing git collections (#16)
- Add /library/{namespace} route with filtered book view (#8)
- Add keywords/tags to ExportedBook and display on book cards (#9)
- Add tag filtering on library pages (/library/{ns}?tag=...)
- Add NamespaceItem struct for import form namespace selection
- Add Import link to navbar
- Library names on /books page now link to /library/{namespace}
- Tags shown as badges on book cards in both /books and /library views

Closes #16, #8, #9
- PDF caching already implemented with content-hash comparison (#14 verified)
- Rebrand search UI to 'AI Librarian' throughout (#13):
  - Search page header: 'Search Results' -> 'AI Librarian'
  - Search placeholder: 'Search books...' -> 'Ask the Librarian...'
  - Index hero text: 'Search across...' -> 'Your AI Librarian'
  - Index button: 'Search' -> 'Ask'
  - Feature card: 'Semantic Search' -> 'Ask Anything'
  - No results: 'No results found' -> 'The Librarian couldn't find results'
  - Breadcrumb: 'Search Results' -> 'AI Librarian'

Closes #14, #13
- Replace bullet list with full table showing all 5 env vars
- Add Required column (GROQ + OPENROUTER required, rest optional)
- Fix HERO_EMBEDDER_URL default port (8080 -> 3752)
- Add SAMBANOVA_API_KEY as optional provider
- Reference env_secrets skill as canonical convention
- Remove .env.example reference (follows source pattern, not dotenv)
- Rewrite serve_import_submit to run complete pipeline instead of scan-only
- Add import_collection_pipeline(): clone/pull, process_collection Q&A,
  upload_collection_embeddings_and_build to hero_embedder
- Add push_ai_metadata(): git add/commit/push .ai/ back to remote
- Add push checkbox to import form template
- Remove 15 tracked .ai/ files (now gitignored, rebuilt each run)
- Rewrite examples/README.md (was stale Atlas references)
- Homepage redesigned: grand library cards with gradient themes and icons
- Each library shows book count, colored theme, 'Enter library' CTA
- Stats bar: '3 libraries, 7 books'
- Navbar: Libraries (home), All Books, Import
- Library page: hero banner with namespace-specific gradient and icon
- Search button scoped to library namespace
- Brand icon: globe2 (knowledge world)
- Add tags field to DocumentMetadata (from topic names with Q&A content)
- Populate tags during process_collection after Q&A extraction
- Aggregate per-page .ai/ tags into book-level tags in get_books_list
- Tags merge with TOML keywords, deduplicated and sorted
Add VISION.md: Phase 2+ roadmap (oschema, Dioxus, MCP, skills)
Some checks failed
Test / test (push) Failing after 2m59s
Details
Test / test (pull_request) Failing after 4m33s
Details
5e34cd1e46
Fix clippy: collapsible_if, type_complexity, should_implement_trait
All checks were successful
Test / test (push) Successful in 3m39s
Details
Test / test (pull_request) Successful in 4m39s
Details
def038167e
Import: live progress log with polling, fix AI JSON parsing robustness
All checks were successful
Test / test (push) Successful in 3m0s
Details
Test / test (pull_request) Successful in 3m56s
Details
e309b52945 
- Import runs in background thread, browser polls /import/log/{id} every 2s
- Live progress page shows running/complete/failed badge + scrolling log
- Log lines color-coded: warnings yellow, errors red, success green
- AI Q&A parser: return empty on bad JSON instead of failing the whole topic
- extract_json: balanced bracket matching instead of rfind
- Debug logging of raw AI response on parse failure
Search: fix relevance %, require library selection, no cross-library search
All checks were successful
Test / test (pull_request) Successful in 3m0s
Details
Test / test (push) Successful in 3m56s
Details
f8a3e1a722 
- Fix relevance_percent formula: use cosine distance range [0,2] instead of [0,1]
  so results show meaningful percentages (100%, 98%, 63%...) instead of mostly 0%
- Remove 'All Libraries' option from search sidebar filter
- Default to first available library when no namespace specified
- Apply library default to all 3 search endpoints (page, API, AI summary)
- Increase relevance bar size for better visibility (80px wide, 8px tall)
Search: move to library context, remove navbar search bar
All checks were successful
Test / test (pull_request) Successful in 3m6s
Details
Test / test (push) Successful in 3m57s
Details
e1ab2b9730 
- Remove global search bar from navbar (base.html)
- Add inline 'Ask the Librarian' search to library page with results
- Library page now handles ?q= parameter for searching within that library
- Fix relevance formula: cosine distance [0,2] scale for meaningful percentages
- Search sidebar library links now redirect to /library/{ns}?q= flow
- All search is now scoped to a single library, no cross-library search
Import: add local path support, code root per library
Some checks failed
Test / test (push) Has been cancelled
Details
Test / test (pull_request) Failing after 5m53s
Details
0f2c0a13c4 
- Add source type toggle: Git Repository or Local Path
- Local path import: scan directory for .collection markers, process, index
- Code root field: specify where git repos are cloned (default ~/code)
- Pass code_root to GitConfig::from_options for per-library clone paths
- Updated import form with source toggle, local path field, code root field
- Git-specific options (pull, reset, push_ai) hidden for local imports
- Updated tips sidebar to mention .book and local path support
UI: remove All Books nav link, add AI Summary to library search
Some checks failed
Test / test (pull_request) Failing after 3m0s
Details
Test / test (push) Failing after 3m5s
Details
c57dc4b89c 
- Remove 'All Books' from navbar (search is per-library now)
- Add AI Summary toggle + panel to library page search results
- AI summary fetches from /api/ai-summary scoped to library namespace
- Includes loading spinner, error handling, markdown formatting
Homepage: remove search bar, search is per-library only
Some checks failed
Test / test (push) Failing after 3m53s
Details
Test / test (pull_request) Failing after 6m33s
Details
2f44af9146 
Users pick a library first, then Ask the Librarian from within that library.
Library: bigger Ask the Librarian search bar below hero section
Some checks failed
Test / test (pull_request) Failing after 3m6s
Details
Test / test (push) Failing after 3m53s
Details
4f1ba7aecd 
Move search from cramped hero banner to prominent centered section with
input-group-lg, primary Ask button, matching homepage search style.
Library: redesign hero with centered icon, title, and search bar
Some checks failed
Test / test (push) Failing after 4m49s
Details
Test / test (pull_request) Failing after 4m49s
Details
ecbae14b0a 
- Large centered icon (3.5rem) above library name
- Light font weight title (2.5rem, 300 weight)
- Search bar integrated inside hero with gradient background
- More vertical padding for spacious feel
- Matches homepage visual style
Library: more spacing between hero/search section and books
Some checks failed
Test / test (pull_request) Failing after 2m57s
Details
Test / test (push) Failing after 3m7s
Details
33d57d952d
Library: clean redesign with spacious layout
Some checks failed
Test / test (push) Failing after 2m59s
Details
Test / test (pull_request) Failing after 3m54s
Details
278c992798 
- Hero section: just icon, name, book count — generous padding (4rem)
- Ask the Librarian: separate centered section with its own heading,
  2.5rem top margin, 3rem bottom margin
- Books section: clear heading, own space below
- Three distinct visual zones with breathing room between them
- Cleaner CSS, removed inline styles
Library hero: rounder corners, more padding, larger text
Some checks failed
Test / test (pull_request) Failing after 2m56s
Details
Test / test (push) Has been cancelled
Details
bd31721c8b 
- border-radius 1rem -> 1.5rem for softer edges
- padding 4rem 2rem -> 5rem 3rem for more breathing room
- icon 4rem -> 4.5rem, title 2.8rem -> 3rem
- more spacing between icon, title, and stats
Library hero: enforce round corners with 2rem, overflow hidden
Some checks failed
Test / test (push) Failing after 3m0s
Details
Test / test (pull_request) Failing after 3m6s
Details
22053a55db
Library hero: inline style for border-radius and text-align to prevent override
Some checks failed
Test / test (pull_request) Failing after 5m6s
Details
Test / test (push) Failing after 5m6s
Details
d470e0d752
Book page: Ask the Librarian vibe with hero banner, library-scoped search
Some checks failed
Test / test (pull_request) Failing after 2m59s
Details
Test / test (push) Failing after 3m53s
Details
1b40924ff1
Fix clippy: match_result_ok, too_many_arguments
All checks were successful
Test / test (pull_request) Successful in 6m0s
Details
Test / test (push) Successful in 6m0s
Details
da5d8cb75d
Book hero: round corners with inline style, more padding to match library
All checks were successful
Test / test (push) Successful in 3m6s
Details
Test / test (pull_request) Successful in 3m54s
Details
3e7d0dfbdc
Book search: remove mic button, widen bar to match library style
Some checks failed
Test / test (push) Has been cancelled
Details
Test / test (pull_request) Successful in 5m36s
Details
1b743a659d
Book search: restore mic button, widen bar to show full placeholder
All checks were successful
Test / test (push) Successful in 3m7s
Details
Test / test (pull_request) Successful in 4m27s
Details
7b8912aaf0
Library + book pages: add mic button with voice search to all search bars
All checks were successful
Test / test (push) Successful in 3m2s
Details
Test / test (pull_request) Successful in 3m12s
Details
8546da78b2
feat: KVS persistence load + relevance bar fix (#23)
Some checks failed
Test / test (push) Failing after 6m7s
Details
Test / test (pull_request) Failing after 6m7s
Details
1ab0f7a1e2 
- Implement load_books_state() to read persisted book state from KVS
- Add compute_book_content_hash() for per-book content hashing
- Enhance persist_books_state() to store content_hash
- Modify index_books_for_search() to skip unchanged books via KVS hash
- Update startup flow: load KVS → index (skip unchanged) → persist
- Fix relevance bar: use raw scores, normalize within result set (40-95%)

Cold start: indexes normally, persists state
Warm restart: skips indexing for unchanged books (fast restart)
rename default namespace display from 'Default' to 'Hero Books'
Some checks failed
Test / test (push) Failing after 3m8s
Details
Test / test (pull_request) Failing after 3m32s
Details
c73b234add 
- Rhai scripts can now start with #!/usr/bin/env hero_books_client
- Client automatically strips shebang line before executing
- Fixes demo_basic.rhai to use correct Rhai API methods:
  - Use collection_names() instead of collections()
  - Use scan_path() instead of scan()
  - Use ensure_index_default() instead of ensure_index()
  - Use process_collection_ai() for collection-level processing
- Simplified demo to match available Rhai bindings

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
fix: Add missing BUILD_LIB and buildenv configuration
Some checks failed
Test / test (pull_request) Failing after 3m5s
Details
Test / test (push) Failing after 21s
Details
ee78733c40 
- Define BUILD_LIB variable pointing to scripts/build_lib.sh
- Add required buildenv.sh exports: PROJECT_NAME, BINARIES, ALL_FEATURES
- Fixes 'make build' and 'make run' commands

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
mik-tf closed this pull request 2026-02-08 13:00:53 +00:00
Some checks failed
Test / test (pull_request) Failing after 3m5s
Details
Test / test (push) Failing after 21s
Details

Pull request closed

This pull request cannot be reopened because the branch was deleted.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
lhumina_code/hero_books!24
No description provided.