coopcloud_code/home
0
0
Fork 0
Code Issues 2 Pull requests Projects Releases Packages Wiki Activity Actions

Layer 5b: Content Regression Tests — No Raw JSON, No Broken Labels, No Placeholders #57

New issue
Closed
opened 2026-04-01 18:06:18 +00:00 by mik-tf · 1 comment
mik-tf commented 2026-04-01 18:06:18 +00:00
Member
Copy link

Problem

Our 7-layer test pyramid has a critical gap between API tests (Layer 3) and visual screenshots (Layer 7). API tests verify the backend sends correct data. Visual tests catch layout shifts. But zero tests verify what the user actually reads on the rendered page.

This gap allowed a bug to ship where the product detail Specifications table displayed raw JSON objects ({"attribute_type":"Number","display_order":3,...}) instead of human-readable values (8, 32 GB, Germany). All 176 API tests passed. All 17 visual snapshots passed. The bug was caught by a human looking at the page.

Root Cause Analysis

Current Playwright tests check:

  • HTTP status codes (API returns 200) ✓
  • Element existence (table exists, button exists) ✓
  • Navigation flows (register → dashboard → buy) ✓
  • Screenshot comparison (layout matches golden) ✓

Current Playwright tests do NOT check:

  • Table cell values (shows 8 not {"attribute_type":...}) ✗
  • Label formatting (shows CPU Cores not Cpu_cores) ✗
  • Data rendering (shows $154.00 not undefined) ✗
  • Empty states (shows "No items" not blank page) ✗
  • Placeholder text removed (no "will appear here") ✗

Solution: Content Regression Suite

New Playwright test file: tests/playwright/tests/content-regression.spec.ts

This test crawls every SPA page and asserts the rendered content is correct.

Test 1: No Raw JSON in Visible Text

For every page, assert that no visible table cell, paragraph, or heading contains raw JSON objects.

const cells = await page.locator('table tbody td').all();
for (const cell of cells) {
  const text = await cell.textContent();
  expect(text).not.toMatch(/\{.*attribute_type.*\}/);
  expect(text).not.toMatch(/^\s*\{.*\}\s*$/);
}

Test 2: No Snake_Case Labels in UI

Assert no visible text contains snake_case patterns that should be humanized.

const body = await page.locator('body').textContent();
expect(body).not.toMatch(/\b[a-z]+_[a-z]+_[a-z]+\b/); // No triple snake_case
// Check specific known offenders
for (const bad of ['Cpu_cores', 'Memory_gb', 'Storage_gb', 'Uptime_percentage']) {
  expect(body).not.toContain(bad);
}

Test 3: No Placeholder Text in Production

Assert no "will appear here", "lorem ipsum", "TODO", "coming soon" in rendered pages.

const placeholders = ['will appear here', 'lorem ipsum', 'TODO:', 'FIXME:', 'placeholder'];
for (const p of placeholders) {
  expect(body.toLowerCase()).not.toContain(p.toLowerCase());
}

Test 4: Product Detail Has Real Specs

Navigate to a product, verify specs table has numeric/text values.

await page.goto('/products/000a');
const specCells = await page.locator('table tbody td').all();
for (const cell of specCells) {
  const text = (await cell.textContent())?.trim();
  expect(text).toBeTruthy(); // Not empty
  expect(text!.length).toBeLessThan(200); // Not a JSON blob
  expect(text).not.toContain('attribute_type'); // Not raw struct
}

Test 5: Wallet Shows Formatted Balance

await page.goto('/wallet');
const balanceText = await page.locator('.card').first().textContent();
expect(balanceText).not.toContain('NaN');
expect(balanceText).not.toContain('undefined');
expect(balanceText).toMatch(/\d+\.\d{2}/); // Formatted number

Test 6: Marketplace Cards Have Price and Name

await page.goto('/marketplace');
const cards = await page.locator('.marketplace-item').all();
expect(cards.length).toBeGreaterThan(0);
for (const card of cards.slice(0, 3)) {
  const text = await card.textContent();
  expect(text).toMatch(/\$/); // Has price
  expect(text!.length).toBeGreaterThan(20); // Has real content
}

Test 7: Empty States Handled Gracefully

Navigate as a fresh user to orders, messaging, etc. and verify friendly empty state messages.

for (const path of ['/orders', '/messaging']) {
  await page.goto(path);
  const body = await page.locator('main').textContent();
  expect(body!.length).toBeGreaterThan(10); // Not blank
  expect(body).not.toContain('Error');
  expect(body).not.toContain('undefined');
}

Test 8: All 44 Routes Render (No Panics, No Blank Pages)

Crawl every SPA route and verify minimum content length, no WASM panics, no router debug output.

const routes = ['/', '/marketplace', '/docs', '/login', '/register',
  '/about', '/privacy', '/terms', '/contact', '/changelog', '/roadmap',
  '/marketplace/statistics', '/products/000a', ...];

for (const route of routes) {
  await page.goto(route);
  await page.waitForLoadState('networkidle');
  const body = await page.locator('body').textContent();
  expect(body!.length).toBeGreaterThan(50);
  expect(body).not.toContain('panicked at');
  expect(body).not.toContain('Unresolved');
}

Pages to Cover

Public (unauthenticated)

  • / (home)
  • /marketplace + each category (/marketplace/compute, etc.)
  • /products/:id (at least 3 products)
  • /docs + /docs/:topic
  • /login, /register
  • /about, /privacy, /terms, /contact, /changelog, /roadmap

Authenticated (after register/login)

  • /dashboard
  • /wallet (balance + transactions)
  • /orders (empty + after purchase)
  • /cart (empty + with items)
  • /messaging (empty state)
  • /profile, /settings
  • /dashboard/nodes, /dashboard/services, /dashboard/apps
  • /pools

Updated Test Pyramid

Layer Tool Count What it catches
1. Compile cargo check — Type errors
2. Unit cargo test 25 Logic bugs
3. API Smoke + Integration Shell/curl 176 Endpoint health, auth, CRUD
5a. E2E Journeys Playwright 48 User flows, purchase, adversarial
5b. Content Regression Playwright ~40 NEW Raw JSON, broken labels, placeholders, empty states
6. Load Shell/curl 7 Concurrency
7. Visual Regression Playwright + Hero Browser 17 Layout, CSS, screenshots

Acceptance Criteria

  • New test file: content-regression.spec.ts
  • Covers all 44 SPA routes
  • Catches raw JSON rendering (the specs bug)
  • Catches snake_case labels in UI
  • Catches placeholder text
  • Validates table cell content (not just existence)
  • Tests empty states
  • Integrated into make test-e2e pipeline
  • All tests pass against dev-app.projectmycelium.org
  • Added to CLAUDE.md test pyramid documentation

References

  • Bug that motivated this: product detail specs showed raw JSON (fixed in v2.1.0-dev)
  • Existing Playwright: tests/playwright/tests/marketplace-e2e.spec.ts (48 tests)
  • Existing admin: tests/playwright/tests/admin-e2e.spec.ts (41 tests)
  • Test pyramid: CLAUDE.md § Testing

— mik-tf

## Problem Our 7-layer test pyramid has a critical gap between API tests (Layer 3) and visual screenshots (Layer 7). API tests verify the backend sends correct data. Visual tests catch layout shifts. But **zero tests** verify what the user actually reads on the rendered page. This gap allowed a bug to ship where the product detail Specifications table displayed raw JSON objects (`{"attribute_type":"Number","display_order":3,...}`) instead of human-readable values (`8`, `32 GB`, `Germany`). All 176 API tests passed. All 17 visual snapshots passed. The bug was caught by a human looking at the page. ## Root Cause Analysis Current Playwright tests check: - HTTP status codes (API returns 200) ✓ - Element existence (table exists, button exists) ✓ - Navigation flows (register → dashboard → buy) ✓ - Screenshot comparison (layout matches golden) ✓ Current Playwright tests do NOT check: - Table cell values (shows `8` not `{"attribute_type":...}`) ✗ - Label formatting (shows `CPU Cores` not `Cpu_cores`) ✗ - Data rendering (shows `$154.00` not `undefined`) ✗ - Empty states (shows "No items" not blank page) ✗ - Placeholder text removed (no "will appear here") ✗ ## Solution: Content Regression Suite New Playwright test file: `tests/playwright/tests/content-regression.spec.ts` This test crawls every SPA page and asserts the rendered content is correct. ### Test 1: No Raw JSON in Visible Text For every page, assert that no visible table cell, paragraph, or heading contains raw JSON objects. ```typescript const cells = await page.locator('table tbody td').all(); for (const cell of cells) { const text = await cell.textContent(); expect(text).not.toMatch(/\{.*attribute_type.*\}/); expect(text).not.toMatch(/^\s*\{.*\}\s*$/); } ``` ### Test 2: No Snake_Case Labels in UI Assert no visible text contains snake_case patterns that should be humanized. ```typescript const body = await page.locator('body').textContent(); expect(body).not.toMatch(/\b[a-z]+_[a-z]+_[a-z]+\b/); // No triple snake_case // Check specific known offenders for (const bad of ['Cpu_cores', 'Memory_gb', 'Storage_gb', 'Uptime_percentage']) { expect(body).not.toContain(bad); } ``` ### Test 3: No Placeholder Text in Production Assert no "will appear here", "lorem ipsum", "TODO", "coming soon" in rendered pages. ```typescript const placeholders = ['will appear here', 'lorem ipsum', 'TODO:', 'FIXME:', 'placeholder']; for (const p of placeholders) { expect(body.toLowerCase()).not.toContain(p.toLowerCase()); } ``` ### Test 4: Product Detail Has Real Specs Navigate to a product, verify specs table has numeric/text values. ```typescript await page.goto('/products/000a'); const specCells = await page.locator('table tbody td').all(); for (const cell of specCells) { const text = (await cell.textContent())?.trim(); expect(text).toBeTruthy(); // Not empty expect(text!.length).toBeLessThan(200); // Not a JSON blob expect(text).not.toContain('attribute_type'); // Not raw struct } ``` ### Test 5: Wallet Shows Formatted Balance ```typescript await page.goto('/wallet'); const balanceText = await page.locator('.card').first().textContent(); expect(balanceText).not.toContain('NaN'); expect(balanceText).not.toContain('undefined'); expect(balanceText).toMatch(/\d+\.\d{2}/); // Formatted number ``` ### Test 6: Marketplace Cards Have Price and Name ```typescript await page.goto('/marketplace'); const cards = await page.locator('.marketplace-item').all(); expect(cards.length).toBeGreaterThan(0); for (const card of cards.slice(0, 3)) { const text = await card.textContent(); expect(text).toMatch(/\$/); // Has price expect(text!.length).toBeGreaterThan(20); // Has real content } ``` ### Test 7: Empty States Handled Gracefully Navigate as a fresh user to orders, messaging, etc. and verify friendly empty state messages. ```typescript for (const path of ['/orders', '/messaging']) { await page.goto(path); const body = await page.locator('main').textContent(); expect(body!.length).toBeGreaterThan(10); // Not blank expect(body).not.toContain('Error'); expect(body).not.toContain('undefined'); } ``` ### Test 8: All 44 Routes Render (No Panics, No Blank Pages) Crawl every SPA route and verify minimum content length, no WASM panics, no router debug output. ```typescript const routes = ['/', '/marketplace', '/docs', '/login', '/register', '/about', '/privacy', '/terms', '/contact', '/changelog', '/roadmap', '/marketplace/statistics', '/products/000a', ...]; for (const route of routes) { await page.goto(route); await page.waitForLoadState('networkidle'); const body = await page.locator('body').textContent(); expect(body!.length).toBeGreaterThan(50); expect(body).not.toContain('panicked at'); expect(body).not.toContain('Unresolved'); } ``` ## Pages to Cover ### Public (unauthenticated) - `/` (home) - `/marketplace` + each category (`/marketplace/compute`, etc.) - `/products/:id` (at least 3 products) - `/docs` + `/docs/:topic` - `/login`, `/register` - `/about`, `/privacy`, `/terms`, `/contact`, `/changelog`, `/roadmap` ### Authenticated (after register/login) - `/dashboard` - `/wallet` (balance + transactions) - `/orders` (empty + after purchase) - `/cart` (empty + with items) - `/messaging` (empty state) - `/profile`, `/settings` - `/dashboard/nodes`, `/dashboard/services`, `/dashboard/apps` - `/pools` ## Updated Test Pyramid | Layer | Tool | Count | What it catches | |-------|------|-------|-----------------| | 1. Compile | cargo check | — | Type errors | | 2. Unit | cargo test | 25 | Logic bugs | | 3. API Smoke + Integration | Shell/curl | 176 | Endpoint health, auth, CRUD | | **5a. E2E Journeys** | **Playwright** | **48** | **User flows, purchase, adversarial** | | **5b. Content Regression** | **Playwright** | **~40 NEW** | **Raw JSON, broken labels, placeholders, empty states** | | 6. Load | Shell/curl | 7 | Concurrency | | 7. Visual Regression | Playwright + Hero Browser | 17 | Layout, CSS, screenshots | ## Acceptance Criteria - [ ] New test file: `content-regression.spec.ts` - [ ] Covers all 44 SPA routes - [ ] Catches raw JSON rendering (the specs bug) - [ ] Catches snake_case labels in UI - [ ] Catches placeholder text - [ ] Validates table cell content (not just existence) - [ ] Tests empty states - [ ] Integrated into `make test-e2e` pipeline - [ ] All tests pass against dev-app.projectmycelium.org - [ ] Added to CLAUDE.md test pyramid documentation ## References - Bug that motivated this: product detail specs showed raw JSON (fixed in v2.1.0-dev) - Existing Playwright: `tests/playwright/tests/marketplace-e2e.spec.ts` (48 tests) - Existing admin: `tests/playwright/tests/admin-e2e.spec.ts` (41 tests) - Test pyramid: CLAUDE.md § Testing — mik-tf
mik-tf commented 2026-04-01 21:09:46 +00:00
Author
Member
Copy link

Done. 53 Playwright content regression tests in tests/playwright/tests/content-regression.spec.ts.

Covers all 44 SPA routes:

  • No raw JSON objects in rendered text
  • No snake_case labels in UI
  • No placeholder/dev text
  • No undefined/NaN/[object Object]
  • Product specs show values not structs
  • Formatted prices with currency
  • Empty states for cart, orders, messaging
  • 404 and edge case handling

All 53 pass against dev-app.projectmycelium.org after deploy.

Total test pyramid: 318 tests (265 + 53 new).

Done. 53 Playwright content regression tests in `tests/playwright/tests/content-regression.spec.ts`. Covers all 44 SPA routes: - No raw JSON objects in rendered text - No snake_case labels in UI - No placeholder/dev text - No undefined/NaN/[object Object] - Product specs show values not structs - Formatted prices with currency - Empty states for cart, orders, messaging - 404 and edge case handling All 53 pass against dev-app.projectmycelium.org after deploy. Total test pyramid: 318 tests (265 + 53 new).
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels   
meeting-notes

Meeting notes and follow-ups

  
meeting-sensitive

Sensitive meeting; no shared archive or AI processing

  
meeting-transcript

Raw meeting transcript; AI drafts the report

No labels
meeting-notes
meeting-sensitive
meeting-transcript
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
coopcloud_code/home#57
No description provided.