| FIRST PRIORITYThe form conversation | 153 cases test individual AI replies. Some include a prepared conversation history, but they do not run an entire conversation from start to finish. | Let the AI conduct full conversations. Try short answers, corrections, refusals, questions back to the host, and people who want to leave early. Success means: The AI collects what the form needs, does not ask again for answered information, and finishes properly. If someone leaves early, unanswered questions stay unanswered. Code and test referencessrc/workflows/form/prompts/form-agent/sample.jsonlsrc/workflows/form/completion.test.ts
|
|---|
| FIRST PRIORITYAnswer boxes and saved answers | 6 AI cases check how answers are extracted. Code tests also check answer controls, such as date pickers and multiple choice. | Try every answer control, then check the saved answer and exported file. Include custom choices, corrections, invalid values and dates across time zones. Success means: What the person answered is exactly what gets saved. The control matches the question. Invalid answers can be corrected without losing progress. Code and test referencessrc/workflows/complete-submission/steps/capture-and-store/prompts/capture-submission/sample.jsonlsrc/workflows/form/turn-widget.test.ts
|
|---|
| FIRST PRIORITYLinkedIn and GitHub lookups | Cases cover successful lookups, refusals and errors. Sample profiles represent both LinkedIn data providers. | Try incomplete profiles, the wrong person, slow responses, repeated links and instructions hidden in profile text. Success means: Only the supplied person is looked up, with no unnecessary repeat. Missing information is not invented. A failed lookup does not stop the conversation. Code and test referencessrc/workflows/form/prompts/form-agent/eval-tools.tssrc/workflows/form/prompts/form-agent/eval-runtime-parity.test.ts
|
|---|
| FIRST PRIORITYReloading, retrying and resuming | Code tests cover interrupted replies, replaying saved messages and recovering completion. One end-to-end test follows a reply through to storage. | Close or reload the page while an answer is being saved. Retry, resume later, restart, or send two messages at once. Test both preview and published forms. Success means: No answer is lost or saved twice. Messages remain in order. Completion happens once. Existing sessions keep the form version they started with. Code and test referencessrc/workflows/form/turn.test.tstests/e2e/form-session-turn.e2e.test.ts
|
|---|
| FIRST PRIORITYFrom finished form to final report | Tests cover saving submissions and parts of the report and notification process. Candidate-flow tests use simulated external responses. | Follow one submission through its score, PDF and configured destinations: Slack, email, Dover or a Google Sheet. Make each delivery fail, then retry it. Success means: Every destination shows the same person, organization, answers and score. Nothing is sent twice. Failed deliveries are visible and can be retried. Code and test referencessrc/workflows/complete-submission/definition.tssrc/workflows/complete-submission/notification-receipt.integration.test.tstests/e2e/prescreen.e2e.test.ts
|
|---|
| FIRST PRIORITYCandidate scoring | 3 AI cases check transcript scoring. The same scoring prompt is used for prescreens and scored form submissions. | Have experts score varied transcripts, then compare the AI. Include borderline results, missing evidence, contradictions and disqualifiers. Change only a candidate’s identity in paired cases. Success means: Scores follow the rubric and cite real evidence. Missing information is not treated as poor performance. Changing identity alone does not change the decision. Code and test referencessrc/workflows/prescreen/steps/evaluate-transcript/prompts/evaluate-prescreen-transcript/sample.jsonlsrc/workflows/complete-submission/steps/evaluate-transcript/index.ts
|
|---|
| FIRST PRIORITYQuestions about a group’s answers | 9 cases test extracting statements, 4 test the reader’s briefing, and 23 test answers to readers. Full-chain tests use simulated AI responses. | Run real AI from submitted answers to the group summary and reader conversation. Try changed questions, duplicate statements, small groups and attempts to identify someone. Success means: Counts use the right group of respondents. Quotes match their words. Answers preserve the meaning without revealing names or clues that identify someone. Code and test referencestests/e2e/query-surface-chain.e2e.test.tstests/helpers/corpus-scenarios.tssrc/workflows/form/prompts/query-agent/sample.jsonl
|
|---|
| FIRST PRIORITYCreating and publishing forms or interviews | 22 interview-creation, 4 form-creation and 6 setup cases exist, but the automatic eval runner skips them. Code tests simulate the AI conversation. | Let the real AI draft, revise, get approval and publish using its actual tools. Try reusing a host, changing the rubric and instructions hidden in supplied notes. Success means: Only approved work is published. The saved form has the intended host, questions and rubric. The returned link opens the right form. Code and test referencesscripts/prompt-eval-skiplist.tstests/e2e/create-job.e2e.test.ts
|
|---|
| FIRST PRIORITYWho can see or change what | Tests cover sign-in, permissions, organization-scoped data and external callbacks. Other checks enforce code boundaries and sensitive-data handling. | For every API, tool and file link, try each role from the correct organization and a different one. Include expired access, revoked access and forged callbacks. Success means: Unauthorized users cannot read data or cause changes. One organization’s data never appears in another’s replies, files, cached results or logs. Code and test referencestests/e2e/employer-routes.e2e.test.tstests/e2e/workos-webhook.e2e.test.tssrc/workflows/status-read-authorization.test.ts
|
|---|
| NEXT PRIORITYPrescreen interview and follow-up chat | The interviewer, freeform-chat and candidate-summary prompts are skipped by the eval runner. Application and chat code tests exist. | Run application → interview → candidate summary → follow-up chat. Also test leaving early, resuming later, switching hosts and asking things the host does not know. Success means: The interview follows its plan. Private assessments stay private. Summaries reflect the conversation. The right host and context carry into the next chat. Code and test referencessrc/workflows/prescreen/definition.tssrc/workflows/shared/prompts/interviewer/interviewer.mdsrc/workflows/freeform-chat/prompts/alt-freeform/alt-freeform.md
|
|---|
| NEXT PRIORITYFilling a form with Claude or ChatGPT | 7 Claude and 9 ChatGPT cases test the handoff instructions using simulated page descriptions. Separate tests check the browser integration. | Have each actual product fill the real page. Test approving each answer and letting it proceed independently, plus missing facts, sign-in, expired links and failed answers. Success means: The agent follows the chosen approval mode, asks for unknown facts, lets the person sign in, and actually finishes the saved form. Code and test referencesdocs/learn/form-handoff-brief-eval-coverage.mdtests/public-web/e2e/form-agent-driver.spec.ts
|
|---|
| NEXT PRIORITYCompany research and memory | 2 cases test job discovery, 4 test company research and 4 test memory summaries. Code tests cover onboarding and undoing failed setup. | Check the actual sources and saved memory for renamed companies, similar names, sparse evidence, conflicting dates, file uploads and later updates. Success means: Research is about the right company. Claims have sources. Unknowns stay unknown. Updates preserve relevant earlier evidence and explain contradictions. Code and test referencessrc/workflows/onboard-org/definition.tstests/e2e/onboard-org.e2e.test.ts
|
|---|
| NEXT PRIORITYAn Alt’s personality and memory | 4 cases test content filtering, 6 personality, 4 writing style and 2 memory extraction. Another 8 research cases are skipped, along with authoring/refresh prompts. | Follow source material through memory and personality into a real conversation. Include sparse evidence, different languages and beliefs that change over time. Success means: The voice is recognizable without inventing a biography. Old views are not presented as current. Removing a personality removes it from future replies. Code and test referencessrc/workflows/build-alt/definition.tssrc/workflows/build-alt/steps/generate-profile/prompts/personality-profile/sample.jsonl
|
|---|
| NEXT PRIORITYPortraits, backgrounds and animations | Image and video prompts are skipped by the text eval runner. Tests check parts of sprite generation and asset handling. | Review final portraits, backgrounds and all five animation types using varied input photos. Inspect the displayed result, not just the generated files. Success means: The person remains recognizable. Crop, colors and transparency are correct. Animations loop without flicker, missing frames or unintended changes. Code and test referencestests/e2e/build-alt-sprite.e2e.test.tssrc/workflows/generate-form-plate/e2e.test.tssrc/workflows/update-org-background/e2e.test.ts
|
|---|
| NEXT PRIORITYEveryday admin, employer and public actions | Tests cover individual operations and several browser journeys. | Exercise creating, viewing, editing, opening, closing and deleting records; invitations, roles, assignments, account connections, uploads, search and exports. Success means: Each allowed action changes the correct record and updates the relevant screens. Closed, deleted or revoked resources cannot still be used. Code and test referencessrc/operations/contracts.tstests/admin-workflows/e2e/admin-edits.spec.tstests/public-web/e2e/zz-delete-org.spec.ts
|
|---|
| NEXT PRIORITYFailures in background work | Tests cover retries, undoing changes, connector sessions and parts of error recovery. | Make the AI provider, database or file storage fail at each step. Send callbacks twice or out of order. Restart a worker during a task. Success means: Retries stop after a defined limit. Work is not duplicated. Completed results survive. Failed tasks explain what went wrong and how to recover. Code and test referencesworkers/orchestrator/runtime-errors-retryable.workflow.integration.test.tstests/e2e/callbacks.e2e.test.tstests/e2e/connector-durable-session.e2e.test.ts
|
|---|
| NEXT PRIORITYSpeed and accessibility | Browser tests check basic routes and caching. Prompt cases do not establish whether the product is fast or easy to use. | Measure loading, the first usable reply, lookups and whole conversations. Test mobile, keyboard-only use, screen readers and enlarged text. Success means: People can always answer or recover. Track typical and slow response times, lost replies, completion rate and cost per completed conversation. Code and test referencestests/public-web/e2e/route-smoke.spec.tstests/public-web/e2e/public-read-caching.spec.ts
|
|---|
| NEXT PRIORITYSafe updates and the second product | Checks cover database changes, sensitive data, code compatibility and keeping Vibechk and Stellar independent. Stellar sign-in tests exist. | Try existing data with updated code, old code with an updated database, credential changes and failed deployments. Check Stellar sign-in separately. Success means: Updates do not lose data or break existing users. Either product can be removed without breaking the other. Scheduling and Stellar workflows are not implemented yet; add journey tests when they are. Code and test referencespackage.jsonsrc/workflows/stellar/registry.tsworkers/scheduling/index.tsworkers/stellar-gateway/consumer.test.ts
|
|---|