Transparency
Methodology
SameAsk helps people pick tools by need, then measures answer similarity across repeated runs of the same prompt. That is not a quality or IQ score.
What we measure
- Same prompt, N runs (2–5), default temperature 0.3 for Live tests
- Pairwise similarity between every pair of successful answers
- Reported score = mean pairwise similarity; we also store min / max / stdev of pairs
- With OpenRouter or OpenAI keys:
text-embedding-3-smallcosine (70%) blended with paraphrase-aware lexical score (30%) - Without embeddings: content-word Jaccard + bigrams + bag-of-words cosine
What we do not measure
- Factual correctness or hallucination rate
- Coding quality, helpfulness, or style preference
- Arena Elo / benchmark leaderboard rank
- “Best model overall” — only “closest answers on this prompt under these settings”
Limits
- 3 runs is directional; 5 runs is stabler but still not a lab eval
- A mediocre answer repeated verbatim can outrank a better answer that rephrases
- Demo mode uses fixed canned variance to teach the idea
- Find-tab “fit” scores are curated matching — separate from Live
Privacy
API keys stay in your browser localStorage and are sent only to the providers you choose for that request. SameAsk does not store keys on a server. See Privacy.