[wwb] Update reranker/embedder tests #2983
Conversation
sbalandi
force-pushed
the
qwen-cls-wwb
branch
from
a1c00ca to
b148c37
Compare
sbalandi
force-pushed
the
qwen-cls-wwb
branch
from
b148c37 to
8331259
Compare
|
@as-suvorov could you please take a look ? |
| similarity = get_similarity(outputs) | ||
| assert similarity >= SIMILARITY_THRESHOLD | ||
|
|
||
| remove_artifacts(outputs_path.as_posix()) |
There was a problem hiding this comment.
Why .as_posix() needed here?
sbalandi
force-pushed
the
qwen-cls-wwb
branch
from
c1008b5 to
6b18943
Compare
| subprocess.run(["optimum-cli", "export", "openvino", "--model", model_id, MODEL_PATH, "--task", task, "--trust-remote-code"], | ||
| capture_output=True, | ||
| text=True) | ||
| def remove_artifacts(artifacts_path: Path, file_type="outputs"): |
There was a problem hiding this comment.
Use file_type or remove from arguments
| def remove_artifacts(artifacts_path: Path, file_type="outputs"): | |
| def remove_artifacts(artifacts_path: Path): |
There was a problem hiding this comment.
Pull Request Overview
This PR updates the embedding and reranking test pipelines, primarily switching from PyTorch's cosine similarity to scikit-learn's implementation to avoid floating-point precision issues that can occur when converting tensors to scalars.
Key changes:
- Replaced PyTorch's
F.cosine_similaritywith sklearn'scosine_similarityfor more consistent numerical results - Restructured test files to add similarity threshold checks and improved artifact cleanup
- Updated logging format for similarity metrics to use consistent precision formatting
Reviewed Changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| whowhatbench/wwb.py | Updated similarity metric logging to use formatted precision (.5) and renamed score list output field |
| whowhatbench/whowhat_metrics.py | Switched from PyTorch to sklearn for cosine similarity calculation and added .item() call for score extraction |
| tests/test_cli_reranking.py | Refactored test structure with new helper functions, added similarity threshold assertions, and improved artifact cleanup |
| tests/test_cli_embeddings.py | Added similarity threshold checks, artifact cleanup helper, and additional output validation assertions |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if document_idx in prediction_scores: | ||
| scores_diff = abs(gold_score - prediction_scores[document_idx]) | ||
| per_query_text.append(scores_diff) | ||
| per_query_text.append(scores_diff.item()) |
There was a problem hiding this comment.
Calling .item() on scores_diff assumes it's a tensor, but scores_diff is set to self.MISSING_DOCUMENT_PENALTY when the document is missing, which is likely a numeric constant (int/float). This will raise an AttributeError. Only call .item() when scores_diff comes from prediction_scores[document_idx].
| per_query_text.append(scores_diff.item()) | |
| if isinstance(scores_diff, torch.Tensor): | |
| per_query_text.append(scores_diff.item()) | |
| else: | |
| per_query_text.append(scores_diff) |
| logger.info("## Similarity:\n%s\n", e["similarity"]) | ||
| logger.info("## Top_n scores:\n%s\n", e["per_text_score_list"]) | ||
| logger.info(f"## Similarity:\n{e['similarity']:.5}\n") | ||
| logger.info("## Difference in scores pre texts:\n%s\n", e['per_text_scores_diff']) |
There was a problem hiding this comment.
Typo in log message: 'pre texts' should be 'per texts'.
| logger.info("## Difference in scores pre texts:\n%s\n", e['per_text_scores_diff']) | |
| logger.info("## Difference in scores per texts:\n%s\n", e['per_text_scores_diff']) |
2 participants
Description
CVS-###
Fixes #(issue)
Checklist: