[RAG] Test qwen3 reranker #2933

as-suvorov · 2025-10-28T15:14:13Z

Description

Ticket:

Fixes #(issue)

Checklist:

Tests have been updated or added to cover the new code
This patch fully addresses the ticket.
I have made corresponding changes to the documentation

as-suvorov · 2025-10-30T08:39:17Z

Windows tests pass:

Optimum: (1, 0.9991921782493591)
GenAI:   (1, 0.9991921782493591)
-----
Optimum: (3, 0.971849799156189)
GenAI:   (3, 0.971849799156189)
-----
Optimum: (2, 0.4064231514930725)
GenAI:   (2, 0.4064231812953949)
-----
Optimum: (0, 0.04272500053048134)
GenAI:   (0, 0.04272500425577164)
-----

Linux and ManyLinux fail:

Optimum: (1, 0.9991921782493591)
GenAI:   (1, 0.9991921782493591)
-----
Optimum: (3, 0.9718499183654785)
GenAI:   (3, 0.7178239226341248)
-----
Optimum: (2, 0.40641993284225464)
GenAI:   (2, 0.20836354792118073)
-----
Optimum: (0, 0.042725469917058945)
GenAI:   (0, 0.03615669161081314)
-----

Cannot reproduce locally with the same packages versions.

@yatarkan @sgonorov fyi

…nker_bad_sim

…nto as/rag_qwen3_reranker_bad_sim

as-suvorov · 2025-11-03T09:36:42Z

Not reproducible any more

as-suvorov · 2025-11-06T13:27:58Z

The issue seems to be with the padding. On win padding is left, on lin - right. Padding needs to be fixed as left.

…nto as/rag_qwen3_reranker_bad_sim

Copilot

Pull Request Overview

This PR adds support for configurable padding side (left/right) in the Qwen3 reranker and includes extensive debugging output for testing. The workflow files are temporarily modified to focus test execution on the specific reranker tests.

Adds padding_side configuration parameter to TextRerankPipeline::Config
Adds debug logging throughout the reranking pipeline
Modifies CI workflows to run only Qwen3 reranker tests with verbose output

Reviewed Changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
tests/python_tests/test_rag.py	Adds test cases for left/right padding and debug print statements
src/python/py_rag.cpp	Exposes `padding_side` config field to Python bindings
src/python/openvino_genai/py_openvino_genai.pyi	Updates type hints to include `padding_side` field
src/cpp/src/rag/text_rerank_pipeline.cpp	Implements padding_side handling and adds debug output
src/cpp/src/debug_utils.hpp	Enhances print_tensor utility with full-print option
src/cpp/include/openvino/genai/rag/text_rerank_pipeline.hpp	Adds `padding_side` field to Config struct
.github/workflows/windows.yml	Comments out most tests, focuses on Qwen3 reranker
.github/workflows/manylinux_2_28.yml	Comments out most tests, focuses on Qwen3 reranker
.github/workflows/linux.yml	Comments out most tests, focuses on Qwen3 reranker

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-11-06T14:20:11Z

tests/python_tests/test_rag.py

        truncation=True,
        return_tensors="pt",
    )
+    print(inputs["attention_mask"])


Debug print statement should be removed before merging to production. Consider using proper logging with configurable levels instead.

Suggested change

print(inputs["attention_mask"])

Copilot · 2025-11-06T14:20:12Z

tests/python_tests/test_rag.py

+    for opt, genai in zip(opt_result, genai_result):
+        print(f"Optimum: {opt}")
+        print(f"GenAI:   {genai}")
+        print("-----")


Debug print statements should be removed before merging to production. Consider using proper logging with configurable levels instead.

Copilot · 2025-11-06T14:20:12Z

tests/python_tests/test_rag.py

+    for opt, genai in zip(opt_result, genai_result):
+        print(f"Optimum: {opt}")
+        print(f"GenAI:   {genai}")
+        print("-----")


Debug print statements should be removed before merging to production. Consider using proper logging with configurable levels instead.

Copilot · 2025-11-06T14:20:12Z

tests/python_tests/test_rag.py

+        "top_n=4, padding_side=left",
+        "top_n=4, padding_side=right",


Inconsistent spacing in test ID. Line 548 uses no space after comma ("top_n=4,padding_side=left"), but this line uses a space. Test IDs should follow consistent formatting.

Suggested change

"top_n=4, padding_side=left",

"top_n=4, padding_side=right",

"top_n=4,padding_side=left",

"top_n=4,padding_side=right",

Copilot · 2025-11-06T14:20:13Z

src/cpp/src/rag/text_rerank_pipeline.cpp

+        std::cout << "Start reranking..." << std::endl;
+        std::cout << "Query: " << query << std::endl;
+        for (size_t i = 0; i < texts.size(); i++) {
+            std::cout << "Text[" << i << "]: " << texts[i] << std::endl;
+        }


Debug output statements should be removed before merging to production. Consider using proper logging mechanisms instead of direct stdout writes.

Copilot · 2025-11-06T14:20:13Z

src/cpp/src/rag/text_rerank_pipeline.cpp


+        print_tensor("scores", scores_tensor);


Debug tensor printing should be removed before merging to production. Consider using conditional compilation or logging levels for debug output.

Suggested change

print_tensor("scores", scores_tensor);

#ifdef DEBUG

print_tensor("scores", scores_tensor);

#endif

Copilot · 2025-11-06T14:20:14Z

src/cpp/src/rag/text_rerank_pipeline.cpp

+        std::string padding_side_val;
+        ov::genai::utils::read_anymap_param(m_tokenization_params, padding_side.name(), padding_side_val);
+
+        std::cout << "Requested padding_side: " << (padding_side_val.empty() ? "not set" : padding_side_val) << std::endl;


Debug output statement should be removed before merging to production. Consider using proper logging mechanisms instead of direct stdout writes.

Suggested change

std::cout << "Requested padding_side: " << (padding_side_val.empty() ? "not set" : padding_side_val) << std::endl;

DEBUG_LOG("Requested padding_side: {}", (padding_side_val.empty() ? "not set" : padding_side_val));

Copilot · 2025-11-06T14:20:14Z

.github/workflows/windows.yml

+          #   timeout: 60
          - name: 'Rag tests'
-            cmd: 'python -m pytest -s -v tests/python_tests/test_rag.py'
+            cmd: 'pip list; python -m pytest -s -v tests/python_tests/test_rag.py -k test_qwen3_rerank_documents'


Workflow configuration has been modified to only run specific tests. This change should be reverted before merging to ensure full test coverage in CI.

Copilot · 2025-11-06T14:20:14Z

.github/workflows/manylinux_2_28.yml

+          #   timeout: 60
          - name: 'Rag tests'
-            cmd: 'python -m pytest -v ./tests/python_tests/test_rag.py'
+            cmd: 'python -m pytest -s -v ./tests/python_tests/test_rag.py -k test_qwen3_rerank_documents'


Workflow configuration has been modified to only run specific tests. This change should be reverted before merging to ensure full test coverage in CI.

Copilot · 2025-11-06T14:20:15Z

.github/workflows/linux.yml

+          #   timeout: 60
          - name: 'Rag tests'
-            cmd: 'python -m pytest -v ./tests/python_tests/test_rag.py'
+            cmd: 'pip list && python -m pytest -s -v ./tests/python_tests/test_rag.py -k test_qwen3_rerank_documents'


Workflow configuration has been modified to only run specific tests. This change should be reverted before merging to ensure full test coverage in CI.

as-suvorov · 2025-11-06T15:06:58Z

@apaniukov On linux hf optimum default padding is left, for genai tokenizer default padding is right.
https://github.com/openvinotoolkit/openvino.genai/actions/runs/19138633911/job/54701069138?pr=2933#step:6:885

Print pip && print results

08828da

github-actions bot added category: GHA CI based on Github actions category: GGUF GGUF file reader category: RAG RAG pipeline components labels Oct 28, 2025

as-suvorov added do_not_merge do_not_review and removed category: GHA CI based on Github actions category: GGUF GGUF file reader category: RAG RAG pipeline components labels Oct 28, 2025

as-suvorov changed the title ~~[RAG] Test qwen3 reranker results~~ [RAG] Test qwen3 reranker Oct 28, 2025

add debug prints

93be0f2

github-actions bot added category: GHA CI based on Github actions no-match-files category: GGUF GGUF file reader category: RAG RAG pipeline components labels Oct 30, 2025

as-suvorov added 5 commits October 31, 2025 10:49

Merge branch 'master' into as/rag_qwen3_reranker_bad_sim

1f72f6e

Fix printing

5c0b0c1

Merge remote-tracking branch 'upstream/master' into as/rag_qwen3_rera…

8c73764

…nker_bad_sim

Merge remote-tracking branch 'origin/as/rag_qwen3_reranker_bad_sim' i…

5eef89a

…nto as/rag_qwen3_reranker_bad_sim

Log in pytest

53578e0

as-suvorov closed this Nov 3, 2025

as-suvorov reopened this Nov 6, 2025

Merge branch 'master' into as/rag_qwen3_reranker_bad_sim

cc4515c

as-suvorov added 2 commits November 6, 2025 15:16

Add padding_side for TextRerankPipeline

a398ff0

Merge remote-tracking branch 'origin/as/rag_qwen3_reranker_bad_sim' i…

ee39432

…nto as/rag_qwen3_reranker_bad_sim

Copilot AI review requested due to automatic review settings November 6, 2025 14:16

github-actions bot added the category: Python API Python API for GenAI label Nov 6, 2025

github-actions bot added the category: CPP API Changes in GenAI C++ public headers label Nov 6, 2025

Copilot AI reviewed Nov 6, 2025

View reviewed changes

	std::cout << "Requested padding_side: " << (padding_side_val.empty() ? "not set" : padding_side_val) << std::endl;
	DEBUG_LOG("Requested padding_side: {}", (padding_side_val.empty() ? "not set" : padding_side_val));

[RAG] Test qwen3 reranker #2933

Are you sure you want to change the base?

[RAG] Test qwen3 reranker #2933

Conversation

as-suvorov commented Oct 28, 2025

Description

Checklist:

Uh oh!

as-suvorov commented Oct 30, 2025

Uh oh!

as-suvorov commented Nov 3, 2025

Uh oh!

as-suvorov commented Nov 6, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

as-suvorov commented Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

as-suvorov commented Nov 6, 2025 •

edited

Loading