Add support for export api #1171
Conversation
WalkthroughAdds an exports path and a new Client.export(...) method, introduces a second Meilisearch instance and client support in tests (client2), adds end-to-end export tests against the secondary instance, and provides a code sample demonstrating an export request. Changes
Sequence Diagram(s)sequenceDiagram
autonumber
participant Test as Test runner
participant Client as Client.export()
participant Source as Meilisearch A
participant Target as Meilisearch B
Test->>Client: call export(url, api_key, payload_size, indexes)
activate Client
Client->>Client: build payload (url + optional fields)
Client->>Source: POST /exports (payload)
activate Source
Source-->>Client: TaskInfo (task created)
deactivate Source
Client-->>Test: return TaskInfo
deactivate Client
Note right of Test: Poll task until completion
Test->>Target: Query indexes / index details
activate Target
Target-->>Test: index metadata & document counts
deactivate Target
Note over Test,Target: Verify UID, primary key, document count match source
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes
Poem
Pre-merge checks and finishing touches✅ Passed checks (3 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (1)
tests/client/test_client_exports.py (1)
13-23: Consider using consistent property access.The test logic is correct and appropriately verifies the export functionality. However, there's an inconsistency: line 22 uses
index.get_primary_key()(method call) while line 21 usesindex2.primary_key(property access). Both work, but consistency would improve readability.Consider using the property consistently:
index2 = client2.get_index(index.uid) assert index2.uid == index.uid - assert index2.primary_key == index.get_primary_key() + assert index2.primary_key == index.primary_key assert index2.get_documents().total == index.get_documents().total
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (7)
.code-samples.meilisearch.yaml(1 hunks)docker-compose.yml(2 hunks)meilisearch/client.py(1 hunks)meilisearch/config.py(1 hunks)tests/client/test_client_exports.py(1 hunks)tests/common.py(1 hunks)tests/conftest.py(3 hunks)
🧰 Additional context used
🧬 Code graph analysis (2)
meilisearch/client.py (2)
meilisearch/models/task.py (1)
TaskInfo(77-106)meilisearch/_httprequests.py (1)
post(98-108)
tests/client/test_client_exports.py (3)
tests/conftest.py (2)
client(16-17)client2(21-22)meilisearch/client.py (5)
index(227-243)export(630-679)wait_for_task(801-828)get_index(185-204)get_indexes(128-159)meilisearch/index.py (2)
get_primary_key(145-153)get_documents(387-425)
🪛 GitHub Actions: Tests
tests/conftest.py
[error] 1-1: isort check failed: Imports are incorrectly sorted and/or formatted.
🔇 Additional comments (9)
meilisearch/config.py (1)
51-51: LGTM!The new
exportspath constant follows the established pattern and correctly maps to the "export" endpoint.docker-compose.yml (2)
9-15: LGTM!The integration of the second Meilisearch instance into the package service is correctly configured with the environment variable and dependency links.
27-34: LGTM!The second Meilisearch service configuration mirrors the primary instance appropriately for test purposes. The port mapping 7701:7700 allows external access for testing.
tests/common.py (1)
5-5: LGTM!The
BASE_URL_2constant follows the same pattern asBASE_URLand correctly defaults to port 7701, matching the docker-compose configuration.tests/conftest.py (3)
20-32: LGTM!The
client2fixture and_clear_indexeshelper function are well-structured and follow the existing patterns in the test suite.
35-44: LGTM!The conditional cleanup logic for the second client is appropriate—it only attempts cleanup when
MEILISEARCH_URL_2is configured, preventing errors when the second server isn't available.
63-70: LGTM!The
clear_all_tasksfixture correctly handles cleanup for both clients when the second server is configured.tests/client/test_client_exports.py (2)
7-10: LGTM!The skip marker appropriately gates these tests behind the
MEILISEARCH_URL_2environment variable, ensuring they only run when a second server is configured.
26-42: LGTM!The test correctly verifies that the export respects index filtering, ensuring only the specified index is exported to the second server.
| export_post_1: |- | ||
| client.export( | ||
| url='https://remote-meilisearch-instance.com', | ||
| api_key='masterKey', | ||
| pay_load_size='50MiB', | ||
| indexes={ | ||
| 'movies*': {}, | ||
| 'books*': {}, | ||
| }, | ||
| ) |
There was a problem hiding this comment.
🧩 Analysis chain
Script executed:
#!/bin/bash
# First, locate the client.py file
fd -t f -n "client.py" .Repository: meilisearch/meilisearch-python
<shell_result>
Repository: meilisearch/meilisearch-python
Script execution failed with unrecoverable error: [CAPABILITY:RUN_SHELL_SCRIPT] Failed to clone repository meilisearch/meilisearch-python: Clone operation failed
</shell_result>
Meilisearch export API payloadSize parameter documentation
<web_search_result>
The export API accepts a payloadSize field that controls the maximum size of each data payload sent during an export. Key points from the docs:
- Type: string (human‑readable size). Example formats: "50 MiB", "100MiB". [1]
- Default: "50 MiB". [1]
- Meaning: maximum size of each single data payload; larger payloads are more efficient but need more RAM/CPU. [1]
- Usage example in request body: "payloadSize": "123 MiB". [2][3]
Sources: Meilisearch Export API docs and release notes. [1][2][3]
References
[1] Meilisearch Export API docs (payloadSize description).
[2] Meilisearch release notes (v1.16 export route examples).
[3] Meilisearch Python SDK transfer_documents (payload_size default and usage).
</web_search_result>
Fix parameter name and value format: pay_load_size='50MiB' should be payload_size='50 MiB'.
The parameter name pay_load_size is incorrect. Per Python SDK conventions, it should be payload_size (treating "payload" as a single word). Additionally, the value format is incorrect—the Meilisearch API expects a space in the size format: "50 MiB" rather than "50MiB".
🤖 Prompt for AI Agents
In .code-samples.meilisearch.yaml around lines 517 to 526, the export call uses
an incorrect parameter name and value format: change the parameter from
pay_load_size to payload_size and format the size string with a space (e.g., "50
MiB"). Update the export invocation to pass payload_size="50 MiB" (keeping other
args intact) so it matches the Python SDK/Meilisearch API expected naming and
value format.
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (2)
tests/conftest.py (2)
20-23: Consider makingclient2explicitly conditional on second-instance configuration.Right now
client2is always constructed, even when a second Meilisearch instance is not configured; usage is guarded later byos.getenv("MEILISEARCH_URL_2"), but a misconfiguredBASE_URL_2could still yield confusing failures when a test directly depends onclient2. You could optionally make this fixture raise/skip with a clear message when the second instance is not configured (e.g., checkMEILISEARCH_URL_2orcommon.BASE_URL_2and callpytest.skip).
25-32: Handle potential pagination in_clear_indexesand reuse existingIndexinstances.
client.get_indexes()returns a dict with aresultslist ofIndexinstances built from the paginated/indexesendpoint; the HTTP layer uses Meilisearch’s default pagination (limit=20) when no parameters are provided.(python-sdk.meilisearch.com) If a test run ever generates more thanlimitindexes,_clear_indexeswill only delete the first page despite the “Deletes all the indexes” docstring.It would be more robust to either:
- Loop over pages using
offset/limituntilresultsis empty, or- At least request a higher
limitthat safely covers expected test indexes.Since
get_indexesalready returnsIndexobjects, you can also simplify the loop by callingindex.delete()directly instead of recreating them viaclient.index(index.uid).(python-sdk.meilisearch.com)Example of a more robust approach (for illustration):
-def _clear_indexes(meilisearch_client): - """Deletes all the indexes in the Meilisearch instance.""" - - indexes = meilisearch_client.get_indexes() - for index in indexes["results"]: - task = meilisearch_client.index(index.uid).delete() - meilisearch_client.wait_for_task(task.task_uid) +def _clear_indexes(meilisearch_client): + """Deletes all the indexes in the Meilisearch instance (handles pagination).""" + + offset, limit = 0, 1000 + while True: + page = meilisearch_client.get_indexes({"offset": offset, "limit": limit}) + results = page["results"] + if not results: + break + for index in results: + task = index.delete() + meilisearch_client.wait_for_task(task.task_uid) + offset += limitAlso applies to: 35-36, 40-44
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
tests/conftest.py(3 hunks)
🧰 Additional context used
🧬 Code graph analysis (1)
tests/conftest.py (3)
meilisearch/client.py (3)
index(227-243)wait_for_task(801-828)delete_tasks(782-799)meilisearch/_httprequests.py (1)
delete(142-147)meilisearch/index.py (2)
delete(88-105)wait_for_task(233-260)
🔇 Additional comments (2)
tests/conftest.py (2)
2-4: Stdlib import ofosis correctly placed.
osis now grouped with the other stdlib imports and used below forMEILISEARCH_URL_2checks; no issues from a style or functionality perspective.
63-70: Symmetric task cleanup forclient2looks good.Extending
clear_all_tasksto delete finished tasks onclient2whenMEILISEARCH_URL_2is set keeps task state consistent across both instances and matches the behavior introduced inclear_indexes.
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
meilisearch/client.py (1)
630-679:exportimplementation looks correct; consider tighteningindexesdoc wording.The method’s behavior (payload construction, field names, and return type) is consistent with other client methods and the Export API, and I don’t see functional issues here.
One small optional improvement: the docstring for
indexessays “A set of objects…”, but the type isMapping[str, Any]. To avoid confusion with Pythonset, you might rephrase to something like “A mapping/dict whose keys correspond to patterns matching the indexes you want to export (see Export API docs for structure).”
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
.code-samples.meilisearch.yaml(1 hunks)meilisearch/client.py(1 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
- .code-samples.meilisearch.yaml
🧰 Additional context used
🧬 Code graph analysis (1)
meilisearch/client.py (2)
meilisearch/models/task.py (1)
TaskInfo(77-106)meilisearch/_httprequests.py (1)
post(98-108)
|
@Strift not sure whether we should wait for meilisearch/meilisearch#6009 or let it go. Locally I was experiencing the same issue and it took quite of my time but here test cases are passing. |
|
Hi @jawad-khan and thanks for this PR 🙏 It's best to hold off until the issue is fixed. Otherwise, we would just be setting ourselves up to get reports in this repository that are unrelated to the SDK 😅 |
2 participants
Pull Request
Added support for the Export API as described in the official Meilisearch docs.
Related issue
Fixes # 1129
PR checklist
Please check if your PR fulfills the following requirements:
Thank you so much for contributing to Meilisearch!
Summary by CodeRabbit
New Features
Tests
Chores
✏️ Tip: You can customize this high-level summary in your review settings.