[2/n] [Serve] Refactor replica rank to prepare for node local ranks #58473
+125
−70
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR refactors the replica rank system to support multi-dimensional ranking (global, node-level, and local ranks) in preparation for node-local rank tracking. The
ReplicaRankobject now contains three fields instead of being a simple integer, enabling better coordination of replicas across nodes.Motivation
Currently, Ray Serve only tracks a single global rank per replica. For advanced use cases like tensor parallelism, model sharding across nodes, and node-aware coordination, we need to track:
This PR lays the groundwork by introducing the expanded
ReplicaRankschema while maintaining backward compatibility in feature.Changes
Core Implementation
schema.py: ExtendedReplicaRankto includenode_rankandlocal_rankfields (currently set to -1 as placeholders)replica.py: Updated replica actors to handleReplicaRankobjectscontext.py: ChangedReplicaContext.ranktype fromOptional[int]toReplicaRankCurrent Behavior
node_rankandlocal_rankare set to-1(placeholder values). Will change in futureBreaking Changes
Rank is changing from
inttoReplicaRank