You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Update finished KV transfer state after every step (#532)
In P/D disaggregation scenario, most of time are decoding forward runs
in decode instances, we need update finished KV transfer states after
decode forward as well (not only prefill forward). Otherwise, even KV
transfer is already finished in prefill instance, while decode instance
cannot get finished state in time (switching state from
`WAITING_FOR_REMOTE_KVS` to `WAITING`) which will increase TTFT.
Signed-off-by: Wuxun Zhang <wuxun.zhang@intel.com>
0 commit comments