@agates reported that he has 100+ items in his queue, and adding/removing items to his queue takes ~30 seconds.
My guess is that I'm doing some iteration through the queue rather than a bulk query (or some other more efficient process).
Ideally this can be fixed just through the backend API, but I'm not sure if the slow down is actually in the front-end handling.