You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Parallelize and simplify the asset processing loop. (#21701)
# Objective
- The previous asset processing loop was very difficult to understand
(IMO).
- The initial processing of tasks would start a bunch of tasks. Then
listening would listen for events and then await on processing tasks one
at a time before continuing to listen to events. Finishing a task would
also add paths to a **separate** `check_reprocess_queue` which would
only be checked after all the current events have been handled.
- Also processing tasks did not occur in parallel - so we'd process
assets one at a time.
## Solution
Approximately throw everything out. The asset processor now does these
things:
1. Initialize the processor: same as before, recover from the
transaction log, initialize the state of all processed assets (so we can
lock them).
2. Queue all the initial processing tasks: iterate through all processed
sources, finding all their assets, and queue a task for them (to recheck
whether they need to be processed, and reprocess them if so). Note we
don't spawn any bevy_task::Tasks here.
3. Spawn the "executor" bevy_task::Task: This task spawns the queued
tasks and updates the overall state of processing (i.e., processing vs
finished).
4. Spawn the source change event listeners: spawns a bevy_task::Task for
each asset source to listen on its event receiver and queued up any new
tasks as source assets change.
So this parallelizes event processing from asset sources, parallelizes
processing each asset, and (IMO) makes the whole processing loop much
simpler.
Also I think it's funny that parallelizing could make things simpler
lol.
### Caveats
- I've removed the public methods for `process_assets` and
`listen_for_source_change_events`. My guess is these were public so that
users can call them outside the context of a running app? I'm not
entirely sure. I think this needs to be rethought though if that's the
case. For one, a running app currently will not be gated on processing
from another app, meaning things will probably get out-of-sync very
easily. If need be, I think we can bring this back fairly straight
forward. There also isn't a migration guide since there's nothing to
migrate to here.
- Parallelizing asset processing **could** be bad for very large tasks.
Some GLTF files can get REALLY big, and managing memory there is very
important (though we're still bad at this). So parallelizing asset
processing can result in many tasks running concurrently consuming more
memory without a way to control it. However I think this is a more
general problem and we should find other solutions than "don't
parallelize".
## Testing
- The asset processing tests still pass!
- The asset_processing example seems to behave the same!
0 commit comments