[Feature] Allow droping models between steps from memory altogether.

[ClearStaff]

Feature Summary

Add toggle that forces each model to be in memory just when needed.

Detailed Description

Hello, as you may know Apple Silicon MacBooks have unified memory architecture. Which basically means kicking models out of VRAM doesnt do anything, when they "return to RAM" on this devices those models still occupy unified memory.

Considering how fast SSD's are on this device it would be great to be able to load CLIP's, encode text, then unload CLIP, load for example WAN, create latent image, unload WAN, then finally load VAE, and generate output.

Unloading models completely changes game for MacBooks, instead of worrying that all you models combined + theirs context must take less space than is available your only limit would be current model + it's context + input from previous model.

I would tackle it, but i don't really use / know C++ i'm afraid, you can still point me in the right direction and i can try to see if i can do something alone if nobody else has time for this.

Alternatives you considered

No response

Additional context

No response

[stduhpf]

That's already the behavior in the main example. The API has a free_params_immediately flag, which is set to true in main.cpp.

The thing is that, while the models are being freed from memory as soon as they stop being used, they're all still loaded at the beginning, before they're actually needed.

The feature that you should be looking for is loading models just in time when you need them.

Edit: Wouldn't the model parts you're not actively using be automatically moved to SSD swap by the OS if you're running out of memory anyways?

[ClearStaff]

That's the problem, even tho models seems to be loaded as mmap first, while it gets used by GPU cores it gets fully loaded into unified memory, then they are pushed back to RAM after usage, not completely dropped (apparently still fully loaded in unified memory) , which overfeeds the unified memory, and forces system to SWAP unused models back to disk, which is absolutely wrong, cause now you have the model in a file on a APFS partition, and also in SWAP, effectively it is just forcing unnecessary writes to SSD, then goes also VAE step which requires crazy amount of memory, and VAE context sometimes gets wrongly swapped (instead of unused latent model for example, or CLIP) and this way somehow VAE steps sometimes gets much longer.

1. So we should double check if free_params_immediately actually frees them from unified memory while using Metal backend. (if so, then perhaps problem lies in macOS that maybe still holds recently used model as cached file..., which is dumb).
2. If we can create a new param called maybe load_models_jit, that loads the params only before they are used, so combining this two params should yield the desired outcome on Mac devices with low unified storage.

EDIT: And actually we should avoid SWAP altogether on Apple Silicon devices, those have nonreplaceable SSD's, so forcing unnecessary writes just kills the hardware quicker, and we speak about 40/50GB of swap at one run... The ideal situation should be loading models right before usage as Mac's SSD's are crazy fast, and ideally completely deleting model and unused parameters after step (like latent model, and CLIP encoding after latent step, before VAE). This way we could avoid SWAP if our model + calculations are not larger than our average free memory available.

[stduhpf]

I see. Maybe its a Metal backend or MacOS specific issue, because on my machine, when the model is unloaded from GPU, the RAM usage doesn't increase while the VRAM drops significantly, on both ROCm and Vulkan backends.

[MrSnichovitch]

Sorry to butt in here, but I'm curious since I haven't used a Mac for more than a decade. Does modern MacOS have an equivalent to the ol' standard sync; echo 1 > /proc/sys/vm/drop_caches one-liner in Linux? It's what I use to flush cached model data from system RAM when switching between large model sets (e.g., FLUX.1, Chroma, WAN), since I only have 48 GB available. Is sudo purge still a viable option with unified memory systems, or is it too broad-stroke to use without causing greater problems? Perhaps it would work as a temporary solution/workaround if "&&"-ed after an sd command string?

[remicar56]

That’s a solid observation — unified memory really changes how model management should be handled on Apple Silicon. The ability to fully unload models between tasks would make workflows far smoother and reduce those frustrating out-of-memory errors. Offloading intermediate data to SSD and reloading only what’s needed could definitely push efficiency closer to desktop GPU setups. It might not be trivial to implement in C++, but the idea makes perfect sense from a resource management perspective — modular loading could be a big win for M-series users.