Loading several checkpoints gives and error

[YerePhy]

Hi,

I am trying to load several checkpoints in order to make an ensemble-like prediction. The __init__ of my LightningModule looks like this:

class VolumetricSemanticSegmentator(pl.LightningModule):
    """
    LightningModule for semantic segmentation models enhanced with MONAI features.
    """

    def __init__(
        self,
        architecture: torch.nn.Module,
        loss: torch.nn.Module,
        optimizer: str,
        optimizer_configuration: Mapping[str, Any],
        postprocessing: monai.transforms.Compose,
        sw_kwargs_test: Mapping[str, Any],
        patch_predictor: Callable[[torch.Tensor, torch.nn.Module], torch.Tensor],
        scheduler: Optional[str] = None,
        scheduler_configuration: Optional[Mapping[str, Any]] = None,
        metrics_evaluator: Evaluator = None,
        train_gpu_augmentations: Optional[torch.nn.Module] = None,
    ) -> None:
        super(VolumetricSemanticSegmentator, self).__init__()
        self.architechture = architecture
        self.loss = loss
        self.scheduler = scheduler
        self.scheduler_configuration = scheduler_configuration
        self.optimizer = optimizer
        self.optimizer_configuration = optimizer_configuration
        self.patch_predictor = patch_predictor
        self.postprocessing = postprocessing
        self.metrics_evaluator = metrics_evaluator
        self.sw_kwargs_test = sw_kwargs_test
        self.train_gpu_augmentations = train_gpu_augmentations
        self.save_hyperparameters(ignore=["architecture", "loss"])

I'm loading the checkpoints like this:

model_cls = cfg.lightningmodule.pop("lightningmodule_type")  # here, I get the model class (no an instance)
model_cfg = cfg.lightningmodule  # here, I get all the arguments in the __init__ instantiated (with facebook-hydra)

# here, I load the checkpoints with the parameters
# of the `__init__` method, I can't do 
# `model_cls.load_from_checkpoint(p)` because 
# it throws a missing parameters error
ensemble = [model_cls.load_from_checkpoint(p, **model_cfg) for p in ckpts]  

Then I get the following error:

Exception ignored in: 
Traceback (most recent call last):
  File "/home/yere/anaconda3/envs/iq_segmentator/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1358, in __del__
    self._shutdown_workers()
  File "/home/yere/anaconda3/envs/iq_segmentator/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1289, in _shutdown_workers
    if not self._shutdown:
AttributeError: '_MultiProcessingDataLoaderIter' object has no attribute '_shutdown'
Exception ignored in: 
Traceback (most recent call last):
  File "/home/yere/anaconda3/envs/iq_segmentator/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1358, in __del__
    self._shutdown_workers()
  File "/home/yere/anaconda3/envs/iq_segmentator/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1289, in _shutdown_workers
    if not self._shutdown:
AttributeError: '_MultiProcessingDataLoaderIter' object has no attribute '_shutdown'

Very weird since I am not using any dataloader in my script.

[akihironitta]

It looks to me that the issue is specific to neither "several checkpoints" nor PyTorch Lightning itself.

• Have you tried running it with a single checkpoint?
• Is it the whole error message?
• Are you able to reproduce the behaviour with the BoringModel? https://github.com/Lightning-AI/lightning/tree/master/examples/pl_bug_report

[YerePhy]

Hi @akihironitta,

Finally I solved the problem by saving all the parameters in __init__ explicitly with save_hyperparameters, even architecture and loss which are torch.nn.Module child classes (in theory, they are saved automatically). If I don't save them explicitly, loss is not saved and throws and error when loading the checkpoint. That error is solved passing loss when loading from checkpoint but then I get the error described. I you want me to try with the boring model let me know, this way we could catch a bug.