Why is it necessary to specify subcommands in the yaml config for the model in CLI mode?

[nicoloesch]

I am currently upgrading from 1.9.4 to 2.x of pytorch_lightning, which includes the removal of the legacy argparse (e.g. Trainer.from_argparse_args). As such I have set up the following test envrionment mimicking my project structure, however with simple modules.

Test Project Structure

In my bigger project, I have an (partial) abstract base class inheriting from LightningModule, which implements general functionality required for both child modules (hence why I specify here ChildModuleA and ChildModuleB).

from pytorch_lightning.demos.boring_classes import BoringModel

class BaseModel(BoringModel):
    def __init__(self,
                 base: int):
        super().__init__()
        self._base = base

class ChildModelA(BaseModel):
    def __init__(self,
                 base: int,
                 child: Optional[int] = None):
        super().__init__(base)
        if child is None:
            child = "None"
        self._child = child
        print(f"A,base| {self._base}")
        print(f"A,child| {self._child}")

class ChildModelB(BaseModel):
    def __init__(self,
                 base: int,
                 child: Optional[int] = None):
        super().__init__(base)
        if child is None:
            child = "Null"
        self._child = child
        print(f"B,base| {self._base}")
        print(f"B,child| {self._child}")

and the instantiation of the CLI as follows:

    cli = LightningCLI(model_class=test_lightning.BaseModel,
                       datamodule_class=BoringDataModule,
                       subclass_mode_model=True,
                       parser_kwargs={'default_config_files': ["test_cfg.yaml"]})

The test_cfg.yaml follows the documentation and is defined as follows:

model:
  class_path: test_lightning.base_model.BaseModel
  init_args:
    base: 2
    child: 5

Problem

However, this configuration works ONLY when run=False within the CLI (and omitting the subcommand for execution). As soon as I try to run the file with python main.py fit --model ChildModelA I receive the following error message:

argparse.ArgumentError: Problem in default config file "test_cfg.yaml": "subcommand" is required but not given or its value is None.

Question

My question now is: Why do I need to specify the subcommands within the .yaml file resulting in an exact duplicate for both fit (and all other subcommands)?

With the following test_cfg.yaml, I can run the command (but the duplication is quite obvious, which becomes even more of a problem with my models and the larger amount of hparams...):

fit:
  model:
    class_path: test_lightning.base_model.BaseModel
    init_args:
      base: 1
      child: 2
test:
  model:
    class_path: test_lightning.base_model.BaseModel
    init_args:
      base: 1
      child: 2

Wouldn't it suffice to have a single configuration that applies to all subcommands and only have the subcommands IF an individual behaviour is wanted?

I am clearly doing something wrong here but I was not able to distinguish which behaviour is required based on the Documentation.

I appreciate any help or a hint to the right direction!

Cheers,
Nico

Envrionment

• pytorch_lighting: 2.0.0 (installed with [extra])
• python: 3.9
• torch: 1.13.1 (intention to upgrade to 2.x)
• jsonargparse: 4.20.0
• lightning-utilities: 0.8.0

[mauvilsa]

There are many possibilities to provide settings to a CLI, and having one yaml where you can specify all subcommand is just one of them. With run=True you can give to the CLI a single config that does not require a subcommand as:

python main.py fit --config=test_cfg.yaml

Note that in the command, the config is given after fit. The subcommand is already known, so the config is expected to not specify it. If the command is

python main.py --config=test_cfg.yaml fit

i.e. the config is given before fit, then the config must specify the subcommand. Similarly it is possible to specify default_config_files for each subcommand (see lightning_cli_advanced_2.html#set-default-config-files), in which case the respective default configs are expected to be like with run=False.

[nicoloesch]

Thanks for your answer! So the best practice would be to either have all configurations in a single .yaml file and pass it through --config or utilise run=False with the default_config_files and manually call cli.trainer.fit(cli.model) as described here?

Just out of curiosity: Why is there a differentiation between passing the config through --config and passing it through default_config_files? It seems like the assumption of replacing --config through passing default arguments does not hold as the run argument needs to be modified according to which method one utilises

[mauvilsa]

There is no best practice for this. You do what best addresses your use case. run=True is less boilerplate, specially if you want to use the other subcommands.

There is no differentiation between --config and default_config_files. Both can be global for all subcommands (in which case the config requires subcommand nesting), or for specific subcommands (no subcommand nesting), see the second option in the link that I sent, i.e. parser_kwargs={"fit": {"default_config_files": ["my_fit_defaults.yaml"]}}.

[nicoloesch]

I assumed it is possible to utilise a single configuration without subcommand nesting in the .yaml file whilst simultaneously supporting all subcommands, i.e. pytorch_lightning 'copies' the configuration under the hood for all subcommands if only a single configuration file without nesting is provided. Nested parser_kwargs are the solution for my specific use case!

Thanks for the clarification!