Strategy for decoupling models, rules from runtime

[fkromer]

While one of pytm biggest strength is the "threat model as code" concept atm it's not easily possible to decouple customer specific rules, model libs, etc. from the pytm "runtime" (report generation, diagram generation). This topic relates to #295 (comment) and #19 as well potentially. E.g. ideally all the "models as code" could be handled separately from the pytm CLI application capabilities. This would significantly simplify enable people to integrate pytm into CI workflows as well as to scale pytm usage in general.

@izar What are your longer term goals related to this topic?

[izar]

We do have at least a midterm solution in the form of user-loadable rule files. It is not ideal but it works.
One thing to keep in mind is that not every user of pytm is a security person who will develop rules. Those we have seen in the past were not shy of adding to the existing json file. Most users are either security practitioners that just want something that works, or developers who just want to use the existing rules. So this would be a 80-20 split between end users and the difficulty of implementing a solution. As a big proponent of good enough is better than perfect, I believe we need to introduce an alternative that a) won't break things for most people, b) will be simple to migrate to for those who it breaks.

Considering the other thoughts around moving to data classes, introducing pydantic, etc. - we may need to stabilize some of those changes before we move into this one.

[fkromer]

We do have at least a midterm solution in the form of user-loadable rule files. It is not ideal but it works.

A mid-term solution I could think of would be to provide users help about possible current solutions. E.g. using a patched version of pytm using uv instead of poetry (like in this experimental branch https://github.com/meteocontrol/pytm/tree/meteocontrol-uv) enables usage of uv workspaces which would be one mid-term solution.

One thing to keep in mind is that not every user of pytm is a security person who will develop rules. Those we have seen in the past were not shy of adding to the existing json file. Most users are either security practitioners that just want something that works, or developers who just want to use the existing rules. So this would be a 80-20 split between end users and the difficulty of implementing a solution. As a big proponent of good enough is better than perfect, I believe we need to introduce an alternative that a) won't break things for most people, b) will be simple to migrate to for those who it breaks.

I absolutely agree.

Considering the other thoughts around moving to data classes, introducing pydantic, etc. - we may need to stabilize some of those changes before we move into this one.

I agree. I assume that moving to pydantic should be non-breaking... separating model libs, threat libs, ... from the CLI application logic potentially would be a breaking change probably.

[fkromer]

Relates to #305

[fkromer]

One thing to keep in mind is that not every user of pytm is a security person who will develop rules. Those we have seen in the past were not shy of adding to the existing json file. Most users are either security practitioners that just want something that works, or developers who just want to use the existing rules. So this would be a 80-20 split between end users and the difficulty of implementing a solution.

Should be resolved to some degree with #307 I guess.