About output head alignment

[congmonkey]

Thanks for the excellent work! I have a question: as we know, different hidden spaces have different dimensions and channel nums — how should the loss function be designed in the OutputHead Alignment training stage?

[CuddleSabe]

Thanks for the excellent work! I have a question: as we know, different hidden spaces have different dimensions and channel nums — how should the loss function be designed in the OutputHead Alignment training stage?

The dim of the embedder's output is same. eg:(f8c16p2)->(f32c32p1), linear(64, 4096) -> linear(32, 4096)

[yankainan]

Thanks for the excellent work! I have a question: as we know, different hidden spaces have different dimensions and channel nums — how should the loss function be designed in the OutputHead Alignment training stage?

The dim of the embedder's output is same. eg:(f8c16p2)->(f32c32p1), linear(64, 4096) -> linear(32, 4096)

how to train the output head?