HN
login
Pretraining Latent Information Feedback Transformers with Teacher Supervisionarxiv.org

Transformer language models (LMs) are feed-forward: deep-layer representations are never fed back to shallower layers, and the only pathway for information to flow downward across generation steps is the decoded token.…

2 points•gmays•about 4 hours ago•0 comments•
#information#teacher#feedback#transformers#latent#pretraining#supervision#pretraining latent

0 comments

No comments yet.

Related stories