Ryan Cotterell - Two views of a language model - IPAM at UCLA
Recorded 01 September 2026. Ryan Cotterell of ETH Zurich presents "Two views of a language model" at IPAM's Foundations of Interpretability Workshop.Abstract: A language model can be read in two ways. Statistically, it is a probability distribution over strings, estimated from finite data and judged by how well it predicts held-out text. Computationally, it is a model of computation—an object in the same lineage as automata and Turing machines—and the question is which algorithms it can implement. These views usually travel together, but they come apart precisely where it matters: on rare, structured, or out-of-distribution inputs, where good average-case prediction says nothing about whether the right algorithm is present. I will begin with a deep dive on this statistical–computational duality and argue that the computational view demands its own mathematical tools—drawn from logic, automata theory, and complexity.
The rest of the talk develops a computational account for the unique-hard-attention transformer (UHAT), a standard and analytically tractable abstraction of self-attention whose expressivity bounds are known to transfer to fixed-precision softmax transformers. On expressivity, fixed-precision UHATs recognize exactly the star-free languages—the class simultaneously captured by linear temporal logic (LTL), first-order logic with order, and counter-free automata. This places transformers strictly below recurrent neural networks (RNNs) as recognizers, which capture all regular languages. Raw expressivity, I will argue, is therefore the wrong lens: it declares transformers weaker than architectures they routinely outperform.
I then turn to succinctness—how compactly a formalism describes a language—as the sharper measure. The main results: UHATs can be exponentially more succinct than LTL and RNNs (hence also state-space models), and doubly exponentially more succinct than finite automata. There exist languages recognized by polynomial-size transformers whose smallest equivalent LTL formula or RNN is exponentially larger, and whose smallest automaton is doubly exponentially larger. I show that these bounds are tight. Finally, succinctness comes at a cost: I show that basic verification questions for transformers—emptiness, equivalence—are EXPSPACE-complete, so no efficient algorithm can reason about transformer behavior in general.
Learn more online at: https://www.ipam.ucla.edu/programs/workshops/foundations-of-interpretability/?tab=overview Receive SMS online on sms24.me
TubeReader video aggregator is a website that collects and organizes online videos from the YouTube source. Video aggregation is done for different purposes, and TubeReader take different approaches to achieve their purpose.
Our try to collect videos of high quality or interest for visitors to view; the collection may be made by editors or may be based on community votes.
Another method is to base the collection on those videos most viewed, either at the aggregator site or at various popular video hosting sites.
TubeReader site exists to allow users to collect their own sets of videos, for personal use as well as for browsing and viewing by others; TubeReader can develop online communities around video sharing.
Our site allow users to create a personalized video playlist, for personal use as well as for browsing and viewing by others.
@YouTubeReaderBot allows you to subscribe to Youtube channels.
By using @YouTubeReaderBot Bot you agree with YouTube Terms of Service.
Use the @YouTubeReaderBot telegram bot to be the first to be notified when new videos are released on your favorite channels.
Look for new videos or channels and share them with your friends.
You can start using our bot from this video, subscribe now to Ryan Cotterell - Two views of a language model - IPAM at UCLA
What is YouTube?
YouTube is a free video sharing website that makes it easy to watch online videos. You can even create and upload your own videos to share with others. Originally created in 2005, YouTube is now one of the most popular sites on the Web, with visitors watching around 6 billion hours of video every month.
![[Loose Ends]: Apple's Trick to Make You Pay Twice](https://img.youtube.com/vi/OJYDb1q3Yhc/maxresdefault.jpg)
