- The Meaning is The Use.
- The Mental States of Language Models
- GPT-3 Semantic Parsing with Simple Sentence Graphs
- The Case Against Grounding
- Quo Vadis Language Model? Will I ever be able to trust you?
- Encoder Autonomy
- Reinterpreting AlphaZero
- Commonsense and Language Models
- A Consistency Theorem for BERT
- The Inevitability of Vector Quantization in Deep Architectures
- Thoughts from TTIC31230: Rate-Distortion Metrics for GANs.
- Thoughts from TTIC31230: Rethinking Generalization.
- Thoughts from TTIC31230: Langevin Dynamics and the Struggle for the Soul of SGD.
- Thoughts from TTIC31230: Hyperparameter Conjugacy
- Superintelligence and The Truth
- Predictive Coding and Mutual Information
- RL and the Game of Mathematics
- The Role of Theory in Deep Learning
- Choice as a Natural Kind Term
- CTC and the EG Algotithm: Discrete Latent Choices without Reinforcement Learning
Author Archives: McAllester
The Meaning is The Use.
Wittgentsein famously stated that “the meaning is the use”. Unfortunately no analytical explanation of this statement is provided in the posthumously published Philosophical Investigations. Wittgenstein’s point seems to be that language is not appropriately approached through logical or analytical means, … Continue reading
The Mental States of Language Models
Behaviorist psychologists refused to talk about mental states or internal computations in human brains on the grounds that everything should be understood in terms of stimulus-response relationships. On the other hand linguists and cognitive psychologists assume the existence of internal … Continue reading
Posted in Uncategorized 4 Comments
GPT-3 Semantic Parsing with Simple Sentence Graphs
For a couple years now Michael Douglas and I have been contemplating a meaning representation — simple sentence graphs — based on breaking sentences down into simple sentence components. Over the last week or so we have been playing with … Continue reading
Posted in Uncategorized 4 Comments
The Case Against Grounding
A recent NOEMA essay by Jacob Browning and Yann LeCun put forward the proposition that “an artificial intelligence system trained on words and sentences alone will never approximate human understanding”. I will refer to this claim as the grounding hypothesis … Continue reading
Posted in Uncategorized 5 Comments
Quo Vadis Language Model? Will I ever be able to trust you?
This spring I had a disagreement with an old friend about AGI. They claimed there was essentially no chance of AGI arriving in the next, say, fifty years. I have always said we just don’t know. They also wanted a … Continue reading
As in previous years, teaching my course on the fundamentals of deep learning has inspired some blog posts. This year I realized that VAEs are non-parametrically consistent as models of the observed data even when the encoder is held fixed … Continue reading
While teaching reinforcement learning I kept asking myself what AlphaZero teaches us about RL. That question has lead to this post. This post generalizes AlphaZero to a larger class of RL algorithms by reinterpreting AlphaZero’s policy network as a belief … Continue reading
Commonsense and Language Models
Modeling human commonsense reasoning has long been a goal of AI. From a 1998 interview with Marvin Minsky we have Marvin Minsky: We need common-sense knowledge – and programs that can use it. Common sense computing needs several ways of … Continue reading
A Consistency Theorem for BERT
Last week I saw Noah Smith at a meeting in Japan where he mentioned that BERT seemed related to pseudo-likelihood. After some thought I convinced myself that this observation should lead to a consistency theorem for BERT. With an appropriate … Continue reading
The Inevitability of Vector Quantization in Deep Architectures
I just read VQ-VAE-2 (Vector-Quantized – Variational AutoEncoders – 2) by Razavi et al. This paper gives very nice results on modeling image distributions with vector quantization. It dominates BigGANs under classification accuracy score (CAS) for class-conditional ImgeNet image generation. For reasons listed below … Continue reading
Posted in Uncategorized 1 Comment