-
Recent Posts
- Sparks of Stupidity
- Guidance and Art.
- Advobots and AI Safety
- The Meaning is The Use.
- The Mental States of Language Models
- GPT-3 Semantic Parsing with Simple Sentence Graphs
- The Case Against Grounding
- Quo Vadis Language Model? Will I ever be able to trust you?
- Encoder Autonomy
- Reinterpreting AlphaZero
- Commonsense and Language Models
- A Consistency Theorem for BERT
- The Inevitability of Vector Quantization in Deep Architectures
- Thoughts from TTIC31230: Rate-Distortion Metrics for GANs.
- Rethinking Generalization.
- The Struggle for the Soul of SGD.
- Thoughts from TTIC31230: Hyperparameter Conjugacy
- Superintelligence and The Truth
- Predictive Coding and Mutual Information
- RL and the Game of Mathematics
Category Archives: Uncategorized
Sparks of Stupidity
It recently occurred to me that I might be able to teach GPT4 to add numbers of unbounded length just by describing ripple-carry addition (in context). I wanted this to be a test of GPT4’s intelligence rather than a test … Continue reading
Posted in Uncategorized
3 Comments
Guidance and Art.
Self-guidance, or simply guidance, is fundamental to systems for image generation including both diffusion-based image generators, such as the well known DALLE generator, and autoregressive image generators such as CM3leon. A technical definition of guidance is given below. While Guidance … Continue reading
Posted in Uncategorized
1 Comment
Advobots and AI Safety
In a 2014 post I argued for the “servant mission” as a solution to AI safety. The basic idea is that there are many AI agents each one of which has the mission of serving a particular person. I gave … Continue reading
Posted in Uncategorized
13 Comments
The Meaning is The Use.
Wittgentsein famously stated that “the meaning is the use”. Unfortunately no analytical explanation of this statement is provided in the posthumously published Philosophical Investigations. Wittgenstein’s point seems to be that language is not appropriately approached through logical or analytical means, … Continue reading
Posted in Uncategorized
Leave a comment
The Mental States of Language Models
Behaviorist psychologists refused to talk about mental states or internal computations in human brains on the grounds that everything should be understood in terms of stimulus-response relationships. On the other hand linguists and cognitive psychologists assume the existence of internal … Continue reading
Posted in Uncategorized
4 Comments
GPT-3 Semantic Parsing with Simple Sentence Graphs
For a couple years now Michael Douglas and I have been contemplating a meaning representation — simple sentence graphs — based on breaking sentences down into simple sentence components. Over the last week or so we have been playing with … Continue reading
Posted in Uncategorized
4 Comments
The Case Against Grounding
A recent NOEMA essay by Jacob Browning and Yann LeCun put forward the proposition that “an artificial intelligence system trained on words and sentences alone will never approximate human understanding”. I will refer to this claim as the grounding hypothesis … Continue reading
Posted in Uncategorized
5 Comments
Quo Vadis Language Model? Will I ever be able to trust you?
This spring I had a disagreement with an old friend about AGI. They claimed there was essentially no chance of AGI arriving in the next, say, fifty years. I have always said we just don’t know. They also wanted a … Continue reading
Posted in Uncategorized
Leave a comment
Encoder Autonomy
As in previous years, teaching my course on the fundamentals of deep learning has inspired some blog posts. This year I realized that VAEs are non-parametrically consistent as models of the observed data even when the encoder is held fixed … Continue reading
Posted in Uncategorized
Leave a comment
Reinterpreting AlphaZero
While teaching reinforcement learning I kept asking myself what AlphaZero teaches us about RL. That question has lead to this post. This post generalizes AlphaZero to a larger class of RL algorithms by reinterpreting AlphaZero’s policy network as a belief … Continue reading
Posted in Uncategorized
Leave a comment