
Recent Posts
 The Role of Theory in Deep Learning
 Choice as a Natural Kind Term
 CTC and the EG Algotithm: Discrete Latent Choices without Reinforcement Learning
 VAE = EM
 Deep Meaning Beyond Thought Vectors
 The Plausibility of NearTerm Machine Sentience.
 Formalism, Platonism and Mentalese
 Comprehension Based Language Modeling
 Cognitive Architectures
 Architectures and Language Instincts
 Why we need a new foundation of mathematics
 The Foundations of Mathematics.
 Friendly AI and the Servant Mission
 AI and Free Will: When do choices exist?
 Metaphor and Mathematics
 Chomsky vs. Hinton
 The Free Lunch Theorem
 Situations
 Tarski and Mentalese
 Wittgenstein and Mentalese
Author Archives: McAllester
The Role of Theory in Deep Learning
This blog post is inspired by the recent NIPS talk by Ali Rahimi and the response by Yann LeCun. The issue is fundamentally the role of theory in deep learning. I will start with some quotes from Rahimi’s talk. Rahimi: … Continue reading
Posted in Uncategorized
5 Comments
Choice as a Natural Kind Term
This is a sequel to my previous post on determinism, free will and the existence of choice. Here I want to consider the semantics of the word “choice” from the perspective of lexical semantics generally. I will focus on the … Continue reading
Posted in Uncategorized
Leave a comment
CTC and the EG Algotithm: Discrete Latent Choices without Reinforcement Learning
Section 20.9.1 of Goodfellow, Bengio and Courville is titled “BackPropagating through Discrete Stochastic Operations”. It discusses the REINFORCE algorithm which requires sampling sequences of discrete latent choices. It is well known that sampling choices yields a high variance gradient estimate. While … Continue reading
Posted in Uncategorized
1 Comment
VAE = EM
I recently realized the connection between the expectation maximization algorithm (EM) and variational autoencoders (VAE). Both optimize the same objective function where VAE performs gradient descent based on a sampling estimate of the gradient while EM performs exact alternating maximization … Continue reading
Posted in Uncategorized
Leave a comment
Deep Meaning Beyond Thought Vectors
I ended my last post by saying that I might write a followup post on current work that seems to exhibit progress toward natural language understanding. I am going to discuss a couple sampled papers but of course these … Continue reading
Posted in Uncategorized
1 Comment
The Plausibility of NearTerm Machine Sentience.
When should we expect “operational sentience” — the point where the most effective way to interact with a machine is to assume it is sentient — to assume that it understands what we tell it. I want to make an … Continue reading
Posted in Uncategorized
2 Comments
Formalism, Platonism and Mentalese
This is a sequel to an earlier post on Tarski and Mentalese. I am writing this sequel for two reasons. First, I just posted a new version of my treatment of type theory which focuses on “naive semantics”. I want to explain … Continue reading
Posted in Uncategorized
Leave a comment