Author Archives: McAllester

The Role of Theory in Deep Learning

This blog post is inspired by the recent NIPS talk by Ali Rahimi and the response by Yann LeCun.  The issue is fundamentally the role of theory in deep learning.   I will start with some quotes from Rahimi’s talk. Rahimi: … Continue reading

Posted in Uncategorized | 5 Comments

Choice as a Natural Kind Term

This is a sequel to my previous post on determinism, free will and the existence of choice.  Here I want to consider the semantics of the word “choice” from the perspective of lexical semantics generally.  I will focus on the … Continue reading

Posted in Uncategorized | Leave a comment

CTC and the EG Algotithm: Discrete Latent Choices without Reinforcement Learning

Section 20.9.1 of Goodfellow, Bengio and Courville is titled “Back-Propagating through Discrete Stochastic Operations”.  It discusses the REINFORCE algorithm which requires sampling sequences of discrete latent choices.  It is well known that sampling choices yields a high variance gradient estimate.  While … Continue reading

Posted in Uncategorized | 1 Comment

VAE = EM

I recently realized the connection between the expectation maximization algorithm (EM) and variational autoencoders (VAE).  Both optimize the same objective function where VAE performs gradient descent based on a sampling estimate of the gradient while EM performs exact alternating maximization … Continue reading

Posted in Uncategorized | Leave a comment

Deep Meaning Beyond Thought Vectors

I ended my last post by saying that I might write a follow-up post on current work that seems to exhibit progress toward natural language understanding.   I am going to discuss a couple sampled papers but of course these … Continue reading

Posted in Uncategorized | 1 Comment

The Plausibility of Near-Term Machine Sentience.

When should we expect “operational sentience” — the point where the most effective way to interact with a machine is to assume it is sentient — to assume that it understands what we tell it.  I want to make an … Continue reading

Posted in Uncategorized | 2 Comments

Formalism, Platonism and Mentalese

This is a sequel to an earlier post on Tarski and Mentalese.  I am writing this sequel for two reasons. First, I just posted a new version of my treatment of type theory which focuses on “naive semantics”. I want to explain … Continue reading

Posted in Uncategorized | Leave a comment