Feb 04

Use Enthought for scientific python

I’ve spent many hours attempting to install numpy and scipy from source on both Linux at work and Mac OS X at home. I could never get the big speedups possible from good numerical libraries like BLAS and ATLAS. Enthought’s one-click install python distribution turned out to be the perfect solution.

Read the rest of this entry »

Jan 22

Why probability contours for the multivariate Gaussian are elliptical

Every 2D Gaussian concentrates its mass at a particular point (a “bump”), with mass falling off steadily away from its peak.  If we plot regions that have the *same* height on the bump (the same density under the PDF), it turns out they have a particular form: an ellipse.  In this post, I’ll use math to show why it is an ellipse.
Read the rest of this entry »

Nov 13

Practical Bayesian Optimization with Spearmint

Recently, I’ve become interested in using Gaussian Processes for hyperparameter optimization. Coincidentally, there’s a great  NIPS 2012 paper by Jasper Snoek, Hugo Larochelle, and Ryan Adams about that very topic. Thankfully, not only is the paper insightful, but they also have Python source code available, called “spearmint” (I guess they chose the name so that it sounds like “experiment”). This post documents my experience using this code on a simple toy example.  This is mostly about the nuts and bolts of using the code, but I definitely recommend reading the paper.

Read the rest of this entry »

Sep 30

Review of STAN: off-the-shelf Hamiltonian MCMC

Recently, some folks at Andrew Gelman’s research lab have released a new and exciting inference package called STAN.  STAN is designed to do MCMC inference “off-the-shelf”, given just observed data and a BUGS-like definition of the probabilistic model.  I’ve played around with STAN in some detail, and my high-level review is summarized here

Good:

  • Installation is involved but straight-forward and quite well-documented
  • STAN has an active and enthusiast developer team. Ask a question on the Google group, and you’ll often get a same day (or even same hour) response.
  • STAN does Hamiltonian MCMC via automatic differentiation and principled automatic tuning, so you (1) don’t need to compute any gradients analytically, and (2) don’t need to set any parameters manually!
  • STAN is also wicked fast on some toy examples it was designed for.

Caveats:

  • STAN (version 1.0) does not support *any* sampling of discrete variables. You’ll need to invest human time in marginalizing out discrete variables, esp. for common machine learning approaches like mixture models.
  • Vectorizing the model definition can work wonders, but requires some detailed human knowledge of STAN’s inner guts.
  • The type system is a bit confusing. For example, to enforce that a variable must be >= 1:
    • This is legal: real<lower=1> x;
    • This isn’t: vector<lower=1> xvec;
  • Some language features I expected were mysteriously lacking.
  • cannot slice-index variables (e.g. MyMatrix[ 1:5,:] )
  • cannot transpose matrices

Punchline: I don’t think STAN is quite ready to be used by Machine Learning researchers as a black-box tool to prototype models.  It lacks fine control over when to do discrete vs. continuous updates, and thus scales poorly to moderately sized dataset (it took *minutes* to run 5 iterations on an LDA topic model with just 100 documents and 7000 words).  However, I do believe that the automatic approach to Hamiltonian MCMC is sensible, and hopefully down the road this package might be more viable.

Read on to see detailed comments and code examples.

Read the rest of this entry »

Sep 01

Using Voodoo to Estimate Handheld Camera Motion

I’m currently investigating fully automatic methods to determine the motion of a handheld cameras in unconstrained environments. Basically, if action recognition is going to work well in unconstrained videos, we need video representations that distinguish between motions that are “signal” (moving actors, moving objects) and motions that are irrelevant noise due to camera motion (panning, zooming, shake, etc.).

Today, I checked out one out-of-the-box solution: an open-source tool called Voodoo. Voodoo seems originally designed with graphics and special effects industries in mind, but . I found it quite easy to install and get up and running. Unfortunately, I found its performance on my test suite of two hand-held camera sequences to be disastrous. Perhaps I’m asking more than it was designed for. Read on for more…

Read the rest of this entry »

Aug 30

Dense Trajectories for Action Recognition

Here, I take a qualitative look at one of the most successful video representations in use today: dense trajectories. You can find the original CVPR 2011 paper by Wang, Klaser, Schmid, and Liu here: Action Recognition by Dense Trajectories.

High Jump with action trajectories

I’m mostly interested in evaluating how well the provided code does at finding good trajectories for videos from the Olympic Sports dataset, which have lots of camera motion, both deliberate (pan, zoom, etc) and accidental (shake).   The example videos I show here show that this approach is really vulnerable to this noise, which probably requires further preprocessing to remove errant trajectories that aren’t firing on the motion of interest. Their “Motion Boundaries” descriptor might help distinguish camera motion from “signal”, but I doubt it will solve the entire problem.

Read the rest of this entry »

Aug 18

Elliptical Slice Sampling for priors with non-zero mean

I’ve been playing around with elliptical slice sampling (ESS) lately.  This is a new MCMC technique developed by Iain Murray, Ryan Adams, and David MacKay. Here is the original AISTATS paper. The punchline is that this method allows exact joint sampling for vectors whose posterior distribution can be expressed using any Gaussian prior and any (non-conjugate) likelihood. ESS does not have any free parameters, and always accepts its proposal.  This makes it favorable compared to, say, Metropolis-Hastings methods, since tuning the lengthscales that control a random walk to achieve good mixing rates requires expensive human time.

One tricky aspect of this method is that it requires a change of variables when applied to sample a variable with non-zero prior mean.  This is mentioned in the original paper, but not all that coherently. I thought this deserved a blog post, so that practitioners make sure they get it right.  The necessary steps are quite simple for experienced statisticians, so there’s nothing really novel or exciting happening. Read on for more.  Hopefully in the near future I’ll report on experiments comparing ESS to other methods on some more interesting models.

Read the rest of this entry »

Aug 14

ICML 2012 Highlights

After attending ICML 2012 in Edinburgh a month ago, I suppose a summary of my favorite papers and presentations is long overdue.

View of Edinburgh, Scotland.Overall, Edinburgh provided a beautiful setting for the conference, and famously dreary Scotland weather even cooperated with a few days of sunshine.

Read the rest of this entry »

Jul 12

CVPR 2012 Highlights

In early June, I attended CVPR, a premier vision conference.  Luckily for me, the conference was right here in Providence just 20 minutes from Brown, so I didn’t have far to go.

Definitely, one highlight was the lobster dinner, served to all 1000+ attendees.  After the jump, I’ll summarize some of my favorite research highlights.

 

Read the rest of this entry »

Jul 12

Blog Basics

This sample post shows how to this site can include typeset mathematics and short code snippets.

Typesetting is possible via the QuickLaTeX plugin. Code via the SyntaxHighlighter Evolved plugin.

Read the rest of this entry »