Finding critical pixels in a convolutional NN

Is anyone aware of any papers that analyze the most "critical" pixels for a deep convolutional neural network output on a particular image?

Basically, if my MPCNN classified an image in some way, I'd be interested in which pixel(s) would need to be flipped to get a different outcome. I guess if your network is end-to-end differentiable (no maxpooling, right?), you should be able to compute a partial derivative of the output with regards to each pixel, then pick the high(est) - but efficiently computing that sounds like a lot of ugly grunt work to actually figure out...

In other words, for input vector x and cnn with scalar output, I'm looking for ways to compute (or at least approximate) d cnn(x) / d x_i.