Diagnosing the way a ML algorithm behaves on a problem is non trivial. Has anyone tried to apply machine learning on itself to automatically solve this?

I got the idea from seeing the title of this paper: Knows what it knows: a framework for self-aware learning.

So, it would be nice to have an extra layer on top of a ML algorithm that would look at its performance and fine tune it: suggest features, number of examples needed, complexity of the model and so on?

At a higher level, this "ML expert" could suggest algorithms (and kernels) that work best on the dataset at hand - why rely on human intuition alone?