There are at least four dimensions of this issue:
(a) the cost of errors,
(b) good models of model competence,
(c) inspectability, and
(d) software processes for certifying ML methods.
(a) We already trust machine learning in many cases where the cost of errors is low (e.g., reading hand-written zip codes).
Doesn't Google's hand-tweaked policy still involve scoring functions (e.g., pagerank) that are computed from data? That means that they are trusting machine learning in some situations.
(b) We need more research on methods by which machine learning algorithms can model their own domain of competence so that they can abstain (and get human help) appropriately. There is lots of room for improvement.
(c) In many situations, I think learning methods would be more trusted if they were more inspectable -- what is the line of reasoning? what features were relevant in the particular case? etc. This is an under-researched area.
(d) We need software engineering processes for evaluating the robustness of learning algorithms. We could trust them more if they were tested through a trusted process. Hold-out data is not enough.
