Mike Grant, How reliable is a model
No prediction we make is certain except for things like death. However, by testing for example a marketing prediction model on a random sample of data not used to build the model we can predict that if no major event has occured which would make future marketing conditions different from the past that our model should be reasonably accurate. (I would expect it to normally perform better than a guess - even a very informed guess - which is what had to be used in the past.)
Tom Dietterich, Trustable Machine Learning
There are at least four dimensions of this issue: (a) the cost of errors,(b) good models of model competence, (c) inspectability, and (d) software processes for certifying ML methods. (a) We already trust machine learning in many cases where the cost of errors is low (e.g., reading hand-written zip codes).
Doesn't Google's hand-tweaked policy still involve scoring functions (e.g., pagerank) that are computed from data? That means that they are trusting machine learning in some situations. (b) We need more research on methods by which machine learning algorithms can model their own domain of competence so that they can abstain (and get human help) appropriately. There is lots of room for improvement. (c) In many situations, I think learning methods would be more trusted if they were more inspectable -- what is the line of reasoning? what features were relevant in the particular case? etc. This is an under-researched area. (d) We need software engineering processes for evaluating the robustness of learning algorithms. We could trust them more if they were tested through a trusted process. Hold-out data is not enough.
David Nettleton, Trusting ML Algorithms
I would have, for example, an odd (5, 7, ...) number of different methods (machine learning, manual, statistical, ..) to produce a prediction, which may include the same algorithm replicated several times with different parametrizations. Then I would do a polling and choose the predicted class by simple majority, with a possible human intervention to take the final decision. This would be for "life critical" situations. In "other" situations, it would depend on what % of error would be acceptable.
Ross Bettinger, trusting ML algorithms
a model is no better than its data. if the data represent all possible outcomes of an event of interest then one may give greater credence to the model's machinations. if the data used to build the model are sparse and incomplete in their representations of the reality which is being modeled, then one must apportion credulity in the model's results sparingly in proportion to and with respect to the absences in observations.
eggyknap, "Full" trust
Interpreting "fully trusted" to mean "this system makes decisions that aren't subjected to examination by some other system", then sure, we can fully trust all kinds of things, dependent on the risk we assign to various failure modes in that system. I fully trust my wife to pick my clothes out for me on those occasions when she chooses to do so, because 1) historically she's been better at it than I, and 2) I don't particularly care if one of her choices happens to be an EPIC FAIL. On the other hand, I don't fully trust a computer to drive my car, largely because the other cars are controlled by humans, which are much harder to predict, and the consequences of failure are particularly dreadful. We already "fully trust" machines to do a lot of things.
Gregory Piatetsky-Shapiro, When can we trust ML algorithms?
One safety mechanism is to have an auxiliary algorithm which can recognize when the input data is strongly different from the training and switch to a simpler backup method. This is also what people do when faced with new conditions, and we also make plenty of mistakes.