Search Results

This chapter describes the different ways of measuring how good a scoring system is. It clarifies that there are three different ways of measuring the systems: their ability to discriminate Goods from Bads, their prediction of the probability of a borrower defaulting, and the accuracy of their categorical calibration. Discrimination, which only requires knowledge of the scorecard itself, is measured using ROC curves, Cumulative Accuracy Profiles (CAP), Gini Coefficient, AUROC, Divergence, Mahalonobis distance, and Somers D-concordance statistic. Probability predictions, which need the population odds as well as the scorecard, are measured using the binomial and normal tests and the Hosmer-Lemeshow test. Categorical calibration, which needs the cut-off score as well as the scorecard, is measured using confusion matrix, swap sets, specificity and sensitivity, and Type I and Type II errors. The chapter also explains how, if one has built a suite of scorecards each on a different segment of the population, one can combine the measures of the different scorecards into an overall measure.

In discrimination, signals can be of two or more types, and the observer is presented with one of them and tries to say which it was. This chapter extends the signal-detection model to discrimination and related to the detection.

After preparing your dataset, the business problem should be quite familiar, along with the subject matter and the content of the dataset. This section is about modeling data, using data to train algorithms to create models that can be used to predict future events or understand past events. The section shows where data modeling fits in the overall machine learning pipeline. Traditionally, we store real-world data in one or more databases or files. This data is extracted, and features and a target (T) are created and submitted to the “Model Data” stage (the topic of this section). Following the completion of this stage, the model produced is examined (Section V) and placed into production. With the model in the production system, present data generated from the real-world environment is inputted into the system. In the example case of a diabetes patient, we enter a new patient’s information electronic health record into the system, and a database lookup retrieves additional data for feature creation.

Having evaluated all the measures and selected the best model for this case, and much of the machine learning process has been clarified, our understanding of the problem context is still relatively immature. That is, while we have carefully specified the problem, we still do not fully understand what drives that target. Convincing management to support the implementation of the model typically includes explaining the answers to “why,” “what,” “where,” and “when” questions embedded in the model. While the model may be the best overall possible model according to selected measures, for the particular problem related to hospital readmissions, it is still not clear why the model predicts the readmission of some patients will be readmitted and that others will not. It also remains unknown what features drive these outcomes, where the patients who were readmitted come from, or whether or not this is relevant. In this case, access to time information is also unavailable––when, so it is not relevant, but it is easy to imagine that patients admitted in the middle of the night might have worse outcomes due to tired staff or lack of access to the best physicians. If we can convince management that the current analysis is useful, we can likely also make a case for the collection of additional data. The new data might include more information on past interactions with this patient, as well as date and time information to test the hypothesis about the effect of time-of-admission and whether the specific staff caring for a patient matters.