KDnuggets » Forums
Latest News

Author Message
Minnie
Guest

 Posted: Sun Jan 22, 2006 8:01 pm    Post subject: Lifts for good model? Hello, I know there is no fixed value for the lift for us to tell a good model - all depends on the industry and data, right? But still how can I know whether the predictive model I got is a good one when the top 10% lift is only 1.58? I used both a logistic regression and neural network to predict the success of insurance agencies based on the agency variables - purchased from D&B. The agency success is based on the mean of two internal performance variables and then coded 1 for the top 30% and 0 for the bottom 30%. Any insights would be appreciated!
Geert
Guest

 Posted: Mon Jan 23, 2006 8:51 am    Post subject: lifts for good model ? Indeed the lift is not a good indicator. Better use the AUC (area under the ROC curve) which is model-independent : 0.5-0.6= useless; 0.6-0.7=poor; 0.7-0.8=fair; 0.8-0.9=good;>0.9=excellent. From my point of view : a lift of the top-10% of 1.58 is rather poor. It also depends if this is the lift of the training data set or of an unseen test data set ( the latter is allways lower). Geert
editor

Joined: 04 Oct 2005
Posts: 124
Location: Boston, MA

 Posted: Mon Jan 23, 2006 2:43 pm    Post subject: Lift rule of thump for good models: Lift(P) ~ sqrt(1/P) A lift of 1.58 in the top 10% is not a good one. We have done a meta-study of lifts in CRM, Banking, and Telecom applications (see G. Piatetsky-Shapiro, B. Masand, Proceedings of KDD-99, www.kdnuggets.com/gpspubs/kdd99-est-ben-lift/) and we found that a good rule of thumb is Lift(P) ~ sqrt(1/P) (+- 1) where P is a percentage of the list as fraction between 0 and 1. So for 10%, P=0.1 and expected lift would be sqrt(1/0.1) ~ 3.2 However, if you tried several methods and your lift does not improve, may be your task is not very predictable, such as predicting which lottery number will win. For measuring lift quality, see also Measuring Lift Quality in Database Marketing, SIGKDD Explorations, Dec 2000 www.kdnuggets.com/gpspubs/sigkdd-explorations-2000-12-lift-quality.pdf Gregory Piatetsky-Shapiro
gsafarz

Joined: 28 Jan 2006
Posts: 3

 Posted: Sat Jan 28, 2006 9:35 pm    Post subject: I would argue that the lift you got may be pretty good given the data at hand. My experience with D&B data is that it can only get you so far. The D&B business data does not contain a lot of behavioral data and some of it is derived(modeled) itself and can be as old as a year since last updated at the business level. This of course depends on the D&B data I guess. I have only used their small business data that contains basic information like sales volume, num employees, bus. type, sic codes, credit score, etc. If this is the same level data, you may need to find other data sources to get better lift.
 Display posts from previous: All Posts1 Day7 Days2 Weeks1 Month3 Months6 Months1 Year Oldest FirstNewest First
 All times are GMT - 5 Hours Page 1 of 1

 Jump to: Select a forum Data Mining and Knowledge Discovery----------------Data Mining Open ForumData Mining BeginnersData Mining ExpertsData Mining Applications Data Mining Algorithms & Software----------------Classification & ClusteringText Mining, Web Mining, Association Rules, and Other Algorithms
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum