Lifts for good model?

[Minnie]

Hello,
I know there is no fixed value for the lift for us to tell a good model - all depends on the industry and data, right? But still how can I know whether the predictive model I got is a good one when the top 10% lift is only 1.58? I used both a logistic regression and neural network to predict the success of insurance agencies based on the agency variables - purchased from D&B. The agency success is based on the mean of two internal performance variables and then coded 1 for the top 30% and 0 for the bottom 30%. Any insights would be appreciated!

[Geert]

Indeed the lift is not a good indicator.
Better use the AUC (area under the ROC curve) which is model-independent : 0.5-0.6= useless; 0.6-0.7=poor; 0.7-0.8=fair; 0.8-0.9=good;>0.9=excellent.

From my point of view : a lift of the top-10% of 1.58 is rather poor. It also depends if this is the lift of the training data set or of an unseen test data set ( the latter is allways lower).

Geert

[editor]

A lift of 1.58 in the top 10% is not a good one.

We have done a meta-study of lifts in CRM, Banking, and Telecom applications
(see G. Piatetsky-Shapiro, B. Masand, Proceedings of KDD-99,
www.kdnuggets.com/gpspubs/kdd99-est-ben-lift/)

and we found that a good rule of thumb is
Lift(P) ~ sqrt(1/P) (+- 1)

where P is a percentage of the list as fraction between 0 and 1.
So for 10%, P=0.1 and expected lift would be sqrt(1/0.1) ~ 3.2

However, if you tried several methods and your lift does not improve,
may be your task is not very predictable, such as predicting which lottery number will win.

For measuring lift quality, see also
Measuring Lift Quality in Database Marketing, SIGKDD Explorations, Dec 2000
www.kdnuggets.com/gpspubs/sigkdd-explorations-2000-12-lift-quality.pdf

Gregory Piatetsky-Shapiro

[gsafarz]

I would argue that the lift you got may be pretty good given the data at hand. My experience with D&B data is that it can only get you so far. The D&B business data does not contain a lot of behavioral data and some of it is derived(modeled) itself and can be as old as a year since last updated at the business level. This of course depends on the D&B data I guess. I have only used their small business data that contains basic information like sales volume, num employees, bus. type, sic codes, credit score, etc. If this is the same level data, you may need to find other data sources to get better lift.