Do you use a database for storing data for data mining ?

[editor]

Does the power of database system outweigh the extra complexity ?
What is your experience ?

[gabrielac]

I often use a database for preprocessing data before actually using it for data mining. Depending on the data mining software, I finally end up using either a text file, or a table with the final preprocessing results.

[TimManns]

I guess it depends largely on the amount of data, available time-frame and subsequent use of results.

I work for a telecommunications company, and we routinely analyse a few million customers. Our analysis is quite adhoc, so having a fixed datamart is not practical. For scaleablity reasons we must do the preprocessing within the database - making use of the indexed and organised nature of the database. We process and summarise billions of rows of data into a single row per customer/account (with a few hundred columns), then sample and use for model building and testing.

Model building on smaller samples is conducted via flat text files. Often the whole customer base is then scored using a predictive model converted into SQL, occiasionally the customer base data is extacted from the database, scored to a text file and then loaded back into the database.

The final database scored table is used by numerous CRM systems and marketing campaigns, which rely on the data being within the database.

If we didn't use the database for preprocessing we wouldn't be able to analyse the data within a single working day (it would weeks rather than hours).

Cheers

Tim