Public Education

chris · Posted: Sat Feb 18, 2006 7:02 pm Post subject: Public Education

I sort of feel out of place in such a distinguished group, but I am trying to use data mining techniques with SPSS's Clementine and Text Analysis for Surveys in public education at the K to 12 level. Several years ago I did a project analyzing test score data for our high stakes testing program. I also used AnswerTree to do some CART and CHAID stuff. I am talking like I have this mastered. Nothing could be further from the truth.

Now I am using text mining to analyze discipline records in our school system. There is a free form narrative that the disciplinarian can fill in to better describe the actual incident. I had a theory that our discipline referrals are compounded by a trivial action becoming a major incident in a heartbeat.

So far I am into the categorizing and synonym phase figuring out what is what. All I had to go on was reading the manual and doing the tutorial. It is becoming interesting, and I am anxious to tie this analysis back to the demographic data that is linked to the text. I always thought that data mining works best when you have a clear understanding of your business. I am anxious to see relationships among variables such as ethnicity, poverty, special education, time of day, and the location of the offense.

I know this is not the kind of project that people in these forums think of when someone says data mining, but it makes sense to me. This certainly isn’t fraud detection or credit card scoring.

I would certainly like some comments if you have them, and I would certainly like some ADVICE from anyone who thinks there is something I need to know.

Chris Bowman
Lafourche Parish School Board

gabrielac · Contributor Joined: 16 Feb 2006 Posts: 8

For my part, I find your project really interesting. Have you taken a look at the "Text Mining & Web Mining" forum. There are interesting links there.
Good luck on your project!

chris · Posted: Mon Feb 20, 2006 3:31 pm Post subject: Got this far -- so far

No, I haven't, but you can bet that I will.

I figured there MIGHT be one person on the planet who would be interested in this, and from your reply, you are the one! As far as my school system goes, they don't know that I am doing this like they think. I have mentioned it to people, but they have no idea what text mining looks like.

At this point, I am much more in the learning phase than the expert phase. Expert is rarely applied to anything that I do. To be honest, this is really interesting. SPSS’s Text Analysis for Surveys is doing something that I can’t do. There were 17,647 records some with and without any text. Some text is a few characters long while other text is near 1,500 characters.

My goal is to get something done categorizing data and looking for links to other categories in order to see how one disciplinary referral goes ballistic in no time flat. I can already see it, but it is becoming very apparent that there are connections among categories. Much is hidden and obscure, but it is plain to see once text analysis comes into play.

My next step would be to export this back to an SPSS formatted file, link it to the appropriate record, and send it to Clementine. Now, I guess I will have to figure out again how Clementine works. I have a feeling that I will find out, confirm, or reject for that matter, many intriguing suppositions or pre-conceived notions. I am not looking to radically alter this school system, but it seems likely that we should look at changing policies.

I know that for data mining to work, a thorough understanding of the business and a well-defined problem are key components. Does it sound like I am doing this correctly so far? Got any advice. What would you do?

Chris

gabrielac · Contributor Joined: 16 Feb 2006 Posts: 8

I can see you are very enthusiastic about this project. I find Text Mining interesting; I've been doing some research, but I'm no expert. I agree with you when you say that "a thorough understanding of the business and a well-defined problem are key components". And in your case, business is how the school system works, and what you can do with the results of the analysis once you have them.
I think you're doing fine so far.
Gaby

chris · Posted: Wed Feb 22, 2006 4:56 pm Post subject: Thanks

Gaby,

Thanks for the reply. It seems as though I am in right direction. I will keep you posted. By the way, what do you do, and where are you?

Chris

gabrielac · Contributor Joined: 16 Feb 2006 Posts: 8

I work in Professional Services for a consultancy firm that specializes in DW and DM in Argentina. I'm also working on a post-graduate degree in DM.