Web Mining Course Unit on Web Log Analysis

Gregory Piatetsky-Shapiro

Here are the teaching modules for a unit on Web Mining with the focus on Web Usage Mining and Web Log Analysis.

Web Mining Course Modules

• Module 1: Introduction to Web Mining
    wm1-web-mining-intro.ppt
• Module 2a: Web Server Log
    wm2a-web-server-log.ppt
• Module 2b: Unix tools for web log analysis
    wm2b-unix-web-log-analysis.ppt
• Module 3a: Hit Analysis
    wm3a-hit-analysis.ppt
• Module 3b: Gawk tools for web log analysis<
    wm3b-gawk-web-log-analysis.ppt
• Module 4a: Visit Analysis; Bot or Not?
    wm4a-visit-analysis.ppt
• Module 4b: Perl tools for web log analysis
    wm4b-perl-web-log-analysis.ppt
  Basic Perl script for web log parsing (web_log_parse.txt)
• Module 5: Behavior modeling
    wm5-behaviour-analysis.ppt

Assignments

• Assignment 1 - news stories.
• Assignment 2 - global analysis.
• Assignment 3 - hit-level analysis.
• Assignment 4 - Perl for visit-level analysis.
• Assignment 5 - bringing all together - final project.

Data

This data can be downloaded from kdlog.zip (0.6 MB) in www.kdnuggets.com/web_mining_course/ directory.

First 100 log lines are in the unzipped file d100.log in the same directory.

Acknowledgments

This project is a continuation of Data Mining Course project, and was funded by a grant from W. M. Keck Foundation, Los Angeles, CA and Howard Hughes Medical Institute, Chevy Chase, MD, as part of Connecticut College Series of Modules in Emerging Fields.

I am grateful to Gary Parker (Connecticut College) for his encouragement and support and to Anand Rajaraman and Jeffrey Ullman (Stanford) for permission to use part of their "Introduction to Web Mining" lecture.

See also