How Big Data is used in Recommendation Systems to change our lives


A Recommendation systems have impacted or even redefined our lives in many ways. It works in well-defined, logical phases which are data collection, ratings, and filtering.



Recommendation systems have impacted or even redefined our lives in many ways. One example of this impact is how our online shopping experience is being redefined. As we browse through products, the Recommendation system offer recommendations of products we might be interested in. Regardless of the perspective — business or consumer, Recommendation systems have been immensely beneficial. And big data is the driving force behind Recommendation systems. A typical Recommendation system cannot do its job without sufficient data and big data supplies plenty of user data such as past purchases, browsing history, and feedback for the Recommendation systems to provide relevant and effective recommendations. In a nutshell, even the most advanced Recommenders cannot be effective without big data.

How does a Recommendation system work?

A Recommendation system works in well-defined, logical phases which are data collection, ratings, and filtering. These phases are described below.

Data collection

Let us assume that a user of Amazon website is browsing books and reading the details. Each time the reader clicks on a link, an event such as an Ajax event could be fired. The event type could vary depending on the technology used. The event then could make an entry into a database which usually is a NoSQL database. The entry is technical in content but in layman’s language could read something like “User A clicked Product Z details once”. That is how user details get captured and stored for future recommendations.

How does the Recommendation system capture the details? If the user has logged in, then the details are extracted either from an http session or from the system cookies. In case the Recommendation system depends on system cookies, then the data is available only till the time the user is using the same terminal. Events are fired almost in every case — a user liking a Product or adding it to a cart and purchasing it. So that is how user details are stored. But that is just one part of what Recommenders do.

The following paragraphs show how Amazon offers its product recommendations to a user who is browsing for books:

  • As shown by the image below, when a user searched for the book Harry Potter and the Philosopher’s Stone, several recommendations were given.

recommendation-content-based

  • In another example, a customer who searched Amazon for Canon EOS 1200D 18MP Digital SLR Camera (Black) was interestingly given several recommendations on camera accessories.

recommendation-collaborative-filter

Ratings

Ratings are important in the sense that they tell you what a user feels about a product. User’s feelings about a product can be reflected to an extent in the actions he or she takes such as likes, adding to shopping cart, purchasing or just clicking. Recommendation systems can assign implicit ratings based on user actions. The maximum rating is 5. For example, purchasing can be assigned a rating of 4, likes can get 3, clicking can get 2 and so on. Recommendation systems can also take into account ratings and feedback users provide.

Filtering

Filtering means filtering products based on ratings and other user data. Recommendation systems use three types of filtering: collaborative, user-based and a hybrid approach. In collaborative filtering, a comparison of users’ choices is done and recommendations given. For example, if user X likes products A, B, C, and D and user Y likes products A, B, C, D and E, the it is likely that user X will be recommended product E because there are a lot of similarities between users X and Y as far as choice of products is concerned.

Several reputed brands such as Facebook, Twitter, LinkedIn, Amazon, Google News, Spotify and Last.fm use this model to provide effective and relevant recommendations. In user-based filtering, the user’s browsing history, likes, purchases and ratings are taken into account before providing recommendations. This model is used by many reputed brands such as IMDB, Rotten Tomatoes and Pandora. Many companies also use a hybrid approach. Netflix is known to use a hybrid approach.