Collaborative Filtering Recommendation Engine with Neo4j & Python
Introduction
In a previous blog, we created a graph database in Neo4j using the DVDRENTAL database which you can download from here.
Here, we will continue our work and build a recommender system using Cypher, the query language of Neo4j. We will use Python to connect to the database and execute the queries. Our system will recommend movies to target users based on the preferences of the users who have a similar taste. I.e. for each target user we will identify the most similar users to them. Then we will use the preferences of these users to generate recommendations for the target user.
This algorithm is Collaborative Filtering.
Github link for the project and data: https://github.com/MNoorFawi/recommendation-engine-with-neo4j
Collaborative Filtering
Image source
Collaborative filtering (CF) is a technique commonly used to build personalized recommendation systems. Some popular websites that use CF technology include Amazon, Netflix, and IMDB. CF algorithm makes predictions about a user’s interests by compiling preferences from similar users.
In order to build our recommender system, we will follow the following steps:
- Select a similarity metric to quantify similarity among users in data.
- For each target user, compute similarity between them and the rest of users.
- Select the top k nearest neighbors based on the similarity metric.
- Identify Movies rented by the top k neighbors that the target user has not rented.
- Rank these movies by the number of renting neighbors.
- Recommend the top n movies to the target user.
The similarity metric we will use here is Jaccard Similarity Coefficient or Jaccard Index. Also known as Intersection over Union.
Jaccard Index
Jaccard Index between two sets A and B is the ratio of the number of elements in the intersection of A and B over the number of elements in the union of A and B.
Our Graph Database
This is how our customers and the movies they rented look like in the graph database.
Let’s now run our database and connect it to python. Then we will query the number of movies per each genre.
from command line run “neo4j console”
from pprint import pprint
from py2neo import Graph
g = Graph("http://localhost:7474/db/data/", password = "password")
query = """
MATCH (cat:Category)<-[g:OF_GENRE]-()
RETURN cat.Name AS genre, COUNT(g) AS number_of_movies
ORDER BY number_of_movies DESC;
"""
g.run(query).to_data_frame()
| genre | number_of_movies
----+-------------+------------------
1 | Sports | 74
2 | Foreign | 73
3 | Family | 69
4 | Documentary | 68
5 | Animation | 66
6 | Action | 64
7 | New | 63
8 | Drama | 62
9 | Games | 61
10 | Sci-Fi | 61
11 | Children | 60
12 | Comedy | 58
13 | Travel | 57
14 | Classics | 57
15 | Horror | 56
16 | Music | 51
Great! We have connected to the database. Let’s now choose a random user and try to make recommendations for them. We will use 25 nearest neighbors to recommend 5 movies to the target user.
Recommendation Engine Model
We first get the most similar users to our target user. Jaccard Index = Movies in common (Intersection) / Movies in total (Union) .
query = """
// get target user and their neighbors pairs and count
// of distinct movies that they have rented in common
MATCH (c1:Customer)-[:RENTED]->(f:Film)<-[:RENTED]-(c2:Customer)
WHERE c1 <> c2 AND c1.customerID = $cid
WITH c1, c2, COUNT(DISTINCT f) as intersection
// get count of all the distinct movies that they have rented (Union)
MATCH (c:Customer)-[:RENTED]->(f:Film)
WHERE c in [c1, c2]
WITH c1, c2, intersection, COUNT(DISTINCT f) as union
// compute Jaccard index
WITH c1, c2, intersection, union, (intersection * 1.0 / union) as jaccard_index
// get top k nearest neighbors based on Jaccard index
ORDER BY jaccard_index DESC, c2.customerID
WITH c1, COLLECT([c2.customerID, jaccard_index, intersection, union])[0..$k] as neighbors
WHERE SIZE(neighbors) = $k // return users with enough neighbors
RETURN c1.customerID as customer, neighbors"""
neighbors = {}
for i in g.run(query, cid = "13", k = 25).data():
neighbors[i["customer"]] = i["neighbors"]
print("# customer13's 25 nearest neighbors: customerID, jaccard_index, intersection, union")
pprint(neighbors)
# customer13's 25 nearest neighbors: customerID, jaccard_index, intersection, union
{'13': [['93', 0.08695652173913043, 4, 46],
['211', 0.07142857142857142, 4, 56],
['379', 0.06521739130434782, 3, 46],
['578', 0.06521739130434782, 3, 46],
['134', 0.06382978723404255, 3, 47],
['8', 0.06382978723404255, 3, 47],
......
['464', 0.05, 2, 40],
['555', 0.047619047619047616, 2, 42]]}
Now, let’s see the top 5 movies that we can recommend to our target user. In addition, we will look at the number of neighbors who rented these movies.
# get the list of the nearest neighbors IDs
nearest_neighbors = [neighbors["13"][i][0] for i in range(len(neighbors["13"]))]
query = """
// get top n recommendations for customer 13 from neighbors
MATCH (c1:Customer),
(neighbor:Customer)-[:RENTED]->(f:Film) // all movies rented by neighbors
WHERE c1.customerID = $cid
AND neighbor.customerID in $nearest_neighbors
AND not (c1)-[:RENTED]->(f) // filter for movies that our user hasn't rented
WITH c1, f, COUNT(DISTINCT neighbor) as countnns // times rented by nns
ORDER BY c1.customerID, countnns DESC
RETURN c1.customerID as customer, COLLECT([f.Title, countnns])[0..$n] as recommendations"""
recommendations = {}
for i in g.run(query, cid = "13", nearest_neighbors = nearest_neighbors, n = 5).data():
recommendations[i["customer"]] = i["recommendations"]
print("# customer13's recommendations: Movie, number of rentals by neighbors")
pprint(recommendations)
# customer13's recommendations: Movie, number of rentals by neighbors
{'13': [['Goodfellas Salute', 5],
['Pacific Amistad', 4],
['Streetcar Intentions', 4],
['Chill Luck', 4],
['Whisperer Giant', 4]]}
Perfect!
Command Line Tool
Now, we have successfully built our recommender system. It can recommend movies to target customers. Why not writing it in a script that we can run in command line using customer IDs as arguments and return recommendations per customer?
## Our Recommender System Script (dvd_recommender.py)
import sys
from pprint import pprint
from py2neo import Graph
cid = sys.argv[1:]
g = Graph("http://localhost:7474/db/data/", password = "password")
def cf_recommender(graph, cid, nearest_neighbors, num_recommendations):
query = """
MATCH (c1:Customer)-[:RENTED]->(f:Film)<-[:RENTED]-(c2:Customer)
WHERE c1 <> c2 AND c1.customerID = $cid
WITH c1, c2, COUNT(DISTINCT f) as intersection
MATCH (c:Customer)-[:RENTED]->(f:Film)
WHERE c in [c1, c2]
WITH c1, c2, intersection, COUNT(DISTINCT f) as union
WITH c1, c2, intersection, union,
(intersection * 1.0 / union) as jaccard_index
ORDER BY jaccard_index DESC, c2.customerID
WITH c1, COLLECT(c2)[0..$k] as neighbors
WHERE SIZE(neighbors) = $k
UNWIND neighbors as neighbor
WITH c1, neighbor
MATCH (neighbor)-[:RENTED]->(f:Film)
WHERE not (c1)-[:RENTED]->(f)
WITH c1, f, COUNT(DISTINCT neighbor) as countnns
ORDER BY c1.customerID, countnns DESC
RETURN c1.customerID as customer,
COLLECT(f.Title)[0..$n] as recommendations"""
recommendations = {}
# cid = [str(c) for c in cid]
for c in cid:
for i in graph.run(query, cid = c, k = nearest_neighbors, n = num_recommendations).data():
recommendations[i["customer"]] = i["recommendations"]
return recommendations
pprint(cf_recommender(g, cid, 25, 5))
Let’s now run the system from command line!
$ python dvd_recommender.py 13 11 19 91
{'13': ['Goodfellas Salute',
'Pacific Amistad',
'Streetcar Intentions',
'Chill Luck',
'Whisperer Giant'],
'11': ['Sweethearts Suspects',
'Tights Dawn',
'Island Exorcist',
'Jason Trap',
'Earth Vision'],
'19': ['Fatal Haunted',
'Crossroads Casualties',
'Ridgemont Submarine',
'Wonderland Christmas',
'Uptown Young'],
'91': ['Forrester Comancheros',
'Anaconda Confessions',
'Bear Graceland',
'Greatest North',
'Hanover Galaxy']}
Leave a Reply