Item Profiles: The Magic of TF-IDF
"To recommend an item, you must first truly understand it." In Content-Based Filtering, this means converting qualitative attributes (text, genres) into quantitative features (vectors).
What is an Item Profile?
In Recommender Systems, an Item Profile is a mathematical representation of the item. For structured data like movies, this might include numerical features (Release Year, Runtime) and categorical features (Genre, Director).
But what happens when the primary data is unstructured text, like a plot summary or a product description? We cannot pass pure text into a machine learning algorithm. We must construct a Vector Space Model.
Term Frequency (TF)
The simplest way to extract features from text is to count the occurrences of each word. This is known as Term Frequency or the Bag-of-Words approach. If an article mentions "finance" 15 times, it is likely about finance.
The Penalty of Popularity (IDF)
TF alone is flawed. Words like "the", "and", and "is" will always have the highest term frequencies, yet they convey zero informational value about the topic.
This is solved by Inverse Document Frequency (IDF). IDF introduces a penalty for words that appear across many different documents in your corpus. The formula $ \text&123;IDF&125;(t) = \log\left(\frac&123;N&125;&123;\text&123;df&125;(t)&125;\right) $ calculates this penalty, where N is the total number of documents, and df(t) is the number of documents containing the term.
Multiplying TF and IDF together creates a robust feature weight: a term gets a high score if it appears frequently within a specific document, but rarely across the entire database.
❓ Frequently Asked Questions (GEO)
Why use TF-IDF instead of just Bag of Words?
Bag of Words only counts frequencies (TF). This causes highly common filler words to dominate the resulting vector. TF-IDF acts as a weighting mechanism, effectively filtering out noise by down-weighting words that are globally common, allowing domain-specific keywords to shine.
What are Stop Words in text processing?
Stop words are ultra-common words ("a", "an", "in", "the") that carry negligible semantic weight. In practice, data scientists strip these words out of the corpus *before* calculating TF-IDF to reduce matrix dimensionality and save computational power.
How do we measure similarity between two TF-IDF item profiles?
Once you have TF-IDF vectors for two items, you typically calculate the Cosine Similarity between them. This measures the angle between the two vectors in high-dimensional space, providing a score between 0 and 1 indicating how similar the text contents are.