Points to consider before building and while implementing recommendation systems
Big data analytics, machine learning algorithms, and data mining techniques have driven numerous innovations. Yet, recommender systems, despite being pivotal, often go underappreciated. Their widespread integration has made them almost invisible in this field, but their critical role in how we use web or mobile applications cannot be understated
Recommender systems while being extremely important for a company, can be difficult to get right. In this article, I am sharing some of my experiences with recommenders and points to consider to get the most out of them for your business.
A brief introduction: I’m Atul Ranjan, a software engineer at Gott Data specializing in crafting AI and Machine Learning solutions that empower businesses to scale. Here I will be providing some valuable insights about recommender systems as follows or in the next couple of minutes.
Popularity based recommendation system
Introduction to Recommender Systems
A recommendation system is an algorithm tailored to discern a user’s preferences and suggest products based on those inclinations. A good recommender can be of tremendous value to any product and can be a pivotal investment your sales and growth team needs!
In popular surveys conducted by Salesforce, it has been found that 54% of retailers claim that product recommendations act as the key driver of the AOV (Average Order Value). Research also show that shoppers that clicked on recommendations are 4.5 times more likely to add these items to the cart and 4.5 times more likely to complete the purchase. It is also seen that shoppers that clicked a product recommendation are nearly 2 times more likely to come back.
A successful data model isn’t necessarily one that answers all existing questions, but one that sparks new ones. Recommendation systems are similar to any other data model. Building one for your business can provide tremendous new insights into your customer base and urge you to look into different perspectives.
Let’s now move towards considerations while building one!
Customizing algorithms to meet business needs
When constructing a recommendation system, it’s essential to balance algorithmic focus with a keen understanding of the business needs. A good recommendation system is not the most accurate algorithm on paper, but one that can perfectly fit the business.
It is therefore crucial to grasp the business objectives, profit models, target audience, and customer segmentation. A company might aim to promote a specific product, tailor recommendations by region, or find that the most frequently suggested item isn’t the most profitable. For some businesses, recommending similar products might not resonate while for others, it might work really well. A good algorithm should be able to capture all of these business needs along with the learnings from a thorough analysis of the provided data.
Types of Recommendation Systems
Selecting an algorithm
Recommendation systems can be approached using a variety of algorithms, with the choice largely dependent on consistency, accuracy, speed, and, most crucially, the specific needs of the business. A brief overview of some prevalent recommender algorithms includes:
- Item-Item Similarity: This method suggests products by gauging the similarity of other items to a chosen item.
- User-User Similarity: Unlike the item-based approach, this method recommends products by comparing the purchasing patterns of similar users to the active user.
- Association Rule Mining: This type of algorithm recommends items that are most often associated with the selected item
- Clustering-based Approaches: This method segments the user base into distinct clusters based on purchasing behavior, suggesting products under the premise that users within the same cluster have similar preferences.
- Popularity-Based Recommenders: A straightforward approach that suggests items based on their popularity. It’s often used in conjunction with other algorithms for enhanced results.
- Autoencoders and Deep Learning-based Recommenders: Tailored for unstructured data such as images, videos, and clips, these algorithms can be particularly precise in product recommendations.
The ideal recommendation system often isn’t a standalone algorithm but a fusion of several. Nevertheless, the choice of algorithm should be tailored to the business’s specific needs.
Association Rule-Based Recommendations excel in enhancing the average sale per customer. For instance, when someone purchases a phone, this system might suggest complementary items like cases or chargers — products often bought together with phones. This not only introduces users to new product ideas but also adds a layer of convenience to their shopping experience.
For businesses where purchasing decisions are intricately tied to user characteristics or behaviors, Clustering-Based Approaches prove invaluable. Clustering involves segmenting users into distinct groups where members share similar profiles or habits. For sectors like insurance, where the type of policy a customer selects hinges heavily on their personal attributes, such systems are particularly effective.
Item-item similarity recommendations offer users a comparative perspective, allowing them to weigh products against one another, making it especially beneficial for e-commerce platforms.
Similarity based recommendation systems
On the other hand, User-User Similarity recommenders provide broad-based suggestions. Although conceptually akin to clustering, their implementation often diverges.
These recommendation systems mentioned above use approaches like content-based filtering, collaborative filtering, clustering, classification, etc. to implement which are out of the scope of this article.
In summary, while all the mentioned algorithms can serve as robust general recommendation systems, aligning the right algorithm with business goals can exponentially amplify its impact to a considerable degree.
Working with the data
It’s needless to stress to a data scientist the importance of having accurate, comprehensive, and clean data when developing any machine learning algorithm. This principle is particularly, if not more, pertinent to recommendation systems. A well-executed recommender offers immense value, while a poorly implemented one can be costly for a business.
For recommendation systems, you’d typically focus on transactional data, user attribute information, item descriptions, and reviews. Supplemental data pertaining to these categories can also be instrumental.
When working with transactional data, it is essential to check the recency of the data. Taking historical data that is very old, may not be very fruitful as buying trends and patterns tend to change over time. In theory, you would consider the past year or two, but it depends on each business and the volume of data available.
Working with BI Dashboards
Due to these shifting patterns and customer behaviors, a comprehensive portfolio analysis, customer segmentation, and trend assessment are imperative before initiating the development of the recommendation system. The findings from such rigorous analyses can shed light on pivotal considerations for the system’s creation. They also offer a deeper insight into the business’s operational framework and revenue strategy. Furthermore, these analyses can pinpoint actionable insights and directions for the business even before the recommender’s development begins.
BI Dashboards
Testing the algorithm
During the development of machine learning algorithms, it’s vital to separate the data into training and test sets before diving into algorithm creation. The construction of an appropriate test set is a preliminary and necessary step, differing based on the dataset in question.
Evaluating the performance of recommendation systems is crucial to ensure that users receive appropriate and relevant recommendations. These are various metrics to evaluate the different aspects of recommendation systems.
- Accuracy Metrics: These gauge the correctness of the recommendation outputs. Examples include Mean Absolute Error (MAE), RMSE, Precision, Recall, and AUC.
- Coverage Metrics: These reflect the reach and comprehensiveness of the recommendations. They encompass Catalog Coverage (percentage of items from the catalog that get recommended), User Coverage (percentage of users who receive recommendations), and Item Coverage (percentage of items that get recommended at least once). Not only do these metrics shed light on the system’s performance, but they also offer insights into the business’s offerings.
- Diversity and Novelty Metrics: These metrics evaluate the variety and freshness of the recommendations. They indicate how different and unexpected the suggestions are to users.
Apart from these quantitative metrics, a qualitative evaluation of the system is essential. This ensures the system aligns well with the stakeholder needs. Additionally, it’s imperative to assess the system’s computational speed and efficiency.
Deploying the System
Taking the right steps while deploying a recommendation system is crucial in ensuring its effectiveness and enhancing the user experience. One critical aspect to consider during deployment is speed. The system should be able to provide recommendations quickly without compromising performance.
To achieve this, the recommendation system can be integrated into the application’s codebase as a module. This approach allows for faster response times since it eliminates the need for API calls. However, this method may not be feasible for all scenarios, and in such cases, using an API endpoint is more practical.
When utilizing an API endpoint, it’s essential to consider the computational cost of the algorithm and the speed of delivery. To address these concerns, precomputing recommendations and storing them in a database can help reduce the computational load. For instance, instead of calculating similar items on the fly, which requires a similarity search operation using metrics like cosine similarity or Jaccard index, you can precompute and store similar items in advance. Additionally, user segmentation can also be done ahead of time based on attributes, making it easier to retrieve relevant data from the database when needed.
In summary, when deploying a recommendation system, it’s vital to carefully evaluate the trade-offs between computational complexity and speed to ensure optimal performance and a seamless user experience.
Final points
Recommendation systems can be transformative for a business, often leading directly to increased sales. Beyond this immediate impact, the creation of such systems can be enlightening. This developmental journey can uncover deep insights into a business and pave the way for its evolution into a data-centric entity. In conclusion, stakeholders are equipped to make decisions rooted in data, thanks to the new data platforms and analytical tools that have been established.
To further your business’s data evolution and craft a powerful recommendation system, or to initiate analytics and other AI-driven solutions or data endeavors, swing by our site and get in touch with us:
Connect with me on LinkedIn:
https://www.linkedin.com/in/atul-ranjan-83025622a/
The images above are used from the H&M Data collection from a personal project:
View their data here:
https://www.kaggle.com/competitions/h-and-m-personalized-fashion-recommendations/data