Implementation of Lambda Architecture: A Restaurant Recommender System over Apache Mesos

摘要

With the rapid development of web, mobile apps and IoT devices, a huge volume of data are created every day. In fact, these data are not only large, but also generated quickly and with a variety of different formats. Many companies are forced to face the problem of processing these large data in near real-time. Within different application domains, some data are required to be processed in real-time, while the others can use offline batch processing. In this research, we will follow the Lambda Architecture to design and implement a restaurant recommender system by using several open-source software. We will use Apache Mesos to manage the cluster automatically instead of manually setting up the cluster, which can allocate cluster resources dynamically and scale up or down easily. Deploying Lambda architecture on Mesos has the advantage of resource sharing and fault tolerance. Since the Lambda Architecture is a generic data processing architecture, the system we built not only can serve as a recommender system, but also is extensible for various applications.