There is an interesting article by Clive Thompson in this Sunday NY Times Magazine on the progress of the $1 million Netflix challenge to improve their recommender results by 10%. The article does not shy away from the mechanics of search algorithms, including a discussion of how Singular Value Decomposition is useful for summarizing the search space. This level of detail often left out of the popular press when describing algorithms.
(By the way, one of the earliest and most cited papers on SVD is Deerwester, S., Dumais, S. T. , Furnas, G. W., Landauer, T. K., & Harshman, R., Indexing by Latent Semantic Analysis, JASIS, 1990.)
Thompson goes to talk about the “Napoleon Dynamite” problem and Netflix’s internal debate of whether hiring cinephiles to tag all 100,000 movies would help. The article ends with Pattie Maes questioning whether computer search alone with ever be enough. Instead, future recommender systems might need to a mixture of algorithmic and social networking tools.
In all, it the article provides a great discussion of rather complex issues about recommender systems. He makes it clear how progress can measured empirically, how difficult it is to improving algorithms even by a fraction of a percent, and science can advance through an open dialogue among highly competitive research teams.
Sunday, November 23, 2008
Subscribe to:
Post Comments (Atom)
3 comments:
I forgot to mention that there is an interesting interview with the author on NPR's On The Media. You can hear it at http://www.onthemedia.org/transcripts/2008/11/21/08.
..."Thompson goes to talk about the “Napoleon Dynamite” problem and Netflix’s internal debate of whether hiring cinephiles to tag all 100,000 movies would help. "
This is like the infamous comment heard 100 years ago as well as today in some circles, that people who like to read will make "good" librarians. [not!]
What is surprising me here is that Netflix even has hope anymore--i thought their biggest challenge lately were those big red boxes outside the supermarkets renting dvds for $1. whatever happened to semi retired engineers giving back to the community i.e. mentoring after school, instead of chasing another $1 million? :-\
The Endeca online catalog at North Carolina State University might be of some interest--they are one of the first to take on similar searching types / if you like this then maybe that etc from corporate retail gurus such as Home Depot and Walmart websites.
http://www.endeca.com/
http://www.lib.ncsu.edu/endeca/faqs.html
Watts in "Six Degrees" has a section that explains why the explanation for things like "Napoleon Dynamite" or "Harry Potter" are random events at the edge of network - rather than the elitist theory that Gladwell presents in “Tipping Point” (Cascades verses cool preppies at some NYC private school) . The network explanation makes one believe that it is probably not possible to in retrospect evaluate an individual’s reaction to a fad.
Post a Comment