The future is no longer an entirely unknowable quantity, it seems. Or, at least, the future as it comes in 140 character chunks on a particular social media service. An MIT professor believes that he has created an algorithm that can predict with almost 100 percent accuracy what topics will end up trending on Twitter.
The algorithm is the work of Associate Professor Devavrat Shah and his student, Stanislav Nikolov, who believe that – if “trained” correctlym it will predict with 95 percent accuracy which topics will trend on Twitter within the next hour and a half before Twitter’s own internal algorithms pick them up, with some topics being predicted as far ahead of Twitter’s own recognition as four or five hours.
MIT’s own press release notes that “The algorithm could be of great interest to Twitter, which could charge a premium for ads linked to popular topics,” which is a mild understatement; it’s worth pointing out, as GigaOM does, that Nikolov is actually a Twitter employee, and therefore may have had a better idea of particular metrics – or what Twitter would find of great interest – than other outsiders. However, the MIT release goes on to suggest that it’s more than just Twitter that might be interested in the math that makes this thing run, saying that the algorithm ” represents a new approach to statistical analysis that could, in theory, apply to any quantity that varies over time: the duration of a bus ride, ticket sales for films, maybe even stock prices.”
Shah believes that his algorithm can be scaled or transferred onto “any sequence of measurements performed at regular intervals,” noting that – although the more complex the system it tries to predict the behavior of, the more sizable the computing power needed to do so accurately – “our computation scales proportionately with the data” and “it is perfectly suited to the modern computational framework.” Using Twitter as a test for the algorithm works in its favor, he says, because “the training sets are very small, but we still get strong results.” Because of the strength of those results, he remains confident that the same algorithm will work in any situation as long as the correct subset of data can be identified to train the system to recognize the right information.
Shah and Nikolov will be publicly presenting the algorithm at the upcoming Interdisciplinary Workshop on Information and Decision in Social Networks later this month at MIT, but even the announcement of its creation has people curious to see how successful it can be. Ashish Goel, a member of Twitter’s technical advisory board told MIT that “People go to social-media sites to find out what’s happening now, so in that sense, speeding up the process is something that is very useful.”