SHIFT: How alternative search engines like Wolfram Alpha and Cuil can succeed

When Wolfram Alpha, a website devoted to providing direct answers to fact-based searches launched earlier this month, it was immediately and widely compared to two websites: Google and Cuil. Though some compared Wolfram Alpha to Google directly (and negatively), the website's creator stressed that Wolfram Alpha would never be a "Google killer." Wolfram Alpha isn't a search engine that will give you tons of links to choose from. Instead, it's all about facts, especially ones pertaining to numbers and math. It will not understand your queries about pop music or help you stalk your ex-girlfriend (for more about what it does do, check out this this excellent animation).

By comparing Wolfram Alpha to Google, writers were wondering if Wolfram Alpha would be a huge, this-will-become-the-most-popular-site-on-the-Internet success. And when they compared it to Cuil, they were asking, will Wolfram Alpha instead be a dismal, good-for-nothing failure.

Cuil, a search engine that claims to index more of the Internet than Google, was widely panned (read my review here) when it debuted last July and was declared all but dead by the end of the year. But the current cycle of hype and subsequent disappointment about Wolfram Alpha got me wondering whether everyone had been right about Cuil. After all, it's still around, isn't it? And though its Alexa rating isn't even in the same stratosphere as third-ranked search engine, it gets as much traffic as say, The New Yorker.

Was Cuil really such a failure? Is there room in this world for non-"Google killer" search engines to thrive or innovate? Has Cuil gotten better since it was so widely dismissed? I decided to take a closer look, and spoke to Tom Costello, the company's CEO and co-founder, earlier this week. Read about our conversation, and my thoughts, after the jump.

First, the Good News…

Costello's most compelling pitch to me was this: "The Web has gotten terribly small. People go to a very select, small group of sites, as opposed to seeing the Web in an enormously rich, broad way. Every time you do a search, you just get Amazon or eBay or Yahoo or Facebook--those are the kind of pages that show up on the front page, as opposed to all of the nooks and crannies that exist on the Web." It's true that while Amazon has a lock on Google's front page for any number of search results, I didn't get even one Amazon search result during my time browsing Cuil this week. That doesn't mean that the sites I found were better. But they were different, and that's a start for people who haven't found what they were looking for in a Google or Yahoo search.

Right now, Costello explains, Cuil is being used as an engine of last resort. I didn't love the results I found on Cuil, but they were frequently decent. And as they get better, Cuil may be able to pick off Googlers a few at a time (especially ones concerned about privacy, since unlike Google, Cuil doesn't collect information about its users).

That's the company's current strategy. Costello concedes that after lots of curious users tried Cuil out at launch — and were disappointed — traffic fell precipitously, and continued to fall until February. But he maintains that traffic has been rising steadily since then, and that that's how search engines, including Google, build traffic: Slowly and steadily.

On the Other Hand…

Cuil's claim to fame when it launched was that it indexes more pages than Google, but its downfall was that nobody could find anything on it. One of the first things people look for when they come to a search engine is their own name. Cuil was not prepared for this in its first days online. In more recent searches, I've been able to find people I couldn't find the first time around, but I couldn't find myself, even though Cuil had indexed specific articles where my byline appears. Tom Costello explained that Cuil can't find me because it doesn't recognize names that start with two initials (though J.P. Morgan doesn't seem to have quite the same problem). Because Cuil attempts to be "a little smarter," by understanding what it's looking for instead of just searching for words on a page, I'm dead to it. Cuil may have gotten a lot better at finding people in the past 10 months, but it still has major, major flaws.

Keeping up to date is another issue that plagues Cuil. On Tuesday, Obama nominated Sonia Sotomayor for the open position on the Supreme Court. Just hours later, a Google search for "supreme court nominee" reflected this, not just in Google News, but on Google's whole front page--the browser takes timeliness into account with its rankings. On Cuil, the top result for the same search was a 2006 CNN article about Democrats grilling Samuel Alito (next to a picture, oddly, of John Roberts). Costello's response is that Cuil has a small staff, and that someone is working on getting the engine to take news into account. More than twenty-four hours later, Cuil's Supreme Court search results are still all from the Bush era.

Staying timely is difficult (TechCrunch observes some of Wolfram Alpha's information was several years out of date just days after it launched), but Cuil hasn't even made many easy fixes that it needs to be competitive. For example, it still doesn't have a spell check, in case you search for the wrong spelling of "meningitis."

Can anyone compete with Google?

Is there room in the market for search engines that are nowhere near as good as the big names, but that nevertheless yield good results? Costello thinks that there is; though it was originally touted as a Google killer, it's now far humbler, hoping for steady growth, and oddball searches from people when they get home from work (Costello says that Cuil is largely used outside of business hours).

Cuil and Wolfram Alpha are unlikely to put Google (or anyone else) out of business, but that increasingly seems like a silly way to judge a website. It's obvious to me that Wolfram Alpha will be able to provide value to its users — it's quite different from anything else out there. Cuil's worth is less clear. It will need to continue to differentiate itself from other search engines, something it's trying to do now by providing timelines for search subjects. Just as Wolfram Alpha specializes in math and charts, Cuil might want to specialize as well, so that it can become the go-to search engine for something (history? academia?). If it succeeds, it could be the sleeper hit of 2012, long after the press has dismissed it as dead and gone.