We’ve invested heavily in advanced research since we started Aardvark two years ago.  This is an unusual move for a small startup company, which is expected to spend all of its time on a panicked frenzy of short-term releases.  So why did we do it?

Well, we noticed that the explosion of online information about people in the Web2 era (on social networks, blogs, tweets, review sites, etc.) was parallel to the explosion of online web pages in the late 1990s.  Back then, the timing was right for a major wave of research into IR-style web search engines, developing algorithms to find the right web page for a query.  Similarly, the timing was right when we started Aardvark for a major wave of research into social search, developing algorithms to find the right person to answer a question.  Naturally, we wanted to be a few years ahead of this curve.

The “Social Search” research problems we’re working on today are related to the “Web Search” research problems that have received so much attention over the past decade, since the components of the underlying search systems are analogous.  However, while the abstract aims of each component in Aardvark are analogous to those in a corpus-based search engine, the means of achieving those aims are quite different:

Query Analysis

• With Web Search, the challenge is extrapolating a user’s intent — the question she really has in mind — based on a keyword search query.
• With Social Search, the challenge is determining the areas of expertise needed to answer a natural language question:  figuring out what subject matters the question concerns.

Crawling and Indexing

• With Web Search, the challenge is crawling large numbers of web pages and indexing them based on the terms and metadata they contain.
• With Social Search the challenge is attracting socially connected users (i.e., crawling large social graphs) and indexing them based on topics of expertise: figuring out what topics people know about, as expressed in their online profiles, the content they have authored, etc.

Ranking

• With Web Search, the challenge is ranking web pages in search results based on relevance (i.e., match to the query) and quality (i.e., an authority score).
• With Social Search, the challenge is ranking candidate answerers to match with the question (i.e., with the relevant expertise) AND to match with the asker:  finding answerers who want to help the asker, and who the asker will trust, based upon their affinities and sense of connection.

User Interface

• With Web Search, the challenge is displaying search results with snippets and highlights in an easily-scannable manner.
• With Social Search, the challenge is facilitating a chat-like interaction between the asker and the answerer: designing a user experience that removes the social and logistical barriers to having a quick conversation with someone you are connected to.

To address these diverse challenges, we throw a diverse set of tools at the problem:  Aardvark makes heavy use of natural language processing, machine learning, and quantitative analysis more generally.  We use a combination of off-the-shelf tools and custom-built frameworks and algorithms.  We are strongly data-driven, and are constantly evaluating the latest tweaks (and outright wacky ideas) to see how they affect the overall performance of the system.

As a result, we’re making great progress on each of these fronts… but there are many more interesting problems ahead.  Our team today includes a number of experienced researcher scientists and research engineers… and we’re still recruiting!  (If you’re a talented researcher, you should consider joining us.)

If you want to hear more about how we’ve approached some of the text processing and semantic analysis research problems, come to my talk on “Text and Meaning” at the Web2.0Expo next week.