Anatomy of a Large-Scale Social Search Engine

By Damon, Lion Tamer | Published: February 2, 2010

Back in October, we wrote a research paper entitled “Anatomy of a Large-Scale Social Search Engine” and submitted it to WWW 2010. We found out last week that it has been accepted, so we wanted to share a preview with you today!

Our paper was inspired by the classic Google paper, “Anatomy of a Large-Scale Hypertextual Web Search Engine”, in which Sergey Brin and Larry Page originally describe the algorithms and architecture of Google. This paper was published 12 years ago in the same WWW conference.

So our goal with our paper is to follow their example by providing a thorough presentation of the approach, architecture, algorithms, interfaces, and issues involved with Aardvark’s new social search paradigm.

The paper describes the fundamental differences between the traditional “Library” paradigm of web search — in which answers are found in existing online content — and the new “Village” paradigm of social search — in which answers arise in conversation with the people in your network. We explain that in social search:

  • Users can ask questions in natural language, not keywords
  • Content is generated “on-demand”, tapping the huge amount of information in peoples’ heads
  • The system is fueled by the goodwill of its users

We demonstrate that there is a large class of subjective questions — especially longer, contextualized requests for recommendations or advice — which are better served by social search than by web search. And our key finding is that whereas in the Library paradigm, users trust information depending upon the authority of its author, in the Village paradigm, trust comes from our sense of intimacy and connection with the person we are getting an answer from.

We also provide a detailed analysis of user behavior, and include dozens of interesting statistics. For example, of the 90,361 users we had in October 2009…

  • 87.7% of questions sent to Aardvark got answered (very high answer rate!)
  • 75.0% of users who asked Aardvark a question also answered a question for someone else (very high participation rate!)
  • 70.4% of answer feedback had a rating of ‘good’ as opposed to ‘ok’ or ‘bad’ (high quality!)

Writing a paper like this requires being more open, and sharing more information, than most small internet startups might be comfortable with. But we recognize that we have benefitted from the open culture of the scientific community, and would like to do our part. Further, we think that the opportunity presented by social search is truly significant, and we’d like to engage with the rest of the research community on the many challenges it presents. There are very interesting problems to explore around question classification, analysis of social relationships, person-to-person matching, maintaining a question/answer economy, and many other areas.

I wrote the paper with my good friend, Sep Kamvar, who started Kaltix, a search company acquired by Google in 2003. He led personalized search at Google for several years, and is now a professor at Stanford — and an advisor for Aardvark. But this paper would not be possible without the hard work and support of the whole Aardvark team over the past few years. And, of course, Aardvark itself would not be possible without the continued enthusiastic contributions of all of you, our users!

We’re very excited about presenting this at the WWW conference, which has been providing a great forum for web research for 19 years, and we hope to see you there in April.  So take a read, and let us know what you think…

(Note: the preview version we’re sharing here has some changes inspired by the great reviewer comments we received; we may make further changes for the camera-ready version that will be presented at the conference.)

26 Comments

Why Machines Need People

By Alison, Purchaser of Exotic Animals | Published: February 1, 2010

Our CTO, Damon Horowitz, was recently invited to speak at the TEDxSoMa event… and here’s the video!

Watch it here: http://www.youtube.com/watch?v=1YdE-D_lSgI

Damon at TedxSoMa

Leave a comment

Feature Update: New Email Designs!

By Rob, Assistant Curator of Birds | Published: January 22, 2010

What we heard from you

“Sometimes it’s hard to read the emails from Aardvark — it’s hard to tell which part is the question or answer, and how you’re supposed to respond.”

“The emails from Aardvark look too boring!”

“I would like to see the same answer options for email questions as IM questions.”  (from our Community Forum)

What we did

We just launched a completely new design for the emails that Aardvark sends you!

The new emails are easier to read, include more details about the person you’re interacting with, and give you all sorts of new options for how to respond. You’ll see special versions of the email when Aardvark sends you a question that you might be able to answer, an answer to something you’ve asked, or a message from another person.

For instance, now when Aardvark sends you a question, it looks like this:

You can always ask a question of your own via email by sending it to aardvark@vark.com — and you can just put the question topic in the subject line.
We hope you like the changes… let us know what you think in the comments or the Aardvark Community Forum!

3 Comments

How Aardvark helped me move

By Anna, Spider Pig | Published: January 13, 2010
Hi everyone!  I’m Anna, and I work in Operations at Aardvark.

The last few weeks have been crazy for me as I passed through a young-adulthood rite of passage: getting my first apartment on my own.  I’d been living with roommates, and there was definitely a lot I didn’t know about finding a place, dealing with landlords, and diagnosing those oh-so-mysterious home repair issues that come up.

Lucky for me, I had the Aardvark community to call on for help every step of the way.  I wanted to share some of my experiences since I think they’re a great illustration of the diversity of experts out there on Aardvark.

Step 1: Choosing a place!  I found a place I really loved, but the property management company had a sketchy reputation.   Aardvark users came to my aid (http://vark.com/t/70166f) and I got a good list of questions to vet the company with.  I also got a great suggestion to ask them for references, and talked to a nice tenant in the same building who put my mind at ease.

Step 2: Packing!  After being in the same place for 3 years, I had a ton of extra stuff.  I turned to Aardvark and had a great pointer in 2 minutes to a local thrift store that would come to my house and pick stuff up: http://vark.com/t/cde5c3

Step 3: Moving my cat!  My cat hates moving — last time she wedged herself in a tiny hole in the laundry room and wouldn’t come out for ages.  Aardvark’s cat lovers gave me great advice on how to make things go more smoothly: http://vark.com/t/d38452

Step 4: Oh no, the water coming out of the faucets had a ton of weird black stuff in it!  Lots of plumbing experts gave me ideas about what might be going on:  http://vark.com/t/afa1d0.  And then there was this other problem with the stove — again, Aardvark users offered a variety of helpful perspectives: http://vark.com/t/b04110.

Step 5: Wow, that old college futon I broke out of storage for my new place is really uncomfortable.  Aardvark users gave me helpful suggestions on how to get a good night’s sleep without investing in a whole new bed: http://vark.com/t/9a7f95

Step 6: Hmm, maybe I should have renter’s insurance.  Other San Francisco renters let me know what companies/coverage had worked for them: http://vark.com/t/156a67

Thanks to everyone in the community for helping me out!  If any of you have had great experiences using Aardvark to help you with life’s larger problems, I’d love to hear about them (or post about them on our Facebook Page!)

4 Comments

Top 10 Aardvark Developments of 2009

By Alison, Purchaser of Exotic Animals | Published: December 21, 2009

Aardvark has seen tremendous improvements since launching a private beta early in 2009. We’ve consistently had major releases every few weeks, and we’re only accelerating. :-)

Since Top 10 lists seem to be all the rage this time of year, I thought I’d share my top 10 favorite Aardvark developments of 2009:

10. All the great press.
A year ago no one had heard of Aardvark: we were in “stealth mode” up until March, when we launched an invite-only version of Aardvark at the South By Southwest conference. Since then we’ve received amazing reviews by some of the top technology journalists in the world, and Aardvark’s founders have been featured at major industry conferences. We’re humbled by the reviews and extremely energized by the attention!

9. Share your conversations with friends.
There are lots of times when you want to share conversations with friends — when you’re coordinating plans, spreading interesting tidbits, or just showing off how much you’ve helped someone! In August we launched the Share Transcript feature… and we’re already seeing almost 100 conversations shared on Twitter every day!

8. Real-Time Twitter integration.
Aardvark is a contact that you can get in touch with from anywhere. We want Aardvark to live where you live, so we integrate with all of the major IM networks, including AIM, Yahoo! Messenger, MSN Messenger (Windows Live), and Gtalk (and Gmail Chat). In July, we launched Aardvark’s integration with Twitter, which lets you Aardvark questions via Twitter by including ‘@vark’ and a question mark in any Tweet. (And ReadWriteWeb listed Aardvark as one of the most important Real-Time Web products of the year!)

7. Ever-increasing ways to share your feedback.
We’ve said it before and I’ll say it again: we *love* user feedback. We read and respond to every piece of feedback we get — written in to feedback@vark.com, or in response to the various user research that we conduct. In June, we opened up our Community Forum, which lets you suggest new features, see other peoples’ suggestions, and add comments and votes. Thousands of people have already contributed to the forum; thanks!

6. Refer a question to someone who knows.
Sometimes you don’t know the answer to a question, but you know that one of your friends has an answer. In June, we launched the refer feature over IM, which lets you send a question to a friend. Soon after we launched refer interfaces on the iPhone and the Aardvark website, too!

5. Research crossed the threshold.
Our research team is constantly making improvements to the AI system that figures out which Aardvark users are the best match to answer a question.  About halfway through this past year, our technology became ready for prime time: we passed the threshold of 90% questions answered, most in under 5 minutes!

4. Aardvark Mobile on the iPhone.
Social Search is invaluable when you’re on the go — it’s the best way to get quick, trusted answers about neighborhood places, travel tips, entertainment recommendations, and lots more. We introduced Social Search on the iPhone in September with our first release of Aardvark Mobile; the response was spectacular, and Apple chose to feature Aardvark as a top iPhone App.

3. Aardvark on the web at www.vark.com.
We initially built Aardvark with an IM interface, and gradually started adding compelling interfaces over email, Twitter, and iPhone. Then in October we released our brand new website, the introduction of Social Search to the web.

2. New team members.
In the past year, Aardvark has grown from 15 people to 30 of the best engineers, researchers, designers and operators in the entire technology industry. And we’re still hiring! We also have gathered together a fantastic team of advisors including some of the best minds in the industry. A bigger and better team has led to faster development of new features, better user support, and of course a smarter Aardvark.

1. All our amazing new users!
Aardvark’s userbase has nearly doubled *every month* since we launched our invite-only version in March. Aardvark users are still kind, helpful, and knowledgeable. Just think of how much helpful knowledge has been shared!

I’m excited for the acceleration in store in 2010!

2 Comments

How we do qualitative user research at Aardvark

By Kavi, Gorilla Therapist | Published: December 14, 2009

Being “user-driven” in our product development is more than an attitude — it’s a rigorous and ongoing process. And it’s ultimately extremely rewarding. We highly recommend it to other product developers.

We’ve written previously about how we incorporate user research into our development process and how we manage user feedback. Recently we’ve been asked to delve into more detail about how we do qualitative user research… we’re happy to!

We primarily use three approaches:

Phone Interviews
When we’re exploring an opportunity to improve Aardvark — before we’ve even started designing features — we usually conduct phone interviews with existing Aardvark users. Talking to users and knowing their expectations ahead of time guides us in the right direction as we come up with ideas. It also helps immensely to pay attention to the language that people use to describe the Aardvark experience. In the past, we’ve conducted phone interviews both with experienced and brand new users. We typically conduct a round of 5-10 phone interviews every few weeks to help us think through a problem.

In-person Testing
Before we build any feature on Aardvark we try to validate it with real people. We invite existing Aardvark users into our office — or recruit people off of Craigslist who have never been exposed to Aardvark — and ask them to participate in 30-minute user tests. We typically run a series of 5 or 6 tests in an afternoon, about once a week. During these sessions we show people potential new features (using click-through prototypes, or paper prototypes, or wizard-of-oz experiments) and listen to their reactions using the talk-aloud protocol, which encourages people to verbalize the actions they’re taking. We jot down every important and memorable quote, while observing the person’s reactions and behaviors. Usually, after only a handful of tests, we notice clear patterns in the way that people react to certain ideas or interfaces. These patterns enable us to tweak the design or abandon the idea altogether.

Email Outreach
The most common type of research we do is via email outreach. When we release a new feature, or a tweak to an existing feature, we typically only introduce it to a small group of users. Then we send emails to 10-20 users who have been exposed to the feature, requesting feedback. These emails usually contain three or four questions related to the feature and are carefully worded to prevent bias from the user. For example, “Did you see feature X?  Did you consider using it?  Why or why not?” We intentionally leave most questions open-ended so that a user can respond with their honest feedback and bring up any concerns that our more pointed questions may not capture. The response rates to these feedback requests are very high (our users rock!), so we typically use this research mechanism several times per week.

Pulling it all Together
When our user research on a specific topic is complete, we summarize our findings and pick out representative quotes from users. We share these with the entire team in a weekly email update.

A huge thanks to everyone who has ever helped us by participating in a phone interview, coming into our office for a test, or responding to one of our emails with feedback!

5 Comments

How you can improve Aardvark in 30 seconds

By Alison, Purchaser of Exotic Animals | Published: December 7, 2009

I know that the Aardvark community is a helpful bunch — that’s what Aardvark is all about!
Here are a few helpful things you can do in 30 seconds or less to improve Aardvark for someone else…

Refer. Can’t answer a question, but know someone who might be able to answer?  Just type ‘refer’ (if you’re talking to Aardvark via Instant Messenger), or refer a question to a friend on the Aardvark website.  Aardvark will send your friend a link to a customized web page where they can answer.  Referring is a great way to help someone find an answer - and a great way to compliment your friend on their expertise!  Read more about referrals here.

Tag. If one of your questions is tagged with the wrong topic, just type ‘tag’ to fix it!  And if Aardvark sends you a question that’s mis-tagged, you can re-tag that, too!  Just type ‘tag:’ followed by a new topic or set of topics (e.g., ‘tag: Berlin, hotels’).  Tagging questions helps people receive answers more quickly!

Tutor. Aardvark is growing quickly - which is awesome! - but it also means that we have lots of new users.  Some people play around with Aardvark at first, to learn what it’s all about.  You can help these people with tips about how to use Aardvark…
If someone asks, ‘Who was the first president of the United States?’ or ‘Why did the chicken cross the road?’, chances are, they’re just testing Aardvark out to see how it works.  I usually give them a tip, like, ‘The first president was George Washington. Just as a tip, Aardvark is better for subjective questions and things that are difficult to find on web search engines. I use it a lot for restaurant recommendations and help with complicated tech issues.”

Flag. We work hard to filter out spam and inappropriate/unhelpful questions and comments, but occasionally offensive messages slip through.  Help everyone in the Aardvark community by flagging inappropriate behavior.  Just type ‘flag’ whenever you see a message that you think isn’t appropriate for the Aardvark community. This will notify the Aardvark team to follow up.

Say “Thanks!” In my opinion, this is the most important habit of all. When you get an answer on Aardvark, it’s coming from a real person who took the time to help you out. Show your appreciation by saying ‘thanks!’. Those six little letters can mean a lot to someone and will help perpetuate goodwill in the Aardvark community.  And it would make your mother proud. :)

Do you have other ideas about how people could help improve the Aardvark community?  Leave a comment and share your thoughts!

10 Comments

Introducing Aardvark Conversation of the Day!

By Alison, Purchaser of Exotic Animals | Published: November 19, 2009

Every conversation you have on Aardvark has a link that you can share with friends.  Looking at recently shared links, I’ve seen many awesome conversations that have given me some great insight, or just made me laugh out loud.  Here are some of my favorites - the helpful and the humorous:

“I am interested in wanting to brew beer. What is the best guide, book or otherwise, to get started brewing delicious beer? I prefer IPA’s and Porters” http://vark.com/t/970b56

“I accidentally dropped my MacBook pro 15″ in a large bin of chili. It was in there for about 2 minutes or so. How can I salvage it?” http://vark.com/t/dd59e3

“I am wearing a full puppy costume to work Friday, and I have a Halloween party that night. Somebody asked me out for a coffee date after work and is sooo cute, and excited. Would it be a bad date if the person you were dating showed up dressed like a cartoon puppy?” http://vark.com/t/36dec5

“I own a internet marketing company (start up) and I am now considering building a blog for the company. I have often seen other social media companies go into full length detail when discussing social media matters. My question is, who should i have in mind as the audience when writing articles for my blog?” http://vark.com/t/18a02e

“How would you figure out how many cats there are in the US that are named Gary?” http://vark.com/t/e8065e

Since a lot of people seem to enjoy these transcripts, we’ve decided to start sharing an Aardvark Conversation of the Day!   Every day, I’ll choose a great Aardvark conversation that someone has shared and post it on Aardvark’s Facebook page and Twitter account.

I’ll need your help!  When you have a great conversation, send it my way… Just find the question in your question history and copy the shared link at the top of the page.  Then email it to me at alison@aardvarkteam.com (or send it to me on Twitter - @AlisonatVark).

If your conversation gets chosen as the Conversation of the Day, I’ll send you an Aardvark t-shirt and stickers as a thanks!

share

2 Comments

The Research behind the Aardvark

By Damon, Lion Tamer | Published: November 12, 2009

We’ve invested heavily in advanced research since we started Aardvark two years ago.  This is an unusual move for a small startup company, which is expected to spend all of its time on a panicked frenzy of short-term releases.  So why did we do it?

Well, we noticed that the explosion of online information about people in the Web2 era (on social networks, blogs, tweets, review sites, etc.) was parallel to the explosion of online web pages in the late 1990s.  Back then, the timing was right for a major wave of research into IR-style web search engines, developing algorithms to find the right web page for a query.  Similarly, the timing was right when we started Aardvark for a major wave of research into social search, developing algorithms to find the right person to answer a question.  Naturally, we wanted to be a few years ahead of this curve.

The “Social Search” research problems we’re working on today are related to the “Web Search” research problems that have received so much attention over the past decade, since the components of the underlying search systems are analogous.  However, while the abstract aims of each component in Aardvark are analogous to those in a corpus-based search engine, the means of achieving those aims are quite different:

Query Analysis

  • With Web Search, the challenge is extrapolating a user’s intent — the question she really has in mind — based on a keyword search query.
  • With Social Search, the challenge is determining the areas of expertise needed to answer a natural language question:  figuring out what subject matters the question concerns.

    Crawling and Indexing

    • With Web Search, the challenge is crawling large numbers of web pages and indexing them based on the terms and metadata they contain.
    • With Social Search the challenge is attracting socially connected users (i.e., crawling large social graphs) and indexing them based on topics of expertise: figuring out what topics people know about, as expressed in their online profiles, the content they have authored, etc.

    Ranking

    • With Web Search, the challenge is ranking web pages in search results based on relevance (i.e., match to the query) and quality (i.e., an authority score).
    • With Social Search, the challenge is ranking candidate answerers to match with the question (i.e., with the relevant expertise) AND to match with the asker:  finding answerers who want to help the asker, and who the asker will trust, based upon their affinities and sense of connection.

    User Interface

    • With Web Search, the challenge is displaying search results with snippets and highlights in an easily-scannable manner.
    • With Social Search, the challenge is facilitating a chat-like interaction between the asker and the answerer: designing a user experience that removes the social and logistical barriers to having a quick conversation with someone you are connected to.

      To address these diverse challenges, we throw a diverse set of tools at the problem:  Aardvark makes heavy use of natural language processing, machine learning, and quantitative analysis more generally.  We use a combination of off-the-shelf tools and custom-built frameworks and algorithms.  We are strongly data-driven, and are constantly evaluating the latest tweaks (and outright wacky ideas) to see how they affect the overall performance of the system.

      As a result, we’re making great progress on each of these fronts… but there are many more interesting problems ahead.  Our team today includes a number of experienced researcher scientists and research engineers… and we’re still recruiting!  (If you’re a talented researcher, you should consider joining us.)

      If you want to hear more about how we’ve approached some of the text processing and semantic analysis research problems, come to my talk on “Text and Meaning” at the Web2.0Expo next week.

      Web 2.0 Expo New York 2009

      5 Comments

      How your Feedback helps

      By Alison, Purchaser of Exotic Animals | Published: November 5, 2009

      You may have noticed that we’re constantly soliciting your feedback here at Aardvark. This is because we strongly believe that Aardvark should be a user-driven company. After all, Aardvark wouldn’t work very well without you. (You can read more about our approach to design and development here.)

      Fortunately, our users are *awesome* and help us a ton with this process. Every day we receive hundreds of pieces of feedback in our community forum, email inboxes, iPhone app, on our Facebook page and via Twitter.

      So what happens with all this great insight?

      Community Forum
      Every time you write something in the community forum, we receive an email. This lets me know to stop by the forum and check out the new suggestions and comments. I try to reply to each post so you know we’re reading and considering everything you tell us. Every couple weeks, a team gets together to look at the top most requested features on the forum. We’ve made it a goal to always either be working on the most requested features, or have a legitimate reason why we can’t and pass this along to our users.

      Email and iPhone Feedback
      One of the most important parts of my job is tracking and responding to the feedback that our users send us directly. Much like in the community forum, we want to make sure that you know we’re getting your feedback, and also let you know what we’re working on. Additionally, it has been a great way to build relationships with our users and understand the aspects of Aardvark that are most important to them.

      Long story short, email me or feedback@vark.com and I promise to write you back. If you email someone else on the team - no worries - your feedback is still forwarded along and tracked.

      Twitter and Facebook
      We use Twitter and Facebook for what we call ‘overheard’ feedback. We don’t just want to know what you’re telling us - we also want to know what you’re telling each other. I constantly have a Twitter stream open that shows me every time someone mentions Aardvark. As I read through them, I mark any tweets that I think the team would find interesting, and respond to anything that I can help with. You can also provide feedback or get help by tweeting to me directly (@AlisonatVark).

      I’m on Facebook all day as well. (Woe is me, I know.) I’m constantly refreshing the page to see the content you share and to respond to feedback, questions and concerns…so be a pal and post more often, so I can spend more of my workday on Facebook… :)

      How it all comes Together
      As cool as it might be if I were the one-and-only all-knowing god of what people thought about Aardvark, that wouldn’t help very much as we build new features. In order to make sure everyone at Aardvark understands the perspectives of our users, I get together with the user research team once a week. We put together a document called the ‘Weekly Learnings’ which we send out to the entire Aardvark team. This includes the nice things people have said about us, the findings from any user tests, features requested by users (with direct quotes), feedback from employees, and the most relevant tweets from Twitter (positive and negative).  Then every Friday, the Aardvark team gets together and discusses these Learnings — and this is how we decide what things we want to prioritize to address next.

      By now, you should know the request that comes next: Send us your feedback! Check out the community forum, shoot us an email, drop me a note on Twitter and/or stop by our Facebook page.

      We really do love hearing what you think — all the positive and the negative — since what we do is driven by user input. Plus, if you don’t leave us feedback, I’m out of a job… :)

      6 Comments