Bing versus Google: Test 1 - Handwriting WPM
At orientation at CMU today, someone asked me about my thoughts on Bing since I'm in Very-Large Information Systems. I said that I personally think Google is better for most searches, but Bing is designed slightly different as a decision engine (it is optimized for tasks that require multiple search queries as opposed to one-and-done search tasks) and it is still a useful tool.
I decided to take notes on some search queries that I actually use. Now to be really fair, I should keep track of every search query I do, but I'm not sure I have time to analyze every query. So we'll start with some notable queries which I consider "reasonably difficult."
Task 1:
I'm considering buying a tablet. One thing I'd like to know before I buy it is how does writing on a tablet compare to writing on paper and typing.
I suppose I'm still in a very Google state of mind so I break this down into several queries.
Query 1.1: how fast do you handwrite
Type - Natural Language Question (quite ambitious)
No-scorlling Analysis
Google #1 - interesting related query, but doesn't answer my question
Bing #1 - pretty much the same related-but-not-the-same topic as Google.
Google #2 - How fast do Japanese people handwrite? That's gotta be major effort points for trying. Unfortunately, I actually care about how fast people write in English. Close, but no cigar.
Bing #2 - How to handwrite nice from Read Fast Now. Wrong topic, I'm not interested in writing nicer, but faster. And the word "Fast" (from "Read Fast Now") is in the title but isn't actually related to the topic of the document - the topic of the document is How to handwrite nice. I checked out the page and the word fast appears once on that page - at the footer of the page. I'm going to say that's a major deduction because that's not what the user expected. I know that it's difficult to disambiguate whether or not I really wanted a website with that name or a website about that topic, so not casting blame here, but truth is, it wasn't what the user expected.
Google #3 - It's about thank-you letters. Definitely not related to what I want. (It came up because they discuss whether thank-you letters should be handwritten)
Bing #3 - Do you have to handwrite to be a doctor. Not what I want, but at least the topic is about handwriting instead of thank-you letters.
No-scrolling verdict: They're both pretty bad, but I'd say Google is less bad because of #2.
First page
Google - nothing useful, but the "Read Fast Now" page came up at #4.
Bing - nothing useful either
Suggestions
Google - No related searches suggested. =( On the positive note, I clicked discussions, and I got some related results about typing versus handwriting. Unfortunately, the snippet text doesn't tell me the answer. However, I clicked on the first such result, and it does contain the answer I'm looking for: "even the fastest handwriters only write max 25 words a min." Unfortunately, this is halfway-down a really long discussion thread (improving search results for discussion threads was the topic of my research at UCLA).
Bing - Top one is "Christian fasting", then a lot about typing fast, and an oddball about getting a loan fast. Now I'm kind of interested about the Christian fasting result. Does Bing know I'm Christian? So to find out: I open up Bing again. I delete my search history. I type in "how fast do you handwrite" again. Christian fasting still shows up. Since it's not related to my history, the top suggested search is pretty bad because it has nothing to do with what I'm looking for. The usage of "fast" as "not eating food" isn't even the same usage I meant ("quick").
Verdict - Google does less poorly. It was possible for me to find the answer I was looking for, but not easily. When I actually did this query, I gave up after looking at the top 3 results and tried a better query. Of course, I don't really blame them because natural language questions are extremely difficult to handle, especially since the query without stop words is either: "how fast handwrite" or "fast handwrite"
Query 1.2: handwriting WPM
Type - single information request
I made a more specific query. I want some WPM metric (average, fastest, approximate) handwriting text. Let's see the side-by-side again.
No-scroll Analysis
Google #1 - There's the answer is right there in the bolded snippet text. 282 WPM in shorthand competitions. It's kind of disturbing that this number is 10x bigger than the other number. That's because it's shorthand. This result meets the criteria I set out: it gave me some WPM metric of how fast people handwrite. Unfortunately, this isn't actually very useful to me, because I'm not a world-class shorthand handwriter. You see why Bing is opting for the multi-searches per task model, because even when Google did it right, it did it wrong. Reading the actual wikipedia page does give the right answer.
Bing #1 - A definition of WPM in the snippet text. Great. Thanks. You know, the snippet text is totally unhelpful, but this was actually a quality link. It has the answer in it, and it's a pretty good page about WPM in typing, 10-key numeric entry, handwriting, and reading,.
Google #2 - Document is a PDF about handwriting speed and legibility. It looks like it'll provide the desired answer. I clicked it, it tells me about the handwriting speeds of elementary to middle school-ers under various styles. Tops out at 18 WPM for 8th graders in a certain style. That fits the desired search criteria, but again, it's not quite what I expected.
Bing #2 - Bing gets the Wikipedia entry that was Google #1, except the answer isn't anywhere in the snippet text. To note, there's a direct link to the handwriting subsection of the Wikipedia page. I clicked the Handwriting subsection and bam! "The average human being hand writes at 31 words per minute for memorized text and 22 words per minute while copying." Exactly the answer I was looking for.
Now, Google actually has this same feature. You didn't know that? Neither did I. Seems to mee like they're putting the links in the wrong place (at the bottom after all the snippet text which I quickly skim through). Bing puts it at the top, and the link to handwriting was in bold, which may be why I was able to notice this feature on Bing but not Google.
Google #3 - I can only see the subject line of the third result (small window, fluid layout - see Side Note 1). It's about handwriting speeds for copying tasks which sounds promising again. The actual document gives me an answer of average 16 WPM for 8th graders.
Bing #3 - It's the WPM page from answers.com, which I personally hate because it just copies over pages from other websites. It's interesting that I don't get the same snippet text as the wikipedia page. It's quite similar to the snippet produced by Google #1.
Scrolling - Lots of promising results for both, answers in snippet text for both.
Suggestions - no need to discuss
Verdict - Google wins Task 1 in my view. Top 3 looked good and delivered. Bing's #3 wasn't that great, only it's #3 snippet performed. However, the top 3 link all did provide the answer if you clicked on the links.
Side Note 1: I'm puzzled why Google's search results are fixed-width (you can see the results go off the window) in the first query, but in the second query, it's fluid (it adjusted to my window size). Bing was fixed both times. Is this good or bad? Well, the advantage of fluid is that I can read the snippet text of everything on the page. The bad is that I can see fewer results. (1 less in this case)
Side Note 2: Another side note - Bing doesn't have Google's Quick View feature, and those PDFs take a long time to download and open on my slow 5.5-year-old computer, while Quick View was speedy. Google - 3.8 seconds. Bing - 10.0 seconds. That's a 2.6 performance improvement.
I decided to take notes on some search queries that I actually use. Now to be really fair, I should keep track of every search query I do, but I'm not sure I have time to analyze every query. So we'll start with some notable queries which I consider "reasonably difficult."
Task 1:
I'm considering buying a tablet. One thing I'd like to know before I buy it is how does writing on a tablet compare to writing on paper and typing.
I suppose I'm still in a very Google state of mind so I break this down into several queries.
Query 1.1: how fast do you handwrite
Type - Natural Language Question (quite ambitious)
No-scorlling Analysis
Google #1 - interesting related query, but doesn't answer my question
Bing #1 - pretty much the same related-but-not-the-same topic as Google.
Google #2 - How fast do Japanese people handwrite? That's gotta be major effort points for trying. Unfortunately, I actually care about how fast people write in English. Close, but no cigar.
Bing #2 - How to handwrite nice from Read Fast Now. Wrong topic, I'm not interested in writing nicer, but faster. And the word "Fast" (from "Read Fast Now") is in the title but isn't actually related to the topic of the document - the topic of the document is How to handwrite nice. I checked out the page and the word fast appears once on that page - at the footer of the page. I'm going to say that's a major deduction because that's not what the user expected. I know that it's difficult to disambiguate whether or not I really wanted a website with that name or a website about that topic, so not casting blame here, but truth is, it wasn't what the user expected.
Google #3 - It's about thank-you letters. Definitely not related to what I want. (It came up because they discuss whether thank-you letters should be handwritten)
Bing #3 - Do you have to handwrite to be a doctor. Not what I want, but at least the topic is about handwriting instead of thank-you letters.
No-scrolling verdict: They're both pretty bad, but I'd say Google is less bad because of #2.
First page
Google - nothing useful, but the "Read Fast Now" page came up at #4.
Bing - nothing useful either
Suggestions
Google - No related searches suggested. =( On the positive note, I clicked discussions, and I got some related results about typing versus handwriting. Unfortunately, the snippet text doesn't tell me the answer. However, I clicked on the first such result, and it does contain the answer I'm looking for: "even the fastest handwriters only write max 25 words a min." Unfortunately, this is halfway-down a really long discussion thread (improving search results for discussion threads was the topic of my research at UCLA).
Bing - Top one is "Christian fasting", then a lot about typing fast, and an oddball about getting a loan fast. Now I'm kind of interested about the Christian fasting result. Does Bing know I'm Christian? So to find out: I open up Bing again. I delete my search history. I type in "how fast do you handwrite" again. Christian fasting still shows up. Since it's not related to my history, the top suggested search is pretty bad because it has nothing to do with what I'm looking for. The usage of "fast" as "not eating food" isn't even the same usage I meant ("quick").
Verdict - Google does less poorly. It was possible for me to find the answer I was looking for, but not easily. When I actually did this query, I gave up after looking at the top 3 results and tried a better query. Of course, I don't really blame them because natural language questions are extremely difficult to handle, especially since the query without stop words is either: "how fast handwrite" or "fast handwrite"
Query 1.2: handwriting WPM
Type - single information request
I made a more specific query. I want some WPM metric (average, fastest, approximate) handwriting text. Let's see the side-by-side again.
No-scroll Analysis
Google #1 - There's the answer is right there in the bolded snippet text. 282 WPM in shorthand competitions. It's kind of disturbing that this number is 10x bigger than the other number. That's because it's shorthand. This result meets the criteria I set out: it gave me some WPM metric of how fast people handwrite. Unfortunately, this isn't actually very useful to me, because I'm not a world-class shorthand handwriter. You see why Bing is opting for the multi-searches per task model, because even when Google did it right, it did it wrong. Reading the actual wikipedia page does give the right answer.
Bing #1 - A definition of WPM in the snippet text. Great. Thanks. You know, the snippet text is totally unhelpful, but this was actually a quality link. It has the answer in it, and it's a pretty good page about WPM in typing, 10-key numeric entry, handwriting, and reading,.
Google #2 - Document is a PDF about handwriting speed and legibility. It looks like it'll provide the desired answer. I clicked it, it tells me about the handwriting speeds of elementary to middle school-ers under various styles. Tops out at 18 WPM for 8th graders in a certain style. That fits the desired search criteria, but again, it's not quite what I expected.
Bing #2 - Bing gets the Wikipedia entry that was Google #1, except the answer isn't anywhere in the snippet text. To note, there's a direct link to the handwriting subsection of the Wikipedia page. I clicked the Handwriting subsection and bam! "The average human being hand writes at 31 words per minute for memorized text and 22 words per minute while copying." Exactly the answer I was looking for.
Now, Google actually has this same feature. You didn't know that? Neither did I. Seems to mee like they're putting the links in the wrong place (at the bottom after all the snippet text which I quickly skim through). Bing puts it at the top, and the link to handwriting was in bold, which may be why I was able to notice this feature on Bing but not Google.
Google #3 - I can only see the subject line of the third result (small window, fluid layout - see Side Note 1). It's about handwriting speeds for copying tasks which sounds promising again. The actual document gives me an answer of average 16 WPM for 8th graders.
Bing #3 - It's the WPM page from answers.com, which I personally hate because it just copies over pages from other websites. It's interesting that I don't get the same snippet text as the wikipedia page. It's quite similar to the snippet produced by Google #1.
Scrolling - Lots of promising results for both, answers in snippet text for both.
Suggestions - no need to discuss
Verdict - Google wins Task 1 in my view. Top 3 looked good and delivered. Bing's #3 wasn't that great, only it's #3 snippet performed. However, the top 3 link all did provide the answer if you clicked on the links.
Side Note 1: I'm puzzled why Google's search results are fixed-width (you can see the results go off the window) in the first query, but in the second query, it's fluid (it adjusted to my window size). Bing was fixed both times. Is this good or bad? Well, the advantage of fluid is that I can read the snippet text of everything on the page. The bad is that I can see fewer results. (1 less in this case)
Side Note 2: Another side note - Bing doesn't have Google's Quick View feature, and those PDFs take a long time to download and open on my slow 5.5-year-old computer, while Quick View was speedy. Google - 3.8 seconds. Bing - 10.0 seconds. That's a 2.6 performance improvement.