Rediscovering Discovery — How We Find Things, and Its Implications « The Scholarly Kitchen

I recently attended the Midwinter ALA conference in Dallas. The longest line was at Starbucks.

What brought me to Dallas was an invitation to participate in a panel sponsored by Sage on discovery. Sage had commissioned a white paper on the topic, which you can find here. It’s a good paper, which I recommend to one and all. The authors of the paper were on the panel as well and spoke authoritatively to the question of how libraries are implementing new means of discovery. I had not realized how complicated the situation was. What was clear to me, though, is that librarians and publishers alike have an interest in improving finding mechanisms: librarians because they want their collections to be used, publishers because such usage translates into a stronger brand, which can help in making the case to purchase the next product.

Tags:

Searching in ebooks: A unique use case that requires a unique approach

This is part of an ongoing series related to Peter Meyers’ project “Breaking the Page, Saving the Reader: A Buyer & Builder's Guide to Digital Books.” We’ll be featuring additional material in the weeks ahead. (Note: This post originally appeared on A New Kind of Book. It’s republished with permission.)


Not all ebook search monocles are equal. Options range from non-existent (Hey, Kobo! If there’s room in the programming budget for virtual reading awards like the Inverted Comma and the BookLover, then it’s time to spring for a search tool, too!), to roughly implemented (Nook), and from nicely polished (Kindle, iBooks) to fully instrumented (Inkling).

To help make sense of what works versus what doesn’t, consider first why readers search. If the title at hand is a reference or how-to book it's often to look up a specific ingredient or procedure. But those kinds of books aren't actually selling that well in eBookLand; publishers are turning those titles into apps. Instead, ebook fans are gobbling up narrative — fiction and non-fiction alike. Those titles all top the charts and so it's worth refining that earlier question: For narrative-style ebooks, why do readers search?

Often it’s for a quick lookup: a character we’re having trouble remembering, a plot twist we want a refresher on, a memorable quote. In each of these cases the searcher’s goal is quite different than what takes place during, for example, a traditional web search. There, you want to leave Google or Bing or wherever. In a book, by contrast, you just want a quick answer … and then an equally quick return to where you left off.

So with that in mind, let’s start off with what I would argue is a not particularly good search implementation. Here’s what a reader sees when searching on the Nook iPad app:

How’s a reader supposed to decide which of these results is the one she wants? Think about how cumbersome it is to tap each item, get whisked off to its location, and then have to navigate back to the starting spot. What a pain. In a print book, at least you’ve got fingers, bookmarks, or a coaster to hold your spot as you leaf around (reviewing index entries, for example). But in an ereader device, the disruption readers suffer after following a link is significant.

Especially for folks in search of a memory jiggle, the most important thing a search tool can do, then, is help quickly decide which result contains the answer they’re looking for. So this means designing a search tool that:

  1. Returns accurate results (duh)
  2. Presents these results with plenty of surrounding context
  3. Highlights the term clearly on the destination page
  4. Makes it easy to quickly return to the original reading position

If 1) and 2) are done well, there are plenty of cases where the reader never has to deal with 3) and 4). So a big part of making a search tool helpful happens in the results list itself. Make it big enough and you bring joy to the reader: they get the memory nudge they need and can get back to the passage where they stopped reading. Some of this stuff is fairly straightforward and can already be found in a few of the more polished e-reading systems.

Consider, for example, the iBooks app on the iPad.

You get a generous sampling of text surrounding the found term (which is itself clearly visible in boldface) making it easy to skim the list and find the result you’re looking for. You also get a tally of the number of times the term was found. For students and other researchers this can be a big help, as they often want a search tool to function like a print book’s index: to help perform a comprehensive review of a particular person or concept. And once the reader taps a result, the page he arrives at displays the term highlighted in yellow. Nice.

But there’s more, still, that can be done. More that a morphable digital screen can do … all in the service of trying to maintain a reader’s flow and minimize the page flipping required on devices that have proven themselves to be pretty poor page flippers.

Here, for example, is one possible solution I’ve sketched out, modeled on the “page preview” feature popularized by Bing, and now used by Google and plenty of other web search sites. (In case you haven’t seen it, it’s where you hover your mouse over any item on a typical search results page and up pops a small window showing the web page you’d see if you follow the search result link; it’s a great timesaver when you’re reviewing a list of results since it lets you do a quick visual skim before clicking any link.)

Proposed design for improved ebook search results
Click to enlarge

Rather than sending the reader to the search result, the idea here is to fetch the result for them. The results list appears just as it ordinarily does in, say, the Kindle app. But there’s that tiny “Show More” option; when tapped, it unfurls the result, increasing the viewable preview. In many cases that’s all a reader needs. For those who need more, the search term remains active and can be tapped to see the target page in full. By giving readers a quick way to review, temporarily, extra content like this, digital books can help preserve the immersive state that a great book induces.

Another way to improve the search results list: sortable search results. Here again progress made in the world of web search can improve how things work within a book. By letting the reader sort the results list in different ways a seemingly overwhelming heap can be reduced to something more manageable. iPad textbook publisher Inkling provides some handy design work in this area.

The tabs at top let the reader sort according to relevance (a software-driven guess at what's most important), content order (where the result occurs, ordered from start of book to finish), and type (audio, video, text, and so on). At the bottom of the search pane two other tabs let the reader switch between chapters they own and the entire book (Inkling sells chapters individually). This last twist is clever — half reader service and half marketing. Because the tool lists a meaningful snippet of the found term, someone searching for, say, bassoon, might see that there’s a video demo of that instrument in a chapter they haven’t yet purchased. Good stuff.

My main takeaway: by all means, let readers search. But give ’em tools that let them get back to what they’re reading as quickly as possible.

TOC Frankfurt 2011 — Being held on Tuesday, Oct. 11, 2011, TOC Frankfurt will feature a full day of cutting-edge keynotes and panel discussions by key figures in the worlds of publishing and technology.

Save 100€ off the regular admission price with code TOC2011OR

Related:

Search is the Web's fun and wicked problem

Peter MorvilleSearch is the Web’s most powerful and frustrating tool. It’s the conduit to unfathomable amounts of information, yet it requires a fair degree of user education to reach its full potential. It’s odd that something so important is so hard to harness.

And it’s not going to get easier anytime soon. We may think of search as static and mature because we’ve used those ubiquitous boxes for years. But it’s a tool in flux. Developments in mobile, augmented reality, and social graphs — to name a few — signal big changes ahead.

Peter Morville, co-author of “Search Patterns” and a long-time observer of the search domain, looks at the next wave of search in the following Q&A. He shows how "weird ideas" will shape search's future, and he also reveals the one recent innovation that unlocked a watershed moment for search (it's not what you'd expect).

Mac Slocum: You’ve called search the “worst usability problem on the Web.” Why is that? What makes it so bad?

Peter Morville: Search is a strange attractor that draws repeat visitors despite poor performance. In the 1990s, Jared Spool proved that when people were banned from using the search interfaces of major e-commerce sites, their success rates improved. But when given the chance, these same folks would choose to use search again and again. This hearkens back to research at IBM in the 1980s showing that users never read manuals. It’s called the “paradox of the active user” and we experience it every time we drive off for a new destination without consulting a map. Instead of taking time to understand the territory and chart an optimal course, we prefer the illusion of speed and simplicity. So we search.

And thanks to Google, Web search works pretty well for basic lookup. But, cross over to categories like e-commerce, enterprise, desktop, mobile, social, and realtime search, and performance falls off a cliff. That’s because search is a really hard problem that relies on language as a bridge. A few keywords can’t provide sufficient insight into the searcher’s intent. They just can’t. So, search isn’t as simple as the box. Search is a complex, adaptive system and an iterative, interactive experience. For designers, it’s a wicked problem, and that’s why it’s so much fun.

MS: Where will future search innovations come from?

PM: In “Search Patterns,” I write about averted vision or “the art of seeing distant objects by looking to their periphery.” This astronomical analogy serves as a powerful reminder that forecasting isn’t simply a matter of extrapolation.

The future of search isn’t Google + 1. To get real-time search, we had to invent Twitter. And to achieve real-world search, an inevitable extension of augmented reality, we’ll need to leapfrog from iPhones to iGlasses, and infuse our surroundings with sensors and spime. So, to anticipate changes in what we search and how we search, we should keep an eye out for emerging technologies and weird ideas that aren’t labeled or categorized as search. In other words, we must look away to see.

MS: Can you point to any examples that satisfy your conclusion that search works best as a conversation?

PM: Here’s a typical scenario. I go to Amazon and ask for a camera.

“What sort of camera would you like?” replies Amazon. “Film, digital, or video?”

I reply that I’m looking for a digital camera.

“Did you have a particular brand in mind? How many megapixels? What can you afford?”

Of course, we don’t converse in natural language. Instead, Amazon suggests related searches and displays a dynamic, personalized map to my search results in the form of faceted navigation. But, the experience is similar to a conversation. It’s iterative and interactive and offers rich opportunities for learning. At its best, search doesn’t simply serve up answers. It helps us formulate the right questions.

MS: On the mobile search side, have you seen any device that takes full advantage of sensors, accelerometers, GPS, speech recognition, etc.?

PM: Mobile search is in its infancy, and it’s already so damn interesting. The opportunity is huge and qualitatively different than anything we’ve seen before. Augmented reality applications on today’s iPhone and Android devices offer the barest glimpse of where we’re going. It’s pretty cool that I can stand on a street corner in Savannah and search for restaurants or hunt for ghosts through the looking glass of my iPhone’s camera. But we haven’t even scratched the surface of what’s waiting to be found at the crossroads of sensors, search, and the Internet of Things.

MS: What do you make of the argument that social search — Facebook, Twitter, “recommendations,” etc. — poses a threat to Google?

Peter Morville: I don’t view Facebook and Twitter as terrible threats to Google, although they’re clearly forcing changes such as the inclusion of social filters and realtime results within the Web search experience.

I think “social search” is a fascinating topic. As research at Microsoft has shown, we often search together. And, social network platforms are making it easier to invoke search as a shout. We can ask our followers on Twitter instead of querying via Google. If you’re interested in this topic, I recommend this presentation by Brynn Evans and Will Evans about designing for sociality in enterprise search. There’s lots to learn. That said, traditional search remains a tremendously efficient way to find, learn, and act. And, it will keep getting bigger and better. I don’t recommend selling short on Google.

MS: Is there a disconnect between search designers and search engineers?

PM: Typically, designers are close to the users. We understand needs and behavior. Engineers are close to technology. They understand what’s possible. We can learn a lot from each other. In fact, I’d argue that search is an unusually interdisciplinary challenge that requires collaboration across the established silos of design, engineering, and marketing. It’s precisely because of these silos that search is often so bad. We need to work together. It’s not easy. It’s even wicked hard. But there’s simply no other way to make search better.

MS: What can users do to improve search?

PM: The real responsibility rests with parents and teachers. I’m convinced that information literacy is among the most important subjects we can teach our kids. They must learn where to search and how to evaluate what they find. Wikipedia, for instance, can catalyze great conversations with our children about authority and trust. In a world where we can increasingly select our sources and choose our news, these analytical skills are mission critical. And, better searchers will make search better. Their behavior will offer valuable feedback that will in turn improve the interfaces and algorithms of tomorrow’s tools for search and discovery.

MS: What’s the difference between search patterns and search engine optimization? Is it about making search work for users as opposed to direct revenue?

PM: SEO is a valuable but narrow lens through which to view search. While it’s important to optimize for Google, Web search isn’t the only way we find. Users also rely on a site’s information architecture and its unique search and navigation systems to find what they need.

Plus, search isn’t only about findability. We created a searcher’s edition of the user experience honeycomb to argue that search must also be useful, usable, desirable, accessible, credible, and valuable. It’s that last point that speaks to revenue. To create value, we must strike the right balance between the needs of users and business, so that our design advances the mission and improves both top and bottom lines.

MS: How does “Search Patterns” connect to your earlier work on “Ambient Findability” and “Information Architecture“?

Peter Morville: The polar bear book explains information architecture from top to bottom. It’s a very practical text. In contrast, the lemur book is an undisciplined exploration of ambient findability, a future at the crossroads of ubiquitous computing and the Internet, in which we can find anyone or anything from anywhere at any time. It’s conceptual and provocative. I like to think of the butterfly book as the love child of the polar bear and the lemur. It offers practical advice for designers while simultaneously provoking readers to think differently about search.

Of course, every child must be unique, and “Search Patterns” is no exception. Jeff [Callender] and I collaborated intensively to create a new kind of book that brings search and discovery to life with colorful and surprising illustrations. And, we’ve made them all available online at searchpatterns.org, so that you can use them in your efforts to make search better.

MS: What’s been the best search/UI development of the last 3-5 years? What really opened things up?

PM: Autocomplete is an old pattern from the desktop that’s found new life in Web and mobile search. Once relegated to the musty modules of “help” in desktop software, autocomplete is now part of our everyday experience. It’s a great answer to the question: why wait for results? It saves time and typos by serving up suggested searches or destinations while we’re still entering our query. And, it’s a simple design pattern with powerful potential. For instance, Yahoo complements basic autocomplete with an offer to explore related concepts. By analyzing query-query reformulation data and post-query browse behavior, Yahoo is able to suggest similar queries that don’t even contain the original keywords. We can find what we didn’t know to seek.

So, what really opened things up? I’d argue it was Google Maps. All of a sudden, designers realized what could be done with programming frameworks like Ajax, and autocomplete was simply one of many interaction design patterns that flourished on the Web soon after that watershed moment.

Note: This interview was condensed and edited.

Digital Campus » Episode 48 – Balkanization of the Web?

What will be the impact of the loss of non-Anglophone books in the revised Google Books settlement? How about the loss of News Corporation content in Google’s search? Or the loss of physical books from the library? And what exactly does the loss of tens of thousands of editors mean to Wikipedia? Mills, Amanda, and Dan discuss these changes to our information environment in a special Thanksgiving edition of the podcast.

[#SOLR-1163] Solr Explorer – A generic GWT client for Solr – ASF JIRA

# Simple query search
# Sorting – one can dynamically define new sort criterias
# Search results are rendered very much like Google search results are rendered. It is also possible to view all stored field values for every hit.
# Custom hit rendering – It is possible to show thumbnails (images) per hit and also customize a view for a hit based on html templates
# Faceting – one can dynamically define field and query facets via the UI. it is also possible to pre-configure these facets in the configuration file.
# Highlighting – you can dynamically configure highlighting. it can also be pre-configured in the configuration file
# Spellchecking – you can dynamically configure spell checking. Can also be done in the configuration file. Supports collation. It is also possible to send "build" and "reload" commands.
# Data import handler – if used, it is possible to send a "full-import" and "status" command ("delta-import" is not implemented yet, but it's easy to add)
# Console – For developme

Scaling Lucene and Solr | Enterprise Search support for Apache Lucene and Solr by Lucid Imagination

While many Lucene/Solr applications will never outgrow a single, well-configured machine, the fact is, more and more applications are pushing beyond the single machine limit due to either index size or query volume. In discussing Lucene and Solr best practices for performance and scaling, Mark Miller explains how to get the most out of a single machine, as well as how to scale out to harness multiple machines to handle large indexes, large query volume, or both.

Why Faceted Navigation is Hard

I was thinking about the relational model today, and I realized that a lot of people probably don't realize that faceted navigation is an example of an application that simply doesn't fit comfortably in the relational model. It also occurs to me that most of you don't care. But I'm going to tell you why anyway.

Faceted navigation is built on faceted classification, which is conceptually very simple. You've got two things: records and categories, and those two things have a many-to-many relationship.

I like to work with concrete examples, so let's take the US states as our dataset. We've got one record for each state. For our categories, let's use adjacent states. That means Massachusetts has the following categories: adjoins-NH, adjoins-VT, adjoins-CT, adjoins-RI, and adjoins-NY. Obviously there are a lot of states that belong to the adjoins-NY category: Pennsylvania, New Jersey, Connecticut, Massachusetts, and Vermont. So clearly this is many-to-many. That's faceted classification.

RDF Aggregates and Full Text Search on Steroids with Solr at Frederick Giasson’s Weblog

The idea is to use the RDF data model and a triples store to populate the Solr schema index. We leverage the powerful and flexible data representation framework (RDF), in conjunction with the piece of software that lets you do whatever you want with that data (Virtuoso), to feed a carefully tailored Solr schema index to optimally perform three things: full-text search, aggregates and filtering. Also, we want to leverage the ontologies used to describe this data to be able to infer things vis-à-vis these indexed resources in Solr. This leverage enables us to use inference on full-text search, aggregates and filtering, in Solr! This is quite important since you will be able to perform full text searches, filtered by types that are inferred!

Read It: Search User Interfaces

Search is an integral part of peoples' online lives; people turn to search engines for help with a wide range of needs and desires, from satisfying idle curiousity to finding life-saving health remedies, from learning about medieval art history to finding video game solutions and pop music lyrics. Web search engines are now the second most frequently used online computer application, after email. Not long ago, most software applications did not contain a search module. Today, search is fully integrated into operating systems and is viewed as an essential part of most information systems.

Zend Framework: Documentation: Chapter 45. Zend_Search_Lucene

Zend_Search_Lucene is a general purpose text search engine written entirely in PHP 5. Since it stores its index on the filesystem and does not require a database server, it can add search capabilities to almost any PHP-driven website. Zend_Search_Lucene supports the following features:

*

Ranked searching – best results returned first
*

Many powerful query types: phrase queries, boolean queries, wildcard queries, proximity queries, range queries and many others.
*

Search by specific field (e.g., title, author, contents)

Zend_Search_Lucene was derived from the Apache Lucene project. The currently (starting from ZF 1.6) supported Lucene index format versions are 1.4 – 2.3. For more information on Lucene, visit http://lucene.apache.org/java/docs/.