Sphinx for best commercial open source project

powered by Sphinx

Sphinx search (www.sphinxsearch.com) is one of the fastest developing search engine. Sphinx is used by Craigslist amongst several popular sites. If you have not already tried Sphinx and you are interested in a building your own search engine, give it a shot. Some of the salient features of sphinx is around easier integration with MySQL, fast indexing and support for distributed searches.

Support Sphinx and nominate it for the community choice awards!

Which companies are impacted by swine flu?

The swine flu epidemic (leading to a pandemic) has terrified the world. It’s impact can be felt across industries – with travel and hospitality being the worst hit. I tried to see which companies report a hit in performance because of swine flu. Using Gridstone’s Search Engine, I searched for swine flu. Here is a sneak-peak of the results:

Search results for swine flu using Gridstone Search

Continue reading

Gridstone Search – a SEC filings and transcripts search engine

Gridstone Search for SEC documents and Transcripts

Gridstone Search for SEC documents and Transcripts

Gridstone Search (http://search.gridstoneresearch.com) is avilable as beta for all the information seekers who have to sieve through SEC documents and company transcripts. Gridstone Search is now loaded with EDGAR filed documents (includes the five most important filing types – 10K, 6K, 10Q, 8K and 14A) starting 2007 and Transcripts provided by Seeking Alpha (through a unique partnership). Continue reading

Welcome the (Yahoo!) BOSS

Yahoo! Inc recently announced the release of BOSS – its open search (web-service based) platform. BOSS (Build your Own Search Service) exposes Yahoo Search through an App ID and a web-service. It will be interesting to see how BOSS matures around two main search engine buzz-words: vertical search and semantic search capabilities. I spent some time trying out BOSS on some of my favorite problems (Yes, I am going to share the URL on my website soon!). Here is a quick catch-up on the ‘cool’ and ‘not-so-cool’ BOSS features.

  1. Vertical Search Engines: It is not easy to construct a vertical search engine and use BOSS for searching on a few thousand sites. BOSS doesn’t support search for multiple sites (in fact it does it wrongly if you try to search for multiple sites through the web-service), nor does it allow regular expressions for sites to search. This doesn’t add up well for building up your own little vertical search engine. Google’s Custom Search Engine scores way higher on this one. With easy to configure websites and ability to tweak with the relevance and configurations Google CSE definitely scores higher. But BOSS has just arrived; I am sure changes will pour in soon.
  2. Semantic Search: Hakia has done it; many more will follow! This is a good value proposition by Yahoo and is primarily aimed at spicing-up the horizontal search market. It provides developers and researchers to try out their own versions of relevance and mash-ups on top of Yahoo’s rich information source. This however operates on a horizontal search model.
  3. Multi-modal Search: Yahoo has done a good job in providing Image and News Search features. Image searches however have relatively fewer use-cases, especially if the search is not based on Image content. A Riya like visual-search would have been awesome.

Currently there are no query limits on BOSS, but with rapidly increasing usage and adoption this is bound to change. The monetization abilities in both Yahoo and Google open search platforms is fairly limited and concentrates around Ads-World. For small enterprises and start-ups the ability to use these applications is therefore cumbersome, especially if the value proposition is being built around a vertical search model.

On a related note, Yahoo’s partnership with Hakia comes at an interesting time with Microsoft completing the Powerset acquisition. The semantic search war is catching up some steam!

Yet another "Microsoft acquires Powerset" blog

PowerSet and MSFT finally announced this rumored acquisition. MSFT has acquired Powerset for close to $100 million. Powerset is a semantic search engine that extends the XEROX-PARC’s licensed linguistic technology. I have tried Powerset when it was a beta and when it got (almost abruptly) released (with search limited to wikipedia and freebase).

The use of semantic technologies for horizontol search engines look highly improbable. The nature of the web is too diverse to be nailed by a single hammer. A related article on making search relevance more meaningful is in this earlier post.

MSFT plans to integrate the Powerset team with its search relevance team and explore advanced search capabilities while taking the Powerset technology beyond wikipedia. MSFT has been creating noise around enterprise and local search, both of which are vertical in nature. This acquisition can add value in making vertical search offerings from live.com smarter and probably scaring the Mountain View behemoth. Whether MSFT takes Powerset’s saplings and nurtures it into its Redmond forests – this only time will tell.

On a related note, I find results from Hakia more useful. Try a sample search for “is EPS the right measure for stock performance” on both powerset and hakia and you will see the difference. Hakia gives results much more relevant to EPS – at least it displays on the very first page results about the use of EPS measure for measuring stock performance. Powerset returned results that had nothing to do with “EPS ~ Earnings per share”. The results from Powerset were purely keyword matches – it mostly matched Extended Play (EP) vinyl records.

In general, most semantic searches seem to be working better on Hakia than on Powerset. I am surprised why Hakia was not approached by suitors if the motivation has been to ramp up semantic abilities for search engines. Or may be it was!

Personalized Search and Disambiguation – The answers to search engine relevance

There is a flurry of announcements around horizontal and vertical search engines offering ‘better’ and ‘more relevant’ content retrieval. SEOmoz.org has an interesting article on the ability (or inability) of search engines to return relevant results. In this entire discussion of retrieving more relevant there are two seemingly related problems – ‘personalized search’ and ‘disambiguation (of search terms)’.

Personalized search is the ability to convolute the search engine results with the user’s profile in order to return the most relevant search results. The user profile can be created by using the following information:

  • User browsing (search, click-through, time-visited on site, etc) history
  • Desktop data (files, content, most-accessed files)
  • User feedback (asking users to rank order the results based on their preferences)

Continue reading