Skip to main content
NCBI home page
Bioinformatics logoLink to Bioinformatics
. 2011 Apr 14;27(12):1734–1735. doi: 10.1093/bioinformatics/btr181

‘Sciencenet’—towards a global search and share engine for all scientific knowledge

Dominic S Lütjohann 1,†, Asmi H Shah 2,†, Michael P Christen 2, Florian Richter 2, Karsten Knese 2, Urban Liebel 2,*
PMCID: PMC3106183  PMID: 21493657

Abstract

Summary: Modern biological experiments create vast amounts of data which are geographically distributed. These datasets consist of petabytes of raw data and billions of documents. Yet to the best of our knowledge, a search engine technology that searches and cross-links all different data types in life sciences does not exist.

We have developed a prototype distributed scientific search engine technology, ‘Sciencenet’, which facilitates rapid searching over this large data space. By ‘bringing the search engine to the data’, we do not require server farms. This platform also allows users to contribute to the search index and publish their large-scale data to support e-Science. Furthermore, a community-driven method guarantees that only scientific content is crawled and presented. Our peer-to-peer approach is sufficiently scalable for the science web without performance or capacity tradeoff.

Availability and Implementation: The free to use search portal web page and the downloadable client are accessible at: http://sciencenet.kit.edu. The web portal for index administration is implemented in ASP.NET, the ‘AskMe’ experiment publisher is written in Python 2.7, and the backend ‘YaCy’ search engine is based on Java 1.6.

Contact: urban.liebel@kit.edu

Supplementary Material: Detailed instructions and descriptions can be found on the project homepage: http://sciencenet.kit.edu.

1 MOTIVATION

Most commonly known search engine technologies (Bing, Google) are based on popularity ranking algorithms. However, scientific research has special requirements for search engines that cannot be addressed by popularity ranking in all cases. Special search engines (for example, Scirus (McKiernan, 2005), PubMed, Google Scholar, Web of Science, Scopus) concentrate more on providing content from scientific journals and literature (Falagas et al., 2008).

Other meta search engines cross-link several centralized databases via a single search interface [for example Bioinformatic Harvester (Liebel et al., 2004), EB-eye (Valentin et al., 2010), Entrez (Schuler et al., 1996), Ensembl (Flicek et al., 2010), STRING (Szklarczyk et al., 2010)]. Today's scientific search queries require searching across different data sources that are geographically distributed. Often different data types, like high content screening (HCS) image data or sequence based data (Birney et al., 2007), require special databases that present a challenge to the global search methods mentioned above.

The latest developments in high content/high-throughput screening microscopy (Pepperkok and Ellenberg, 2006) and next-generation sequencing technologies (Metzker, 2010) routinely produce experimental datasets in the terabyte (TB) range resulting in millions of data files. To the best of our knowledge, there is no central database to encompass all experiment datasets due to the fact that large-scale data handling is a challenge for any known data publication platform. Uploading all this data to a centralized database is currently too time consuming and expensive (Schadt et al., 2010). Also, maintaining a centralized infrastructure over the years is costly (Ball et al., 2004). Consequently, it is likely that no single library alone will be able to index the entire science web (Lewandowski and Mayr, 2006). Research strongly benefits from accessible data that provides a valuable resource for comparative and novel studies (Campbell, 2009). Thus, a decentralized search and publishing network that can handle multiple data types at different locations will significantly improve the scientific research process.

2 DESIGN AND IMPLEMENTATION

We designed Sciencenet, a distributed peer-to-peer search engine network that can incorporate many different scientific data types like text, large-scale image datasets (Swedlow and Eliceiri, 2009), DNA/protein sequences (Ansorge, 2009) and mass spectrometry (MS) data (Gstaiger and Aebersold, 2009), which are published on web servers. It facilitates linking search results to other related heterogeneous data sources.

To ensure the scalability of the data space, documents are located via a Distributed Hash Table (DHT) (Balakrishnan et al., 2003). This avoids asking every peer to receive a complete search result. Our DHT rule allows storing index elements for a single search request on several peers. Due to concurrent queries, the more peers contribute, the better the response time gets.

The distributed Sciencenet software platform has the following key elements:

(1) A large-scale index technology capable of handling billions of documents belonging to the scientific web. Based on KIT's 350 000 web pages and currently 6471 known scientific sites in the whitelist, we estimate a total number of over 2 billion documents to be integrated.

Our startup environment for Sciencenet consists of 30 commodity PCs, equipped with 2–24 cores, 4-64 GB of RAM and 500 GB hard disks, each capable of handling 15 million documents, which would just require a total number of about 200 peers for the estimated data space. The operating system is standard Ubuntu 10.04 with Java. This architecture was chosen to mimic a global distributed search engine.

These Sciencenet PCs (peers) are configured to crawl (load and analyze) distinct scientific web sites and import repositories that provide an Open Archive Interface (OAI) (Lagoze et al., 2002). OAI is a standard to import data sources in a fast and structured manner. Currently, 240 million web pages and documents are in the index of our machines. Furthermore, 1 TB of image-based data are available.

The scientific community can easily provide server capacity to expand the index and improve search performance.

(2) A community-driven method to manage the integration of institutional web sites, databases and journals to improve the quality of the scientific search index. Any scientific web site can be submitted by anyone, and registered users can be part of the process to accept these suggestions to support the growth of the index.

(3) A simple ‘one-stop’ search interface for all users. The Sciencenet web site (http://sciencenet.kit.edu) provides a search portal without installation. The search results are presented along with a domain navigator and a tag cloud to refine the search.

Alternatively, users can download the free open-source Sciencenet-YaCy client software package, allowing them to access the search network from their machines, perform search queries and access published scientific experiment data from others. The result list can be exported via an Application Programming Interface for further processing in external tools.

Due to the preselected index, we consider every search result to be relevant, so pre-computed ranking, like PageRank (Brin and Page, 1998), is not used. The results are ranked using a default ‘ranking matrix’ consisting of a set of 28 statistical ranking criteria, such as ‘word distance’ or ‘appearance in title’ (see Supplementary Material). For each search query, users can customize the values of the ranking matrix with no increase in the overall complexity.

(4) An easy to use software tool that allows data publishing and sharing. Users are able to publish and share their own scientific data or web sites. We provide an example module (the ‘AskMe’ tool) for non-text based data integration in the downloadable client. The tool handles large-scale image datasets from HCS experiments by providing a dataset preview. All collected meta information is presented in corresponding experiment descriptor files in both human and computer readable form. Hence, we use the embedded Resource Description Framework RDFa (Birbeck and Adida, 2008). This data publication method follows the principle of a Linked Open Data architecture (Berners-Lee, 2006) and is already the foundation for a semantically enriched web (Jensen and Bork, 2010).

3 CONCLUSION AND OUTLOOK

The combination of the technologies mentioned above makes it possible to search thousands of heterogeneous data sources with billions of documents and datasets. Our decentralized peer-to-peer approach overcomes the performance and capacity limitations of centralized data repositories. Ideally, future modules would allow users to rank and comment on search results.

ACKNOWLEDGEMENT

The authors thank YaCy's open-source community, Björn Kindler for technical support, Stefan Bräse (IOC) for general support, Gary Davidson, Caitlin Howell and Markus Niermeyer for comments on the manuscript.

Funding: DOPAMINET Molecular Networks of Dopaminergic Neurons in Chordates. BOLD (Biology of Liver and Pancreatic Development and Disease) - Marie Curie Initial Training Network (238821 to A.H.S.); Dopaminet FP7 (Seventh Framework Program) (223744 to U.L.).

Conflict of Interest: none declared.

REFERENCES

  1. Ansorge WJ. Next-generation DNA sequencing techniques. Nat. Biotechnol. 2009;25:195–203. doi: 10.1016/j.nbt.2008.12.009. [DOI] [PubMed] [Google Scholar]
  2. Balakrishnan I, et al. Looking up data in P2P systems. Commun. ACM. 2003;46:43–48. [Google Scholar]
  3. Ball CA, et al. Funding high-throughput data sharing. Nat. Biotechnol. 2004;22:1179–1183. doi: 10.1038/nbt0904-1179. [DOI] [PubMed] [Google Scholar]
  4. Berners-Lee T. Linked Data. W3C Design Issues. International Journal on Semantic Web and Information Systems. 2006;4 W3C, 1. [Google Scholar]
  5. Birbeck M, Adida B. RDFa Primer. W3C Notes. 2008. W3C Working Group Note 14 October 2008. [Google Scholar]
  6. Birney E, et al. Identification and analysis of functional elements in 1% of the human genome by the ENCODE pilot project. Nature. 2007;447:799–816. doi: 10.1038/nature05874. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Brin S, Page L. The anatomy of a large-scale hypertextual Web search engine. Comput. Networks ISDN. 1998;30:107–117. [Google Scholar]
  8. Campbell P. Data's shameful neglect. Nature. 2009;461:145. doi: 10.1038/461145a. [DOI] [PubMed] [Google Scholar]
  9. Falagas ME, et al. Comparison of PubMed, Scopus, Web of Science, and Google Scholar: strengths and weaknesses. FASEB J. 2008;22:338–342. doi: 10.1096/fj.07-9492LSF. [DOI] [PubMed] [Google Scholar]
  10. Flicek P, et al. Ensembl 2011. Nucleic Acids Res. 2011;39(Database issue):D800–D806. doi: 10.1093/nar/gkq1064. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Gstaiger M, Aebersold R. Applying mass spectrometry-based proteomics to genetics, genomics and network biology. Nat. Rev. Genet. 2009;10:617–627. doi: 10.1038/nrg2633. [DOI] [PubMed] [Google Scholar]
  12. Jensen LJ, Bork P. Ontologies in quantitative biology: a basis for comparison, integration, and discovery. PLoS Biol. 2010;8:e1000374. doi: 10.1371/journal.pbio.1000374. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Lagoze C, et al. The Open Archives Initiative Protocol for Metadata Harvesting, Version 2.0. The making of the Open Archives Initiative Protocol for Metadata Harvesting. 2002;21:118–128. Library Hi Tech, Vol. [Google Scholar]
  14. Lewandowski D, Mayr P. Exploring the academic invisible web. Libr. Hi Tech. 2006;24:529–539. [Google Scholar]
  15. Liebel U, et al. ‘Harvester’: a fast meta search engine of human protein resources. Bioinformatics. 2004;20:1962–1963. doi: 10.1093/bioinformatics/bth146. [DOI] [PubMed] [Google Scholar]
  16. McKiernan G. E-profile: Scirus: For Scientific Information Only. Libr. Hi Tech News. 2005;22:18–25. [Google Scholar]
  17. Metzker ML. Sequencing technologies - the next generation. Nat. Rev. Genet. 2010;11:31–46. doi: 10.1038/nrg2626. [DOI] [PubMed] [Google Scholar]
  18. Pepperkok R, Ellenberg J. High-throughput fluorescence microscopy for systems biology. Nat. Rev. Mol. Cell. Biol. 2006;7:690–696. doi: 10.1038/nrm1979. [DOI] [PubMed] [Google Scholar]
  19. Schadt EE, et al. Computational solutions to large-scale data management and analysis. Nat. Rev. Genet. 2010;11:647–657. doi: 10.1038/nrg2857. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Schuler GD, et al. Entrez: molecular biology database and retrieval system. Methods Enzymol. 1996;266:141–162. doi: 10.1016/s0076-6879(96)66012-1. [DOI] [PubMed] [Google Scholar]
  21. Swedlow JR, Eliceiri KW. Open source bioimage informatics for cell biology. Trends Cell. Biol. 2009;19:656–660. doi: 10.1016/j.tcb.2009.08.007. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Szklarczyk D, et al. The STRING database in 2011: functional interaction networks of proteins, globally integrated and scored. Nucleic Acids Res. 2010;39(Database issue):D561–D568. doi: 10.1093/nar/gkq973. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Valentin F, et al. Fast and efficient searching of biological data resources–using EB-eye. Brief. Bioinform. 2010;11:375–384. doi: 10.1093/bib/bbp065. [DOI] [PMC free article] [PubMed] [Google Scholar]

Articles from Bioinformatics are provided here courtesy of Oxford University Press

RESOURCES