Advertisement
If you have a new account but are having problems posting or verifying your account, please email us on hello@boards.ie for help. Thanks :)
Hello all! Please ensure that you are posting a new thread or question in the appropriate forum. The Feedback forum is overwhelmed with questions that are having to be moved elsewhere. If you need help to verify your account contact hello@boards.ie
Hi there,
There is an issue with role permissions that is being worked on at the moment.
If you are having trouble with access or permissions on regional forums please post here to get access: https://www.boards.ie/discussion/2058365403/you-do-not-have-permission-for-that#latest

Distributed search engine, is it possible?

  • 05-05-2007 11:31pm
    #1
    nobodythere
    Closed Accounts Posts: 2,349 ✭✭✭


    Do you think that it's possible and feasible for an internet search engine to be distributed in much the same way that SETI is?

    We already have implentations for decentralised searching in protocols like Gnutella.

    the more I think about it, the more possible it seems. Security problems could be fixed by having two hosts verify each URL in the index. It wouldn't take a particularly large number of computers to index it (in the hundreds probably), it's the actual searching of a decentralised database that's the challenge.

    If the DB was 8 billion pages and each record took up 500 bytes that would be a 4TB database. If there were enough people and organisations willing to act as 'supernodes' the security issue of trust would be irrelevant and if you split it up into 10 sections we'd be looking at 400GB per node.

    The more evil google gets the more I think that this is the next step in the evolution of the internet. A sort of internet socialism so to speak.


Comments

  • #2
    flowgar
    Closed Accounts Posts: 1 ✭flowgar


    Have a look at FAROO ( http://www.faroo.com ), a peer-to-peer web search engine. The users are connecting their computers, building a worldwide, distributed p2p web search engine. No centralized index and crawler are required anymore. Every web page visited is automatically included in the distributed index of the search engine.


Advertisement