Got an Idea but I'm stuck there!

OfflerCrocGod

Wondering how to implement this....... I need to write a program that can hit a search engine with a query request i.e.

www.search.com/formated%20query

where the "formated query" part would be the search term I'm looking for, once the search engine returns the search query answers I want to be able to look through ALL the returned web sites and pass by all the web sites I'm not interested in to a web site that could be very deep in the returned list. Once I find this site I then need to hit/open the site in such a way that the search engine which is tracking me KNOWS I went for that site.

So in google I type in "red ball" and the URL would be
in my above example
www.google.com/"red ball"
or in real life
www.google.com/search?q=red+ball

and then I look through all the results and go very deep into them, result 60,000 lets say, I look for its website(lets call it www.x.com) and hit that link that google returns at (for this example) position 60,000, while still letting google know I used there link to go to that website. I want to be able to leave this loop for as long as possible ( I have a loop condition in my head ) and for it to always go for the same link, the index(60,000) could be variable so I afto search the actuall URLs.

Now I'm not asking for anyone here to actually do this I'm just wondering what would YOU use to do something like this (surely its possible??), I have one guess...PHP? am I miles away??. My small amount of coding knowledge is mainly in Java so application layer stuff where I dont mess around with Internet stuff to much I would be doing it over x-mas. So if PHP is the language I need to learn I would get me a few books to look at for x-mas. So does anyone here know if there is something else which would be better at this?. Can what I'm saying be done?, surely it can cause Meta-search engines do something kinda like this.

Anyway any help or hints will be appreciated, ta!.

Zaltais

Ok, not sure if it's the way you've phrased your question or because I'm half twisted (I'm guessing the latter), but I've not quite understood what your trying to achieve.... I understand the first part of your methodology... but just not quite what you're trying to do with it, so apologies for the probable uslessness of this answer...

Given you're a self confessed java head, my first point of call would be google's own api's :- http://www.google.ie/apis/

And I'd follow that up with a search on CPAN (given I'm a Perl head)
http://search.cpan.org/search?query=google&mode=all

Given that you're looking specifically for info on PHP & google my next port of call would be the PHP resource index (which has a 'Remote Searches' section)
http://php.resourceindex.com/Complete_Scripts/Searching/Remote_Searches/

Google Search may be of particular interest....
http://www.jappc.com/google.php

Basicially I'm always of the opinion that the best language for the job is the language you know best... so personally I'd be inclined to stick with Perl (or Java in your case) unless there is some reason why you can't use Java (or have to use PHP)

OfflerCrocGod

Well no I'm not going to be using this on google, that was to illustrate my point thats all:p, this is an experiment and it may achieve what I want and it may not I cant tell until I run my code . "Java head" would be the wrong term for me I have some experience in Java thats about it. Thats why I would be reluctant to use it cause its not fast and this could take a while and also I just dont see how Java could do this for search engine X (lets call it), by the way thanks for that Java google API thing! I'll give it a look at x-mas. Thats why I though this would be easier to imlement with a language that lives and breathes Internet, PHP (which I know nothing of) sounded like it could be my ticket. I just wanted to sound this off everyone else here before going and learning it.

uberwolf

google is the only SE with an API, AFAIK. The rest you'll have to write a wrapper to communicate with. Which means studying how each SE responds to a query and then artificially generating that URL, connecting to it, parsing the page which is returned and then either link to the next pae to get more results or move on to the nxt query.

This is fairly ok to in Java. As for PHP - I don't know

OfflerCrocGod

Ummm OK then I'll check it out in Java aswell I'm not planning to do it now cause I just dont have the time but during x-mas when I'm bored out of my skull it would be an interesting problem to work on thank uberwolf for your help the way you explained it was the only way I could see it being done I was just hoping that there was an easier way:D.

Zaltais

If you want to try Perl instead of PHP these wrappers are already available for loads of search engines, including Altavista, Yahoo, AlltheWeb, Lycos, etc. as well as other sites like Ebay, RpmFind, all you need to do is use them....

They are all derivitives of WWW::Search, a free Perl module available on CPAN.

seamus

I have an implementation of a engine queryer in Java, that myself and uberwolf used for a project last year.

You can fire a few PMs at me if you need some help when going to do this

OfflerCrocGod

I've heard great things about perl so maybe I'll look into it aswell.....so this x-mas I will be learning PHP, Perl and improving my Java while at the same time learning C++ ( unrelated to this) and also celebrating x-mas......Ummm I dont know wheter I can do ALL those things at once:D. I'll just skip the eating and sleeping stuff, seamus if I decide to do this in Java and have great difficulties I may take you up on your offer, ta!.