linux host : web mine/grab query

DannyD · 19-11-2007 01:36PM #1

Hi,
I'm hosting a website on a linux box. I am looking for a php script that will check a webpage periodically and grab a page and make it available on my site. I game across a product called mine the web but this costs $150. I'm hoping for a free script. Any ideas? Thanks in advance

Trojan · 20-11-2007 04:15PM

Write a cron entry that runs at whatever your time period is. Make the command "wget http://www.xyz.com/the-page".

daymobrew · 20-11-2007 04:18PM

You can use wget to download a file from another site. A cronjob could run this.
In PHP you can use fopen to download remote files.

Getting the page to your site is a little more complicated because the permissions of the web server process will not have write permission to your web space (unless you make it world writeable - not a good idea).

If you run a cronjob as your web space user then this is not an issue.

What exactly are you going to do with the web page that you download?
What about copyright issues?

gortronovich · 26-11-2007 11:12PM

You can use the PHP function file_get_contents() to get the source of a remote URL:

http://de2.php.net/file_get_contents

Then use cron to schedule this as often as you like:

http://www.adminschoice.com/docs/crontab.htm

linux host : web mine/grab query

Comments