scraping a website for info

OU812 · 29-05-2012 08:54AM #1

Apologies if this has been asked before.

I'm in the process of having a website redeveloped & would like to be able to scrape some other sites for info (product pricing) to display on my own site.

In the manner of "how long is a piece of string", would this be a difficult thing to do ? Would it be expensive to have developed into the site ?

Your advice/assistance is appreciated.

jester77 · 29-05-2012 10:37AM

Scraping could leave you with some legal issues unless you have permission from the sites owner. If you have permissions then ask them to expose the data through an API. Not only would it be quicker to access the data, it would also be easier to develop and you don't have to worry about UI changes on the site.

I've not used any scraping tools, I have used HTMLUnit for testing but you could easily use that for scraping. It's very straightforward to use and you can access attributes by IDs, CSS or XPATH.

Freddio · 29-05-2012 10:39AM

It would all depend on how good the html you are scraping is

The total is &#8364; <span id="total">99.99</span>

is a format that would have more consistant results than:

The total is &#8364; 99.99

If the sites you are scraping change you will have to rewrite your parsing code.

Having said that, it doesn't take very long to write scraping code (in php anyway)

OU812 · 29-05-2012 10:54AM

Thank you both for your replies. I'm not in the development industry & would preferable like to speak to someone face to face / over the phone before diving in. Is there an area on here where I can contact someone to explain exactly what is needed prior to getting a quote for work ?

Freddio · 29-05-2012 11:00AM

pm sent

scraping a website for info

Comments