How would you go about developing a savings aggregator?

Alligator Wine

I'm a bit stumped here about where to start looking. There's a strong possibility I need to build an aggregator similar to what bonkers.ie have for comparing savings accounts: http://www.bonkers.ie/compare-savings-accounts/

I'll more than likely hire a developer for this, but would like to know for initial conversation with the client what would be involved. Are there any companies that offer a plugin like this or would it have to be developed from scratch? Any help greatly appreciated.

Graham

I think you're going to find this is going to be a fairly manual process.

I doubt any of the institutions present the information on their savings product in a standard format (or make the information available as a data feed). If that's the case you are going to be left with 2 choices.

1) Manually enter the details for each financial institution and update the information manually every day or two.
2) Develop a scraper to go to the website of each particular institution and scrape the data you require.

The problem with option 1 is obviously that it's a manual process. The problem with option 2 is each time a website changes its layout, your scraper breaks.

With either solution you're also going to have to keep an eye out for new providers and put in place a process for including them.

Apart from bonkers, consumerhelp.ie also has a comparison service for savings accounts but I doubt either provider would be too happy if you just lifted your data from them directly.

Make sure you and/or your client allocates sufficient resources to update/maintain whatever solution he decides to go with. This is not going to be a write it and leave it running project. Unless there is significant commercial advantage to having this functionality, it's not something I'd rush to recommend. A broken comparison service will not reflect well on the organisation providing it.

Talisman

It would be developed from scratch.

The price comparison sites are brokers for the services they list. How they make money:
1 - Advertising on the site
2 - Paid/Sponsored search results
3 - Click through/commission on sales of service

If you are setting up such a service you will need to establish a relationship with the service providers. Your server would communicate with their web services - each one would require it's own adapter.

jmcc

As Graham said above, it is a very manual process initially. You also need very good programmers who can think in the kind of coding needed to extract data from various websites if they don't have a data feed. The other aspect is that permissions might be required to aggregate data. Starting such a project is the easy part. The hard part is to keep it going and coping with changed data formats. There are a few books that might help but I am not sure about your programming languages or level.

Regards...jmcc

The Corinthian

One point to add is that you're going to have to maintain your data and/or application.

As Graham suggested, data harvesting could be done either manually or using screenscraping, but either way you'll need to have a resource in place to either input new or update data or fix your screenscraper whenever it inevitably 'breaks' (this will happen whenever the source site changes format significantly).

As such you should factor that into your ongoing costs.

Jonny Blaze

Beware of establishing a plan that relies solely on scraping websites too as it's possible that many institutions limit programmatic access to their content via robots.txt rules on the server. If other aggregators are achieving their results programmatically though you should be ok.

Just make sure to do your homework before you hire a developer to build anything I would suggest.

Graham

Jonny Blaze wrote: »

Beware of establishing a plan that relies solely on scraping websites too as it's possible that many institutions limit programmatic access to their content via robots.txt rules on the server. If other aggregators are achieving their results programmatically though you should be ok.

Robotx.txt is just a file, it doesn't control anything. It's more of a friendly note saying please do/don't do something or other.

Problems occur when they block your IP address/scraper or point the legal cannons in your direction.

Other aggregators may well have an agreement (paid or otherwise) to access source data. Don't assume that because someone else is doing it that it's ok for you to do it.

Jonny Blaze

Graham wrote: »

Robotx.txt is just a file, it doesn't control anything. It's more of a friendly note saying please do/don't do something or other.

Problems occur when they block your IP address/scraper or point the legal cannons in your direction.

Other aggregators may well have an agreement (paid or otherwise) to access source data. Don't assume that because someone else is doing it that it's ok for you to do it.

Oh I know it has no actual power to enforce the rules but I just figured any responsible, professional software developer would respect them is all!

You would still be better off trying to get in partnership with a company if your opener was "I wanted to scrape your site but then I saw your rules which asked me not to do it. So i'm asking you now if I can have some access to your data."

jmcc

The other aspect is that some site owners may ban IP ranges from certain data centres such as Amazon. This is more common amongst large site owners though.

Regards...jmcc

Talisman

I always block traffic with dodgy user agent strings in the web server configuration.