best way to parse data from webpage

Placebo

Need to retrieve data from a stock site.
What would we the best way to retrieve data from it [and store it in a db?]

php - curl ? parse by looking at page structure, i.e if it uses tables etc?

Thanks

Eoin

As in a stock price / financial website? Sounds obvious, but see if they have an RSS or atom feed you can use. Or they have a ticker which might be nicely marked up and easy to parse.

Placebo

Yeah, stock price / financial website.
No API/RSS/

ticker =?

Eoin

You know the scrolly things that show the stock prices? That might use a feed you can use.

RandiR

You can automatically get stock prices using a script similar to one below. This script is written in biterscripting. It downloads a stock's price from finance.yahoo.com.

# [B]SCRIPT: Price.txt[/B]
# Input argument.
var string symbol
 
# Get the first table from finance.yahoo.com for the stock symbol.
var string URL, table
set $URL = "http://finance.yahoo.com/q?s="+$symbol
script ss_webpagetocsv.txt page($URL) number(1) > $table
 
# Get the first row.
var string row ; lex "1" $table > $row
# Get the first column.
var string column ; wex "1" $row > $column
# $column now has stock price in string format. Convert it to real.
var real price ; set $price = makereal(str($column))
 
# Print price. Or insert into DB or excel.
echo $price

Store the script in C:\Scripts\Price.txt. Call it from biterscripting as follows.

script Price.txt symbol("<your stock symbol here>")

If you want to load the prices into DB instead, insert an sql statement in the above script. You can do this for multiple stock prices by writing a script that calls the above Price script once per stock symbol. Also, you can schedule this activity daily/hourly/etc. using task scheduler or cron.

As far biterscripting ( http://www.biterscripting.com/install.html ), it is just a general scripting language. It has nothing to do with stock prices. You can use any scripting language. Just code the above functionality in the scripting language of your choice.

Randi

(Make sure any site you use to download prices from, that it is ok with them to download prices using such batch approach.)

Josh223

For Stock prices in Excel, I use iXBRLAnalyst Addin for Excel. It adds a new built-in function "SharePrice(ticker, date)".