Running a data refresh from an online provider

TinCool

Here's my little pondering.

I am currently working on a project with an ASP .NET Frontend and MSSSQL backend. I have a data refresh process that I need to run hourly which pulls data from a couple of external API's. Since the datasets being pulled are quite large, it's not possible to call them from website calls initiated by the client as the response times are too slow.

I have written a console app to run the refresh but I am running this manually right now on demand. I'm wondering is there any cloud based/online solution where I could schedule an executable to run to do the data refreshes on a scheduled basis. I'd rather not have my own or another PC running 24/7 to accomplish this.

Suggestions ?

Fluffy88

A quick google returns this https://www.setcronjob.com/ no idea is it safe or reliable.

If you have your own website/server you could just setup one on your own server.

TinCool

Fluffy88 wrote: »

A quick google returns this https://www.setcronjob.com/ no idea is it safe or reliable.

If you have your own website/server you could just setup one on your own server.

Thanks, it'd probably work but not exactly what I was looking for. I think I'll just schedule it on my own web server at home.

Seth Brundle

Can you set up a scheduled job within MS SQL that pulls the data?
http://msdn.microsoft.com/en-us/library/ms191439.aspx

TinCool

kbannon wrote: »

Can you set up a scheduled job within MS SQL that pulls the data?
http://msdn.microsoft.com/en-us/library/ms191439.aspx

No, my Hosting Provider does not provide access to the SQL Agent on their shared SQL Server instance, so no jobs can be setup/scheduled. I would have gone done this road if it wasn't for this restriction.

Fluffy88

By console app what do you mean?

I usually write an update script in PHP, Perl, Bash or whatever language suits then just run a cronjob on the webserver to execute the script. I've never used Windows server and I'm guessing that's what your using so I am probably not the person that should be trying to help here.

But if you have a home server I assume you let it run 24/7 so you might as well use it
And if you have a hosting service you should definitely use it, they surely have some method of automated tasks.

ressem

Since the datasets being pulled are quite large, it's not possible to call them from website calls initiated by the client as the response times are too slow.

So as Fluffy suggests, there's no technical reason why a page on your web server and the backend code can't carry out the operation aside from taking a minute or so to return the result and update the cache for your clients?

You're working in c# ? What versions of .net does your hosting provider allow?
Have you seen whether there is support for the application level BackgroundWorker?
http://code.msdn.microsoft.com/CSASPNETBackgroundWorker-dda8d7b6

You have to be aware that the application using this might be garbage collected if quiet.

TinCool

Fluffy88 wrote: »

By console app what do you mean?

I usually write an update script in PHP, Perl, Bash or whatever language suits then just run a cronjob on the webserver to execute the script. I've never used Windows server and I'm guessing that's what your using so I am probably not the person that should be trying to help here.

But if you have a home server I assume you let it run 24/7 so you might as well use it
And if you have a hosting service you should definitely use it, they surely have some method of automated tasks.

Basically, I'm pulling data from a few external API's. I have written a vb console application (i.e. an .exe that runs in a dos window (windows)) which does all the data refreshing in the database.

The easiest option for me is to just schedule it on my server at home.

TinCool

ressem wrote: »

So as Fluffy suggests, there's no technical reason why a page on your web server and the backend code can't carry out the operation aside from taking a minute or so to return the result and update the cache for your clients?

You're working in c# ? What versions of .net does your hosting provider allow?
Have you seen whether there is support for the application level BackgroundWorker?
http://code.msdn.microsoft.com/CSASPNETBackgroundWorker-dda8d7b6

You have to be aware that the application using this might be garbage collected if quiet.

This is also an option. I'll read up on this backgroundworker. I'm running .NET 4.0 and writing my .NET code in VB.

Thanks for the suggestions.

Procasinator

If you can't access a task scheduler of some sort, you can use a cache invalidation trick in ASP.NET.

http://www.codeproject.com/Articles/12117/Simulate-a-Windows-Service-using-ASP-NET-to-run-sc

It is a hack, but it could get the job done.

While inserting dummy data into cache, set a timeout and a method to callback when the cache is modified. When this callback is called, you have hit your interval and execute what ever task you want.

This task is not tied to a request, but it is tied to the ASP.NET thread pool.

Freddio

Could you put the update service on Amazon and call it up from a cron through http on a.n.other hosting plan you have lying around?

TinCool

Procasinator wrote: »

If you can't access a task scheduler of some sort, you can use a cache invalidation trick in ASP.NET.

http://www.codeproject.com/Articles/12117/Simulate-a-Windows-Service-using-ASP-NET-to-run-sc

It is a hack, but it could get the job done.

While inserting dummy data into cache, set a timeout and a method to callback when the cache is modified. When this callback is called, you have hit your interval and execute what ever task you want.

This task is not tied to a request, but it is tied to the ASP.NET thread pool.

Hmm, this sounds interesting. I'll check it out. Cheers !

TinCool

I found this website: https://scheduler.codeeffects.com which will work very nicely indeed. Plus you get 1 x free schedule which is all I need. Tested it and it works as expected. Data is refreshing as I type.