Clusters

Ziycon

Can anyone tell me how to go about making a cluster from a few pcs i have lying around??

SouperComputer

what do you want to do with the cluster? This will decide the OS\Software you are going to use.

I used plan9 a few years ago.

Ziycon

It would be for storage. Then maybe be able to map the items to a drive over the network.

Tom Dunne

Are you confusing a cluster with a server, by any chance?

A clustered pair, for example, consists of two computers and a shared set of drives. The drives can switch between either node in the cluster, but not both. The cluster is accessed through a single address, the idea being that if one computer in the cluster fails, the other takes over and the data on the drives is still available.

Clusters are designed for high availability of the drives (and applications) - would you have need for this in your situation?

Ziycon

tom dunne wrote:

Are you confusing a cluster with a server, by any chance?

A clustered pair, for example, consists of two computers and a shared set of drives. The drives can switch between either node in the cluster, but not both. The cluster is accessed through a single address, the idea being that if one computer in the cluster fails, the other takes over and the data on the drives is still available.

Clusters are designed for high availability of the drives (and applications) - would you have need for this in your situation?

Im not confusing it with a server, what you described above is exactly what i need it for!

masteroftherealm

Ziycon wrote:

Im not confusing it with a server, what you described above is exactly what i need it for!

What exactly is the usage as clustering solutions iare very usage dependednt to reccomend u one I kinda need to know what u using it for.

aidan_walsh

Ziycon wrote:

Im not confusing it with a server, what you described above is exactly what i need it for!

Sounds like you need a RAID array, more than a cluster.

Cryos

aidan_walsh wrote:

Sounds like you need a RAID array, more than a cluster.

However with limited budget i suspect he needs a cluser as he said PC's, not big hunking servers with raid arrays

masteroftherealm

Blitz wrote:

However with limited budget i suspect he needs a cluser as he said PC's, not big hunking servers with raid arrays

Ummmmmm nope.
Raid aint got nuthing to do with how big the server is.
I suspect raid also.
Clustering is only needed if u need data crunching or high scale reliability for a specific aplication in which case u aint making it outta old PCs anyway.

Ziycon

Okay, basicly im doing some research for my fourth year project, basicly im deciding what project i want to do and im thinking of writing a backup application with a client part on the client pc monitoring certain folders but im trying to figure out what is the best storage method and i want to look into Clustering as it seems the best way forward for a data backup application.

Capt'n Midnight

clusters *me missing the whole point*
http://www.beowulf.org
http://clusterknoppix.sw.be/ - way out of date

Tom Dunne

Ziycon wrote:

Okay, basicly im doing some research for my fourth year project, basicly im deciding what project i want to do and im thinking of writing a backup application with a client part on the client pc monitoring certain folders but im trying to figure out what is the best storage method and i want to look into Clustering as it seems the best way forward for a data backup application.

I have some experience of clustering, but I am having trouble understanding how it could be applied to a data backup solution.

Don't get me wrong, I am in no way trying to undermine you or your project (quite the opposite, I'm trying to help ). But I still get the feeling you might be going off in the wrong direction with your intention to use clustering.

zilog_jones

Yeah, I'd always think of clustering as a way of combining processing power for common tasks. I suppose it may be useful for redundancy, but just replication would be easier for that.

layke

Clusters are generally used for failsafes for data. Are you talking about something along the lines of the way a domain controller and a secondary domin controller replicate data between them? Or from your first two posts you are looking to create a SAN to store data?

ressem

Clusters are generally used for failsafes for data.

As discussed above, more usually it's for providing an application with CPU time from multiple machines. Distributed file systems are often used in conjunction with a cluster to provide some redundancy and a shared namespace, eg MS DFS, Google FS, or the ones studied in distrib sys textbooks like andrewfs and coda.

SouperComputer

im still a bit confused myself......

How much storage do you need? Surely RAID 5 will suffice? IE take the drives out of all the PC's, throw them in one PC and use LVM.

Failing that, SAN is what you might be looking for:

http://sourceforge.net/projects/broke-aid
http://sourceforge.net/projects/netserva

Ziycon

i want the server storing the data to be on a cluster, i was only picking a cluster as one avenue for the application, just keeping my options open!

masteroftherealm

Yeah.
Clustering is very comlicated, youd have to produce a veriosn of your application that was highly parralell in order to use it on a cluster.
Clustering aint as easy as stringing a few boxes together. Its generally highly specialised.

ShiverinEskimo

I thought of clustering as a failsafe for the AVAILABILITY of the data and not the actual safe keeping of it. In otherwords ensuring the service is available even after system crash or system failure whereby another node can take the load and still access the same data.

A clustering system with a RAID setup for data security is common ensuring disaster recovery and resiliancy.

I could be wrong.

Tom Dunne

ShiverinEskimo wrote:

I thought of clustering as a failsafe for the AVAILABILITY of the data and not the actual safe keeping of it. In otherwords ensuring the service is available even after system crash or system failure whereby another node can take the load and still access the same data.

A clustering system with a RAID setup for data security is common ensuring disaster recovery and resiliancy.

I could be wrong.

No, you are spot on. High availability is one of the key functionalities of a cluster.

@Ziycon: Is there a particular reason you need the server storing the data clustered? Sounds like Linux is the way to go. I think clustering options for Windows cost money (I know Microsoft's own clustering does).

Ziycon

tom dunne wrote:

@Ziycon: Is there a particular reason you need the server storing the data clustered? Sounds like Linux is the way to go. I think clustering options for Windows cost money (I know Microsoft's own clustering does).

No particular reason at all, its just an option, i need to figure out what type of server system the server side of the application will run on so i can develop the application for the server system if you understand me!?

masteroftherealm

Your trying to develop for a cluster based system?
Or your trying to delevop a program that can use a cluster to process?

Ziycon

Ok how to explain this, the application im developing, i want the server part of the application to run on a server and that server wheter its cluster or RAID im not sure yet, i basicly want to decide what the best option for the server is for this type of application and i thought of cluster because of the failover as it is a critical application which will be updating constintly!

I was going to use RAID but in work a few weeks ago there was a server with 8 hotswap drives and 7 out of the 8 failed in one go which put me off it.

masteroftherealm

Clustering will prvide ni more failsafe than an 8drive raid if thats what your thinking?
And are u aware how much extra effort/impossibility your putting on your coding skillls to make an app cluster aware and ip fallover compatible.

Ziycon

masteroftherealm wrote:

Clustering will prvide ni more failsafe than an 8drive raid if thats what your thinking?
And are u aware how much extra effort/impossibility your putting on your coding skillls to make an app cluster aware and ip fallover compatible.

I know there would be a lot more work involved, maybe i'll stick with a RAID as i do have only a few months to code the project.

Thanks to all.

masteroftherealm

Id reccomed goin with the raid mate
saves yourself a lotta bother.

zilog_jones

Ziycon wrote:

I was going to use RAID but in work a few weeks ago there was a server with 8 hotswap drives and 7 out of the 8 failed in one go which put me off it.

Sickener. I suppose that's one of the big flaws in RAID - if the disks are all bought at the same time and you get a bad batch of disks you're screwed.

nadir

www.lustre.org/ tbh.

SouperComputer

zilog_jones wrote:

Sickener. I suppose that's one of the big flaws in RAID - if the disks are all bought at the same time and you get a bad batch of disks you're screwed.

Yup, thats one of the reasons to always try to get drives from different batches.

I think start off with a RAID setup, get the framework for your app setup and then look into a cluster setup.

Capt'n Midnight

novell has been able to mirror servers since the early 90's if not earlier but you needed identical hardware - they used to drop anvils on them at expo's

linux can be setup that way

You could do something in windows by using rsync / robocopy running continuously. You have an IP address of the share
you'd assign TWO ip addresses to each server. one address would be fixed and the servers would use that to talk to each other. the other would be a generic address . if ping or whatever failed then the server would use NETSH to reprogram the second address, to the one the clients use (when the other server came back on line it would see that address in use and not hijack it)
that would be possible even in win 98 if you saved and loaded reg settings.

enterprise should do something similar but over kill unless you have $$$$$'s for licenses

Capt'n Midnight

Ziycon wrote:

I was going to use RAID but in work a few weeks ago there was a server with 8 hotswap drives and 7 out of the 8 failed in one go which put me off it.

yeah your one point of failure now becomes the backplane. also you can't pop the RAID drives into another server to read the data unless lots of things are the same, especially the raid controller and it's bios.

Have's a look at this :eek: :eek: :eek:
http://support.dell.com/support/topics/global.aspx/support/dsn/en/document?docid=10DA10AD1A67ADD9E040030ABD62644F&c=us&l=en&s=gen

Higher than expected failures rates have been reported on the following Maxtor Blackbird ULD (unleaded or lead free) SCSI hard disk drives and Maxtor Genesis ULD (unleaded or lead free) SCSI hard disk drives.

If the hard disk drive (HDD) is unable to complete commands, this may result in the controller reporting the HDD offline due to the timeout condition. The primary failure modes have been the Maxtor Blackbird ULD SCSI HDDs and the Maxtor Genesis ULD SCSI HDD failing to successfully rebuild and also failing after a rebuild has completed.

The following Dell part numbers and model numbers are affected. Dell considers this as an "Urgent Update" and recommends proactively updating the firmware to avoid interruptions.

NOTE: The Maxtor Blackbird HDD is also known as the Atlas 15KII. The Maxtor Genesis HDD is also known as the Atlas 10K V.

Maxtor Genesis ULD SCSI HDD Part Numbers

* YC951 – Model 8J073L0020851 - 73GB 10K 68 pin
* CD807 – Model 8J147L0040852 - 146GB 10K 68 pin
* FD456 – Model 8J300L0080853 - 300GB 10K 68 pin
* GD084 – Model 8J073J002075E - 73GB 10K 80 pin
* YC952 – Model 8J147J004075F - 146GB 10K 80 pin
* CD808 – Model 8J300J008075G - 300GB 10K 80 pin

Maxtor Blackbird ULD SCSI HDD Part Numbers

* FD457 – Model 8K036L0021451 - 36GB 15K 68 pin
* GD086 –Model 8K073L0041452 - 73GB 15K 68 pin
* YC953 – Model 8K147L0081653 - 146GB 15K 68 pin
* CD809 –Model 8K036J002135E - 36GB 15K 80 pin
* FD458 – Model 8K073J004135F - 73GB 15K 80 pin
* GD088 - Model 8K147J008155G - 146GB 15K 80 pin

The following drives can benefit by having the JT00 revision of the firmware installed but is considered as a "Recommended Update" rather than "Urgent" as listed for the unleaded Blackbird and Genesis models.

Maxtor Genesis SCSI HDD Part Numbers

* T4350 – Model 8D073L0022851 - 36GB 10K - 68 pin
* U4006 – Model8D147L0042852 - 73GB 10K - 68 pin
* R4784 – Model 8D300L0082853 - 300GB 10K - 68 pin
* CC315 – Model 8D073J002495E - 73GB 10K - 80 pin
* FC271 – Model 8D147J004495F - 146GB 10K - 80 pin
* CC317 – Model 8D300J008495G - 300GB 10K - 80 pin

Maxtor Blackbird SCSI HDD Part Numbers.

* T4357 – Model 8E036L0025451 - 36GB 15K - 68 pin
* N4723 – Model 8E073L0045452 - 73GB 15K - 68 pin
* W4019 – Model 8E147L0085653 - 146GB 15K - 68 pin
* FC272 – Model 8E036J002C15E - 36GB 15K - 80 pin
* CC319 – Model 8E073J004C15F - 73GB 15K - 80 pin
* FC273 –Model 8E147J008C35G - 146GB 15K - 80 pin

stevenmu

I don't see why some limited custom form of clustering can't be an option if you're writing your own code for it, and seeing as you are writing your own code it doesn't matter too much what platform you go with.

The app you write to sit on your 2, or however many, servers would negotiate which server would be master and which would be slave(s). If a new server joins the cluster it would join as a slave and the master would transmit it a fresh copy of all clustered data. You would need some kind of scheme whereby if the master goes down the remaining slaves would detect this and negotiate a new master amongst themselves, possibly all servers could periodically ping each other or the master could periodically send an "I'm still alive" singal to all the slaves.

The client software would need some kind of scheme for finding the current master, such as broadcasting to the local subnet for a master of a given cluster id. It could then transmit data to the master as needed, the master would then pass the data on to the rest of the cluster. If the master goes down the client would return to scanning the subnet for the new master.

For bonus points consider a scheme to allow a client use a 'local' master in the case of a cluster spread across multiple offices

Actually, reading back over that, replication is definitly a more appropriate term than cluster.