Freenas 9.3 dodgy disk issues !!

dbit

So a year ago I built my own frenass with 4 x 2TB discs on raid 5 (Iperc6) 8GB of ram booting from 16GB pen and using zfs for volumes . All this presenting luns to esxi cluster (Using iscsi) and some nfs for home stuff.

Disks were seagate barracuda's, of the 4 I bought around 4 months ago one died and had it replaced , yesterday another disk goes in the raid , so i think phew close one , ten mins later another one fails . So i have lost my entire lab at home and now im pissed 3 of the 4 discs rma'd in under a year WTF amazon ?? what would you do ??


Western digital discs better for always spinning ?? any one have any experience with brands and freenas ??

Gone Drinking

RAID 5 with large SATA disks isn't recommended anymore. The rebuilds are too entensive and SATA isn't reliable enough. You're not the first person I've met who's fallen victim to RAID5 with SATA disks. I've seen people lose terrabytes of data. Go for RAID 6 next time around.

In regards to why they failed; It's doubtful both disks failed due to a hardware problem (unless they were part of a defective batch of disks). Are there any disk firmware updates avaialble for that model of disk? What about freenas, were you running the latest version of that?

dbit

Gone Drinking wrote: »

RAID 5 with large SATA disks isn't recommended anymore. The rebuilds are too entensive and SATA isn't reliable enough. You're not the first person I've met who's fallen victim to RAID5 with SATA disks. I've seen people lose terrabytes of data. Go for RAID 6 next time around.

In regards to why they failed; It's doubtful both disks failed due to a hardware problem (unless they were part of a defective batch of disks). Are there any disk firmware updates avaialble for that model of disk? What about freenas, were you running the latest version of that?

Yup latest version at the time of build , these disks are grinding complete shower of sh1te from seagate. first time around when one popped i was on raid 0 - fast but not what i needed . so deploying on raid 5 i thought haaa haaa now im covered , I am not seeing stability from the disks and didnt want to fork out for ssds. The iperc 6 is sas to sata adaptor and the controller is good . Raid 6, jesus more disks , more stripes man im pissed .


My lab was only spun up say 50 times in a year and only for a couple of hours at a time . Its the disks nothing else first one = grinding and got that rma'd . The two that went yesterday one is grinding and the other logic board is dead. Absaloute rubbish and all you get is crappy refurbs while under warranty, load of aul sh1te.

ED E

Seems strange to have such a high failure rate. You shouldnt come close to that until after about 3yrs of daily use. Are you certain they're all dud?

As above, RAID5 isnt a great idea. Get some WD RED 2TB's. Build a RAID6 and stress test the hell out of it before commissioning.

Also worth checking that there isnt an environmental reason why drives are dropping, excessive vibration, high operating temps etc.

dbit

ED E wrote: »

Seems strange to have such a high failure rate. You shouldnt come close to that until after about 3yrs of daily use. Are you certain they're all dud?

As above, RAID5 isnt a great idea. Get some WD RED 2TB's. Build a RAID6 and stress test the hell out of it before commissioning.

Also worth checking that there isnt an environmental reason why drives are dropping, excessive vibration, high operating temps etc.

Thanks Ed,
Ye looking over it all now ZFS seems to favor not using a raid card, loads of communities report the disk type seagate ST2000DM001 as a dud in general , some guys who bought 12 sent 5 back and then eventually all 12 . I was wondering why the box never seemed to use the 8 GB of ram ( 1gb of ram per TB of storage used) . reading up again on the freenas forums they all say go raidz1 or 2 if cost is a factor with no raid card. I do realize that they are only desktop range HD's but Jesus they pop so bloody quickly . The two that went yesterday are deffo dead , one grinds very loud at spin up and the other isin't even picked up on windows nor in bios if direct connected- thus logic board is kaput.


I am going to look up those WD's you mentioned next . And possibly opt for native sata's no raid card adn see how zfs handles them native.

ED E

Yeah, with plenty of RAM just fill them in a JBOD fashion and have freenas do all the heavy lifting.


So far, touch wood, the WD Red 3TBs have served me well, but I wouldnt expect problems due to wear for another 3yrs+.


SMART has limited use, but as you get it for free with all drives its worth setting up alerting for it. Every time my soft raid updates itself I get a log in my email, and if theres ever a serious issue detected by smart I'll get a mail so I can shut it down and intervene before the disk dies completely. Wont always catch a fault, but when it does I may as well know about it.

dbit

The only sh1tter i have now is that the native sata's on the mobo i used 2 are 6/GBs and 4 are 3/gbs. A bit sh1t having to replace that as i built it thinking hardware raid was the job. Dammit.

dbit

Guess i could go gold tier for zfs volume with vm's (On the 6/GBs) and slower tier of zfs volumes for smb /nfs stuff. Possibly locked then to a mirror only of 2TB (2x2tb) to the esxi cluster .

ressem

dbit wrote: »

Thanks Ed,
Ye looking over it all now ZFS seems to favor not using a raid card, loads of communities report the disk type seagate ST2000DM001 as a dud in general , some guys who bought 12 sent 5 back and then eventually all 12 . I was wondering why the box never seemed to use the 8 GB of ram ( 1gb of ram per TB of storage used) . reading up again on the freenas forums they all say go raidz1 or 2 if cost is a factor with no raid card.

Usually you would use a Raid card (battery backed ideally, with NVRAM), but in JBOD mode (just a bunch of disks); rather than the motherboard controller which have caused issues for others.

I'd wonder about the ability of the drive enclosure that you're using to dispose of heat when the drives are under load (during a rebuild), combined with desktop grade drives and their lower heat tolerance; but you do seem to have a bad batch.

dbit

ressem wrote: »

Usually you would use a Raid card (battery backed ideally, with NVRAM), but in JBOD mode (just a bunch of disks); rather than the motherboard controller which have caused issues for others.

I'd wonder about the ability of the drive enclosure that you're using to dispose of heat when the drives are under load (during a rebuild), combined with desktop grade drives and their lower heat tolerance; but you do seem to have a bad batch.

Temps are super cool inside the box , I even mounted a shield and extra fan at the back of the disk array inside the case that would increase air flow - before i did this they were getting a little hot , after i did this the temps are very low. This was all done before i had any disk failures and at build time.

Ye ok i see i can use the Iperc 6 in jbod and have freenass do the heavy lifting yes understood. at least i will retain the faster channels . (And my iperc 6 is battery backed)

dbit

It has been suggested all over that the jbod raid card on zfs should not push the disks as hard any opinions on that aspect ?

ED E

IIRC ZFS only spins up the drives it needs, like flexraid that I'm using, so it would be similarly easy on the disks.

Sata3 isn't a limit for HDDs btw, its only a concern for SSDs.

dbit

ED E wrote: »

IIRC ZFS only spins up the drives it needs, like flexraid that I'm using, so it would be similarly easy on the disks.

Sata3 isn't a limit for HDDs btw, its only a concern for SSDs.

Seriously the IO has no factor against HDD's ?

dbit

Wow i never realized that :-
In terms of everything else, we saw basically no difference from the same single drive being plugged into a SATA 6Gb/s or SATA 3Gb/s port. All of this is obviously due to the hard drive not being able to actually take advantage of the 6Gb/s bus.

dbit

IS there any win to me using the iperc 6 then ? I dont see that there is unless software raid on mobo bottle necks right ?

ED E

The mobo won't raid across chips, but can JBOD them.

If you go ZFS the lperc is redundant, but if you keep hw raid then keep it as itll be far more reliable and probably faster.

dbit

ED E wrote: »

The mobo won't raid across chips, but can JBOD them.

If you go ZFS the lperc is redundant, but if you keep hw raid then keep it as itll be far more reliable and probably faster.

I did build zfs volumes on the raid 5 setup via the iperc and did not see expected ram utilization on the freenas.

I am taking that you mean use the iperc but in jbod and then zfs volumes to be used in freenas ?

dbit

Woooo Hooooo All is well , the second disk i thought was dead is not the usb caddy i was testing it with is not giving it enough power , popped back into iperc and found for some reason the second disk was moved to "Foreign" on the iperc , ( Imported the config again) A quick tidy up, remove a lun or two, and power down till the grinder gets rma'd

Thank fook for that so much config time went into that lab. YaaaaaY!

I WAS looking at loss of VC, VCD, NSX, vds's, DSM and array of DSVA's and a good chunk of VM's all back .

dbit

Thought about it a little more after your poke in the ass Ed , true not likely to have 2 go within minutes of each other.

ED E

Only know about half of them :P

Yeah, they only usually fail together due to an impact or a surge, or some other catastrophe.


Now remember the lesson here, RAID is not backup

dbit

ED E wrote: »

Only know about half of them :P

Yeah, they only usually fail together due to an impact or a surge, or some other catastrophe.


Now remember the lesson here, RAID is not backup

Yes , Daddy.

Cuddlesworth

Pick up a 6Tb usb disk and run a scheduled backup to it if the data is important.

On the topic of the drives, what usually happens is drives are bought in one big purchase and so usually come from the same batch of disks made. If you're unlucky to get a manufacturing flaw, then the risk the disks will all die within a close period of time increases substantially.

I've seen enterprise disk arrays have a spate of failures in a short period and nothing after for a long time. They usually have more then enough redundancy to cope with it though.

dbit

Cheers for all the input lads . Much Love DBIT. ( Terry Tibs voice used)