Building a Lab

Dr4gul4 · 07-07-2020 9:32pm #1

Good evening folks,

So, I currently have an N54L running Synology, it's old, very old ! and I find it's not really up to the task's I set it many years ago.
Coupled with out dated software i now find I have a need to spool up some virtual machines for pentest purposes.

So, I have some ideas in mind, and a budget, something relatively silent to run 24/7 with a single GPU or dual GPU's based on cost and if i can get them working in parallel for hashcat.

Disk space is not a concern really, I dont require a huge amount of space for plex purposes, say 1/2 TB there.

No backup, I dont care if it all falls down, as long as i can rebuild it all again within say 1 /2 days .

I've a feeling 32 Gb of memory and a min of 6/8 cores, or dual chips maybe suit., and some form of SSD for transcoding / cache in plex ( if im correct )

Further details below of course, nothing too fancy case wise, it's not build for drag racing, it's build for power

1. What is your budget? [€1000 - 1250]

2. What will be the main purpose of the computer? [ Esxi / Proxmox host - Plex on lan, Several Vm's for Pentesting, Hascat VM. ]

3. Do you need a copy of Windows? [No]

4. Can you use any parts from an old computer? [ Nope ]

5. Do you need a monitor? [No]

5a. If yes, what size do you need. [N/A]

5b. If no, what resolution is your current monitor and do you plan to upgrade in the near future? [1920x1080/1440x900/etc.] [N/A]

6. Do you need any of these peripherals? [N/A]

7. Are you willing to try overclocking? [Yes/ maybe]

8. How can you pay? [All of the usual methods i guess.]

9. When are you purchasing? [In 30-60 days]

10. If you need help building it, where are you based? [ No building assistance required]

Dr4gul4 · 08-07-2020 7:33pm

Yikes, nobody brave enough to take this one on given my budget maybe ?

Serephucus · 08-07-2020 8:23pm

I have a 1950X and X399 mobo that's not getting much use if they're any use to you as a base?

Dr4gul4 · 09-07-2020 8:18am

Serephucus wrote: »

I have a 1950X and X399 mobo that's not getting much use if they're any use to you as a base?

Tempting, 8 /12 or 16 cores ? Both are still regarded as heavy hitters i believe, if 18 /24 months old if im correct ?

google just told me it's the 16 core version

Coyler · 09-07-2020 6:14pm

Just an FYI, the SSD will do nothing for transcoding. That's one of those tech myths. However, look up how to cache VMs in the SSD though. That really helps.

If you want to improve transcoding, be aware *nix based plex only works with Intel iGPUs and Nvidia cards. But you are putting in GPU, so just get an Nvidia one.

Dr4gul4 · 09-07-2020 7:32pm

Coyler wrote: »

Just an FYI, the SSD will do nothing for transcoding. That's one of those tech myths. However, look up how to cache VMs in the SSD though. That really helps.

If you want to improve transcoding, be aware *nix based plex only works with Intel iGPUs and Nvidia cards. But you are putting in GPU, so just get an Nvidia one.

Fair play for the caching tip at the os level, Had pidegon holed an Nvidia card, would love 4 1080 TI's of course, but, it's not practical ! .

Plex use is really in house only, but im stuck at 720p, cant go higher on the res at the mo, and the version is old, so any boost will be an improvement.

Still not sure on the underlying Os, freenas maybe, had a better look at proxmox, and i see it's nas based ? so, not sure how well that might suit, i'd rather one host, JOBS/ RAID, and carve up the storage, locally, via NFS shares.

Coyler · 09-07-2020 8:22pm

Proxmox or Unraid is your answer here. Proxmox is a hypervisor first and file server second. Unraid is the other way around. However, gun to my head, Unraid is a better choice for a beginner. Much bigger community, more mature and a ton of how-to videos on YouTube. Proxmox is good as well, not knocking it, but it might not be the first option to go for.

As for transcoding, once you have any halfway decent CPU in there even the software transcoding can do several streams so don't fret about it too much. Just be sure to check the Nvidia NVENC support matrix when you are grabbing a GPU.

Serephucus · 10-07-2020 5:17pm

As someone who's using unRAID currently, I can recommend it. I've also played with ESXi, and have yet to play with Proxmox properly.

unRAId should do everything you want. As Coyler says, it's a storage box first, an hypervisor second, but it does all the stuff with VMs you'd expect. Slightly more fancy things like snapshots are possible, but you'll need to go through CLI for that rather than the (very user-friendly) GUI. It has a really nice Docker implementation though that makes setting things up really simple. Have a look at Spaceinvader One on Youtube for tutorials an ideas.

And apologies, I meant to get back to your last post; the 1950X is indeed a 16-core chip.

tazzzZ · 10-07-2020 11:46pm

I third unRAID... it doesnt support all filesystems natively.... as serephucus alluded too ie zfs. but there is a really helpful community for setting that and a multitude of other things up. Only thing is its not freeware (trials are available though). but its been worth every penny to me. even if it has resulted in me spending a stupid amount of money on becoming a data horder!

Also if consider plex transcoding to ram if you can. saved un needed writes to hdd or ssd. Also if plex is in use only in the house surely direct plays are the way to go. no transcoding required and little resources req to do that. just make sure you have players that support the video format.

Dr4gul4 · 11-07-2020 10:05am

Serephucus wrote: »

And apologies, I meant to get back to your last post; the 1950X is indeed a 16-core chip.

1950x and Board is nice, I hope you wont think me rude, but i'd prob aim for new hardware, simply down to life span, that is of course unless you're giving them away for a song

tazzzZ wrote: »

I third unRAID... it doesnt support all filesystems natively.... as serephucus alluded too ie zfs. but there is a really helpful community for setting that and a multitude of other things up. Only thing is its not freeware (trials are available though). but its been worth every penny to me. even if it has resulted in me spending a stupid amount of money on becoming a data horder!

For $60 i'd happily invest, i took a brief look this morning and it does appear to be a good alternative, with great community support.

tazzzZ wrote: »

Also if consider plex transcoding to ram if you can. saved un needed writes to hdd or ssd. Also if plex is in use only in the house surely direct plays are the way to go. no transcoding required and little resources req to do that. just make sure you have players that support the video format.

Jesus, talk about a moment of clarity ! I'll have to check, but i've a feeling it's all direct play on lan, but the bottle neck is on the N54L CPU right now, buffering over 720p .

I'll do some diggin, maybe it's config related, but either way, this old girl need to be sent to the farm soon.

Great advice so far folks, really appreciate it ...

Just need to match up some hardware now soon

Inviere · 11-07-2020 10:17am

I'll 4th Unraid. Serephucus convinced me last year to go this route instead of an all in one Synology unit....I'm sooooo glad I did. I've got Unraid doing more than I originally envisaged, and as such, would have wasted money on a Synology. The community for it is superb I must say, always help on hand if needed.

I've just upgraded the other day to a Plus license, added in a SAS card & two new Ironwolf 6TB drives*....was a little daunting doing it as I haven't messed with the array config since initial set up & I'd forgotten everything, but their forums & subreddit are a wealth of info. Job done in a few mins, parity rebuild overnight last night (swapped my original parity drive WD Red out into the data pool, & put an Ironwolf in as parity instead...damn SMR bs by WD...Seagate for me from now on), and everything back running sweet now.

*Serephucus also laughed at me when I said 18TB would be plenty for me...he was right....again

Dr4gul4 · 11-07-2020 10:27am

Inviere wrote: »

*Serephucus also laughed at me when I said 18TB would be plenty for me...he was right....again

18TB

I've lived off 1 TB for the last 6 years

But of course it's all personal pref, in my case, VM's and pure grunt is more important right now, of course i'll prob put something a little bigger than 1 TB in for storage needs anyway. but performance would be my key focus.

Didn't realize WD RED had gotten so bad these days,

Serephucus · 11-07-2020 12:22pm

Dr4gul4 wrote: »

1950x and Board is nice, I hope you wont think me rude, but i'd prob aim for new hardware, simply down to life span, that is of course unless you're giving them away for a song

That's entirely fair. I'd have a look at ASRock Rack's X470D4U board. I'm not sure if you need remote management for your setup, but I find it really handy.

You could pop any cheap Ryzen CPU in there for the moment, then when you figure out more clearly how much power you need, you can stick a 4th gen chip in there when they launch later in the year.

Inviere wrote: »

I'll 4th Unraid. Serephucus convinced me last year to go this route instead of an all in one Synology unit....I'm sooooo glad I did. I've got Unraid doing more than I originally envisaged, and as such, would have wasted money on a Synology. The community for it is superb I must say, always help on hand if needed.
[snip]
*Serephucus also laughed at me when I said 18TB would be plenty for me...he was right....again

It's really is tough being right all the time... I don't know how I manage really. :cool:

If you do go unRAID, the rule of thumb for storage is go for the single biggest capacity drive you can afford (that isn't an absolutely absurd €/TB). You'll pay a little more for the storage now, but it makes expansion later on that bit easier because you don't have to go swapping parity drives and the like, you need less drives in your box overall, less noise, heat, all that.

tazzzZ · 11-07-2020 3:10pm

I started the same.... i had been using windows home server 2011 up until whenever i changed over... i cant remember exactly but i believe i was sitting at < 4TB storage at that time... i now have an array of 91TB... about another 15TB in unnassigned devices, non array devices (for non unraiders out there), in ssd's and external hdd.

It becomes an absolute money pit if you let it. I am currently using a ryzen 1700 and im getting limitted by PCIe lanes atm. I want to have 2 nvme drives for caching but adding the 2nd is knocking out the last pcie slot on the mobo which is my 10gb nic. So now, like an absolute mental person, im looking at threadrippers and epyc setups... just to add a 2nd nvme... admittedly i do have a second HBA card in mind for a jbod connection and possible ssd zfs array...

You can see by my tangent how much of a rabbit hole this can become. but i literally started off with an old amd phenom 3 core that i unlocked to a 4 core and i got a 3x5.25inch to 5x 3.5inch HDD cage and went from there. Actually still have that whole system just sitting here behind me unused XD

Serephucus · 11-07-2020 3:38pm

tazzzZ wrote: »

I started the same.... i had been using windows home server 2011 up until whenever i changed over... i cant remember exactly but i believe i was sitting at < 4TB storage at that time... i now have an array of 91TB... about another 15TB in unnassigned devices, non array devices (for non unraiders out there), in ssd's and external hdd.

It becomes an absolute money pit if you let it. I am currently using a ryzen 1700 and im getting limitted by PCIe lanes atm. I want to have 2 nvme drives for caching but adding the 2nd is knocking out the last pcie slot on the mobo which is my 10gb nic. So now, like an absolute mental person, im looking at threadrippers and epyc setups... just to add a 2nd nvme... admittedly i do have a second HBA card in mind for a jbod connection and possible ssd zfs array...

You can see by my tangent how much of a rabbit hole this can become. but i literally started off with an old amd phenom 3 core that i unlocked to a 4 core and i got a 3x5.25inch to 5x 3.5inch HDD cage and went from there. Actually still have that whole system just sitting here behind me unused XD

Truer words have never been spoken. I started with 2x2TB drives on Server 2016, running in a basic desktop full of the cheapest hardware I could get my hands on at the time. I'm now in a 4U rackmount case with 84TB array storage, and probably 3-4TB of SSDs and unassigned devices, and am mulling over a second server for GPU passthrough with Parsec and a ZFS array.

I'll say what I said the last time someone asked me about server advise: Ye be warned. :pac:

AlmightyCushion · 11-07-2020 5:50pm

Telling yourself 'I'm just gonna build a small server with a few TBs storage' is probably the same as saying 'I'm just gonna take a little bit of heroin'.

Dr4gul4 · 11-07-2020 5:54pm

I'm just putting it out there, but 'Storage Anonymous' might not be a real thing, but you guys floating about at 70-100 TB , crazy ! I couldn't even imagine holding that much 'data' for in house use these days.....horses for courses

Serephucus wrote: »

I'd have a look at ASRock Rack's X470D4U board. I'm not sure if you need remote management for your setup, but I find it really handy.

God damn, that is a serious board ! I'll have a dig to see where the cpu limitation are on it, plus the cash concerns

tazzzZ · 11-07-2020 6:58pm

Dr4gul4 wrote: »

I'm just putting it out there, but 'Storage Anonymous' might not be a real thing, but you guys floating about at 70-100 TB , crazy ! I couldn't even imagine holding that much 'data' for in house use these days.....horses for courses

We all started out the same way... it will suck you in when you get going! first for me it was about getting a nice automated setup. Then having plex running as fast and as well as possible. then i had a few friends join my plex so i wanted to push the transcoding side of things... then i wanted to have 10gbe networking between my main pc and server.... then.... you see where im going. And dont even get me started on the money im throwing into this to keep myself entertained during lockdown... i ended up changing my whole home networking gear to ubiquiti...

Serephucus · 11-07-2020 10:12pm

Dr4gul4 wrote: »

God damn, that is a serious board ! I'll have a dig to see where the cpu limitation are on it, plus the cash concerns

Then it's a good thing they absolutely don't make a TRX40 board, no sir. Very good thing. :pac:

tazzzZ wrote: »

We all started out the same way... it will suck you in when you get going! first for me it was about getting a nice automated setup. Then having plex running as fast and as well as possible. then i had a few friends join my plex so i wanted to push the transcoding side of things... then i wanted to have 10gbe networking between my main pc and server.... then.... you see where im going. And dont even get me started on the money im throwing into this to keep myself entertained during lockdown... i ended up changing my whole home networking gear to ubiquiti...

I built a whole new server during lockdown. It's dangerous I tells ya.

tazzzZ · 11-07-2020 11:18pm

not to hyjack the thread anymore than we already have... but would you go epyc or threadripper??? bare in mind i dont use it for vms really.... so higher clocks arent as important. i just cant justify the cost of the lowest core threadripper... i can do epyc for alot cheaper it seems. biggest thing im looking for is a good mobo with 10bge and many many pcie slots... asrock epyc server boards look nice.

z0oT · 11-07-2020 11:19pm

Serephucus wrote: »

That's entirely fair. I'd have a look at ASRock Rack's X470D4U board. I'm not sure if you need remote management for your setup, but I find it really handy.

You could pop any cheap Ryzen CPU in there for the moment, then when you figure out more clearly how much power you need, you can stick a 4th gen chip in there when they launch later in the year.

I have that board, and I'm mostly happy with it. It was my introduction to IPMI, and having had it now I would never go back to not having it. Being able to actually debug hardware problems or install an OS remotely is so insanely handy once you get used to doing it that way.

I have had some problems with PCIe passthrough on that board though, and the BIOS revision is seriously lagging from ASRock. They haven't released a BIOS with the newest AGESA version yet, but you can reach out to their support directly to get a newer Beta version if you want. I haven't tried this myself.

For the OS on my system, I went with a headless install of Debian administered entirely via SSH over the command line. Debian is kind of bread and butter for me at this stage, having used it for so long. The cool thing is that I've managed to replicate probably most of the functionality of something like Unraid in doing so, with the main exception being the real time parity.

For me it's kind of fun to really learn the inner workings of things which not having a GUI forces you to do. Some things like PCIe passthrough can be very challenging though which is a downside. Not necessarily recommending going this way, it is of course not for everyone.

I started with my own server in a serious way probably earlier this year. Previously I had an Athlon 5350 setup with a few big HDDs in it that just ran Plex, then I upgraded that to a R3 2200G doing the same thing. The warnings of it being a drug are really spot on. I'm getting seriously tempted to purchase an Eypc based system to get real server class hardware now. You'll never start small and stay small!

As for reccommendations, it might be worth trawlling second hand stuff given the budget. The real server class stuff is the best to get, but the prices can be killer.

Inviere · 12-07-2020 7:48am

Dr4gul4 wrote: »

18TB I've lived off 1 TB for the last 6 years

Hell, if I could store myself on Unraid at this stage I think I would (parity protected of course!). Yeah, the requirements of an Unraid server have a tendency to grow, the more you realise the cool and different things it can do...so your storage needs are inclined to follow suit.

This brings a different problem....parity itself shouldn't be viewed as redundancy...so you're left with a large Unraid array and the need to have its data protected. I use offline usb external backups of everything....so yeah, moar hdd's!!

Inviere · 12-07-2020 8:02am

AlmightyCushion wrote: »

Telling yourself 'I'm just gonna build a small server with a few TBs storage' is probably the same as saying 'I'm just gonna take a little bit of heroin'.

Dr4gul4 · 12-07-2020 4:24pm

Damn storage junkies

Ok, so the basic premise would be, buy the storage, it's cheap, ish. you'll be thankful in the long term.

I've been looking at a few diff builds and see some using M.2 drives ( 2 or 3 where applicable ) and some 2.5 SSD's along with the usual SATA 3/4/5 + TB drives.

As im an unraid noob, I'm wondering how they all fit together, i'm guessing the M.2's could be used to cache OS's and plex data ? the standard SATA's are hard storage. the 2.5 .... boot for unraid maybe ?

Any rough suggestions ? or " you should look at this concept for performance. "

tazzzZ · 12-07-2020 4:55pm

Dr4gul4 wrote: »

Damn storage junkies

Ok, so the basic premise would be, buy the storage, it's cheap, ish. you'll be thankful in the long term.

I've been looking at a few diff builds and see some using M.2 drives ( 2 or 3 where applicable ) and some 2.5 SSD's along with the usual SATA 3/4/5 + TB drives.

As im an unraid noob, I'm wondering how they all fit together, i'm guessing the M.2's could be used to cache OS's and plex data ? the standard SATA's are hard storage. the 2.5 .... boot for unraid maybe ?

Any rough suggestions ? or " you should look at this concept for performance. "

Ok well ill give you my 2 cents on this and others im sure have their own ideas.

Firstly unRAID runs directly off the usb stick so no boot device req. which is great as everything is then storage.

As for cache drives i personnally have found that ssd's have been best for me. I have recently tried M.2(nvme) and i find (or i think) it is getting limitted by the DRAM cache in it. ie when it fills it slows down. This can happen on bigger transfers ive found. And as a result i have had huge fluctuation of speed with them. And even my top end speed with them was similar to the SSD anyway. Im on 10gbe and both seem to max out at about 400MBps but SSD will sustain that for the whole transfer where nvme died off in my case. So in the end i've settled on the following setup..

Array/Parity - HDD
Cache - SSD
docker/VM - nvme (including appdata)

I also assign certain dockers their own appdata drive to ensure other dockers dont effect it. ie plex runs off its own SSD.

Serephucus · 12-07-2020 7:20pm

So unRAID has 4 main device types, if you like:

1) Parity - This can be one drive or two, and gives you that number of drives worth of tolerance in the event of a failure of a drive in your array.
2) Array - Your basic unit of storage. The size of your largest data drive, cannot exceed the size of your parity drive.
3) Cache - Usually a faster drive to store things like VMs and Docker containers, and can also serve as a write cache to speed up transfers to your server. These can be used as a single drive, or a pool of drives using BTRFS to offer RAID 1 redundancy.
4) Unassigned Device - You technically need a plugin for this, but it's such a core feature I'm including it here. It's what it sounds like: A disk you add that's separate from the array and the cache to be used as you like.

Some notes, on cache and storage arrangements:

At the moment, only 1 cache is possible (whether a single drive or a pool), but in the latest version of unRAID (currently in beta) multiple caches are possible. This lets you do things like keep your VMs on a separate disk/disks from the day-to-day stuff you're writing to the array, or from your Docker containers, for example.

There is also currently a bug that's causing massive write amplification on BTRFS cache devices.
(A single cache disk can be formatted as XFS - like the array - or BTRFS, but to be in a pool of more than one device, the drives must use BTRFS)
In my case, this meant I was writing close to a terabyte a day to my cache, which wasn't great, so I dropped a couple of drives and moved to a single XFS cache, with hourly backups of stuff that's cache-only to the array to give some form of protection.
For context, this is probably the worst bug I've seen in unRAID since I've used it, but they're working on it in the betas of the next release at the moment.

Dr4gul4 · 12-07-2020 7:31pm

Fair play guys, i think i've some good clarity on the over all layout now, given the info ye have shared, and some YT video's ..

Back to my hardware search soon ...

Coyler · 13-07-2020 12:40am

Something came to me regarding plex and switching from 720p to 1080p. What router are you using? Chance ares, as tazzzZ says, you will be able to direct play most content anyway and you might be hitting a bottleneck with your router. Best confirm before you buy all this fancy new gear. Now, while we'd all love to get a ubiquiti setup like tazzzZ (Jealous), you can still get some very decent routers for a lot less than you think. Now, you may end up moving the router functionality to your server eventually but good Wifi is just something you have to pay for.

Anyway, what I'm saying is don't forget your network. Your server is only as good as it is.

Dr4gul4 · 13-07-2020 8:46am

Coyler wrote: »

Something came to me regarding plex and switching from 720p to 1080p. What router are you using? Chance ares, as tazzzZ says, you will be able to direct play most content anyway and you might be hitting a bottleneck with your router. Best confirm before you buy all this fancy new gear. Now, while we'd all love to get a ubiquiti setup like tazzzZ (Jealous), you can still get some very decent routers for a lot less than you think. Now, you may end up moving the router functionality to your server eventually but good Wifi is just something you have to pay for.

Anyway, what I'm saying is don't forget your network. Your server is only as good as it is.

On lan only Coyler, Standard unmanaged switch 10/100, backed with Cat6e.

I was doing some investigation over the weekend and can see plex is direct streaming my 720p, I wonder if it's the audio on "other" files that maybe causing an issue with the CPU. i'll do some 1080p tests soon to try and identify whats spiking the CPU.

Either way ! the upgrade is required, the core reason being, i'm running home assistant on a Pi3, which has reached it's limitation unfortunately, so i'd like to move that to a docker + the hashcat etc requirements..

tazzzZ · 13-07-2020 11:09am

Dr4gul4 wrote: »

On lan only Coyler, Standard unmanaged switch 10/100, backed with Cat6e.

I was doing some investigation over the weekend and can see plex is direct streaming my 720p, I wonder if it's the audio on "other" files that maybe causing an issue with the CPU. i'll do some 1080p tests soon to try and identify whats spiking the CPU.

Either way ! the upgrade is required, the core reason being, i'm running home assistant on a Pi3, which has reached it's limitation unfortunately, so i'd like to move that to a docker + the hashcat etc requirements..

Im assuming you just forgot to put the /1000 on the end of the switch bit... either way shouldnt be a bottleneck for 720p.

In fairness its worth the upgrade. plenty of good bargains to be had on hardware out there. In the past I had a very nice middle ground cpu with the 4790S(also a T varient) thats low power and has onboard igpu which you can use for transcoding if thats something you ever do and dont want a seperate gpu for it. I would look for the more modern equivalent. Maybe a 9900t or lower tier or previous gen varient?

The reason id choose this over a ryzen is quicksink works well with plex if i remember correctly and can do even 4k transcodes (but i wouldnt advise this for other reasons). It has a marked tdp of 35W which is nice and managable... even if these are very loose guidance of power usage.

If you have another gpu to use in this then id be going ryzen all day long personally. cost to preformance is just a no brainer on this IMO.

AlmightyCushion · 13-07-2020 11:27am

tazzzZ wrote: »

Im assuming you just forgot to put the /1000 on the end of the switch bit... either way shouldnt be a bottleneck for 720p.

In fairness its worth the upgrade. plenty of good bargains to be had on hardware out there. In the past I had a very nice middle ground cpu with the 4790S(also a T varient) thats low power and has onboard igpu which you can use for transcoding if thats something you ever do and dont want a seperate gpu for it. I would look for the more modern equivalent. Maybe a 9900t or lower tier or previous gen varient?

The reason id choose this over a ryzen is quicksink works well with plex if i remember correctly and can do even 4k transcodes (but i wouldnt advise this for other reasons). It has a marked tdp of 35W which is nice and managable... even if these are very loose guidance of power usage.

If you have another gpu to use in this then id be going ryzen all day long personally. cost to preformance is just a no brainer on this IMO.

I have a 4570k in my current PC. When I eventually build a new pc (probably won't be until next year) I plan on turning my current one into a plex server. Obviously it's not as good as the 4790S but hopefully it's not far off it.

Building a Lab

Comments