Multicore (12+) build machine

srsly78

You can even get jobs as a "build engineer" these days, which means optimising and working with CI a lot. Optimizing a game engine for runtime performance and optimizing a build system are two completely different things...

KyussBishop

It's a project that (accumulatively) probably has well over a hundred people working on it at the moment (including a whole build team); it's as efficient as it will get.

I've checked out a few different ways of optimizing the speed locally, and processor power is definitely the bottleneck - combine that with restrictions from the toolset the project uses (making distributed compiles costly - using Incredibuild), and the best solution for me is a faster processor (which is what the thread is about - not about the build setup).

smacl

Sorry, but not buying it...

A C++ software project built using Visual Studio that involves over a hundred people in development is obviously going to be made up of large number of smaller Visual studio projects in one or more solutions that build the components required. Anyone with any amount of build experience would tell you that the fastest way to get this built is across a number of PCs in parallel. Even if you had to rebuild the whole lot (unlikely) on a very regular basis, doing so across a large array of PCs with one or more projects per PC is going to result in a total build time that is an order of magnitude faster than what you could hope to achieve on even the fastest dual or even quad xeon set-up. The fact that such a project would be run without some kind of CI in this day and age, and that you have a build team just does not add up.

Putting one fast PC under a single developer in this scenario, even if they are a lead developer, doesn't seem liable to significantly influence overall build times.

farna_boy

Would you consider just upgrading your graphics cards and using them to process the data?

I know it is applicable in some scenarios, especially so when you can do parallel processing but I'm not sure if it would work in your case. Nvidias CUDA seems to be the basis for a lot of this (from the quick search I did) but it may be a viable alternative considering how much the Xeons will cost.

KyussBishop

I'm not asking for advice on the project guys, I asked for advice on the computer - which I got, and don't need more help with.

Honestly, if you want to offer help to someone, start by not making unfounded assumptions about what they are doing, and then ignoring the person when they tell you your assumptions are wrong (thus wasting everyones time).

smacl

KyussBishop wrote: »

Honestly, if you want to offer help to someone, start by not making unfounded assumptions about what they are doing, and then ignoring the person when they tell you your assumptions are wrong (thus wasting everyones time).

You started off with a budget of €1500-€2000 looking for a 12 core system to improve build performance on a large visual studio app. You talk about compilation causing a bottleneck as you are mucking about a lot with header files -> you're talking about a C++ project (possibly C but unlikely). You've never heard of CI, yet there are a hundred people working on this project which includes a dedicated build team. You've been told by multiple posters that high end multi-core processors will most likely not provide a reasonable return on investment, and have been provided with a number of cost effective alternatives.

The PC you're talking about is about 3-4 times over your budget and will give you at best about 60%-75% increase in build time over the PC that is inside your budget. Anyone with any nous at build optimization for large C++ projects would likely achieve a 300%-500% improvement for the same budget, and given that you haven't even heard of continuous integration, my guess is you know exactly squat about the subject of build optimization yourself.

And we're wasting your time? :rolleyes:

btkm8unsl0w5r4

Op check out this eview to see general performance http://www.tomshardware.co.uk/precision-t5600-xeon-e5-2687w-workstation,review-32805.html
I am sure such a big project would consider 10k for a machine small change. If they won't buy you a machine they shouldn't trust you to compile the code
If not the 4930k with quad channel memory is a real workhorse, I have one totally stable with all cores at 4.6ghz on a closed loop watercooler.

Gumbi

Sunny Fluffy Stockroom wrote: »

Op check out this eview to see general performance http://www.tomshardware.co.uk/precision-t5600-xeon-e5-2687w-workstation,review-32805.html
I am sure such a big project would consider 10k for a machine small change. If they won't buy you a machine they shouldn't trust you to compile the code
If not the 4930k with quad channel memory is a real workhorse, I have one totally stable with all cores at 4.6ghz on a closed loop watercooler.

May I ask what voltage?

KyussBishop

Sunny Fluffy Stockroom wrote: »

Op check out this eview to see general performance http://www.tomshardware.co.uk/precision-t5600-xeon-e5-2687w-workstation,review-32805.html
I am sure such a big project would consider 10k for a machine small change. If they won't buy you a machine they shouldn't trust you to compile the code
If not the 4930k with quad channel memory is a real workhorse, I have one totally stable with all cores at 4.6ghz on a closed loop watercooler.

That (4930k) is what I'll be going with for the moment, yes (almost identical setup actually); that should give me at least a 3-4x (5x if lucky, after overclock) performance boost over what I have now, so hopefully will work out well.

BloodBath

Maybe up the ram to 32gb. I don't know how memory intensive it is but it sounds like it might be. Maybe try and get some idea and go for more ram if you need it. You can get a different board that supports 64gb of quad channel memory either.

Could be total overkill but you will have to find that out.

KyussBishop

BloodBath wrote: »

Maybe up the ram to 32gb. I don't know how memory intensive it is but it sounds like it might be. Maybe try and get some idea and go for more ram if you need it. You can get a different board that supports 64gb of quad channel memory either.

Could be total overkill but you will have to find that out.

It shouldn't be all that memory intensive, the bottleneck definitely does seem to be CPU here (on my current setup, I never really eat far into the 8gb of memory I have).

I'm looking at similar specs as your suggestion but different components, and have a board which should support 2 quad sets (8 slots), so will just get 16gb, and (if I might have use for ramdisks or multiple VM's at some stage later - probably not) may replace that with a full 64gb 2-quad set if the memory gets much cheaper.

smacl

BloodBath wrote: »

Maybe up the ram to 32gb. I don't know how memory intensive it is but it sounds like it might be. Maybe try and get some idea and go for more ram if you need it. You can get a different board that supports 64gb of quad channel memory either.

Could be total overkill but you will have to find that out.

Given the cost of the rig, I'd go 32gb at least, probably 64gb for a bit of future proofing. Worth remembering that multicore compilation uses a separate instance of the compiler for each process, and each of these can consume quite a bit of memory if you've got complex headers in play. To keep all cores busy all the time, VS always has more compilation processes on the go than available cores. Having precompiled headers and object files on a ram disk speeds up link times over and above what you'd get with a fast SSD. My PC uses 6.8gb compiling with 8 cores, with no RAM disk and not that much else open. While I could get away with 16gb most of the time, 32gb give a bit of breathing space, and that's for a ~€1400 build.

The difference in price between 8GB and 32GB will be a few hundred euro, so on a PC costing 3k-4k it is a total no brainer.

btkm8unsl0w5r4

Gumbi wrote: »

May I ask what voltage?

Ok I have to admit I am a lazy bastard and I allow my mobo (asus formula 2011) to decide that for me. I dont think the voltage is raised, but the memory run a little slower to stay stable. The great thing about the asus mobo is that is still allow the speedstep to work, so its not always at full tilt. Temperatures are never above 70 degrees across all cores using prime 95 for a few hours.

Oh yeah OP there is a chap called blades, or bladesof glory selling a bunch of high end memory, CPU, GPUs, mobo etc. over on adverts

Gumbi

Sunny Fluffy Stockroom wrote: »

Ok I have to admit I am a lazy bastard and I allow my mobo (asus formula 2011) to decide that for me. I dont think the voltage is raised, but the memory run a little slower to stay stable. The great thing about the asus mobo is that is still allow the speedstep to work, so its not always at full tilt. Temperatures are never above 70 degrees across all cores using prime 95 for a few hours.

Oh yeah OP there is a chap called blades, or bladesof glory selling a bunch of high end memory, CPU, GPUs, mobo etc. over on adverts

I would check if I were you man, just in case it is over applying. You could well be applying 1.4v which is totally unnecessary.