Deployment dilemma

pushpop

Hi there

I started working for a very young and small software company a few weeks ago, where the product is a web-application. I got a panicked email from my boss last week saying that someone had checked in code which broke the site and he had to revert the live site, and said that developers should branch on Git for all untested code and merge when it's complete.

We currently don't have any QA staff so the developers themselves are verifying their code, which isn't ideal for obvious reasons, and there is no intermediate testing going on between the time the code is committed to Git, and the time it's deployed.

I think this is just a teething problem with a growing company that hasn't nailed down some necessary processes yet, but it flagged concerns with me because he (a) didn't communicate he was deploying new code to the other developers (b) he's deploying code that has only been tested locally on developer machines and hasn't been tested on a staging-server (c) branching for all new code is an inversion of my understanding of the traditional SCM workflow, where you only branch for releases and major changes.

I'm new to this server-side stuff so I'm finding my bearings too, so I don't claim to be an expert by any means, but it seems to me that the deployment strategy is far too casual and passive at the moment, and that formal QA and deployment processes need to be implemented.

I'm hoping to implement SCRUM as the dev process in the company from this week on, and we currently don't have a bug tracker or project management CRM so I'm hoping to have one in place this week.

I'm wondering though what other server-side devs who use this site do when deploying. Any suggestions would be appreciated.

srsly78

Automated deployment via Jenkins or whatever - use Selenium or other tool for test automation. Use VMware scripting to dynamically create/destroy test servers as needed.

Start with simple sanity tests (deploy code, try to start server up) - improve by adding more complex tests as you go along.

I happen to know one consultant that is currently available to set all this up for you

Procasinator

pushpop wrote: »

(c) branching for all new code is an inversion of my understanding of the traditional SCM workflow, where you only branch for releases and major changes.

I don't agree with your here. First off, you are not using traditional SCM workflow, you are using a DVCS workflow.

Gits power lies in branching often, and easy merges.

Your master branch (or equivalent) should always be stable.

philologos

pushpop wrote: »

Hi there

I started working for a very young and small software company a few weeks ago, where the product is a web-application. I got a panicked email from my boss last week saying that someone had checked in code which broke the site and he had to revert the live site, and said that developers should branch on Git for all untested code and merge when it's complete.

We currently don't have any QA staff so the developers themselves are verifying their code, which isn't ideal for obvious reasons, and there is no intermediate testing going on between the time the code is committed to Git, and the time it's deployed.

I think this is just a teething problem with a growing company that hasn't nailed down some necessary processes yet, but it flagged concerns with me because he (a) didn't communicate he was deploying new code to the other developers (b) he's deploying code that has only been tested locally on developer machines and hasn't been tested on a staging-server (c) branching for all new code is an inversion of my understanding of the traditional SCM workflow, where you only branch for releases and major changes.

I'm new to this server-side stuff so I'm finding my bearings too, so I don't claim to be an expert by any means, but it seems to me that the deployment strategy is far too casual and passive at the moment, and that formal QA and deployment processes need to be implemented.

I'm hoping to implement SCRUM as the dev process in the company from this week on, and we currently don't have a bug tracker or project management CRM so I'm hoping to have one in place this week.

I'm wondering though what other server-side devs who use this site do when deploying. Any suggestions would be appreciated.

Get JetBrains TeamCity if possible. Its a build server that takes a personal build and only if that build is successful and no unit tests fail is it committed to SVN and then autodeployed to our IIS server. We use ASP.NET but it is compatible with lots of languages and runs on Linux and Windows.

It works by uploading the code to the build server before commit. So if people use it, no broken repository and no branching required.

There are client plugins for Visual Studio and Eclipse and other IDEs.

Once you set up your build environment correctly these errors will go away.

Sparks

Saw that kind of environment first hand a few years back. Quick questions - do you actually have a test server to deploy to, and do you have a completely automated deployment process yet that deploys to both test and production (and I mean *completely* automated - manual steps are always forgotten sooner or later)?

Also, scrum's nice when it works, but it won't fix a problem like this because your problem's not that you have too top-heavy a development process, it's that you don't have a deployment process.

Ideally, you want to deploy to test; run automated tests against the test server (there's a lot of software that will test web apps out there and setting up automated tests should be easy enough); and then deploy from test to production. A developer should not be able to deploy from their machine directly to production, and you should limit the number of people who can deploy to production to as low as you can, though not to just one or a sick day would cause problems (yes, I know, ideally the dog should be able to deploy to production for an Agile/Scrum/whatever development model, but if you've got problems because you're deploying broken code to production and you're worrying about Agile, then you really are trying to sprint before you can walk. Lock it down first, then open it up later when the test metrics say the problem is resolved).

pushpop

Thanks for the responses guys. I suppose it is early days in the company and we have to make the best of what we have with our limited resources. Definitely going to look into Selenium and Jenkins for continuous integration anyway, and put a deployment process in place. As well, we've agreed that we will branch for new code and merge into the trunk when the branched code is stable.

Cheers

jester77

First off you need a CI server, jenkins is popular, free and has great plugins. You can use git hooks to launch your build once someone has commited to whatever branch. Do you have unit tests? If not, then you should.

It would be an idea to have just one person responsible for build/deployment.

Ye seem a bit all over the place with how git should work. Have a look at git-flow, you don't have to use it but it will give you an idea of what is good practice.

Selenium is great for automated tests.

Trojan

Here's a setup in my previous employers:

Development local gate -> Local test gate with build server setup -> Team test gate with build server setup (optional) -> main gate on main build server.

Run nightly tests on your main build server (or a clone, if it's too busy), and always have a single-command build setup, and preferably single-command test setup.

For 'putback' to the main gate you must pull the latest code from the server, apply your deltas, and then run the tests. Full automated cdiff and sdiffs generated to show during a local code review from peer, plus a code review from a high level engineer in another group (one of ~20). Code review immediately failed if test results not available or if your delta shows you built on an older version of the base code - you re-sync, then go again.

Whoever breaks the main build server babysits it until the next breakage occurs - in our case, that wasn't allowed, but you did get called out by name if you broke it, do it twice and you got the ignominy of having to ask a colleague to do your putbacks until the build engineers decided you could go again.

These days I just edit live websites with vi via ssh.

Procasinator

Trojan wrote: »

These days I just edit live websites with vi via ssh.

What a great build up to that line. :P

Trojan

I had also considered "via telnet"

Sparks

Telnet? You young folk, don't know you're born. I remember Glass Teletype on an OPD at 300 baud, and that was considered new-fangled at the time...

Sparks

/goes off to google image search a morse code key in response to Trojan's predicted response