How do you measure coverage when creating tests

grahamor

Hi, we are using some Selenium based tools to write UI regression tests

We are looking for a way to define all our tests and use this definition as a list to calculate the coverage amount.

The list is currently just a spreadsheet and items are being added based on their priority.

I'm interested in hearing how small dev teams have incorporated UI regression testing and how coverage is measured.

Thanks for any info

srsly78

You use a coverage tool - because who can be bothered writing **** into a spreadsheet? This typically involves running instrumented tests and then analysing the output. Will give you a report saying X% of code covered.

For c++ on linux "gcov" is an example of such a tool. Depending on your platform and toolchain there are other tools that may be applicable.

14ned

srsly78 wrote: »

You use a coverage tool - because who can be bothered writing **** into a spreadsheet? This typically involves running instrumented tests and then analysing the output. Will give you a report saying X% of code covered.

For c++ on linux "gcov" is an example of such a tool. Depending on your platform and toolchain there are other tools that may be applicable.

He's looking for UI test coverage, not code line coverage.

Niall

Paranoid Bob

Google this:
"What is Testivus' wisdom concerning the proper percentage of test coverage?"

(I can't post a link).

TD;DR:
Test coverage numbers are only useful to managers who don't understand testing.


More generally: Best practice would be to make your tests self-documenting. Each test should describe itself, with a reference to the requirement it is testing. Always call back to requirements (if you don't have formal requirements then just describe the expected behavior clearly).
Use a spreadsheet (or any other tool you like) to build your to-do list of tests yet to be implemented, but once the tests are implemented then running the test suite should give you all the information you need.

14ned

Paranoid Bob wrote: »

More generally: Best practice would be to make your tests self-documenting. Each test should describe itself, with a reference to the requirement it is testing. Always call back to requirements (if you don't have formal requirements then just describe the expected behavior clearly).
Use a spreadsheet (or any other tool you like) to build your to-do list of tests yet to be implemented, but once the tests are implemented then running the test suite should give you all the information you need.

There appears to be a profound lack of understanding as to what the OP is asking for (fair enough, his chosen title is awful).

He'll likely have some website made up of various web controls. Users, or bots, will traverse said site punching and selecting various things, and he wants a UI regression test framework which will punch and select various things which a continuous integration server can run on any modifications before those modifications are pushed to live users.

He's written them up in Selenium, which even I as a non web developer have heard of and would probably suggest as a good default. He's asking how far do you take the testing. Unlike writing test code, these tests are NOT self documenting. They are like "click here, tick this, hit this button, tick this other thing, hit another button". It's basically an acyclic graph of events. What that means is meaningless to anyone reading it, indeed if somebody modifies the HTML then the test could mean something completely different.

To try actually answering the OP, and I really must stress I am not a web developer nor have web development experience, but what I'd do is create a dummy back end which records the AJAX and other HTTP POST sent to me. I'd then have Selenium run every permutation of every possible operation on some web page, and record all the things which are supposed to go to the server. I have now got a "fingerprint" of the existing website.

When somebody makes a change, you run all the permutations again and get a new fingerprint, and force a human to reconcile the two before the changes get pushed live. If a more real backend were needed, you could snapshot a VM of the backend services and reset it to the snapshot before each regression test so it plays out exactly right each time.

Now I've no idea if the above would work, I am not a web developer. But it's the best I can think of which could be called "complete UI regression test coverage". No idea if it's practical, or if anybody even does that. Let's hope someone knowledge of this stuff chimes in next.

Niall

counterpointaud

Yeah there are tools in Javascript land to do this using various methods. Some using instrumentation to check LoC covered (e.g. https://www.npmjs.com/package/grunt-protractor-coverage) some use elements interacted with as a metric (e.g. https://www.npmjs.com/package/protractor-e2e-coverage)

Also I think JSCover has some way of injecting JS to check coverage. See this blog post: https://innovatorsdelight.wordpress.com/2014/05/05/collecting-javascript-code-coverage-during-a-selenium-automation-test-run/

I've never used any of these, and honestly I think it's a bit nuts to do it that way.

For UI tests I prefer covering features, not LoC or elements / components. Keeping track what is covered and what isn't inside of whatever your project management tools is fine IMO.

biko

I think you are already doing it well. An excel sheet with numbered UI test cases will give you an overview.

This is part of a writeup I did for a company recently
Automation testing best practices
* Don't assume, get everything clarified on paper. Steps, browsers, OS etc Do POC's
* Automate production critical features first. Anything with high customer impact like login. Look for and focus on most visited areas of the sites
* Automate low hanging fruit, get quick wins
* Automate error handling verification, this is often overlooked.
* Many small scripts with quick feedback is better than a test suit that runs 5 hours before feedback

Atomic Pineapple

14ned wrote: »

He's asking how far do you take the testing. Unlike writing test code, these tests are NOT self documenting.

Cucumber can help document, while not exactly self documenting we have found it a great bridge between Product, Developers & QA - https://cucumber.io/