Satya's blog - 2012/04/

In response to a post at http://37signals.com/svn/posts/3159-testing-like-the-tsa

Don't aim for 100% coverage.

Nope, please test as much as you can. It's ok to break out things like:

function goto(url) {
 window.location = url;
}

and not test the goto function, though. Also, don't test other people's code.

Code-to-test ratios above 1:2 is a smell, above 1:3 is a stink.

Perhaps. but what's the metric? I hope it's not LOC.

You're probably doing it wrong if testing is taking more than 1/3 of your time. You're definitely doing it wrong if it's taking up more than half.

I think we do this sometimes. If we can't imagine how to test something, I think that's a smell of testing other people's code, or testing too granularly. Perhaps an integration-style test, or a behavior-driven test, would work better. Do we really need to test that this method sets the right call stack and calls the right 45 methods in ActiveRecord? Or do we want to start with a CSV file and empty DB, run the method under test, and at the end have those records in the DB?

Don't test standard Active Record associations, validations, or scopes.

Don't test other people's code, unless incidentally in a behavior-driven test.

Reserve integration testing for issues arising from the integration of separate elements (aka don't integration test things that can be unit tested instead).

Sure, unless you're behavior-driving.

Don't use Cucumber unless you live in the magic kingdom of non-programmers-writing-tests (and send me a bottle of fairy dust if you're there!)

Cucumber is non-trivial.

Don't force yourself to test-first every controller, model, and view (my ratio is typically 20% test-first, 80% test-after).

Well, depends. If you test-first, you get better coverage and leaner code (code that does only what the test calls for). Behavior-driven development (BDD) is correlated with test-first and better coverage.

The 37signals blog then quoted Kent Beck:

"I get paid for code that works, not for tests, "

So true.

Last updated: Apr 14 2012 23:04

I work in an agile environment. We have daily standups, weekly iterations, no sprints (as far as I can tell or care). We churn through stories in our story tracker. We work towards a release marker. We try to do acceptance testing early and often. We have a separate and powerful (as in, they can reject stories and we must fix) QA department. We also have a separate Product Lead, and she determines the features. We front-load technical debt by having stories near the top of the backlog/queue for things like "there must be a QA server", "there must be a test harness page in the app".

We pair all day, though the pairs can be loose. We rotate pairs frequently, the nominal rate being once a day. Pairing station setup is dual keyboard/video/monitor with mirrored view. MacOSX, RubyMine. We happen to use Ruby, Padrino, some Rails, some Java, some shell. We're all polyglot, full-stack programmers comfortable between Ubuntu, Mac, command line, ruby, HTML, CSS, Javascript (though we use HAML and SCSS).

That's just background. We have, in this environment, used both Pivotal Labs' Pivotal Tracker and Atlassian's JIRA, in that order. We have tried to use the same methods while using both products, modified by the product's requirements, limitations, mind-sets. What follows is my own opinion and does not reflect the company or team's.

Tracker shows a unified, single view of all stories in a project. There are various queues, like current, backlog, icebox, but everyone sees the same view. Not so JIRA. We're still having arguments like "but I see this story thus," "oh but I see it this way, and oh bah...."

Tracker also forces the stories to be sorted one way. Users can drag and drop stories above and below each other, very easily, and we take that to mean the priority. Can't (easily) do that with JIRA (unless our's isn't set up to do that -- and we want it to be, so I wonder why not). In JIRA, you can drag stories to change their order -- in that view only. There's some kind of disconnect between Green Hopper and Rapid Board, apparently. An upgrade may close the gap. But, it's a source of annoyance and complication, confusion and delay.

Tracker's forced sorting is also useful in another way. I once used Gforge for issue tracking. The manager would set every gosh-darn bug to "top priority". That's a good way to get the developers to pick their own priority. What we really need to know is the urgency of the bugs (issues/stories) relative to themselves.

The good points about JIRA, which AFAIK PTracker can't do, is things like link issues together and group issues by team. Also, JIRA can link issues across projects. It has nice dashboards, too. This issue linking thing is great for Change Management tickets. CM tickets are linked to the issues that are being promoted to production, and before we announce the change, we make sure all associated tickets are complete, and we get a nice list of changes to go in the email or whatever.

Bottom line, I prefer Pivotal Tracker. JIRA is this enterprise-level thing (like most things about Java). It's a lumbering beast. Tracker feels agile, lean, and has been the only issue tracker that didn't get in my way.