Early Excellence in Failure

There is a point at which I hope to do research which doesn’t fail. There’s probably a way to write code that would tell me when that point might be. Of course, I don’t code. I fail at that too.

At this point I can see a giant set of sparkling white self-help teeth spouting horrible affirmations about our failures being the first sunbeams of success (damn late night TV).  Not quite hitting the expected mark in research is clearly very much part of the standard story, even though it feels as graceful as this …

I remember feeling like this for about  a year. [via hdwpapers.com]

I remember feeling like this for about a year. [via hdwpapers.com]

I was part of a research team that undertook a failed bit of research. When I look back at it, it reeks of inexperience as a researcher. And I have to confess that the failure was a ‘valuable learning process’ . Yes, that hurt as much to write as it did to read. Of course, even research that looks implausible isn’t really a failure as there’s value to be found everywhere. Possibly though, before sharing the myriad ways we got it wrong a little background might help.

The Project

The trigger for the whole thing was a worthy question about burns care. In anaesthesia we focus a lot on managing symptoms. When most people think burns, they think of the pain. Which is fair enough because it’s a big issue but we have an arsenal of pain management options. The other really annoying question is what to do about the itch.

Somewhere north of 80% of people with sizeable burns (which we’ll define as more than 10% of their body surface area) will report itch at some point and it’s often severe. A variety of cells involved in the healing process tend to release substances which, while important to the healing, stimulate pathways in the nervous system that trigger that annoying bug-crawling feeling we’ve all cursed.

The itch caused by burns often seems to turn up a week or so down the track as healing gets into swing. Not only is it an issue because feeling like you want to scratch your own insides out with nothing more than a sweet potato. The itch is classically more disturbing at night (and if you’ve ever tried to sleep in a hospital, you’ll understand distractions aren’t welcome) and causes lots of distress. It can even lead to failure of wound healing or disruption of skin grafting as patients scrabble away at their dressings.

In addition to all of that, a lot of available early treatments are as effective as an old cattle dog trying to catch flies. So we set out to see if we could make something different work – a drug originally introduced to help with seizures, gabapentin, also helps with pain relief and in some cases itch. Having decided to try and establish its role for burns itch a little better we figured that even better than stopping itch would be preventing it ever happening. So we embarked upon a study to test it out. Just a little double-blind randomised controlled trial to test it all. That would be easy, right? Actually that’s where our problems started and we ended with incomplete data that was really hard to work with. Each one of our missteps has informed subsequent work though with some really obvious big themes for clinical research.

1. Know your area

A wide-ranging group of clinicians were involved in setting the trial up. They provided lots of useful input on things like how severe itch was, how many kids put up with it and crucially, how many eligible patients we’d have. Except those experienced clinicians were wrong. In particular, the ones saying how many kids were admitted on a month to month basis were wrong by a factor of ten. ‘Feelings’ about an area are no help where data is required. Which leads on to …

2. Make no assumptions

Rose-hued anecdotes are no substitute for numbers when trying to plan. To get onto the heart of the project, have actual numbers or pilot work to inform every bit of the later planning. The only acceptable assumption is that your gut is wrong. This also leads to …

3. Question everything and everyone  …

Accepting no assumptions means not allowing any of the little things through without consideration. You may end up having to accept a consensus view on an item of the plan, but do this with eyes open, not through an oversight. The follow-up is that you can’t assume people you work with are clever at this bit of research just because they’re clever at other things or other research. One of our fatal assumptions was that those with a bit more experience knew the numbers accurately or, in some cases, knew how to make this research work best. Some of the time they were working off a bit of first principle logic themselves.

Even as a junior researcher, question them. Not in an  “I want the truth!”/”You can’t handle the truth!” style, but to test their reasoning and clarify the planning. There’s only two outcomes from those discussions: you highlight a potential flaw or you learn from the beauty of their explanation.

4. Resources

Research needs actual resources not an assumption that your effort, enthusiasm and goodwill can get you there. All that stuff will wane. Other realities of life will impose themselves. The effort at the start doesn’t allow autopilot later. There is only continuous work so plan it like that. This also relates to …

5. Love isn’t universal

No one will love your project like you do. You won’t even love your project like you do when you start. If you set up a system (like we did) where a variety of people will be involved in the trial assume that all of them will have, at best, a passing interest in the work. One assumption is fair – they will probably devote as much brain space focussing on it as a night security guard with the TV remote in hand.

As a group, we learnt lots from undertaking the study. We ended up with better resourcing for research generally. It still didn’t poison us against future work and I’m now involved in work where we searched hard for actual numbers to inform our planning and we’ve spent way longer asking questions, preparing the logistics and working on having enough people continuously devoted to it.

Here’s hoping this latest project looks a fair bit more elegant. Or at least provides more refined failures.

I bet the second time they did a Tupac unicorn tattoo, it was awesome. [via the4sp.com]

I bet the second time they did a Tupac unicorn tattoo, it was awesome. [via the4sp.com]


The Flying PhD Guide to Grant Failure

If the life lessons learned from reality TV mean anything, and they’d better given how many use these shows as life guides, then through the fire of failure comes redemption and meaningful success. The narrative of the hero being tortured by a poorly risen soufflé or undersheened macaroon only to earn their place in successful society with a superior pannacotta comes from a dog-eared script. Even this reality TV star failed last time around.

In the attempt to gain grant funding for this research, I’ve put together 6 grant applications so far. 3 have been decided and we’re a perfect 0 for 3. I guess that means I am approaching reality TV promised land. Given this good fortune, I feel a certain sense of duty to share my secrets of glorious grant failure so that all may enjoy the experience. Of course, none of the funding groups has provided feedback, so all I can share is what I’ve done. Here are the tips that seem to have worked for me in this personal crusade to reach the discard pile.

The model sun = the total funding pool. The planets = the chance of  success for a researcher. The planet at the left would be someone with more chance than the writer. [via bobs-spaces-net]

The model sun = the total funding pool. The planets = the chance of success for a researcher. The planet at the left would be someone with more chance than the writer. [via bobs-spaces-net]

1. Be the Work Experience Kid

The last thing you want to be if you’re going to fail is experienced at research. Researchers who are successful with grant applications tend to be beyond the early researcher stage. In fact, the average age of first success in schemes like the NIH have been climbing for a while now. So first step, apply when you have pretty much zero games in the league. The more you’re like the annoying research work experience kid still fetching misplaced lunch orders the better.

2. Do Other Stuff For A While

It probably looks really good when you’ve demonstrated steady progress through a field of scientific endeavour. That might jeopardise your chance of failure, so what you really want to do is spend about a decade getting other qualifications for non-research related work. Doing a medical specialty for example. It’s probably just as much of an issue for those who take time out to have kids or look after other people. It may be just one of the challenges that has led to success rates for women making applications being lower (which warrants a much larger discussion). While processes are there to allow mitigation for gaps, it’s unclear to me how well they’re working.

3. Work in a Little Field

Everyone wants to fund the cure for cancer. Trying for a grant in a relatively niche area, like prehospital health care, involving things that sound not that serious, like helicopters, might just bump up your chance of failure. Or maybe choose an obscure genetic disease with a vanishingly low incidence to focus on. That would probably do it too. The more the field is in a niche, the less likely the reviewer will be able to understand the full scope of what you’re doing.

4. Be a Novice

Along the same lines as point 2, you don’t want to come with a track record. Early career researchers have success rates of less than 20% (take the ARC Discovery Early Career Researcher Awards as an example). You can drive this lower. Having other grants already in play and lots of related publications is the stuff of pros and might jeopardise your chances of cathartic failure. From personal experience, a total of 5 publications and no grants is a good starting point.

5. Be a Bit Left Field Too

Being in a little field might not be enough. Trying to look at something that doesn’t have lots of preceding literature, like noninvasive prehospital brain monitoring, might just give the impression that it’s an area no one else wants to look at. Extra implausibility points always help.

6. Work Outside the Established System

Obviously you’re probably going to have some sort of linkage to a Uni or established research group. If you can nest your research within an organisation that doesn’t fit that description, your lack of reputation is only reinforced. If you can make it a charity where previous research has been funded successfully through alternate means leading to no grant history, this is an extra bonus. That way, not only is your chance of failure enhanced, you get to feel that you risk bankrupting the charity*. The self-flagellation adds to the bitterness. Excellent.

7. Be a Part-Timer

Knowing that a researcher is going to work full time on the project you’re funding must make it easier to back them. Being a part-timer plants the useful idea in the reviewers’ head that you’re actually some sort of uncommitted wheatgrass-chugging hippie with other agendas that may distract. It’s an obvious differentiation point.

7. Actually,  Just be a Researcher

The brutal truth is none of the above may be the issue. Grant monies have a limit, and really good research teams miss out. When they miss the big national competitive grants, they have to hit the smaller Foundation grants. This only enhances the failure rate of the novice researcher flaunting all the attributes above. Even all the measures taken to try and address any imbalances can’t entirely make up for the high number of applications vs the actual number funded. It’s not that reviewers are trying to thwart good research. They’re just stuck with reality and the need to differentiate. The landscape isn’t likely to change without substantial increases in funding either.

With overall success rates of about 20% for major grant projects, research funding applications are going to involve failure at least some of the time. This is actually one of the things I’ve already learnt from the PhD. Having said that, even saying that you can learn from the failure is a sentiment with more mole whiskers than a hormonal bridge troll. I’ve no doubt there’s some really excellent researchers who get knocked back just enough to necessitate vacating the field of endeavour. The question is how many times can perseverance overcome the setback?

At the moment, it certainly feeds the Impostor Syndrome. Perhaps I should take it as positive feedback on that front – “We write to take the chance to validate your thinking … we also don’t think you belong here.”  At least I’ll be super-resilient if I burn the toast.

* To be clear, my charity has survived for more than 25 years. They’re way too bright to actually let me jeopardise anything and are extremely clever at making things happen. It’s one of the reasons I love working for them.

Failure Fetish and Research

There is  a pernicious evil that takes over at night. It gnaws at the brains of innocent people innocently filling up the witching hours. It is the infomercial. Most annoyingly, the hours now spread out so that at almost any time an infomercial can be found seeking to sell the latest hair removal  product to the nearly hairless.

The version I hate the most is the one peddling a self-help solution. The one with a scripted interview between someone who was almost a journalist at some point and a set of teeth speaking on behalf of a haircut attached to a person. The one where they stitch together various insights gleaned from the fortune cookies at a particularly lavish yum cha and package them into a motivational “life rejuvenation handbook”.

A Confession

Embarrassingly I caught one of these briefly the other night where the interviewer was asking the teeth about failure. The glib lines started spilling forth – “Failure is your greatest opportunity!” “The only failure is in not failing enough!” [Insert appropriate Sideshow Bob grumble-mutter right here].

I’ve seen this a bit when reading around tips and insights for research. Not the really extreme version, but still a healthy amount of “embracing failure” self-help infested advice. My problem is that occasionally this extends to the point where failure is celebrated as the key achievement one can aspire to.  There is a world of difference between accepting that failure is this thing that happens sometimes and teaches a lesson or two and making a fetish of it. And lest it be forgotten, there are occasions where it’s not that clear if the big fail has really driven a key life lesson home.

Not so visible to the wearer that they'd actually get it, of course. [Picked up via Buzzfeed]

Not so visible to the wearer that they’d actually get it, of course. [Picked up via Buzzfeed]

Too often this nuance is lost although this spot gets it, if not the header, around the mark.

Shiny New Failure

The other thing that brought on this failure thinking was my latest personal fail. This time it is on the grant front. I’ve just been through the immense joy of providing responses to some foundation grant reviewers. I’m not necessarily calling the grant response a fail but the tenor of the comments from the reviewers of the original project was not encouraging.

Were I to paraphrase the responses they would look like:

Reviewer 1: This is quite a big project. Do you really think you’re the person for the job?

Reviewer 2: This is an interesting project that should be contemplated, but in that contemplation I would do every single element of it differently.

(Sidenote: I’ve heard the word “interesting” used way too much in hospitals to disguise genuine thoughts to take it at face value – like when prospective parents indicate the name of their child will be “Abserdee” spelt “A-B-C-D-E” and the midwife responds by describing this as “Interesting”.)

At that point, there was a part of me thinking any rebuttal was likely to be a waste of time. Then I sat down and wrote a response. That response then went to my co-investigators. Some of those co-investigators have had serious grant success before, and have the sort of background that makes me wonder why they speak to me at all instead of getting on with revolutionary stuff. Those guys then gave me a 3 day masterclass in actually writing a proper reply. So I might have kept just this side of making failure useful.

At the end of this, I still feel a little like this guy …

This aardvark invites open scrutiny of their latest project [via flickr]

This critter invites open scrutiny of a current project [via flickr]

but I guess I’m hoping that I’ve learnt lessons that will help me out when I get to the national competitive grants and get to take on the formidable odds against success with the NHMRC cycle (see an earlier bit on this here). Or I guess I could get some dental work and hit an infomercial channel.