Working with the Company

For a brief time as a youngster, I took gymnastics quite seriously. Serious enough to be able to pull off a range of contortions and tricks and even know the actual names of them in the days before Roy and HG. Back then I thought that all the really tough balancing acts involved an apparatus. It was only some time after a request to give up my Sundays had terminated my acrobatic interests that I figured out there are far more challenging balancing acts.

Big Industries and Research Bucks

In an environment where research funds generally are hard to find, navigating the relationship with companies that sell stuff is tricky. As I’m from a medical background my instinct is to run a mile from any rep selling as much as nostril tweezers. This is because I know there’s been a long term problem with companies, particularly the ones that make various colours of pill, unduly influencing medicos with flattery and gifts. This has finally led to widespread campaigns to stop the perpetual Kris Kringle act from companies and some responses to curb unethical conduct.

I also know doctors are well trained to spot the tiniest speck of dust in someone’s eye but correspondingly terrible at noticing timber plantations wedged deep beyond their own iris. It’s described nicely in relation to advertising influence in this review – ‘Other people must be – otherwise advertisers wouldn’t bother – but I’m not’. Bizarrely, doctors seem to struggle with the idea that we’re all pretty much suckers.

In clinical research land, doctors do tend to downgrade the quality of industry-linked research. There has also been recent investigations regarding conflicts of interest that have raised the profile of what is a serious problem (although in the most prominent recent case the NEJM suggested there was more smoke than fire at their end, even if the involved company settled for $40 million).

So for the researcher the answer should be simple – when you see a company rep offering a sweet deal, pull out your handy cross and string of garlic cloves and back away, right?

This guy, Dean Potter, also has incentives to get the balance right [via redbull.com]

This guy, Dean Potter, also has incentives to get the balance right [via redbull.com]

The Case for Big Business

The problem with a “destroy all industry approaches” setting on your phaser is that research can’t happen without those companies being part of the picture. In Australia in particular, development of good basic ideas (research) into things that make a difference (development)  is a major stumbling block (or as so well phrased here by Bill Ferris who served on the McKeon review, “Australia has internationally competitive ‘R’ and bugger all ‘D'”).

The McKeon review spent some time examining this issue and specifically pointed out the need to improve on commercialisation of research (and the delivery of research findings back into patient care). Looking at the report now (and here’s the summary version, where you may want to head to page 41) is like glimpsing a blurry alternate reality that will never come to pass (insert your own movie reference here). There is no evidence to support the idea that these recommendations will actually become policy, but the message that researchers need to engage more with industry isn’t going away.

How to strike a balance?

For this PhD project, we were confronted with this problem quite acutely. We’re trying to conduct research with a commercially available monitoring technology, Near Infrared Spectroscopy. We’re not just testing whether its useful with patients with injuries (particularly head injuries) we’re testing whether it can be used in patients with those injuries in the rough and tumble world outside the hospital. Which means trying to figure out which monitor actually fits our purpose most accurately.

To do this requires a bit of vetting of what’s being sold in the shops. Which means lots of dealing with companies and reps. We can’t feasibly do the research without a monitor, but we also need to be able to produce credible research.

So how did we do it?

A Non-Exhaustive List of Ways to Do Research with Companies

1. Check the Facts

Companies are in the business of selling stuff. They tend to present things in a way that will be best for them. That’s their job. Complaining about it is a bit like complaining that large, hairy spiders are too good at being large and hairy. It’s the job of the clinicians and researcher to check their work. Examples? The easiest one with NIRS oximetry is that companies tend to let you get away with certain inferences and assumptions.

Most of the companies making these things on the market have applied the principles in a slightly different way to produce a number on a scale that looks about equivalent. They don’t stress that there are always a few assumptions in generating the magic number. They will tell you that their technique is better. They don’t tend to point out that it’s hard to transfer every study done in NIRS to their individual monitor seeing as they use a different technique.

All the usual ploys are at play. The team that make this monitor have references to reductions in strokes using the technology without really specifying it’s from a study using another device or in which patients. The guys making this monitor probably rely on the fact you’ll never actually check their validation data which is performed against an older generation monitor with not many samples.

That’s just them doing their job. It’s the job of the research team to really check out what stacks up and understand the limitations.

2. Avoid the money

Companies know money makes research happen and are likely to offer to provide support. Pretty enticing when you’re trying to figure out how to make the dollars add up. Also a little self-defeating as anything produced in the early stages is likely to be tainted. So paying for what you get like an everyday purchase is one way to approach this and that’s what we did. It’s a decision that will cost lots of money now but the value of it will come in later.

3. Get things in writing

We paid for a version of the monitor that conceals the monitoring values at the time the monitor is on (as we know that clinicians wouldn’t be able to resist trying to look at the number and changing their care). In paying for this, we did provide guarantees that we wouldn’t try to figure out any underlying intellectual property belonging to the company. In return, we have a signed guarantee that the company will not have access to any trial data. We’ve also specified that the company cannot seek to limit publication of anything related to the study (and we specifically covered publishing unfavourable results). Once a company owns the data, objectivity can’t be assumed.

4. Disclose everything

Lack of transparency only goes one way – badly. Declare every possible relationship. Disclose that one time you smelt the smell of ink from a drug company pen. Just disclose. The last thing you want to look like if you have nothing to hide is the researcher hiding something.

5. Think about the later stages

At some point, we will have to think about engaging with some company. Our end goal (assuming we show there is a potential clinical role for the monitoring) is to have a monitor that displays useful information so clinicians can provide better patient care. Eventually that will require people who build monitors. How to plan for that is already part of our thinking.

 

That last one needs more thought. Nor is this list an answer for every situation or a suggestion that all research emerging from companies must automatically be mistrusted. It should be examined closely though. And for the individual researcher, the trip should be way less bumpy if the balance is right.

 

PS: Here’s the disclosure

We purchased a version of the Nonin EQUANOX monitor that conceals the monitor values. We paid full price for the monitor, cables required, the computers we use for the processing and every disposable sensor that gets attached to the patient. We have an agreement that we can publish anything we find, including results the technology doesn’t work in this setting and isn’t helpful, and the company will not seek to influence that in any way. The agreement covers the possibility that new intellectual property will be generated, and that intellectual property will belong to those independently assessed as having generated that new bit of stuff. We have not, and will not, accept any funding for conferences, education or presentation from the company that makes that monitor. We have no commercial agreements with that company.

 

 

 

Advertisements

Research and a New Big Brother

Finally. The sausages are done. No more putting up with partisan barrackers. An end to putting up with the petty point scoring and amateur analysis. You don’t even have to bite your tongue at that one nut who always goes over the top. Things can go back to normal. Yep, the kids’ footy season is over.

In the meantime, there was this election thing. As with any change people survey the landscape and try to figure out what it means. For researchers that started days ago, with the announcement that the incoming government will look to cut $103 million by ditching “ridiculous research grants”. They then pulled the standard trick of quoting a couple of titles in the humanities without context to make the point. The culture wars never start with guns, but with mockery.

The research community was quick to jump. This is perhaps because they know the sort of malarkey that has gone in the US (covered earlier here) and because the last time a government got talking like this, things got ugly (more on that below). At the heart of this are a few assumptions:

  • Peer review isn’t rigorous enough.
  • There is more value in applied research than all other types.
  • Politicians and community reviewers know better.

He's probably just misunderstood, right? [via mriddett.com]

He’s probably just misunderstood, right? [via mriddett.com]

Slapdash Peer Review

Peer review is not a boozed up work BBQ where old mates share out the funding 6-pack. It’s a rigorous, very drawn out process (check an outline here). The application gets put together over hundreds and hundreds of hours. It then meets a whole group of independent reviewers who are subject matter experts in the same field. There’s collation of the reviewers’ feedback and the chance for the applicants to swear a lot when they get that feedback. They get to respond and say as politely as possible “get stuffed, this is actually seriously excellent and you just need a few more facts”.  Then there’s final reviews and for the successful applicants the decisions about how much will actually be provided.

After all that, the ARC only grants about 20% of applicants any money. That’s not because only 20% were actually decent, it’s because there’s a limit on the money (the projects are generally part-funded) and other highly rated projects miss out. It’s so rigorous that effectively hundreds of years are expended by researchers on applications that won’t be successful.

Peer review weeds out the crap. Is it absolutely perfect? Of course not. But the assertion that there are large numbers of wasteful, irrelevant projects being waved through by an independent body has no basis in fact.

I’d wager that the pollies could do with peer review themselves. If all pollies’ thought bubbles had to go through this process you can guarantee there’d be a whole lot less rubbish getting out there. Like special plans for northern Australia or buying boats from Indonesian fishermen to prevent people smuggling. Or my favourite, the internet filtering policy developed by up to 9 MPs, screened by HQ, checked by the leader the night before then shown to the responsible minister with a little time to digest it. The policy was junk and was abandoned 3 hours later.

Even when they peer review, they don’t know how to peer review. But they reckon the ARC process is rife with problems.

Don’t Argue With Death

Thrown in as a bonus is the idea that there is medical research, and then all the other less valuable frippery.

For starters, what dolt thinks medical research happens without other research? It’s a creationists’ world view. Medical research doesn’t suddenly appear like a precious flower in a featureless wasteland.  It relies on many years of industrious activity from those working on basic science. Applied science exists because basic research gave it a shot. Argue against devoting funds to medical research though, and risk getting accused of being OK with people missing out on the next big cure.

The group at real risk is the humanities. So often seen as “not contributing”, the exploration of “non-science” areas is no less valid a way to explore and understand our world (I defer to the excellent Patrick Stokes here). So what’s the point of pitting research communities against each other? It can only be to try and divide the voices of those who might speak up to protect an independent process.

The Selection Panel

But maybe those voicing concern are just being paranoid? Well, not that long ago a Coalition government started applying the “has to please the gasfitter” rule and projects approved by peer review were vetoed. The list hasn’t been confirmed but is thought to have included projects such as a review of how politicians had used the media in various scandals. Of course, no one can understand why modern use of media might be of interest, right?

Until I see everyone suggesting a community cabinet should help veto Australian cricket team picks, where there’ s a more obvious need for auditing, I’ll be unconvinced there’s merit to the idea.

There’s a longer piece by Gideon Haigh covering the period in plenty of depth. It does a good job of recording the charged history that unfolded (and includes some of the areas where the ARC process isn’t perfect). The fear is that once decisions are vetoed because they please or displease political masters, the tenor of all applications becomes tainted. Nobel laureate Peter Doherty is quoted in the article decrying the risk of shutting down debate:

“What these people are saying is that certain things shouldn’t be looked at … and I don’t believe that at all. In a free society, we should be able to look at anything. A free society should welcome debate – which is one of the very depressing things about this society, that it’s trying to close debate down, which is always a sign of mediocrity, and mediocrity at the highest political level. What happens next?”

Here’s what has to happen next – the entire research community has to be ready for the detail and speak up. Anyone who thinks they can afford not to be a communicator had better think twice. The indications are it will be on like Donkey Kong. Is everyone ready to raise their voice?

Where are the deep pockets?

There was a period of time where I only used the name Bill Gates thus: “F#@$ing Bill Gates! Bloody Microsoft hunk of junk!” The accompanying visual was most likely to be the blue screen of death.

These days I have entirely different feelings about him. I’m likely to be singing his praises when I’m hanging in a nursing home. Through his new career in philanthropy (along with the very impressive Melinda Gates), he has proven that he can use his ridiculous level of wealth and formidable mind to make a real impact in world health, education and development. They’ve already achieved amazing things through the Bill and Melinda Gates Foundation. Chances are, when we solve problems like malaria, the Gates Foundation will have been a big part of that story.

So, why the love letter? Well, the world’s richest geek was on Q and A this evening (the video isn’t up at the ABC iView site yet, but I don’t doubt you’ll find it soon). Beyond having a very high word count on “research”, as he is clearly of the view that research is the key to progress, there’s a couple of things about his philanthropy that are interesting.

The first is the level of engagement with the subject. It is very evident that he is not just throwing money from Olympus. He is seriously engaged with the details of what they’re doing, and the difference they hope to make. Seeing a multibillionaire who clearly has some concept of the challenges facing the broader world, or the potential benefits of addressing global health and equality issues was more refreshing than a faux Scandinavian deodorant ad. It also provided a chance to reflect on all the Australian überwealthy similarly engaging with the challenges facing the world, beyond just reminding us that African miners want to work for $2 a day. Yep, just think of all of them at a big party (and feel free to grab that spiky tumbleweed rolling on by to use for an acupuncture session and release the negative energy that’s building up).

The other interesting thing was the foundation’s readiness to consider “blue sky” ideas – bolt from the blue thoughts that might seem outlandish but just maybe have a chance of making a difference. Precisely the sorts of ideas that get suffocated by the oppressive national competitive grants system.

I can’t help but reflect again on the need for novel ideas and opportunities to fund the research that will drive the development of cultures everywhere. In an environment where government-funded options are ever more competitive, and ever more pressured by political realities, what other options are out there?

Could direct philanthropy or charity organisations provide the way forward? Well, there are some groups out there supporting research (the Brain Foundation, the Ramaciotti Foundation etc). I can’t think of any particular big ticket philanthropists engaging with research in a similar fashion though.  The other challenge for not for profit organisations is the way we think about charities and how they should raise and spend their money (and an extra disclosure here, I spend some of my time working at CareFlight, a registered charity which devotes some of the money it raises to research).

That issue is a complex one itself, but I can’t do better than refer anyone who has a spare 20 minutes to watch this awesome TED talk from Dan Pallotta (seriously, clear 20 minutes) in which he very clearly states exactly the obstacles we put in the way of charities growing and carrying out their goals. Most particularly, we tend to focus on the proportional amount they spend on “frontline services” rather than growing bigger projects or re-investing to raise larger overall amounts. For the US (and, I believe the UK), total donations to charity as a proportion of GDP haven’t changed much for 20-30 years. So as much as there might be an interest from NFPs to get involved, there are issues that need to be addressed.

What about other novel ideas? We need to start exploring different mechanisms to fund research, or at least expand the range of opportunities for researchers to raise funds. Hell I’d even consider a special research economic exclusion zone if it would help. If social impact bonds (sometimes called social benefit bonds) can be used to fund preventive programs or services, could this model be used to fund basic research (or other forms of research)? Or wouldn’t it be good if more clever people who understand economics came up with suggestions like this from Chris Becker where he lays out preliminary thoughts for other sorts of investment bonds to support research (with some of the fancy economics lingo that makes my eyes glaze over when I go to see the accountant, but I like the concept).

The point is that research funding options in Oz (and I’m sure lots of other places) remain a bit monochromatic in their offering. There are plenty of novel ideas coming from researchers. Is it asking too much for a few clever types to exercise some imagination and deliver on new ways to fund research? Anybody out there got a clever idea? (Yep, I’m looking at all 6 of you who will actually have a look at this entry, I want answers dammit!)

PS The whole area of social impact bonds is pretty interesting anyway. Having been lucky enough to meet the very clever Emma Tomkinson a while back, you’ll find her blog has a plethora of stuff related to these instruments and examples of where they’ve been used. It’s well worth a look.

And I really would love to hear about any other novel options for funding research.

The False Science of Picking Winners

I have nothing specific against Michel de Nostredame. I’d probably happily share a game of petanque with him if the chance came up. Having said that, a condition of us hanging out would probably be a strict “no prophecies” rule (yep, if you missed it, he’s otherwise known as Nostradamus). This is partly because quatrains are a fairly irritating means of conducting a conversation. The other thing, though, is that predictions of such a nature are fairly tiresome when founded on not much more than an incantation inspired by the direction of the breeze rather than robust examination of real data.

The flawed nature of such predictions came to mind this week when I saw this coverage. Let’s put to one side the bizarre situation that sees a climate change denying legislator heading the House of Reps science committee. The more substantive issue is the fact that such a lawmaker is seeking to supplant the whole peer review process. Before you say this is an “only in America” issue, concerns have been raised within the last 12 months by some members of the Australian parliament regarding the appropriateness of some of the funding decisions of the NHMRC and ARC, as well as the need to ensure research is in areas with obvious potential for driving innovation and growth of the nation (bit of a rough cut at the humanities).

The stated reason is always to ensure that “public money is being spent wisely”. They will insist that there must be a clear sense of almost immediate payoff. There are a few obvious problems with this. The first is that peer review is a rigorous process that’s unlikely to be replaced effectively by a committee of legislators. Those charged with peer review of a grant application take their job seriously, wading through large amounts of submitted paperwork, reading additional information and drawing on their pre-existing expertise. I’ve had the chance to be involved on both sides of the ledger, and it’s exhaustive.

The next problem is with the described preconditions being proposed. Let’s examine those. The conditions include that the research must be in the interests of advancing the national health of the US. You also have to make sure your research is groundbreaking and of the finest quality, answering questions of the utmost importance to society. Oh, and you can’t be duplicative.

The problem is that basic research can be a little shy about staking its claim. History is riddled with publications whose importance was only appreciated over subsequent decades. Just one example lies in the works of Max Planck, who released the initial work upon which quantum physics is built at the turn of the 20th century. Humbly called Thermodynamik (1897) and Theorie der Wärmestrahlung (Theory of Heat Radiation) (1906), in describing the relationship between energy and the frequency of radiation, he made a huge leap inundertanding many observeddiscrepancies in existing theories within the field of physics. You might not have guessed that from those titles.

Another great example lies in the comments of the linked article above. Consider for a moment the 1978 publication of Limits of Cosmic Radio Bursts with Microsecond Time Scales”. I’m sure the link to the subsequent innovation is really obvious. WiFi of course .In fact this is the original name of that paper. Every WiFi device on earth was built thanks to the sharing of that research.

Is it any surprise that we don’t do well at predictions (and I mean all of us, but now consider how that looks when we only consider the politicians)? Need further proof, how about these quotes (which I duly acknowledge may represent the clunkers that got written down, while lots of other voices were raised in opposition)?

– “Louis Pasteur’s theory of germs is ridiculous fiction.” Pierre Pachet, Prof of Physiology at Toulouse, 1872.

– “We don’t like their sound, and guitar music is on the way out.” – Decca Recording Co. rejecting the Beatles, 1962.

– “Who the hell wants to hear actors talk?” – HM Warner, Warner Brothers 1927. 

– “This ‘telephone’ has too many shortcomings to be seriously considered as a means of communication. The device is inherently of no value to us.” – Western Union internal memo, 1876. 

– “There is no reason anyone would want a computer in their home.” – Ken Olson, president, chairman and founder of Digital Equipment Corp., 1977. 

“I think there is a world market for maybe five computers.” – Thomas Watson, chairman of IBM, 1943.

The problem with predictions is (at least) two-fold – we are so often blessed with a total lack of imagination as to assume that only today’s applications make sense when a new factoid is discovered. The other problem is that we can’t measure the value of such discoveries within the short-term currency of the political cycle. Look at the lag between that paper in 1978 and WiFi becoming standard. We’re all glad they looked at cosmic microbursts now though, aren’t we?

The other big problem inherent in imposing the proposed standards from Lamar Smith is with the requirement to be “groundbreaking”. As much as I quite like the idea that all science types are quietly diligent superheroes, shattering the falsely constructed frontiers limiting the pursuit of knowledge and wisdom, the truth is a little more low key than that. Research pursues small steps, not massive strides in most instances. Most researchers will add a very particular tile in the mosaic they’re working on, while others working in the same field add their own multicoloured irregularity.  Research rarely breaks the ground, it slowly claims it after the model of the inchworm.

So let the experts review the grants and leave the politicians playing out their small time dramatisations after Machiavelli. There is no tougher judge of an application than can be found in the peer review (whether it’s too onerous is for another day). Once we start demanding results delivered by bedtime for the MTV generation, or titles we can decipher the future from, we erode the ability of the journey of research to deliver unforeseen possibilities (not to mention the struggle of the humanities, where it’s harder to measure the market benefit derived from your work). And if the politicians must try to tell the future, let’s verse them in obscure literary forms (haiku, anyone?) and suggest they write down those predictions in a vague poetic style that leaves room for numerous interpretations. That way, they might just have people chatting about them in a few centuries time . Then let the researchers get on with the innovation that will guide all of our futures.