Top Tips for Curating Research to Support Your Cause
When looking at research and statistics it’s always important to do a bit of digging (as I’ve pointed out before). This hit home for me again recently when I was working through some newly released research on the impact of gender identity conversion therapy (you can read my critique here). As I worked through the research report, I was struck that it could be used as a textbook example of how to create and report on a survey so that it supports your cause. (Obviously I can't know whether that was deliberate in this case!) So, inspired by that example, here are my top tips for curating research to serve your cause.
1. Use a voluntary, non-probability sample.
These first few tips are all about how you get your data.
The make-up of the group who answer your survey is important. To get the results that best support your cause you want to combine tips one and two.
A voluntary, non-probability sample means people volunteer to complete the survey in response to your invitation. It is non-probability because you don’t do anything to try and ensure there’s an equal chance of any person completing the survey. (In fact, as tip #2 will show, we want to do the opposite.) This means you won’t be able to know how representative of any larger group your sample is, but that’s not something to worry about. Most people won’t realise.
2. Find respondents from a group likely to share or at least be sympathetic to your views.
Here you want to think strategically about where you advertise your survey and how you invite people to respond to it. Perhaps there is a noticeboard or email mailing list that might reach people with the right sort of views. Or you may find the social media channels of a special interest group helpful.
3. Use retrospective, self-reporting.
We’re now thinking about the ways you’re actually gathering data from your sample of respondents.
Retrospective reporting asks people to look back and remember something from the past. It matches well with a carefully sourced voluntary sample because our more recent experiences and current beliefs are likely to shape our perceptions of the past. If you can find people with strong views now, there’s a good chance their retrospective report will be influenced by those strong views.
Self-reporting is when the response is based on the individual’s own perception (rather than, for example, a formal diagnosis or a widely recognised set of diagnostic criteria). It’s an approach that will often put more respondents in a certain group than approaches that use more rigorous diagnostic methods. It can be particularly useful if you’re looking for evidence of psychological harm since that’s a fairly subjective measure and people might well over-report.
4. Don’t ask questions that might give awkward answers.
Think carefully about what you actually ask. If challenged later down the line, it might be hard to hide unhelpful data, but if you don’t have the data because you never asked the question you’ll be in a much safer position.
What are the things that could undermine your position and throw serious doubt on your overall conclusions? Be careful not to ask questions on those unless you are confident that you won’t end up with unhelpful data.
5. Exclude unhelpful responses.
The next two tips are for the data processing stage.
If you find that you’ve received some unhelpful responses even after following the previous tips, you may be able to exclude some of these.
Is there any way in which you could cast doubt on the integrity of those answers? Anything that you could give as a defense for thinking the data is of poor quality or was given in bad faith? You may be surprised how many unhelpful responses you can exclude if you think about this carefully.
6. Involve an independent research monitor.
An independent research monitor is a person with some relevant qualifications who will keep an eye on your analysis of the data. When you release your results, being able to show that the analysis was independently monitored can give it an extra air of credibility.
Where possible, you want to try and find someone who is sympathetic to your perspective. But even if this isn’t fully possible, if you’ve followed the tips above, it shouldn’t really matter. If your data has been sourced carefully, you should still be able to get results conducive to your position even if it is analysed objectively.
7. Present the figures but don’t comment on those that are unhelpful.
These final tips are about how you report your results.
Your report will probably contain a mixture of tables, charts and comments. Think carefully about what you want to report. You may find there are some things it’s best to omit completely and hope no one notices.
You can also hide unhelpful results. People are less likely to notice such figures in charts and tables because they require a little more thinking. You’re probably safe to include some unhelpful results in these and then just make sure that your comments only draw on the helpful results. Your comments can provide a good distraction.
8. Be accurate but strategic in your comments.
Try and avoid stating things that are clearly not true from the data (although you might get away with a few such statements across a report), but obviously you want to highlight the things that are helpful to you. Remember to think about whether a fraction or a percentage will more effectively make your point. The same figure can feel quite different depending on how it’s communicated.
And always remember that correlation is your friend. If you can point out a clear correlation, people will assume causation even if your data could never prove it. Usually, you don’t even need to state anything that isn’t true; just present the correlation in a carefully worded statement and your readers will join the imaginary dots themselves.
Follow these simple steps and you too can curate research to support your cause. (Or you can use them to evaluate the research of others. May that’s the better use for them!)