General Principles

• Disaggregate the question. On some occasions, we have a very specific question - “how many people in Nigeria own a car” - but in other cases, we’re more interested in a broader area (“what do we know about the economic impact of remote working”). In the latter case, its worth taking the time to explicitly break out multiple questions that you might look for (commuting times, office rental values, labour productivity etc) and could each have different search terms.
• Keep a running tally. If you don’t find the answer in the first 10 minutes - in other words, this is going to take some extended work - it can be worth keeping running notes on what you’ve tried: search terms you’ve used, sources you’ve checked and so on. Even if they weren’t much help, that will make it much easier for someone else you ask for advice later on to see what you’ve already looked at and what you might have missed.
• It may take some time. Sometimes a single Google Search will get you the right number - but other times, you’ll have to try many approaches, brush up on the basics of something or skim a couple of books. If you haven’t spent at least 2-3 hours trying to find, its hard to say that something definitely doesn’t exist.
• No source is perfect. Even official statistics often changed drastically as new data comes in. You should always use your common sense, and check if this feels approximately right.
• Understand where the data comes from. Don’t just take anybody’s number on face value - try to understand how they got to it. What is the original source of data, and how reliable do you think that is?

As a rough guide to levels of reliably:

• 5 - People from vastly different ideologies agree on this one. We can measure it directly. (This rarely happens outside of Physics).
• 4 - People might disagree, but we have multiple Randomised Control Trials (if we care about causation) or a pretty reliable way to exhaustively sample and measure (think the census).
• 3 - We have a mature literature. While we don’t have true randomisation, we have data based on a sophsticated regression strategy: eg based on a natural experiment, or regression discontinuity design.
• 2 - We have hard data. We might not be able to completely control for causation, but our data isn’t just based on people’s opinion.
• 1 - We have survey data, but it’s at least as well designed it can be to deal with the limitations of survey data.
• 0 - We have badly designed survey data, or numbers inappropriately extrapolated from a survey.
• You don’t have to reinvent the wheel. If a literature review already exists, you should draw on and update it, rather than start from scratch. It’s always worth starting by searching variants of “[TOPIC] literature review”, “[TOPIC] systematic review”, “[TOPIC] handbook” Often if you just find the most recent paper on a topic, it will contain a reasonable literature review at the start.
• Find the experts. In academia particularly, there tends to be 2-3 experts who are the go to experts on a particular topic. Once you’ve found these experts, you should check their website or on Google Scholar, and see what they’ve written mostly recently: these are the papers that are least likely to be in someone else’s literature review.
• Figure out who disagrees and why. Once you’ve found your 2 experts, you might start to notice, they tend not to agree with each other… (There are a lot of literatures which are just people trading back papers back and forth for years or decades.) But it’s still helpful to disentangle what the positions in a debate are.
• In the back of your head, try to keep track of what we don’t know. It might be that no good data exists on a particular topic: but that’s also worth knowing, and can suggest topics for future research.
• Look for the right search term. This might differ by type of source (eg academic, consultancy etc) - but for literature reviews, jargon is your friend. The more specific a term people have come up for calling something, the easier it is to Google.