Methods ≠ Methodology

Every so often I get asked a research-methodology question. In those moments two of my many inner voices begin arguing with each other: one screaming at me to run in the opposite direction and take cover; and the other urging me to stay, learn more about the project, and ask the all too important (albeit selfish) question: if I help, can I get my hands on your data?
 
Now, I am always happy to offer my opinion on research design or provide quick statistical guidance to a colleague in need. However, as someone who has had significant training in statistics, measurement, and modes of inquiry, I know all too well the dangers of offering advice without understanding the larger context of a project.

One memorable example that comes to mind was an interaction I had several years ago where I was put on the spot to answer a ‘simple’ question: how to translate a hypothesis into a testable model. Of course, I could have easily come up with a response that would have satisfied the person, but I chose to take a different path; I responded with a question of my own:

What do you want to know and why?

​Surprisingly, the person struggled to provide an answer. But with a single question, I was able to identify the root of their problem: they had yet to flesh out a methodology.
 
As I began to formulate my response, I was interrupted by a bystander who believed that they could provide an answer without any additional information about the project’s objectives. I smiled and gestured for them to continue. Months later I would learn that despite getting an answer from the bystander, the person still had difficulty developing and articulating a methodology, preventing them from moving forward with the project.
 
To many, my unwillingness to provide a quick and useful response is a failure on my part. More people than I would like to admit have questioned my experience and minimized my skills because I prefer to listen first then ask questions before giving advice. And as I enter a new phase of my career—Ph.D. in tow— and reflect on that experience and others since then, I realize that much of my success as a ‘data person’ and social researcher can be traced to a conversation I had with a mentor nearly a decade ago. I had approached them for advice while struggling to develop a questionnaire for my first independent project, and they offered the following words of wisdom:

What do you want to know and why?

At the time, I was irritated by the advice and thought it was yet another way academics try to confuse students while hiding behind the four walls of the Ivory Tower. Weeks later a light bulb lit up in my mind, and I suddenly understood what their seemingly cryptic words meant:

A methodology is more than the methods used to collect, analyze, and interpret data.

Put another way, they were encouraging me to spend more time thinking about what I intended to study and why, as answering these two questions would form the basis of my project and guide me through the research process.
 
In short, they were right. When I reviewed my research aims and thought about what I wanted to study and why, everything fell into place— including the study instrument. And while it would take me several years to fully appreciate the advice, now when I consider collaborating with others on projects or begin a conversation with a colleague about their work, I always ask some version of that question.
 
While my story grappling with issues of methodology ended well, many don’t. And despite the complexity of designing studies— particularly in applied research settings, I am always surprised at how little attention is given to a study’s overall methodology— until something goes wrong. Yes, we all like to play around with data and create compelling visualizations to aid in the interpretation of the results. But it is equally important to identify a topic for study; formulate appropriate questions; choose and justify the selection of a study design and data collection method etc…

​So, whether you’re developing a large, grant-funded research project or conducting an internal evaluation, consider partnering with a methodologist who is familiar with your field of research or your organization’s philosophy and procedures. Ideally, you should hire this individual during the planning stages of a study. That way, they can work with you to develop your project and ensure that study objectives are well aligned with the set of methods and procedures you intend to employ. By partnering with a methodologist, you can also gain insight into whether the project is feasible and achievable given your budget and time constraints.