fabricating knowledge
Science and Technology have proven to be a successful enterprise.
With it, humans have grasped reality in a much clearer and coherent way than with previous traditions, as it has allowed us to completely change the world around us.1 These facts made me wonder about what knowledge is in what I do as a researcher, the role knowledge plays in organizations and what types of knowledge we value over others. However, to try to sketch an answer for these questions, I need to contextualize the background in which this role (researcher) exists.
reducing uncertainty:
Creating technology is an extremely uncertain endeavor. We can never be sure if our technology will get adopted or even understood, yet our species continues to create technology at an ever-increasing rate. Additionally, we deal with economic pressures that need clarity of our efforts and their impact.
Thus, we do research to gain knowledge and reduce uncertainty. We use concepts, methods, and ways of constructing knowledge borrowed from science. Though we are interested in reality in a different way than academic institutions, living inside organizations that create technology, we are mostly interested in practical knowledge. And more specifically, knowledge that moves our organization forward, which is shaped by how the organization makes decisions and the values they embody.
As part of our division of labor in the organization, our goal as researchers is to gain knowledge about the world, and to improve the correspondence of what we know is true to what is actually true. And we do this by running studies and by simplifying the world into language and numbers. In other words, we bring context and clarity to our organization's information needs by sharing the insights we found out in those studies.
types of knowledge
Because we are interested in human behavior there are many ways of acquiring knowledge about it. In general we categorize all the methods in two: quantitative and qualitative research. The first type "derives from the scientific method (empiricist and positivist traditions) and used in physical sciences", it uses a deductive approach, it relies heavily in statistics, it's used to quantify or measure phenomena, and it's design allow us to generalize its findings to a particular population. On the other hand, the goal of qualitative research is to "describe certain aspects of a phenomenon", thus it's scope is bigger, more flexible, and looks for a deeper understanding of the subject, allowing subjects to raise topics that the researcher may have not thought of or included.2
People usually have a preferred method as one might have for either vanilla or chocolate ice-cream, though it's not about what one prefers as to what it is that we are trying to find out. Normally those that prefer one method, choose the quantitative regardless of context, and ironically, they don't normally care about being statistically sound, which is the whole purpose of using it, to be able to generalize findings. These biases can run strong in an organization, and as a researcher one needs to understand what those are in order to justify and communicate the research strategy effectively.3
challenges when scaling
One way to know what type of knowledge we value is by looking at how ResearchOps (research operations) manage scaling. In 2020, Brigette Metzler, published an article in EPIC synthesizing a three part research they did in the ResearchOps community of over 6k members at that time (now there are more than 16k members). One of the main challenges their community faces is to manage the demand for research whilst still maintaining the quality of the work.
The author points out how organizations have tried to address this challenge by scaling the number of people performing the research. Though these frameworks always fail to contextualize the whole operation, as they just focus on one aspect, and therefore they are not really scaling. Additionally they are not useful to diagnose strengths and weaknesses of the research practice. Thus she points out a new framework so that the organization can have a more realistic map, depth, and detail of the practice.4
On a similar note Kate Towsey, ResearchOps Manager of Atlassian and author of upcoming book Research that scales, has a similar understanding of what scaling should be for a ReOps. For her, the notion of treating scale just by incrementing the number of people who can perform research, is not enough. Her experience and the needs of her organization tells her otherwise, there are a myriad of variables that need to be taken care of in order to have a successful operation that meets the demands of the organization. 5
Both of these cases show how mature organizations think about their research practice. They are also the exception, they are the leading example of how or what a ReOps should be like, but the reality of research practices lies more in those frameworks that try to scale their research operations with just one variable. Furthermore the ReOps community is valuable to this conversation as their role differs a great deal with the researcher itself. They have gone beyond discussions of which method is better (quant or qual), they understand the role each method has for their teams and therefore they try to build the capacity, tools, and practices that can help everyone.
what we value
We are biased or forced towards certain methods; it's not uncommon to see an organization that focuses their efforts in just one space (either problem or solution space), or organizations that don't trust qualitative research or quantitative research, or organizations pressured to only do very short research cycles. In Metzler's research, for example, she describes how organizations may fall into imbalances as they cope with the speed demanded by the business:
"Turbulence in the system that is off balance can be seen clearly in the tension noted previously between the speed demanded by business and the time taken to do contextual, generative research. A symptom of imbalance, is researchers needing to spend so much time on faster evaluative types of research that they cannot gain the time or by-in for generative research favoured by ethnographic research methodologies for example. Or researchers spending so long doing the deeper layers of research that they are unable to respond quickly or lack the skills or infrastructure to do evaluative research when it is required."
Organizations try to cope with these and many other factors all the time and it's the reason why a framework to diagnose the practice is such an important tool to any ReOps.
Furthermore, an interesting implication of this and how ReOps understand their practice is that all types of knowledge are valuable and necessary.
grounding
We are culturally shaped by our education and what we think counts as knowledge. Though, as we mature as researchers, knowledge seems to grow in definition. Not only because we become aware of the distinct nature of being a researcher in industry, and thus we understand that the type of knowledge we are after is different from academia, but also because if we only do a certain type of research we know our picture of reality it's incomplete. Therefore as gatekeepers of the organization's knowledge we know what different methods can bring and how they can allow us to move forward.
I'm not suggesting or saying that anything goes. Within each area of inquiry and method there are rigor and standards that we must uphold in order to do good and ethical work, though we live in the realm of the tech development rhythms. Therefore different levels of certainty and confidence are part of our basic survival toolkit.
It shrinked philosophy's domain and has put aside many other ways of knowing the world.↩
Definitions come from this paper: The strengths and weaknesses of quantitative and qualitative research: what method for nursing?↩
I think that the whole quant vs qual discussion gets in the way of talking about what is really at stake and instead it ends up being a discussion about what bias we lean into when we think about knowledge. I think a more fruitful way of framing research is if we talked about it from the area of inquiry: solution and problem space. In short, the solution space we want to know if what we have created (concept or the product itself) works, and in order to do that we try to answer all kinds of questions, like: is the product usable, does the new journey make sense for users, if a different call to action creates more leads, etc. On the other hand, in the problem space, we want to know the context in which our solution could take place, and the questions we answer here are more like: how are they currently solving their problem, what is the problem, what are their needs, what challenges they face, etc. I believe this framing allows a better discussion.↩
Her framework is very interesting as it allows Managers to oversee many areas. For example if all their efforts go to a certain type of research, if they should focus more on governance (consent, ethics), if their knowledge management is helping their team or if previous studies gets lost, if the recruitment process could be improved, etc.↩
In her talk @ PWDR, Kate explains how each study is unique and therefore we can't really create an industrialized process and expect good results. Rather, we must take into account: the information that makes the study necessary, who is performing the research, and who will be reading it. When we take these variables into consideration, it changes how we set up the tools, practices, and processes so everyone can do their research and help their teams to move forward. From teams that are highly academic and therefore rigor is important, to product managers that need to talk with customers to understand them and empathize with them.↩