I recently attended TMRE@Home, a 3-day virtual conference with many talks about many topics. The third day focused on restech - obviously an interest of mine. The first speaker launched a poll asking the audience, "Is AI ready for market research analysis?"
At first, nearly 75% of the responses were "No." After a couple of minutes of the poll being up, it dropped to 68%.
Intriguing that the swiftest reaction was an overwhelming "NO," tempered a bit by those who gave it a bit more thought.
Either way, more people said no than said yes. That led to me posting about it on LinkedIn with thoughts about the need for AI literacy and the limitations AI currently has that limit its ability to be fully adopted for market research analysis.
The reaction to the post was mixed. People seemed to either be in full agreement that AI isn't ready for market research analysis, or else were bullish on its readiness, and more likely to suggest that people themselves were the ones not ready for AI.
To be fair, during the conference, I had posted a note to the effect of, "It's actually more nuanced. I think it's ready for qualitative analysis, but we aren't quite there yet for quantitative analysis." To explain further: LLMs are evolving well for analyzing open-ended text for themes and pulling quotes and clips of videos to support the themes, and for allowing people to interact with the data using natural language prompts. This isn't true for quantitative analysis; while we have had many tools available to do quantitative analysis, what we're missing is the ability to use natural language prompts to interact with the data.
But I found the "it's already here," or "it's inevitably going to take over," versus the, "I've tried it and it's very hit or miss," intriguing.
I kept thinking about this as the day progressed, and, suddenly, it dawned on me.
Why not ask the tools themselves?
Does AI think AI is ready for market research analysis?
The methodology
I started with ChatGPT. I prompted it twice. The first prompt was: "Do you think AI is ready for taking over a market research analyst's job? I'd like to know the readiness for quantitative analysis that could be done using natural language queries, and qualitative analysis using natural language queries."
The second prompt was: "Let me ask a slightly different way: do you think AI is ready to do market research analysis? I'd like to know if this wording changes your assessment of readiness for quantitative and qualitative analysis in market research."
I let ChatGPT answer both prompts, then copied the conversation, and opened Copilot, Microsoft's GenAI tool.
I entered this prompt into Copilot: "Hi, I had an exchange with ChatGPT about AI's readiness for market research analysis. I'm going to put the copy of the exchange below. I'd like you to critique ChatGPT 's response - where do you agree and where do you disagree, and why?"
I let it answer, and then I went to Gemini, Google's GenAI tool.
I entered the same prompt I'd given Copilot into Gemini. "Hi, I had an exchange with ChatGPT about AI's readiness for market research analysis. I'm going to put the copy of the exchange below. I'd like you to critique ChatGPT 's response - where do you agree and where do you disagree, and why?"
Lastly, I went to Claude, Anthropic's GenAI tool. I entered the same prompt. "Hi, I had an exchange with ChatGPT about AI's readiness for market research analysis. I'm going to put the copy of the exchange below. I'd like you to critique ChatGPT 's response - where do you agree and where do you disagree, and why?"
The data
Here are the critiques from each of the GenAI tools afterwards.
The results
TL;DR
ChatGPT was very middle-of-the-road. AI is ready to augment a market researcher, but certainly not ready to replace one. Routine tasks, great. Nuance needed? Not close.
Copilot thought ChatGPT was being too conservative and was pretty bullish on AI's readiness. However, this readiness was really about near-future advancements, not current capabilities.
Gemini thought ChatGPT was too bullish on AI's readiness, citing the human involvement still needed to make sure data is clean and that complex analyses for quant are set up correctly.
Claude also thought ChatGPT was too optimistic about AI's readiness, saying natural language interfaces are still pretty basic and any analysis is still highly dependent on data quality.
Detailed results
ChatGPT focused on AI's current ability to handle routine work well, like summarizing, processing, and visualizing data. This was true for both quantitative and qualitative research. In the quant world, AI handles large datasets well, but conjoint analyses, experimental analyses, and advanced weighting are still human-dependent. In the qual world, AI's ability to identify themes is getting better, as is the ability to generate insights. However, the human is still absolutely needed to interpret the data and understand the nuance, so that it could be correctly applied to a business strategy.
Copilot agreed with the idea that AI is a great augmentation tool to a market researcher. However, where it disagreed with ChatGPT was in how advanced the tools are or are becoming. Basically, tools are evolving for both quant and qual at such a speed that capabilities are being added rapidly and changing how ready AI is for taking over market research outputs. However, the human is still absolutely needed to interpret the data and understand the nuance, so that it could be correctly applied to a business strategy.
Gemini and Claude were fairly similar in their critiques of the ChatGPT response. Like Copilot, they agreed that AI is a great augmentation tool and can handle routine tasks really well. Where they disagreed was how advanced the tools are, especially for qual, where they both suggested natural language tools are still fairly basic and surface-level in their application. As with the others, they stated the human is still absolutely needed to interpret the data and understand the nuance, so that it could be correctly applied to a business strategy. Additionally, humans still have work to do to improve data quality, and have the upper hand on the ability to apply relevant industry-specific knowledge to the data as part of the data interpretation.
The takeaways
AI is advancing, of that there is no doubt. If we look at the question of AI readiness for market research analysis as a point-in-time question, though, GenAI tools generally agree there is still room to grow before it can be used for more than routine tasks like summarizing data, for both quantitative and qualitative data.
If we look at the question of AI readiness for market research analysis as a near-future-capability question, we expand the capabilities for AI to do more with the data. Perhaps, as Copilot indicated, AI tools will become better at understanding human nuance in responses.
However, every GenAI tool agreed: humans still have the key skill of strategic thinking, nuanced thinking, and knowing how to interpret data within a given context. That particular capability will, for now, remain a human capability, and a critical capability.
So, today and in the near future, the sweet spot is human PLUS tools that integrate artificial intelligence. I think that's a pretty great win.
