After having fun with the LLMs last week, asking if they thought they were ready for market research or not, I decided to play around with the "deep research" functionality for each of the LLM tools.
Ready to dive in?
The Deep Research claim
Deep Research is basically LLM on hard mode. The idea is that Deep Research modes will be more structured about answering a query, digging deeper into sources, citing sources, and delivering more than a paragraph or sentence in response to your query. Instead, it will do a thorough search and write a report for you.
The assumption is that hallucinations will be fewer, as the data included in the report is cited; at least that is my takeaway from including a list of sources at the end of the report (in some cases, but not all).
The feedback I've heard about this Deep Research function is that it's impressive: it's definitely going to put market researchers at risk of losing their jobs because anyone can use it to dig into a topic and have a report delivered in far less time than it would take a human to deliver a report of similar breadth, using a similarly robust set of sources for the findings.
The methodology
I decided on a prompt that would be difficult for a human to examine thoroughly, but that a computer would be able to do much faster. Here is the text of the prompt.
What are the best tools available right now to help market research professionals do their jobs faster or cheaper? Please specify if AI is being used in these tools and how AI is being used in the tools. I'd also like a summary of the ways AI is most helpful in market research today and when to NOT consider using AI in market research - both quantitative and qualitative. I am looking for tools that could be used for either B2B or B2C research. I also am interested in tools for any stage of the market research process - survey question ideation, data collection, data analysis, or finding insights across multiple sources based on a natural language query. Please focus on tools that are budget-friendly but have global capabilities.
I entered the first part of the above prompt into ChatGPT, and it responded with clarifying questions:
- Are you looking for B2B or B2C tools? Are you looking for tools at a specific stage of the market research process?
These led to the reply in bold which was then added to the original prompt and used for the following Gen AI tools.
Microsoft Copilot "Think Deeper"
Gemini Deep Research
Perplexity Deep Research
NOTE: I did not use Anthropic Claude's Deep Research because it requires a subscription to their paid tier to access the tool, and I was using only what was available for free.
The output
Interestingly, Gemini Deep Research told me the steps it would take to run the research and asked if I had changes I'd make to the proposed steps. It was the only Gen AI tool to do this.
General results
ChatGPT: ChatGPT did what I asked - gave me restech tools applicable to various stages of the market research process, and specified if and how AI was implemented into the tools. The list wasn't exhaustive - that would have made the report far too long, but some of the tools it suggested did surprise me, especially for "AI Insight & Reporting Tools." Fascinating that it included itself and other LLM tools (but not Microsoft's LLM, Copilot, nor Google's Gemini, but it did include Notebook), but not any of the myriad of other insight and reporting tools that I'd have expected, like Displayr.
Copilot: What a let-down. It was short, it was anything BUT exhaustive as it suggested it would be in its own introduction, and it just felt so lacking. Given that I don't think "Think Deeper" is meant to be the equivalent of the "Deep Research," I suppose this would make sense, but then where IS Microsoft's "Deep Research" equivalent?
Gemini: This felt more exhaustive than ChatGPT's report. However, it only focused on tools where AI was somehow integrated into the tool. I purposefully made the first sentence of my prompt broad, attempting to include non-AI tools in the mix, but it seemed Gemini missed that. It was also too wordy! I honestly wanted to run it through "regular" Gemini to get a summary. Thankfully, it had a summary in the table, at least for the tools and uses of said tools, but the tools it chose to include were interesting. No Gen AI as "insights and reporting" tools in this one!
Perplexity: Similar to Gemini, Perplexity went directly to tools that used AI in some way. And, again, I was surprised by the tools it decided to name for each of the market research process stages.
Each report generally cited the same areas where one should be wary of using AI: times when human nuance is needed; times when data quality might be questionable; times when the AI tool is relied on for data more than humans; bias in data as models are trained.
The takeaway
1. Not a single report accurately captured the stages of the market research process. None included project management, business problem definition, customer or stakeholder management, checking data quality, or even deciding on the methodology to be used to gather the data or identifying the audience to be researched! MAJOR FAIL!!!
2. Great table summaries - weird selection of example tools. Generally, aside from survey programming tools Qualtrics and SurveyMonkey, there wasn't much consensus on tools for data collection, insight development, or reporting.
3. Gemini needs an editor to make its content a bit more approachable.
Conclusion
Like any LLM-based AI tool, a report created by a Gen AI Deep Research should not be considered a de facto source of truth, no matter the number of sources cited. The information may be a bit more than you'd get from a typical back-and-forth with the "quick answer" version of the Gen AI, but I think I need to play with it more to see where it really shines.
