In a , we find that LLMs offer significant efficiency and effectiveness gains in the marketing research process for both qualitative and quantitative research. We show that LLMs serve as excellent assistants for insights managers through different stages of the research process: study design, sample selection, data collection, and data analysis.  

The AI-Human Hybrid Approach  

Consider a business context in which a brand manager collaborates with a consumer insights manager to formulate the problem the research is trying to address and come up with a set of research questions. The two may collaboratively agree on a research design that, for example, begins with exploratory research (e.g., in-depth interviews) followed by descriptive research (e.g., a survey). These first two steps of the research process are largely led by humans. Although the brand and insight managers could consult an LLM to gather secondary research on the topic and explore use cases that could help inform the research questions or research design, they would still largely rely on their knowledge of the business context to formulate the research problem, questions, and design.  

Our central premise is that a human–LLM hybrid approach can lead to efficiency and effectiveness gains in the marketing research process. For this study, we partnered with a Fortune 500 food company and replicated two studies the company had conducted using an LLM. The first study was qualitative and centered around business questions for the Friendsgiving celebration. The second study focused on testing a new refrigerated dog food. For each study, we treated the original (human) studies as the “ground truth” and benchmarked the LLM generated studies against them. This approach allowed us to objectively evaluate the quality of synthetic data and investigate the role LLMs could play in knowledge generation.

For qualitative research, we find that LLMs are excellent assistants for data generation and analysis.  

• On the data generation front, LLMs effectively create desirable sample characteristics, generate synthetic respondents that match those characteristics, and conduct and moderate in-depth interviews. Our results show that LLM-generated responses are superior in terms of depth and insightfulness.  

• On the analysis front, LLMs perform well, matching human experts in identifying key ideas, grouping them into themes, and summarizing information. Although LLMs missed some themes that humans detected, they generated some that humans did not. Expert judges find that human–LLM hybrids outperformed their human-only or LLM-only counterparts. The upshot here is that LLMs and humans bring unique, complementary insights that managers should leverage. 

A Handy Research Assistant 

An LLM can be an excellent starting point for creating the first draft of a survey and can generate survey introductions, screener questions, and demographic questions with relative ease. The LLM can focus on the laborious, repetitive, and uninteresting tasks while the human expert can use this time savings to think more creatively about answers to the business questions and the quality of the insights. 

A significant advantage of LLMs as an assistant is their low cost. We believe that this single factor will contribute toward rapid adoption of LLMs for insight generation. The gains here are likely to be higher for hard-to-reach respondents (e.g., doctors, senior managers) because synthetic respondents do not get tired and can provide lengthy answers to many questions. In the B2B arena where the end users and buyers are not easy to reach, LLMs could be quite helpful in supplementing the information gathered from human respondents. As an intelligent engine, an LLM could be a revolutionary generator of prior information for a wide variety of business questions at a low cost. 

It is important to note that LLMs can be wrong, biased, or hallucinate when not trained on the relevant data. Therefore, a human supervisor is a necessary part of the marketing research knowledge production process. For example, the human can make decisions about when not to ask an LLM for help. This could occur when the information sought is new not only to the company but also to the world. Other examples include marketing research in cultural contexts to understand local customs and traditions, topics with ethical considerations such as targeting vulnerable populations, and obtaining insights from data containing personal information, where LLMs may lack the necessary safeguards for data security and privacy.  

Source: Neeraj Arora, Ishita Chakraborty, and Yohei Nishimura, “,” Journal of Marketing.

Go to the Journal of Marketing

The post The Ultimate Research Assistant: How Marketing Researchers Can Effectively Collaborate with LLMs appeared first on ���������.

Imagine your friend suggests that the latest iWatch is a great purchase that you should consider. When you visit the store to purchase it, what questions do you think you will ask the store associate?  Will you ask about the product features?  Can you be influenced to look for more positive or more negative information based on the prior information you received? How can marketers use the information that they provide to consumers to their advantage? These questions served as an inspiration for the authors Kinshuk Jerath and Qitian Ren for their article “.”

Consumer big data provides marketers with information about every touchpoint in a consumer’s journey, including the feedback after the actual purchase. Using artificial intelligence, marketers conduct multiple analyses to provide targeted information to consumers regarding various facts about the product, like its pricing, attributes, and other information that can enable a consumer to make an effective decision. Marketers also have the responsibility of providing factual yet persuasive information that results in a beneficial transaction to both consumers and the firms. The authors have found that strategically placing the information in ways that resonate with consumers’ own beliefs, such that it confirms both the positive and negative information they have about a product, may result in more purchases. Contrary to intuitive belief, the authors suggest that providing negative information about the product along with its positives thus becomes advantageous to the firms. The authors build a mathematical model that illustrates the benefits of providing these two types of information for both marketers and consumers.

The authors have found that strategically placing the information in ways that resonate with consumers’ own beliefs, such that it confirms both the positive and negative information they have about a product, may result in more purchases.

The authors construct their framework based on “rational inattention theory,” which states that when information processing is costly, consumers optimally process only a part of it. Using this as the theoretical basis for the model, the authors provide a mathematical solution that optimizes the consumers’ attention allocation toward both favorable and unfavorable product information. Research in psychology has found that people tend to focus on information that affirms their beliefs (confirmation bias), and the authors show that this may occur for consumers when information processing is costly. Also, the authors show that marketers may benefit from both favorable and unfavorable product information, challenging the intuition that sellers cannot profit from negative information of the product. The negative information becomes essential to consumers especially when deciding the fit of the product to their needs, and if a sufficient amount of negative information is not available, then consumers may not even start their search process.

Q: The application of mathematical modeling using confirmation bias is very interesting. What aspects of the modeling did you enjoy the most? Why?

A: We had an open-ended research question that motivated us. We wanted to understand what information consumers seek when they are making purchase decisions. I think the enjoyable part was the emergence of the phenomenon known as confirmation bias as the result of optimal consumer information processing behavior. This confirmation bias can be a reason why the firm should provide unfavorable information about the product: if you as a consumer know that the firm gives only positive information and will never give me any negatives about the product, you are likely to feel that you will not be able to make a confident decision using this information. If you feel that the information given is not going to help you make a decision, you might not search for any information at all. In addition, if you don’t search for information, you will not be confident about buying the product. Therefore, the availability of negative or unfavorable information can induce purchases. 

Q: What types of websites in your opinion would benefit from the information design? Are there any websites that are exempt from this framework? Was there any particular website you were thinking of when developing this research?

A: The framework would be applicable to most e-commerce websites.  When consumers have a product in mind, they seek information. They can visit e-commerce websites like Amazon, where they can find the product ratings and can sort by the star rating. When you click on the 5-star link you will see all the 5-star reviews, or you click on the one-star link, and you see all the one-star reviews. I was doing the same to see what unfavorable information was there on Amazon. Essentially, e-commerce websites like Amazon were used as a frame of reference where primarily information gathering is important. However, if it is a discussion forum, where the objective of the website or the person who runs the forum is just to provide information and it is not easy to sort this information as positive or negative, then this model is less relevant. Some examples include Wikipedia or Quora. 

Q: The current framework studies many interesting concepts like confirmation bias, attention allocation, the valence of information. What would you say were some of the challenges you faced while developing this research?

A: There were many challenges. We wanted a strong theoretical foundation that would be supported by data. One reason I like this paper is that it starts off with the idea that consumers are rational, but it ends up showing that they can do things that might appear irrational, like confirmatory search or confirmation bias. Most people know and understand that consumers are not fully rational, right? If you take a rational consumer, they incur a reasonable cost, like the cost of thinking or the cost of processing information, then you can see the behavior as predicted by our framework. This action is essentially rational but appears irrational. Putting it all together in a framework that is theoretically solid, intuitive, and yet technically solvable was a challenge, but it was rewarding.

Q: How would you think information design would change with respect to using AI devices. What are some of the aspects to think about in this scenario?

A: It depends on how it is used. One aspect is that you can use AI systems on the fly. I think AI can cut both ways. For example, a person could be looking at certain information and the AI  makes suggestions which is the next best piece of information, helping the consumer make a good decision. However, you could also have  it the other way around where the consumer has seen certain information and AI gives them a different piece of information or makes it easier to search, increasing the chance that they end up buying the product. AI is a big tool that firms can use for consumers’ benefit. In order to be used for consumers’ rather than just for the firms’ benefit, there must be some regulations around that.

Q: Visuals and graphics form a large portion of website information. What are your thoughts about including visual information? How would this change the framework?

A: Visuals can really help with attributes that otherwise can be difficult to understand. Think of a car. Visually I can decide, I like red more than blue. Visuals can help a lot in conveying information when it is about softer match attributes or information on horizontal attributes. Information on vertical attributes is often information that can be digitally conveyed well in text or in some other numerical format. Visual information is a very good complement to textual information. Even complicated information can be given visually to make it is easier to understand. Sometimes it can be “in place of” textual information and sometimes it can be “complementary.”

Q. We were also curious to know how marketers can avoid negative perceptions arising from information management.

A: This is a big debate going on in privacy circles, which is all about information. Consumers are worried that there is a lot of information that marketers have. It’s a very important issue and it is a matter of trust. I trust some websites more than others. It’s about reputation. Coming back to this paper and talking about favorable and unfavorable information, there should be some of both. First of all, as a marketer, you should be giving different kinds of information and not make it too difficult to understand or find information. This also helps with reducing negative perceptions. Over time a firm will build the reputation of being fair. Marketers should not make finding unfavorable information especially hard. They can always highlight the positive things, but I think marketers should make unfavorable information readily available as well.

Read the full article:

Jerath Kinshuk, and Qitian Ren (2021), “,” Journal of Marketing Research. 58 (2), 343–62. doi:10.1177/0022243720977830

The post Do You Think Your Decision to Buy Was Rational? appeared first on ���������.

As a vivid example, we describe a quasi-experiment that occurred when eBay shut down all the paid search advertising on Bing during a dispute with Microsoft, but lost little traffic. These quasi-experimental results inspired a follow-up field experiment where eBay randomized suspension of its branded paid search advertising and found results consistent with the quasi-experiment.
 
We begin by establishing various type of quasi-experimental variation at the individual, organizational, and market-levels. In each type, given the lack of an experiment, some individuals, companies, or markets receive an action or policy (i.e., treatment group) and some do not (i.e., control group). For example, some markets are affected by a new policy and some are not. The question is how the markets receiving the treatment would act if they had not received it (i.e., the counterfactual). The unobservability of the counterfactual means assumptions are required to ensure that differences (both observed and unobserved) are as untroubling as possible, thereby mimicking random assignment as closely as possible. 

We discuss how to structure an empirical strategy to identify a causal relationship using methods such as difference-in-differences, regression discontinuity, instrumental variables, propensity score matching, synthetic control, and selection bias correction. We emphasize the importance of clearly communicating identifying assumptions underlying the assertion of causality and establishing the generalizability of the findings.

We examine the following topics with the goal of helping researchers and analysts use quasi-experiments more effectively.

• Topic 1 – Research question: Do we care whether x causes y? The first and hardest stage in this process is identifying a question in which marketing scholars, managers, or policy makers actually care whether x causes y.
• Topic 2 – Data question: How to find data with quasi-experimental variation in x? Much of the work using quasi-experimental variations in marketing harnesses easily understood events such as contract changes; ecological issues such as the weather; geography; and macroeconomic, individual, organizational, and regulatory changes. The key is to consider why each of these sources of variation can approximate random assignment.
• Topic 3 – Identification strategy: Does x cause y to change? The researcher must first explain where the variation claimed to be exogenous comes from. Second, the researcher needs to demonstrate that the relationship between the variation and the outcome of interest is very likely driven by the relationship between x and y and not by some other factor.
• Topic 4 – Empirical analysis: How to estimate the effect of x on y? We discuss three different regression analysis frameworks using quasi-experiments: difference-in-differences, regression discontinuity, and instrumental variables.
• Topic 5 – Challenges to research question: What if variation in x is not exogenous? We outline three methods—propensity score matching, synthetic control methods, and selection bias correction—with steps to take when comparability between the control and treatment groups is violated.
• Topic 6 – Robustness: How robust is the effect of x on y? The idea here is to show that the sign, significance, and magnitude of the estimate remain broadly consistent across a vast range of possible models. A few examples of robustness checks include different controls, functional forms, choices of time periods, dependent variables, the size of the control group, and a placebo test.
• Topic 7 – Mechanism: Why does x cause y to change? Understanding how the process of the change unfolds adds insight that can drive new knowledge and stronger actions.
• Topic 8 – External validity: How generalizable is the effect of x on y? The external validity discussion in a paper should recognize the assumptions required for the analysis to capture the average treatment effect across the population of interest rather than a more local effect that is an artifact of the data sample or the source of quasi-experimental variation.
• Topic 9 – Apologies: What remains unproven and what are the caveats? Any identification strategy relies on assumptions that need to be explicit throughout the paper. While apologies do not mean all is forgiven, the objective should be to clarify the boundaries of claims made in the paper.

.

From: Avi Goldfarb, Catherine Tucker, and Yanwen Wang, “,” Journal of Marketing.

Go to the Journal of Marketing

The post The Benefits of Quasi-Experiments in Marketing [Research Methodology] appeared first on ���������.

The Paul E. Green Award recognizes the article in JMR that demonstrates the greatest potential to contribute to the theory, methods, and practice of marketing.  It honors the S.S. Kresge Professor Emeritus of Marketing at the Wharton School at the University of Pennsylvania. On behalf of the ���������, this year’s Green Award selection process was managed by Professor Michel Wedel, Distinguished University Professor at the Robert H. Smith School of Business at the University of Maryland. The committee overseeing the selection process consisted of Russ Winer (New York University), Kusum Ailawadi (Dartmouth College), and Mary Frances Luce (Duke University). ​In recognizing the winning paper, the committee made the following comment:

In their paper, “Bayesian Synthetic Control Methods,” Kim et al. significantly improve the application of synthetic control methods (SCMs) to marketing and other social science problems where the lack of a randomized control group inhibits the ability to estimate the treatment effect.  SCMs are very useful tools for such quasi-experimental research that creates a “synthetic” control unit as a weighted average of a set of controls where the weights are determined by getting as close as possible to the pre-treatment outcome in the treatment unit.  It has three limitations: restrictive constraints on the weights, no formal theory for statistical inference, and what is called the “large p, small n” sparsity problem, in which there are more parameters than observations.

The significant contribution of this paper is the use of Bayesian methods to address these three limitations. It relaxes the constraints on the weights, provides exact statistical inference, and uses shrinkage priors to solve the sparsity problem.  In addition, the authors’ methods incorporate analogs of the frequentist SCM variants used in prior research.  The authors provide computer code for their SCM approach so that it can be implemented by practitioners and other academics.  With respect to the practical applications, the example in the paper of the impact of a soda tax imposed on Washington State consumers in 2010 illustrates how their method can be applied in the real world.  Thus, the paper makes important advances both methodologically and substantively as it gives analysts a new tool to improve the measurement of the causal effects of a variety of marketing, policy, and other interventions where randomized controlled tests are either infeasible or expensive.

In addition to the winning paper, the three other excellent finalists for the award were as follows:

The post Kim et al. Receive Journal of Marketing Research 2020 Green Award appeared first on ���������.