Research Quality Forum
Forum Co-Chairs

Gian Fulgoni
Gian Fulgoni
comScore
Bio »

Don Gloeckler
Don Gloeckler
ARF
Bio »

Research-on-Research Leaders

Michael Fallig
Michael Fallig
Cogenti Applied Strategies, Inc.
Bio »

Chuck Miller
Chuck Miller
DM2
Bio »

John Ouren
John Ouren
Independent Consultant
Bio »

George Terhanian
George Terhanian
Toluna
Bio »

ARF Executives

Don Gloeckler
Don Gloeckler
EVP Innovation, Research Quality, & Executive Member Liaison, ARF
Bio »

Research Quality Forum

Upcoming Meetings

Foundations of Quality 2: Launching What We’ve Learned

Thursday, May 08, 2014
9:00AM - 3:00PM ET

Join us in New York City for a one-day symposium in on how to activate learnings from ARF's FoQ2 initiative. Learn from the FoQ2 leaders, network with your peers and industry experts, and share your own points-of-view on this important ARF initiative. Go back to your desk with immediately applicable ways to drive research quality and reduce business risk. 

This is a free event for ARF members.

More details coming soon!

 




Previous Meetings

Members: Miss a meeting?
Click here to download past presentations via your My ARF account!

Hear the Latest From FoQ 2: The Journey Continues

Tuesday, October 15, 2013
2:30PM - 4:30PM EST

Registration
To attend via webcast, register via My ARF »
If you have any questions, please contact Zena M. Pagán at forums@thearf.org.

Join us via webcast on October 15 for the next meeting of ARF’s Research Quality forum, as we welcome back Steve Gittelman (Sample Source Auditors), Robert Walker (Surveys & Forecasts) and Bill Cook (e-Strategic Advantage) to present the latest learning from Foundations of Quality 2. Topics will include accounting for social desirability in cross-modal comparisons and revelations about surveys taken on mobile devices.

Presented by:
Steve Gittelman – President, Sample Source Auditors, a division of Mktg, Inc.
William Cook – Principal, e-Strategic Advantage
Robert Walker – Principal, Surveys & Forecasts

Turning New Answers into New Solutions

Thursday, June 20, 2013
12:30PM - 2:30PM EST

Registration
To attend in-person or via webcast, register via My ARF »
If you have any questions, please contact Zena Pagán at forums@thearf.org.


Agenda
12:30–12:40PM ET

Welcome, Introductions and ARF Updates
William Cook – EVP, Research and Standards, ARF
Zena M Pagán – Forums and Ogilvy Awards Manager, ARF

12:40–12:45PM ET
Agenda Overview
Gian Fulgoni – Executive Chairman & co-Founder, comScore

12:45–1:00PM ET
Behaving Badly: Implications of Respondent Behavior on Data Quality across Sample Sources and Providers

  • How sub-optimal respondent behaviors compare across providers and sample sources.

  • Searching for an optimal combination of indicators to help improve data quality (speeding, straightlining, trap failures, response patterns, and primacy effects).

Randall K. Thomas – VP, Online Research Methods, GfK

1:00–1:15PM ET
Will the Real Data Set Please Stand Up?
Understanding the results of various machine and ID validation techniques
Melanie Courtright – VP, Research Services, Research Now

1:15–1:30PM ET
Truth or Consequences: The Use of Benchmarks
Critical to our analysis of the FOQ 2 is having a way to keep score – benchmarks. Team 1’s preliminary report will bring to light how well FOQ 2 measures mesh with the benchmarks.
Steve Gittelman – President, Sample Source Auditors, a division of Mktg, Inc.

1:30–1:45PM ET
Some of these Things are Just like Others
Certain online sample sources produce results that are similar to other online sample sources while being meaningfully different than other sources. This talk sets out some of the characteristics of sample sources that produce similar results.  
John Bremer – Chief Research Officer, Toluna

1:45–2:00PM ET
What Respondents Wish Surveyors Understood
Suppose respondents were given more control of the research process, what would they change?
Robert Walker – Principal, Surveys & Forecasts

2:00–2:25PM ET
Panel Discussion
John Bremer – Chief Research Officer, Toluna
Melanie Courtright – VP, Research Services, Research Now
Steve Gittelman – President, Sample Source Auditors, a division of Mktg, Inc.
Randall K. Thomas – VP, Online Research Methods, GfK
Robert Walker – Principal, Surveys & Forecasts

2:25–2:30PM ET
Next Steps and Adjournment
William Cook – EVP, Research and Standards, ARF

“Respondent Motives – Do you really know them?” Another FOQ 2 Production

Tuesday, April 23, 2013
2:30PM - 4:30PM EST

 

Registration

To attend in-person or via webcast, register via My ARF »
If you have any questions, please contact Zena Pagán at forums@thearf.org.
 

Social Channels
Facebook Link: https://www.facebook.com/events/177119085772088
Twitter: @the_ARF; Hash tag #arfforums

Join us on April 23 as this upcoming meeting of ARF’s Research Quality Forum welcomes Pete Cape – SSI’s Global Knowledge Director and Bob Walker – Founder of Surveys & Forecasts.

We will begin with Pete who will discuss how we normally assess the impact of rewards by looking to maximize response rates for least cost. But, since response rates are universally poor anyway, and clients are always looking to improve costs, what is the impact of doing away with rewards altogether? In this presentation we examine the downside of removing rewards. Who notices? Is it a sustainable strategy or a trick that may backfire?

We will then continue with Bob with an early analysis of the ARF’s landmark Foundations of Quality 2.0 research showed meaningful differences in survey enjoyment across panel companies. Data also showed that monetary incentives, while important, may be less persuasive than previously thought. How are these related? How do non-monetary factors relate to survey enjoyment, the level of sub-optimal behavior, or various panel characteristics? As the survey research industry grapples with ways to engage respondents, what new strategies might be considered?

Agenda

2:30–2:40PM ET
Welcome, Introductions and ARF Updates
William Cook – EVP, Research and Standards, ARF
Zena M Pagán – Forums and Ogilvy Awards Manager, ARF

2:40–2:45PM ET
Agenda Overview
Gian Fulgoni – Executive Chairman & co-Founder, comScore

2:45–3:25PM ET (Presentation 2:45-3:15; Q&A 3:10-3:25PM)
Money for Nothing?
Pete Cape – Global Knowledge Director, SSI

3:25–4:05PM ET (Presentation 3:25-3:55; Q&A 3:55-4:05PM)
Survey Enjoyment: An Oxymoron?
Robert Walker – Principal and Founder, Surveys & Forecasts

4:05–4:25PM ET
Update on Other FoQ 2 Activity
George Terhanian – President North America, Toluna

4:25–4:30PM ET
Next Steps and Adjournment
William Cook – EVP, Research and Standards, ARF

4:30–6:00PM ET
Wine and Cheese Reception

 

Research Quality - FoQ 2: Learning has now Accelerated

Tuesday, February 26, 2013
2:30PM - 6:00PM EST

Registration
To attend in-person or via webcast, register via My ARF »
If you have any questions, please contact Zena Pagán at forums@thearf.org.

Download the upcoming Research Quality Forum meeting agenda

Join us on February 26 as the first meeting of ARF’s Research Quality Forum will unveil the first look at the FoQ 2 field study. At our upcoming George Terhanian, President of Toluna NA, John Ouren and Joan Treistman, President of The Treistman Group, will review the findings from the respondent selection, behavior and motivation, verification, the survey design and the dynamics and experiences. Then, we will have Bob Walker, Principal and Founder of Surveys & Forecasts, update the analytic plan and FoQ2’s next steps.

Agenda

2:30–2:40PM ET
Welcome, Introductions and ARF Updates
William Cook – EVP, Research and Standards, ARF

2:40–2:50PM ET
Agenda Overview
Anne Hedde – Managing Director, Schlesinger Associates

2:50–3:20PM ET
Summary of FoQ2 Field Facts
Randall Thomas – VP, Online Research Methods, GfK

3:20–3:50PM ET
First Look at Findings
George Terhanian – President, Toluna NA
John Ouren
Joan Treistman – President, The Treistman Group

3:50–4:20PM ET
Updated Analytic Plan and Next Steps
Robert Walker – Principal and Founder, Surveys & Forecasts

4:20–4:30PM ET
Q&A

4:30–6:00PM ET
Wine and Cheese Reception

 

Research Quality - Dec 11

Tuesday, December 11, 2012
12:00PM - 1:00PM EST

FoQ 2: From Data Stream to Knowledge Flow

Free to members and non-members!
This Research Quality Forum meeting will be 100% via Webex.

Download the Research Quality Presentation from the December 11, 2012 Forum.

The ARF’s largest-ever research initiative has coursed from Questions to Models to Designs to Data Collection.

Now the FOQ 2 sponsors and volunteers are laying out the priorities for the many analyses, reports, and learning modules to be shared with sponsors and eventually the industry.

  • Outreach Team’s early report on sponsor interviews
  • Research Issue Teams recommend early focus topics

Join us on December 11 at Noon EST to advance the paving of new knowledge avenues for Foundations of Quality 2.0.
 

Research Quality Forum - Draft Analytic Plan

Thursday, October 25, 2012
1:00PM - 3:00PM EST

FOQ 2 Design and Analytic Plan

Download the Research Quality Presentation from the October 25, 2012 Forum.

The value of the ARF’s largest-ever research initiative is growing clearer each day. The proposed revisions to the FoQ 2 research design are driving fresh insights and dialogue among a global body of survey research experts. Join our virtual forum meeting on October 25, and hear Bob Walker, the Principal & Founder of Surveys & Forecasts, LLC, and the rest of the team link the latest project design to a draft analytic plan.

Building on two years of hard work by the Research Quality volunteers, the pace of progress on Foundations of Quality 2.0 is accelerating and heartening. Come see what FoQ 2 will mean to you and your company. This Research Quality Forum meeting will be 100% via Webex!

Agenda

1:00–1:05PM (5 minutes)
Welcome and ARF Updates
William Cook – EVP, Research and Standards, ARF
Zena M Pagán – Forums and Ogilvy Awards Manager, ARF

1:05–1:20PM (15 minutes)
Agenda Overview and Introductions
Don Gloeckler – Chief Research Officer, ARF

1:20–1:35PM (15 minutes)
Revised Integrated Design for FOQ 2 Study
George Terhanian – President North America & Group Chief Strategy Officer, Toluna
Joan Treistman – Principal, The Treistman Group

1:35–2:00PM (25 minutes)
Proposed FOQ 2 Analytic Plan
Robert Walker, Principal & Founder, Surveys & Forecasts, LLC

2:00–2:10PM (10 minutes)
Proposed Survey Design & Goals
Randall Thomas – VP, Online Research Methods, GfK
Philip Garland – VP Methodology, Survey Monkey

2:10–2:40PM (30 minutes)
Attendee Discussion
Don Gloeckler – Chief Research Officer, ARF

2:40–2:55PM (15 minutes)
The FOQ 2 Communications Framework: It's Bracing!
Anne Hedde – Managing Director, Schlesinger Interactive, Schlesinger Associates

2:55–3:00PM
Next Steps and Adjournment
 

Research Quality - Inside FOQ 2: Field Launch Readiness

Tuesday, September 18, 2012
2:30PM - 4:30PM EST

Inside FOQ 2: Field Launch Readiness

Download the Research Quality Presentation from the September 18, 2012 Forum.

At this forum session, we will present a draft of the Preliminary Integrated Foundations of Quality 2 Study Design. Previously reviewed by the leaders of the eight initiative teams and the Foundations of Quality 2 (FoQ 2) sponsors, the study evaluates the impact of eight factors on survey response quality and translates those quality impacts into improved research practices.

Sponsors and attendees will then have the opportunity to comment on the draft, and finally, we will learn about the path to integrating the FoQ 2 study design.

Featured Speakers:

  • Anne Hedde – Managing Director, Schlesinger Interactive, Schlesinger Associates
  • John Ouren – EVP & General Manager, Zoom Businesses, SurveyMonkey
  • George Terhanian – President, North America, Group Chief Strategy Officer, Toluna
  • Randy Thomas – VP, Online Research Methods, GfK

Agenda

2:30–2:35PM
Welcome and ARF Updates
William Cook – EVP, Research and Standards, ARF
Zena M Pagán – Forums and Ogilvy Awards Manager, ARF

2:35–2:50PM
Agenda Overview and Introductions
Gian Fulgoni – Chairman & co-Founder, comScore
Don Gloeckler – Chief Research Officer, ARF
William Cook – EVP, Research and Standards, ARF

2:50–3:10PM
Preliminary Integrated FOQ 2 Study Design
George Terhanian – President, North America, Group Chief Strategy Officer, Toluna
John Ouren – EVP & General Manager, Zoom Businesses, SurveyMonkey

3:10–3:20PM
Sponsor Comments

3:20–3:30PM
Attendee Comments
Gian Fulgoni – Chairman & co-Founder, comScore
Don Gloeckler – Chief Research Officer, ARF

3:30–3:50PM
Survey Drafts
Randy Thomas – VP, Online Research Methods, GfK

3:50–4:10PM
Path to FOQ 2 Implementation
Anne Hedde – Managing Director, Schlesinger Interactive, Schlesinger Associates

4:10–4:30PM
Next Steps and Adjournment
 

ARF Foundations of Quality 2

Thursday, January 26, 2012
2:30PM - 4:30PM EST

FOQ 2 Kickoff to Continue the Industry Leading Work on Online Research Quality

Invitation to Dinner and Research Quality Forum

You are cordially invited to attend the ARF’s next Research Quality Forum meeting, January 26, 2:30-4:30 p.m. EST.  Following, at 6:00PM, the TrueSample team will host cocktails and dinner for attendees, hosted by John Ouren (TrueSample) and Dave Goldberg (Survey Monkey).
Please note that dinner is open to forum attendees only.

Research Quality Forum
January 26, 2012 2:30–4:30 PM EST
Open to members and non-members

Registration
To attend the Forum meeting in person or virtually, register via My ARF » If you have any questions, please contact Zena Pagán at forums@thearf.org.

ARF Foundations of Quality 2.0 Pushes Ahead with Integration of Eight Initiatives

Please join leaders of the eight FoQ2 Initiatives to review the integration plan and share your thoughts on how it might be made even more effective and efficient.  Leaders of the eight FOQ 2 Initiatives [Click Here] will be meeting prior to the January 26 Forum meeting to produce a strategy for integrating their separate initiatives into a coherent program for advancing online research quality across the industry. They look forward to your participation and thoughts.

The TrueSample team invites forum attendees to attend cocktails and dinner following the ARF Foundations of Quality 2 Integration meeting, hosted by John Ouren of TrueSample, and Dave Goldberg, CEO of SurveyMonkey.  Cocktails will be served starting at 6:00 p.m.  We hope you can join the SurveyMonkey leadership, the TrueSample team, and other important global leaders in online data quality for the forum and dinner!  

Please click on the link below to RSVP! (Please note that the below link is the RSVP for dinner and cocktails ONLY. You must register for the forum meeting separately through My ARF.)

Dinner Registration:  http://truesample.wufoo.com/forms/registration-for-dinner-in-ny-january-26th/

Forum Registration:  http://my.thearf.org           

 

Research Quality - FOQ 2: Building the Teams and Fueling the Engines

Tuesday, November 08, 2011
2:30PM - 5:30PM EST

Research Quality Forum

Download the Research Quality Presentation from the November 8, 2011 Forum.

FOQ 2: Building the Teams and Fueling the Engines

In the September Research Quality Forum meeting FOQ 2 was launched with the objective of evaluating the impact of each of the following on survey response quality:

  1. Respondent selection procedures
  2. Post-hoc weighting procedures
  3. Procedures to reduce/eliminate undesired within-survey behavior
  4. Identity verification (and exclusions based on non-identification)
  5. Survey participation frequency
  6. Respondent motivations and incentives
  7. Sample sources (panel, river, community, etc.)
  8. Router sample optimization practices (impact on sample and survey results)

Now we encourage you to attend the forum and to push ahead with defining mission, priorities and process of FOQ 2.

FOQ 2 – People Powered

  • Why FOQ 2?
  • Why now?
  • How will FOQ 2 build on FOQ 1 and benefit the industry, both buyers and sellers?
  • Proposed FOQ 2 timetable
  • How should we prioritize the current 8 objectives if we need to make priority calls due to resource constraints?

For each of the objective areas, a committee chair will review a draft proposal, answer questions, gather suggestions, and accept volunteers to join the team.
 

Building on the Foundation - The debut Meeting for the Research Quality Forum's 2011 Research-on

Monday, September 26, 2011
2:30PM - 4:30PM EST

Download the Research Quality Presentations here and the Research Quality Notes from the September 26, 2011 Forum.

The 2008-09 Foundations of Quality provided the industry with critical answers and actionable steps forward for Online Survey Research quality. However, several areas were not addressed, and now the Research Quality Forum’s Research-on-Research Committee and Router Committee are prepared to roll out their proposals to the industry for new critical research programs.

Your participation in this kickoff meeting for the Forum’s new initiative is vital. Please arrange your calendar now to be at this meeting in person. We will notify you when we have posted the committees’ proposals on the ARF website next month. Please discuss the proposals with your colleagues and come to the Forum meeting prepared to offer your company’s support and recommendations.

What Online Survey Research-on-Research Does the Industry Need?

Thursday, April 07, 2011
11:00AM - 1:00PM EST

A Report from University of Michigan’s Institute for Survey Research
Speakers:
* Fred Conrad
* Dave Vannette

"How Do We Ensure the Quality of Routed Surveys?"
Panelists from the new Router Committee:
* Nancy Brigham, Ipsos
* Pete Cape, Survey Sampling International
* Michael Fallig, GfK
* Bob Fawson, Opinionology
* Chuck Miller, DMS Insights, uSamp
* Nitin Patel, Toluna
* Efrain Ribeiro, Lightspeed Research

Knowledge Briefs

Tuesday, January 11, 2011
2:30PM - 4:30PM EST

Foundation of Quality Knowledge Briefs and their respective Executive Summaries are now available. Knowledge briefs are available free to ARF Members and to non-members on My ARF. Download under “Publications” at my.thearf.org.

Taking 3 Steps along the Quality Diving Board

Join Reg Baker and a Reaction-Proaction Panel in discussing the AAPOR Task Force Report on the Uses and Misuses of Online Panels, and steps that the RQC can take to address urgent needs re:Representivity, River Sampling, and Routers.

Featured Speaker: Reg Baker, Market Strategies, International and member AAPOR Online Panel Task Force

Reaction-Proaction Panelists will be:
Mitch Eggers, GMI
Steve Gittelman, Mktg Inc.
Jackie Lorch, Survey Research International
George Terhanian, Toluna

Bring your best ideas for advancing our understanding of the 3 Rs (Representivity, River Sampling, and Routers), and join in the discussion.

ARF Research Quality Council Charter

The mission of the ARF Research Quality Council is to:

  • Provide a forum for thought leaders to identify and discuss a broad range of issues and improvement opportunities related to research quality.
  • Create opportunities for interested parties to collaborate in the creation and evaluation of potential solutions to research quality issues.
  • Recognize advances and innovations that contribute to improving the overall quality of consumer and market research globally.
  • Facilitate the articulation, integration and dissemination of research quality knowledge in the form of ARF Industry Standards and Best Practices.

Why is this Council needed?
In the face of the escalating pressure for faster and cheaper research, the quality of research methodologies, processes and deliverables is increasingly at risk as new approaches and opportunities to acquire and manage consumer and market information – many not validated – explode on the scene. In particular, research must establish quality standards for the emerging platforms, social media, digital, mobile that will once again redefine how companies tap into their consumers and shoppers to drive insights.
What is to be accomplished by this Super Council?

The council will offer a forum for sharing and possibly commissioning new research-on-research and new solutions to quality issues and will address practices as well as methods.  By identifying, creating, evaluating and sharing best practices, the RQC will become a powerful catalyst for advancement.

What is the benefit to ARF members and to the industry?
ARF member companies will demonstrate leadership in their industries as their gains in research quality translate into improved decision making. Research buyers and sellers will have a stronger foundation on which to match client quality needs with supplier quality assurance.

What issues will be explored by the RQC?
The RQC will explore a broad range of research quality related topics, possibly including (but open to Council input and revision):

  • The unique challenges to research quality in online research and advances in method and practices
  • The relationship between sample representativeness and total research quality
  • Social media as a source of representative consumer insights
  • Impact of real-time metrics on decision quality
  • Challenges to research quality with mixed-mode data collection and best practices
  • Fraud in research process including interviewer fraud
  • Challenges and contributions to quality of global research including ISO certification
  • Contributions of neuro- and bio-metrics to insight and decisions

What will be the structure of this Super Council?
The RQC will be guided by a highly-credentialed Steering Committee, driven by individual working committees (to be determined), including:

  • Online Research Quality – Research on Research
  • NeuroStandards Collaboration

Meeting/Registration

  • Initially, the RQC will meet bimonthly and subsequently on a quarterly basis.
  • Council meetings will be open to ARF members only.
  • Meetings will be listed in Eastern Time (US) and held in the ARF New York City office unless otherwise noted.


ARF Research Quality Initiatives

The Research Quality Council was launched as the Online Research Quality Council on June 17, 2008. It led the largest research-on-research activity in ARF's history, entitled the Foundations of Quality Initiative which was kicked  off by sending five RFPs to over 600 executives at 400 plus companies and academic institutions.

Industry leaders representing both buyers and sellers of online research have joined together to address issues affecting online research quality and accountability. The council’s mission is to develop a set of standards for research buyers and sellers to follow. The RQC is enlisting support from the global market research industry.
Three current initiatives include:

To address quality of online research, the RQC will seek to promote knowledge sharing from across the industry by working in concert with sister associations:

  • American Marketing Association (AMA)
  • Council of American Survey Research Organizations (CASRO)
  • Council for Marketing and Opinion Research (CMOR)
  • ESOMAR International Organization for Standardization (ISO)
  • Interactive Marketing Research Organization (IMRO)
  • Market Research Association (MRA)

To find out how you can contribute to the ARF's ongoing research quality efforts, simply email Bill Cook at Bill@theARF.org.

Forum's mission

The mission of the Research Quality Forum (RQC Forum) is to:

* Provide a forum for thought leaders to identify and discuss a broad range of issues and improvement opportunities related to research quality.

* Create opportunities for interested parties to collaborate in the creation and evaluation of potential solutions to research quality issues.

* Recognize advances and innovations that contribute to improving the overall quality of consumer and market research globally.

* Facilitate the articulation, integration and dissemination of research quality knowledge in the form of ARF Industry Standards and Best Practices.

ARF Forum sessions are closed, member only sessions and cannot be covered by the press without prior approval from the ARF President and CEO. Learn more about the benefits of ARF membership.

Meetings are listed in Eastern Time and held in our New York City office unless otherwise noted.

The Advertising Research Foundation

432 Park Ave South, 6th Floor
New York, NY 10016

ARF Forums Overview
Download ARF Forums Overview