Online Research Quality Council

ARF Launches Online Research Quality Initiative

Industry leaders representing both buyers and sellers of online research have joined together to address online research quality and accountability. The new council’s mission will be to develop a set of standards for research buyers and sellers to follow. The ORQC will enlist support from the global market research industry, and will be led by senior executives from both the buyer and seller communities. The council’s initial focus will be on online research panels.

NOTE: ORQC RFP DEADLINE EXTENDED TO JULY 23rd
Given the 4th of July holiday, we are extending the RFP deadline. No further deadline extensions will be made.
ARF Online Research Quality Issues Five RFPs »

Knowledge-based research initiatives will be led by three new ARF committees focusing on:

• How is quality defined? Defining Quality Committee »
• Does quality matter? Quality Matters Committee »
• What metrics can all stakeholders use to consistently assess the quality of the information derived from online research? Quality Metrics Committee »

The fourth working committee of The Council will develop funding and engage in outreach to prospective partners globally.

• Outreach & Funding Committee »

To address quality of online research, these action committees will lead new research and seek to promote knowledge sharing from across the industry including that from sister associations:

• AAPOR
• AMA
• CASRO
• ESOMAR
• ISO
• IMRO

Volunteers are requested for each of these areas of contribution. To join a committee simply email bill@theARF.org.

Upcoming Meetings

SEPTEMBER 16, 2008 • 2:00–4:30PM EDT

Topic to be decided.

DECEMBER 9, 2008 • 2:00–4:30PM EDT

Topic to be decided.

Previous Meetings

JUNE 17, 2008 • 2:00 – 4:30PM EDT

Deduping Panel Participants

Join us on June 17th for brief presentations by the finalist companies who replied to the RFP for Panel Duplication and Measurement of Respondent Performance.

In addition, progress reports from three of the ORQC Working Committees will be presented:

• Defining Quality
• Quality Metrics
• Reliability, also known as Accountability

PRESENTATIONS

Getting Started with The Duplicate Respondent Search (PDF, 120kb)
OptimusID - Digital Fingerprinting for Market Research (PDF, 686kb)

FEBRUARY 12, 2008 • 2:00–4:30PM EDT

The Continuing March to Improve Online Research Quality

Download Meeting Notes (PDF, 296kb)

Jeff Hunter, Director of Consumer Insights at General Mills, will be presenting, “Keeping Pace with the Consumer,” while Ron Gailey, Director of Consumer Insights at Washington Mutual, will be presenting “Understanding ‘Undesirable’ Survey Respondents.”

COMMITTEE REPORTS:

Define Quality

• Research buyers survey on issues and value points – discussion of topline results.
• Overview of bibliography reporting R-on-R on multi-panel membership and respondent engagement.
• Brief presentation by IPSOS and TNS on their joint work on impact of heavy responders.
• Brief presentation by Harris Interactive on multi-panel membership impact.

Quality Accountability
Online Research Reliability – Progress report on synthesizing existing R-on-R studies.

Quality Metrics
Report on metrics currently in use leading online research panels to monitor panel quality.

Outreach
Potential funding needs for R-on-R studies under consideration plus around the Industry Panelists’ Focus Group concept.

DECEMBER 11, 2007

Knowledge Sharing

Download Minutes (PDF, 67kb)

The meeting continued ORQC knowledge sharing on research-on-research to drive increased quality. Three speakers reported valuable research on online research:

• Jackie Lorch, Survey Sampling International
• Patrick Glazer, CMOR
• Dana Stanley, Research Now

This was followed by the Committee co-chairs sharing updates on the plans and progress of their respective working committees.

PRESENTATIONS

Creating a Respondent Cooperation Playbook (PDF, 554kb)

New Dimensions of Quality in Online Panels (PDF, 2.1mb)

Innovations in the Active Research Experience: Flash Tools and Their Impact on Data Quality (PDF, 838kb)

NOVEMBER 13, 2007

Knowledge Sharing Continued
Updates on the Four Working Committees

Download Complete Minutes (PDF, 296kb)

The first half of the November 13th ORQC meeting featured a panel discussion with five speakers from the Client Advisory Board: Tom Evans (ESPN), Leroy Leiker (Bank of America), Keith Stevens (Kraft), Stan Sthanunathan (Coca-Cola), and Jeffrey Todder (CitiBrands). Three topics addressed were the expectations of online research providers, the advertiser commitment to improve fit between expectations and deliverables, and the actions that advertisers would like to see taken both immediately and in the near-to-medium term. During the discussion, several significant challenges to improving online research quality were identified – including proper survey design, collecting and analyzing data. Furthermore, several on-going issues relative to the conduct of on-line surveys were discussed: reduced respondent engagement due to multi-tasking; the inappropriateness of demands placed on respondents by lengthy and convoluted surveys; and the lack of back-checking of the data. It was agreed that improving the quality of online research requires open communication and transparency, while the data collected is in need of reliability and consistency.

In the second half of the meeting the co-chairs of the four working committees discussed their ideas and plans for progress. The Define Quality committee reached a consensus that the overarching mission is to put the "science" back into market research. They’ve developed goals to identify client expectations about quality and define key dimensions of quality tied either to the research process or to client business problems. They agreed that they need both minimum and maximum standards since there are gradations of quality.  The Accountability committee (formerly Quality Matters) agreed that “quality matters” because companies are making multi-million dollar business decisions based on research findings, while flawed findings can lead to wrong decisions.  The committee recognized that accountability for quality is an increasingly important issue for the client community, and thus renamed the Committee from “Quality Matters” to “Accountability.”  The Quality Metrics group agreed to study what other professional bodies have come up with in terms of both 'quality issues' and corresponding metrics. To the group, it is essential that the ORQC evolve a system of multi-tiered metrics that can be recognized and used globally.  It will include fast production of interim metrics that can be utilized within, say, 3-6 months, followed by developing a definitive set, based on the upcoming research on research. Finally, the Outreach committee agreed that they will likely need a better understanding of what the other committees are setting as goals for the group to be able to establish a more concrete set of deliverables.

The next Online Research Quality Council meeting will be on Tuesday, December 11, 2007, from 2:00 – 4:30 PM.  Three speakers will report valuable research on online research: Jackie Lorch from Survey Sampling International, Patrick Glaser from CMOR, and Dana Stanley from Research Now; then Committee Progress Reports will follow.

The Steering committee and the Client Advisory Board will hold a joint meeting immediately prior to each of the upcoming ORQC meetings.

For questions regarding the Council, please email Inna Sokolyanskaya at Inna@thearf.org.

SEPTEMBER 10, 2007

Kickoff Meeting

The September 10th kickoff meeting of the ARF Online Research Quality Meeting involved 125 executives from a diverse set of companies including three dozen research buyer companies and a like number of research providers.

Robert Tomei lead the meeting with a presentation (see link below) sharing the proposed mission of the Council, its four committees, and its time table for producing knowledge on which to base the needed standards.

David Anthony, Global Research Analytics, Proctor & Gamble, spoke about the needs of online research buyers and acknowledged the steps being taken by the research provider firms which they are working with, and of the ARF in forming the ORQC.

To illustrate three basic directions being pursued to date, three research companies presented research on research which demonstrates the utility of approaches and tools in use now.

• Jeffrey P. Miller – Chief Operating Officer, Burke, Inc.
• Patricia Graham – SVP, Client Service and Business Development, Knowledge Networks
• Dan Fitzgerald – Managing Director, Global Accounts and Services, Global Market Insite, Inc. (GMI)

PRESENTATIONS

Welcome – Barocci & Tomei (PDF, 136kb)

Panel Quality: Tips to Achieve Pure Sample, from Recruitment to Long-Term Engagement – Fitzgerald (PDF, 612kb)

Using Known Benchmarks to Inform the Accuracy of Online Research – Graham (PDF, 164kb)

Burke Panel Quality R&D Summary – Miller (PDF, 1.1mb)

ISO Standard on Access Panels – Wiegand (PDF, 336kb)

Please add your learning to the ARF Online Research Quality Knowledge Base

The ARF created the Online Research Quality Council (ORQC) to improve the accountability of online research and to set standards and guidelines for research suppliers and buyers. To date, ten companies have shared knowledge. The information shared by these companies addresses a range of measures, causes, and effects relevant to online research quality.

You are encouraged to contribute to expanding that base of shared knowledge by submitting to us a summary of your research-on-research, analyses, case study, or whatever research-based learning you may have attained regarding online research quality. Our knowledge sharing program requires minimal effort from individual contributors and will benefit all of us. With your prompt response, we can achieve the goal of publication on November 23rd.

We are attaching a template to assist you in putting together your contribution to the knowledge base. Please let us know if you have any concerns or questions. To submit your knowledge base contribution, please email it to Inna Sokolyanskaya.

We sincerely thank you for your work on this important task.

ORQ Council Leadership

Steering Committee
Josh Chasin – Chief Research Officer, comScore
Renee Smith – Vice President, Panel Quality, Harris Interactive
Efrain Ribiero – COO, Ipsos Interactive Services
Jonathan Jephcott – Executive V.P., ViewsNet, Synovate
Steve Coffey – Chief Research Officer, The NPD Group
Robert Tomei – EVP & Global Director of Access Panels, TNS

Client Advisory Board
Bob Sanders – Sr Manager – Marketing Strategy, Measurement and Insights, Allstate Insurance Company
Leroy Leiker – Sr Vice President, Brand Marketing Research, Bank of America
John Willard – Director, Market Research & Consumer Relations, Bayer Consumer Care Division
James Mendelsohn – Vice President, Corporate Strategy, Capital One
Jeffrey Todder – Senior Vice President, Director of Advertising Analytics, CitiBrands
Stan Sthanunathan – Vice President of Global Knowledge & Insights, Coca Cola
Tom Evans – Vice President of Digital & Cross Media Research, ESPN
Keith Stevens – Sr. Brand Research Manager, Consumer Insights & Strategy, Kraft
Michelle Salazar – Sr. Director, Global Consumer & Business Insights, McDonald's
Clifford Thumma – Sr. Director, Team Leader, US Market Analytics, Pfizer, Inc.
Amy Raihill – Consumer Insights Manager, Shell
Jim Carter – AT&T, Executive Director, Advertising Rsch & Metrics
Kimberly Dedeker – Procter & Gamble, Vice President, Global Consumer & Market Knowledge
Robert Atencio – Wal-Mart Stores, Inc., Vice President, Insights and Customer Strategy

Define Quality
Justin Fromm – Sales & Strategy Research, ABC Television Network
Don Bruzzone – President, Bruzzone Research
Paul Shellenberg – Director of Sales, Bruzzone Research
Lew Alpert – CEO, Chilmark Digital
Thomas C. Evans – VP, Digital & Cross Media Research, ESPN
Jennifer Cripe – Sr. Research Analyst, Consumer Insights, Fox Interactive Media
Dan Fitzgerald – VP Americas and Global Accounts/Services, GMI
Renee Smith – Vice President, Panel Quality, Harris Interactive
Efrain Ribiero – COO, Ipsos Interactive Services
Birgi Martin – Research Development Director, Lightspeed Research
Chris Stevens – Chief Research Officer, Lightspeed Research
Jeff Stewart – Director, Panel Mgmt & Development, MarketTools, Inc.
Amanda Abry – Senior Interactive Research Manager, Meredith Corporation
Gordon Wyner – Executive Vice President, Millward Brown
Ellen Jacobs – Senior Analyst, Strategy & Evaluation, Salt River Project
Andrew Susman – CEO, Studio One Networks
Jackie Lorch – Product Manager, Survey Sampling Int'l
Mike Donatello – Director of Research, USA TODAY
Chad Bohnert – VP, Marketing and E-Commerce, Zogby International

Quality Matters
Leroy Leiker – Sr Vice President, Brand Marketing Research, Bank of America
Stan Sthanunathan – VP, Global Knowledge & Insights, Coca Cola
Josh Chasin – Chief Research Officer, comScore
Felicia Rogers – Executive Vice President, Decision Analyst
Chris Stevens – Lightspeed Research
Birgi Martin – Research Development Director, Lightspeed Research
Doreen Harmon – Vice President - Client Operations, Millward Brown
Steve McHugh – Senior Vice President - Client Operations, Millward Brown
Andrew Jeavons – Managing Director, Nebu USA, Nebu.com
John Toomey – SVP, Operations, The Nielsen Company

Quality Metrics
Patrick Glaser – Director of Respondent Cooperation, CMOR
Marjette Stark – Senior Vice President, DMS Research
Thomas C. Evans – VP, Digital & Cross Media Research, ESPN
Mitch Eggers – Chief Technology Officer, GM Panel Business Unit
Keith Stevens – Sr. Brand Research Manager, Consumer Insights & Strategy, Kraft
Nancy Kienzler – VP Research Operations, North America, Lightspeed Research
Geeta Mallya – Account Executive - Client Operations, Millward Brown
Bob Shullman – Senior Vice President, Monroe Mendelsohn Research
Ali A Moiz – Director & Founder, PeanutLabs.com
Jonathan Jephcott – Executive V.P., ViewsNet, Synovate
John Toomey – SVP, Operations, The Nielsen Company
Susan Frede – Online Access Panel Team, TNS
Chad Bohnert – VP, Marketing and E-Commerce, Zogby International

Outreach
Jeffrey Todder – Senior Vice President, Director of Advertising Analytics, CitiBrands
Lynnette McCarthy – Marketing Director, North America, Lightspeed Research
Steve McHugh – SVP, Millward Brown Client Operations, Millward Brown
Clifford Thumma – Sr. Director, Team Leader, US Market Analytics, Pfizer, Inc.
Steve Coffey – Chief Research Officer, The NPD Group
Robert Tomei – EVP & Global Director of Access Panels, TNS