Featuring
Sherrell Mane – SVP Industry Services, Interactive Advertising Bureau,
Paul Lavrakas – IAB Consultant, Northern Arizona University &
Joseph Laszlo – Dir. of Research, Interactive Advertising Bureau
Knowledge sharing has been a critical success factor over the three years of ORQC growth. Today sharing is highly visible in the ARF Quality Enhancement Process (QeP) workshops and in the ongoing partnerships of research buyers and sellers. Through the workshops and the companies applying the QeP we continue to improve the QeP templates and process. One of the leading client companies will share learning from their quality initiatives – Procter and Gamble.
The need for new knowledge extends beyond the bounds of the original Foundations of Quality research, and the ORQC is planning for added research in Foundations of Quality II. The planning for FoQ II will be discussed in the Council meeting. We will also hear about the areas of need pursed by our sister associations, the Interactive Advertising Bureau (IAB) and the Council of American Survey Research Organizations (CASRO).
Featured Speakers
Donald Gloeckler – Manager, Next Gen Research, Procter & Gamble
Paul Lavrakas, PhD
Peter Milla – Technical Director, CIRQ
Welcome and Introductions
Steve Coffey – Chief Research Officer, The NPD Group Inc.
Industry Solutions Committee
David Schmidt – Director Portfolio Management, The Coca-Cola Company
Bob Walker – President, Surveys & Forecasts, LLC
Outreach Committee
John Willard – Director, Market Research & Consumer Relations, Bayer Healthcare
Art Klein – EVP Director, MSW Research
Research-on-Research Committee
John Ouren – EVP & General Manager, Zoom Businesses, MarketTools Inc.
George Terhanian – President of Global Solutions, Harris Interactive Inc
The ARF Quality Enhancement Process (QeP) is a facilitation program to help clients, research firms, consultants, and panel suppliers engage in a structured conversation to improve online data quality. The process consists of a set of forms and templates that are used to establish critical metrics to help guide online data quality. This involves goal setting, training, deployment, and evaluation via a collaborative approach between buyers and sellers.
On January 19, we reported synthesized results of the QeP Pilot Test evaluations that were carried out by Coca-Cola, Unilever, Kraft, Capital One, Bayer, General Motors, Microsoft, and General Mills along with their major suppliers and research partners on specific research programs. This phase of pilot testing will result in a ‘sharing of experience’ via the ORQC, and will define future versions of the QeP.
Combined Presentations:
QePProgram Evaluation
Sexy Questions, Dangerous Results?
ORQC Update on the ORQC and Quality Enhancement Process Pilot Testing
Steve Coffey – Chief Research Officer, The NPD Group Inc.
Stan Sthanunathan – VP Marketing Strategy & Insights, The Coca-Cola Company
Joel Rubinson – Chief Research Officer, The ARF
Panel – Contemporary Science of Online Data Quality
Panel Moderated by Bob Lederer – Editor and Publisher, Research Business Report
George Terhanian – President of Global Solutions, Harris Interactive, Inc.
Charles DiSogra – Chief Statistician, Knowledge Networks
Steve Gittelman – President, Mktg, Inc.
Doug Rivers, CEO, Yougov America
This special meeting was open to all – ARF Members, Non-Members and Press.
We introduced the ARF Quality Enhancement Process – Version 1.0. This is the culmination of a million dollar research project, called The ARF Foundations of Quality Study, and the input, expertise, and collaboration of professionals from both the client and supplier side, under the auspices of the ARF.
The QeP Panel Template, Sample Template, and ARF Quality Enhancement Process Guidebook are available on My ARF under Publications.
The ARF Quality Enhancement Process
Combined Presentations:
The Industry Solutions Committee
Foundations of Quality Study
Analytic Plan for the Foundations of Quality Project
The Foundations of Quality Project (PDF, 64kb)
Online Panel Metrics Interim Recommendations Highlights (PDF, 60kb)
Releasing FoQ Results: Lessons Learned from the Netherlands (PDF, 192kb)
ORQC - Client and Knowledge Driven Standards and Metrics (PDF, 48kb)
P&G’s Kim Dedeker hosted a critical meeting of the ORQC at The Cincinnatian Hotel in Cincinnati, Ohio. Kim has been one of the leading voices in the pursuit of online research quality and was recently named chair of the ORQC’s Client Advisory Board.
Leading research buyers and sellers have joined the ORQC’s Foundations of Quality research-on-research initiative, a highly collaborative effort that is seeking to build on the work underway by leading research associations in the US and Europe. We urge you to come add your experience, passion, and commitment.
Sample Factors that Influence Data Quality (PDF, 120kb)
General Mills Data Quality Update (PDF, 686kb)
The Foundations of Quality Project (PDF, 96kb)
On-Line Research Quality Council: Interim Proposals from the Quality Metrics Committee (PDF, 124kb)
Survey Programming and Reporter (PDF, 240kb)
On June 17th for brief presentations were made by four representatives of companies who replied to the RFP for Panel Duplication and Measurement of Respondent Performance: Market Tool, OTX, Peanut Labs, Sigma Validation.
The Defining Quality Committee co-chairs gave a detailed presentation on the "Foundations of Quality" initiative they have launched. For details see ORQC Five RPFs.
Sigma Validation: Getting Started with The Duplicate Respondent Search – Weber (PDF, 295kb)
Peanut Labs: OptimusID – Digital Fingerprinting for Market Research – Morgan (PDF, 347kb)
Download Meeting Notes (PDF, 296kb)
Ron Gailey, Director of Consumer Insights at Washington Mutual, presented “Understanding ‘Undesirable’ Survey Respondents.”
WaMu: Understanding ‘Undesirable’ Survey Respondents – Gailey (PDF, 512kb)
COMMITTEE REPORTS:
Define Quality
Quality Accountability
Online Research Reliability – Progress report on synthesizing existing R-on-R studies.
Quality Metrics
Report on metrics currently in use leading online research panels to monitor panel quality.
Outreach
Potential funding needs for R-on-R studies under consideration plus around the Industry Panelists’ Focus Group concept.
Download Minutes (PDF, 67kb)
The meeting continued ORQC knowledge sharing on research-on-research to drive increased quality. Three speakers reported valuable research on online research:
This was followed by the Committee co-chairs sharing updates on the plans and progress of their respective working committees.
SSI: Creating a Respondent Cooperation Playbook – Lorch (PDF, 554kb)
CMOR: New Dimensions of Quality in Online Panels – Glazer (PDF, 2.1mb)
Research Now: Innovations in the Active Research Experience: Flash Tools and Their Impact on Data Quality – Stanley (PDF, 838kb)
Download Complete Minutes (PDF, 296kb)
The first half of the November 13th ORQC meeting featured a panel discussion with five speakers from the Client Advisory Board: Tom Evans (ESPN), Leroy Leiker (Bank of America), Keith Stevens (Kraft), Stan Sthanunathan (Coca-Cola), and Jeffrey Todder (CitiBrands). Three topics addressed were the expectations of online research providers, the advertiser commitment to improve fit between expectations and deliverables, and the actions that advertisers would like to see taken both immediately and in the near-to-medium term. During the discussion, several significant challenges to improving online research quality were identified – including proper survey design, collecting and analyzing data. Furthermore, several on-going issues relative to the conduct of on-line surveys were discussed: reduced respondent engagement due to multi-tasking; the inappropriateness of demands placed on respondents by lengthy and convoluted surveys; and the lack of back-checking of the data. It was agreed that improving the quality of online research requires open communication and transparency, while the data collected is in need of reliability and consistency.
In the second half of the meeting the co-chairs of the four working committees discussed their ideas and plans for progress. The Define Quality committee reached a consensus that the overarching mission is to put the "science" back into market research. They’ve developed goals to identify client expectations about quality and define key dimensions of quality tied either to the research process or to client business problems. They agreed that they need both minimum and maximum standards since there are gradations of quality. The Accountability committee (formerly Quality Matters) agreed that “quality matters” because companies are making multi-million dollar business decisions based on research findings, while flawed findings can lead to wrong decisions. The committee recognized that accountability for quality is an increasingly important issue for the client community, and thus renamed the Committee from “Quality Matters” to “Accountability.” The Quality Metrics group agreed to study what other professional bodies have come up with in terms of both 'quality issues' and corresponding metrics. To the group, it is essential that the ORQC evolve a system of multi-tiered metrics that can be recognized and used globally. It will include fast production of interim metrics that can be utilized within, say, 3-6 months, followed by developing a definitive set, based on the upcoming research on research. Finally, the Outreach committee agreed that they will likely need a better understanding of what the other committees are setting as goals for the group to be able to establish a more concrete set of deliverables.
The next Online Research Quality Council meeting will be on Tuesday, December 11, 2007, from 2:00 – 4:30 PM. Three speakers will report valuable research on online research: Jackie Lorch from Survey Sampling International, Patrick Glaser from CMOR, and Dana Stanley from Research Now; then Committee Progress Reports will follow.
The Steering committee and the Client Advisory Board will hold a joint meeting immediately prior to each of the upcoming ORQC meetings.
For questions regarding the Council, please email Bill Cook at Bill@thearf.org.
The September 10th kickoff meeting of the ARF Online Research Quality Meeting involved 125 executives from a diverse set of companies including three dozen research buyer companies and a like number of research providers.
Robert Tomei lead the meeting with a presentation (see link below) sharing the proposed mission of the Council, its four committees, and its time table for producing knowledge on which to base the needed standards.
David Anthony, Global Research Analytics, Proctor & Gamble, spoke about the needs of online research buyers and acknowledged the steps being taken by the research provider firms which they are working with, and of the ARF in forming the ORQC.
To illustrate three basic directions being pursued to date, three research companies presented research on research which demonstrates the utility of approaches and tools in use now.Eric Wiegard, Chair of ISO Standards on Access Panels, outlined the ISO process and his committee's efforts.
Welcome – Barocci & Tomei (PDF, 136kb)
GMI: Panel Quality: Tips to Achieve Pure Sample, from Recruitment to Long-Term Engagement – Fitzgerald (PDF, 612kb)
Knowledge Networks: Using Known Benchmarks to Inform the Accuracy of Online Research – Graham (PDF, 164kb)
Burke: Panel Quality R&D Summary – Miller (PDF, 1.1mb)
ISO Standard on Access Panels – Wiegand (PDF, 336kb)
