Online Research Quality Council Previous Meetings
SEPTEMBER 16, 2008
Knowledge Sharing and Proposed Metrics
P&G’s Kim Dedeker hosted a critical meeting of the ORQC at The Cincinnatian Hotel in Cincinnati, Ohio. Kim has been one of the leading voices in the pursuit of online research quality and was recently named chair of the ORQC’s Client Advisory Board.
Leading research buyers and sellers have joined the ORQC’s Foundations of Quality research-on-research initiative, a highly collaborative effort that is seeking to build on the work underway by leading research associations in the US and Europe. We urge you to come add your experience, passion, and commitment.
PRESENTATIONS
Sample Factors that Influence Data Quality (PDF, 120kb)
General Mills Data Quality Update (PDF, 686kb)
The Foundations of Quality Project (PDF, 96kb)
On-Line Research Quality Council: Interim Proposals from the Quality Metrics Committee (PDF, 124kb)
Survey Programming and Reporter (PDF, 240kb)
JUNE 17, 2008
Deduping Panel Participants
On June 17th for brief presentations were made by four representatives of companies who replied to the RFP for Panel Duplication and Measurement of Respondent Performance: Market Tool, OTX, Peanut Labs, Sigma Validation.
The Defining Quality Committee co-chairs gave a detailed presentation on the "Foundations of Quality" initiative they have launched. For details see ORQC Five RPFs.
PRESENTATIONS
Sigma Validation: Getting Started with The Duplicate Respondent Search – Weber (PDF, 295kb)
Peanut Labs: OptimusID – Digital Fingerprinting for Market Research – Morgan (PDF, 347kb)
FEBRUARY 12, 2008
The Continuing March to Improve Online Research Quality
Download Meeting Notes (PDF, 296kb)
Ron Gailey, Director of Consumer Insights at Washington Mutual, presented “Understanding ‘Undesirable’ Survey Respondents.”
PRESENTATIONS
WaMu: Understanding ‘Undesirable’ Survey Respondents – Gailey (PDF, 512kb)
COMMITTEE REPORTS:
Define Quality
- Research buyers survey on issues and value points – discussion of topline results.
- Overview of bibliography reporting R-on-R on multi-panel membership and respondent engagement.
- Brief presentation by IPSOS and TNS on their joint work on impact of heavy responders.
- Brief presentation by Harris Interactive on multi-panel membership impact.
Quality Accountability
Online Research Reliability – Progress report on synthesizing existing R-on-R studies.
Quality Metrics
Report on metrics currently in use leading online research panels to monitor panel quality.
Outreach
Potential funding needs for R-on-R studies under consideration plus around the Industry Panelists’ Focus Group concept.
DECEMBER 11, 2007
Knowledge Sharing
Download Minutes (PDF, 67kb)
The meeting continued ORQC knowledge sharing on research-on-research to drive increased quality. Three speakers reported valuable research on online research:
- Jackie Lorch, Survey Sampling International
- Patrick Glazer, CMOR
- Dana Stanley, Research Now
This was followed by the Committee co-chairs sharing updates on the plans and progress of their respective working committees.
PRESENTATIONS
SSI: Creating a Respondent Cooperation Playbook – Lorch (PDF, 554kb)
CMOR: New Dimensions of Quality in Online Panels – Glazer (PDF, 2.1mb)
Research Now: Innovations in the Active Research Experience:
Flash Tools and Their Impact on Data Quality – Stanley (PDF, 838kb)
NOVEMBER 13, 2007
Knowledge Sharing Continued
Updates on the Four Working Committees
Download Complete Minutes (PDF, 296kb)
The first half of the November 13th ORQC meeting featured a panel discussion with five speakers from the Client Advisory Board: Tom Evans (ESPN), Leroy Leiker (Bank of America), Keith Stevens (Kraft), Stan Sthanunathan (Coca-Cola), and Jeffrey Todder (CitiBrands). Three topics addressed were the expectations of online research providers, the advertiser commitment to improve fit between expectations and deliverables, and the actions that advertisers would like to see taken both immediately and in the near-to-medium term. During the discussion, several significant challenges to improving online research quality were identified – including proper survey design, collecting and analyzing data. Furthermore, several on-going issues relative to the conduct of on-line surveys were discussed: reduced respondent engagement due to multi-tasking; the inappropriateness of demands placed on respondents by lengthy and convoluted surveys; and the lack of back-checking of the data. It was agreed that improving the quality of online research requires open communication and transparency, while the data collected is in need of reliability and consistency.
In the second half of the meeting the co-chairs of the four working committees discussed their ideas and plans for progress. The Define Quality committee reached a consensus that the overarching mission is to put the "science" back into market research. They’ve developed goals to identify client expectations about quality and define key dimensions of quality tied either to the research process or to client business problems. They agreed that they need both minimum and maximum standards since there are gradations of quality. The Accountability committee (formerly Quality Matters) agreed that “quality matters” because companies are making multi-million dollar business decisions based on research findings, while flawed findings can lead to wrong decisions. The committee recognized that accountability for quality is an increasingly important issue for the client community, and thus renamed the Committee from “Quality Matters” to “Accountability.” The Quality Metrics group agreed to study what other professional bodies have come up with in terms of both 'quality issues' and corresponding metrics. To the group, it is essential that the ORQC evolve a system of multi-tiered metrics that can be recognized and used globally. It will include fast production of interim metrics that can be utilized within, say, 3-6 months, followed by developing a definitive set, based on the upcoming research on research. Finally, the Outreach committee agreed that they will likely need a better understanding of what the other committees are setting as goals for the group to be able to establish a more concrete set of deliverables.
The next Online Research Quality Council meeting will be on Tuesday, December 11, 2007, from 2:00 – 4:30 PM. Three speakers will report valuable research on online research: Jackie Lorch from Survey Sampling International, Patrick Glaser from CMOR, and Dana Stanley from Research Now; then Committee Progress Reports will follow.
The Steering committee and the Client Advisory Board will hold a joint meeting immediately prior to each of the upcoming ORQC meetings.
For questions regarding the Council, please email Inna Sokolyanskaya at Inna@thearf.org.
SEPTEMBER 10, 2007
Kickoff Meeting
The September 10th kickoff meeting of the ARF Online Research Quality Meeting involved 125 executives from a diverse set of companies including three dozen research buyer companies and a like number of research providers.
Robert Tomei lead the meeting with a presentation (see link below) sharing the proposed mission of the Council, its four committees, and its time table for producing knowledge on which to base the needed standards.
David Anthony, Global Research Analytics, Proctor & Gamble, spoke about the needs of online research buyers and acknowledged the steps being taken by the research provider firms which they are working with, and of the ARF in forming the ORQC.
To illustrate three basic directions being pursued to date, three research companies presented research on research which demonstrates the utility of approaches and tools in use now.Eric Wiegard, Chair of ISO Standards on Access Panels, outlined the ISO process and his committee's efforts.
- Jeffrey P. Miller – Chief Operating Officer, Burke, Inc.
- Patricia Graham – SVP, Client Service and Business Development, Knowledge Networks
- Dan Fitzgerald – Managing Director, Global Accounts and Services, Global Market Insite, Inc. (GMI)
PRESENTATIONS
Welcome – Barocci & Tomei (PDF, 136kb)
GMI: Panel Quality: Tips to Achieve Pure Sample, from Recruitment to Long-Term Engagement – Fitzgerald (PDF, 612kb)
Knowledge Networks: Using Known Benchmarks to Inform the Accuracy of Online Research – Graham (PDF, 164kb)
Burke: Panel Quality R&D Summary – Miller (PDF, 1.1mb)
ISO Standard on Access Panels – Wiegand (PDF, 336kb)