Doc. no.:  N1631=04-0071
Date:         11 April 2004
Project:     Programming Language C++
Reply to:   David Abrahams <dave@boost-consulting.com>
                 Beman Dawes <bdawes@acm.org>
                 Jeremy Siek <jsiek@osl.iu.edu>

Electronic Review Process

Introduction
Process Overview
Prerequisites
What to include in Review Comments
Results
Notes for Proposal Authors
Notes for Review Managers
FAQ

Introduction

This document describes an electronic review process designed to augment the physical review process used by the C++ committee's working groups (WG's) at meetings to reach a consensus.

Under ISO, national body, and committee rules, official action on an issue or proposal is taken only by formal vote of the full committee.

To ensure that proposals which come before the full committee are truly ready for a formal vote, working group review occurs among domain experts meeting physically. Proposals are only brought forward to the full committee when the WG chair determines that sufficient working group consensus exists to warrant a full committee vote.

The committee wishes to extend this process to allow reviews to occur electronically between meetings. The intent is to improve the process by ensuring:

• More eyes on each proposal.
   
• Faster review/response cycle.
   
• More work done between meetings and fewer stalls waiting for review responses.
   
• Work is not completed at the last minute before a mailing deadline.
   
• Decisions are not made in haste; all arguments, especially those of domain experts, are given time to "gestate" between meetings.

A separate "review" mailing list reflector will be set up to encourage wide participation in electronic reviews.

Regardless of whether review occurs electronically between meetings or in an actual working group meeting, final acceptance of any proposal always requires a formal vote by the full committee.

Process Overview

1. Proposal authors request on the review reflector that a review be scheduled. WG chairs may also nominate issues (with their proposed resolutions) for electronic review.
2. The appropriate WG chair appoints a review manager.
3. The review manager coordinates the review schedule with authors and WG chairs via private email. Reviews are usually scheduled to last at least 14 days, to allow full participation, but may be shorter or longer as experience and the needs of specific proposals dictate.
4. The review manager begins the review period by posting a notice of the review, including a summary of the proposal, on both the review reflector and appropriate WG reflector.
5. Review comments are posted by review reflector participants.
6. After the conclusion of the review period, the review manager posts results on the review and WG reflectors.
7. At the next meeting, the WG chair in consultation with the working group makes the final determination to submit the proposal to the full committee for a formal vote.

Prerequisites

• Enough preliminary discussion has occurred to make it clear that the proposal has significant interest and support.
   
• The document to be reviewed normally must contain full wording for any proposed change, written in standardese, and identifying the exact document to be changed (WP, TR, etc.) and location for the change. WG chairs may grant exceptions for reviews of incomplete proposals if such a review would expedite WG progress.
   
• Only numbered committee documents are reviewed.
   
• The document to be reviewed must be transmitted to the committee vice-chair for inclusion in the next mailing.
   
• The document to be reviewed must be available on a web site. FTP is discouraged because it is often blocked by corporate firewalls. [Comment: It would be nice to have a common web site to post proposals, but it would have to be one that could be updated by multiple committee members so that no one person could become a bottleneck.]

What to include in Review Comments

A reviewer's comments may be brief or lengthy, but basically the review manager needs your evaluation of the proposal. You can even vote with no additional commentary, though comments are valuable and strongly encouraged. If you identify problems along the way, please note if they are minor, serious, or showstoppers.

Here are some points you might want to touch on in your review:

• What is your evaluation of the motivation and potential usefulness of the proposal?
   
• What is your evaluation of the concepts and overall design underlying the proposal?
   
• What is your evaluation of the proposed wording? Is the standardese relatively clear?
   
• What problems did you detect?
   
• Are there any other suggested improvements?
   
• Are there any questions for the authors?
   
• How much effort did you put into your evaluation? A glance? A quick reading? In-depth study?
   
• Are you knowledgeable about the problem domain?

And finally, every review should answer this question: "Should the proposal be accepted by the committee?"  Be sure to say this explicitly so that your other comments don't obscure your overall opinion. It may also be helpful if your answer is complete enough to give guidance for further work. Possible responses:

• I support the proposal as written.
• I support the proposal, but feel it needs to be revised to address issues raised in review.
• I do not support the proposal. I might possibly support a future revision that made major changes to address issues raised in review.
• I do not support the proposal and feel that it has enough problems that I am unlikely to support even a much revised version.

Results

At the conclusion of the review period, the review manager will post a message with the final tallies of the reviewer's votes and a summary of the main issues that were raised during the review. This vote plays the same role as the straw polls that occur during working group meetings; it provides a mechanism for measuring consensus. As always, the final determination of whether or not sufficient consensus exists to warrant forwarding a proposal to the full committee is made by working group chairs in consultation with their WG's.

If there are suggestions or conditions that must be met in a revised version of a proposal, they should be stated in the review summary.

Notes for Proposal Authors

A proposal should remain stable during the review period; it will just confuse and irritate reviewers if a revised document appears during a review period.

Authors should respond to reviewer's detailed comments. For example, if a reviewer suggests some change in the proposal, the authors might:

• Disagree, and give rationale.
• Be undecided, and ask for other views.
• Agree, and indicate plans to make the change in a revised paper.

Notes for Review Managers

Before a proposal can be scheduled for electronic review, someone not connected with it must volunteer to be the "Review Manager" for the proposal.

The review manager for an electronic review performs similar functions to a working group chair during in-person working group reviews; they coordinate the process and tally the votes for whether or not enough consensus exists to forward the proposal to the full committee for a formal vote. They also mediate if discussion becomes overheated.

It is expected that working group chairs may act as review managers for proposals relevant to their working group. WG chairs may ask someone else to act as review manager when the chair is the proposal's author, or to spread the workload.

The review manager:

• Checks the proposal to make sure the prerequisites are met.
• Finalizes the schedule with the working group chair and the submitter .
• On the first day of the review, posts a notice of the review schedule on both the review reflector and the working group reflector.
  • The notice should include a brief description of the proposal, to let readers know if it is a proposal they are interested in reviewing.
• Urges people to do reviews if they aren't forthcoming.
• Follows review discussions regarding the proposal, moderating or answering questions as needed.
• After the review period ends, posts a message on both the review reflector and the working group reflector informing members of the review results.

In other words, it is the review manager's responsibility to make sure the review process works smoothly.

FAQ

Who can participate in an Electronic Review? Anyone with access to the review reflector. As with any committee reflector traffic, committee officers monitor discussion to ensure acceptable use. And as with other reviews, WG chairs factor in any divergence between straw voters and voting members when determining if enough consensus exists to warrant a full committee vote.

Why are the review periods so short/long? The Boost experience with electronic reviews is that there is a tension between the desire to give reviewers plenty of time to prepare reviews, and practical factors such as reviewers delaying until near the end of the review period, workload for submitters and review managers, and lack of discussion focus when review periods are lengthy. The length of review periods can be adjusted as committee experience with electronic reviews grows.

Why aren't the reviews scheduled right after between meeting mailings? The initial strategy is to spread the committee's workload out over time, rather than concentrate it in a few specific time periods, and to decouple reviews from other events like mailings to simplify the process. Schedule timeslots can be adjusted as committee experience with electronic reviews grows.

Why aren't documents to be reviewed posted on the committee web site? Logistics. We don't want the webmaster to become a bottleneck or to be overloaded. Perhaps committee web site procedures can be worked out, or a permanent committee Wiki can be established, but we don't want to delay progress until these administrative details are resolved.

The prerequisites are tilted toward mature proposals. Won't this inhibit discussion of "half-ready" proposals? Yes, and that is intended. The individual working group reflectors are the appropriate place for discussions of "good ideas", "trial balloons", "half-ready" proposals, and the like. Electronic reviews involve enough expenditure of committee effort that they are best reserved for mature proposals, or for less-mature proposals important enough to justify electronic review.

How can an electronic review replace all meeting review time? It can't. The goal of electronic reviews isn't to eliminate all meeting review time, but rather to allow  meetings to be more productive. Electronic review allows authors to refine proposals, meeting participants to vote in confidence, and with minimal time be spent in meetings on review aspects which can be handled well electronically.

Revised 11 April, 2004