Following is the Test Plan for interoperability,
comprised of two documents: the first part is updated
from a draft distributed at the New York meeting, and
the second part is from Jay Martin at Underscore
(unedited).
Test Plan
for Products Implementing the
Printer MIB, RFC 1759
(DRAFT 1.2, 12.29.96)
The primary goal of standardization is to achieve
interoperability of systems developed by independent sources.
The IETF standardization process as defined by RFC 2026
"The Internet Standards Process -- Revision 3"
(http://ds.internic.net/rfc/rfc2026.txt) requires
interoperability testing (see section 4.1.2) before that
specification can be advanced from Proposed Standard
to Draft Standard on the three level IETF standards
track. (Note that this requirement differentiates the
IETF standards process from many other organizations
that define standards.)
A lack of interoperability can be due to a poorly specified
standard and/or to low quality implementations of the
standard. Both of these causes must be corrected before
interoperability can be achieved. However, the IETF
standardization process only requires that non-interoperable
due to poor specification be corrected and leaves
"correction" of low quality implementations to "the market."
Implementation of an SNMP MIB requires implementation of a
network transport protocol, such as UDP, and implementation
of the SNMP protocol. The SNMP protocol requires two
entities, one acting in a "manager" role and the other
acting in the "agent" role. An SNMP MIB is implemented in
both an agent and a manager. Demonstrating interoperability
requires that both agents and managers be implemented.
Unlike other products implementing SNMP (or other IETF
protocols), printers are largely receive only devices and
do not have a network pathway out to other products as hubs,
routers, etc. do. With the exception of traps, printers are
passive devices from a network management perspective. For
this reason, the most significant aspect of interoperability
will be consistent operation (returning the same values)
under like operating conditions, rather than the ability of
two devices to send and receive packets.
Requirements to Demonstrate Interoperability
I. Values:
In order to move RFC1759 forward to "Draft Standard",
each of the six implementations must be tested for:
(1) consistent predictable traps
(2) consistent predictable scalars (non-table variables)
(3) consistent predictable table values
A document reporting the results of all tests for each
coded printer (stripped of manufacturer identity) will be
provided to the IETF.
II. Applications:
Underscore will provide the Print Alert software. If other
software applications become available, they will be
incorporated into the testing.
III. Heavy/Load and Heavy Traffic Conditions:
Tests in I and II should be repeated under heavy traffic
conditions to insure that operation remains consistent.
IV. Protocol Conformance Testing
In addition to all printer MIB specific issues, the general
SNMP agent should be implemented correctly and capable of
passing a suite of protocol conformance tests.
However, in as much as RFC 1759 is one of the largest MIBs
with 179 objects, it is likely that there would not be
sufficient time in the three day period to include
this. Also RFC 2026 does not strictly require proof of
correct protocol implementation, so the testing may
have to be confined to consistent implementation of
the MIB objects.
PURPOSE
The purpose of the testing is to demonstrate satisfactory
interoperability of six implementations of the Printer MIB
RFC 1759, so that RFC 1759 may be advanced through the
IETF standards process.
GOALS
The outcome of this testing should be an independent,
unbiased report that states the results of all tests for
each coded printer (stripped of manufacturer identity).
The report must show that 90% of all results were
consistent. All inconsistent results (the remaining
10%) must be addressed by the Printer MIB Working Group
and corrective action taken.
METHODOLOGY
Schedule an interoperability test event for a three day
period that vendors or their proxy are required to attend.
The fee to use the Stardust lab facility for three days is
$9,000 and could be split among the attendees.
The following is from Jay Martin at Underscore:
----------------------------------------------------------------------
-- JK Martin | Email: jkm at underscore.com --
-- Underscore, Inc. | Voice: (603) 889-7000 --
-- 41C Sagamore Park Road | Fax: (603) 889-2699 --
-- Hudson, NH 03015-4915 | Web: http://www.underscore.com --
----------------------------------------------------------------------
------------------------------------------------------------------------------
Alert Table Test Plan for PWG Printer MIB Bake-off
Version 1.0
This part of the bake-off focuses on problematic events commonly
encountered with printers. In particular, the Alert Table components
of the Printer MIB will be exhaustively tested.
It is in the best interests of the PWG and other other implementors to
define a reasonably complete list of "typical" alert conditions and a
set of corresponding guidelines for indicating which MIB alerts should
be posted for those conditions. Such guidelines should serve to
maximize interoperability between agents and management applications
for this critical aspect of the Printer MIB.
For each of the defined "Test Scenarios" described below, the entire
set of Printer MIB variables (and other associated MIB variables) will
be captured twice, both before and after the test is performed. (The
need to capture all variables comes as the result of prior testing of
some implementations in which certain "description" fields were
incorrectly changed as the result of an alert condition.)
The analysis of each test will involve the creation of a "differences"
file that shows the precise change in the agent database as the result
of the specific test. It is hoped that most analyses will be
performed during the course of the testing period; however, depending
on the speed of the agent implementations, some analyses may have to
be deferred until after the event.
Vendors are strongly encouraged to bring two additional toner
cartridges, where one is nearly empty (for "toner low") and the other
is empty (for "toner out"). Also, it will be quite useful for each
vendor to bring as many printer options as possible, such as addtional
input/output trays, to test how various dimension variables are
reported.
Test Scenarios
--------------
This draft only lists the basic Id and Title of each Test Scenario;
additional structuring and detail will be provided before the event.
For each Scenario, special attention must be paid to the various
subunit status values, if appropriate, as well as the impact on the HR
MIB status values.
Id Title
--- ---------------------------------------------------------------
1.1 Input Tray Removal
1.2 Input Tray Reinsertion
1.3 Input Tray Insertion with Different Tray Dimensions
1.4 Input Tray Missing
1.5 Input Tray Nearly Empty
1.6 Input Tray Empty with Demand
1.7 Input Tray Empty without Demand
1.8 Input Tray Jam
2.1 Output Bin Nearly Full
2.2 Output Bin Full
2.3 Output Bin Jam
3.1 Marker Toner Low
3.2 Marker Toner Out
3.3 Marker Toner Cartridge Missing
4.1 Media Path Jam
5.1 Cover Open
5.2 Cover Closed after Open
6.1 Printer Offline
6.2 Printer Online after Offline
6.3 Printer Reset
6.4 Printer Power-Saver Mode Entry
6.5 Printer Power-Saver Mode Exit
6.6 Printer Idle
6.7 Printer Currently Printing
Outstanding Issues
------------------
- How to test these tough situations:
* Removal of alerts when the table becomes full
* Any of the Interpreter-related alerts
- How are the data files to be named? Need to take into consideration
these things:
* Product Id
* Test Id
* Before vs. After
- Which software tool(s) to use for these tests?
- Must take into account ALL the MIB objects pertaining to the
Printer MIB. In particular, we need to focus on the many related
HR MIB variables.
- Should we suggest that all vendors publicly provide a "list of
supported alerts" similar to the way the MIDI spec is handled in
the marketplace?
# # # # #