Test Plan
for Products Implementing the
Printer MIB, RFC 1759
(DRAFT 1.4, 01.26.97)
The primary goal of standardization is to achieve
interoperability of systems developed by independent sources.
The IETF standardization process as defined by RFC 2026
"The Internet Standards Process -- Revision 3"
(http://ds.internic.net/rfc/rfc2026.txt) requires
interoperability testing (see section 4.1.2) before that
specification can be advanced from Proposed Standard
to Draft Standard on the three level IETF standards
track. (Note that this requirement differentiates the
IETF standards process from many other organizations
that define standards.)
A lack of interoperability can be due to a poorly specified
standard and/or to low quality implementations of the
standard. Both of these causes must be corrected before
interoperability can be achieved. However, the IETF
standardization process only requires that non-interoperable
due to poor specification be corrected and leaves
"correction" of low quality implementations to "the market."
Implementation of an SNMP MIB requires implementation of a
network transport protocol, such as UDP, and implementation
of the SNMP protocol. The SNMP protocol requires two
entities, one acting in a "manager" role and the other
acting in the "agent" role. An SNMP MIB is implemented in
both an agent and a manager. Demonstrating interoperability
requires that both agents and managers be implemented.
Unlike other products implementing SNMP (or other IETF
protocols), printers are largely receive only devices and
do not have a network pathway out to other products as hubs,
routers, etc. do. With the exception of traps, printers are
passive devices from a network management perspective. For
this reason, the most significant aspect of interoperability
will be consistent operation (returning the same values)
under like operating conditions, rather than the ability of
two devices to send and receive packets.
PURPOSE
We are testing that every single one of the 179 MIB objects are
implemented in at least two printers, and that the implementations are
consistent among all the printers. This is the IETF requirement to
advance to Draft Standard.
We are *NOT* testing conformance, compliance, boundary conditions
(except indirectly), MIB II and its instrumentation, performance, data
representation (except for inconsistencies among implementations),
etc.
GOALS
The outcome of this testing should be an independent,
unbiased report that states the results of all tests for
each coded printer (stripped of manufacturer identity).
The report should have a list of recommendations of corrections,
clarifications, and resolutions derived from the three days of
testing. The Printer MIB Working Group must take corrective action
to incorporate these recommendations, as appropriate into the
RFC.
METHODOLOGY
The methodology is comprised of three parts: (1) Questionnaire,
(2) Propositions, and (3) the procedures we will use to achieve
the goals.
(1) Questionnaire. Generally we do testing to find
misinterpretations and ambiguities in the RFC as evidenced in
the product implementations. In the case of RFC1759, we are
operating at a coaser level of granularity, meaning that we will
find many of these areas through simple dialog. What did you
use as a definition of critical alert? Do you send a trap on
all critical alerts? It will not be considered acceptable to
consider these issues "implementation dependent".
(2) Propositions. To save time during the testing and in
evaluating the results, we would like to have all outstanding
ambiguities framed as propositions. In formal logic this means
a statement in which the subject is affirmed or denied by the
predicate, e.g. "Alerts are entered into the alert table only
once, and not re-entered." This will make recording of decisions
easier after the technical discussion.
(3) Procedures. Each printer will be attached to a TCP/IP network and
act as the product-under-test. Various software tools and products
contributed or loaned by vendors will be used to automatically run
tests against each product-under-test. In these cases, specific
functionality will be automatically tested and logged. In some cases,
all the values of the relevant tables will be read and stored, and then
certain fault conditions will be manually generated (e.g. pulling the
paper tray out), and the affected tables will be read and saved in a
file. For many of the test cases where the tester will be manually
generating the fault condition, we will use the Castlerock Computing's
Windows-based SNMPc network management package. (Please thank John
Sancho of Castlerock for his generosity in providing a loaner version
of this product to all participants in the testing.)
STEP-BY-STEP TESTING SCENARIO
Sunday, Feb 2, 1997:
1. Set up printer and Win95 laptop (See detailed list of
required items.)
2. Check for link state.
3. Use Dr. Watson (see www.cavebear.com) to do ARP and ping to
each printer and to check all IP adresses, subnet masks, etc.
4. Install all test tools and products.
5. Answer questionnaire.
Monday, Feb 3, 1997:
1. Initialization.
Do a MIB walk on each product and count the returned objects.
2. Alert Table Test.
Use SNMPc to read all the tables (including subunit status and
HR MIB status values) and log the results. This will initialize the
beginning values. The tester performs the manual action listed
below. After each action, use SNMPc to read all the tables and save
the results.
Id Title
--- ---------------------------------------------------------------
1.1 Input Tray Removal
1.2 Input Tray Reinsertion
1.3 Input Tray Insertion with Different Tray Dimensions
1.4 Input Tray Missing
1.5 Input Tray Nearly Empty
1.6 Input Tray Empty with Demand
1.7 Input Tray Empty without Demand
1.8 Input Tray Jam
2.1 Output Bin Nearly Full
2.2 Output Bin Full
2.3 Output Bin Jam
3.1 Marker Toner Low
3.2 Marker Toner Out
3.3 Marker Toner Cartridge Missing
4.1 Media Path Jam
5.1 Cover Open
5.2 Cover Closed after Open
6.1 Printer Offline
6.2 Printer Online after Offline
6.3 Printer Reset
6.4 Printer Power-Saver Mode Entry
6.5 Printer Power-Saver Mode Exit
6.6 Printer Idle
6.7 Printer Currently Printing
3. Channel Table Tests
Verify that the Printer MIB Channel Table functions properly.
This test checks each entry in the Channel Table for valid enumerated
values for Type and State and non-error status. For channels with
ifTable references it then has the tester print to check ifInOctets.
The test then disables the channel and has the tester print again and
finally has the tester cause an alert on the selected channel.
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
[The remainder of this document contains proprietary information
(detailed test specifications) and can only be sent to those parties
who have executed Non-Disclosure Agreements. If you wish to see a copy
of the detailed test plans, please contact me to arrange it.]
The entire detailed test plan is approximately 12 pages of faxed
material.
-----------------------------------------------------------------------------
--==--==--==- Chris Wellens President & CEO
==--==--==--= Email: chrisw@iwl.com Web: http://www.iwl.com/
--==--==--==- InterWorking Labs, Inc. 244 Santa Cruz Ave, Aptos, CA 95003
==--==--==--= Tel: +1 408 685 3190 Fax: +1 408 662 9065
-----------------------------------------------------------------------------