>For one thing, we have yet to nail down any kind of precise test plan
>(other than what I had already submitted way back in November, dealing
>only with the Alert Table), so it's a bit premature to assume how long
>it will take to perform whatever tests are defined.
I think the two main problems with the "pre-bake" in Portland were
1. Lack of a detailed plan
2. Not enough time
Given that, I believe we will need all 3 days for testing.
>In other words, I believe we can accomplish considerable interop
>testing in 2.5 days. On the other hand, I can also see us taking
>2.5 YEARS if we try to perform all possible pertubations of the data
>model. I don't think we will be trying to do that...will we?
No, I don't think we should attempt the full combinatorial set - in
fact, it would take light-years.
However, I think we should do MUCH more than just test the Alert
Table! I think there should be a straight run through the entire set
of objects, perhaps with a standardized test configuration or two
that demonstrates each printer knows how to respond with the defined
set of objects. I also think we need some basic SNMP protocol assurance
like not latching in sets in VarBinds if one of the objects fails etc.
Beyond this, yes, we need to define configuration and status transitions
and check each printer's status and event behavior.
I expect there to be 7 printers and as many as 3 or 4 software
applications that want to demonstrate interoperability.
All this can add up to quite a lot of time and effort.
>Let me say this. Interop testing without a debrief meeting to discuss
>results, resolve issues and provide clarifications is POINTLESS.
Agreed, ideally we would debrief in person immediately after (or
perhaps several times during) the testing period. Since it *is*
possible to debrief via phone conference, however, my goal is to
maximize the actual test time so we don't fall short there.
>Unless, of course, all we are trying to do is a "check off" of the
>IETF rule for having an Interop Test in the first place.
I don't think checkoff is the goal. I think we could accomplish this
without even meeting. Aren't we checked off with one application that
manages 2 or more printers? I think I can find this. If this isn't
enough for the IETF, then we better find out *exactly* what will
satisfy them *prior* to our test!
>So, exactly what is our goal for this testing?
Speaking as one who has invested a lot in developing and implementing
to this standard (and I think we all fall into this category) as well
as one who is going to be paying to attend this test (well... a few
of us fall into this one)... my goal is to determine the level of
compliance to the RFC and (thereby) interoperability of the set of
hardware and software products which have currently implemented. Of
course, I'm most interested in learning any areas that I might have
to correct in my own product to gain fuller interoperatiblity as well
as any changes that might be needed to the RFC as it moves forward.
Harry Lewis - IBM Printing Systems