Zig Minutes Sep 23-25: Introduction and Status Reports Eric Ferrin, Penn State Julie Mills, US Government Peter Ryall, Mead Data Central Oren Sreebney, BRS - Working to interface Z39.50 to BRS/Search Ralph LeVan, OCLC - Server been talked to by nearly everyone. Ported system to Sparc-10 Server can act as gateway to other servers including Z39.50-88 Has Gopher Gateway Has Client API, and has run client against a couple people Tom Dopirak, CMU - Many internal organizational changes. No external progress Bill Cattey, MIT - Working to hook together Kunze's Z39.50 code to BRS/Search on the server side and U Washington's Willow on the client side. Bob Waldstein, ATT - Starting to give his client and server to real users. Implementing diagnostics. Having interoperablity problems with defaults. Les Wibberley, Chemical Abstracts - Doing a Scientific and Technical attribute set. Working on chemistry application. Client based on Stanford code, server based on Berkeley code. John Amoss, Chemical Abstracts - Project leader of one of the efforts. Eric Bivona, Dartmouth - Z39.50 Gateway from DCIS CWIS. Is distributing WAIS gateway. Hope soon to be able to distribute Z39.50 gateway. Sara Randall,NOTIS - Server online since August. Michael Thwaites, UC DLA Todd Perry, VTLS - Working on client. Just starting on Server. Dwight Lillie, Molecular Design Ltd.- Client for Chemical Abstracts server. Ray Denenberg, LC Raplh Orlik, LC - client sending canned searches to nearly everyone. Terry Sullivan, Center for Library automation - Doing Z39.50 in pure OSI Environment. Has nobody to talk to yet. Harold Finkbeiner, Stanford - Client Up and running. Got rid of ISODE to make it more portable. Hoping to make code available shortly. Supports Info-1. Mitra, Pandora - Client/Server database to run on MAC. Liv Holm - Nordic Project SR over ISODE over TCP/IP. Dennis Lynch, ESL - Politics and program startup rather than coding. Put together a WAIS front end to existing DB to show that Z39.50 works for general DB access method. Brad Mclean, Gaylord Info Systems - V2 prototype Client/Server. (Has to give up office phone line to turn on system.) Marjorie Tibbitts, UC DLA - Server up 24hrs. Clifford Lynch UC DLA - Client which is publicly accessible. Lennie Stovel, RLG - Server contacted by nearly everybody. And a few other sites we've never heard of. Has had their real Bibleographic DB available for the past month. Planning to have Penn State be one of their early real users. They will be supporting Browse. Demoed at LITA along with Penn State and Melvyl John Kunze, UCB - Client and Server running. Server just returns one record. Expecting to release new version in late November. Cecilia Preston, UC DLA - "Token Librarian" Janet Vratney, Apple Library - Orchestrated connectathon at LITA. Tony Malia, information Access - Coming up to speed. Interested in Document Architecture issues. Beau Parker, Stanford - Gave a couple public demonstrations of CWIS system hooked into Melvyl and ??? Geoff White, Carlisle systems - Just getting starting on Z39.50 project Prototypes with WAIS I code and toy servers to come up to speed. Expect internet connection in 2 months. Rich Fuchs, RLG Slavko Manojlovich, Stanford Steve MacNeil, Acadia University David Brown, ESL Julie D' Angelo, ESL Prakash Sinha Geac Bruce Park, Geac - Investigation phase. Plan to do both OSI and TCP. Simon Spiro (here in spirit) - We're anxiously anticipating the fruits of his WAIS II labor. Joe Zeeman, Software Kinetics Ottawa - Signed contract to develop SR Server and Client by different people. Starting work within the month. Expect to have products in May. Making available a Kernel Implementation of Origin and Target protocol machine. Will to interface to Oracle SQL. (KA University library will be mapping SQL to BIB-1) National Library convening an SR interest group starting Oct 8. People are welcome, especially software vendors. Wayne Davison, RLG Peter Ciofetti, Silver Platter Stuart Soffer, DRA Craig Summerhill, CNI Mark Needleman, UC-DLA ZIG Minutes Sep 23-25: Day 1 Going through the Adjenda 1. Introduction and Progress Reports (see other part so-titled) Issues raised during intros and leading into Item 2: A. Do we want to start using the Implementation Name field in the INIT APDU. -- No. Resellers may want to use this field. B. Z39.50-88 Interoperability - Trivial examination of first packet in. C. SR Interoperability -- May require ASCE support. Would be nice if we could tell at INIT time. You could then swap between two OID tables. (There is no OID in the init.) Maybe some clever approach to examining the INIT will do: Maybe SR won't come in on port 210. Needs simple, agreed upon hook. (Discussion Item for next meeting.) D. Modeling documents with records. (Deferred to Adjenda 14.1) E. Multiple servers on one host. Somebody has to serve as a multiplexor. (Mailing list issue.) 2. Version 3 survey Results -- Revised in the course of the meeting. "YES" for version 3: OtherInformation added to all APDUs Diagnostic Format is made EXTERNAL addinfo added to DiagRec attribute set identification within RPN query (formerly called attributes plus term - now much nicer.) Named result sets EXPLAIN SCAN General Preferred record syntax Search Result set SORT (both as separate PDU and piggy backed on SEARCH -- without a facility for removing duplicates.) BIB-1 Additions BRIEF Element set Concurrent Operations (now well understood and all conducted through the reference-id) Add CLOSE and ABORT APDUS to facilitate serial dialogues (provided no new issues jump out at us on this.) Segmentation-0 (segmentation on a record boundry) Segmentation-1 (target specified segmentation not on a record boundry.) Subrecord Addressing - AKA Define Element Set name "NO" for version 3: periodic query suspend cost limits indeterminate message length ID Authentication in INIT response - we might later decide to do this anyway. Items that started out neither YES nor NO: "MAYBE - yes" concurrent operations - the Canadians want it. Serial dialogs (reinitialization) - it's simple to do and the Europeans want it. diagnostic format external addinfo octet string Attributes plus term ID Authentication in INIT response - low impact, even if not popular Named result sets - low impact even if not widely required Sort result set "MAYBE - no" Define Element Set name - needs review and discussion Save Result set Persistent Objects (Objects) Regular Expressions ---- The V2/V3 compatibility issue showed itself to be one with a divergence of understanding. Ray will propose to the list an approach to answering the question. V2/V3 Compatibility: <> Discussion: What is a minimum amount of V3 stuff must you support to be able to say you are V3 and expect to interoperate? Should various features be negotiated explicitly? How much of it? Should all ADPU fields be recognizable even if no action is taken on them? If you claim to be V3, you must have compiled the v3 ASN.1 so that you can receive all the ADPU's without an ABORT. Every version is a formal superset of the previous version until the case can be made that such is impossible. ** Questions: Should the feature be in V3? Can one negotiate to say that the ADPU is not going to be recognized. Can one negotiate to say that the ADPU is recognized but not acted upon. Should the implementation be required to recognize the ADPU without negotiation? Should the implementation be able to silently ignore the ADPU recognized or not? Further discussion deferred to laterin the ZIG meeting ---- 3. Version 3 draft 4 discussion. Much discussion ensued when the segmentation/fragmentation topic was raised. We'll need to revisit this when we're supposed to discuss this. (Don't forget to discuss modifying the SEARCH APDU as well as the PRESENT APDU.) Concurrent operations: Need to figure out what to do about ASCE for TCP. Add one? How much does it cost? -- Searching a Result set -- What does it mean? We named it "universal" because a database name was required for SR. Rename to Reserved Database Name "Complete". <> Seems devisive. We're considering dropping this from V2. Les says Chem Abstracts needs it. He will define semantics for it and bring it to the next meeting. Otherwise it may be dropped from the standard. ---- 4: V3D4 Details -- In order of Yes, Maybe Yes, No, Maybe No. SCAN: to SCAN backwards, if you specify the last term as the one to start from, you are SCANing backwards. Preferred position begins with 1 not 0. Add "Step Size not supported" diagnostic. New tag numbers: (Compatible with ISO) 36 is SCAN request 37 is SCAN response Kunze asks: Can't we do SCAN as a particular case of SEARCH? Yes, but SR is doing SCAN, and we may expand it into a richer SCAN in the future. There is also the concern about overloading the SEARCH. Q: Should we make Entry a CHOICE {entryInIndex, EXTERNAL } A: No. That puts on the slipery slope of adding piecewise the functionality of SEARCH. Keep it simple for now. -- Other Information: Defer discussion of negotiation to list. Q: What about interoperability for Other Information? A: There will be a registry of OIDs. -- Generalized Preferred-record-syntax -- Defer Element and Element set name until Tomrrow morning when we're fresh. -- BIB-1 Additions -- Put them in. They make SCAN much more useful. Q: Why these chars and not ISO CCL? A: They are not the same as NISO CCL. There was considerable interest in just putting in a small useful subset that people were familiar with. CCL was deemed not powerful enough. The CCL syntax could be registered later (V4?). -- Segmentation: Segmentation of PRESENTs. Records will not span segments. Zero or more SEGMENT PDU's come followed by a PRESENT response. This method should also be OK for a piggy-back Search response This is not a solution to getting SEARCH results before the search is complete. That can be done with a local agreement on use of a RESOURCE REQUEST. Q: Should we negotiate Max number of Segments in INIT or PRESENT. A: Doing it in INIT limits the changes to the PRESENT APDU. Q: Do we need a maximum number of Segments negotiation at all? A: No. If you multiply the maximum record size by the number of records requested, you get the maximum aggregate buffer size if you wanted to hold all the segments. ZIG Minutes Sep 23-25: Day 2 - Going through the Adjenda Recap: We agree on Segmentation for the easy case of sending subsets of the whole set of records from a PRESENT (with no record spanning a segment) as zero or more Segment pdus followed by the PRESENT response. We "officially" call this "segmentation at record boundries" or "Segmentation-0". You are guaranteed to get back no more Segment PDUs than records you requested. We also agree that for situations in which early partial results from a large, time consuming, SEARCH are required the mechanism is to use a local EXTERNAL definition for a RESOURCE CONTROL request and response to get early results. -- Segmentation discussion: Problem: Determining where to make the cuts: A. On a semantically meaningful logical boundry. (May require logic for understanding the insides of a record.) B. On an arbitrary boundry which must be decoded or reassembled before the chunk can be made sense of. (Requires encapsulation or re-assembly.) -- Segmentation-0: Addresses the problem of a response containg a large number of records. Subsets of the whole set of records from a PRESENT are sent by the target as zero or more SEGMENT PDUs followed by the PRESENT response. Target guarantees to cut on record boundries. We "officially" call this "segmentation at record boundries" or "Segmentation-0". You are guaranteed to get back no more SEGMENT PDUs than records you requested. Segmentation-1: Address the problem of receiving very large records, like TIFF images. You get multiple SEGMENT PDUs followed by a PRESENT response. Records span segment pdus, and there is a special wrapper transfer syntax. (You may get more segments back than records requested in this case.) To simplify operating with Segmentation-0 and 1, Segmentation-1 is contrained NEVER to cut on a record boundry. For that it uses Segmentation-0 Segmentation-0 and Segmentation-1 are seperate negotiations in INIT. If Segmentation-1 is supported, it directly implies that Segmentation-0 is supported. A Max-seg-count is put into the PRESENT ADPU. This allows the various PRESENTs to specify how big or small the aggregate record is for the particular PRESENT (allowing different contexts to specify the aggregate record size without tearing down and re-establishing the session.) Suggest the standard defines a particular transfer syntax for fragmentation. Ray will take the discussion and issues raised on the various sizes and counts and will write up a proposal and send it to the list. Some of the discussion: With segmentation supported, the meanings of Maximum Record Size changes. Preferred Message size is the approximate maximum size of a PDU. Maximum record size is ordinarily a buffer size issue. (It enables you to send out a single record that's bigger than Preferred message size.) Without segmentation, Maximum Record Sized must equal or exceed the preferred message size. With segmentation, Maximum Record size must equal or exceed the aggregate record size. Each segment is less than or equal to the preferred message size. In the PRESENT request there will be an optional max-logical-record-size and a max-{response,aggregate}-segments. Kunze wants to be able to say that the maximum number or segments is indeterminate. (ASN.1 has a way to specify a number is a positive integer value of infinity.) -- Origin specified segmentation comes under the heading of the Define Element set name topic -- deferred until that discussion. -- Additonal SEARCH Information: will be added as an optional additional EXTERNAL parameter in the SEARCH. Clifford Lynch will try and come up with a starter record. -- concurrent operations: All the things being multiplexed by the origin are keyed on the reference-id. In concurrent operations, all threads in the same assocation have the same access control capabilities. If the target needs to know something about the meaning of reference-id values, it should be done by local agreement. Doing it in standard would be bad. It is the responsibility of the origin performing the multiplexing to keep seperate the result sets belonging to different users by manipulating the result set name. There is a desire to have seperate "threads" within an association. A thread would look like a regular Z39.50 session (rather than having to have an origen keep track of all the properties for concurrent users within a session.) The thinking on this issue is to add in the OSI Presentation context idea outside the Z39.50 application layer protocol. There would be several threads, each with its presentation context on the same association. This would imply that there should be a CLOSE APDU distinct from the IR_ABORT APDU added to the Z39.50 application protocol for proper interfacing to the presentation layer. Thinking on how this would be implemented: Have a thread APDU which contains an INIT APDU. TCP based implementations would add a wrapper around the Z39.50 PDUS to add the moral equivalent to a presentation context. -or- Add a thread ID to every APDU except init. The list will discuss which of these two implementations should be done. Concurrent operations moves to YES for V3. -- Reinitialization (Serial Dialogues) Add CLOSE and ABORT to do serial dialogues. Which is a provisional YES for V3 if no other issues come up. Ray will dust off the V2 prose on the CLOSE and ABORT PDUs and post it to the list. -- Diagnostic format EXTERNAL - A simple feature with a groundswell of support - moves to YES for V3. -- addinfo moves to YES for V3. Clifford will generate the appropriate description and appendix. addinfo will become a CHOICE of Integer, Visible String, OCTET String, etc. -- attributes plus term (AKA) attribute set identification within an RPN query. Moves to YES for V3. There is an issue of some people wanting to hang an attribute globally off all terms in a SEARCH. But there is resistance to adding this into the protocol. This is really the Global Search Qualifiers issue -- deferred. -- ID Authentication in INIT response: No compelling need for this mechanism for authenticating the identity of the target. -- Named Result sets: Moves to YES for V3. (As a bit within the Options parameter) -- Sort Result set: Should we have a Sort Attribute set or should we add Main Entry and Relevance score to BIB-1? Should we use Attribute plus Term with BIB-1 and a small SORT Attribute set? (Would require revising ASN.1 to match RPN query. Would create a cross dependency on Attribute plus term feature.) ASN.1 for sort should be revised to match V3 anyway. Clarification: You are allowed t o specify any attribute set for SORT. Q: Do we want to implement SORT: as a seperate PDU only? as an Add on to SEARCH only? Both? A: BOTH. Should the Small Set / Medium Set / Large Set parameters from present to appear in the SORT? Two bits in INIT for negotiation: SORT PDU / SORT in SEARCH. In the case of sorting two sets together, need a way to select either interleaved or non-interleaved sort. Mark H. Objects to use of undefined terms like Field and Element, and wants to remove sortField. What about eliminating duplicates? (Only in case of identical records.) Q: What about if record comes from two different databases? A: OCLC says that different user communities have different notions of identical. Add set of attributes related to eliminating dups. SORT response needs field for size of result set in the case of eliminating dupes. Let's think of de-dup as a separate service which may, later on, be subsumed into SORT. SORT without de-dup moves to YES for V3. de-dup becomes a separate issue. A champion would have to have a clear proposal before the next meeting. ---- There was an adjenda item for going through all the V3 features and determining what sort of V2 compatiblity there would implemented: requiring the PDU to be understood by anyone saying they are V3? negotiating in INIT never to send the PDU? understanding the PDU but negotiating whether or not the feature is supported? requiring the feature to be supported without negotiation by anyone saying they are V3? At issue here was deciding various migration pat hs for V2 systems into V3 features: advertising V2 but containing a V3 feature. advertising V3 but containing the minimal V3 feature set. deciding what ASN.1 must be understood by anyone advertising V3. This discussion has been OFFICIALLY deferred to the list. ZIG Minutes Sep 23-25: Third Day - Going through Adjenda Explain: Question: How should the EXPLAIN-1 attribute set be done? Discussion: There is a lot in common with BIB-1. Use BIB-1 instead? Include all of BIB-1 into EXPLAIN-1? Add things to BIB-1? Do Attribute Plus Term and get BIB-1 directly with a small attribute set? (People prefer making it as easy as possible to put EXPLAIN in V2 systems, so this precludes use of Attribute Plus Term.) Mark H: So much has been finessed through EXPLAIN that the EXPLAIN-1 attribute set is more important to be standardized and stable than even BIB-1. How do we balance stability and extensibility? A core static set of Attributes and a dynamic extensible set. There may be systems that support EXPLAIN that don't support BIB-1. Answer: Defer to next meeting. Most popular solution: A separate autonomous EXPLAIN attribute set. Under the covers, the implementation could map the BIB-1 and EXPLAIN-1 names to the same tables of attribute sets. -- Mitra asks for a list of all the little things that one must finesse through EXPLAIN and how to use EXPLAIN to do them. Example: I have come across a database with an unknown attribute set. How do I use EXPLAIN to discover the Attribute set? Search on a combination of the database name and the attribute to find out if the attribute is supported. Overall caveat: What EXPLAIN returns are human readable strings. Conveying semantics is difficult. For dynamic attribute sets: There is some provisions for getting lists of attributes. But we've not really understood the relationship between Info-1 and EXPLAIN yet. Does support of Info-1 require EXPLAIN? Does support of EXPLAIN require Info-1 in order to get back the answers to the queries? Writing this list will be taken to the list. -- There is an important set of issues related to extensibility: What should the target do if it gets attributes it has never seen before? There is a distinction between unsupported but valid attributes, and invalid (perhaps nonsense) attributes. The intelligent thing for a target getting an out of range attribute during EXPLAIN is that it should return "Unsupported". There was some concern that a "purist" model would require tearing down the connection. This is not an applicable model because there is no ASN.1 explicitly defined for these attributes that would have to be recognized, so the standard does not require tearing down the connection. Can Clients ignore fields that they do not recognize? I.e. can the EXPLAIN record structure be changed for extensibility? This IS problematic because these ARE ASN.1 structures and are required to be validated by the presentation layer. New fields are added by having a sequence of OtherInfo added to the EXPLAIN record and defining new data structure syntaxes which have OIDs. A client searching the EXPLAIN database can specify a list of record syntaxes that contain the various extended record types. If the target understands the ASN.1 to assemble extended record type, it will respond. With the requested syntax. The right place for the various OtherInfo fields to allow exension of the fields will be discussed on the list, and will become clearer as we gain experience. Clifford will post a new base version of EXPLAIN to the list. -- There is additional stuff that is SORT related that must be added to the spec. ---- Progress report from Objects group. Working to simplifiy the services definition as much as possible: CREATE GET MODIFY LIST Service model is: You're storing information, and then side effects happen which are outside of the standard. The semantics will be defined and make their way into the non-normative appendix to the standard at varying rates (depending on the package). Example: persistent result set will get wide consensus and go into an appendix to the standard very quickly. Expect a concrete and fairly complete proposal in on the order of three weeks. The subgroup will be meeting tomorrow morning. Ray wants to see an operational model sent to the list. This is the essence of the semantic stuff that Ralph intends to post. How would Objects get into Z39.50? Could it make it to V3? An addendum? (No, not an addendum -- that would require waiting until the V3 balloting is done which would be the same as waiting until V4.) As a separate standard? (Potentially too time consuming. Might also have implementation ramifications for hooking together the two standards.) Is this service going cause problems in the Z39.50 V3 ballotting. As designed, it does not impact V3 -- it has no effect on any of the other services. (It does have result set naming as a corequisite service.) It is designed to be totally optional. Going to try and get this together in time for V3. Want to pursue this feature as a version-independent feature. ---- The CNI ftp server will hold the documents online to deal with the problem of not being able to access the ZIG documents on THINK.COM. Oren and Craig will send EMail when the server is available. How might we be able to search the OIDs? A file with the OID and a line that describes what the OID is that refers to the doc. deferred. Ray will work with CNI on this. Everyone with private OIDs: send them to Ray with a line describing them. ---- subrecord addressing: - moves to YES for V3. Element-set-name ::= CHOICE {VISIBLE STRING, EXTERNAL} ? The notagsplease BOOLEAN -- Are we sure this is relevant to an Element set? Yes. It refers to classes of transfer syntaxes which might be required to lack tags. Ralph wants to be able to ask for scattered members of a field that occurs in multiple places. Whoever proposes a record syntax must also propose a syntax for fragmentation. Removing the Info-1 specific aspects from the proposal is required. It is an open question that the Info-1 stuff is sufficient. So it now becomes EXTERNAL possibly to appear in an appendix. Ray will assign an ID for the Element Set definition syntax and post it. Call it: ES-1. Missing functonality? We need to gain experience defining the various element sets that people use. Mark H. Wants a better definition of "Field" than a comment in the ASN.1. "Field" is deleted from the spec. Need clear definition of Element. "a component of a record" is not sufficient. A component of an abstract record syntax that can be referred to by a name in that syntax. (Implies that some syntax is defined, even for generated pieces of the record.) (See also 3.2.2.1.4 for element set name definition.) Mark H. will try to post appropriate Element set name stuff for MARC. There was much discussion about Variants. It looks like these all are able to be taken care of by reference to the ObjectID which will name the particular transfer syntax. In the case of Info-1, there is also a Qualifier component which enables further choice. ---- Next meeting is December 9-11 at FCLA in Ganeville FL Meeting after is April 14-16 in Washington DC. ---- For next meeting: Applying attributes to result sets Protocol-active elements Global attribute list semantics Changes to type 101 query ----