Caltech/CERN/HP Project Report

For June/July/August 1997

Julian J. Bunn

256-48 High Energy Physics

Caltech

1200 E.California Blvd.

Pasadena

CA 91125

USA

Tel.: +1 818 395 6681

Latest News

• The latest reports and documents to do with the Project are being stored at: https://julianbunn.org/ If you have comments about any of the material therein contained, please let JJB know as soon as possible.

• A new Project title is proposed: GIOD (Globally Interconnected Object Databases). Your comments are welcome.

• Included in the Web page above is a link to set of transparencies JJB prepared, and which were shown by Harvey Newman at the CMS Software and Computing Technical Board in June.

• Also included is a set of transparencies that JJB showed from Caltech via a VideoConference link to the RD45 workshop in July.

• The page also includes links to a demonstration of JJB and Rick Wilkinson (RW) loading the SPP Exemplar with Objectivity queries, links to plots of the scaling performance of Objectivity on the Exemplar, NT machines and HP workstations here at Caltech, and a description of the test application called "Stars" that JJB developed with Roy Williams.

• JJB and RW successfully ported the whole of the CMS H2 Test beam analysis code (based on Objectivity) to Windows/NT at Caltech. All data so far collected at the test beam have been copied across to Caltech, and inserted it in an Objectivity database here. We are currently debugging the code. The OO promise of more maintainable and easier to understand software does not seem to apply in this case!

• JJB and RW have rebuilt all CMS libraries on the Exemplar, and made an SPP-version of CMSIM (111), which we are preparing to run in background production to use spare cycles on the machine.

• JJB and RW have replicated the test beam database to the Exemplar, but are encountering Objectivity problems when trying to access the data in the copy.

• The HPSS system at Caltech has been used to test Objectivity with an NFS interface. At the request of RD45, and together with James Patton (CACR), JJB created a test database on an NFS-exported filesystem on an HP workstation in the HEP department. The NFS filesystem was exported from the HPSS system in CACR. The test database was migrated out to HPSS tape, and an attempt made to then access it using a simple Objectivity application. Initially this failed, but by increasing the RPC timeout value used by Objectivity, it was possible to have the HPSS system recall the data from tape before the application timed out. This test shows that a fallback solution for managing large Objectivity databases on HPSS storage is to use the NFS interface (see test details below).

• The non-DCE HPSS client work is progressing well at San Diego Supercomputing Centre, and we expect to receive some code to test on machines here very shortly.

• Tests have begun to monitor the activity of the Caltech HPSS robot remotely, using a small camera situated inside the robot, connected to a PC running the Caltech "Virtual Rooms" software. Since, when using HPSS, it is impossible at a software level to tell whether ones request for data is causing robot activity, the idea with "Project Cyclops" is to allow one to connect to a "Virtual Vault" and observe the robot directly. More details on this little project are at: https://julianbunn.org/Cyclops.html

• JJB still waiting to meet Ram Appalaraju, who has had to postpone his visits to Caltech for various reasons. Ram is the HP person responsible for the project funding.

• Caltech was obliged to buy a license for Rogue Wave Tools.h++, since the CMS test beam software relied on it. Following great difficulty in actually obtaining the software (at considerable expense), the supplied installation procedures for Windows/NT failed, and we were obliged to consult Rogue Wave's customer support at length. The problems are still not resolved. It seemed particularly irritating because the product was "only" being used for a couple of fairly simple class definitions.

• JJB will travel to the CMS week in Madison/Wisconsin, in September. The trip is being paid for by Caltech, at JJB's request. Vincenzo Innocente will visit Caltech for one week following the CMS week, in order to develop the H2 test beam software, and to assist us with the data replication from CERN. This trip is being funded as one of the RD45 trips foreseen in the Project proposal (this has been agreed to by RD45).

• JJB met Jeremy Walton (NAG) at Caltech, where Jeremy was presenting Iris Explorer. Caltech have a site license for Explorer, which we will use during the tests with the CMS test beam analysis code.

• JJB has installed ObjectStore and Versant (rival Object Databases) on machines at Caltech, in preparation for possible comparison tests with Objectivity. Contact has as well been made with Computer Associates, who will shortly announce a fully functioned multimedia Object Oriented database called Jasmine.

• The availablity of the "aCC" compiler on SPP/UX is in question. This compiler will be supported by Objectivity in future releases of their product for HP machines. The currently supported compiler, "CC", is poor on C++ template support. It may be important that Objectivity continue to support "CC", as well as "aCC", although the timescales for the latter being available on HP/UX 11, and the convergence of SPP/UX and HP/UX 11, are sufficiently vague that this is not yet an issue.

• Objectivity has been installed on the following platforms at Caltech: HP 755 workstation, Pentium Pro 200 MHz PC, Pentium II 266 MHz PC, HP Exemplar SPP 2000. The latter installation is not supported by Objectivity and is in any case outside the framework of the CERN licensed platforms scheme. The licensing issues are being actively addressed by Paul Messina (CACR) and the Objectivity salesman for Southern California (John Jarrell). The other platforms are being used as supported platforms with CMS licenses.

• The FTO and DRO (fault tolerant and database replication options) were installed at Caltech, for the purposes of allowing Eva Arderiu from RD45 to make tests of Wide Area Network database replication. The tests involved replicated database images from CERN to Caltech at regular intervals during the day and night. These tests were highly successful, and Eva reported fully on them at the RD45 workshop in July.

• Several problems with Objectivity have been uncovered:
  • Objectivity version 4.0.2 is supported on Windows/NT/95 only with the Microsoft Visual C++ compiler version 4.2, which is no longer being sold. It is possible to use Objectivity with the latest Visual C++ (version 5.0), but it involves some dangerous "fix-ups".
  • It was not possible, even with the above "fix-ups", to build the CMS test beam code (that uses C++ templates) using VC++ version 5.0, and we were obliged to revert to VC++ 4.2. This is highly inconvenient as it means either giving up the new features (particularly the Visual Studio development environment, which is compatible with Visual J++ and Digital Visual Fortran), or trying to make the two products co-exist on the same system.
  • When allowing many simultaneous applications to operate on an Objectivity database, it is found that, unless very careful (and non-intuitive) steps have been taken in the design of the schema, strange and incomprehensible error messages are issued by some database applications at run time, some of the time.
  • Cleaning up after applications "go bad" is often tedious: pending locks tend to remain from dead applications, and they are especially hard to remove if the application was running on a remote node when it died.
  • There are some "known" bugs (i.e. bugs that are documented somewhere, but that place is hard to find), that affect how collections of objects should be iterated over. An application hitting one of these bugs may not fail, but will produce very poor performance. An example is the use of object iterators over containers (works fine) and over databases (doesn't work fine). Another example is the use of indexes over sets of objects ... declaring the indexes improperly does not cause any warning or error during compile or run time, but just produces the wrong set of objects!

Other News

• JJB prepared a poster for the SPP Exemplar "dedication" in June.
• RW attended Objectivity course at Mountain View.
• JJB attended RD45 meeting at CERN on July 3rd.
• Caltech HEP suffered several network outages recently
• There is a possibility that P.Marchesini and J.Draskic-Ostojic will join the Project to work in the HPSS area.
• Details of the HPSS/NFS/Objectivity tests

•  1) HPSS filesystem mounted over Ethernet LAN on HP workstation running HP/UX 10.20
  2) Objectivity 4.0.2 installed on HP workstation
  3) Federation created on the HP, with two databases of size 0.5 MBytes each on local disk.
  4) Application runs on HP against database correctly
  5) Both databases moved to reside on the NFS-mounted HPSS filesystem, using
  oochangedb -db [name] -filepath /hpss/[name].db [bootfile]
  6) Application runs on HP against databases with new location correctly
  7) On the HPSS machine, both database files forced to second-level storage class in HPSS (i.e. forced to tape in this case).
  8) Application runs on HP against database: we observ the following 
     a) HPSS system mounts tape in order to restore the (two?) database file(s)
     b) Objy application waits for a while while this is going on
     c) Objy application fails with the following error message:
  
  cithe702 29: date;stars_match;date
  Fri Aug  1 13:23:31 PDT 1997
  With predicate query
  ** Error #4515: ooHandle(ooDBObj)::open(): Storage Manager: Error reading page.
  size = 8192 page = 0 RPC: Timed out (103204)
  Cannot open DB2
  Fri Aug  1 13:23:59 PDT 1997
  
  The conclusion seems to be that the Objy RPC request times out for the 2nd.
  database. 
  
  (...later...)
  
  Further to my mail of 1st. August, we have now re-run the test with 
  an RPC timeout of 1200 seconds (10 minutes). This evidently gives 
  HPSS enough time to restore the database files (which were forced 
  offline prior to the test): 
  
  cithe702 36: date;stars_match;date
  Tue Aug  5 17:28:16 PDT 1997
  With predicate query
  Checksum -1961861
  Time to search for nearest stars 898 secs
  Tue Aug  5 17:43:17 PDT 1997
  
  This indicates that the NFS/HPSS interface works for Objy 
  when an appropriate time out value is set.
  
  Note that the elapsed real-time for the application is 15 minutes,
  of which 900 secs comprise the application searching all objects 
  in the two databases. 
  
  The RPC timeout is set using a call to ooRpcTimeout, apparently 
  undocumented in the Objy API