Project Organization Plan for the USCMS Software and Computing Project

Draft

Table of Contents

Figures and Tables

The goal of the USCMS Software and Computing Project is to provide the software and computing resources needed to enable US physicists to fully participate in the physics program of CMS. Additionally it should allow US physicists to play key roles and exert an appropriate level of leadership in all stages of the computing related activities from development of the reconstruction programs and software infrastructure to the extraction of physics results. This capability should extend to physicists working at their home institutions.

The software and computing effort for CMS exceeds in scale and complexity anything that has so far been achieved in High Energy Physics. Even the new generation of experiments that will be coming on in 1999 and 2000 will not approach this scale. Because of the large number of participants and their wide geographical distribution CMS will need to employ what is essentially a new model of distributed computing and data analysis which has no precedent in HEP. It will do so during a period of rapid change in software practices and hardware technologies. The USCMS Software and Computing Project is the responsse of US physicists in CMS to this challenge.

A key element in achieving this goal is to develop the software and to construct the facilities to provide an integrated environment for remote collaboration that would make possible central US roles in the data analysis. This includes:

1. providing the resources to support participation in the development of software associated with the design, calibration, commissioning, and analysis of the detectors in which US CMS members are involved;

2. providing the resources to support the participation in the development of reconstruction, simulation, and analysis frameworks and other physics applications infrastructure at a level appropriate to the size, capabilities, and experience of the USCMS physicists;

3. providing the resources and facilities for participation by USCMS physicists, especially those who wish to remain based in the US, in all analysis efforts and activities of interest to them.

The word 'resources' is meant to include personnel for support, operations, and development; hardware; commercial software purchases and contracts for other services required to achieve these goals. This includes resources, support services, and facilities to couple smoothly to CERN central computing so that US physicists can work productively whether at their home institutions or at CERN.

USCMS functions within the context of the full CMS experiment which in turn functions as an experiment of CERN. It is essential that this project stay well aligned with both the scientific goals of USCMS and with the policies and approaches of CMS and CERN. CERN has stated clearly its policy that significant resources to support data analysis must come from sources external to CERN. This project responds to that policy by marshaling US national resources to support the analysis activities of US physicists on CMS. The US expects to do this in a cost effective way by leveraging the knowledge, talent, and experience with HEP computing that exists within US universities and Fermilab, which is the US CMS host institution.

The project organization chart is presented in Figure 1.

The Software and Computing project has two subprojects:

• Applications Software
• User Facilities

The Applications Software Subproject will develop software

1. to support the development, modeling, calibration and commissioning of software related to detectors being constructed by USCMS;

2. to provide its share of the framework and infrastructure software required to support data analysis and simulation for CMS;

3. for remote collaborative tools to enable the distributed model of computing that enables members of USCMS to carry out analysis while they are at home in the US or resident at CERN; and

4. to satisfy any specialized needs required to carry out data analysis activities of interest to members of USCMS.

The goal of the User Facilities Subproject is to provide the enabling infrastructure of software and computing that will allow US physicists to fully participate in the physics program of CMS. To this end the subproject will acquire, develop, install, integrate, commission and operate the hardware and software for the facilities required to support the development and data analysis activities of USCMS. This subproject will include a major 'Tier1' regional computing center at Fermilab to support US physicists working on CMS. It is appropriately sized to support this community which comprises 20% of the full CMS collaboration.

A significant part of the plan, which affects many of the details that follow, is that main US-CMS regional center will be a tier1 center located at Fermilab. The Fermilab regional center will include substantial CPU, data storage and data access facilities. It will be able to deliver that data to other USCMS institutions through high-speed networks and will have a high bandwidth network connection to CERN. It will also include user support personnel, personnel to manage licenses and license acquisition, to contract for needed services. It will have the responsibility and personnel to develop or acquire any software that is required to carry out its production and operation activities. It will also provide support for many development activities during the detector construction period before data taking begins.

There are no specific plans for lower tier centers but it is reasonable to expect they will be proposed and to define how they will be included in the organization. A 'Tier2' regional center has more limited, but still broad, capabilities and about 10-20% of the capacity of a 'Tier1' center. Other 'special purpose' centers may also exist. These are focused on providing facilities that address only a limited area of analysis computing, such as Monte Carlo production. If these lower tiers come into existence, they will be integrated into this subproject so that all US 'production' facilities can be operated in a manner that focuses on the efficient use of resources in the achievement of the goals of the US CMS physicists.

Networks and networking will play a key role in the distributed computing model that will be implemented by this project. Key elements will be the high-speed connection to CERN and excellent connections among sites within the US. Since policy in this area is in a state of flux, it is not possible to specify how exactly the networking will be handled but it is clear that it will be an important part of the project.

The upper level project management entities are:

• the USCMS Software and Computing Board (SCB);
• the USCMS Project Management Group (PMG);
• the Fermilab Computing Division (CD);
• the Level 11 Project Manager (L1PM);
• the Level 22 Project Manager for the Applications Software subproject; and
• the Level 22 Project Manager for the User Facilities subproject.

The USCMS Software and Computing Board, the SCB, is a board whose membership is defined by USCMS. It is expected to be composed of physicists in USCMS with strong involvement and expertise in computing and data analysis. It may include other members such as outside experts or ex officio members from the project management or from CMS at USCMS' discretion. While the detailed nature of this board must be worked out by CMS and incorporated in its official management plan and its constitution, we present here an initial concept based on discussions that took place in the course of developing this plan. When the final plan is approved by CMS, it will be incorporated into a revised version of this document.

In the 'initial' concept, the SCB will be chaired by a Director of Software and Computing (DSC) who is elected by USCMS and is responsible for communicating US CMS' official position on software and computing issues, and for representing the SCB in various management organizations. The Board includes a Software and Computing Executive Committee (SCEC), chaired by the DSC, whose membership is composed of

• the DSC;

• the Technical Director of the USCMS project;
• the Level 1 Software and Computing Project Manager;
• the two Level 2 Software and Computing Project Managers;
• One technical expert; and
• One CMS institutional representative.

In addition, there will be at least one ex officio member

• The head of the Fermilab Computing Division.

The membership of SCB will be defined by US CMS. In our initial concept, it includes

• the DSC;
• the Level 1 and Level 2 Software and Computing Project Managers;
• the Spokesperson of US CMS;
• the Technical Director of the US CMS Construction Project;

• approximately 5 technical experts appointed by the DSC with the concurrence of US CMS and FNAL; and

• approximately 5 representatives of collaborating institutions, including the Physics Coordinator and Educational Outreach Coordinator, and additional representatives appointed by the US CMS Executive Board with concurrence of the Collaboration Board.

and as ex officio members

• the CMS Software and Computing Project Manager;
• chairperson of the CMS Computing Institutional Board; and
• the Head of the Fermilab Computing Division.

Except in unusual circumstances, meetings are open to all members of US-CMS, although only members of the SCB may participate in votes.

3.1.2 Roles and responsibilities

The SCB sets scientific and technical policy for software and computing in USCMS consistent with the scientific direction of US CMS and CMS. Technical, scientific, operations and resource allocation issues are to be discussed frequently by the SCEC, with the aim of addressing problems and creating policy as required for effective execution of the SC Project, and making recommendations to the Level 1 Project Manager. Scientific issues with a major impact on the physics results to be obtained by US CMS, will be brought to the full Executive BoardCommittee by the SCB chairperson. Issues affecting international CMS will be brought by the DSC to the international Software and Computing Board (CMS SCB) and by the USCMS Level 1 Project Manager to and the CMS Software and Computing Project Manager to the international Software and Computing Board (CMS SCB) and the international Software and Computing and Technical Board (CMS SCTB) as appropriatee.

Working with the SCB, the Level 1 Project Managerevel 1 PM, in consultation with US CMS, creates an approved plan for software and computing.

The SCB has the specific responsibility of providing the interface to USCMS in setting the requirements and developing the project plan and any overall policies associated with it. The L1 PM, working with the SCEC, has primary responsibility for the detailed formulation and execution of the plan, consistent with the policies determined by the SCB and US CMS. The SCB, including the L1 PM, works with US CMS physicists and the PMG to establish a project plan that is compatible with the available resources. The development of a sound project plan, which meets the needs of the collaboration and can be achieved within the available time and budget, is a key to the success of this project. The SCB further has the responsibility of making sure that the project plan is consistent with the plans and policies of CMS.

Once the project plan is developed, the SCB would work with the Level 1I Project manager to develop initial funding profiles, resource loaded schedules, budgets, Memoranda of Understanding (MOUs) and Statements of Work for the "baseline" project. These will be presented by the Level I Project Manager to the PMG and eventually to the funding agencies for approval.

Changes to the baseline plan will be covered by a "change control procedure" described below.

The SCB provides scientific and technical oversight of the project. The SCB periodically reports to the USCMS Executive Board on the status of the project. It addresses the issue of whether the project is indeed carrying out the policies set by USCMS and whether it is faithfully executing the project plan. The L1PM PM, as a member of the SCB, reports on whether the project is on schedule, on budget, meets the technical goals, and is being properly implemented. He/she informs the US CMS of the results of all reviews, internal and external, of the project and explains the actions that will be taken to address any problems that are identified. The SCB chairperson (DSC) participates in reviews by the funding agencies and in all deliberations and reviews of the PMG. In these reviews, the SCB through its Chair is especially responsible for addressing issues related to the successful implementation of scientific and technical policies of USCMS, while the L1 PM is particularly responsible for issues relating to the successful execution of the project plan.

Even after the approval of the project plan, it may be necessary to change the scope of the project or to change scientific or technical policies. This could occur if new collaborators were added, if technology changes made it necessary to make changes in technical direction, or if the physics results indicated that a change in the scope or balance of resources was required. In such cases, the SCB would propose such changes and work with the Level 1I Project manager to understand the implications of the proposed changes on the scope, budget, and schedule of the project. The DSC and the L1 manager would then present the change to the PMG, which would have the final approval on major changes in project scope or direction. Thresholds for change control are described below in section 4.

The Level 1I Project Manager is responsible for completing the project by achieving the approved scope within budget and on schedule. He/she must also insure that the individual deliverables of the experiment conform to the technical specifications that are set for them. Finally, he/she is responsible for doing all this in manner consistent with the scientific and policy decisions of USCMS.

The L1PM will be jointly appointed by Fermilab and will receive the concurrence of the US SCB, the DOE, and the NSF. During his/her tenure, the L1PM will be a permanent or term member of Fermilab's regular, full time staff. Administratively, the L1PM will be a member of the Fermilab Computing Division and his/her staff organization will reside with the CD. It is expected that the L1PM will have had experience and background in HEP experiment, software development, management, and operations issues and skills that would predict success as a project manager in this role.

The specific responsibilities of the L1PM include

1. providing technical input to the development of the baseline project plan especially with respect to budget and personnel requirements and with respect to the schedule, L1 milestones, and L1 deliverables;

2. executing the approved project plan and conforming to the technical and scientific policies of the USCMS;

3. developing the integrated Cost and Schedule Plan including setting, in agreement with USCMS, International CMS, the PMG and the US funding agencies, the Level 1I milestones and deliverables;

4. establishing and maintaining the project organization, within the Fermilab Computing Division, required to manage procurements, maintain schedules, submit reports, develop budgets, carry out quality assurance, maintain the official version of the project plan and the record of all revisions to it, maintain safety standards and records, etc;

5. developing the annual budget request to the DOE and NSF. This request is reviewed by the SCB and must be approved by the PMG;

6. acting as liaison between the project and the CMS Software and Computing Technical Board (SCTB);

7. appointing a deputy and the two L2 managers in concurrence with the SCB and the PMG. Consulting on and concurring with the appointment of L3 managers by the L2 managers;

8. developing or adopting general technical and quality assurance standards which deliverables must conform to. This includes making sure that USCMS software and facilities conform to applicable CMS and CERN standards and practices and can interoperate with them without unnecessary additional integration effort;

9. providing coordination and oversight to the Level 1I and Level 2II projects. Oversight may be accomplished by requiring appropriate regular reports, following and tracking the results of technical reviews, and conducting reviews using both internal and external committees or experts. Coordination may involve making sure that the User Facilities subproject can supply to the software development project test beds for major development activities;

10. reviewing and approving adjustments to MOUs and SOWs with collaborating universities and laboratories. These MOUs and SOWs are initially formulated as part of the project plan but adjustments may be necessary in the course of the project. Adjustments and modifications are reviewed by the SCB and the PMG;

11. allocating money and resources within the project. These allocations are largely set by the project plan and the annual budget request. However, adjustments are always necessary. The project plan will include a management reserve that can be committed by the L1PML1 PM during the course of the year based on the performance and need of the various subprojects. The L1PM must justify all major adjustments to agreed upon allocations to the PMG;

12. reporting variances from the scope, schedule, or cost estimates to the PMG and developing action plans for dealing with them. The L1PM informs the SCB of such variances;

13. exercising change control authority as granted in this plan and bringing change issues that exceed his/her authority to the attention of the PMG;

14. establishing advisory committees where appropriate;

15. providing reports and organizing reviews in conjunction with the funding agencies; respond to requests for information from USCMS, Fermilab, and the funding agencies; and

16. In conjunction with the Level 2II managers, with the advice of the SCB, and in conjunction with similar efforts at CERN, developing a technology tracking plan so that the project remains open to new opportunities that may arise even during the period of its execution and that it implements in its technical approaches the ability to change technologies easily. This is especially important given the length of the construction phase and the expected length of the operations phase of this project. This may involve working with the participating institutions to develop R&D, testing, and prototyping projects within CMS, in collaboration with CERN, other HEP or non-HEP academic institutions, or in collaboration with industry.

The Level 2II Project managers are appointed by the Level 1I Project manager. The SCB and the PMG concur in these appointments. The main task of the L2 Project managers is to successfully produce the deliverables of the project on schedule, within budget, and within their technical specifications.

3.3.1 The Applications Software Subproject

The L2 manager of the Applications Software Subproject is appointed by the L1PM with the concurrence of the SCB and the PMG. This position will normally be held by a physicist who is a member of USCMS and is expert in software development and HEP data analysis. The subproject will be responsible for developing software components, carrying out evaluations and feasibility studies, and performing various tests.

Responsibilities of the L2 Project Manager for Applications Software include:

1. developing, in collaboration with the SCB and the L1PM, the definition of the milestones and deliverables of the subproject;

2. developing, subject to review by the L1PM and the SCB, the technical specifications of each component and deliverable of the subproject;

3. defining, in consultation with and subject to the concurrence of the L1PM, the organizational substructure of the subproject. This includes defining the L3 projects and proposing, subject to the approval of the L1PM, the L3 project leaders;

4. developing, under the guidelines provided by the L1PM, the annual budget proposal for the subproject;

5. identifying resource imbalances or deficiencies within their projects. Making adjustments within prescribed limits of resources within their subprojects. Proposing adjustments to SOWs with collaborating universities and laboratories to the L1PM. Requesting allocations of management reserve and contingency from the L1PM where required to meet technical or schedule goals.

6. delivering the scope of the subproject on schedule, within budget, and in conformance with the technical specifications and quality assurance guidelines of the project;

7. organizing (and documenting) internal reviews to make sure that deliverables will meet the technical specifications and the quality assurance standards of the project;

8. being accountable for all funds allocated to the subproject;

9. maintaining the Cost and Schedule Plan for this subproject; and

10. providing reports as required to the L1PM, the SCB, and the PMG.

3.3.2 The User Facilities Project

The Level 2 manager of User Facilities Project is appointed by the L1PM with the concurrence of the SCB and the PMG. He/she will be experienced in and knowledgeable about HEP physics analysis production issues, hardware acquisition, operations issues, and the development of software components as relates to operations and production.

The Level 2 manager will have significant responsibilities connected with the USCMS Regional Computing Center at Fermilab and will normally be a permanent or term member of Fermilab's regular, full time staff.

Responsibilities of the L2 Project Manager for the User Facilities Project include:

1. developing, in collaboration with the SCB and the L1PM, the definition of the milestones and deliverables of the subproject;

2. developing, subject to review by the L1PM and the SCB, the technical specifications of each component and deliverable of the subproject which will include the acquisition of hardware components, contracting for services (such as network connectivity), acquiring software for operations or monitoring, and developing software to integrate components and to manage production or operations activities where required;

3. defining, in consultation with and subject to the concurrence of the L1PM, the organizational structure of the subproject. This includes defining the L3 projects and proposing, subject to the approval of the L1PM, the L3 project leaders. Each Tier2 and special purpose center will presumably be a Level 3 project and have its own Level 3 manager who will report to this L2 manager;

4. developing, under the guidelines provided by the L1PM, the annual budget proposal;

5. negotiating adjustments to the MOU with Fermilab, in particular with the Computing Division, for support of USCMS Regional Center;

6. negotiating annual MOUs with Tier2 or special purpose centers should they be approved;

7. delivering the scope of the subproject on schedule, within budget, and in conformance with the technical specifications and quality assurance guidelines of the project. This will include developing RFI's, RFPs, acquisition plans, implementation plans, technical specifications, acceptance tests and criteria, etc;

8. organizing (and documenting) internal reviews to make sure that deliverables will meet the technical specifications and the quality assurance standards of the project;

9. being accountable for all funds allocated to the subproject;

10. maintaining the Cost and Schedule Plan for this subproject;

11. providing reports as required to the L1PM, the SCB, and the PMG as required; and

12. working with potential candidates for Tier2 and special purpose centers in consultation with the SCB and the L1PM to prepare their proposals.

In response to the Department of Energy's and the National Science Foundation's request that Fermilab exercise management oversight for the USCMS Software and Computing Project, the mandate of the Project Management Group for the USCMS Construction Project, which is chaired by the Deputy Director of Fermilab, will be extended. Specific members will be added to the PMG for their expertise in software, computing, and data analysis issues. These will include

• The Chair of the SCB (DSC) and

• The L1 and two L2 Project Managers,

The US CMS Spokesperson, the Fermilab Associate Director for Information and Technology, and the Head of the Fermilab Computing Division are already members of the PMG with a particular interest in the Software and Computing project. Other members or observers with specific expertise or interest in this area and who represent USCMS, Fermilab, or the funding agencies may be added.

These members will comprise the PMG subgroup for the US CMS Software and Computing Project. The PMG Chairperson may chose to hold meetings of this subgroup, of the group associated with the Construction Project, or of both groups together based on the issues to be addressed. The chair of the PMG, in consultation with the L1PM and the DSC, prepare the agenda for these meetings.

The PMG also serves as a high level Change Control Board. The PMG receives the reports of the L1PM of the USCMS Software and Computing Project.

Oversight of the project is implemented in part through reviews. Along with providing routine interactions with the project management, the PMG will identify actions and initiatives to be undertaken to achieve the goals of the project including the allocation of both financial and human resources. The PMG also functions as the Baseline Change Control Board for the project.

Detailed change control thresholds are established in three areas: technical changes, schedule changes, and cost changes. Table 1 is a modified version of the change control thresholds from the USCMS Construction Project Plan for the detector and represents an initial plan for the US CMS Computing and Software Project.

The CMS collaboration has organized the software and computing efforts as a subsystem just like any other. This system is represented on the CMS Steering Committee by the "L2 Manager" for software and computing. The Technical activities of subsystems in CMS are managed in a "Subsystem Project Office". For this particular subsystem, this entity is called the Software and Computing Technical Board (SCTB) which is chaired by the L2 manager. The collaborative aspects of a subsystem in CMS are handled by an "Institutional Board". To create a strong linkage between the USCMS Software and Computing Project and the overall CMS Project, the CMS L2 manager and the chairperson of the CMS Computing Institutional Board, the SCB will be ex officio members of the USCMS SCB. The DSC will have the responsibility of acting as liaison between the USCMS Computing Project and the CMS Software and Computing Board and the USCMS Software and Computing Project Level 1 Project Manager will act as liaison to the CMS Software and Computing Technical Board.

The US CMS Project is responsible for the construction of specified elements of the CMS detector as designed by US physicists. The scope of the Project extends from the detector elements to transducers to front end electronics, through L1 pipelined triggers, L2 and L3 triggers and data acquisition including the L3 computing farm and data logger. The US Software and Computing Project concerns the steps needed to complete these tasks all the way to physics analysis. Necessarily, the CMS experiment is a unitary enterprise and the two distinct projects must be well coordinated so that the needs of US CMS physicists are seamlessly met as they work on tasks from the detector elements to the physics analysis.

The USCMS Project Management Plan for the construction of the detector contains a detailed description of the interaction between the project staff and the funding agency (DOE and NSF) personnel. Key elements of this are the Joint Oversight Group, the U.S. LHC Program Office, the Chicago Operations Office of the DOE, the US LHC Project Office, the Fermilab Director, and the Fermilab Deputy Director. While the details of the oversight by the agencies will be worked out during the formulation of the project, we assume an arrangement similar to the one that is in place in the construction project. In particular, the proposed organization chart shows that the PMG reports to the Joint Oversight Group.

The Computing Division is home to the USCMS Software and Computing Project within Fermilab. The Division supports the project in many ways. The L1PM and the L2PM for the User Facilities subproject are members of CD. The 'project office', which provides administrative resources to the project management, resides within CD. The Head of the Computing Division is a member of the PMG and is an ex officio member of both the SCB and its SCEC. The staff for the Tier1 Regional Center and much of the staff for the User Facilities Project are members of CD.

This plan applies to the 'initial development phase' for the software and facilities. Once that phase is complete, the software and the facilities go into the "operation, support, and development" phase. This should occur about a few years after CMS starts taking data. At that point, there should be a new 'operations plan' which would replace this plan. It is very important to recognize that software development and hardware evolution will continue throughout the life of CMS. The resources required for the ongoing operation of the facilities and evolution of the software and hardware are quite significant. The operation of the Regional Center is probably a 15-20 year commitment. For at least 2/3 of its lifetime, it will be in an 'operations' rather than 'construction' phase but continual investment in software development, technology tracking, and R&D throughout this period will be essential if the facilities are to continue to serve the interests of USCMS as physics and computing technology move forward. Similarly, the scientific software will be in a state of continual development and evolution throughout the operations period. This will be driven both by changes in physics goals and analysis techniques and by changes in underlying software and hardware technologies.