GriPhyn Collaboration

LIGO-T010166-00-ELASER INTERFEROMETER GRAVITATIONAL WAVE OBSERVATORY- LIGO –CALIFORNIA INSTITUTE OF TECHNOLOGYMASSACHUSETTS INSTITUTE OF TECHNOLOGYDocument TypeAnnual ReportLIGO- T010166-00-E30 June 2001LIGO DCC Archival Copy:GriPhyN Annual Report for 2000–2001NSF Grant 0086044GriPhyn CollaborationCalifornia Institute of Technology LIGO Laboratory - MS 18-34Pasadena CA 91125Phone (626) 395-212Fax (626) 304-9834E-mail: info@ Massachusetts Institute of Technology LIGO Laboratory - MS 16NW-145Cambridge, MA 01239Phone (617) 253-4824Fax (617) 253-7014E-mail: info@www: /Final VersionJune 30, 2001GriPhyN Annual Report for 2000–2001NSF Grant 00860441 The GriPhyN Project (2)1.1 Introductiongoals (2)and1.2 Participants and first year funding (2)structure (3)1.3 Management2 Activities and Findings for 2000−2001 (5)2.1 Creation of a Working Grid (5)2.2 Meetings (5)2.3 Testbeds (5)activities (6)2.4 Research2.4.1 ComputerScience (6)2.4.2 Virtual Data Toolkit (9)2.4.3 ATLAS Research Activities (9)2.4.4 CMS Research Activities (11)ResearchActivities (12)2.4.5 LIGO2.4.6 SDSS Research Activities (13)2.5 Education and Outreach (14)2.6 Connection to other efforts and the wider community (14)2.6.1 Coordination with Other Data Grid Projects (14)Coordination (15)2.6.2 Network2.7 External Advisory Committee meeting (15)3 Products resulting from our work (16)3.1 Publications (16)documents (18)3.2 Internalsite (18)3.3 Web4 Plans for next year (2001 − 2002) (18)Appendix I: Participants in Project (asterix marks institutional contact) (19)Appendix II: CMS Information Release About Large-Scale Test (23)1 The GriPhyN Project1.1 Introduction and goalsGriPhyN is a collaboration of information technology (IT) researchers and experimental physi-cists who aim to provide the IT advances required to enable Petabyte-scale data intensive science in the 21st century. Driving the project are unprecedented requirements for geographically dis-persed extraction of complex scientific information from very large collections of measured data. To meet these requirements, which arise initially from the four physics experiments involved in this project (ATLAS, CMS, LIGO and SDSS) but will also be fundamental to science and com-merce in the 21st century, the GriPhyN team will pursue IT advances centered on the creation of Petascale Virtual Data Grids (PVDG) that meet the data-intensive computational needs of a di-verse community of thousands of scientists spread across the globe.This report is being written at the start of GriPhyN’s ninth month of funding. We have started substantial research IT activities in areas such as data management, query optimization, data grid architecture, as was reported at our all-hands meeting in April and described in Section 2.3 be-low. We have also performed substantial early experiments, notably a large CMS computation across a Data Grid spanning Caltech, UW, and NCSA. Much initial effort has also been spent on recruiting and hiring students, postdocs, scientists and staff; writing detailed requirements docu-ments; setting formal milestones; implementing a management structure and organizing our col-laboration in line with these milestones; and interacting with other Grid projects and the four ex-periments which compose GriPhyN. During this time, we have had two collaboration wide meetings, at which students and postdocs made significant contributions, as well as numerous smaller meetings for CS research groups and CS-experiment interactions. The report below dis-cusses these topics in more detail.1.2 Participants and first year fundingA table showing a list of participants and their affiliations is shown in Appendix I. We have ex-pended much effort in finding and hiring personnel, but have experienced a somewhat slower rampup in personnel than we expected when the proposal was submitted, even though we had assumed 50% of our total FTE count in the first year. At the time of this report (June 2001) we have hired or made offers to approximately 80% of our full complement of hires (including peo-ple who were promised as matching contributions at Florida, Boston U and Indiana. We have had a particularly difficult time finding suitable candidates for the Project Coordinator position, in spite of a large cross-disciplinary search. Nevertheless, we have identified one and possibly two candidates for the position and we hope to have a Project Coordinator hired sometime in July. On the other hand, we have identified an outstanding candidate for the Outreach Coordina-tor (Manuela Campanelli) and she will start in Fall, 2001.Although our hiring has gone somewhat more slowly than expected, we have incurred some ad-ditional costs for our External Advisory Committee, travel and web development. We estimate that slightly more than 20% of our funds will not be expended at the end of the first year, though it is hard to be precise more than three months before that date (12 subcontracts need to be tracked and the dates of some hires are not precise). We are setting up procedures to track our total expenses every three months to aid our planning and reporting.1.3 Management structureThe management of the GriPhyN project has been conducted largely in accordance with the management plan presented in the proposal and in the more detailed GriPhyN Management Plan1 prepared in November, 2000. The organization chart from the latter document is repro-duced below. We describe briefly the activity in each of the chart's sections.• The Project Coordination Group is chaired by the Project Directors Avery and Foster. It meets once or twice each week by telephone conference to review and plan the major ac-tivities of the project.• The Applications Committee has not had formal meetings, but its members, the represen-tatives of the four component physics collaborations, have worked actively with the CSResearch Committee to define requirements and goals for integration of GriPhyN mid-dleware into each collaboration's testbed as a function of time.• The Virtual Data Toolkit Committee is chaired by Miron Livny. It has worked actively with the CS Research Committee to define the first release of the Toolkit as described in Section 2.2.2 below.• The CS Research Committee, chaired by Carl Kesselman, has met as needed during the past several months to coordinate work on the four areas of CS research.• The Technical Coordination Committee has worked only informally so far.• The External Advisory Committee has been formed and has met to review the progress of the GriPhyN project during the general GriPhyN meeting at ISI in April. The EAC re-port2 can be accessed at the GriPhyN web site.3• An active search has been underway for the Project Coordinator position. We expect to fill the position by the middle of July 2001.2 Activities and Findings for 2000−20012.1 Creation of a Working GridThis past spring, a collaborative effort involving physicists and computer scientists from Caltech, the National Center for Supercomputing Applications (NCSA), and the University of Wisconsin-Madison produced a working example of the Grid in action. The details are described in Appen-dix II, which will appear in NCSA Magazine.2.2 MeetingsThe size, scope and interdisciplinary nature of GriPhyN require building effective communica-tion channels between the members of the collaboration. Early on we decided to use face-to-face meetings to establish and maintain such channels. In early October we had our first general meet-ing at Argonne National Laboratory4. The main objective of this meeting was to introduce the different groups to each other and to agree on a plan for jumpstarting the collaboration. Each of the physics groups gave an overview of the overall scientific mission of the group and the design principals and structure of their software and hardware infrastructure. Carl Kesselman presented a strawman architecture for a Virtual Data Grid and outlined a plan for the first year. A key ele-ment of this plan was a series of meetings between each of the four Physics experiments and small groups of Computer Scientists to discuss in details experiment specific technical and con-ceptual aspects of Virtual Data Grids.More than a dozen such meetings took place in the following six months. The format, dynamics and frequency of these meetings varied from one experiment to the other. However, they all con-tributed to a collection of extremely useful documents – use cases and requirements on the phys-ics side and a Virtual Data Grid architecture document on the computer science side. Two meet-ings in December 2000 – one only of computer scientists and one of representatives from the four experiments and the computer scientists – lead to the formulation of the GriPhyN data grid reference architecture.Our second “all-hands” meeting5 was held April 9-11, 2001 at USC/ISI in Marina del Rey. More than 60 members of the collaboration attended the three-day meeting, plus a few visitors from other projects and the External Advisory Committee. The first day was devoted to presentations and discussions on the requirement and architecture documents. Early results from the computer science efforts were reported on the second day. Three planning working groups took place in parallel on the morning of the third day. Each group focused on a key component of Virtual Data Grids. The afternoon was devoted to two invited talks that challenged the attendees with vision-ary ideas about distributed computing and data management. Details and the contents of almost all talks and supporting reference6 documents can be found at the meeting web site.5Another Computer Science – application group meeting is planned for August 2-3, 2001 in Chi-cago. The next all-hands meeting will be held the week of October 15, 2001 in California.2.3 TestbedsThe activities for developing and deploying testbeds to test components of the GriPhyN architec-ture have been pursued separately by the different application groups, as discuss in the subsec-tions in Section 2.3. The testbeds for CMS and ATLAS are integrated with the production activi-ties of their international collaborations and are expected to draw on these international facilities for limited GriPhyN tests next year. These application testbeds will be operated independently for the near future. We are exploring the idea of conducting occasional wide scale tests that would require two or more testbeds working coherently, but we anticipate that these kinds of tests would take place after 2002 when more tools are available.2.4 Research activities2.4.1 Computer ScienceComputer science research during the first nine months of the GriPhyN project has focused on two main activities:• definition of a baseline virtual data grid architecture• research on basic virtual data grid technologies, such as scheduling, execution manage-ment, and virtual data instantiation.We review the progress in each of these areas in the following sections.2.4.1.1 Virtual Data Grid ArchitectureArchitecture development is a focused activity whose goal is to define a virtual data grid refer-ence architecture that can guide the development of the virtual data toolkit as well as provide a framework for performing initial application experiments. We have produced two main docu-ments: “A Data Grid Reference Architecture” and “Representing Virtual Data: A Catalog Archi-tecture for Location and Materialization Transparency.” Contributions to these documents were made by many of the participating GriyPhyN institutions.Our initial reference architecture is based on existing data-grid technologies including metadata catalogs, replica catalogs, software catalogs, Grid resource and data management protocols. By heavily leveraging existing data grid components, we anticipate rapid construction and deploy-ment of a virtual data grid testbed, which Array will enable application experiments. Thefigure to the left illustrates the main com-ponents of the reference data gridarchitecture. One key element is thespecification of a set of catalogs used todetermine if virtual data has beenmaterialized and if not, how to locate andinvoke the transformations required togenerate the requested data. A significantissue yet to be resolved is how to “name”virtual data so that its materialization statecan be determined. A number of mined. A number of alternatives are currently being investigated, including the use of applica-tion or domain-specific naming strategies that leverage existing naming conventions within anexperiment.2.4.1.2 Research in Virtual Data TechnologiesTo date, research activities on various specific virtual data technology areas have been largely decoupled. The reference architecture will provide a integrating framework for many of these efforts, and will form the basis of cross-project collaboration. Specifically, during the next year, we anticipate that more mature research activities will be integrated into the overall system archi-tecture and be deployed as part of our testbed environment for evaluation within the context of application experiments. We highlight some of these ongoing research activities below.Agent based scheduling. Request management in virtual data grid environments is complicated by the complexity of the requests, the potential duration of request execution, and the dynamics of the underlying execution environment, including component failure. For this reason, request planning and execution management for virtual data grids must be very flexible, and capable of preplanning request execution in the case of failure. A promising approach is to exploit Intelli-gent Agents as a technology for implementing request management. At USC/ISI, we have initi-ated an investigation of this approach using the Electric Elves software agents as a framework for information gathering, planning and execution monitoring. To date, we have demonstrated the integration of Grid resources into the Elves environment and have used KQML (Knowledge Query and Manipulation Language) based commands to initiate Grid operations. Our next step will be to build a Agent knowledge base that performs intelligent recovery from failed data-movement operations.Improving Query Performance. Research at UC Berkeley has been focused on developing techniques to reduce latency and improve the interactive performance of access to virtual data products. The work performed to date has focused on two promising directions: 1) Query relaxa-tion and associated caching techniques, and 2) dynamic prefix caching for large files.Query relaxation is aimed at providing more answers to user queries faster in cases where there are no existing (or readily available) virtual data products that exactly match a given query speci-fication. In this situation, the query relaxation module looks for cached data products that may overlap or be closely related to the requested data. The user is presented with a menu of such products and in indication of the expected time required to obtain them. He/she may then choose to view one of these alternative products while waiting (or perhaps before asking) for the pro-duction of a high-latency data product. The ability to relax queries changes the caching problem somewhat, as now individual data products can "cover" multiple queries. Thus, caching policies to maximize query coverage have been developed and an initial simulation study has been per-formed.Dynamic prefix caching aims to improve the cache hit rate by caching only the initial portion of large data items. When a request for such an item is received, the initial portion is sent to the user while the remainder of the item is retrieved from the slower location. If the prefix size is sufficiently large, then the latency of accessing the slower location can be completely hidden. This approach, however, may have the side effect of increasing the load on the slower site. Thus, a dynamic approach as been developed, that can balance load vs. cache hit rate by increasing the prefix size for highly popular items. A simulation study of this technique is currently under way. XML-based data manipulation: The virtual data grid requires mechanisms for the creation of derived data products. At the same time, the virtual data grid requires the ability to query the collections of derived data products. An emerging standard for querying and manipulating XML files is Quilt. A standard version of Quilt is being developed called XQuery. Of great interest iswhether XQuery can be extended to support scientific data types. This will make it possible to manipulate an XML encoded digital object directly within the same language that supports in-formation discovery.At SDSC, Xufei Qian, under the direction of Amarnath Gupta, added data types to XQuery, added overloaded operators to support scientific manipulations, and interfaced the system to the Extensible Scientific Interchange Language (XSIL). XSIL is an XML markup language used to describe LIGO data, developed by Roy Williams (Caltech). The resulting system was used to query and manipulate XSIL encoded data files. In particular, array operations were added to XQuery, including accession, element summation, array summation, and subsequence com-mands for concatenation. The resulting prototype demonstrated that it is possible to integrate query and manipulation commands for XML encoded digital objects.Data Management: There are a number of research activities that are investigating various as-pects of location transparency, specifically with respect to creating and managing replicas of ma-terialized data. Work at the University of Chicago by Kavitha Ranganathan has used simulation studies to examine alternative replication and caching strategies on application response time and bandwidth requirements. A simulator has been developed that represents data movement within a hierarchically organized data grid, and 5 different replication strategies have been implemented and compared via simulation studies. Complimentary activities at Lawrence Berkeley Labora-tory are using analytical and simulation studies to understand the performance of alternative caching strategies specifically targeted towards hierarchal storage managers. Additional simula-tion studies by Matei Ripeanu at the University of Chicago have examined TCP performance in typical replication scenarios involving parallel bulk transfer of many files. Our next step in the work will be to take the knowledge gained from simulation studies, and instantiate it in request planning strategies that can then be evaluated within the context of specific data access scenarios generated by one or more of the physics experiments.Flexible Storage Technology: At the core of virtual data grid technology is the need for effi-cient and manageable physical data storage and movement. Along these lines, at Wisconsin we are currently developing NeST (Network Storage Technology). NeST software transforms a commodity workstation or PC running a commodity operating system into an easy-to-administer and high-performance storage appliance.To operate effectively in a Grid environment, NeST will support multiple data transfer protocols, enabling integration into local sites with specific protocol preferences (e.g., NFS). However, standard Grid lingua franca such as GridFTP will also be supported. One of the main challenges of NeST is the combination of these protocols into a single, efficient, and yet easy-to-maintain software base. We are also developing the necessary software technology to allow NeST to con-figure itself to the vagaries of the underlying software and hardware platforms. In particular, NeST can statically choose between multiple concurrency architectures depending on which is best on the given platform, and in the future NeST will be able to select the most appropriatefile-access methods given a particular storage platform.We are currently investigating the following additional research issues within the NeST domain: how to build generic support for third-party transfers and wide-area caching, how to integrate NeST effectively into an opportunistic environment, and how to incorporate resource limitation and management into the NeST framework in a robust and yet flexible manner.Performance Monitoring and Analysis: The work at Northwestern University has focused on the development of an infrastructure, called Prophesy, to automate the process of generating ana-lytical models. Prophesy automates the following processes: instrumenting of code, recording of performance data into a relational database, and developing analytical models based upon em-pirical data. Prophesy also accepts data generated by other tools such as FMMPI, gprof, pixie or Rabbit. The data from these tools is placed in the performance database, which includes infor-mation about the context in which the data was collected as well as the data. This data is used by the data analysis component to produce an analytical performance model at the level of granular-ity specified by the user, or answer queries about best implementation of a given function.We are also developing a performance requirements document for the GriPhyN project. The goal of this document is to identify the performance monitoring and modeling requirements for the four HEP experiments and the CS related activities. Our initial focus is on the ATLAS ex-periment; future work entails extending the requirements to other three HEP experiments. In terms of requirements, this document attempts to identify (1) what system components and ap-plication features require monitoring and recording of performance data and (2) what perform-ance models are needed and the required accuracy. Once these requirements are identified, we can leverage from current work with performance tool development to identify the suite of tools needed.2.4.2 Virtual Data ToolkitIn preparation for the release of the first version of the Virtual Data Toolkit (VDT) by the end of the year, the Globus and Condor Teams focused on adapting, enhancing and hardening existing software tools. The original timetable and scope of this version were modified to reflect the sig-nificant reduction in first year funding allocated to the VDT activities. The VDT work was guided by the requirements and timelines gathered from the experiments and the design and im-plementation principals established by the GriPhyN architecture document. Catalogs play a key role in this architecture. A number of different data catalogs were prototyped and carefully evaluated and tested. All catalogs use a common software foundation provided by the Globus Toolkit. The performance and robustness of the GridFTP component were also addressed. We expect this component to be the “work horse” of any Data Grid and therefore pay special atten-tion its performance and robustness. The runtime support infrastructure of Condor was enhanced to fully utilize the capabilities of GridFTP and the security infrastructure of Globus. New fea-tures were added to the Directed Acyclic Graph Manager (DAGMan) of Condor and extensively tested. The DAGMan provides advanced job control services and is expected to serve as the agent responsible for dynamically materializing Virtual Data. Early versions of VDT compo-nents were successfully used for distributed production and reconstruction of simulated events for CMS.2.4.3 ATLAS Research ActivitiesATLAS-GriPhyN activities proceeded along two fronts. First, ATLAS physicist-developers and GriPhyN computer scientists met on two occasions7 to discuss computing methods in high-energy physics, the architecture and status of the Athena8,9,10 analysis control framework, and virtual data concepts specific to the Athena framework. A prototype virtual data query use case was developed, and development of associated Athena and ATLAS database software has been incorporated into the ATLAS Software and Computing Project Grid Plan11. Second, severalmore practical projects were undertaken: grid testbed development, prototype Tier 2 center selec-tion and development, grid account registration software, and grid access software.The testbed development proceeded throughout the year, with a kick-off workshop12 at the Uni-versity of Michigan, followed by bi-weekly meetings. There are five university sites participating in the activity: Boston University, Indiana University, University of Michigan, Oklahoma Uni-versity, and the University of Texas at Arlington; three national laboratories participate: Argonne National Laboratory, Brookhaven National Laboratory, and Lawrence Berkeley National Labo-ratory. Approximately 15 users are participating in this development. Each site has contributed existing facility resources to be dedicated for this purpose, and site administrators have deployed current releases of grid toolkits including Globus and Condor. Accounts have been created and grid certificates exchanged for the current group of testbed participants. Each site has installed a complete ATLAS software environment, and work is ongoing to develop an ATLAS “kit” for simple distribution and update to future grid nodes. The primary applications to be run on the system in 2001 include fast Monte Carlo simulations executed within the Athena control frame-work, and full detector simulations using legacy FORTRAN codes. The infrastructure will be used during Mock Data Challenges in 2002 and 2003 to validate the LHC Computing Model13. Indiana University and Boston University were selected through a competitive process as the prototype Tier 2 centers for the U.S. ATLAS-GriPhyN project. Development of the first centerat Indiana University has begun. A system consisting of a 16-node (32 processors) Linux farm, 500 GB RAID disk, plus access to HPSS hierarchical storage facilities for archival data has been deployed. The system is managed by professional IT staff in a machine room adjacent to theI2/Abilene Network Operations Center (NOC) in Indianapolis. Current releases of grid packages have been installed. A dedicated AFS server ring will be installed this August.GriPhyN and ATLAS users will use a secure facility to support grid account request and authori-zation. GRIPE14, (Grid Registration Infrastructure for Physics Experiments) was implemented by four computer science graduate students at Indiana University under the supervision of Profs. Gardner and Bramley. The system facilitates account creation across all administrative domains. GRIPE users choose trusted sponsors and grid site-specific resources and submit account re-quests through the system. Site administrators interact with GRIPE to communicate to users and exchange Globus certificates. The system is being tested this summer by the testbed working group, and is the primary mechanism for account creation at the Tier 2 centers.A web-based portal for ATLAS users is currently under development. GRAPPA15 (Grid Access Portal for Physics Applications) will provide ATLAS and GriPhyN users a single point of access to grid resources. The system provides a simple utility for job submission to Condor flocks and other available grid resources. The user interface will use script management tools based on Common Component Architecture Toolkit (CCAT) software being developed by Profs. Bramley and Gannon at Indiana University. The first prototypes will be available during Summer 2001. Summary of meetings held related to ATLAS-GriPhyN project, with strong participation from GriPhyN personnel:• 2nd International ATLAS Grid Workshop16, CERN, Geneva, Switzerland, September 29-30, 2000.• 3rd International ATLAS Grid Workshop17, CERN, Geneva, Switzerland, January 25, 2001.。