8 2 v 8 4 1 8 1 x Z < @ | 4 u2 u2 2 ^ ^ ^ ^ 2 2 2 $ V 2 2 2 V 2 2 2 Evaluation Criteria for CMOCs Background JCOMM-4 adopted Resolution 2 with Annex 2 having to do with a Marine Climate Data System (MCDS) and with an entity called a CMOC. The Resolution required the development of accreditation criteria for new CMOCs and evaluation criteria for previously accredited CMOCs. This document provides these criteria. Questions regarding the background for the discussion and adoption of Resolution 2 should reference JCOMM documents, and subsequent documents from MCDS meetings. It is accepted that the criteria presented here are a first attempt to clarify how CMOCs will be assessed. The intention is to make the accreditation and evaluation processes as transparent as possible. It is recognized that the process laid out here will put additional burdens on staff who already volunteer activities to support the international data systems. Both the criteria and the process are likely to require alterations as experience is gained. Required changes will need the same formal approval process by JCOMM and IODE as these criteria undergo. Accreditation Process for a new CMOC The agency proposing to host a CMOC must follow the procedures listed here. They must prepare a document that clearly addresses all of the obligations (i.e. capabilities, functions and a commitment of a minimum time period of operation), capabilities and corresponding functions described in Annex 2 to Recommendation 2 of JCOMM-4. Explicitly, this should include descriptions of the proposed outcomes/outputs (services, products) and how are these contributing to WMO and IOC requirements for managing marine-meteorological and oceanographic climatological data. The proposing agency should also describe what their annual report will contain. To conduct an accreditation of a proposed CMOC, a committee will be formed of at least three members appointed by the JCOMM DMCG. Terms of Reference for the Accreditation Committee are found below. Further actions will proceed as described in Annex 3 to draft Recommendation 2 (JCOMM-4) Accreditation Criteria Annex 2 and 3 to Recommendation 2 of JCOMM-4 are quite extensive in describing the obligations, capabilities and functions for a CMOC. The following criteria have been devised to test these conditions. They are presented in the form of questions, for which a simple yes or no is possible. Generally, if there is uncertainty in achieving a yes response, the criterion should be judged as unmet. Does the scope of activities (rescuing, collecting, controlling quality, calibration and bias correction, processing, archiving, sharing, distributing and mirroring data, metadata and information, products and services) have any unnecessary overlaps with existing activities of an agency operating within the JCOMM data system, with activities of an IODE NODC, with the High Quality Global Data Management System for Climate, with an existing World Data System centre, or with some other well established data management activity? If so, is the added value of the overlap activity well explained and does it warrant the establishment of the CMOC? If the scope of activities is regional, is there evidence of support from Member/Member States from the region (e.g. expressions of support)? Does the proposal for the CMOC explain clearly how its activities will be coordinated with other relevant, existing systems (such as with well described procedures, letters of cooperation, expressions of support from major data providers)? Is the proposed CMOC activity well defined, scientifically sound (e.g. supported by a publication record), and does it fill a clearly articulated and real gap in formal WMO or IOC data management activities? Are the variables to be treated not GCOS ECVs? If they are ECVs, is the added value that the proposal brings to management of these variables sufficient to warrant the overlap and the creation of a CMOC? Are the processes for assessing and assigning quality indicators well documented and is this documentation easily available? Will the proposed procedures ensure that the quality within all of the CMOC data sets are internally consistent? Are there any restrictions on access to the data, metadata and information served by the proposed CMOC? If so, do these go against the spirit of free and unrestricted access? Are the infrastructure, experience, financial resources and assigned staff for the proposed CMOC sufficient to meet the planned operations? Interoperability means the data, metadata and information are widely visible and available through the WIS and/or ODP. Will this interoperability function be met by the proposed CMOC? Does the CMOC proposal clearly describe the data domains of its operations in type(s) of data, geographic, and temporal coverage? Are there any domain specific procedures to be applied by the proposed CMOC? If so, is their purpose (such as enhancing interoperability, ensuring data quality and consistency, improving access, improving coordination, or other functions) well described, useful and is documentation of these procedures easily available? Are the proposed choices for procedures, standards and best practices to be followed suitable and adequate for data quality and management? Where applicable do they use procedures of the Ocean Data Standards and Best Practices Catalogue? If not, will they propose a new standard or best practice for consideration? Is there a clear description of what the CMOC will undertake to mirror their processes, data and metadata? Is there evidence (e.g. a letter of agreement) of a cooperative arrangement with an existing CMOC or another established and ongoing data management system for this mirroring? Is the mirroring process sufficiently robust to be reliable and timely? Are the proposed methods of version control for data sufficient so that identical copies of data may be distinguished from near identical copies? Are the proposed methods of version control for metadata sufficient to distinguish different versions of metadata? Are the proposed methods of version control for processes sufficient so that data users can be certain about the processing steps through which data have passed? Annual Reporting and Performance indicators These are intended to be used to demonstrate that the CMOC is meeting its obligations and functions. The CMOC will report in writing to the chair of DMCG annually no later than 31 Jan of each year. The report can be structured in any way so long as the information listed here is all included. Other information that is helpful in informing on the operations of the CMOC in the past year is welcome. Reports should be limited to 20 pages or less. Provide statistics on the type and volume of data processed in the past year and compare this to previous years of operation. If reprocessing of data already received in a previous year is included in these statistics, provide an explanation of the reasons for the reprocessing. If CMOC operations have been changed to include new kinds of data, metadata and products or to exclude previously handled items, provide an explanation for the changes. Provide statistics on the type and volume of data, metadata and products actually served in the past year and compare this to previous years of operation. If there have been changes in serving operations, provide an explanation. Ensure that data and information served through ODP and WIS are clearly described. Describe the mirroring functions of the CMOC both for its own data and metadata and in support of another CMOC. Provide statistics to demonstrate the robustness and timeliness of the mirroring operations. Provide information to describe the coordination activities undertaken with other CMOCs, IODE NODCs, with the High Quality Global Data Management System for Climate, with an existing World Data System centre or with another established data management system. Describe any changes in infrastructure or staff in the past year. If one or more variables are GCOS ECVs, describe the coordination activities in cooperation with the data system handling the ECV and explain the value added operations provided by the CMOC for these variables. Provide an updated list of documentation for quality management, standards or best practices as practised at the CMOC. Highlight those produced or updated in the past year and describe how all documentation is made available. Has there been any noticeable changes in aspects (e.g. Quality, timeliness, new instrumentation, etc.) of the data received and processed in the past year compared to other years. If so, describe the change and explain the reason. Describe any actions taken as a result of this change. Have there been any changes in access restrictions to the data, metadata, products or services. If so, explain them. Provide statistics by type of data on the spacial and temporal coverage of the data managed by the CMOC during the past year and contrast these to previous years. Describe any changes in processing of data or information during the past year. Explain why these changes were made. Describe how the activities of this CMOC reflect common procedures employed by others in the CMOC network. Describe any interactions with other individuals or organizations where the CMOC responded with assistance or provided advice regarding standards and best practices (e.g. on data rescue, collection, processing, archival, and distribution of marine- meteorological and oceanographic data, metadata, and products) to Member/Member States. Provide, as appropriate, a list of citations, or statements from users that show usage of the CMOC operations, products or services. Provide statistics or descriptions of the mirroring process that demonstrates it is functioning as described in the original proposal for the CMOC. The Evaluation Review Process for an existing CMOC Annex 3 to draft Recommendation 2 (JCOMM-4) states that the performance of an existing CMOC will be reviewed once every 5 years by the DMCG. This review will be conducted by a committee of at least 3 members appointed by the DMCG. It may be necessary for one or more members of the review committee to visit the CMOC. In this case, it is expected that the CMOC will finance the visit. A review committee will be formed of at least three members appointed by the JCOMM DMCG. Terms of Reference for the Review Committee are found below. Further actions will proceed as described in Annex 3 to draft Recommendation 2 (JCOMM-4) 5 Year Review Criteria Is the objective of the CMOC or scope of activities (rescuing, collecting, processing, archiving, sharing, distributing and mirroring data, metadata and information, products and services) of the CMOC still relevant given any changes that may have taken place in the management of met-ocean data in the past 5 years? Is the coordination of CMOC activities with other CMOCs and existing systems functioning at a sufficient level? Has the CMOC been active in coordination activities and meetings in the CMOC network? Is the infrastructure and assigned staff continuing to support CMOC operations adequately? Have there been any improvements over the past 5 years to enhance operations? Is there any documented community support from groups outside the CMOC network for the operations of the CMOC? Have there been any changes in GCOS ECV designations that impact the operations of the CMOC? Does the CMOC continue to demonstrate added value to managing GCOS ECVs? Do the processes used for quality management continue to be sufficient? Has the CMOC provided written reports on or before 31 Jan of each year of its operation? Are there any changes to restrictions on access to the data, metadata and information, products or services? If so, do these go against the spirit of free and unrestricted access? Interoperability means the data, metadata and information are widely visible and available through the WIS and ODP. Is this interoperability function being met? Have other interoperability operations been established? Are the data domains still clearly described in type of data, geographic and temporal coverage? Do the data and information management procedures applied (such as enhancing interoperability, ensuring data quality and consistency, improving access, improving coordination, or other functions of a CMOC) continue to be well described and useful? Does the scope of activities have any overlaps with existing activities of IODE NODCs, with the High Quality Global Data Management System for Climate, with an existing World Data System centre or another well established data management activity? If so, is the added value of the overlap well described and warrants the continuing operations of the CMOC? Do the procedures, standards and best practices that are followed continue to be suitable and adequate for defining data quality and management? If the standards or best practices chosen at the time of accreditation or the last review were not part of the JCOMM Catalogue of Standards and Best Practices, has a proposal been made to the Catalogue to incorporate the new procedures? Does the mirroring arrangement with another CMOC, or another established and ongoing data management system continue to function in an appropriate, robust and timely manner? Is the documentation of the accreditation evaluation and annual reviews easily available? Are the quality processes and procedures of the existing CMOC appropriately in agreement with the rest of the CMOC network? Are the methods of version control for data, metadata, products and processes sufficient for a user to adequately distinguish differences between versions? _______________ Annex 1 of Annex VI Terms of Reference of the CMOC Accreditation Committee To conduct an accreditation of a candidate CMOC, an independent committee will be formed of at least three members appointed by the JCOMM DMCG. The Committee shall: Elect a chair; Review the document submitted by the CMOC candidate, with special attention to the accreditation criteria. The proposal will be judged against each of the criterion and will be scored as described in the document on establishing a CMOC. To receive accreditation, a proposal needs to satisfy the committee that all criteria are fully met; there must be unanimity among committee members; Decide what expressions of support and from which groups are appropriate for a proposal (criteria #2, #3). The rationale will be explained in the committee report; If required, designate someone to consult and/or undertake a visit of the candidate CMOC in order to (i) inform the candidate about possible elements to be clarified, and seek clarification, (ii) verify specific functions and capabilities of the candidate CMOC, (iii) negotiate needed changes to the proposal, and (iii) submit a report back to the committee with recommendations within the time period specified in Resolution 2 and its annexes; Prepare a written report of their evaluation that explains the results. In particular, if the proposal is deemed insufficient to pass one or more criteria, the committee will explain their reasons and may suggest a possible remedy. Submit the report to the DMCG and to the authors of the proposal. Upon request, provide copies of the proposal and evaluation report to any JCOMM or IODE member requesting it. The Chair of the Committee shall: Report the results of the evaluation to the DMCG. Membership: IODE Representative JCOMM Representative (WMO side) Other representative(s) as needed _______________ Annex 2 of Annex VI Terms of Reference of the Review Committee for an existing CMOC To conduct the review of an existing CMOC, an independent committee will be formed of at least three members appointed by the JCOMM DMCG. The Committee shall: Elect a chair; Review the annual reports from the CMOC under review. The reports will be used to evaluate if the CMOC is continuing to meet all of the criteria used in the original evaluation. If there have been changes in the accreditation criteria, these will also be used. As necessary, the committee may seek additional information from the CMOC on its activities. The committee may also wish to read the report of the accreditation committee and any previous CMOC reviews. All of these documents should be made available by the CMOC. If required, designate someone to consult and/or undertake a visit of the CMOC in order to (i) inform the CMOC about possible elements to be clarified, and seek clarification, (ii) verify specific functions and capabilities of the candidate CMOC, (iii) negotiate needed changes to the proposal, and (iv) submit a report back to the committee with recommendations within the time period specified in Resolution 2 and its annexes; Prepare a written report of their evaluation that explains the results. In particular, if the proposal is deemed insufficient to pass one or more criteria, the committee shall explain their reasons and may suggest a possible remedy. The committee may suggest enhancements (e.g. new or improved products) to operations of the CMOC though the CMOC is not obligated to implement these. Submit the report to the DMCG and to the authors of the proposal. Upon request, provide copies of the proposal and evaluation report to any JCOMM or IODE member requesting it. The Chair of the Committee shall: Report the results of the evaluation to the DMCG. Membership: IODE Representative JCOMM Representative (WMO side) Other representative(s) as needed APPENDIX 7 Proposed Terms of Reference of IODE Global Data Assembly Centres (IODE GDACs) To IODE Global Data Assembly Centres (IODE GDACs) shall: receive and assemble meteorological and/or oceanographic data (real or delayed-mode) and metadata from the appropriate data streams identify duplicates and if possible resolve by keeping the best copy of dataset; apply data processingquality control procedures according to the international standards amd methods provide feedback to the resource on data quality issues make data accessible through IODE/ODP make discovery metadata available to IODE/ODP forward data and metadata to the appropriate CMOC(s) in agreed format(s) within defined timescales contribute to WMO and IOC Applications by collecting and processing worldwide marine-meteorological and oceanographic data and metadata documented in appropriate WMO and IOC publications report to the IODE and JCOMM Committees on its data management status and activities ( 1 U } ) w 3 q^K8 $hq h ~` CJ OJ PJ QJ ^J aJ $h`3 h ~` CJ OJ PJ QJ ^J aJ $h? h ~` CJ OJ PJ QJ ^J aJ $hTb h ~` CJ OJ PJ QJ ^J aJ $hF. h ~` CJ OJ PJ QJ ^J aJ $hF. h ~` CJ OJ PJ QJ ^J aJ *hYT h ~` 5CJ OJ PJ QJ \^J aJ $hYT h ~` CJ OJ PJ QJ ^J aJ -hYT h ~` 5;CJ OJ PJ QJ \^J aJ )hYT h ~` 5;CJ OJ QJ \^J aJ ) 0 1 V ) w C & $ & F & 6m I %\ *$1$ 7$ a$gd ~` $ & 6m I %\ 7$ a$gd ~` $ & 6m I %\ 7$ a$gd ~` $ S# 1$ a$gd ~`
w w $ & 6m I %\ 7$ a$gd ~` $ & 6m I %\ 7$ a$gd ~` & $ & F & 0` P@ *$1$ 7$ a$gd ~` & $ & F & 6m I %\ *$1$ 7$ a$gd ~` = = D D D D D D E E hL iL jL kL lL uL vL דl]L>3+ h ~` CJ$ aJ$ h, h ~` CJ$ aJ$ h ~` CJ OJ QJ ^J aJ hYT h ~` CJ OJ QJ ^J aJ hYT h ~` PJ ^J mH sH $hYT h ~` CJ OJ PJ QJ ^J aJ 'hYT h ~` ;CJ OJ PJ QJ ^J aJ -hYT h ~` 5;CJ OJ PJ QJ \^J aJ -hYT h ~` 5;CJ OJ PJ QJ \^J aJ *hYT h ~` 5CJ OJ PJ QJ \^J aJ $hYT h ~` CJ OJ PJ QJ ^J aJ *hYT h ~` 5CJ OJ PJ QJ \^J aJ Z> o> ~> @ @ bB KC C C C D PD QD ]D & $ & F & 6m I %\ *$1$ 7$ a$gd ~` & $ & F & 6m I %\ *$1$ 7$ a$gd ~` $ & 6m I %\ 7$ a$gd ~` ]D qD D D D D D D E E E ~ $$ & 6m I %\ 7$ a$gd ~` $ & 6m I %\ 7$ a$gd ~` $ & 6m I %\ 7$ a$gd ~` & $ & F & 6m I %\ *$1$ 7$ a$gd ~` E E G I jJ K CK K K K L L L & $ & F & 6m I %\ *$1$ 7$ a$gd ~` $ & 6m I %\ 7$ a$gd ~` & $ & F & 6m I %\ *$1$ 7$ a$gd ~` L 'L GL iL jL kL wL L L M M 8N pN N N (O O 8P $ & F S# 1$ a$gd ~` $ S# 1$ a$gd ~` $ S# 1$ a$gd ~` $ 7 7 1$ a$gd ~` & $ & F & 6m I %\ *$1$ 7$ a$gd ~` vL wL L L L L L 7P 9P :P ;P
H 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 0 = vL
]D E L 8P
H . . >H 9 *urn:schemas-microsoft-com:office:smarttagsState 9 *urn:schemas-microsoft-com:office:smarttagsplace 2 ^ g " " 9 9 0@ 1@ E E ,F /F >H F Q m } ! % % N. S. 2 2 u3 z3 >H 3 3 3 3 3 3 3 3 3 3 _" t" g# # ) ) Z6 ~6 l8 8 Q< < = = D lD >H >H
N~Vk6
^`OJ QJ ^J 8 8^8`OJ QJ ^J % ^`OJ QJ ^J % ^`OJ QJ ^J p p^p`OJ QJ ^J % ^ `OJ QJ ^J % @ @ ^@ `OJ QJ ^J ^ `OJ QJ ^J % ^`OJ QJ ^J % ^`OJ QJ ^J 8 8^8`OJ QJ ^J % ^`OJ QJ ^J % ^`OJ QJ ^J p p^p`OJ QJ ^J % ^ `OJ QJ ^J % @ @ ^@ `OJ QJ ^J ^ `OJ QJ ^J % ^`OJ QJ ^J % ^`OJ QJ ^J 8 8^8`OJ QJ ^J % ^`OJ QJ ^J % ^`OJ QJ ^J p p^p`OJ QJ ^J % ^ `OJ QJ ^J % @ @ ^@ `OJ QJ ^J ^ `OJ QJ ^J % ^`OJ QJ ^J % ^`OJ QJ ^J 8 8^8`OJ QJ ^J % ^`OJ QJ ^J % ^`OJ QJ ^J p p^p`OJ QJ ^J % ^ `OJ QJ ^J % @ @ ^@ `OJ QJ ^J ^ `OJ QJ ^J % ^`OJ QJ ^J % ^` . 8 8^8` . ^` . ^` . p p^p` . ^ ` . @ @ ^@ ` . ^ ` . ^` . ^` . 8 8^8` . ^` . ^` . p p^p` . ^ ` . @ @ ^@ ` . ^ ` . ^` . ^` . 8 8^8` . ^` . ^` . p p^p` . ^ ` . @ @ ^@ ` . ^ ` . ^` . ^` . 8 8^8` . ^` . ^` . p p^p` . ^ ` . @ @ ^@ ` . ^ ` . ^` . ^` . 8 8^8` . ^` . ^` . p p^p` . ^ ` . @ @ ^@ ` . ^ ` . ^` . h ^`OJ QJ o( h H - h ^`OJ QJ ^J o( h H o h p^p`OJ
QJ
o( h H h @ ^@ `OJ QJ o( h H h ^`OJ QJ ^J o( h H o h ^`OJ
QJ
o( h H h ^`OJ QJ o( h H h ^`OJ QJ ^J o( h H o h P^P`OJ
QJ
o( h H
N~Vk
W W 8 N u m 5 W W 8 N u m 6 W W 8 N u m 8 W W 8 N u m 9 W W 8 N u m 1 4 W W 8 N u m 1 5 W W 8 N u m 1 6 W W 8 N u m 1 7 W W 8 N u m 1 8
ZV* T+ F. 9 >U> %dC #F Z ~` q v \[ ? I m Tb DB "[ [? `3 ? 9 jD kD :H >H @ % % ^e % %