III. Policy Requirements
Agencies management of information resources must begin at the earliest stages of the planning process, well before information is collected or created. Early strategic planning will allow the Federal Government to design systems and develop processes that unlock the full value of the information, and provide a foundation from which agencies can continue to manage information throughout its life cycle.
Agencies shall take the following actions to improve the management of information resources throughout the information’s life cycle and reinforce the government’s presumption in favor of openness:
1. Collect or create information in a way that supports downstream information processing and dissemination activities
Consistent with OMB Circular A-130, agencies must consider, at each stage of the information life cycle, the effects of decisions and actions on other stages of the life cycle. Accordingly, to the extent permitted by law, agencies must design new information collection and creation efforts so that the information collected or created supports downstream interoperability between information systems and dissemination of information to the public, as appropriate, without the need for costly retrofitting. This includes consideration and consultation of key target audiences for the information when determining format, frequency of update, and other information management decisions. Specifically, agencies must incorporate the following requirements into future information collection and creation efforts:
a. Use machine-readable and open formats19
Agencies must use machine-readable and open formats for information as it is collected or created. While information should be collected electronically by default, machine-readable and open formats must be used in conjunction with both electronic and telephone or paper-based information collection efforts. Additionally, in consultation with the best practices found in Project Open Data and to the extent permitted by law, agencies should prioritize the use of open formats that are non-proprietary, publicly available, and that place no restrictions upon their use.
b. Use data standards
Consistent with existing policies relating to Federal agencies’ use of standards20 for information as it is collected or created, agencies must use standards in order to promote data interoperability and openness.
c. Ensure information stewardship through the use of open licenses
Agencies must apply open licenses, in consultation with the best practices found in Project Open Data, to information as it is collected or created so that if data are made public there are no restrictions on copying, publishing, distributing, transmitting, adapting, or otherwise using the information for non-commercial or for commercial purposes. 21 When information is acquired or accessed by an agency through performance of a contract, appropriate existing clauses 22 shall be utilized to meet these objectives while recognizing that contractors may have proprietary interests in such information, and that protection of such information may be necessary to encourage qualified contractors to participate in and apply innovative concepts to government programs.
d. Use common core and extensible metadata
Agencies must describe information using common core metadata, in consultation with the best practices found in Project Open Data, as it is collected and created. Metadata should also include information about origin, linked data, geographic location, time series continuations, data quality, and other relevant indices that reveal relationships between datasets and allow the public to determine the fitness of the data source. Agencies may expand upon the basic common metadata based on standards, specifications, or formats developed within different communities (e.g., financial, health, geospatial, law enforcement). Groups that develop and promulgate these metadata specifications must review them for compliance with the common core metadata standard, specifications, and formats.
2. Build information systems to support interoperability and information accessibility
Through their acquisition and technology management processes, agencies must build or modernize information systems in a way that maximizes interoperability and information accessibility, to the extent practicable and permitted by law. To this end, agencies should leverage existing Federal IT guidance, such as the Common Approach to Federal Enterprise Architecture, 23 when designing information systems. Agencies must exercise forethought when architecting, building, or substantially modifying an information system to facilitate public distribution, where appropriate. In addition, the agency’s CIO must validate that the following minimum requirements have been incorporated into acquisition planning documents and technical design for all new information systems and those preparing for modernization, as appropriate:
The system design must be scalable, flexible, and facilitate extraction of data in multiple formats and for a range of uses as internal and external needs change, including potential uses not accounted for in the original design. In general, this will involve the use of standards and specifications in the system design that promote industry best practices for information sharing, and separation of data from the application layer to maximize data reuse opportunities and incorporation of future application or technology capabilities, in consultation with the best practices found in Project Open Data;
- All data outputs associated with the system must meet the requirements described in part III, sections 1.a-e of this Memorandum and be accounted for in the data inventory described in part III section 3.a; and
- Data schema and dictionaries have been documented and shared with internal partners and the public, as applicable.
3. Strengthen data management and release practices
To ensure that agency data assets are managed and maintained throughout their life cycle, agencies must adopt effective data asset portfolio management approaches. Within six (6) months of the date of this Memorandum, agencies and inter-agency groups must review and, where appropriate, revise existing policies and procedures to strengthen their data management and release practices to ensure consistency with the requirements in this Memorandum, and take the following actions:
a. Create and maintain an enterprise data inventory
Agencies must update their inventory of agency information resources (as required by OMB Circular A-130) 24 to include an enterprise data inventory, if it does not already exist, that accounts for datasets used in the agency’s information systems. The inventory will be built out over time, with the ultimate goal of including all agency datasets, to the extent practicable. The inventory will indicate, as appropriate, if the agency has determined that the individual datasets may be made publicly available (i.e., release is permitted by law, subject to all privacy, confidentiality, security, and other valid requirements) and whether they are currently available to the public. The Senior Agency Official for Records Management should be consulted on integration with the records management process. Agencies should use the Data Reference Model from the Federal Enterprise Architecture 25 to help create and maintain their inventory. Agencies must describe datasets within the inventory using the common core and extensible metadata (see part III, section 1.e).
b. Create and maintain a public data listing
Any datasets in the agency’s enterprise data inventory that can be made publicly available must be listed at www.[agency].gov/data in a human- and machine-readable format that enables automatic aggregation by Data.gov and other services (known as “harvestable files”), to the extent practicable. This should include datasets that can be made publicly available but have not yet been released. This public data listing should also include, to the extent permitted by law and existing terms and conditions, datasets that were produced through agency-funded grants, contracts, and cooperative agreements (excluding any data submitted primarily for the purpose of contract monitoring and administration), and, where feasible, be accompanied by standard citation information, preferably in the form of a persistent identifier. The public data listing will be built out over time, with the ultimate goal of including all agency datasets that can be made publicly available. See Project Open Data for best practices, tools, and schema to implement the public data listing and harvestable files.
c. Create a process to engage with customers to help facilitate and prioritize data release
Agencies must create a process to engage with customers, through their www.[agency].gov/data pages and other necessary means, to solicit help in prioritizing the release of datasets and determining the most usable and appropriate formats for release. 26 Agencies should make data available in multiple formats according to their customer needs. For example, high-volume datasets of interest to developers should be released using bulk downloads as well as Application Programming Interfaces (APIs). In addition, customer engagement efforts should help agencies prioritize efforts to improve the discoverability and usability of datasets that have already been released to the public but are not yet fully “open” (e.g., they are only available in closed, inaccessible formats). See Project Open Data for best practices and tools that can be used to implement customer engagement efforts.
d. Clarify roles and responsibilities for promoting efficient and effective data release practices
Agencies must ensure that roles and responsibilities are clearly designated for the promotion of efficient and effective data release practices across the agency, and that proper authorities have been granted to execute on related responsibilities, including:
Communicating the strategic value of open data to internal stakeholders and the public;
Ensuring that data released to the public are open (as defined in part I), as appropriate, and a point of contact is designated to assist open data use and to respond to complaints about adherence to open data requirements;
Engaging entrepreneurs and innovators in the private and nonprofit sectors to encourage and facilitate the use of agency data to build applications and services;
Working with agency components to scale best practices from bureaus and offices that excel in open data practices across the enterprise;
Working with the agency’s Senior Agency Official for Privacy (SAOP) or other relevant officials to ensure that privacy and confidentiality are fully protected; and
Working with the Chief Information Security Officer (CISO) and mission owners to assess overall organizational risk, based on the impact of releasing potentially sensitive data, and make a risk-based determination.
4. Strengthen measures to ensure that privacy and confidentiality are fully protected and that data are properly secured
Agencies must incorporate privacy analyses into each stage of the information’s life cycle. In particular, agencies must review the information collected or created for valid restrictions to release to determine whether it can be made publicly available, consistent with the Open Government Directive’s presumption in favor of openness, and to the extent permitted by law and subject to privacy, confidentiality pledge, security, trade secret, contractual, or other valid restrictions to release. If the agency determines that information should not be made publicly available on one of these grounds, the agency must document this determination in consultation with its Office of General Counsel or equivalent.
As agencies consider whether or not information may be disclosed, they must also account for the “mosaic effect” of data aggregation. Agencies should note that the mosaic effect demands a risk-based analysis, often utilizing statistical methods whose parameters can change over time, depending on the nature of the information, the availability of other information, and the technology in place that could facilitate the process of identification. Because of the complexity of this analysis and the scope of data involved, agencies may choose to take advantage of entities in the Executive Branch that may have relevant expertise, including the staff of Data.gov. Ultimately, it is the responsibility of each agency to perform the necessary analysis and comply with all applicable laws, regulations, and policies. In some cases, this assessment may affect the amount, type, form, and detail of data released by agencies.
As OMB has noted, “The individual’s right to privacy must be protected in Federal Government information activities involving personal information.” 27 As agencies consider security-related restrictions to release, they should focus on information confidentiality, integrity, and availability as part of the agency’s overall risk management framework. They are required to incorporate the National Institute of Standards and Technology (NIST) Federal Information Processing Standard (FIPS) Publication 199 “Standards for Security Categorization of Federal Information and Information Systems,” which includes guidance and definitions for confidentiality, integrity, and availability.28 Agencies should also consult with the Controlled Unclassified Information (CUI) program to ensure compliance with CUI requirements, 29 the National Strategy for Information Sharing and Safeguarding, 30 and the best practices found in Project Open Data. In addition to complying with the Privacy Act of 1974, the E-Government Act of 2002, FISMA, and CIPSEA, agencies should implement information policies based upon Fair Information Practice Principles and NIST guidance on Security and Privacy Controls for Federal Information Systems and Organizations. 31 For example, agencies must:
Collect or create only that information necessary for the proper performance of agency functions and which has practical utility; 32
Limit the collection or creation of information which identifies individuals to that which is legally authorized and necessary for the proper performance of agency functions; 33
Limit the sharing of information that identifies individuals or contains proprietary information to that which is legally authorized, and impose appropriate conditions on use where a continuing obligation to ensure the confidentiality of the information exists; 34
Ensure that information is protected commensurate with the risk and magnitude of the harm that would result from the loss, misuse, or unauthorized access to or modification of such information; 35 and
Take into account other publicly available information when determining whether particular information should be considered PII (as defined in part I of this Memorandum).
5. Incorporate new interoperability and openness requirements into core agency processes
Consistent with 44 U.S.C. 3506 (b)(2), agencies must develop and maintain an Information Resource Management (IRM) Strategic Plan. IRM Strategic Plans should align with the agency’s Strategic Plan (as required by OMB Circular A-11), 36 support the attainment of agency priority goals as mandated by the Government Performance and Results Modernization Act of 2010, 37 provide a description of how IRM activities help accomplish agency missions, and ensure that IRM decisions are integrated with organizational planning, budget, procurement, financial management, human resources management, and program decisions. As part of the annual PortfolioStat process, 38 agencies must update their IRM Strategic Plans to describe how they are meeting new and existing information life cycle management requirements. Specifically, agencies must describe how they have institutionalized and operationalized the interoperability and openness requirements in this Memorandum into their core processes across all applicable agency programs and stakeholders.
- 19 The requirements of this subsection build upon existing requirements in OMB Statistical Policy Directives No. 1 and No. 2, available at http://www.whitehouse.gov/sites/default/files/omb/assets/omb/inforeg/directive1.pdf and http://www.whitehouse.gov/sites/default/files/omb/assets/omb/inforeg/directive2.pdf. ↩
- 20 See OMB Circular A-119, available at http://www.whitehouse.gov/omb/circulars_a119, and OMB Memorandum M-12-08, Principles for Federal Engagement in Standards Activities to Address National Priorities (Jan 27, 2012), available at http://www.whitehouse.gov/sites/default/files/omb/memoranda/2012/m-12-08.pdf ↩
- 21 If a data user augments or alters original information that is attributed to the Federal Government, the user is responsible for making clear the source and nature of that augmentation. ↩
- 22 See Federal Acquisition Regulation (FAR) Subpart 27.4—Rights in Data and Copyrights, available at https://acquisition.gov/far/current/html/Subpart%2027_4.html ↩
- 23 Office of Management and Budget, Common Approach to Federal Enterprise Architecture, available at http://www.whitehouse.gov/sites/default/files/omb/assets/egov_docs/common_approach_to_federal_ea.pdf ↩
- 24 See OMB Circular A-130, section 8(b)(2)(a). ↩
- 25 Office of Management and Budget, Federal Enterprise Architecture (FEA) Reference Models, available at http://www.whitehouse.gov/omb/e-gov/fea ↩
- 26 OMB Statistical Policy Directives 3 and 4 describe the schedule and manner in which data produced by the principal statistical agencies will be released. Statistical Policy Directive No. 4: Release and Dissemination of Statistical Products Produced by Federal Statistical Agencies, available at http://www.whitehouse.gov/sites/default/files/omb/assets/omb/fedreg/2008/030708_directive-4.pdf; Statistical Policy Directive 3: Compilation, Release, and Evaluation of Principal Federal Economic Indicators, available at http://www.whitehouse.gov/sites/default/files/omb/assets/omb/inforeg/statpolicy/dir_3_fr_09251985.pdf ↩
- 27 See OMB Circular A-130, available at http://www.whitehouse.gov/omb/Circulars_a130_a130trans4/ ↩
- 28 NIST FIPS Publication 199 "Standards for Security Categorization of Federal Information and Information Systems", available at http://csrc.nist.gov/publications/fips/fips199/FIPS-PUB-199-final.pdf ↩
- 29 Executive Order 13556, Controlled Unclassified Information, available at http://www.whitehouse.gov/the-press-office/2010/11/04/executive-order-13556-controlled-unclassified-information. ↩
- 30 The White House, National Strategy for Information Sharing and Safeguarding (December 2011), available at http://www.whitehouse.gov/the-press-office/2012/12/19/national-strategy-information-sharing-and-safeguarding ↩
- 31 See NIST Special Publication 800-53 "Security and Privacy Controls for Federal Information Systems and Organizations", available at http://csrc.nist.gov/publications/drafts/800-53-rev4/sp800-53-rev4-ipd.pdf ↩
- 32 See OMB Circular A-130, section 8(a)(2). ↩
- 33 See OMB Circular A-130, section 8(a)(9)(b). ↩
- 34 See OMB Circular A-130, section 8(a)(9)(c). ↩
- 35 See OMB Circular A-130, section 8(a)(9)(a). ↩
- 36 OMB Circular A-11, available at http://www.whitehouse.gov/omb/circulars_a11_current_year_a11_toc ↩
- 37 Pub. L. No. 111-352 (2011) (codified as 31 USC § 1120 note). ↩
- 38 In March 2012 OMB established PortfolioStat accountability sessions, engaging directly with agency leadership to assess the maturity and effectiveness of current IT management practices and address management opportunities and challenges. For FY13 OMB PortfolioStat guidance, see OMB Memorandum M-13-09, Fiscal Year 2013 PortfolioStat Guidance: Strengthening Federal IT Portfolio Management (Mar. 27, 2013), available at http://www.whitehouse.gov/sites/default/files/omb/memoranda/2013/m-13-09.pdf. ↩