A Standards-based Approach to Leveraging Big Data
The business value of information is often limited by the specific dataset being analyzed, because although data characterizations and correlations can reveal patterns, observations and conclusions can be drawn only on that set of data. Consequently, the integration of multiple heterogeneous data sources is the key to unlocking the full potential of Big Data capabilities.
What is NIEM and How Does It Work?
NIEM is not a database or intrusive to existing information technology and systems. It is also not software or a technology stack. Rather, it’s a framework for exchanging information that allows systems to talk to each other even if they use different programming languages or operating systems.
NIEM specifies a set of reusable information components for defining standard information exchange messages, transactions, and documents on a large scale—across multiple communities of interest, mission areas and lines of business. In doing so, NIEM ensures the sender and receiver of information share a common, unambiguous understanding of the meaning of that information. This consistent understanding establishes a fundamental baseline for improved data sharing interoperability.
Benefits of NIEM
NIEM provides a consistent starting point for exchanging data between disparate IT systems, and includes a data model, normative technical specifications, governance, training, software tools, technical support services, and an active community to assist users. Among its many benefits and characteristics, NIEM:
- Provides a federated design supporting the implementation of automated, machine-to-machine information exchange without need for a central controlling function between systems;
- Fully supports external standards (e.g., security markings, data tagging, geospatial, etc.);
- Functions independently of transport methodology;
- Does not require changes to either the producing or consuming data store;
- Employs a Whole-of-Government approach;
- Can rapidly modify the data exchange composition/scale; and
- Is compatible with common programming languages and tools (e.g., Java, C#, Ruby, etc.).
Information Exchange Package Documentation (IEPDs)
The NIEM data model provides reusable data components for building exchange specifications referred to as Information Exchange Package Documentation (IEPD). IEPDs are a collection of artifacts (e.g., schema documents, business rules, mapping) that define and describe the structure and content of a data exchanged between a producer and consumer(s).
IEPDs provide the basis for traditional web-services, micro-services, application program interfaces, other point-to-point exchanges, etc. IEPDs are developed with a repeatable engineering process using widely available software tools, checked for conformance, registered in repositories such as the NIEM IEPD Clearinghouse, and are available for search and discovery, and reuse by NIEM practitioners.
An Information Exchange Package (IEP) is an instance document defined by an IEPD, and IEPs are typically instantiated by some form of computer software (e.g., executable software application, web service, etc.).
NIEM Specifications & Guidance
The underlying technical architecture supporting NIEM is described in a set of normative specifications and guidance that includes among others:
- High-Level Version Architecture (HLVA): Specifies the NIEM version architecture.
- Model Package Description (MPD) Specification: Specifies rules for organizing and packaging MPDs in general and IEPDs specifically.
- NIEM Code Lists Specification: Establishes methods for using code list artifacts with NIEM information exchange specifications.
- NIEM Conformance Specification: Defines general conformance to NIEM
- NIEM Conformance Target Attribute Specification (CTAS): Specifies XML attributes to establish a claim that the document conforms to a set of conformance targets.
- Naming and Design Rules (NDR): Specifies principles and enforceable rules for NIEM-conformant schema documents, instance XML documents and data components.
- Non-Normative Guidance in Using NIEM with JSON: Guidance for using NIEM with JSON-LD specified by RFC4627. Note: A normative NIEM-JSON specification under development and scheduled for release in Dec 2017.
Helpful NIEM Resources
- NIEM technical architecture
- NIEM governance structure and processes
- Communities that currently manage content in the NIEM model
- On-demand, no-cost online training
- NIEM Tools Catalog
- Current NIEM release
- Technical support and resources
For additional information on Big Data refer to the NIST Big Data Information.