Collecting Oceans Of Data – Problem Or Opportunity?

Sensor, packet, video and signal data is not only all-digital and collected around the clock but it is also scaling up in size and resolution. The resulting massive surge in storage required to save it all is either a problem for your organization – or a major opportunity.

Plotting geophysical trends, tracking industrial changes or monitoring surveillance imagery requires both broad area spectral analysis and high resolution digital images. Ideally, you can combine multiple inputs for richer analysis to help make the most informed decision as close to real-time as you can get.

Better imagery and capture techniques from ever-higher resolution platforms available around the clock mean that your raw data ingest is massive. Given competitive pressures, your need to collect, route and make sense of that data quickly is sharper than ever. Throwing data away that you can’t handle is not an option, but neither is pumping data into an archive with no way to retrieve it and manage it intelligently.

Handling the influx of data requires a scalable storage platform that allows multiple
computing environments to share satellite feeds or other collected data. Most organizations use multiple applications running on different operating systems for different purposes. For example, a Linux-based system might be used to provide high-performance computing horsepower for ingesting data, while a Windows-based system might enable another team to visualize and produce a final product. And since a single data set is often used in multiple workflows to generate varying types of images, simultaneous sharing among systems is critical.

High-performance shared file systems can serve as the glue to bind ingest, processing, and distribution systems together. A shared data pool can allow direct, Fibre Channel speed data access without the scalability and performance bottlenecks found in most NAS solutions. The architecture also provides the flexibility to add additional systems as application requirements and customer demands evolve. In some instances, the streamlining of production workflows might be sufficient, but in the majority of cases, creating imagery products might only be half of the goal.

The Need for Content Retention
Because it is essential to have access to the most complete historical data record possible, preservation of all geospatial data has been a critical requirement. A single snapshot in time is not adequate to facilitate temporal and spatial comparisons that enable accurate analysis of change and its impact over time. If a complete record is not maintained, extremely valuable content that is essential to many applications may be lost. Snapshots need to be retained indefinitely because you never know when that one archived image may be extremely critical for a project.

In industries where trend analysis is increasingly important, companies involved in managing geospatial imagery also need to keep their data sets stored and readily accessible based on evolving data analysis demands. This could apply to changes in plankton levels in our oceans, changing global environmental patterns that could impact the health of food supplies, or activities in a foreign nation that could impact our national security. This would also apply for natural resources where petabytes of location-referenced seismic data have been collected from around the world that need to be preserved for the petroleum industry for current and future exploration and extraction activities. Spatial as well as temporal comparisons are needed to answer important inquiries. And with the increasing resolution of imagery and complexity of the various types of geospatial data, retention and data management becomes extremely critical.

Automated Data Movement
Different data sets are treated differently depending on the project so companies need the ability to tweak movement based on specific project requirements. A data management platform with a rich policy engine could be tuned to handle multiple data sets in unique, project specific manners. A robust policy engine also makes it possible to build a second, long term storage repository. Now as projects are completed, the raw satellite feeds, refined imagery product, and browse thumbnails can be collated and transferred to a long term storage pool. Data can live on high-speed disk for short term repurposing needs and can then transparently move to long-term storage, such as a tape library or disk-based object storage, for faster, more efficient global access to historical data. Such a tiered storage strategy can cost much less than an all disk solution and provides standardization across all projects.

Simplified Data Access
When data analysis must be made quickly, for example in the case of disaster relief support where maps are used to assist relief workers to visualize where specific damages occurred and determine quickly how to route resources, time can’t be wasted when accessing a file. It is critical to find a solution which can mask data location regardless of if it has moved to a tertiary storage device. Within long term storage repositories, all data should look like it lives on the primary storage. This eliminates the need to reconfigure applications to look for data in different locations, causing upheaval to business processes.

A high-performance shared storage platform, such as Quantum StorNext, can ingest and process an ever growing amount of data in a timely fashion and have the ability to store data in a simplified, standardized long term storage repository. With a modern architecture, companies can scale faster and evolve with the next generation of visualization and interpretation techniques. There is a need to have the ability to produce quality imagery and provide trend data that enables better planning, smart analysis and quick action on evolving geophysical and geopolitical conditions. This kind of strategic implementation may not always be able to anticipate demand, but one can now safely plan for it – and turn what is a problem for other organizations into your major opportunity.

Tags: Quantum, Skip Levens

Category: Quantum

This entry was posted on Monday, February 24th, 2014 at 3:13 pm. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.