Blog
How a New Generation of Data Warehouse Technology Keeps You from Drowning in the Data Lake
As the healthcare industry contends with greater volume, variety and velocity of data, there has been some frustration with traditional data warehouse solutions.
Francesca D'Angelo, Dir, Information Management Commercial Offering
Jul 02, 2020

As the healthcare industry contends with greater volume, variety and velocity of data, there has been some frustration with traditional data warehouse solutions. The rigidity of their structures can slow down the process of achieving new insights. As new data types emerge, it’s especially challenging with data sources such as unstructured social media, or with the impacts of new privacy laws that alter the regulatory landscape and force changes to compliance reporting.

In this context, the efforts and costs of data preparation have in many cases become a serious burden. By one estimate, organizations are spending over $450 billion on preparation alone. As the volume of data continues to grow, so do concerns about long term sustainability. This paradigm has led to new storage architectures, including the “data lake.”

  • Sixty percent of IT professionals spend half or more of their time at work on data quality assurance, cleanup or preparation.
  • Fifty-nine percent of IT professionals and data analysts believe that the majority of the data analysts in their organization are dependent on IT resources to prepare or access data.
  • Eighty-two percent of analysts believe that they would be able to drive increased value from their analysis projects with a decreased dependency on IT.1

Over the last few years, some saw data lake technology as a solution to the supposedly inherent limitations of data warehouses. By storing information in a central repository without preconfigured models, data lakes would save up-front costs associated with preparing it for use in analytics and business intelligence tools. This seemed promising, in theory. Unfortunately, many companies that invested in data lake technology quickly discovered that their lakes turned into little more than swamps, where data accumulated but remained unused.

Despite these realities, the conversation around data lakes rightly brought the spotlight on some traditional data warehouse shortcomings, specifically in the areas of design and implementation. To keep pace with the rapid changes occurring in the healthcare industry – and to open new opportunities for how data can be analyzed – flexible and scalable models must become the standard.

A new generation of data warehouse solutions addresses these concerns by delivering on the need for more flexibility without sacrificing the valuable aspects of modeling in the process. This new class of data warehouse combines prebuilt data models with a flexible architecture that permits rapid reconfiguration and extensibility as needs increase and analytics tools become more sophisticated. This combination both relieves the dependency on internal IT teams within the organization and increases operational agility.

Compared to hardwired data models that are inherently inflexible and unable to integrate shifts in business climate or accommodate new sources, these new data warehouse solutions offer configurable models that capture promotional effectiveness, regional formulation preferences or local compliance requirements and can accommodate increasing volume and diversity. Purpose-built models reduce preparation costs by organizing information that is typically fragmented across organizational silos for analysis or business intelligence reporting, while also being sufficiently flexible to comply with changing needs and requirements.

The benefits of these ultra-flexible data warehouse solutions include cost savings during the data preparation and maintenance phases. They also provide a higher level of confidence in the data fueling business-critical decisions throughout the enterprise. This provides an enhanced ability to spot trends and proactively adapt to changes in the industry and the populations being served. All of this translates into the ability to make the right decisions and improve health outcomes – which is why we’re all in this business.

1. A.R. Guess, “Global Organizations Wasting Billions of Dollars on Data Preparation,” DATAVERSITY website, May 21, 2018, http://www.dataversity.net/global-organizations-wasting-billions-dollars-data-preparation/.

Related solutions

Contact Us