Leading Boston Hospital Cuts Query Time by 75 Percent and Improves Access to Big Data
August 9, 2014
Beth Israel Deaconess Medical Center (BIDMC), a leading teaching and healthcare organization, wanted to improve performance for mission-critical applications but struggled with a growing volume of data. To speed queries and gain better insight, BIDMC is implementing Microsoft SQL Server 2014 Enterprise and is looking at ways to extend its on-premises infrastructure with Microsoft Azure HDInsight and Microsoft Power BI for Office 365. It has boosted performance significantly with In-Memory columnstore technology, cutting query time from 45 to 10 seconds. BIDMC is also improving access to historical data and can query decades of information on-demand with HDInsight. Workflow is improving as well: Business users can work directly with large data sets without IT assistance, and the IT team can quickly implement new features without rewriting applications.
SituationLocated in Boston, Massachusetts, Beth Israel Deaconess Medical Center (BIDMC) is a teaching hospital for Harvard Medical School and a leader in biomedical research and patient care. Regularly cited as a “best hospital” in multiple specialties by U.S. News & World Report, BIDMC is also known for its innovative use of information technology in healthcare.
The BIDMC IT team relies on Microsoft technology for approximately 200 application databases, including mission-critical clinical software and a security system. BIDMC sees more than 275,000 patients each year, and as the volume of health records and other data grew, the IT team became concerned about system performance. “As a healthcare organization, performance is a key requirement,” says Don Wood, Manager of Database Administration at Beth Israel Deaconess Medical Center. “We manage a lot of patient care systems, from obstetrics to the intensive care units to the emergency departments. As the volume of data expanded and performance slowed, we looked for ways to improve our services.”
BIDMC maintained data warehouses that included multiple data sources and hundreds of millions of table rows. It could take up to 45 seconds to run a query, a frustrating delay for both the BIDMC IT team and end users. “For example, if we’re facing an immediate threat, and it takes the better part of a minute for security staff to run a query, it’s not acceptable,” says Wood. “And by the time they get results back, the data might no longer be relevant, so they could have to run the query over and over again.”
Database administrators worked continuously to optimize performance, but faced a tradeoff. For instance, to meet service level agreements with its security event monitoring system, the team had to limit the stored records to information collected in a 30-day period. As a result, it could be difficult to look for recurrent patterns over longer periods of time, such as a history of failed logon attempts from a specific user or IP address. Data warehouses used for clinical systems faced similar constraints.
BIDMC wanted to speed query performance, and it also sought better access to historical data to improve predictive analytics and other business intelligence (BI) tasks. In addition, the IT team hoped to simplify management tasks, including reducing the time spent aggregating data and building reports for end users.
SolutionIn February 2013, BIDMC decided to become an early adopter of Microsoft SQL Server 2014 Enterprise software, and to try out new cloud-based offerings such as Microsoft Azure HDInsight and Microsoft Power BI for Office 365. HDInsight is an Apache Hadoop implementation that runs in the cloud, and Power BI provides a set of online analytics and reporting tools.
BIDMC believed that SQL Server 2014 would offer multiple benefits, including enhanced query processing without any applications changes. The IT team was particularly interested in the in-memory columnstore feature, which improves transactional performance and data compression. For the first phase of database project, the team created a data warehouse based on SQL Server 2014 for the system that it uses to monitor security events.
The solution, which went live in February 2014, works with Active Directory Domain Services to collect data, including logon attempts and IP addresses, throughout the hospital network. Security and help-desk staff use the system daily to look for unsuccessful logons or patterns that could indicate an attack. Instead of relying on the BIDMC IT team to aggregate the information as it had in the past, security and help-desk employees can work directly with the large data sets by using Power Query for Excel. The IT team also created an interactive report with dashboards that returns results within seconds.
BIDMC plans to take advantage of built-in data compression technologies and increase the retention period in the database from 30 to 90 days before archiving the information.
The hospital will migrate its in-house- designed clinical applications from previous versions of SQL Server to SQL Server 2014, and expects to complete the project in 2015. BIDMC is also implementing Hyper-V virtualization technology with SQL Server 2014 to ensure high levels of scalability.
In addition, BIDMC is performing a proof of concept with HDInsight and SQL Server 2012. The POC stores large amounts of unstructured log data in the HDInsight Blob Storage, processes it with HDInsight, and then uses on-premises BI tools to analyze the information. In a separate project, BIDMC business users are trying out Power BI tools such as Power Q&A, a natural- language query feature that automatically generates reports based on questions asked.
BenefitsWith Microsoft technology such as SQL Server 2014 and HDInsight, Beth Israel Deaconess Medical Center has boosted query performance, improved access to data, and eased IT management.
Cuts Query Time from 45 to 10 Seconds
BIDMC is significantly improving performance with SQL Server 2014, cutting query execution time by 75 percent in some cases. “Some of our queries have a 45-second execution time,” notes Wood. “With SQL Server 2014, that dropped to 10 seconds. That’s a huge improvement.”
BIDMC anticipates that it can maintain high levels of performance even when it triples the volume of historical data in its warehouse. As a result, the hospital can gain better insight faster, helping it meet its service level agreements and keep the hospital running smoothly and securely. “The volume of data always keeps growing,” says Wood. “With SQL Server 2014, we can improve our availability and performance, and ultimately provide a better data service for patient care.”
Improves Access to Big DataThe hospital is looking at new ways to handle Big Data by extending its on-premises infrastructure with cloud services. “What I’ve found with HDInsight is that it removes a lot of administrative headaches,” says Wood. “It’s a straightforward and easy process to scale out and add new resources. HDInsight can be used for storage and provides an end-to-end solution with the built-in BI stack in SQL Server 2012.”
The solution can also be used to meet stringent requirements for data retention while keeping the information available for analysis. “It’s part of our audit requirements that we keep data for seven years, and some information has to be retained for as long as 30 years,” says Wood. “With HDInsight, we can store more data and query it as needed.”
BIDMC is exploring possibilities for improving access to data with online BI tools as well. “Our users love Power BI,” says Wood. “They like the interactive, visual aspect, especially Power Q&A—a natural language query feature—which lets you type in a friendly question and then builds a report for you.”
Simplifies IT Management
The new Microsoft technologies are also easing management for the BIDMC IT team. In the past, the team spent hours pulling together data and producing reports for end users. Now, hospital staff can query information directly without waiting for IT assistance. “Aggregating hundreds of millions of records was a painful process and it took forever,” says Wood. “But with SQL Server 2014 and features like the in-memory columnstore index and Power Query for Excel, we can eliminate that step. Users can query the raw data themselves.”
The IT team can also easily implement new capabilities without rewriting applications. “Making changes is a seamless process for our end users, who can continue work without interruption,” says Wood. “We can take advantage of built-in features to enhance the back end without spending weeks or even months rebuilding an application. So between the performance boost and the minimal changes required to implement new features, SQL Server 2014 is a big gain for us.”
Unlock insights on any dataMicrosoft business intelligence (BI) solutions simplify access to virtually any type of data, whether it resides in the business or the cloud. Powered by Microsoft SQL Server, and built into familiar programs such as Microsoft Excel, BI tools speed insight into data from multiple sources, including business applications, blogs, and sensors.
For more information about unlocking insights on any data, go to:
For More InformationFor more information about Microsoft products and services, call the Microsoft Sales Information Center at(800) 426-9400. In Canada, call the Microsoft Canada Information Centre at (877) 568-2495. Customers in the United States and Canada who are deaf or hard-of-hearing can reach Microsoft text telephone (TTY/TDD) services at (800) 892-5234. Outside the 50 United States and Canada, please contact your local Microsoft subsidiary. To access information using the World Wide Web, go to:
This case study is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY.