Integrating Data from Both Sides of the Firewall

Organizations are beginning to recognize the need to monitor multiple real-time information sources regularly and automatically. The Web is one of the major sources of data that must be monitored and analyzed by todayï¿1⁄2s enterprises. Additionally, critical information is located in internal records and data; it is stored throughout an organization in databases, email stores, office documents, spreadsheets and other data repositories located behind the company firewall. To be useful, organizations must look at all these data side by side, because analyzing Web data and internal data separately is highly inefficient and will produce misaligned information.

Who needs these data?
Take a fictitious pharmaceutical company (for these purposes weï¿1⁄2ll call it A+Pharma) that is the premier manufacturer of diabetes drug treatments. A highly competitive and regulated industry like the pharmaceutical space must be aware of what is happening externally in the news, in consumer groups, in clinical trials, with regulatory bodies and with their competitors. Much of the data that need to be considered for strategic decision-making and growth are found outside of the firewall. Due to the amount of information and its multiple sources, this information needs to be automatically gathered and seamlessly integrated with internal intellectual property for knowledge, awareness, analysis and action.

The problem
A+Pharmaï¿1⁄2s most popular drug (weï¿1⁄2ll call it A+Drug) is used to regulate blood sugar in type-2 diabetics. A+Drug has proven to be very effective, and A+Pharma is making a large percentage of its overall revenue from the sale of this drug. Unfortunately for A+Pharma, A+Drug is nearing the end of its patent period, meaning that generics, which will be sold at a much lower price point, will soon be available. A+Pharma needs to make a strategic next move or it risks losing a large portion of its revenue. A+Pharma fully recognizes the need to examine all data available, including its own internal intellectual property, potentially competing product information, regulatory body information, adverse events and consumer sentiment and behavior.

A+Pharma must focus on closely monitoring the status of various relevant clinical trials currently under way by competing drug manufacturers to ascertain whether any replacement drugs are close to approval by the FDA. In order for these data to be helpful, they must be integrated into A+Pharmaï¿1⁄2s internal database so they are accessible for analysis. Additionally, the company needs to gather and integrate daily news information and data from regulatory bodies such as the FDA for a complete view of what is going on both inside and outside of the organization.

(Click for larger image)

Along with the information from the Web, A+Pharma also needs to analyze internal information such as its drug pipeline to understand what potential replacements are available and where they are in clinical trials and the approval process. The internal information is vital for analysis and must be considered in conjunction with data gathered from the Web for a 360-degree view of the situation. All this information needs to be brought together in such a way that it can be monitored and analyzed in one location, so that decisions can be made and strategies devised to maintain the companyï¿1⁄2s competitive edge.

How should these data be collected?
Considering how much information needs to be monitored, the data should be collected automatically using data extraction and integration technology. In the past, analysts would be responsible for manually gathering data from various resources like the Web or internal documents, and then copying and pasting this information into internal systems for analysis. One group might be looking at clinical trials statuses, one at competitor information, one monitoring adverse events, etc. Knowing that internal data are also located in disparate repositories throughout the organization, itï¿1⁄2s easy to see that all data, regardless of where they reside, need to be automatically gathered and integrated if they are to be useful and actionable.

The Solution
QL2 Software has developed products and tools that automate data extraction and the process of integration, giving enterprises a 360-degree view of critical information. WebQL, QL2ï¿1⁄2s flagship technology, allows companies around the world to seamlessly integrate data, regardless of where those data reside.

With the use of WebQL, A+Pharma can achieve all of its data collection goals. Dynamic websites can be searched on a continual basis by WebQL for real-time data needs. WebQL is capable of accessing data from any number of sites, even those that require user logins or forms, and thus, can access data from a deeper layer of the Web. WebQL delivers critical information from news sites, clinical trials, adverse events monitoring, competitor websites, etc. directly to the organization and integrates that information into any application or solution required.

Additionally, WebQL can navigate internal networks and data repositories, collect data and integrate them with applications, solutions or other data repositories across the organization. These features and capabilities transform A+Pharma from an organization with many data repositories into one that has all information available in a central, consolidated grouping.

The end benefit of WebQL is the ability to extract data for a complete 360-degree view of not only whatï¿1⁄2s happening in the market place or with the competition, but also with the internal intellectual property which contains critical data about each possible competitive drug. Through processing and managing multiple feeds and by seeing all aspects of other drug lifecycles, alerts, customer acceptance and regulatory filings, A+Pharma can make the most informed decision possible about the future of A+Drug.

Only in recent years have companies recognized the need to automate data collection efforts. Companies are recognizing that antiquated methods such as manual collection arenï¿1⁄2t producing the elevated quality and quantity of data needed to make critical company decisions regarding product strategy. Beyond that, companies are ascertaining that data collection alone isnï¿1⁄2t providing a complete solution. Integrating collected data with the proprietary data that is already housed within organizations has become paramount to developing the enlightened data analysis needed to make any business decision.