Needed: Automated Database Integration

 It’s long been a truism that data is one of the key assets, if not the key asset, for an enterprise. Our recent conversations with customers are putting a new spin on this insight. They are telling us why database integration and automation are increasingly important to them to not only reduce costs, but to increase agility. Integration Requirements By integration we mean the ability to quickly and easily combine data from multiple source relational databases into one, or more, central databases while keeping the original databases intact. Such integration allows customers and suppliers to share production and parts information without the expense of implementing EDI. Field auditors at a federal agency are using our own DataPortal to easily share relational databases of audit records to give managers a cross-geography view of agency performance. An online retailer could use such integration to combine information from multiple suppliers and distributors into a single database so web customers can get an up-to-date view of stock on hand and ship dates. Much of this information is stored in relational databases. Our customers are telling us they need to integrate these databases without stripping out the table structures and table references that allow ad-hoc queries. They also need to retain the complex data types (anything from engineering change orders to marketing images) they store in relational platforms. Customers also need to be able to identify which database fields originated in which source database. Such tracking by the source of information is important for anything from identifying which supplier has the lowest prices or most inventory to identifying the source of cost or sales issues within multiple business units. Customers also tell us they need to perform this integration easily and quickly. Ideally, they’d like to see it automated. This not only reduces data management costs, but helps an organization respond with greater agility to changing business needs. For example, the more quickly a major aerospace manufacturer can see database information from new suppliers, the more quickly it can fill its needs for critical parts. The more quickly a CFO can see integrated cost and sales databases from various business units, the more quickly he can see and fix problems.  Our Solution Traditionally, database integration has required days, weeks or months of work by skilled database administrators using database management tools. These days, that is unfeasible both from a cost and a time perspective....

Read More

Database Transfer: Why You Need It, How To Do It

Database Transfer: Why You Need It, How To Do It

Among all the confusing terms we use when talking about information, database transfer is one of the least understood. But regardless of which industry you’re in, you’re probably doing database transfer every day. And you’re likely spending way more time, money and effort on it than you need to. Database transfer means moving a database from one physical, or logical, location to another for any reason. It can cover a wide variety of use cases, which we’ll describe below. Unlike database sharing, database transfer doesn’t imply that different parties are sending and receiving the data. But like database sharing, the ease and speed of how you transfer data is critical to saving money and focusing on your business, while giving your employees, customers and business partners the information they need. Let’s look at some of the use cases that fall under the heading of data transfer, and at the common technical requirements they share. Migration: Moving data from one location to another, without leaving a copy at the source. Replication: Creating multiple copies of the same database and reconciling differences among them. Backup and restore: Creating a second copy of a database for use in disaster recovery or business continuity, and copying it back to a production system to replace a lost primary copy. Supply chain: Making production data available to suppliers, customers or multiple internal facilities to coordinate production or shipments or to compare pricing. Government data sharing: Making government economic, environmental, scientific, legal or other information easily available to the public. Integrate and Automate Integration is the combination of data from multiple internal or external sources. It allows users to create, view and analyze data in their own database platforms and then combine their data with that of others for a broader view of a problem or a process. Such fast, easy and cost-effective information transfer is a critical requirement for a growing number of public and private information users. One federal agency, for example, resorted to mailing thumb drives and cutting and pasting data in spreadsheets to integrate audit data from regional facilities. A major aerospace manufacturer sought our DataPortal software to share detailed parts requirements with potential suppliers. In developing countries, retailers and manufacturers could transfer data without the expense and effort of implementing legacy technology such as EDI. What’s common across all these use cases? That you don’t want anyone from your employees to your...

Read More

Data Sharing… Also Known as Data Transfer, Data Migration, Data Replication, etc.

Data Sharing… Also Known as Data Transfer, Data Migration, Data Replication, etc.

Like everyone else in the tech industry, we in the database field can get a little sloppy when describing the various actions we take with our data stores. We use words like “migrate,” “copy,” “move” and “share” to mean many of the same things…except when we don’t. We at DataPortal run into this problem when we describe our solution as providing “database sharing.” Audiences sometimes look at us as if we were suggesting they share, oh, their ATM passcodes or their car keys. Being specific is important because, as our chart below shows, using the wrong term could lead you to spend more time, effort, and money than you need to. Here’s what, in our view, the various forms of “sharing” do and don’t mean, and where Data Portal does and doesn’t fit the bill. Remember, as you compare this chart to others you may have seen, when we say we can “share” databases we mean even the largest relational databases, in their full relational form, without adapting the databases, using custom protocols or reworking security architectures. Buzzword What It Means Can DataPortal™ Do It? Transfer Change the physical location of a database so only one copy of exists in a single place. Yes Copy Make an exact replica of a database so a copy or copies of it exist in multiple locations. Yes Migrate Copies a database from one database server platform (e.g. vendor, version) to another Yes Edit Allow one or more users or applications parties to view and make changes to the database. Yes Update Allow one or more users or applications to replace existing data with newer data. Yes Append Add additional columns and/or rows to an existing database. Yes Move Change the location Yes Replicate Make changes in multiple copies of a database to reconcile changes among them. Yes* Share Allow multiple users and/or applications to receive their own local copy of a database on their preferred database server platform. Yes *Even better, database updating can be automated so that all child databases are updated whenever the parent source database changes. You see we snuck “share” back into our list at the end, but for a good reason.  To us, database sharing simply means giving users appropriate access to view and change information that is stored in the form of a relational database, which is one of the most direct and efficient ways that...

Read More

Sharing Government Data in a Way it Can Be Useful for Everyone

Sharing Government Data in a Way it Can Be Useful for Everyone

There’s no Statue of Liberty (yet) welcoming government data to the Home of the Free and the Land of the Brave. But if there was, this might be the inscription on it. The rise of the Internet and the demand for more open government has resulted in initiatives such as the U.S. government’s Data.gov database sharing site. To cite just a few examples, with this data: Businesses can use Census data on housing, income and education to better plan everything from store locations to hiring, sales and production. Consumers can track the status of product recalls, government rankings of health care providers or crime rates when looking for housing. Citizen or advocacy groups can track the status of legislation, government contracts and political contributions. Investors, planners, policymakers and interest groups can use economic information such as gross national product to measure economic growth, and better plan government and public investment and economic policies. To meet the rising demand for data and government “transparency,” as of 2012 at least forty-three governments and international organizations worldwide have made more than one million data sets available, in areas ranging from education to health, energy and commerce. Not Just Data: Insights But pre-chosen subsets of data from a government agency are just snippets of reality. They may not contain all the data a user may need or present it in an easily usable format. These partial data sets in unusable formats make it difficult, if not impossible, to combine different forms of data in creative ways. It is such “mashups” that unleash insights from data, allowing users to answer new questions in new and unpredictable ways. Using “mashups,” for example: Citizen advocates can overlap emissions and weather data on maps to identify sources of pollution for remedial action. The unemployed can use the Employment Market Explorer app, a Google Maps mashup, to compare unemployment rates among communities. Home buyers can view a database of crime by type before choosing a neighborhood in which to     look for a home. Parents in Chicago can assess their child’s chances of getting into a selective public school based on their address. Consumers can check the quality ratings of care facilities before choosing a provider for an aging parent. Health care providers, vaccine producers or payers can track disease outbreaks in real time. Economists can use financial databases to build sophisticated models to forecast market prices and employment levels....

Read More

Six EDI Hassles You Never Have to Face Again

Six EDI Hassles You Never Have to Face Again

By Howard W. Sabrin, executive vice president. I recently got off a one hour call with an EDI (electronic data interchange) veteran, discussing his experiences with the venerable data sharing format. EDI has been around since the ‘60s, and for some purposes it’s absolutely the way to go. But hearing the questions this EDI user asked made me realize how much extra work EDI customers are doing, especially if all they need to do is share relational databases. Folks who’ve been using EDI for years may not realize how much of the work that costs extra time and dollars in EDI is handled automatically by relational databases. Here are six quick examples of EDI hassles you can avoid, without damage to your business, through direct database sharing. Converting data to a standardized EDI form on the sending end, and converting it back to a form the recipient’s systems can use. If you’re sharing relational databases, all both sides need to do is agree on a common database design (e.g. tables, columns, data types). Since all work is done within the database, there’s no need to get network, security, firewall, or software development staff involved. If any data must be pre- or post-processed, it can be done using standard database tools by your database staff. Avoiding duplicate entries and other data integrity issues. Database transaction processing capabilities prevents data from being written unless it can be written without error and repeat data writes until they are successful. Each data entry is tagged with a unique identifier set, preventing the same data from being entered repeatedly. No further coding or other work with specific EDI formats is required. Creating an audit trail to assure no loss or corruption of data. Using familiar database queries, the source and destination data can easily be compared to the last byte to assure integrity. Unlike EDI systems where transactions must be tracked through multiple components, the results of these simple queries serve as your audit trail. Integration with ERP and CRM systems. Since many of these systems are built on relational databases, business data from one partner’s database platform can be transferred directly and securely over the Web to another partner’s database. No integration or reformatting of data to match specific EDI formats work is required. Agreeing on, and implementing, common security protocols. DataPortal uses secure Web protocols (e.g. HTTPS) and has layers of password...

Read More

Getting Big Data Where It Needs To Go

Getting Big Data Where It Needs To Go

Information is a critical, if not the, critical business asset in a fast-changing world. The more information an organization has, the better it can make decisions about which products and services to offer, how to price and distribute them, where it can reduce costs and where it can find new customers. Big Data is the process of creating insight from the increasing masses of raw data every organization generates.  Most Big Data is made up of loosely structured or unstructured data such as text, photos, genomic or other scientific data. But some of it also originates in structured relational databases. In addition, many of the results of a Big Data analysis are distributed in the form of a relational database. Whenever Big Data relies on relational data as either a source of information or a way to distribute Big Data insights, DataPortal can share those relational databases quickly, easily, cost-effectively and securely. “Point and Click” Database Sharing DataPortal is a patented tool for moving, storing, and sharing database data across platforms and over the Web. It was designed to be used by people at all technical levels. It requires no coding, entering command lines, opening ports or installation of software. Using DataPortal, each database is published to a DataPortal server. From there it is available over the internal LAN or the Internet to any authorized DataPortal client. This server-to-client communication avoids the security risks of exposing the database directly to a WAN or the Internet. Any user can transfer a full relational database or a subset of that database, in relational form, with the push of a button. All they need is a Web browser and authority to share the database. Using DataPortal, subject matter experts can stay focused on their critical, value-added analytic work without wasting time on the mechanics of sending and receiving data. DataPortal for Data Gathering DataPortal is an efficient way to gather structured data from various database sources at a central location for Big Data integration, analysis, conversion and storage. DataPortal complements Big Data frameworks such as Hadoop, which allow users to access massive quantities of data spread over hundreds or even thousands of processors. These frameworks allow queries to be performed using the familiar SQL used in relational databases.  The results of such queries can also take the form of structured data. Sharing the result set returned by Hadoop with end users can be...

Read More