Archive for the ‘Data Warehousing’ Category

Shade Under The Cloud

July 20, 2011

More and more companies and consumers are increasingly using use data-intensive applications driven by cloud technology that may have been previously unavailable due to overall cost and complexity.  Now, even mom and pop shops can store data, deploy complex applications using cloud technology.  They are finding that this is often the best way to provide optimal service, reduce failure points, ensure data integrity, reduce operating costs, and create a more easily managed environment.

(more…)

Currency Conversions in SSIS

May 3, 2010

Capturing Real-Time Currency Conversions in SSIS

This article shows how to convert currency using SSIS. A web service task makes an external call to a web service to capture up-to-date exchange rates. The really cool part about this article is that it shows how to configure: web service tasks, recordset destinations and a for-each loop to shred a recordset.

http://www.sqlservercentral.com/articles/Integration+Services+(SSIS)/69768/

BP(K*D) = i

March 25, 2010

Hey readers,  here is an excerpt from another blogger Jeff McQuigg

The full post can be found here.

I think the most interesting part about this post is discerning the difference between BI and DW.  It reminds me of the telescope/microscope analogy:  BI is about making reactive and more importantly proactive business decisions.  While some BI may involve analyzing historical trends, the newer the data, the more relevant it is.

(more…)

Data Quality Part II

February 26, 2010

In Part I of this series, I showed how to create rules around data in a table. This post will expand on the idea of data quality and the need for data quality integration in your systems and applications. Ideally, a good data quality solution will help to reduce the amount of ETL (and associated overhead) required to incorporate data sets between applications, and will help to create data that is both efficient and valuable to your organization. (more…)

Extract Variable Records to the Same File

January 22, 2010

Hello my faithful reader(s).  I have been away for a while but do not despair:  I am back and 2010 promises some really cool posts.  Today’s post centers on extracting variable record types to a single flat file.  This applies to data warehouse type files that have  header, detail & trailer records somewhat similar to those I have seen for terradata and other warehouse applications.  There will also be a future post in this blog that will describe how to load this data type into your warehouse along with some built in intelligence.

(more…)

Data Quality Part – I

December 1, 2009

Since the blog is named data integrity, I figured I would look up what “data integrity” truly meant.  In searching for the definition on google, data integrity came to be defined as ‘data that is complete or whole. All characteristics of the data including business rules, rules for how pieces of data relate, dates, definitions and lineage must be correct for data to be complete’.  With that in mind, I am going to start a number of posts on Data Quality and Data Profiling over the coming weeks. 

(more…)

Comparing tables with TABLEDIFF

October 30, 2009

Recently, I stumbled across a pretty cool utility called tablediff.  This utility is shipped standard with Microsoft SQL Server and is used to compare the data in two tables for non-convergence (dissimilarities).  The contingencies for this tool require that the tables being compared both have a primary key or a similar field.  This utility can be used from the command prompt or in a batch file which means it can be called from SSIS.   (more…)

Performance Tuning in SSIS

October 2, 2009

Tuning SQL Server 2005 Integration Services for EBCDIC Data Conversion

This article talks about tuning the dataflow for 5 million EBCDIC records at 497 bytes per record.  Impressively, the execution time for the dataflow task was cut almost in half by employing a number of tuning practices. 

Read the article at:

http://www.sql-server-performance.com/articles/per/ssis_ebcdic_conversion_p1.aspx