ETL Lifecycle – An Interesting Thought

During this ETL Framework session, Larry Barnes (the person with whom Donald Farmer is co-presenting) raised an interesting point:

ETL processes tend to stick around forever, because people are horrified to
touch them.

He used the image of a Perl script that runs on an Oracle server pulling data from a Sybase database and loads it into Informix (can’t you just picture that?) but I have no trouble visualizing the same thing happening with SSIS as well.

This may not be technical or of interest to anyone but me, but it strikes me as being profound in some way. I tend to be very aggressive in my commenting, annotation and documentation, but this really drives home how important this is for SSIS. In addition, this really demonstrates the importance of having a well-documented and understood process for making changes, moving from dev to test to prod just like with traditional application code, because only through this process will people truly be comfortable making changes to such business-critical code.

Advertisements

About ssimagine

My name is Matthew Roche, and I am a Senior Program Manager with the SQL Server product group at Microsoft. I work on Master Data Services and Data Quality Services, and have previously worked on SQL Server Integration Services. Although I work for Microsoft and will be posting on technical topics, I want to stress that this is a personal blog, and any opinions posted here are mine and mine alone. I built my career around SQL Server and Microsoft technologies for well over a decade before I joined Microsoft as an employee, and I plan on using this blog to share my personal experience and opinions. They may well be shaped by my experience on the SQL Server team, but they’re still mine, and not that of Microsoft, disclaimer, disclaimer, etc., etc..
This entry was posted in Conference, Quote, SSIS. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s