April 28, 2008

"‘To be is to do.' - Immanuel Kant

"‘To do is to be.' - Jean Paul Sartre

"‘Do-be-do-be-do' - Frank Sinatra"

--Kurt Vonnegut, Jr. (Nov 1922 - Apr 2007)

In the news today: the Compress Engine

In 1783 Immanuel Kant wrote, "David Hume woke me up from my dogmatic slumbers," and revolutionized the way humanity thinks about metaphysics. Almost 220 years later, Netezza set out to achieve a similar goal - redefine analytics. When the first NPS® data warehouse appliance was introduced, the market released itself from yet another dogmatic slumber and realized that there is a different, better way to do data warehousing; a way without compromise, a way without limits.

Netezza has helped to reenergize the data warehouse market in creating and leading the data warehouse appliance category.

• "Every time you turn around you see another industry that's facing a tidal wave of data and they need to understand what this data is saying. Many of them have data volumes in this range that they haven't been able to afford to analyze, as much as they'd like to. ... Netezza can deliver that analytic capability, and at a very attractive price." - Richard Winter, Winter Corporation, from Netezza will scale its appliance to petabyte range, InfoWorld (January 2008)

• "This is what Netezza has done in the data warehousing market: it has totally changed the way that we think about data warehousing... So the bottom line is not just that Netezza's entry into the market was a black swan event but that that event has not ceased to unfold." - from Netezza: a black swan by Philip Howard, Bloor Group (October 2007)

• "Appliances are here to stay and are revolutionizing the data warehouse industry." - from Business Analytics Appliances Are Here to Stay, by Dan Veset, IDC (June 2006)

• "The term data warehouse appliance was coined by Netezza, and this vendor has blazed a trail by proving the concept and educating the market." - from Defining the Data Warehouse Appliance, by Philip Russom, TDWI (August 2005)

Since 2002, Netezza has been repeatedly breaking the latency barrier and challenging the boundaries of data analytics. Since our first release, we have been continuously refuting the alleged mutual dependencies that became the building blocks of the industry's dogmatic misconceptions; namely the expensive nature of performance, the necessary complexity of the analytics architecture and the unavoidable limits of scalability. With today's announcement of the Compress Engine, Netezza disproves yet another myth - the inverse relationship between data compression and query performance.

The architectures of traditional data warehouses, steeped in a legacy of serving OLTP applications, were not designed to handle the ever-growing amounts of data combined with larger and more complex user workloads and shrinking data latency requirements that characterize the modern enterprise. Regulatory compliance, electronic commerce and the need to process and analyze all data in a matter of seconds has pushed the capabilities of traditional data warehouse systems to their limits. In reaction to the data capacity pressures, vendors introduced compression; not as an enhancement but as a compromise solution that allows for further data growth at the cost of processing performance.

Traditional compression approaches, used by several of the competing data warehouse vendors, typically result in performance degradation to accomplish the compression effect. Netezza's addition to the FPGA-Accelerated Streaming Technology (FAST) Engines framework - Compress Engine - utilizes its innovative streaming architecture^TM not only to increase the system's storage capacity by 2-4X but actually boost overall streaming query performance by a factor of about 2X (100%). All this is achieved without requiring any tuning or administration, and it is in fact a software-only upgrade that enables Compress Engine on the Netezza appliance.

It's actually really cool technology, obviously something we love to rave about. Late last year, I wrote about FAST Engines in this blog. We'll use that as a starting point and dig a level deeper into how Compress Engine works. I'm sure it will tickle the fancy of the geek in you!

The NPS system employs a patent-pending method for compiling (yes, compiling) columnar data in all the tables of the database as it is being written to disk e.g. during load, insert or update operations. The process converts row-based data into column streams that are independently compiled to replace the original data in the columns with a stream of "instruction sets" for the FPGA. The "instructions" themselves are much smaller in size than the data they replace, resulting in a highly compressed data stream emerging from the process.

While the compression occurs on columnar data because of the inherent compressibility within database columns, the compressed data is reassembled in rows before being written to disk. Row-wise storage of tables avoids the data scan complexity associated with columnar stores and ensures that scanned data can be efficiently parsed and processed without the need to reconstitute it from multiple sources. The compressed data uses disk much more efficiently and increases the data density of NPS systems by 2-4X - in some cases substantially higher - allowing customers to scale their NPS data warehouse systems into the hundreds of terabytes of user data.

But if the NPS system's data compression and scale brought the system's performance to its knees or severely limited performance speedup due to compression (as it does on many of those other systems), it wouldn't be so great, would it? The beauty of the Netezza way of providing data compression is that not only does it have no negative impact on performance, but it actually increases query performance by up to 100%!

As the compressed data is read off the disk, it is passed through the Compress Engine which applies the instructions embedded in the data stream to restore it to its original form. Our compilation algorithm ensures that this decompression process can be performed entirely in silicon, at wire speeds. Each physical block scanned from the disk can mushroom into 2 to 4 or more times its size in memory without incurring any overhead in processing time - i.e. 2 to 3 times more data is scanned in the same amount of time without any increase in system hardware! Our internal benchmark testing reflecting real customer configurations and workloads has shown an overall 2.2X increase in streaming query performance through the use of Compress Engine.

This software-only enhancement, enabled by our unique architecture, is only the beginning. As we continue to develop our platform, we are investigating further enhancements to the Compress Engine or the addition of new FAST engine(s), aimed at directly increasing streaming performance on the NPS system.

Our philosophy and aim is to continue to shake the industry out of its dogmatic slumbers by extending the price/performance advantages of our products; showing that there's a different way to do data warehousing and advanced analytics. One where performance and scalability are neither the result of expense nor complexity, where you can get more performance from compression, where you do have the power to question everything^TM ...

by Jit Saxena, Netezza Chairman and CEO

"If you are a Big Dog and you are not persuaded by data, then in God we trust...but everyone else, bring data." - Jane E. Shaw, retired Chairman and CEO of Aerogen, Inc. and current member of the Intel Board of Directors (quoted from PowerSpeaking Inc.)

More and more companies recognize the power of analytics as part of their competitive strategy. But most solutions only provide a glimpse of what can be achieved. What is the potential impact when performance barriers fall away? In this post, I’d like to explore the possibilities and introduce a few examples of companies leveraging the intelligence in their data in new and unexpected ways. After all, competing is good, but winning is better. In finance, the term arbitrage refers to the ability to find and exploit market disparities (hedging strategies monitoring currency or securities fluctuations being prime examples). Most arbitrage opportunities are very time-sensitive - you have to recognize value in an overlooked stock then swoop in to buy it before others take notice, get the same idea and drive up the price. On Wall Street, an arbitrage virtuoso, able to consistently spot untapped potential that others miss, is worth his or her weight in gold.

Leaping through Tiny Windows

The term Information Arbitrage has many similarities to its finance equivalent, and it’s a good way to think about the impact that analytics can have on a company or even an entire industry. Information arbitrage is about finding game-changing intelligence buried in vast, unappreciated data assets, and exploiting it to leap ahead of the competition. Like a financial investor, the Information Arbitrager takes advantage of an opportunity before the window slams shut (which can be very fast indeed). Companies in certain industries make particularly good arbitrage candidates. These are companies dealing with "Big Data" - tera-scale or even peta-scale databases, and a constant flood of incoming data. Telecommunications, eBusiness, RFID retail applications and online advertising are a few segments that come to mind. Often the operational data is changing very quickly, and key insights are only found at a very granular level. Now suppose this normally takes hours or days, and one company can suddenly do it in minutes, seconds or even sub-seconds. As Netezza customers well know, this kind of intelligence disparity can have dramatic implications, both for that company and its market.

For example, telecommunications is a high-volume, low-margin business. Constant changes in network utilization demand real-time decisions about rating and pricing structures for an operator to stay competitive. By running pricing scenarios against billions of call data records, and by examining individual customers to determine their current calling patterns and preferences, iBasis, a major telco provider, knows exactly what options to offer each customer. In contrast, competitors might only see that customer as part of a larger segment measured at some time in the past, and come up short with their offers and pricing.

There are several challenges to Big Data analytics that make arbitrage opportunities hard to pursue. Predictive modeling, optimization and other analytic applications are much more processor intensive than the SQL queries used in standard business intelligence applications. When complex algorithms and gargantuan databases converge with real-time business demands, something usually has to give.

Many companies find they are unable to fully exploit their growing data holdings, and have to make do with sampling or high-level summaries rather than the complete, granular data they often want to examine. But using partial or high-level data can be dangerous; even the most powerful algorithms can suggest spurious or meaningless conclusions when they are applied to insufficient data. Companies may also lose hours offloading data from the data warehouse to an external cluster of processors to run the analysis. With all these approaches, the result is an incomplete solution that provides just a hint of the possibilities of analytics, because that’s all the current technology is capable of delivering.

Consider the problem of optimization, for example. Optimization solutions play a key role in helping companies target the right customers, make the right offers, determine manufacturing volumes or accurately price products to take full advantage of market conditions while minimizing expenses. Depending on the problem being addressed, an accurate optimization solution needs to account for many variables and constraints such as products, branches, budget, time, contact channels, offer history, market segmentation and privacy preferences, to name a few.

Due to the multiple permutations and combinations among the different elements, even a simplified optimization model limited to only a month of data, a thousand customers and ten different offers results in an astronomical solution search space of 2 to the power of 10,000. Just to put things in perspective, the number of atoms in the observable universe is about 10^81, just a few more variables away.

The "Big Math" at the heart of this kind of analysis pushes most processing technology to its limits and beyond. As the number of variables and restrictions increases linearly, the algorithm amplifies exponentially, often reaching the complexity class NP-Complete. As a result, companies are forced to compromise in the thoroughness of the analysis and/or the response time they are willing to tolerate. Most optimization efforts look at small snapshots of the total data available (for example, only the last month’s data), and make use of a range of techniques such as Linear, Dynamic and Integer Programming, Lagrange Multipliers and Cluster Analysis that reduce the level of complexity in various ways, all in an attempt to reach an actionable result in a realistic timeframe. But even with these approaches, companies are faced with costly infrastructure requirements, incomplete views of their data and lengthy response times resulting in stale data or missed arbitrage opportunities.

But what if you could bypass the existing performance limitations and get crucial intelligence much faster than before? For example, what if a database marketing company could use complex algorithms to get accurate optimization results days before the market could adjust? Or a retail franchise could precisely adjust the prices of thousand of products daily for each of its stores? Or a credit card company could run customer scoring algorithms one hundred times faster than its competitors? Or a financial services firm could run real-time Monte Carlo simulations on terabytes of data to manage risk? What impact could advantages like these have on a business? It’s fair to say the difference would be game-changing, providing a major competitive advantage and the ability to enter new markets previously out of reach.

These capabilities are not just marketing fantasies or future visions - they’re in use today.

Making these Information Arbitrage opportunities possible is precisely what Netezza does. Our streaming analytic appliances are built for running complex mathematical models on huge data sets, with results in a fraction of the time required by other technologies. Sophisticated analytic applications run "on stream" in the data warehouse, against all the records and detail that need to be examined. There’s no need to settle for summary data or aggregations, or ship data to another system for analysis. (We’re also constantly making our appliances better. Our recent doubling of performance is just the latest Netezza breakthrough.)

Through the Netezza Developer Network, we’re helping developers worldwide use the Netezza architecture to create a new generation of analytic applications that were previously impractical, unaffordable or simply impossible. When exploiting an arbitrage opportunity means leveraging Big Data and Big Math, Netezza’s streaming architecture is simply inherently faster and more efficient than other technologies. Of course, our customers already know this - and with appliance simplicity and low purchase price, information arbitrage pays off even more.

The bottom line is: when Big Data meets Big Math, great things become possible for our customers and their businesses, enabling them to:

• Use Information Arbitrage to take advantage of time-sensitive opportunities

• Rapidly run multiple scenarios and sensitivity analyses in near real-time

• Make use of all the available data, all the time while their competitors are still struggling with reduced visibility from sampled or aggregated dataWhen the first Netezza appliances burst on the scene in 2002, their ability to query giant databases with unprecedented speed upset a lot of preconceived notions about the limitations of technology and what companies can do with their data. Advanced analytic applications take processing complexity to a much more challenging level, and once again the capabilities of our appliances are revolutionizing the market and capturing the imagination of our customers.

Jit Saxena

November X, 2007

Issue 14: Supercomputing Conference Brings Life to Reno

"Computing is not about computers anymore. It is about living."

- Nicolas Negroponte

Negroponte's words certainly rang true, in more ways than one, at the recent SC07 conference in Reno. In a city known for little other than its lackluster "old Vegas" character, its proximity to Tahoe and the Reno 911! Comedy Central series (which is actually filmed in southern California), the Supercomputing conference breathed new life into the city from November 12-15, while providing extensive displays of technologies that are changing many facets of life as we know it.

Moments after stepping off the plane, the fluorescent "Welcome to SC07" signs that followed me from the Reno airport all the way to my hotel made it clear that this conference was going to be a pretty big deal. My intuitions were confirmed upon entering the Reno-Sparks convention center, with an impressive array of meeting space, food stations, souvenir "shops" and an exhibit hall encompassing 200+ rows of endless exhibitor booths. The annual Supercomputing conference does a great job of bringing together growing numbers of cutting edge technologies, commercial enterprises, government organizations, national labs and graduate students, year after year. With its extensive technical program consisting of several awards, Birds-of-a-Feather sessions, "[challenges|http://sc07.supercomp.org/?pg=challenges.html]" and more, along with a plethora of industry and research exhibits and sessions, the show continuously delivers something of value to everyone who attends. The result: increased momentum and record-breaking numbers of both attendees and industry exhibitors at SC07. The collaboration of masterminds from all different backgrounds, locations, age groups and organizations produces an amazing fusion of ideas, solutions, partnerships and technologies.

Netezza participated in the conference with booth space in addition to a session in the SC07 exhibitor forum, featuring our very own Justin Lindsey along with John Johnson from Lawrence Livermore National Lab (better know as LLNL). Despite the modest space Netezza occupied with our 10x20' booth in the massive exhibit hall, our presence felt considerably larger as we attracted a continuous stream (no pun intended) of visitors throughout the week, including those who had never heard of Netezza along with those who'd set out on a resolute mission to find our booth.

Our bright purple SPUBox, featuring an animated motorcyclist and "Netezza Speed" written in graffiti, was by far the biggest draw to the booth. It was exciting to see so many passers-by stop for a closer look at the SPUBox, often asking a question or two about the box, how they might get their hands on one, and what exactly the Netezza Developer Network is all about. Several visitors applied to join the NDN right then and there, hoping for a chance to win a SPUBox at the end of Justin and John's exhibitor forum session. All the NDN buzz created quite a "Netezza high," if I do say so myself.

"What Netezza is doing is... going a step further: score the data as it is streamed into the appliance and before it even hits the database... However, it is not just the performance gain that is significant. This initiative means that developers are embedding analytic software into the Netezza Data Warehouse Appliance so that it becomes, in effect, an application appliance."

Philip Howard, Director of Research, Technology, Bloor Research - from his 5th October posting, "The Netezza Developer Network"

Pardon the title's riff on the late-1970s Elvis Costello hit song What's So Funny 'Bout Peace, Love and Understanding, but a recent mini-dustup got me to thinking about providing a bit of insight into why Netezza's approach to "Streaming Analytic™ Appliances" is different from others' entries in the market. It seems the recasting of Netezza's mission in terms of streaming analytics rather than the more-limiting data warehouse appliances, along with the launch of the Netezza Developer Network (NDN), has caused something of a hullabaloo among some of our competitors (refer to recent stories from Teradata/SAS, Greenplum, IBM/SPSS and industry analyst, Curt Monash).

And well it should. While some would seem to declaim Netezza's positioning on the topic as 'nothing more than UDFs', and argue that what matters is supporting them effectively, we must beg to differ (and to differentiate). In short, we feel the Netezza approach to Streaming Analytics opens the door to dramatically change the way data warehouse systems are viewed, used and even deployed.

The positioning of some of our larger, (recently) publicly-traded competitors may suggest that they see themselves not just as expert in the domain of data warehouse systems, but also as experts in the ways of CRM, advanced scoring and analytics, etc. They seem to have bolted on homegrown software packages as extensions of their data warehouse offerings in the market. That may well be the case, but we don't really see how it's possible for one vendor to "corner the market" on innovation - a view that we think is borne out in recent announcements of closer UDF-based partnerships. Still others, more from the new-entrant category, claim that the only thing required is simply to support basic UDF functionality as an extension to the database. We think both ends of that argument are incorrect.

Instead, we at Netezza think it best to "stick to our knitting". Our aim is to provide the high-performance infrastructure along with a technical and community foundation to enable others much more expert than we are to drive the algorithmic and application-level innovation by their ability to exploit the performance of our streaming analytic appliance. To again provide a riff on something from the late-70s and early-80s BASF ad (a campaign that has recently been rekindled in that company's marketing), our vision could be summarized as, "We don't make your advanced applications; we make your advanced applications 'run like rockets'**."

^** "Raw SPU functions are called like any other SQL function... ...and they run like rockets by exploiting the Netezza architecture."

— Justin Lindsey, Chief Technology Officer, Netezza, speaking at the 2007 International Netezza User Conference, 26th September, 2007

<font color="#008000">What Netezza Provides</font>

What Netezza provides in this mix is an extremely high-performance system, particularly well-suited at storage-intensive operations (like data warehousing) and in particular, operations that (like data warehousing & BI) can benefit from a data streaming architecture in which critical reduction of unnecessary data can be accomplished as rapidly as it is read from the storage elements - allowing for greater processing efficiencies. We've written extensively about this before (see Spotlighting FPGAs, parts one, two and three) and won't repeat the arguments here.

Another key lever that Netezza provides by way of the NPS^® appliance is the fact that our intelligent storage elements known as Snippet Processing Units (SPUs) are really each compute nodes. They are capable of running compiled C or Java code, with the added task-by-task "customizability" of an FPGA that can further accelerate performance, operating in an MPP compute grid but with the simplicity of the our appliance approach.

Consider this: if those 100s of SPUs in an NPS appliance could be used to run C code to execute SQL query processing tasks, why couldn't they equally be tasked to perform tasks that go well above and beyond those enabled (encumbered?) by the set-based, structured-data logic of SQL? Where others may use UDF or even UDA functionality in the data warehouse systems to collect up and standardize use of SQL functionality across users, the streaming analytics enabled by Netezza allows users to "draw outside the lines" of SQL.

Another thing Netezza provides for NDN members is a set of some basic building blocks - functions and an algorithmic work area that form the foundation for more advanced work to be produced. In so doing, some of these appear to be greatly in common with the standard fare of "traditional" SQL functional extensions: record-level functions or User-Defined Functions (UDFs) and aggregate-level functions or User-Defined Aggregates (UDAs) are part of the foundation. But some of the other parts go far beyond those definitions allowing for developers to implement functions retaining a sense of state or to cascade multiple complex algorithmic processes to build even more powerful solutions, all making use of the streaming nature of the NPS analytic appliance to push performance even further.

And finally, what Netezza provides is a simple development appliance platform on which NDN members can develop and verify their algorithms, including the performance impacts of operating in parallel. Affectionately known by the decidedly non-marketing name, SPUBox, the platform is a fully-functional version of the NPS appliance, with four Snippet Processing Units, a host processor and network connectivity. Weighing in at a little over 40 pounds (18 kilos), one might call it a "0^th generation luggable analytic appliance" but one that only consumes about as much energy as two 75W light bulbs. We granted more than ten of them to new NDN members at our September global user conference in Boston, some with special decal "wraps" to stand out above the ordinary compute platforms you may be used to.

<font color="#008000">It Takes a Vilage...</font>

I think the word potential is important when applied to streaming analytics, because what we're doing is opening the door to the potential of the data warehouse to be used in a very different way. Those extended uses are being made possible by Netezza and our community of users, developers and partners that is being fostered and growing, virtually with each passing day.

What we are seeking to unleash is a new level of performance and innovation in the use of storage-intensive analytical computing, but the important bit is that Netezza is not looking to do this alone. In fact, we do not picture ourselves as having cornered the market on analytical algorithm writers. Instead what we launched with the NDN is intended to evolve as a cooperative and competitive global web of experts who will build on their own and one another's innovations. Here, the term, "coopetition" seems trite; I'd prefer to think of the NDN as an opportunity for innovative "mashups" at the building-block, advanced algorithmic and applications levels.

Foundational elements are used together to enable basic value-add functions to be built. Those are mixed and matched, typically but not always with standard SQL fare, to enable more complex algorithms to be realized. And, in turn, the algorithms enable very high-performance applications and new uses of the NPS appliance to be realized. In some cases, one entity may do most or all of the above work. In many others, we are already seeing cooperation among members to use and reuse modules developed elsewhere to extend the capabilities.

Like what has been accomplished by artists with a humble plastic child's toy such as the Lego, these capabilities can be mixed and built-upon to create innovations we may not even be able to imagine today.

The opportunity is helped along by the network effects of an open community and members (by last count, in excess of 50 spread around the world) spanning entities from university professors and graduate students to BI applications providers to end-customers of the Netezza Performance Server^? appliance - and everything in between. This is where the true industry expertise lies. This is also the source of innovation for what can be possible with the opening up of Netezza's architecture to more than "just" data warehouse and BI.

Where will all of this lead? To advanced text, image, bioinformatics or video processing? Perhaps. Into the domain of the 'what if' Monte Carlo or Genetic algorithm simulations for risk analysis and predictive resource optimization? That's another possibility. But we're confident that people are going to use the NPS appliance in new and innovative ways as a result of Streaming Analytics and the NDN - and in ways which may well help shape the features and functionality of the appliance in releases to come.

<font color="#008000">What's So Special?</font>

What's so special about all that? Well with these foundational building blocks, imagine being able to develop customer- or threat-scoring algorithms that could be accomplished in as little as one pass through record data in a data warehouse instead of multiple passes required to denormalize or pivot data, or worse still, large extracts of the data from the warehouse to an off-board computing complex in order to perform the denormalization and scoring tasks. What if this single-pass technique yielded a 10X speedup in processing? What if it could be more than 100X - perhaps even allowing a task that formerly was accomplished in over 10 hours to be done in less than 20 minutes? Might that change the way that particular analytical task was used? Might that change someone's business? We think it could. More importantly, so do many of our customers, partners and prospects.

To date, the Netezza Developer Network has dozens of active partners participating in the program globally, with more than 100 applications to become part of the program pending [note: if you're thinking of your own really exciting "on stream" application ideas, you can apply online at http://www.netezza.com/ndn]. We think from the combined innovation and expertise of this group, the NDN has the potential to take the NPS analytic appliance to new levels of performance and new applications domains that will continue to include, but may go far beyond, the standard Data Warehouse Appliance of our roots.

by Ellen Rubin - Netezza, Vice President of Marketing

"Hang on - It's starting again
Hang on - There's no shelter from the wind
Hang on - Like a fire from the sky
Winds of change are blowin' by"

- closing chorus from "Winds of Change", by The Jefferson Starship and Grace Slick, 1982
[[click here for YouTube video]|http://www.youtube.com/watch?v=Z50G07A6HzE]

Witches, Elvis impersonators, bikers and the odd clown rush by, while outside, strong winds are blowing and the sky is dark and stormy.

Yes, it's Halloween at The Data Warehousing Institute's Orlando conference, and as 600 attendees try to celebrate without their families (actually, no one seemed too upset), Hurricane Noel is blowing in.

In fact, for the data warehouse community the storm has already hit. The appliance revolution has taken place and the impact is causing some extreme after-shocks. The talk around the halls was about appliances, and there were two half-day courses dedicated just to this topic. One was led by Richard Winter and Rick Burns of WinterCorp, providing an overview on appliances as well as a comparison of the different architectures and products on the market.

There’s certainly a need for this kind of information. Since Netezza launched the appliance category, built a community with well over 100 large customers and became publicly-listed, pretty much every major vendor in the industry has launched its version of an appliance, and at the TDWI show, several new players were clamoring for attention. It's pretty confusing for people who are just beginning to consider the appliance approach. Unlike the attendees at Netezza user conferences and events, the typical TDWI attendee is probably thinking something like this: "Boy, I’ve been hearing a lot about appliances lately, there seems to be a lot of news about them from TDWI and in the press, and Gartner says they're becoming mainstream. I better find out more about what they really can do and whether I need to think about this for my organization."

I guess when you've been evangelizing about the appliance concept for more than five years it's good to be reminded that in many ways, this is still a new frontier.

Back at the WinterCorp course, Richard talked about the market trends that have created a need for appliances, and made the point that "what you really want is to answer any question on any level of your data at any time." I couldn't agree more. Actually, he told a joke that really made the point: A doctor, a lawyer and a statistician went deer hunting. The doctor shot at a deer and was two feet too high and to the right. The lawyer shot and was two feet too low and to the left. The statistician said, "No need for me to shoot - according to the statistics, I've already hit the deer!" In case you missed the punchline, Richard added, "Sometimes, highly aggregated data does not get you the right answer for many business questions." Again, couldn't agree more: all the data, all the time is what appliances are about.

I also had a chat with Wayne Eckerson, a leader at TDWI and expert on predictive analytics. I was describing how, through our Netezza Developer Network, Netezza is opening up our appliance to developers all over the world who are doing new and cutting-edge analytics "on stream," leveraging Netezza’s streaming architecture. Wayne pointed out that there has really been an evolution over time from purpose-specific desktop systems just for the analysts in an organization who do the heavy quant work, to now, embedding some of that functionality in the data warehouse, and eventually, being able to combine it with the more traditional reporting and analysis work done by BI users. The new frontier in appliances is all about this broader role of analytics - including more groups of users, types of data and analytic algorithms - that can be done "inside the appliance," and as usual, Netezza customers and partners are at the forefront of the revolution.

Ellen Rubin