Thinking Inside the Box

SAS Goes Inside the Netezza Appliance

pfrancisco — Mon, 16 Nov 2009 16:07:06 GMT

“The best vision is insight.” -- Malcolm Forbes (1919-1990), publisher of Forbes magazine, New Jersey state senator and adventure hobbyist.

A couple of big announcements from our friends at SAS today. For the industry at large, SAS’ commitment to in-database analytic processing is a confirmation of trends that we have been discussing for over two years: more and more, the “data warehouse” is becoming the hub of all analytics processing for the enterprise. While that announcement covers multiple database vendors, today’s other announcement from Cary, NC on the availability of the “SAS Scoring Accelerator for Netezza” means that we and SAS are immediately putting this recommitted strategy into action.

Of primary importance to Netezza’s customers is the fact that with SAS’ intensification of In-Database functionality, SAS and Netezza will continue working together to deliver ever more advanced analytic capabilities inside the Netezza appliance. And the first step on that path is an excellent one: the availability SAS Scoring Accelerator for Netezza means that Netezza’s customers are able to execute SAS scoring models directly within the Netezza appliance and in-line with other SQL query processing on their data. The SAS Scoring Accelerator for Netezza will be Generally Available in early 2010, and Netezza and SAS are already working with a small number of early adopter customers such as Catalina Marketing, as they begin to benefit from this powerful functionality.

These scoring models are used in virtually every vertical market in which Netezza sells our products for fraud detection, credit and risk analysis and market segmentation. By embedding them in the Netezza appliance, customers will get the same 10-100X market-leading performance on scoring their data as they do on query processing. By running in-database customers can score all their data and not be reliant on only using samples or aggregates for expediency. And the in-database scoring also means that the inherent delays, or latency, in getting at the data to score it has been eliminated. The best way to deal with the large amounts of data being loaded in today’s data warehouse systems is not move it unless necessary, so Netezza’s AMPP architecture and method of moving the data processing as close as possible to where data is stored delivers huge performance gains for in-database analytics.

The on-going partnering work with SAS, and specifically the Scoring Accelerator, are part of the conversation with customers, partners and the market in general that Netezza began back with our Enzee Universe world tour in September regarding our vision for the industry and for Netezza. It’s known as “Netezza Insight” and CEO Jim Baum used his keynote addresses in seven cities around the world to begin the dialogue of taking Netezza and the concept of data warehousing “deeper”, “higher”, “wider” in a “unified” enterprise-wide platform approach together with other partners in the community. In smaller settings with customers, partners and analysts since then, we’ve continued that dialogue since the Enzee Universe and generated real excitement as they come to understand the full breadth of what Netezza is enabling in the market.

In coming days, we’ll be writing more about Netezza Insight and how it is manifest in product platforms, features and applications. But for today, let’s just say that SAS and Netezza customers are already able to do more, faster, with our combined products than ever before and that this is just a step toward even more powerful capabilities.

As Rick (Humphrey Bogart) said in the closing scenes of Casablanca, “I think this is the beginning of a beautiful friendship.”

[UPDATE: Rather than just reading what I have to say, you can watch SAS Executive Vice President and CTO Keith Collins describe his take on the value of in-database processing and the Scoring Accelerator for Netezza in the following video from the Enzee Universe 2009 show in Boston.]

A "FUD-machine" in overdrive?

pfrancisco — Tue, 04 Aug 2009 21:31:18 GMT

A loyal customer alerted us toan Oracle blog by Jean-Pierre Dijcks earlier today that showed the Oracle FUD machine is fully revved-up and ready to go. I'd like to offer a rebuttal, however in the interest of not intruding on Jean-Pierre's entry with an overly-long comment, I've just put a short response on his blog post with a pointer to this one.

Misconceptions and Misunderstandings, or Errors and Plain-old FUD?

I’m writing to correct *just a few* of the misconceptions about what is really important in high-performance, scalable data warehouse systems, errors, or just plain-old pure “competitive FUD” points from Jean-Pierre's posting earlier today. We certainly have posted some information recently about the TwinFin product and Curt Monash’s postings late Thursday provided more info. If his readers are interested in learning more, or even signing up for a “Test Drive”, they should visit www.netezza.com.

First off, I think this is a “banner day” for Netezza. We believe that TwinFin (and the other products in the new product family) extend both our performance and price-performance advantage over our competitors. We stand by our marketing statements that we regularly demonstrate 10-100X performance advantages over our competitors, particularly competitive offerings of the major incumbent DW system vendors (“Just who are those incumbents?” Jean-Pierre's readers may ask. Well let’s just say that we see Oracle as the incumbent system and/or a challenger system in over 50% of our deal flow.).

Regarding his claims about DBM being “faster than Netezza” (and I can only assume he meant at “real” data warehouse tasks) - we’re ready whenever Oracle feels up to actually taking one of their Database Machines onsite to a customer for a fair, open customer benchmark. So far, Oracle have been, shall we say, “a little reticent” to do on-site benchmark testing against Netezza.

Next, given the large number of incorrect points in the original posting, I think perhaps that just a few of them will be useful enough for readers to get the gist of just how far afield some of the ‘facts’ are:

“It all comes down to data scan rates per rack”: Would that it were true that all of data warehousing boiled down to full-stream data scans (as if the entire world of analytics relied on “select count(*) from lineitem” types of queries), then we could all measure “goodness” on how many GB/sec of data could be burst-scanned in our systems. But that’s not the case. So we build Netezza’s data and analytic appliances to deliver the best possible overall performance at the best price and power requirements. As a consequence, and following from those same numbers as-posted, a single rack of TwinFin can process (not just scan) about 400 million rows of data per second. That’s process, as in: “scan, decompress, project, restrict, AND join, etc.”. Need more processing firepower? Netezza’s system performance scales linearly with the addition of more S-Blades: at the low-end, the TwinFin 3 can deliver as much as 100M rows/second of processing horsepower, while the TwinFin 120 can provide you with 4 billion rows/second. Does a system that still relies on using SMP-based servers running “plain old” Oracle 11g RAC scale similarly for data warehousing?

“Non-open Linux running on FPGAs”: I’m really not sure what (if anything) was meant by this, but saying that Netezza’s FPGAs “are apparently running non-open Linux” is oxymoronic on at least two different levels (FPGAs don’t typically “run” an OS and, “non-open Linux” - really?)

“User data & compresssion”: I also enjoyed the accounting of all that “user data” available to DBM users in the Oracle table and the various comments about compression. When Netezza quotes user data capacities in our systems, the numbers reflect real raw user data space, not space that will be further reduced because of required indexes in an attempt to boost performance. Furthermore, Netezza’s compression & decompression techniques allow us to extract “pure performance” from their use. By not relying on CPU cycles to decompress the data before we can process it any further, the FPGA engines decompress the data, on-the-fly, as fast as it streams off the disk drives. Can Oracle make either of those claims?

“Tolerating node failures without downtime”: In perhaps the most bald-faced inaccuracy, the Oracle blog claimed, that Netezza “continues to lack the ability to tolerate node failures without downtime”. This I can only chock up to pure competitive “FUD-ism” as our capabilities in this area have been quite strong throughout the four generations of Netezza appliances and are further strengthened in TwinFin. Netezza is a fully-redundant system with no single point of failure, even in our smallest systems. Failover in the presence of failures of the disk drives, S-Blades, internal networking or host processors (in short, everything) is automatic and done in-service, with hot-swappable replacement throughout.

“Appliance simplicity”: One thing Jean-Pierre didn’t address that might have been humorous to see his take on is the notion of “appliance simplicity” - basically the ability to build, support and maintain large to very large-sized data warehouses, with heavy workloads, with no or minimal tuning, partitioning, indexing or other “performance duct tape” required. Routinely, this capability in the Netezza systems is what delights our customers most and we have customers managing systems with several hundreds of terabytes of user data (not indexes + data, mind you - real data) with fractions of an FTE (full-time employee) devoted to them.

I hope that clears up some of the misconceptions. If any of Jean-Pierre's readers or Oracle customers would like to see or hear more about TwinFin for themselves, we definitely would invite them to come stop by our booth (#207) at TDWI or come to one or our regional Enzee Universe events coming to a location near you.

Shooting the Curl in San Diego

pfrancisco — Tue, 04 Aug 2009 14:28:03 GMT

"You stay classy, San Diego." -- Ron Burgundy (Will Ferrell) in "Anchorman" (2004)

This morning a few others from the Netezza Marketing and Product Management teams and I are ensconced by the Marina in sunny San Diego, CA for the TDWI World Conference and for an news announcement or two. And who better to bring us "Breaking News!" than the Number 1 newsman in all of San Diego, Ron Burgundy. [For those of you who might have been "hoping for more" from Ron in a quote about San Diego, you can check out the IMDB database for some great ones, including Ron's own historical (and hysterical) etymology for the city's name.]

Though it’s not exactly a state-secret at this point, today we’re launching the 4th generation of Netezza data warehouse and analytic appliances and the first of four initial product lines in it: TwinFin™.

Some of the core characteristics of the TwinFin and the overall platform are:

Resetting Netezza’s price-performance leadership position in the market and extending Netezza’s performance lead;
Disrupting the competitive data warehouse market among the incumbents, just as we did with our initial systems in 2003/’04;
Moving to a commercially-available, blade-based server and storage platform; and
Opening Netezza’s aperture on the broader market with a multi-product platform design to match customers’ data warehouse and analytics needs across their enterprise

After the market disruption Netezza caused with the introduction of the NPS® in 2003 and since, we have seen the entry of dozens of new startups in our wake and virtually every major incumbent data warehouse vendor has retooled its portfolio to include a “response” to the Data Warehouse Appliance (DWA) in a suddenly reenergized market. Several of them, to their credit, have advanced their value propositions and improved their competitive position.

Now it is Netezza’s time once again. With the introduction of TwinFin and the other members of the new family of products, Netezza is once again changing the game; widening the applicability of our systems to more types of customers, applications and partners in the market.

As stated in my response to Curt Monash, my response to Curt Monash last week, we think of this 4^th generation of the Netezza appliance as using “the same architecture with a new physical implementation”. Starting with TwinFin, we moved to a commodity blade-server based system framework, but one that still uses Netezza’s “secret sauce” to deliver as much as a 5X increase in performance over the previous generation of Netezza systems, namely:

· our balanced design and streaming architecture;

· the use of Field Programmable Gate Array (FPGA) technology as a query processing “turbocharger”; and

· our advanced MPP management and optimization software.

And there are more innovations and performance gains on the way! TwinFin, quite simply, will serve as a platform for expanding Netezza’s performance and price-performance advantage in the industry and as the basis for advancing the state-of-the-art for in-database, analytically intensive data processing; all without sacrificing any of the appliance simplicity with which our company is synonymous.

As a couple of us said last week, Netezza has served as “the benchmark” for high-performance DWA pricing in the industry and we are now leading “the market in pivoting to a new competitive price-performance level”. With these new systems, we have embraced a trend that has been happening around the industry – the movement of marginal cost of a bit of disk storage toward $0 – with system-sizing, pricing and even system numbering focused on the performance delivered by a given platform.

We think the net effect of the new, simplified pricing structure for TwinFin and the other members of the Netezza product family will create a major disruption in the market. With starting (US-based) prices that equate to under $20,000 per terabyte, TwinFin’s list price is a fraction of other competitors’ performance-system pricing (after they’re all done playing price-obfuscation games around mirror, swap and index storage).

TwinFin and the other new Netezza data and analytic appliance products give us the opportunity to continue to lead the market and provide our customers with the best value and performance possible for all of their data warehouse and analytic processing needs. Netezza TwinFin - because two fins are faster than one.

Change, but no Change

pfrancisco — Fri, 31 Jul 2009 05:53:11 GMT

Just trying to clarify. Curt Monash's informative blog on the coming Netezza system and family of products includes the following:

<snip>

Beyond the switcheroo in components, Netezza is making substantial changes to its hardware architecture. In current Netezza products, the FPGA plays the role of a disk controller on steroids — it receives data, does some SQL or other analytic operations on it, and then throws it over the wall to the CPU for the rest of the processing. The new Netezza product family, however, adds an actual disk controller. More important, it adds fast interconnects between the FPGAs, the disk controller, and RAM — specifically, as Phil Francisco put it in an email,

using multiple parallel channels of PCIe with much faster interconnection rates and lower contention between the blade server and the “DB accelerator card” with the FPGAs.

DMA (Direct Memory Access) technology also fits into the picture somehow.

<snip>

...which seems to beg further clarification.

While Curt suggests big changes are afoot in Netezza's “architecture” - I think a more appropriate viewpoint would be that it's “the same architecture with a new physical implementation”. That is, the concept of data streaming from disk through the system is just as important now as it ever was.

True, we did move the "disk controller" function to a pair of HBA (Host Bus Adapter) cards that interface with the disk enclosures using multiple, redundant SAS (Serial-Attached SCSI), and providing more than ample bandwidth to stream all the drives per rack continuously to the blades. For those who click-thru on Curt's blog, this function is embedded in the device labeled “SAS Expander Module” (one on both the blade server and the "DB accelerator") in the 3rd chart of the PDF file (and also shown above) and allows data to stream from disk through to memory and then on to the FPGA without delay.

To move data between the blade server and the DB accelerator cards, we use IBM's expansion card (formerly known as "sidecar") technology to provide multiple parallel high-speed PCIe (peripheral component interconnect express) channels delivering the data streams from the disk drives to the memory on each blade server and providing very high-speed interconnect between the FPGA devices and that same memory, using DMA (direct memory access) to effect high-speed memory access without encumbering the CPU to get at it.

With all this high-speed interconnectivity, Netezza has been able to alter the data flow so that data streams to the memory first and then to the various FAST engines (see above diagram and/or refer to Issue 16: The Latest Addition to Netezza's FAST Engines Framework) in the FPGA. Those engines act as a "turbocharger" for query processing, implementing data decompression, restricting, projecting and applying the appropriate visibility rules in a pipelined process; typically filtering out well over 95% of the data scanned. From the FPGA, the resulting reduced data set is passed on to the CPU memory for additional processing to complete the process.

So, the logical streaming model of data from from disk to FPGA to CPU is retained, with significantly higher throughput as a result. But there's an added benefit: the fact that the originally-scanned data can remain in memory, still in compressed & unfiltered form, to be used as a cache avoiding disk scan activity where possible and helping boost system performance even more. In short, "Change, but no Change."

I hope that helps - with Curt's architecture viewpoint as well as with questions about our use of PCIe interconnects to raise performance.

"Catch a wave and you're sittin' on top of the world"

pfrancisco — Fri, 31 Jul 2009 02:00:52 GMT

"Don't be afraid to try the greatest sport around

(catch a wave, catch a wave)
Everybody tries it once
Those who don't just have to put it down
You paddle out turn around and raise
And baby that's all there is to the coastline craze
You gotta catch a wave and you're sittin' on top of the world"
– from "Catch a Wave" by The Beach Boys (1963)

Surf's up! Summer seems to finally have arrived in the Boston area and a number of vendors in the data warehousing and analytics space are hoping to catch a wave riding on a flurry of industry announcements. A few trends continue to build in the news:

Data sizes continue to grow alongside the pressure to increase performance & shrink data latencies;
Workload complexity and user counts continue to grow;
More and more, customers are seeing the value of running advanced analytical processing directly in their primary data repository (see item #1 for reasons why); and
Industry prices for data warehousing and analytics have begun another shift downward.

Today I'd like to address this last point. According to more than one industry analyst, over the last several years, Netezza has served as "the benchmark" for DWA pricing in the industry. Several of our competitors have sought to match and/or undercut Netezza pricing in the market. Some of the incumbent players have tried to, with very limited success, hinge their pricing off Netezza prices, match the performance of the Netezza Performance Server® system, or inoculate their pricey "flagship" products by adding less-expensive, feature-deficient products to their portfolio. But Netezza has continued to succeed in the marketplace, becoming a profitable, publicly-traded company with nearly 300 customers and 400 employees worldwide and one that is listed among the "Leaders" in the Gartner Magic Quadrant.

When we disrupted the data warehousing market with our first generation product in 2003 and 2004, Netezza was one of very few startups in an otherwise moribund industry. Now, with established "street cred" and hundreds of loyal customers, we intend to once again upset our competitors and lead the market in pivoting to a new competitive price-performance level. We're about to launch the fourth generation platform of our data warehouse and analytic appliances, which will advance Netezza's performance leadership and once again establish a new price-performance benchmark.

Admittedly, we won't be the first vendor offering high-performance data warehouse systems to move to a lower pricing plateau. That task is usually done by early-stage start-ups looking to find a way to differentiate themselves. True to form, Dataupia probably can claim establishing a lower price point first and recently another multiyear "start-up" has also started lower. But those are offerings from very modestly-sized startups with no established market "track record". Netezza will be the first company with proven product maturity, customer base and financial viability to do so.

Just how and what are we doing to cause this disruption? Well, let's just say things around the "briefing table" have been quite hectic, and that I and others will have more news about that to follow shortly.

[As you might imagine, it's been getting more and more difficult to keep things under wraps – in recent weeks we've even had to fight people off from getting early "sneak peeks". ]

Until then hey, it's summertime! So here's what I'd recommend –

"So take a lesson from a top-notch surfer boy

(catch a wave, catch a wave)
Get yourself a big board
But don't you treat it like a toy
Just get away from the shady turf
And baby go catch some rays on the sunny surf
And when you catch a wave you'll be sittin' on top of the world

Catch a wave and you'll be sittin' on top of the world"

Twin Fin: A short board (usually 5'8" - 6'8") with a wide tail for maneuverability and a fin near each rail for stability in radical turns.

Purpose: A wider tail area provides more planing area and lift, which creates more speed by efficiently utilizing wave energy. Milking speed and energy from smart surf with extremely sensitive and responsive turning ability are this design's strong points

Setting the Record Straight

pfrancisco — Tue, 16 Dec 2008 23:49:21 GMT

We came across a series of blog posts the other day which seemed to insinuate that Netezza and other competitors might be trying to shape our 10-100X performance message on the backs of comparisons to antiquated, end-of-service life systems and not comparing to current competitors' platforms . When we got to this one - "Database Customer Benchmarketing Reports" - I felt we just had to correct the record, so I wrote a response to Greg Rahn's posting to give Netezza's side of the story, namely that

we are as up-front as possible with prospective customers and use the customer benchmark testing/POC process to prove out Netezza's performance, value and simplicity value propositions;
the results of other products' performance come from our prospects/customers and not the result of Netezza running the tests on those platforms;
not only do we test against the incumbent systems, but there is almost always at least one other current competitive system that is included in the POC process;
the PowerPoint deck on which Greg was doing his analysis contained some rather ancient (in enzee-years, anyway) comparisons with versions of the NPS appliance that we have not sold in as much as 4.5 years & was really not much of a data set on which to base his analysis; and
the "proof of the pudding is in the tasting" - Netezza's success rate of converting prospects to customers through the customer benchmarking process remains very strong.

In short - we make every effort to keep the Netezza website contents both accurate and clear and we definitely feel confident in standing by our 10-100X performance claims. It would be great to have more than the Netezza "product marketing guy" clarify things - while I know some of the excellent results recent customers have seen in POC, no one knows them better than our customers & SI partners themselves.

"Netezza Underground" - NPS at your bookstore

pfrancisco — Wed, 26 Nov 2008 20:37:04 GMT

We had quite a surprise the other day when it came to our attention that Netezza and the NPS data warehouse appliance are now the subjects of a new book: Netezza Underground: The unauthorized tales of derring-do and adventures in resilient data warehousing solutions, by David Birmingham (ISBN: 1-4392-0743-7 and now available in paperback version for $31.54 at Amazon.com).

This is not the first instance of the NPS system being the subject of a book sold by Amazon (e.g., SAS/ACCESS(R) 9.1.3 Supplement for Netezza), but this particular publication certainly brought feelings of both fun and reaching into the mainstream with it, starting right from it's very clever cover art (above) to David's clever turns of phrase and real-life examples.

As the title suggests, it was not written or coordinated with any Netezza authorization. So of course we bought a copy and read/skimmed through it as quickly as we could. I will say this, David's self-publication skills are great - he keeps what could easily have been a boring, heavy technical tome both engaging and fun to read while still imparting lots of great information about the NPS system, its performance and its ease of operation. And the book's publication is incredibly current - with references to Netezza Developer Network and "BI Appliance" announcements made only as recently as the Enzee Universe user conference in September.

While I certainly could quibble with a point made here or there about the system, in general I thought it was an excellent book and even put up the following recommendation for it on the Amazon site:

I commend David Birmingham on a book that is at once as lightly entertaining and interesting to read as it is chock full of details about just the kind of performance and operational simplicity that is possible with the Netezza Performance Server (NPS) system. Straightaway from the opening pages, Birmingham's effusive, engaging style and excitement about Netezza's system is apparent, "It inhales, crunches and publishes Libraries-of-Congress-at-a-time - and fast."

He also captures the essence of the NPS appliance in an ultra-succinct two-sentence paragraph explaining just why his "Administration Stuff" chapter is so short, "It's an appliance. Put it in the corner and let it work." I couldn't have said it better myself!

This book is comprehensive and current - even reflecting some of the more recent announcements from Netezza regarding OnStream programmability, the Netezza Developer Network and analytic appliances.

As the guy who is responsible for projecting the Netezza products and our technology direction forward, I want to recommend David Birmingham's book to current and prospective customers and partners alike, or as David himself says on the book's Dedication page, "to Enzees everywhere".

--Phil Francisco, VP Product Management & Marketing, Netezza Corporation

So "to Enzees everywhere", have a read of David's book and welcome to the "Netezza Underground".

Oracle’s Entry to the DWA Community – just some of the Snarkalicious Commentary

pfrancisco — Fri, 26 Sep 2008 06:34:29 GMT

Okay I'll admit that my first posting about the new Oracle Data Warehouse Appliance (DWA) tonight was a tad on the "snarky" side. But I have to say that I think it was because of all influences in the environment all around me. Straight away since the announcement yesterday afternoon, there's been a healthy degree of skepticism from industry insiders.

Beyond his commentary on Larry Ellison's hairstyle, Gavin Clarke of the UK's Channel Register virtually flogged Larry for flogging the "Oracle server appliance alliance with HP". Some of the best snippets included:

Gavin's subtitle: "(Not) a hardware provider"
"And so to chief executive Larry Ellison, who Wednesday afternoon announced Oracle's third effort in 10 years bundling his company's software with someone else's hardware. This time, it's a high-performance, Oracle data and storage server stack locking arms with old favorite Hewlett-Packard."

And after taking several informative paragraphs to expound on Oracle's two previously-failed attempts at ‘appliantization' - most recently the "Network Computer" initiative circa-2000 - to draw the clear analogy to yesterday's announcement, Clarke closed out his piece with this stinger:

"In a telling sign of how much faith Ellison places in his latest appliance, he did not sit down for his traditional, open-mic smack-down session with OpenWorld attendees to field questions."

Analyst/blogger Curt Monash summarized more than a few skeptical digs in his Oracle Exadata and Oracle data warehouse appliance sound bites posting earlier today. For example, here are a few "bites" from Curt's post:

"Oracle Exadata Release 1 is hardly going to put Teradata, Netezza, or Greenplum out of business."
"After long denying it, Oracle has finally admitted that putting more than 10 TB on Oracle had been an extremely painful thing to do."
"Oracle's Exadata-based appliance doesn't have the out-of-the-box simplicity that other appliances and analytic DBMS do."

VP & Global Marketing CTO Chuck Hollis of EMC weighed in with a couple good shots on his Chuck's Blog post: Oracle does hardware (emphasis mine):

"Of course, there's little in the way of performance comparisons to help us evaluate just how fast this beast might go, except the ‘Up To 10x Faster' which smells a bit optimistic, never mind that it's Oracle comparing with itself, rather than other data warehousing appliances."
"Every year at Oracle Open World, we hear about many "new initiatives" from Oracle. Well, not to be harsh here, but it's my impression that very few of them get talked about at next year's Oracle Open World. I routinely dig up past announcements from previous years, and it's relatively consistent pattern. I think it's fair to ask the question -- just how serious is Oracle about all of this?"

But the lead cynic was none other than Oracle CEO Larry Ellison himself. After years of denying performance issues at scale with various generations of Oracle DBMS software for data warehousing, Larry dropped this 11g-megaton bombshell about Oracle's data warehouse scalability, pre-Exadata - laying out the fundamental reason why Netezza has become the industry leader in Data Warehouse Appliances (source: ZDNet's Larry Dignan):

"Ellison, speaking at Oracle's OpenWorld conference, said large databases are creating a fundamental problem: Disk storage systems can't cope with data that has to be moved off of drives to database servers. He called it a ‘data bandwidth problem.'

"As data gets larger the slowdowns become more unbearable. At one terabyte you will notice data bandwidth slippage. At 10 terabytes, storage systems crawl. ‘At one terabyte the problem rears its ugly head and it gets worse every year,' said Ellison."

And that's not all - the barbs, skepticism and "bites" go on in site-after-site, and commentary-after-commentary. So please forgive my snarky-ness - I blame it on the "nuture" of my environment, not my personal "nature", per se.

Please Send More Product Brochures!

pfrancisco — Fri, 26 Sep 2008 03:29:40 GMT

It was an odd email exchange. Only 30-minutes earlier, at approximately 3:04pm US-PDT, Oracle CEO Larry Ellison, head of one of the most powerful database technology companies on Earth, had publicly launched Oracle's entrée into the Data Warehouse Appliance marketplace: "the HP Oracle Database Machine and the Oracle Exadata Data Storage Server" - while simultaneously "sporting a curiously Romanesque hair style".

Larry Ellison & Julius Caesar - separated at birth? (Wikipedia: Julius Caesar)

Perhaps we should have been cowered by such a goliathan announcement? Perhaps we should have quivered? Well that's when the email showed up. You see, Netezza had a booth (or "stand" - as I'm writing this from London tonight) in the exposition area of Oracle's big OpenWorld show in San Francisco. Within minutes of Larry's presentation, in which Netezza figured prominently albeit with substantially erroneous information across Mr. Ellison's charts, the Netezza stand was completely deluged with people saying things like, "I had never talked to your company about data warehousing before, but if Larry is going to spend 10 minutes talking about you, I need to know more." And the Netezza product brochures starting flowing - not in a trickle like a leaky pipe, but like water through a burst dam.

Larry hadn't just brought up Netezza but had spent some "quality time" extolling the strengths of the Netezza architecture - moving query processing horsepower as close as possible to the storage elements of the system, and his commentary had marked Netezza as the leader in the Data Warehouse Appliance (DWA) approach. Within the hour, our team's supply had run out. Undeterred by the lack of the product brochures - the team had moved on to distributing our glossy fold out "BI Emergency Survival Guide".

But what this anecdote from the floor of a 50,000-person trade show really meant was that a sea-change had happened in the industry. No less than Larry Ellison had put his imprimatur on the DWA industry segment and in so-doing had also summarily marked Netezza as the industry's leading vendor in the segment.

Since then, phones have rung off the hook and email exchanges have approached the immediacy of Instant Messaging, with in-bound requests for more information about the Netezza Performance Server^®. Whatever doubt that existed in the market that DWAs were a force in the marketplace was eradicated yesterday... at approximately 3:04 pm US-PDT.

"Please send more product brochures," indeed! Thanks for all the sales leads, Larry! We'll get around to correcting all your misconceptions about our product shortly.

Diving Deep into Analytics from the Netezza Platform - Part Deux

razi — Wed, 03 Sep 2008 14:02:24 GMT

In theory there is no difference between theory and practice. In practice there is.
-- Jan van de Snepscheut, 1953-1994, computer scientist and educator, California Institute of Technology

So, yesterday I wrote about "the Netezza's" transformation into a platform for deep analytics. Now I know a platform is only as good as the applications available on it, which brings me to our announcement this morning.

Last September, we got together with a handful of visionary partners and customers and created the Netezza Developer Network (NDN) with the goal of developing truly innovative analytic applications. We announced the first wave of these offerings today, with 5 NDN partners delivering game-changing applications built using Netezza's OnStream analytics. Let me highlight a couple of them here.

Systech Solutions' profitability analysis application for retail and CPG companies provides cost and revenue analysis at the detailed SKU and customer level. It gives business users the ability to build and run profitability models using a GUI, instead of relying on IT to do it for them. This is pretty unique, because traditionally something like this would take huge amounts of time - measured in many months - not to mention the resources required. Their app cuts this down by orders of magnitude! So you not only get very fine-grained profitability analysis, but it's available very, very quickly. That makes all the difference between gut-feel decisions and data-based ones about which products and customer to keep, which prices to re-negotiate and how to truly impact the bottom line.

Imagine if telco service providers could analyze each and every one of the many millions of call detail records they collect and store, before making very important decisions - the kinds that can dramatically alter their earnings statements. That's what RateIntegration's app offers - a tool for business users that allows them to model the impact of competitors' pricing and regulatory changes to figure out the most optimal rate plans. Business analysts can also directly implement custom scoring algorithms for customer segmentation and profiling using their flexible rules engine.

Apart from these, we have Multi-Threaded Inc's fuzzy name and text matching app for critical anti-terrorism, money laundering and digital forensics operations; HCL Technology's implementation of Monte Carlo simulations for pricing derivatives; and Edge Associate's library of SQL functions to speed up migrations to "the Netezza". Make sure you check out the brand spanking new applications webpage to get more details about each of these members and their applications, and don't forget to stop by their booths at the User Conference.

"As long as one does not have to wait minutes to hours between computational gestures, something amazing happens; one gets problem solving at the speed of human insight"
-- Data-Centric Computing with the Netezza Architecture, Sandia National Laboratories

While the new applications developed by NDN members are unique and serve very different markets - retail, telecommunications, financial services and government - they have remarkable similarities in the value they offer to customers. The applications power complex analytics orders of magnitude faster than economically feasible before, allowing users to perform "what-if" analyses to more accurately predict future outcomes. These analyses can be performed on large volumes of detailed data, providing unique business insights that would otherwise be lost in sampled and summarized data. The deployment and management of the overall solution is greatly simplified, freeing up business users to focus on results rather than worrying about tuning and maintaining the system.

What's really neat about Netezza's open platform approach is the ideas and innovation it is generating and the differentiated applications it is helping launch. Now that's what platform innovation is all about, isn't it? At Netezza, it's about bringing the power of analytics to the mainstream.

Diving Deep into Analytics from the Netezza Platform

razi — Wed, 03 Sep 2008 02:27:40 GMT

"The milk of disruptive innovation doesn't flow from cash-cows "
-- David Isenberg, Blogger, Musings About Loci of Intelligence and Stupidity

Dare I say ... "orders of magnitude performance" for data warehouse applications is old news as far as Netezza customers are concerned! It became fairly obvious to me at the Netezza European User Conference, held a few months ago. In presentation after presentation, customers talked about the performance and simplicity benefits they got from "the Netezza" - how the proof-of-concept (against their favorite legacy data warehouse vendor) seemed unbelievable at first, but certainly proved true in production; the fact that they did indeed get orders of magnitude better performance; and how all this changed the way they did business. Brian Ganly of The Carphone Warehouse used this chart to highlight Netezza performance during his talk about the "Netezza Experience." I think it captures the sentiment really well ...

It's not that data warehouse performance is not important any more, or that somehow the 100X performance that Netezza delivers is "enough". In fact, what the Netezza customers were alluding to, in a customer's own words, is: "Netezza does what it says on the tin!" We talk about blisteringly fast performance without requiring tuning and aggregations at half the cost of other systems, and we deliver. Once customers see for themselves what "the Netezza" can do for their data warehouse, they get intrigued about the possibility of what else it could do for their business. And that quickly leads them to look beyond raw performance for data warehouses and apply "the Netezza" to new and interesting big-data analytic problems.

"... the best products become platforms at some point."
-- Bob Warfield, author of the SmoothSpan Blog

As the data warehouse market continues to evolve, more and more companies are looking to use information as a competitive lever across their organizations. The most successful will be those that make use of information to exploit arbitrage windows in the marketplace and predict future outcomes more accurately. These companies will differentiate themselves by making high performance analytics pervasive, providing employees, partners and vendors access to the kinds of analytics that are only available to a select few in the enterprise today.

What's needed to deliver on the promise of advanced analytics is a platform that can overcome the challenges of doing deep analytics on large data volumes - performance, complexity and cost. Let's look at how advanced analytics are done on traditional systems. In most cases, these poor data warehouses are so overtaxed that adding any more processing is a certain way to bring them to their knees. And so the usual approach is to extract huge data sets onto an outsized SMP server or compute grid, perform the analytic computation on it and load result sets back to the data warehouse for querying. You can clearly see the problems with this approach. It's expensive, especially when you're talking about a large SMP or grid; it's complex since you have more systems to maintain; but most importantly you get poor performance even if you spend tons of time and money on the infrastructure. The data movement back and forth introduces the same latency and performance bottlenecks that still plague traditional data warehouse architectures.

What we've done with "the Netezza" is created just such a platform that overcomes these complex analytics challenges. The idea is quite simple actually. Algorithms for analytic tasks such as scoring, text and spatial processing, image and video analysis and financial simulations can be run directly on the intelligent nodes inside the Netezza. So these algorithms can act on the data where it resides, rather than sending it off-board for processing. You not only get the benefit of fully parallelized execution across hundreds of processors resulting in orders of magnitude better performance for analytics, but also the simplicity and economy of an appliance. Plus the Netezza is able to handle all this extra processing because of the spare processing capacity built into each of its intelligent nodes. Let me refer you to Phil Francisco's blog for a blow-by-blow version of how "[OnStream analytics|p-1032]" works in practice.

This is all great so far - I mean any platform that provides these kinds of advantages has to be quite extraordinary! But the true value of a platform is determined by the applications that run on it and how innovative and differentiated they are. That's where there is a lot of interest and excitement in the enzee community. More on that very soon ...

Netezza - Now with Added CLARiiON!

pfrancisco — Tue, 20 May 2008 13:49:59 GMT

"And one of his partners asked, 'Has he vertigo?' and the other glanced out and down and said, 'Oh no, only about ten feet more.'"
— Ogden Nash, American writer and humorous poet (1902-1971)

Today's News: Netezza and EMC Partner to Simplify Data Warehousing for the Enterprise
This is hot! No, I mean it's really hot. I'm here in Las Vegas this week, attending the EMC World show in support of today's partnership announcement with EMC and yesterday the temperature crossed 110 degrees Fahrenheit (43 Celsius). Oh yes, and our announcement here this morning is hot as well - stirring up interest around the EMC World show floor. We had several great discussions with EMC field staff, their partners and some customers yesterday. In the imperfect domain of trade show tsotchkes as metric, in just four short hours yesterday we ran through nearly 400 tee-shirt give-aways. Of course, as an aside, I would subjectively say our shirt was clearly a "best-in-show" candidate here - witness these front & back photos:

Complementary Technology and Co-marketing for Success
"So what is this announcement all about?" you ask. From the viewpoint of our customers and prospects, it's primarily about Netezza partnering with the industry-leader in information infrastructure to bring the performance horsepower of the NPS^® data warehouse appliance into the enterprise data center in an even simpler, more efficient way than we already have been. It's consistent with their continuing evolution of data warehouse appliance deployments - as business-critical systems that are used enterprise-wide. And it provides them with both operational flexibility and performance in supporting the requisite data management functions.

From Netezza's business perspective this makes the NPS appliance even more well-suited to a broader sweep of customers. We'll provide provide familiar, enterprise-class data backup, replication and disaster recovery capabilities that extend Netezza's simple appliance approach by embedding EMC's CLARiiON^® storage arrays and Navisphere^® & MirrorView™ software. But just as important, we'll work with EMC on co-marketing initiatives in bringing these configurations to market. As a result we anticipate being able to extend our market penetration, both with new customers and broadening our footprint in current accounts.

How does this partnership change an NPS system?
The partnering initiative with EMC involves two basic configurations of the CLARiiON AX4 storage arrays. Both configurations will be deployed within the NPS data warehouse systems themselves and will require no additional data center footprint, with very minimal impact on the system's very low power and cooling requirements.

First off, let me say straight-away that this partnership in no way changes the basic NPS system architecture that Netezza has made use of through three generations of the its data warehouse appliance. The NPS system still will achieve its scalable, high performance through the unique AMPP™ architecture, including the Snippet Processing Units (SPUs) whose design is unaffected as a result of this partnership.

Instead, the EMC storage arrays will be used to provide near-line storage for the NPS systems in order to stage data for all of the primary bulk data movement operations: loading, unloading, backup, data replication and disaster recovery. In short, the most basic functional block diagram of the NPS appliance will evolve
from this: to this:

The two types of embedded CLARiiON configurations are as follow:

Storage Pad™
Deployed as part of the standard equipage in our two-rack (NPS 10200) and larger systems, the Storage Pad configuration will support up to 5 TB of near-line data capacity for staging ETL data loads, data unloads and incremental backup images. Applying an approach that I've come to call Tivo for data warehousing, the Storage Pad allows customers to time-shift data management functions to suit their operational requirements. For example, customers might make use of the Storage Pad to move backup data from the data warehouse rapidly and then move the backup data from the Storage Pad to a tape or disk archive at the rate that the data center network, media and operations scheduling will allow.

By comparison, other vendors may charge as much as $100,000 for just 1.5 TB of capacity for similar functionality.

Storage Pad XL™
As the name suggests, this optional configuration is scalable to high capacities that scale in-line with the NPS systems in which they will be deployed. The Storage Pad XL configurations will be available on all NPS 10000 series models and will support up to 10 TB of near-line data capacity per NPS rack - up to 80 TB for the 8-rack NPS 10800 system. Just like above, the Storage Pad XL can be used to for data staging, but now full images of the NPS tables or databases can be captured for high-speed backup.

In addition, this configuration will also be support the deployment of EMC's MirrorView software package for enterprise-class data replication and DR.

"Netezza - Now with added CLARiiON!" is the fun spin we put on the marketing look of our booth at EMC World this week but we think this partnership will provide our enterprise customers with an excellent, simple toolset for bringing the appliance paradigm to data management functions; and through our co-marketing initiatives with EMC, it will bring the NPS data warehouse appliance into more of our customers' data centers.

]]>

Issue 19: The Compress Engine - The Netezza Philosophy

admin — Thu, 15 May 2008 19:48:45 GMT

April 28, 2008

"‘To be is to do.' - Immanuel Kant

"‘To do is to be.' - Jean Paul Sartre

"‘Do-be-do-be-do' - Frank Sinatra"

--Kurt Vonnegut, Jr. (Nov 1922 - Apr 2007)

In the news today: the Compress Engine

In 1783 Immanuel Kant wrote, "David Hume woke me up from my dogmatic slumbers," and revolutionized the way humanity thinks about metaphysics. Almost 220 years later, Netezza set out to achieve a similar goal - redefine analytics. When the first NPS® data warehouse appliance was introduced, the market released itself from yet another dogmatic slumber and realized that there is a different, better way to do data warehousing; a way without compromise, a way without limits.

Netezza has helped to reenergize the data warehouse market in creating and leading the data warehouse appliance category.

"Every time you turn around you see another industry that's facing a tidal wave of data and they need to understand what this data is saying. Many of them have data volumes in this range that they haven't been able to afford to analyze, as much as they'd like to. ... Netezza can deliver that analytic capability, and at a very attractive price." - Richard Winter, Winter Corporation, from Netezza will scale its appliance to petabyte range, InfoWorld (January 2008)
"This is what Netezza has done in the data warehousing market: it has totally changed the way that we think about data warehousing... So the bottom line is not just that Netezza's entry into the market was a black swan event but that that event has not ceased to unfold." - from Netezza: a black swan by Philip Howard, Bloor Group (October 2007)
"Appliances are here to stay and are revolutionizing the data warehouse industry." - from Business Analytics Appliances Are Here to Stay, by Dan Veset, IDC (June 2006)
"The term data warehouse appliance was coined by Netezza, and this vendor has blazed a trail by proving the concept and educating the market." - from Defining the Data Warehouse Appliance, by Philip Russom, TDWI (August 2005)

Since 2002, Netezza has been repeatedly breaking the latency barrier and challenging the boundaries of data analytics. Since our first release, we have been continuously refuting the alleged mutual dependencies that became the building blocks of the industry's dogmatic misconceptions; namely the expensive nature of performance, the necessary complexity of the analytics architecture and the unavoidable limits of scalability. With today's announcement of the Compress Engine, Netezza disproves yet another myth - the inverse relationship between data compression and query performance.

The architectures of traditional data warehouses, steeped in a legacy of serving OLTP applications, were not designed to handle the ever-growing amounts of data combined with larger and more complex user workloads and shrinking data latency requirements that characterize the modern enterprise. Regulatory compliance, electronic commerce and the need to process and analyze all data in a matter of seconds has pushed the capabilities of traditional data warehouse systems to their limits. In reaction to the data capacity pressures, vendors introduced compression; not as an enhancement but as a compromise solution that allows for further data growth at the cost of processing performance.

Traditional compression approaches, used by several of the competing data warehouse vendors, typically result in performance degradation to accomplish the compression effect. Netezza's addition to the FPGA-Accelerated Streaming Technology (FAST) Engines framework - Compress Engine - utilizes its innovative streaming architecture^TM not only to increase the system's storage capacity by 2-4X but actually boost overall streaming query performance by a factor of about 2X (100%). All this is achieved without requiring any tuning or administration, and it is in fact a software-only upgrade that enables Compress Engine on the Netezza appliance.

It's actually really cool technology, obviously something we love to rave about. Late last year, I wrote about FAST Engines in this blog. We'll use that as a starting point and dig a level deeper into how Compress Engine works. I'm sure it will tickle the fancy of the geek in you!

The NPS system employs a patent-pending method for compiling (yes, compiling) columnar data in all the tables of the database as it is being written to disk e.g. during load, insert or update operations. The process converts row-based data into column streams that are independently compiled to replace the original data in the columns with a stream of "instruction sets" for the FPGA. The "instructions" themselves are much smaller in size than the data they replace, resulting in a highly compressed data stream emerging from the process.

While the compression occurs on columnar data because of the inherent compressibility within database columns, the compressed data is reassembled in rows before being written to disk. Row-wise storage of tables avoids the data scan complexity associated with columnar stores and ensures that scanned data can be efficiently parsed and processed without the need to reconstitute it from multiple sources. The compressed data uses disk much more efficiently and increases the data density of NPS systems by 2-4X - in some cases substantially higher - allowing customers to scale their NPS data warehouse systems into the hundreds of terabytes of user data.

But if the NPS system's data compression and scale brought the system's performance to its knees or severely limited performance speedup due to compression (as it does on many of those other systems), it wouldn't be so great, would it? The beauty of the Netezza way of providing data compression is that not only does it have no negative impact on performance, but it actually increases query performance by up to 100%!

As the compressed data is read off the disk, it is passed through the Compress Engine which applies the instructions embedded in the data stream to restore it to its original form. Our compilation algorithm ensures that this decompression process can be performed entirely in silicon, at wire speeds. Each physical block scanned from the disk can mushroom into 2 to 4 or more times its size in memory without incurring any overhead in processing time - i.e. 2 to 3 times more data is scanned in the same amount of time without any increase in system hardware! Our internal benchmark testing reflecting real customer configurations and workloads has shown an overall 2.2X increase in streaming query performance through the use of Compress Engine.

This software-only enhancement, enabled by our unique architecture, is only the beginning. As we continue to develop our platform, we are investigating further enhancements to the Compress Engine or the addition of new FAST engine(s), aimed at directly increasing streaming performance on the NPS system.

Our philosophy and aim is to continue to shake the industry out of its dogmatic slumbers by extending the price/performance advantages of our products; showing that there's a different way to do data warehousing and advanced analytics. One where performance and scalability are neither the result of expense nor complexity, where you can get more performance from compression, where you do have the power to question everything^TM ...

Issue 18: Teradata's "Me-too" Model 2500 – welcome to the Data Warehouse Appliance club ...finally

admin — Tue, 13 May 2008 13:58:01 GMT

April 21, 2008

"Imitation is the sincerest of flattery."

- Charles Caleb Colton (1780-1832), from his Lacon, Vol. I, published in 1820

Welcome to the Data Warehouse Appliance club - another validation of an important, growing market segment

Well, well, well! "Only" eight years after Netezza coined the term and invented the market segment, Teradata today finally officially entered the Data Warehouse Appliance market. Though it's a bit late, and certainly behind a number of other vendors, perhaps today's entry will put an end to Teradata's vacillating over whether they 'invented' the concept or not, were an appliance or not, or whatever. In the past couple of years, it seems Teradata spokespeople have gone out of their way to say their product was simultaneously a data warehouse appliance and absolutely not one - even booking appearances on panels of data warehouse appliance "vendors". Certainly their announcement is another validation that the role of Data Warehouse Appliances is an important and growing one not only in the current market, but for the future as well.

Derivative Marketing and a "Repackaged, Warmed-over" Product?

Teradata is positioning this new product as being, "simple, powerful and cost-effective" - which to our way of thinking sounds much more than a little derivative from Netezza's["Performance, Value and Simplicity"|http://www.netezza.com/customers/data-warehouse-appliance-customers.cfm], but I'll leave it to the reader to decide if you think so. Our reading of the Teradata announcement sounds like just another larger vendor's "repackaging" alternative to respond to the competition. Like others before them such as IBM and Oracle, it appears that with the 2500 model Teradata has done nothing more than cobble together a collection of elements from the company's model 5500 systems, repackaged and sold as an appliance.

Powerful. Um, How's That Again?

And while anyone who is serious about the appliance segment of the data warehouse market (like Netezza) has focused on delivering systems that can scale to highly complex, enterprise-wide, high performance systems, we think the 2500 will struggle to deliver even modest performance for just 6 TB in a single equipment rack.

While Teradata is quoting just over 6 TB of user capacity per two nodes in this new system, let's remember that they have been advising customers for the past year not to put more than 1.5 TB against each of those same dual-core CPU nodes. Which is it? Is the 2500 underpowered for its 6 TB data capacity per dual-node rack, or has Teradata been advising its model 5500 customers to pay at least 2X too much for their data warehouse systems for the past year?

Time will tell whether Teradata has made other compromises to the 2500 model in an attempt to limit its impact on its flagship products (5500 and the new 5550). Beyond its underpowered nodes, have they sacrificed anything else like workload management or system availability, or even the system's ability to handle highly-interactive, operational applications? As the days and weeks help raise the shroud covering the model 2500 further, we'll know more. For now though, it just feels like "me-too" imitation.

Issue 17: Information Arbitrage - When big data plus big math pays off

admin — Wed, 16 Jan 2008 03:00:00 GMT

by Jit Saxena, Netezza Chairman and CEO

"If you are a Big Dog and you are not persuaded by data, then in God we trust...but everyone else, bring data." - Jane E. Shaw, retired Chairman and CEO of Aerogen, Inc. and current member of the Intel Board of Directors (quoted from PowerSpeaking Inc.)

More and more companies recognize the power of analytics as part of their competitive strategy. But most solutions only provide a glimpse of what can be achieved. What is the potential impact when performance barriers fall away? In this post, I’d like to explore the possibilities and introduce a few examples of companies leveraging the intelligence in their data in new and unexpected ways. After all, competing is good, but winning is better. In finance, the term arbitrage refers to the ability to find and exploit market disparities (hedging strategies monitoring currency or securities fluctuations being prime examples). Most arbitrage opportunities are very time-sensitive - you have to recognize value in an overlooked stock then swoop in to buy it before others take notice, get the same idea and drive up the price. On Wall Street, an arbitrage virtuoso, able to consistently spot untapped potential that others miss, is worth his or her weight in gold.

Leaping through Tiny Windows

The term Information Arbitrage has many similarities to its finance equivalent, and it’s a good way to think about the impact that analytics can have on a company or even an entire industry. Information arbitrage is about finding game-changing intelligence buried in vast, unappreciated data assets, and exploiting it to leap ahead of the competition. Like a financial investor, the Information Arbitrager takes advantage of an opportunity before the window slams shut (which can be very fast indeed). Companies in certain industries make particularly good arbitrage candidates. These are companies dealing with "Big Data" - tera-scale or even peta-scale databases, and a constant flood of incoming data. Telecommunications, eBusiness, RFID retail applications and online advertising are a few segments that come to mind. Often the operational data is changing very quickly, and key insights are only found at a very granular level. Now suppose this normally takes hours or days, and one company can suddenly do it in minutes, seconds or even sub-seconds. As Netezza customers well know, this kind of intelligence disparity can have dramatic implications, both for that company and its market.

For example, telecommunications is a high-volume, low-margin business. Constant changes in network utilization demand real-time decisions about rating and pricing structures for an operator to stay competitive. By running pricing scenarios against billions of call data records, and by examining individual customers to determine their current calling patterns and preferences, iBasis, a major telco provider, knows exactly what options to offer each customer. In contrast, competitors might only see that customer as part of a larger segment measured at some time in the past, and come up short with their offers and pricing.

There are several challenges to Big Data analytics that make arbitrage opportunities hard to pursue. Predictive modeling, optimization and other analytic applications are much more processor intensive than the SQL queries used in standard business intelligence applications. When complex algorithms and gargantuan databases converge with real-time business demands, something usually has to give.

Many companies find they are unable to fully exploit their growing data holdings, and have to make do with sampling or high-level summaries rather than the complete, granular data they often want to examine. But using partial or high-level data can be dangerous; even the most powerful algorithms can suggest spurious or meaningless conclusions when they are applied to insufficient data. Companies may also lose hours offloading data from the data warehouse to an external cluster of processors to run the analysis. With all these approaches, the result is an incomplete solution that provides just a hint of the possibilities of analytics, because that’s all the current technology is capable of delivering.

Consider the problem of optimization, for example. Optimization solutions play a key role in helping companies target the right customers, make the right offers, determine manufacturing volumes or accurately price products to take full advantage of market conditions while minimizing expenses. Depending on the problem being addressed, an accurate optimization solution needs to account for many variables and constraints such as products, branches, budget, time, contact channels, offer history, market segmentation and privacy preferences, to name a few.

Due to the multiple permutations and combinations among the different elements, even a simplified optimization model limited to only a month of data, a thousand customers and ten different offers results in an astronomical solution search space of 2 to the power of 10,000. Just to put things in perspective, the number of atoms in the observable universe is about 10^81, just a few more variables away.

The "Big Math" at the heart of this kind of analysis pushes most processing technology to its limits and beyond. As the number of variables and restrictions increases linearly, the algorithm amplifies exponentially, often reaching the complexity class NP-Complete. As a result, companies are forced to compromise in the thoroughness of the analysis and/or the response time they are willing to tolerate. Most optimization efforts look at small snapshots of the total data available (for example, only the last month’s data), and make use of a range of techniques such as Linear, Dynamic and Integer Programming, Lagrange Multipliers and Cluster Analysis that reduce the level of complexity in various ways, all in an attempt to reach an actionable result in a realistic timeframe. But even with these approaches, companies are faced with costly infrastructure requirements, incomplete views of their data and lengthy response times resulting in stale data or missed arbitrage opportunities.

But what if you could bypass the existing performance limitations and get crucial intelligence much faster than before? For example, what if a database marketing company could use complex algorithms to get accurate optimization results days before the market could adjust? Or a retail franchise could precisely adjust the prices of thousand of products daily for each of its stores? Or a credit card company could run customer scoring algorithms one hundred times faster than its competitors? Or a financial services firm could run real-time Monte Carlo simulations on terabytes of data to manage risk? What impact could advantages like these have on a business? It’s fair to say the difference would be game-changing, providing a major competitive advantage and the ability to enter new markets previously out of reach.

These capabilities are not just marketing fantasies or future visions - they’re in use today.

Making these Information Arbitrage opportunities possible is precisely what Netezza does. Our streaming analytic appliances are built for running complex mathematical models on huge data sets, with results in a fraction of the time required by other technologies. Sophisticated analytic applications run "on stream" in the data warehouse, against all the records and detail that need to be examined. There’s no need to settle for summary data or aggregations, or ship data to another system for analysis. (We’re also constantly making our appliances better. Our recent doubling of performance is just the latest Netezza breakthrough.)

Through the Netezza Developer Network, we’re helping developers worldwide use the Netezza architecture to create a new generation of analytic applications that were previously impractical, unaffordable or simply impossible. When exploiting an arbitrage opportunity means leveraging Big Data and Big Math, Netezza’s streaming architecture is simply inherently faster and more efficient than other technologies. Of course, our customers already know this - and with appliance simplicity and low purchase price, information arbitrage pays off even more.

The bottom line is: when Big Data meets Big Math, great things become possible for our customers and their businesses, enabling them to:

Use Information Arbitrage to take advantage of time-sensitive opportunities
Rapidly run multiple scenarios and sensitivity analyses in near real-time
Make use of all the available data, all the time while their competitors are still struggling with reduced visibility from sampled or aggregated dataWhen the first Netezza appliances burst on the scene in 2002, their ability to query giant databases with unprecedented speed upset a lot of preconceived notions about the limitations of technology and what companies can do with their data. Advanced analytic applications take processing complexity to a much more challenging level, and once again the capabilities of our appliances are revolutionizing the market and capturing the imagination of our customers.

Jit Saxena