[ www.netezza.com ]

Blog Posts

Blog Posts: 129
Items per page
Statistics: Blogs: 11 Blog Posts: 129   1 2 3 ... 5 Previous Next
0

As some of you know, here in Texas the sport of football is a bit more than a team sport. In some locales, it's practically royalty. Back in high school and college, I was ever-aware of the sports fans that liked to wear the team colors. One in particular invited me to lunch while his parents were in town (I already knew them from back home) but I never knew that his father owned a car that was actually painted in Dallas Cowboys colors. Another colleague of mine was stuck in a parking garage with a dead battery as I was leaving work, and asked if I could help him with a jump. Upon saying yes, he produced a set of maroon-and-white jumper cables, honoring Texas A&M University. Oh yeah, he was a fan.

 

Then several of my friends kept large jars of team-color makeup and would smear it all over them before a game. No, not to go see the game at the stadium, but in their apartment! Yes, the team makeup was always a big hit, and their parties were solely centered on the tiny color TV in the center of their rather modest apartment. I had once wondered what the people next door thought of their shouts, whoops and antics, until I learned that the folks next door were just as rowdy with their own team. Hey, ya gatta be a fan.

 

Okay that's not the team makeup I'm talking about here, but those were just such funny memories I thought I'd share. And to make a point of course. The enthusiasm of your developers and architects, and their desire to cheer you on with your goals to achieve, are directly dependent on the team makeup. Of people, that is. So what does it take to make a data warehouse? Or for that matter, what does it take to effectively roll out a new one, or migrate an old one?


What most people fail to estimate in taming such a beast, is the level of testing required to make it a reality. As an example, many of you must produce documents as part of your regular workday. Those documents are often hard to write, but even more work to proofread. In fact, proofreading is the same form of content-and-context testing we would do for a data warehouse. The chief reason is the product - information and knowledge. Business intelligence is the same way - it has a way of taking on a life of its own, but the only way we can reliably roll out a viable business intelligence platform, is to test, test and test some more. Eyeballs-to-page however, may be required for a book or document, but it won't scale for a warehouse.

 

Many people don't realize that the testing portion of the warehouse can take as much as 80 percent of the project's resources. While we can compress this somewhat with agile methods, we cannot afford to test such quantities of data with simplistic manual approaches. And by that I mean eye-ball examination or screen-shot testing. No, the majority of testing is in the data itself, and on a Netezza machine it's in the billions of records. Eyeballs don't have the bandwidth. We need to use the actual power of the machine to scale this mountain.

 

So what should the team itself look like? May I suggest that for every Architect you would have several developers, and for every developer you would have two or more testers. Ideally three testers for each developer, regardless of how many developers are actually doing the work. I will also suggest that you keep the total count of developers rather lean. Five is perhaps overkill for the back end. For the front end, three solid developers can be an army, and five is about the upper limit. The reason is simple: logistics. If you have five developers on the back end and five on the front end, and three testers in the wake of each, this is a team of 40 people - which quite frankly is overkill in any sense of the word. Not to say that an overall team might not be comprised of 40 people once we include all of the infrastructure folks, but not for pure develop-and-test. We can and should make it leaner.

 

As an aside, I had the rather disturbing experience, numerous times, of encountering folks who worked these things out with overblown spreadsheets that they normally used for application development estimations. A data warehouse gig is completely different from an application development gig. But of course, if one of these spreadsheet guys ever showed up and plugged in his metrics, he would spout off that we need a 30-person team to migrate a couple of tables from one machine to the new one. Once this number is in the air, it becomes the de-facto standard by which all discussions are measured, even though it is completely wrong. In another setting, another spreadsheet-guy plugged in his numbers and characterized a project as a $900k gig when our competitors were bidding $300k for the same work. Knock yourself out, dude, because the client ain't a-bitin' three times more expensive projects because they like our faces. True to form, the $300k bid actually won. But the irony was, that the potential client had no desire whatsoever to pay more than about $400k, so the bid fit their budget just fine. The eventual winner of the bid took a bath, however. The truth is always somewhere in the middle.

 

I still say, watch those spreadsheet-guys. Somethin' up with that.

 

Perhaps it goes without saying that an architect needs to lay out a framework so that all can work comfortably in the same sandbox. This is a challenge and should not be left to the developers to forge on their own. Harnessing it later will be impossible, because too many opportunities for flow-based consolidation will be lost. Workarounds and repetitive logic will become the rule. Let's not go there.

 

If we have say, three solid developers in the back and front end each, they can and should cross-test each other's  work. In this case, we have the senior developers and architect working on the core logic, and the junior developers bench-checking their work and zipping it up for a formal testing team. Here we have a synergy, that a senior developer can crank out ten times the work and quality of the juniors (so says Demarco and Lister, your actual mileage may vary)  but nonetheless, we would not want to put a junior developer in the front seat of this chaim because the testers will be waiting on him more often than not. But with the ten-times-more-power driving the front like a locomotive, the junior developers can wrap up the many tactical areas of the warehouse and cross-test each other, but also receive the work products from the senior developer.

 

Now think about what this kind of model means. The senior developer is literally force-feeding the pipeline with work products and is doing it with the highest quality the team has to offer. The junior developers are learning from the senior without injecting their own inexperience into the mix, which will invariably have to be reviewed by the senior developer anyhow. No, the senior developer is more productive and experienced, so let him/her drive. Seems like every senior developer I talk to, they really, really want to develop and have the testing tedium off their plates. The junior developers really, really want to learn from a senior developer, and of course want to do some development themselves. I'm not saying this is off-limits, but the senior developer can delegate-what-he-knows to the junior developers because they cannot go too far astray without his guidance anyhow, so it's a win-win. And of course, I and every other person who was ever a junior developer had to pay our dues, so not everyone can be the leader. I don't say this dismissively, but we know in a business intelligence project there has to be a driving mind. Too much consensus means too little leadership, and in the famous words of Margaret Thatcher "Consensus is the absence of leadership".

 

But for the people doing the testing, they need something that will scale. To billions of records. And it had better be solid on the first round or they will be playing catch-up for every round after that. While writing and proofreading a document is an eyeballs-only model, don't you think I could at least do myself a favor and run a spell-check and grammar-check on the contents? Such set-based operations resolve a world of problems and let me focus my eyeballs on the harder stuff. But in a data warehouse, our eyeballs will never have enough bandwidth, and will never scale to the necessary heights. Set-based testing is all we have, but it's also all we need. And with a Netezza machine, we're so in the zone.

 

Now testing of the report screens can involve eyeball-based activities but doesn't have to be so egregious. Automated testing tools go a long way to mitigate the necessity for eyeballs on these as well (for the subjective parts like positioning, banners or colors especially). However, if the data is wrong, no amount of pretty-pretty will fix it. As Murphy would say "Beauty is only skin deep, but ugly goes to the bone."

 

Now, no sooner will I write this than I will get feedback from those junior developers who say that they have been relegated, but not to fear. This particular article is in context of a high-productivity bubble of work, normally found with new projects or migrations. The priority is not to make people feel better about their role, but to get past the workload so that everyone can feel better about the work products. I am always looking for opportunities to stretch the developers, both junior and senior. When a junior is ready to sit in the driver's seat of the locomotive, it's because he's passed the Demarco and Lister smoke test. Now what the heck is that, anyhow?

 

Get a copy of Demarco and Lister's Peopleware, a classic in every decade. Something they have empirically measured, is that a junior developer will start out at one level of productivity, and then in a sudden epiphany will transform into someone who is ten times more productive than before. Something mentally and/or emotionally clicks and they get this whoosh. They claim it is different timing for each developer, but usually takes about two years to make this transition. This is perhaps one reason why so many job-search requirement listings show "X years of experience in Y" and the "X" is never less than two years. Not because the poster has ever read Peopleware, but we who are in the field want folks who are 2 years along because we already know they have (at least) transitioned into a high-productivity asset.

 

But this is the mechanism driving the team makeup - and the experience of the developers and their known levels of productivity should help us find the right role for them on the team. We don't want a low productivity person in the locomotive chair. But having one in the wake of a strong developer only makes them stronger and exposes them to practices that will accelerate their transition into the higher productivity person we always wanted anyhow. And then, of course, once the person has made the 10x transition and is self-aware of their value, we have another problem: They are self-aware of their salary level too! Making someone stronger makes them more valuable. Be prepared to recognize the value (or rest assured that your competitor will). But all this, is the nature of the beast we purport to tame, no?

 

Back to set-based testing. This has as much to do with using the right data as it does the right method. The right data means - select a subset of known data that will deliberately exercise all of your business rules and software paths. Nothing is worse than realizing such errors in production. Then, we need set-based testing methods. This means we need three primary assets: (a) source data that we can sluice through our application transforms to get a result (b) a saved baseline result to compare this recent result against and (c) tested components that compare these two results in a reliable manner so that we get a statistical report on what passed and what didn't, and a detailed report on what specific records didn't make the cut. Counts, amounts, checksums and summaries all reveal deviations, especially for regression testing. You might recognize this as an exception report, and this is exactly the spirit of the effort. Our testing has to deal with statistical exceptions, because it is the only practical and scalable way to validate billions of rows.

 

And also notice that such a practice would be the kiss-of-death in many other "secondhand" data warehouse platforms. Such platforms are in no wise optimized to compare monstrous sets of data to each other column-for-column, row-for-row. Queries like that can dim-the-lights and may not return for hours, if not days. We cannot afford a protracted testing phase, and with Netezza we don't have to. Scan times and comparison times are very objective and knowable. The tests will take the same amount of time each time they run, and we always have the option to optimize them further with the Netezza performance model. Power is in the physics.

 

And again, why all the focus on testing? I have seen data warehouses blind-side an organization that accounted only for the opposite equation - 80 percent development and 20 percent testing, when more often than not, exactly the opposite is true. This would mean that if a two-month development effort were characterized with one model (the wrong one) it would look like at most a three-month effort. Why then does it metastasize into a ten-month effort? Because 20 percent (2 months) tranlates to 80 percent (8 months) of testing.

 

That is, if we just embrace the standard model. By embracing the aforementioned model, we get the development out of the way quickly and deliberately, entering the testing phase much sooner, and if more heads are deliberately dedicated to set-based testing we can close this part off even sooner. I have watched very-large-scale projects, with a Netezza team in the middle and strong developers in the locomotive seat, enter their first UAT phase within two months of the project's inception. The funny thing is, the model requires rapid turnaound that only the Netezza workhorse can provide, Try pulling off this team makeup with any other lower-productivity technology, and it won't sing the same key. A high-productivity developer is meaningless on low-productivity technology. And high-productivity testing methods are useless if enslaved to a low-productivity technology.

 

Start it, shape it, ship it. Netezza is the ticket home.

0 Comments Permalink
0

      

In a recent blog, Greg Rahn of Oracle responded to Phil’s “Oracle Exadata and Netezza TwinFin Compared” eBook; before commenting on an Oracle engineer’s views, I’ll restate the eBook’s larger themes.

 

Exadata connects Oracle’s RAC database, its architecture designed for online transaction processing (OLTP), via a fast network to a massively parallel processing storage tier. As an OLTP database paired with a specialized storage subsystem, tuning Exadata to function as a data warehouse is complicated and demands skilled, highly trained, experienced technical staff. Mitigating the shortcoming of an OLTP database pressed into service as an analytic database with expensive network and storage makes Exadata costly: to acquire; to design, tune and maintain as an optimally-configured data warehouse; to run in the data center.

 

Netezza TwinFin, designed as an analytic database, brings the power of massively parallel processing to manage and exploit data at terabyte-to-petabyte scale. TwinFin is an appliance–easy to install, easy to operate and easy to manage. TwinFin offers value: fast performance for advanced analytics at an affordable price.

 

Now I’ll discuss the detail of Greg’s blog and respond from a Netezza perspective.

 

Claim: Exadata Smart Scan does not work with index-organized tables or clustered tables.

 

Greg responds that “IOTs and clustered tables are both structures optimized for fast primary key access, like the type of access in OLTP workloads, not data warehousing” and suggests our intent was to mislead by quoting from an old Oracle datasheet. It wasn’t. Oracle 11g Release 2 documentation reads “Index-organized tables are suitable for modeling application-specific index structures. For example, content-based information retrieval applications containing text, image and audio data require inverted indexes that can be effectively modeled using index-organized tables.” Elsewhere the documentation states “Index-organized tables are useful when related pieces of data must be stored together or data must be physical stored in a specific order. This type of table is often used for information retrieval, spatial and OLAP applications.” In the eBook Phil discusses first and second generation data warehouses; many of the applications described by Oracle as candidates for IOTs are typical of those our customers run on TwinFin – these are second generation data warehouse applications. Greg believes Exadata smart scan not working with index-organized tables has zero impact on Exadata customers. Is it reasonable to conclude that Exadata is not being used for second generation data warehousing?

 

Claim: Exadata Smart Scan does not work with the TIMESTAMP datatype.

 

Since we published the first edition of the eBook Christian Antognini, the original source of this information, goes to the heart of the matter in his blog: “The essential thing to understand is that this limitation is due to bug 9682721. The fix is expected to be part of 11.2.0.2. According to my test cases (that Greg Rahn was so kind to execute against an early release of 11.2.0.2), offloading works correctly for all datetime functions but for the following three predicates.

 

  • months_between(d,sysdate) = 0
  • months_between(d,current_date) = 0
  • months_between(d,to_date(‘01-01-2010’,’DD-MM-YYYY’)) = 0”


Note that the MONTHS_BETWEEN function can basically be offloaded. The problem in these cases is that the offloading does not work when, for example, SYSDATE is used as a parameter.

While happy to let this one pass, I have a question. Do organizations accrue value or cost from a technology requiring its administrators understand all combinations of functions, their predicates and their parameters before they are capable of designing queries to be processed in parallel?

 

Claim: When transactions (insert, update, delete) are operating against the data warehouse concurrent with query activity, smart scans are disabled. Dirty buffers turn off smart scan.

 

In my opening comments I compared TwinFin’s simplicity to the complexity of Exadata. All queries submitted to TwinFin are processed in its massively parallel grid; no tuning, no special database design. This is appliance simplicity. In Exadata whether a query benefits from smart scans (massively parallel processing) can depend on the state of the data being read. Exadata requires developers to understand at great depth the physical path a query takes to access data. This is complexity.

 

While Greg concedes Exadata’s MPP processing is disabled for those blocks containing an active transaction he is confident that “Not having Smart Scan for small number of blocks will have a negligible impact on performance”. My experience with Netezza’s customers and their applications prompts me to take a more circumspect view. I’ll explain why in the next section.

 

Claim: Using [a shared-disk] architecture for a data warehouse platform raises concern that contention for the shared resource imposes limits on the amount of data the database can process and the number of queries it can run concurrently.

 

Greg argues contention for shared disk is not a problem for Exadata and cites Daniel Abadi’s blog in his defense. Let’s take a look at what Daniel says on this subject “If you are going to make an argument that shared-disk causes scalability problems, you have to make the argument that contention for the one shared resource in a shared-disk system is high enough to cause a performance bottleneck in the system - namely, you have to argue that the network connection between the servers and the shared-disk is a bottleneck.” This is the argument Phil makes in our eBook. Consider a query analyzing correlations between equity trades in a sector of a stock market. The algorithm calculates Spearman’s rank correlation coefficient (Spearman’s rho), measuring statistical dependence between two variables by assessing how well the relationship between them can be described. This analysis creates valuable insight in to whether specific equities influence behavior of other equities in the same market sector within a window of one to ten minutes.

 

The customer loads a massive volume of trading data into TwinFin and constantly trickle feeds data from live markets into the warehouse. The query is run and re-run constantly to assess behavior of different equities in dynamic markets. Each time TwinFin completes a Cartesian join between all the equities in the sector while at the same time calculating a Volume-Weighted Average Price and a Return From Previous Close value for the equity under investigation. The results pass to Spearman’s rank correlation coefficient function to calculate the Population Covariance and the standard deviation of every equity combination for the time period. Netezza executes every step of the query in parallel utilizing all TwinFin’s hardware and software resources. Netezza’s intelligent storage selects only the rows needed for that market sector and projecting only the columns needed for assessment. The join result is directly streamed to the code implementing the statistical analysis which TwinFin downloads to every processor in its MPP grid, running the complex calculations in parallel. Results from each node in the MPP grid are returned via the network to the host for final assembly and rendering back to the requesting application. TwinFin completes the analysis in a few minutes, and then runs it again, and again for as long as the market is open.

 

After several hours Oracle 10G was still attempting to complete its first round of analysis. What difference will a new version of the Oracle database paired with an MPP storage system and a fast network make? Exadata’s MPP storage grid is unable to process Cartesian joins, the first step of in this analytic process, meaning it brings no performance gain but must put all records on the network and send them across to Oracle RAC. Even if it we able to process the join Exadata cannot push down user defined functions, used to implement the calculations, to MPP - in Oracle functions always execute on the RAC servers. In processing the algorithms Oracle must create and manage temporary data sets and write these out of memory for storage. Exadata’s flash cache may play some role here, but the size of the data sets and the complexity of the algorithms will force database processes to write to disk. This flow from Oracle RAC is back across a network still clogged with coming from the MPP storage tier data, queued and unprocessed waiting for attention from a fully-consumed Oracle RAC. I contend that Exadata’s network connection between the servers and the shared-disk is a bottleneck. Not Exadata’s only bottleneck. TwinFin demonstrates how a true MPP architecture excels in calculating Spearman’s rank correlation coefficient - a real workload on a real dataset. Oracle’s OLTP database, simply not designed to process large-scale analytics, is overwhelmed. Exadata suffers contention on its network and in its database system’s shared disk architecture.

 

Back to the previous point about Exadata’s MPP processing being disabled for blocks containing an active transaction – the customer is constantly loading new market data and analyzing it in comparison with a massive volume of historic data. While entirely appropriate for transaction processing, Exadata’s architecture of disabling an entire block from parallel processing when a single record in the block is being updated can only hinder and never help in the data warehouse. The very point of a data warehouse is that all data should be available to the business as quickly as extract-transform-load processing allows. By pressing an OLTP database in to service as an analytical database Oracle unnecessarily burdens customers with creating database designs to work around this complexity and, developing a thorough understanding of how each query accesses the data model. While not having Smart Scan for small number of blocks may or may not impact performance, as an unnecessary complexity demanding the attention of database specialists, it costs customers real money.

 

Claim: Analytical queries, such as “find all shopping baskets sold last month in Washington State, Oregon and California containing product X with product Y and with a total value more than $35” must retrieve much larger data sets, all of which must be moved from storage to database.

 

Greg shows some nice SQL to demonstrate how Exadata processes the beer and pizza query. Give the business an answer and they always come back with a new question: “Greg, what was the total value of Brand #42 beer’ sold in each basket?” Greg can now update his SQL with the clause:

 

sum(case when p.product_description in ('Brand #42 beer') then td.sales_dollar_amt else 0 end) sum_productX,

 

and re-run the query. Business users love IT when we give them a fast performing system but are less forgiving when a query, that yesterday ran blazingly fast, today slows to a snail’s pace. Exadata cannot push down the newly introduced sum for parallel processing by its storage nodes as the join must be processed first, and the storage nodes cannot process joins. Any function or calculation that uses columns from two or more tables must be evaluated on the RAC database servers. The query performance is going to degrade significantly sending the database expert back to the Oracle documentation in an attempt to find a new way to resolve the amended query so it completes at a time acceptable to the business.

 

Claim: To evenly distribute data across Exadata’s grid of storage servers requires administrators trained and experienced in designing, managing and maintaining complex partitions, files, tablespaces, indices, tables and block/extent sizes.

 

While conceding Oracle Automatic Storage Management automates the task of striping partitions across all available disks, the ASM administration team must still create partitions, configure and manage disk groups for shared storage across instances, choose and implement either 2-way mirroring or 3-way mirroring, and configure Allocation Unit sizes. Additionally, Exadata configuration requires administrators create and manage tablespaces, index spaces, temp spaces, logs and extents.

 

In conclusion, Netezza entered the data warehouse market convinced the products offered by the dominant vendors, in particular Oracle, were ill-suited to meet the challengers of Big Data and of such complexity to make them exorbitantly expensive to acquire and use. Exadata only increases the complexity and expense of an Oracle warehouse. Greg draws his readers’ attention to the excellent blog at http://dbmsmusings.blogspot.com/ where Daniel Abadi muses “Both Oracle and Teradata are too expensive for large parts of the analytical database market.

 

Greg’s blog reveals one path available to organizations wishing to generate greater value from their data. CIOs willing to build, train, and permanently assign a team of technical experts to choosing just the right combination from a myriad of settings, can be continuously employed coercing a database designed for OLTP to function as a data warehouse. I’ll close this blog with a manager’s perspective, from someone who focuses an organization’s limited resources on its highest priorities. Peter Drucker, who introduced us to the concept of the knowledge worker, gave us a pragmatic measure to evaluate our own and our team members’ activity - am I merely efficient (doing things right) or truly effective (doing the right thing)? All the workarounds and clever tuning demanded by Exadata simply don’t exist in TwinFin, Netezza has proven them unnecessary.

0 Comments Permalink

Hi,

 

 

  following is my code, I am using netezza JDBC driver 5.0.10,this  driver support BATCH UPDATES, so with the following code i should get BatchUpdateException but I am getting SQLException would any  one please help me.I am first time working with Netezza

 

                             

                             

 

                                                      try{

                                                                     for(String dBstat: DbStatments)

                                                                    {

                                                                            statement.addBatch(dBstat);

                                                                    }

 

                                                                   noOfRecordsUpdated=statement.executeBatch();

                                                              

 

 

                                                         }

                                                         catch(BatchUpdateException  bue)

                                                        {

                                                               logger.error(bue.getMessage());

                                                        }

 

 

 

                                                       catch(SQLException sql)

                                                       {

                                                               logger.error(sql.exception);

                                                      }

 

0 Comments Permalink
0

Netezza Director of Product Marketing Razi Raziuddin is blogging today.


     

I’ve been at The 2010 TDWI World Conference in San Diego this week, where the theme is "agile BI that delivers data (I would use the term ‘insights’) at the speed of thought.” Timing is everything when it comes to making decisions – and influencing other to make decisions we’d like to see.

 

We’ve all experienced Red Car Syndrome at some point or another. You test drive a red car. You like it. Suddenly, you start noticing red cars everywhere – not because the number of red cars has increased, but because the experience of driving a red car is now personalized. Online advertisers use Red Car Syndrome to connect consumers with the products they genuinely want, as I was reminded first-hand recently. While searching for kitchen fixtures online, I noticed that many of the ads featured a pair of pricey fixtures that initially caught our eye, but that we had rejected as exceeding our budget. But the ads seemed to know our tastes better than we did, and ultimately we succumbed and made the purchase.

 

Red-Car-psd38311 6.jpg

 

The experience brought home the power of right-time analytics. Speed is critical in making analytics actionable and delivering real value to the business. The trifecta of huge data volumes, complex analytics and query performance is an increasingly common thread in the BI and data warehousing world. It is true not just for online marketers, but cuts across industry lines. Whether it is an insurance provider trying to prevent fraud, a telco determining the cheapest and best path to route a call or a government agency unearthing criminal activity, time to insight from big data makes the difference in every case.

 

Doug Henschen recently wrote a good article on this topic for InformationWeek in which he calls out success in the Big Data era as the ability to get faster insights from huge data sets. The article highlights Catalina Marketing’s  petascale data warehouse environment and the fast insights they derive from a huge database of 195 million consumers.

 

Although not every enterprise has a data warehouse environment quite that large, the need to perform complex analytics and derive insight in the shortest time possible is common in every environment, big or small. While scalable MPP architectures address the big data problem quite well, the big math problem associated with complex and advanced analytics is what many customers still wrestle with. There’s general agreement that in-database processing, especially in scalable MPP systems, is the right solution to the big math problem. Doug’s article again highlights Catalina’s use of in-database analytics to radically streamline their analytic modeling environment and gain efficiencies of 10X as a result.

 

However, not every data warehouse platform is geared up for the challenges of performing in-database analytics at scale. The first and obvious challenge is the additional processing overhead required to run advanced analytic algorithms alongside the traditional data warehouse workload. You need a system architecture that is not overwhelmed by the data volumes typical of data warehouses in the Big Data era. Then there is the question of what analytics you want to perform. The majority of commonly available analytic libraries are written for in-memory processing in SMP systems and need to be parallelized in order to take advantage of MPP architectures. The analytic system should not only offer parallelized versions of the analytics you desire, but also provide primitives to easily parallelize advanced analytic algorithms while hiding the complexity of parallel programming from developers.

 

Finally, the dearth of universally accepted standards in the advanced analytics world poses yet another challenge. A typical analytic environment may consist of a mish-mash of commercially available tools such as SAS and SPSS, open source ones such as R and Hadoop (which are gaining popularity), and tons of application code written in various languages such as Java and Python. The underlying system must offer tremendous flexibility in integrating with a wide array of analytic tools and support for a variety of frameworks and languages.

 

In subsequent posts, I’ll talk about Netezza’s advanced analytic capabilities to enable big math on big data. In the meantime, as you plan your analytic infrastructures for the Big Data era, tell us what challenges you are coming up against.

0 Comments Permalink

Hi,

 

  would any one please tell me how to get the list of SQLErrorCODES netezza  will return

 

Thx

0 Comments Permalink

Netezza SQL ERROR Codes in Enzee Universe

Posted by vgss Aug 10, 2010

Hi,

 

  I am trying to access Netezza from my JDBC code, would any one please tell me as where can I look for  NETEZZA SQL ERROR CODES

 

 

THX

vgss

0 Comments Permalink
0

I’ve been at Netezza now for just over three months and I still feel a bit like the innocent abroad but at the same time, I’m clearly a veteran, as other people join our growing team after me. We have our own EMEA Telco Solutions Manager (Chris Smith) and EMEA Alliances Manager (Kate Tickner). I’m just jealous they seem to be on top of their missions, much more quickly than me. It’s great that we have more dedicated resources for working with partners and customers, but i was struck last week by some other the stuff we do, working with customers, that is really much less visible.

Over every weekend I get cc’d on a bunch of weekly reports from folk in the org, which I scan, occasionally raise a question about and file. And amongst the reports are those from our technical account managers. These reports list customer accounts, each with a status - green, amber and red (not many of these – no really), the activities in the account this week, issues outstanding and what’s happening next. All sensible stuff and my input is sometimes to ask for a briefing if an account is at amber for two weeks or more, which generally they aren’t, and at red even more infrequently (but not never – get thee behind me corporate).

Then i rewalized that what I’m seeing is every single customer having their account reviewed every single week – proactively. I even read a report the other week in which a customer had requested less frequent contact; a sort of “everything is fine, don’t call me, i’ll call you”. And I was stunned. This is a customer complaining in that their supplier’s customer service is too responsive! Now that’s not a situation you meet often in my experience. Yet Netezza regard that level of customer service as standard process.

This was the next report for that customer:

 

Customer:                                        XXX

Status:                                            Green

Activity this week:                          No activity this week. Model customer J

Issues on hold/carried forward:     None

Next steps:                                     Light touch.

 

I just loved that ’Light Touch’.

0 Comments Permalink
4

Today Netezza is launching a new eBook entitled, “Oracle Exadata and Netezza TwinFin™ Compared”. As the name implies, this eBook provides a comparison of the Netezza TwinFin data warehouse appliance and Oracle’s “appliance-like” database machine offering.ebook_tfexam_thumb.jpg

 

Certainly Netezza is not the first company to compare/contrast its flagship system with Oracle’s most recent entry. Richard Burns, a consultant over at Teradata did a laudable job exposing the technical shortcomings of the Exadata v2 machine as they pertain to data warehousing in a May 2010 whitepaper. And there have been several recent pieces written on Oracle’s apparent success although the publicly named customer-list has struck some as a bit underwhelming.

 

Netezza continues to compete (and win) against Oracle regularly in the marketplace, including in competition with the Exadata v2 product and so, we felt it was high time to put our own comparison story together with today’s eBook and with this little blog posting. Let me know what you think.

 

So where to begin? Let’s start with the fact that the Netezza TwinFin is built to excel at a specific purpose – as the best price/performance platform for Data Warehousing and Analytics in the market. Conversely, Oracle has tried to “kill two birds with one stone” in the Exadata v2 – aiming it primarily at the On-Line Transaction Processing applications space, but also making bold claims to performance as a Data Warehouse with it’s Sun-based Oracle Database Machine (DBM) and Exadata Storage Server, version 2 (Exadata).

 

So why does it matter that Oracle is aiming to do both OLTP and DW in the same system – apart, that is, from at least two decades of people trying-and-failing to do exactly that with the likes of Oracle in previous software and hardware instantiations? Let’s start with the workload requirements of the two application areas:

  • OLTP systems execute many short transactions, typically of extremely small scope (touching only a handful of records) and in extremely predictable, well-understood access and query patterns. They need to excel at handling these small transactions in very high volume, combined with equally small writes to the database in the form of updates, insertions and deletions. This limited scope, high throughput and “regularity” of the access patterns make OLTP systems great candidates for intelligent caching and (multiple) secondary data structures, such as indices to speed their processing.

 

  • Conversely, DW systems are typically asked to perform “read-heavy” queries and operations against the current and deep historical data sets. Rather than analyzing just a few records, a DW query might look at millions, even billions, of rows from a single table, combined with join logic with multiple other tables. Data warehouse systems are used by company analysts and managers to find the “needle in the haystack” in guiding enterprise decision-making in a more comprehensive and often ad-hoc manner – frequently mitigating the ability to use “tricks of the trade” such as results caching and/or indices.

 

So the two applications tend to lead to very different system/platform implications. No special “news” there – as I said earlier, people have been trying-and-failing to use a single system for both applications for years.

 

Without stealing any more of the thunder of our electronic publication today, let me just lay out what I believe are the fundamental differences between Netezza’s TwinFin and the Oracle Database Machine/Exadata as simply and plainly as I can:

 

Netezza TwinFinOracle Database Machine / Exadata v2
True MPPHybrid "SMP-plus" Approach
Data Streaming with a Hardware AssistCPU-intensive Processing for Basic DB Operations
Deep Analytics ProcessingCentral Cluster-based Approach
No-Tuning-Required SimplicityComplex Array of Knobs and Levers

 

In my view, these are "big deal" differences. They're not the result of a simple feature gap to be closed in an upcoming point-release, but rather go directly to limitations at the heart of the Oracle DBM/Exadata system architecture and/or business culture. To address them would require a major rearchitecting, or at least refactoring, of Oracle's decades-old DBMS code base. They also happen to be highly visible to customers and prospects, which makes for some interesting comparisons in head-to-head on-site Proofs of Concept (POCs).

 

1) True MPP vs. a Hybrid "SMP-plus" Approach

Netezza’s TwinFin uses a full MPP approach to data warehousing, pushing all of the processing down as close as possible to where the data is stored and maximizing the processing horsepower of MPP for scalability, throughput and performance – for even the most complex workloads. Using the MPP method of dividing the workload and attacking query problems in parallel, Netezza has been able to demonstrate market-leading data warehouse price-performance across four generations of data warehouse appliances.

 

Oracle’s DBM/Exadata takes a hybrid approach adding Exadata Storage nodes largely to handle data decompression and predicate filtering tasks, but still relying primarily on the SMP cluster of Oracle RAC to handle most of the data warehouse tasks, including complex joins. In addition the SMP cluster also must act as the central distribution point for any data that needs to be redistributed between and across Exadata nodes. To try to minimize this, Oracle and Sun’s solution was to “throw hardware at the problem” (quoting Teradata’s Mr. Burns), over-engineering interconnections, processor rates and other elements required because of all of this data movement, rather than refactoring and solving a fundamental software architecture issue.

 

The difference between the two is akin to an 8-lane continuous streaming superhighway in the TwinFin instance versus multiple freeways converging on and necking down to a two-lane country road via a “traffic roundabout”. I live in Massachusetts and can attest to the negative impact of taking multiple highways down to a single road – it happens every weekend at the gateway to and from Route 6 on Cape Cod.

 

2) Data Streaming with a Hardware Assist vs. CPU-intensive Work for Basic DB Operations

In addition to the advantages of the MPP architecture for data warehousing, the TwinFin system makes use of hardware acceleration for increased query and analytics performance. Coming in the form of the "DB Accelerator" that is part of each S-Blade in the TwinFin system architecture, providing four dual-core Field-Programmable Gate Arrays (FPGAs) on each DB Accelerator, this hardware acceleration takes care of fundamental processing steps such as decompression, predicate filtering and ACID-compliant data visibility at the full scan rate of the data from disk. The fact that this device is placed as close as it is to the disks for which it is performing its processing gives the TwinFin system much more performance leverage because data can be filtered, processed and value-added before undergoing any unnecessary CPU processing or having to be transported across an expensive network.

 

And the fact that it is a field programmable device means that Netezza can use it to introduce additional features and performance through a simple upgrade to our NPS software/firmware – as Netezza has with the introduction of two phases of hybrid column/row-level compression technology (with Release 6.0, scaling as high as 32:1 compression, depending on data patterns) first introduced in 2005, and our high-performance implementation of row-level security. Because it's performed in the FPGA in TwinFin, "Compression = Performance"; so if a customer's data is compressed by a 4:1 factor, the effective data streaming rate for processing queries is increased four-fold.

 

Conversely, the DBM/Exadata system relies entirely on CPU processing. In fact, the great majority of the functionality provided for by the Exadata nodes in the DBM/Exadata system is to replicate the functionality included in each FPGA core of the TwinFin - data decompression and predicate filtering. Because of the CPU-intensive nature of decompressing data in the DBM/Exadata system, Oracle "strongly suggests" lesser compression when data is required for high-performance data warehousing vs. "cooler" queryable archive purposes. Again, the heavy-lifting for query processing and analytics is left to the central SMP cluster nodes rather than parallel Exadata nodes, forcing Oracle to "throw hardware at the problem".

 

3) Deep Analytics Processing vs. Central Cluster Analytics

Netezza brings analytics to where the data is stored – as close as possible to where it is stored to do the processing – not just to decompress it and do predicate filtering, but to complete as much of the complex analytics as is possible, in parallel. That’s as true of the “traditional” OLAP analytics of SQL-based data warehousing as it is of the advanced and predictive analytics enabled by the new capabilities of i-Class in the “Second Wave of TwinFin”.

 

With i-Class, Netezza introduces a comprehensive, scalable and high-performance approach to advanced analytics for both our customers and partners, spanning Linear Algebra/Matrix manipulation, and engines for R and Hadoop along with several programming languages including C, C++, Java, Python and even Fortran. The i-Class functionality also offers plug-ins and packages for the Eclipse IDE and R GUI, and pre-built, analytic functions engineered to deliver performance at scale spanning data preparation, mining, predictive analytics and spatial functions together with access to analytics functions from the GNU Scientific Library and R CRAN repository. Extended by the i-Class embedded analytics capabilities, TwinFin allows our partners and customers to push-down applications, functions and algorithms going well beyond standard set-based SQL, at scale with high performance, freeing them of the latency and sampling requirements demanded by off-board processing platforms for advanced analytics.

 

The Oracle DBM/Exadata performs the majority of the OLAP analytics in the central cluster (RAC) nodes, after traversing the "traffic roundabout". And apart from basic scoring functionality, virtually ALL of the advanced analytics are performed in the cluster nodes as well. Placing the predominance of processing in the central SMP cluster means that both the functionality and scale of the analytics are limited by the capacity and performance that the SMP cluster can provide - typically limited to the elements included in Oracle's own "Data Mining" package.

 

The DBM/Exadata’s requirement for shipping the data from the storage arrays to the central cluster for analytics is akin to backhauling full massive truckloads of materials from a mining site to pick out the gold at a central headquarters rather than sifting out the most important nuggets in parallel and sending only those valuable elements back in the case if TwinFin.

 

4) No-Tuning-Required Simplicity vs. a Complex Array of Knobs and Levers

For a long time, the simplicity of the Netezza data warehouse appliance has shone through most strongly in the extremely limited tuning requirements it imposes on administrators of the system, particularly as compared to Oracle-based systems. Simplifying the system management is core to Netezza’s “appliantization” of the data warehouse and analytics platform. Rather than managing a “coordinated collection” of technology assets, the system and database administrators of TwinFin interact with a single appliance and use the redundant Linux-based SMP host nodes as the interaction point for all activities. Everything from database configuration, data distribution, data mirroring, monitoring, software upgrade and day-to-day management are simplified (in the words of one TwinFin customer, “It’s Netezza-easy – it just works.”).

 

No indexing is necessary (or even supported) in TwinFin to achieve high performance. Just about the only requisite “tuning” of the system is the definition of the distribution key for spreading data across all the S-Blades – typically the primary keys of the tables. Even in the internal management structure of TwinFin, our system management has been configured to get the maximum performance from the commodity subsystems (blades, chassis, disk arrays and network) by connecting them in novel ways and then managing them at a system level, rather than at the subsystem or rack-level.

 

While it is true that Oracle has simplified some of the tuning knobs and levers in the DBM/Exadata, prospective customers should ask them if they really have moved into the domain of requiring only a small handful of tuning knobs & settings; or whether they still require, or more colloquially, “strongly suggest” the use of dozens or even hundreds of settings (depending upon the number of objects being maintained and optimized). How many dozens of IP addresses are needed to configure and manage the DBM/Exadata (TwinFin requires only two)? Oracle even have a special service to help DBM/Exadata customers migrate and tune their systems and databases for performance and some of their leading Performance Architects even talk about the requirement of using functions like the Oracle SQL Tuning Advisor as an inevitable fait accompli.

 

By Oracle’s own admission, the time-savings that customers can expect to achieve in managing and tuning the DBM/Exadata system in Oracle 11g r2 is only 26% less than in Oracle 11g. Contrast that with installation after installation of Netezza appliances where 100s of terabytes of data under management in a data warehouse(s) are being maintained by two or even less then one FTE, rather than a team of Oracle specialists. It all depends on one’s perspective and philosophy in building a real appliance for the data warehouse market. Where others may see the need to tune, partition, index and sub-index data sets for performance purposes as an inevitability, Netezza sees that same need as reason to enhance TwinFin’s capabilities in order to obviate it.

 

All of this really adds up quickly to a significant price-performance advantage for customers of TwinFin – and with our limited tuning and simplified operations, also translates into much more rapid time-to-value for Netezza’s customers, too. So that’s it – four simple fundamental differences that really set the TwinFin appliance apart from the DBM/Exadata. Agree? Disagree? Let me know what you’re thinking. And now, go over and have a look at today’s eBook release for the rest of the story.

4 Comments Permalink
0

     

The "Intelligent Economy" is much more than a trendy buzz phrase or the name of a business school seminar.


It's the new reality for enterprises today.


The Intelligent Economy is about fundamentally changing the way we view our businesses and our customers, about inviting change when we decide change is needed – but also about recognizing what we do that makes sense and valuing it accordingly.


A June, 2010 IDC Vendor Spotlight noted that the “demand to respond faster and with greater insight to ongoing internal and external events based on facts is increasing,” and that “data management and analytics challenges of the intelligent economy are likely to overwhelm organizations unprepared for the emerging changes.” (You can download the PDF of the IDC report here.)


Gut checks and intuition are still valuable when it comes to making management decisions – but actionable, on-demand information from analytics is shaping up to be a key differentiator when it comes to competing these days.


ENTERPRISES MAKE UP THE INTELLIGENT ECONOMY


The Intelligent Economy is made up of "Intelligent Enterprises" – companies and organizations that not only recognize the new reality – but are thriving because of it.


For these enterprises, data isn't viewed as a problem that must be managed and contained. For them, data is a gift that exposes hidden truths about their futures.


We know at Netezza – based on the real-life experiences of our customers – that data is a valuable, sustainable resource that can change the way you view your customers and run your business in fundamental ways - forever.

The ideas that stem from the analytics solutions we enable for our customers are much more than abstract formulas on a virtual chalkboard. They result in real, positive, lasting change. Our technology at Netezza becomes a source of inspiration, a catalyst to a new way of thinking about the way people interact with the world and with each other.


BUSINESS ANALYTICS COME OF AGE


The energy and enthusiasm we experienced at 2010 Enzee Universe this June on Boston's waterfront proved to me that we have reached a tipping point when it comes to accepting data analytics on a much deeper, broader level.

 

 

As I said in my keynote address, Netezza embraces a set of core principles that are simply hard for others to reproduce:

  • Your success as our customer is our top objective
  • We embrace the fact that we are part of an ecosystem of partners and technologies that provide you with a complete solution
  • Simplicity is the key to making analytics more accessible and a fundamental driver behind everything we do
  • We want to be a company that is easy to do business with and one that is viewed as a partner, not just a vendor
  • We value our customers’ input and integrate it into our thinking every day


Many of our customers are solving problems they could previously only DREAM of with a level of agility and simplicity that has allowed them to apply their energy to the business opportunity, freeing them from the complexities of the platform to focus on what really matters, the business solution.

This, in a nutshell, is what is enabling the Intelligent Economy to emerge: the enormous amounts of human time and energy previously required to simply process information can now go toward inspiration, learning, and improving.


Over the next few blog posts, I will be writing about the real-life experiences of some of the Intelligent Enterprises that make up this emerging Intelligent Economy – organizations that are using high-performance data analytics to make critical time-sensitive decisions, predict future events, and change they way they do business for the better, forever. Stay tuned.

0 Comments Permalink
0

In the past several projects, the issue of using views has consistently arisen, as to when to use, not to use and what to expect. Views are one of those mainstay workhorses that we love to hate and sometimes hate to love, but used correctly, can save a world of hurt and lost development time.

 

So we would ask the question - why use a view at all? Isn't the table definition good enough? And what of a synonym? Isn't this just as good?

 

Well, synonyms are handy for configuration management and invaluable for testing. For BI, however, they don't pass-thru the metadata of their underpinning relation's metadata for consumption by the BI tool, so this can be problematic. We also cannot refresh a synonym easily. It has to be dropped and then created in two operations, where view gives us the concurrency-protected operation of "create or replace view" and is muey bien. More on synonyms in another blog entry.

 

Views can easily reach across databases, giving us the ability to stand-up a consumption-point that contains one-part tables and one-part views without having to push data around (very handy for say, a reference database that we want as an on-demand resource of fresh information). I'm a big fan of setting up consumption-point databases so that a user comes to a pre-designated place, not the master repository, to fulfill their information needs. This decouples the user from the master repository and gives us enormous freedom in the ongoing enhancement of their user experience.Views are the vehicle towards this goal.

 

Views also let us do on-demand case/when conversions and typecasting that can be completely encapsulated from the consuming process.


And of course a really cool part about Netezza views - is that we can include as many columns as we want in its "select" clause, the view will not fetch them all, only the one's mentioned in the select that consumes the view - this is a win-win because otherwise it would fetch all of the columns and then drop the majority on the floor to deliver a few.

 

Views have the lightweight nature of a single SQL statement that can be easily installed, where a stored proc often contains multiple SQL statements. Both of these mechanisms serve to hide the logic from the BI tool. But think about this - would we use the stored proc as part of another join? Or would we expect to just select from the stored proc and consume an answer? The more complex the operation, the more we need to just select-and-consume, and take the burden off the BI tool to know more than it has to.

 

A pernicious part of integrating BI tools is just that - expecting that it will know all it needs to know to interact with the Netezza MPP. This is - as you may painfully discover - a false expectation. Case in point, we might have a very creative intersection table between two large fact tables, and we can formulate a query that will browse the information-we-want in mere seconds. Then we plug in our BI tool and ask it to manufacture a query to do the same thing, but it struggles. Now we have to make a call - do we deploy the BI tool in the hopes that later releases will resolve this, or do we install a view or stored procedure that adapt the BI tool to our data model, and then wait for the BI tool to get better in a later release? You see, we can always toss the adaptation when our BI tool gets better. But we cannot allow our user-experience to languish on the same terms. More on this in another essay.


So before I jump into a lot of other things we like about views, I'll address some of the above in their more malignant form.

 

I'll loosely divide views into two buckets - simple and complex. The simple view consumes a single table and may have columnar transforms on it. A complex view, simply put, has more than one table in the join logic.

 

A simple view cannot be easily misused, but a complex view can be misused so easily it will make the head spin on your best troubleshooter. For example, I cannot count the times I've seen a case where a master query joining on a view, which in turn joined on a view, which in turn joined on a view etc. How deep can you go? This is not the issue at all. The issue is in treating the view as though it is a reusable, inheritable object rather than a standalone select-and-consume capability. So where do we draw the line?

 

Transactional thinking - that is - the notion that we can install nested (inherited) views because they handle transaction-at-a-time anyhow and any given instance of them will have a negligible performance problem - is completely washed away when dealing with multi-billion-row scales on a Netezza platform. It's not a transactional platform, so each view potentially initiates a full table scan. Multiply these nested upon nested views and we have nested tables scans - sometimes several separate scans on the same table. Which is more efficient, to look at a multi-billion row table one, or multiple times?

 

One customer had a query that started running very slow one day. We went through a process of discovery to find out what had changed. Seems that a new version of an existing view had
been installed, and the bad query was consuming this view deep under the covers. The bad query and view  were both accessing one of the largest tables in the database, the bad query was now scanning the big table twice, taking a double-hit on the master query itself. Even worse, the changed view did not leverage the big table's zone maps or its distribution key. So a change in one place dramatically affected unchanged functionality of a master query.

 

Because we are embracing economies of extraordinary scale, dynamic objects have a propensity to lose performance integrity over time. What worked yesterday may not work today, so we have to tune it. Netezza is so efficient that this tuning necessity may not arise for years after the implementation. (In one case, four years afterward). By that time, the knowledge of the system's dependencies are not fresh on everyone's mind, so it is easy to make a spot-fix on the view and deploy it. In so doing, we may create a cascading effect for all the other places that consume the view and do so with the expectation of original behavior. In short, the latent nested view architecture is a minefield. We should not implement it because it creates trouble from day one, even though nobody has stepped on mine just yet.

 

At one customer site we had to sift through six levels of view logic to find the performance problem. The customer wanted to know what they should do to fix the problem, but "the problem" was in the overall inplementation and the nested views, not the one bad view, or for that matter, the recent performance symptoms of a minefield implementation.

 

Views can behave as traditional objects if they are single-table views or they leverage additional tables that are small and inconsequential to performance. Don't ever include a big-fat table in a view as part of a performance boosting strategy unless you can designate that the view is in fact a standalone entry point and not something that can arbitrarily participate in the JOIN clause of another master query. Why is this? Invariably we will forget the complexity of the view and then attempt to join it in another operation. For a BI tool, this could be highly problematic as well, because a view that was once simple could spontaneously go complex, and if it affects performance, we'll be pulling our hair out to find the problem through what reduces to a scavenger hunt, or worse, a submarine hunt.

 

Many BI tools simply choke on automatically forming a "complex" Netezza query because there is an implicit assumption of indexes via primary keys, and if these don't exist, the BI tool does the best it can, which in many cases is the least-common-denominator of a query structure. This this doesn't play well on the SPUs for large-scale queries. I cannot count how many times I've seen a convoluted query that we just de-engineered and simplified, and ran an order-of-magnitude faster than the one conjured by the BI tool, yet nothing the tool folks could do seemed to make the BI tool form it the same way. To the rescue: a view that did the right thing - and that was that.

 

What's that? Putting together a view diminishes the flexibilty of the query? Only marginally, and since we're dealing with billions of rows, we don't have much runway for "ultimate" flexibility anyhow. The larger the datasets, the more we need to make sure the queries are as efficiently formed as possible. And since this means as simply formed as possible, we're not talking about BI Tool query engineering, but query de-engineering.


To avoid pain and injury, don't treat all views the same. If we have a complex view, we should tune and designate it as standalone. No matter how much we like its results, it is better not to just arbitrarily include it in another join. the primary reason being - most views are not set up to regard a distribution. So when we include it with our other join, the resolution of distribution might take the form of of least-performing, lower-denominator. We don't want that.

 

One alternative, oddly, is to CTAS - execute such a view in context and insert its data into a temporary table, then use the temporary table in the master join. This affords us the option to (a) leverage the view's normally small output (b) preserve the distribution or (c) align distribution to the next operation (d) simplify the implementation. Of course, your BI tool may not support this, or may support it in an inefficient fashion. Most of the major BI tools will accommodate advanced scenarios, so get your product support rep on the wire and have a heart-to-heart.

 

Yet another alternative is to use the view like an in-line view, except in the where-clause in correlated sub-query. This can often take the form of a where-not-exists clause or the like and can also be very efficient.

 

Another alternative is to break apart the view's logic and assimilate it into the larger view so that all logic is preserved. But you'll be maintaining that logic in two places, right? Not necessarily. We have a lot of view DDL executables that do not directly spawn from a modeling tool. Several of those being in BASH script, which provides for parameterization of logic. If we put the logic into a parameter, then produce the views by including the parameterized logic, we will maintain the core logic in one place (script) but actually deploy two views that leverage it. This is essentially what happens under the covers with many object-oriented environments anyhow. Multiple objects will consume another class and deploy an instance that includes that class, so this approach embraces that inheritance pattern. Not in the dynamic run-time of the view, but in the view's initial DLL-level deployment.

 

MYLOGIC=$( cat <<!
a.limit1 between 50 and 60 and
a.limit2 between 1000 and 50000 and
a.tran_amt < 10000 and
b.employee_id <> 9999
!
)

 

view1="create view view1 as select col1, col2, col3 from mytable a where $MYLOGIC ;"


view2="create view view2 as select col1, col2, col4 from mytable a join yourtable b on a.id = b.id where a.col1 = b.col1 and $MYLOGIC ;"

 

If our modeling tool supports this capability as part of its functionality, and we should leverage it before simply bolting a view into a join. If our modeling tool does not support it, this scripted DDL scenario is easy enough to formulate and leverage without a lot of overhead. The objective: two views that both behave as optimized joins, rather than one view that behaves as a join-with-a-view.

 

Either way, there is a theme here, that simply including the complex view as part of another join's  logic - as though it was a table - is risky and can, even at the outset, offer up such bad performance as to be a non-starter. So a plain-vanilla practice should be to make the complex view behave in a standalone query-and-consume fashion by default. Make no assumptions that it is okay to arbitrarily include it in a larger query's join clause.

 

The further downside is that a de-facto join-with-a-view can work really well at the outset, but the scale of the data can catch up to the even the most robust of implementations, and wiring up the complex view dependencies creates a problem that will not scale, but will only become obvious over time (a minefield)

 

One group invoked a standard for view naming conventions. The simple views would have no prefix at all, so they would look like tables to the casual user. Fair game and all that. The complex views were labeled as v_<viewname> as a cue to a user or report builder: don't use it in the join of a larger query. You'd think that if there was an implicit rule to avoid using anything "v_" prefix that people would play nicely. But not so, since your reporting users may have come from a RDBMS background where it's perfectly okay to mix views into the master query. Awareness of the standard is one thing, but actually embracing it is another. We cannot protect our systems from people who either don't know the rules, don't understand them, or cannot map their experiences from an RDBMS to an MPP.

 

So a suggestion here would be to name the view in a manner that is a departure from common view nomenclature. Calling it an sp_(NAME) might draw the ire of your admins who want stored procs named for what they are, and not obfuscate their names. But if our views are not really common views, and have caveats on their usage, we need a safer naming convention, one that aligns with the goal we are trying to achieve - that of adapting the BI tool to the MPP. One group used a naming convention of "bi_", while another used "rpt_", and still another used the common acronym for their given BI tool. The point is to adopt a convention that is somewhat unconventional, so that those with conventional thinking are able to transform their thinking without finding themselves in a minefield.

 

Nothing is worse than overlooking a minefield - it's a scary view - a view to a kill.

0 Comments Permalink
0

I mentioned in my previous post that Netezza is excited about our partnership with Cloudera and Hadoop because we’ve already seen some of our customers benefit from the synergy of Hadoop and Netezza TwinFin™ technologies working together.

 

As I noted, these types of strategies play to the strengths of both technologies and roughly break down into two categories: 1) the use of a Hadoop Cluster for data ingestion, and 2) using a Hadoop Cluster for long-term data retention, which I’m addressing today.

 

Netezza TwinFin with a Hadoop Cluster Used for Queryable Archive Analytics

The second pattern we have seen customers deploy is one in which the Hadoop Cluster is used for long-term data retention, or as a “queryable archive”. Here one could think of Hadoop as a complementary analytic extension of the Netezza TwinFin when there is far less premium placed on low-latency or high-performance. In addition to the weblog and unstructured data analysis discussed in Pattern 1, the queryable archive could also retain long-term copies of structured data that had previously been loaded into the high-performance TwinFin appliance.

 

Hadoop-NZ 3.jpg

Hadoop Cluster Used for Queryable Archive

 

 

With a mix of structured, semi-structured and unstructured data loaded across the two complementary systems, customers can alter the level of granularity and data retention periods across each and typically use TwinFin for processing “hot” data and the Hadoop Cluster for processing “cool” or “cold” data, perhaps with specialized analytics. A deployment of this pattern could look like the following diagram:

 

Hadoop-NZ Arch 3.jpg

 

 

Readers should view this pair of posts as a “point-in-time” look at the market. Our customers continue to innovate and make use of the complementary strengths of TwinFin and Hadoop. And Netezza will continue to innovate both inside the appliance – adding performance, scale, workload management capabilities and especially with the advanced analytics of i-Class, through partnerships like the one announced with Cloudera a week ago, and through expansion of our platform, software and virtualization capabilities beyond the TwinFin and Skimmer™ appliances. Those innovations should help alter and/or enhance some of the deployment directions discussed here.

 

Now, as I said at the outset of these two posts, I’d like to hear from you on your Netezza & Hadoop co-existence deployment and/or compatibility wish-list ideas. What would you like to see?

0 Comments Permalink
0

Two things before I begin:

  • I’ll begin this posting with a call for inputs. Below I will list a few of the most common Hadoop/Netezza co-existence deployment patterns we have seen to date. But I would like to hear from others. As you see the continuing deployment of Hadoop in the enterprise and as the Second Wave of TwinFin™ comes on with the advanced analytics capabilities of i-Class, how do you see the evolving deployment patterns happening in your environment?

  • A special hat-tip to Krishnan Parasuraman, Netezza’s Chief Architect for our Digital Media group, for his excellent help in aiding and abetting this post! I have used his guidance gratefully and (with his permission) stolen freely from some of his inputs.

 

You may have noticed a partnership announcement made by Cloudera and Netezza late last week. Together with Cloudera, Netezza will open up data movement and transformation between Cloudera’s Distribution for Hadoop and the Netezza family of appliances applications and data flows for integration of the two systems. We expect that our partnership with Cloudera, together with the Hadoop support in Netezza’s i-Class™ set of advanced analytics capabilities that are included as part of the upcoming release 6.0 software release, will lead to some very innovative and expansive applications for our customers and for both companies.

 

Even today, Netezza customers are doing some very interesting things with deployment of Hadoop and our TwinFin data warehouse appliance. Far from being the “Hadoop v. SQL” battle that some people might like to make the current market out to be, we have instead noticed a growing number of “co-existence” deployment strategies and design patterns already at work with our customers – particularly among customers in the “Digital Media” vertical market.

 

These types of strategies can play to the strengths of both technologies and roughly break down into two categories: 1) the use of a Hadoop Cluster for data ingestion, which I’ll write about in further detail today; and 2) using a Hadoop Cluster for long-term data retention, or as a “queryable archive,” for which I’ll go into further detail in a post later this week.

 

Using a Hadoop Cluster for Raw Data Ingestion

The use of a Hadoop Cluster as the engine for data ingestion is the most common “co-existence” pattern we see in our customers’ mutual deployments of Hadoop and Netezza. The deployment pattern typically arises when the customer has hit specific performance and processing throughput scalability limitations with their existing Data Integration or ETL implementation.

 

Raw weblog data is the primary data source for most Digital Media analytics and reporting requirements. Weblogs are data rich (e.g., page views, impressions, click-throughs and demographics collected from applications servers). They are typically semi-structured and collected and stored in flat files.

 

There are some critical facts about weblogs that present real performance challenges in processing them:

  • sheer volume: millions of rows of weblog data collected throughout the day and loaded daily into the data warehouse;
  • complex query processing: parsing and decoding encoded character strings requires text processing, pattern matching, tokenizing type capabilities within the ETL process
  • non-conformed dimensions: collecting page views or impression data defined and represented differently by various systems makes fitting them into conformed dimensions is another very common data ingestion & processing challenge.

 

There are two common variants of this pattern – dealing with semi-structured (e.g., weblogs) and unstructured (e.g., text) data and often customers will have versions of both variants in operation simultaneously.

 

Hadoop-NZ 2.png

Semi-structured data ingest via Hadoop

 

Semi-structured data is parsed (and possibly aggregated as well) in the Hadoop Cluster and then loaded into a TwinFin where the performance and workload scaling of the appliance is important for deeper analysis, higher throughput and faster reporting.

 

 

Hadoop-NZ 1.jpg

Unstructured data ingest via Hadoop

 

Unstructured data in this pattern is contextualized (classified, mined, keyworded and indexed) in Hadoop and then moved into a Netezza TwinFin appliance for the low-latency, high-performance analytics used to drive business decisions.

 

 

A Hadoop Cluster provides a scalable ingestion mechanism that is well suited for addressing the challenges described above. The Cluster can be incrementally scaled to handle ingesting the massive volumes of weblog data and it can support text processing and complex data processing through programming languages such as Java or Python. [Note that with the coming i-Class set of analytics functionality, the programmability and some of the complex data processing may also be possible on the TwinFin, depending on a customer’s applications needs or preference.]

 

Following the data ingest steps, processed weblog information is brought into TwinFin as atomic event information or as summarized tables, depending on the size of the appliance and analytic maturity & scale of the organization where it is deployed. A typical deployment might look like the following diagram:

Hadoop-NZ Arch 1.jpg

 

 

An alternate, far less common, deployment design of the above co-existence pattern is used by some of our customers. That is the use of an external elastic MapReduce cloud (such as the Amazon Cloud) for the data ingestion purposes.

 

In cases where the customer may have its application servers in the Amazon’s EC2 cluster, they may also choose to use Amazon’s S3 web services for retaining weblog data. In that case, Amazon would provide the elastic MapReduce infrastructure for the data ingest process into the TwinFin appliance. This alternative deployment scenario would look something like the following:

Hadoop-NZ Arch 2.jpg

 

 

The bottom line is that the different strengths of TwinFin and Hadoop lend themselves to complementary deployments – and some of our customers have already discovered innovative ways to leverage them together to maximize the value of both their investments.

 

In my next post, I’ll discuss the second pattern we’re noticing: one in which Netezza customers are using the Hadoop Cluster for long-term data retention.

0 Comments Permalink
1

I wasn’t planning to blog about EMC’s acquisition of Greenplum since Phil Francisco has commented here and many others, more well qualified than me, have had their say (eg here), but it did occur to me that one point this illustrates is how data warehousing got interesting again after ten years as a bit player in the drama of information technology. Suddenly, led i have to say by Netezza back in 2003, a whole generation of disruptive innovators have entered what was a stagnant market of established players and are redefining the segment. Richard Hackathorn said as much at EnZee Universe.

And it's what i had that in mind last week talking with someone from one of Greenplum’s software-only competitors. His take was that the choice was data warehouse machine or software only on a self-assembly hardware configuration (sounds like a grid from IKEA – i’d like to see the allen key in that kit). And Greenplum had decided warehouse machine was the way to go. Of course i wouldn’t be a good Netezza corporate citizen if i didn’t observe that there’s two classes of data warehouse machine: the true appliance (Netezza, out-of-the-box) and the customized data warehouse machine (either vendor-assembled hardware configuration or workload-specific tuned database on commodity hardware, or both).

That doesn’t alter what i took away from the conversation, which was that Greenplum tried the software-only route and then plumped for the machine option. Of course they may well have been made an offer they couldn’t sensibly refuse (in the original sense not the horse’s-head-in-the-bed sense). If so, i guess it’s a case of EMC not being content to be the optional storage component in a configured data warehouse machine and indulging in a bit of supply chain integration. These are interesting times for vendors, customers and, as ever in such situations, analysts and consultants.

1 Comments Permalink
0

I took a little time out yesterday to think about the implications of a possible huge upsurge in mobile BI apps. You don’t have to share Michael Saylor’s unwavering beliefthat iPad and iPhone will be the dominant deliveryplatform in order to acknowledge that there is a real opportunity here. The use cases that I have heard talked about seem to fall into two distinct classes. The first type would be role-based dashboards, for example top ten best movers/worst movers/best aggregate margin etc for a retail store manager. These might be database-intensive queries, but they would generally be cached because they would be slow moving. The second type of query would be ad-hoc requests for specific data. For example ‘what’s the inventory for product xxx?’ These could be run very frequently, with different parameters, but would not be database intensive. And both kinds of queries would be pre-baked into the app that the user downloads to their mobile device. All of this seems eminently doable for MSTR using their new mobile app development toolset and their intelligent server run time.

And the same might well be true for any other comparable app development platform. The critical success factor here is not the deployment platform, though some folks may get burned there depending on how the fight for dominance plays out. The critical success factor is fast response time. A BI specialist running operational reports or complex predictive analytics across the whole dataset may be happy with a multi-second or even minutes response time. But a mobile user has to have the answer quickly or it’s not worth having. From a Netezza perspective this is all music to my ears. Any kind of app that needs fast access to a mass of data is going to need the fastest database they can put at the back of it. We have already partnered with Microstrategy and Quantisense to deliver the Retail Analytics Appliance. It will be interesting to see how the market for mobile BI apps develops. I think it’s clear that retail is a fertile segment, but not the only one.

As you do, if you’re a partner-vendor exhibitor at a conference, you bring along what i’ve always called a twomm (to rhyme with from) - total waste of marketing money, otherwise known as the stand giveaway. Of course they are not a waste of money they represent a unique opportunity to build lasting positive brand identification, blah, blah, blah. But it being world cup semi-final week and with it being my choice, we have vuvuzelas as our giveaway. And give ‘em away we have. I’m not even sure i’ll have one to take back to my Dutch footballing friend who i was texting with as I watched their game (1-0 pleased : 1-1 worried: 2-1 confident:3-1 triumphant: 3-2 clenching) against Uruguay at the conference beach party (thanks Microstrategy – good party). Next to me throughout the second half was a tall Dutch (from appearance, mien and allegiance even if she had said nothing) woman doing a great job with a Netezza vuvuzela. Thank you madam for promoting our brand, and congratulations on a (just about) deserved victory. I’m hoping for more branding opportunities tonight. Not sure who i want to win: hugely talented Spain or unexpectedly fluent and uninhibited Germany. Or should i say who would i prefer to lose: envy-inducing talented Spain or England-crushing Germany.

Anyone get that somewhat torturous music reference in the title?

0 Comments Permalink
1

News broke on Tuesday that EMC plans to acquire Greenplum to focus on data warehousing and analytics on “big data”. The idea is that by doing so, EMC is officially throwing its hat into the competitive ring for the ‘Data Warehouse Appliance’ (DWA) market – something of a defensive mechanism now that virtually all of the major data warehouse vendors are now selling their own versions of a DWA – and consequently greatly reducing sales pull-through of EMC storage for data warehouse deployments.

Some referred to the merger as “
a good fit for a storage vendor with appliance-y ideas” and others hailed it as follows, “the market has shifted as of late moving toward integrated appliances and this move gives EMC a very important arrow in its quiver” and labeled Greenplum as a purveyor of “very high performance database systems”.

One can also reasonably assume that this acquisition not only is intended to shore up a product offering weakness, but that it is also destined for affiliation with EMC’s other major initiative announced earlier this year – the
Acadia Virtual Computing Environment (VCE) Joint Venture with Cisco Systems and headed up by Michael Capellas. The Acadia JV includes EMC’s storage and its VMWare virtualization software as well as Cisco Systems’ compute nodes and networking. VCE is built on the concept of modular building blocks, called vblocks that marry computing horsepower to storage capacity. All that’s missing from that story is a data warehouse DBMS to make it a full-on data warehouse appliance, right?

There are two big problems with these assumptions…


Performance: For all the discussion about “scale” and  “big data” in the EMC announcement, there is no mention of how either party can address the real issues that mainstream enterprises face every single day with their data warehouse systems – how to get maximum performance out of a complex, highly concurrent operational environment where hundreds if not thousands of users are banging away on the system, night and day.

  • The fact is that the actual Greenplum target market has clearly NOT been one that focused on high-performance analytics over the past several years. Instead, the few wins publicly announced by the company have been for very high capacity, limited compute platforms – applications more commonly referred to as “queryable archive”.
  • Curt Monash today again mentioned Greenplum’s lack of support for the “high-concurrency” requirements of a mainstream data warehouse.
  • This looks much more like adding a very basic set of storage-centric data warehousing capabilities in a move to find a broader channel for EMC’s traditional storage products rather than any strategic move into the world of high performance data analytics. Further to this point, neither company has done much of anything to address a very strong trend in the mainstream data warehouse market – the marriage of advanced, predictive analytics into the busy data warehouse systems.
  • David Vellante confirmed that to be successful the EMC/Greenplum marriage will need to yield, “optimized sytems[sic]; smokin’ fast performance; reference architectures; scale;” and “federation capabilities; not just big honking systems.” We couldn’t agree more but one can’t help but notice that neither Greenplum nor EMC have brought any of those characteristics to market for data warehousing to date.


Appliances: Since the acquisition is fairly transparent in its defense against moves by the likes of Oracle, Teradata and IBM (as well as Netezza seven years ago) to the appliance model, it’s hard to see how either EMC or Greenplum are effectively equipped now to do battle against those established players.

  • EMC have never really “sold” data warehousing to anyone previously and Greenplum have nearly prided themselves in going after “Greenfield” high capacity applications rather than head-to-head competition vs. established players. And one need look no further than the limited market penetration of H-P’s NeoView to understand that it takes more than simply deep pockets to succeed in the data warehousing market.
  • Greenplum is not a purveyor of “integrated appliances” and at best, they can hope to infuse in EMC the ability to make their joint product offering a little more of an “appliance-y idea” (hat tip to Dr. Monash for coining the term) to the market. Instead, Greenplum have fashioned themselves over the past several years as a software only solution.
  • Assume that the Acadia VCE and “vblock” application is a big piece of this strategy. Neither Cisco nor EMC would claim that their servers, networking or storage arrays offer the lowest price-per-bit or price-per-performance alternative in the market. So one needs to think about what that means in terms of the price-performance competitiveness of this new “appliance-y” joint product.


In short, Greenplum joins the pantheon of “interesting” acquisitions for EMC as it will certainly stir some news cycles and drive some analysts and bloggers to create “fresh, new” content; but it’s not really something that I think will register on the Richter scale of customer market share.

1 Comments Permalink
0

Up at ten to five to get an early flight to Nice to attend the MIcrostrategy World conference in Cannes. Very interesting keynote presentation by MSTR CEO Michael Saylor at the launch of their iPad/iPhone mobile BI apps platform. Saylor was passionate about iPad; he regards it as a game changer; the biggest thing since the .COM boom, even bigger he suggested. I tweeted it and got a 2 to 1 vote in favour of iPad vs ‘it’s a fad’ (no, literally 2 to 1). I think I can regard that as a statistically valid sample. So bet the company Michael, you have the technorati with you all the way. And we’ll just call the rest the troglodati. The more I play with an iPad the more I feel myself agreeing with him. But later I came across an intriguing argument for why IPad BI apps will succeed. Think of all those managers who’ve bought or want to buy iPads; the perfect justification is ‘so I can be connected to my KPIs and access my critical performance reports wherever I am’. That’s going to get the expense approved, no brainer; well better than ’but it’s so cool, and everyone else has one’.

Then I came across another example at dinner. There was a boys-and-their-toys moment during the hiatus before dessert and everyone is slapping his phone on the table and arguing the case. Most of which was legitimate geekery but it reached its climax when someone observed that the key differentiator for his iPhone4 over his colleague’s Android Samsung with touchscreen (Omnia 2?) was ‘yeah, but it’s Apple’. And his justification was ‘you have similar job offers from two companies and one gives you a Peugeot company car and the other offers you an Audi. Which one are you going to pick?’ The quality, function, form and whatever can all be overridden by brand. And Apple is certainly the brand of the moment.

But the highlight of the dinner was discovering a Frenchman whose favourite film is ‘Holy Grail’. You haven’t laughed quite enough at the French taunting scene, until you’ve heard a genuine Frenchman do it. It’s still making me smile as I draft this later.

In fact i was drafting that much too later. For a bloke who started by complaining about how early he had to get up, I made a poor job of catching up on my beauty sleep. I’ve been on the booth in the partner exhibition hall from early this morning. I made it through my morning shift without falling asleep on the shoulder of a prospect while I listened to his earnest questions about query times for cartesian product joins. Just as well, I might have snored and dribbled on his suit and that’s just plain unprofessional (you can see I have high standards).

0 Comments Permalink
0

Whew! The Enzee Universe this year was quite an experience.

 

I would like to offer my sincerest thanks to all of you who attended the Best Practices session held in marathon-form on Monday before the keynote. Over 300 people signed up, and many of you arrived the evening before, and at the end of the session, you were still there!

 

Afterwards we took a checkpoint and then added more material to the powerpoint presentations that we used during the sessions, and these will be added to the content of the Enzee Universe downloads for those who attended.


Some of you also asked me about the music selections we played at the intro and during the breaks. These were selected in terms of "Your Theme Songs", because some of them were from superhero movies, and some from action-adventure flicks. Here they are, in no particular order, the tune, origin and reason for selection:

 

Theme from "The Incredibles" - because we come from a family (The Enzee Universe) of mult-talented superstars

Theme from "Batman/The Dark Knight" - because sometimes we have to work in a thankless role (however personally rewarding, with high-tech toys)

Theme from "Superman" - because to competitors, Netezza is like Kryptonite, and to the rest of us, it solves World Problems and makes us look good without having to wear our underwear on the outside

Theme from "Spirit, Stallion of the Cimarron" - surging music for those who entered the frontier on a Mustang

Theme from "Surf's Up" - probably enough said, here

Theme from "Mission Impossible" - a congenially offered rebuttal to those naysayers who say it can't be done

Theme from "James Bond" - because he, like many of you, is an MTBA - That's Multi-Talented Bad A**

Herbie Hancock - Rockit - because that's what you do

 

I would also like to thank Netezza for the opportunity to share these ideas, many of which I have gathered over the course of my Netezza derring-do from people just like you, so some of the information is what-is-practiced in the field, and some of it is idea-works that we have re-synthesized into practices that seem work well as a sort of "adaptive composite". The objective of course is to share with you what others are doing, to enrich your base of ideas, but are certainly not hard-and-fast rules. The Netezza appliance is one that unlocks creativity, harnessing it for the Good of All Mankind. So guidelines and practices give us more critical mass to solve problems.

 

Likewise I would like to thank Netezza for the Enzee Community Voice Award presented to me on Tuesday night at the Gala, in recognition for being such a vocal supporter. But my words then apply as now "The people and the product create a synergy that's like electric current. I love interacting with the Enzee Community, and being a part of it".

 

I also noted that a larger number of independent consultant/contractors were present on this go-round. In the best practices sessions, there are a wide range of professionals, from those who are Netezza customers, to consultants working for firms, independent consultants, analysts and the like. The Enzee Universe has various video screens constantly running slow-motion surfing videos in keeping with the TwinFin Theme. One day you might be lookin' at that guy on the wave, thinkin' about the Twinfin and wondering if you're on the wave, or just watching it pass.

 

As with all professions, you might be in the zone where your project is just ending, or is about to, and you're wondering about the next great thing. And as you know, I'm always on the hunt for bright architects and engineers, especially in this economy, so if any of you independent types are looking for an opportunity, give me a shout. I also extend the invitation to anyone else who is reading, with the qualified apology that I am not lurking at the doorways to steal away your company's valuable resources. But I have seen in the past that some bright folks find themselves tapping a pencil on their desk, coming down from the exhilaration of a Netezza 'experience" and wishing for more. I can say - the work is out there. I'm often in contact with people who need someone just like you - hooked on the technology. Hey, who isn't?

 

Finally I offer a simple salutation to everyone who "gathered round the grill" this past week, sampled the wares, wishes and whatcha-ma-callits of the various vendors, trainers and speakers, and came away enriched and enabled to dream a little stronger, solve a little simpler, and crush those waves with the shredding confidence of parallel power. So I'll either see you in your natural habitat, interact with you here, or catch up with you in person when the Enzee Universe cranks up another adventure.

0 Comments Permalink
3

Netezza Migrator.jpg

It may have been the result of a misunderstanding or a comment heard out of context. But whatever the background for the commentary, let me simply state that Netezza is completely committed to the success of the Netezza Migrator and all the other Netezza products and functionality launched at Enzee Universe 2010 this past week. Migrator eliminates a potential barrier to TwinFin™ adoption (i.e., migration costs) and logically should lead to easier acceptance and broader system sales for Netezza. Furthermore, our partnership with EnterpriseDB at both the corporate and technical levels has been and remains extremely solid and strong.

As I stated in the
announcement of the product, “The Netezza Migrator product allows organizations to make data warehouse migration decisions independent of proprietary software lock-in. Organizations using data integration and BI applications with embedded Oracle-proprietary database constructs, interfaces and utilities can now more easily manage their migration from Oracle to a TwinFin appliance. The Netezza Migrator will allow our customers to achieve the performance, scale and cost advantages of their TwinFin systems while maintaining their prior investment in proprietary software.” The Netezza Migrator is specifically designed to reduce the time, complexity and costs required of our customers to move their IT applications to the Netezza TwinFin platform.

With Migrator, Netezza’s customers will be able to extract themselves from the dreaded “Oracle lock-in” of functions and procedures written using Oracle-proprietary techniques and they can decide which of their applications to migrate directly to Netezza and just when, at their own pace. Its capabilities go well beyond the extremely limited capabilities provided by Oracle’s own ‘Database Gateway for ODBC’. Migrator provides an Oracle compatible wrapper around Netezza that is optimized in ways that Oracle could never hope, nor deign, to provide with its "Heterogeneous Services" functionality: including support for Netezza syntax pushdown, high speed API, and Netezza user defined functions.

Migrator makes it even easier for Netezza’s customers to move all of their data warehouse from Oracle to Netezza. In short, this is something we feel is extremely valuable for Netezza and particularly “liberating” for our customers.

3 Comments Permalink
0

When he wasn’t being distracted by the footie (today I use a Brit term – you can work out what I’m talking about) and amongst a lot of other themes, Curt Monash made a few observations on privacy in his EnZee Universe presentation. I find this topic scary but fascinating. As I wrote previously, there’s some fantastic opportunities to use customer data to deliver better, more customized service. My feeling is that we’ll willingly make the trade of our data, and by implication our privacy, for the advantage of being better served, provided we are better served. Although we might, each of us, respond differently to our grocery store knowing more about what we want to buy this morning than we do ourselves. We saw what happened at Facebook when they overstepped the mark on re-purposing our data, but I feel there is a subtle difference between on the one hand re-purposing data gathered in an intentional relationship to improve that relationship and on the other selling it to third parties to create new uninvited relationships. But from the reactions I saw of my own non-tech friends in that instance at least part of the resentment was the shock at how much privacy they had inadvertently traded for the value they get from the site.

There’s a lesson for us all to learn about the need for pro-active transparency about data, privacy and re-use, certainly with my generation. Maybe generation-Y just don’t care.

I’m drafting this in breaks between sessions of the Netezza Advisory Council that has been running since the conference ended. It’s a forum for interested and innovative customers with Netezza engineering, product management and product marketing management (not sure how I snuck in). The objective is to help Netezza prioritize development by taking input from customer pain points and use cases and from seeking customer feedback to research projects that are maturing in the NZLabs. In the course of the discussion we looked at some very exciting new technologies, but more of them another day. It’s time for me to head for the airport. Knackered and keen to get home.

Hi to anyone i met at the conference. Especially the guys who wanted to get revenge on me for sticking a mic under their noses when i was doing my roving-reporter impersonation by sticking a phone video camera in may face late at night in a pub. Cheers guys. I hope you got some lovely footage. I doubt it.

0 Comments Permalink
0

In Jim Baum’s keynote at the EnZee Universe conference  Intelligent Economy was central and it featured elsewhere in the conference. Not always by that name, but with the same essential characteristics – a mass of data and imaginative use of that data beyond the core application it was gathered for. I’ve written before about the analogy with human intelligence - that babies have all the data but they haven’t developed the capability to deal with it intelligently. With us humans (and I’m assuming we have that in common, no offence to my more exotic readers if it’s a false assumption) we gradually learn to do that sort-of automatically until we reach the rational point where we can choose to learn more skills, acquire more data. Organizations entering the nascent Intelligent Economy are already rational (ok, stay with me, i know there are exceptions) and can choose. That was Jim’s final slide on Monday, “think big”. The organizations that choose to exploit the data they have more fully, will have an advantage. For example, retailers who have big loyalty card programs started them primarily to buy loyalty (there’s a clue in the title) but increasingly they, unlike their competitors with less well-developed loyalty schemes, can now do market basket analysis over time for an identified individual customer – genuine micro-segments of one! And that's Jim’s real message, echoed elsewhere, is what do you do next? What new capability do you choose to acquire? What further data do you choose to collect?

Sticking with my retail example, is the next step in-cart technology for the customer to insert their loyalty card into so it can display their usual shopping list? Or custom suggestions? Or shortest route through the store for the list? Or automatic scanning and tick-off of each item as it drops into the cart? Stephen Baker’s book is packed with examples of mind-boggling, but possible ideas like this.

One of the threads that Stephen Baker and others have emphasized is the importance of the big-brained analysts who will be needed to build the models that enable the complex analytics that drive some of these scenarios. But it occurred to me that the limited supply of such mathematicians would be a barrier to progress. A bit like the early assessment of telephone adoption - that if everyone in the US had a phone then one third of the population would be working as switchboard operators (Is that an urban myth? I couldn’t track it down. Let me know if you can source it). That didn’t happen and i suspect the parallel scenario won’t happen; apart from anything else, quantitative analytics is a bit more intellectually demanding than operating a manual switchboard. In the BBBT forum on Tuesday Richard Hackathorn suggested one way to achieve the necessary scalability will be the development and productization of industry-specialized models. These might then be operated, combined, incorporated into business processes. I’m looking forward to the technology advances, Richard’s suggestion and/or others, that will create the breakthrough for mass adoption of deep analytics. But meanwhile the existing technologies and opportunities provide ample scope for the next wave.

And yes i know i'm a Brit using an American English word in my title, but i suspect i have more readers for whom Diaper is the right word.

0 Comments Permalink
0

Curry was up to scratch last night, definitely the best I’ve eaten in the US, but I spent most of my time at the event with a microphone in my hand and a cameraman with a big, professional-looking shoulder-cam right beside me while I interviewed customers and partners about the conference. If they weren’t just being polite, I’d say the consensus is it’s the best EnZee Universe yet. It’s certainly the biggest.

As well as the interviewing I did last evening, we’ve had a number of people wandering around the conference with flip video cameras in their hands. I’m one of them so I have been trying to do it justice by recording clips in the talks I have been at and sound-bites from Netezza technologists. Today I have been following up the exercise i kicked off yesterday. I toured the sponsor stands in the partner pavilion asking them if they would record a 30 second elevator pitch to camera – to go on the community web-site and to show at the conference. I offered a prize for the best one and I said I’d be back today to record them. So this morning i have been discovering just how wide a range of reactions a little hand-held video camera can have on folks.

Basically my modus operandi was to walk onto the stand just say “I’d like to shoot a 30 second elevator pitch to camera for our conference website, do you want to do it now, or shall i come back later?” Only one person said ”Ok, let’s go”, plenty of people said “Can you give me  a few minutes to think what I want to say?” several said “We need to get so-and-so, she/he is the best person to do it”. All of which struck me as pretty reasonable reactions. And i predicted the “let’s go” guy just by his appearance – a frustrated actor in a marketing job i suspect (and no, it wasn’t my alter ego, though maybe it could have been).

One guy on a small booth did turn me down. He said his CEO had left the conference early and he was a techie who didn’t feel confident to do the pitch. Well fair enough, but what really surprised me was the reaction on one of the bigger stands. They delayed while they conferred with the other staff they had at the conference, but in the end they said thanks but no thanks. It just shows that people who can be great one-on-one don’t all have the thespian gene when it comes to one of those scary little lenses.

Off to capture some more mischievous footage and to some more serious meetings.  I need more time, which i don’t have right now, to write up some of the surprising things i learned in the keynotes on Monday and a panel on analytics this morning. That should be coming this evening or tomorrowI

0 Comments Permalink
0

Beer and Curry in An Innocent Abroad

Posted by dai clegg Jun 21, 2010

Well this is a baptism by fire for me. When i read my schedule at the weekend i assumed that the 7am /10pm start/finish was just the conference activities from registration desk opening to the evening’s entertainment concluding. I hadn’t realised i would actually be on the go for the whole 15 hours. Six hours in and so far i’m lasting the pace.

I don’t know how many people at the conference know about the Netezza UK tradition of Beer and Curry evenings. And i don’t know for sure that’s what inspired the curry banquet tonight but i suspect it did, so for those who don’t know the background let me elucidate...

It’s a tradition that started some years ago, and basically ‘beer and curry' is just an informal and, we hope, convivial get together of Netezza staff, customers and prospective customers for beer and curry. It’s as likely to be wine nowadays and it will be a pretty classy curry (the last one was here) but we remain true to the spirit of informality. When i first met the idea i was amazed that Netezza just lets prospective customers talk to existing customers freely and without intercession from staff. I went to a beer and curry evening before i joined Netezza and it just confirmed i’d made the right decision to escape the DeathStar for a company so sure of its technology that it doesn’t need an army of marketing and sales staff mediating any interaction between customers and prospective customers.

I guess tonight is the same intention writ large and i’m looking forward to it. I’ve eaten curry in a lot of places, including Heston Blumenthal’s favourite curry house, but i’m expecting this to be a bit special.

I'm trying to think of analogies from consumer sales to this idea of customers and prospective customers mixing freely. In retail the stores are all next to each other and they are competing on every sale to build loyalty. Maybe a better example is the insurance industry. I bet not many insurance companies would host an unmediated forum of their customers to direct you to when you requested a quote for new business. Did i just invent Which for the blogosphere? Or is it already out there and i didn't notice?

0 Comments Permalink
0

At EnZee Universe this week in Boston, Netezza’s user conference. My first one, of course, so a great opportunity to learn directly from the customers what they are doing and what they want. And i have a challenging role to play. I get to be a flip holder, one of a small group of Netezza staffers who will be wandering the conference with flip video cameras in hand. I’ll be shooting clips from presentations, and any attendees who will talk to me. We’re going to put the clips up on the conference website (attendees only access). And i’ll be in the partner pavilion looking to shoot the slickest, most diverting elevator pitches to upload.

And since i have a very long history of data modelling i’m going to keep an eye open for any presentations on the topic. In Netezza we’re pretty agnostic about logical data modelling; we only get concerned about the physical database design, but i have heard stories about physical warehouse designs that have been optimized for their original target database and continued to evolve over the lifetime of the implementation, By the time we see them they are a long way from the original TNF model. I wrote a little while ago about this and why it’s no barrier to a successful Netezza implementation, but what i haven’t tracked down yet is the best practise for building a Netezza warehouse from scratch. Any pointers/resources/suggestions?

I’m going to try to post every day from the conference, so i hope that flip will help me gather material. If you want to help out with your own Netezza anecdote, i’ll be the slap-head Brit in the red shoes. And a glass of champagne to the first person who correctly tells me who made my shoes. And another for who tells me why the title.

0 Comments Permalink
0

I’m at a telecoms conference this week, and in previous years i’m told the talk here had all been of networks and switches and bandwidths and copper. This year it’s all about content and customers and services and customization. Very interesting how the industry is moving from building infrastructure & acquiring market share to adding value for customers and delivering content.

The new watchword 'is the customer data is king'. I talked to an infrastructure provider who is planning to collect data on the content they deliver on behalf of customers and then sell access to that data back to advertisers. We talked about an analytics-in-the-cloud type service for the advertisers. And another innovative mobile provider was very interested in the churn management application one of our partner has, although what she really wants is predictive analytics that tell her which subscribers to call, because they might become dissatisfied. The example was subscribers who live near the edge of a cell and get higher than normal dropped calls. Does that need spatial data? Not sure, but even if it does it’s not massively complex. I suggested that combinations of call history to dynamically identify ‘friends and family’ of very recent defectors might be interesting and we talked about a few other possibilities. I don’t think she realized it was feasible to deliver demanding analyes of huge data volumes quickly enough to support this kind of agility in the business. More than one person I spoke to today appreciated that if they didn’t push the boundaries with how they get value from their data, their competitors will.

Of course there’s a big advantage to Netezza (with our newly enriched in-database analytics) for customers to identify more and more complex ways to analyse more and more data. But that doesn’t mean I would steer the conversations in that direction. No sirree, not me. (where’s that Pinocchio emoticon when you need it).

I had to get to the conference at 7am to set up the stand. About 10:30 the previous night i got a call from the sales executive who was supposed to be there to help me to tell me she’d be delayed. I’m still a newbie so this is my first time building and manning the exhibitor’s stand. Good job there were instructions in the case. That way I could ignore them and struggle – well I am a man. We can’t be following instructions until all other possibilities have been exhausted.

And my predictive analytics tell me my sales executive colleague – who incidentally worked her (insert your own inappropriate simile here) all day – will have another urgent appointment come tear-down time tomorrow. We’ll see.

PS. Top conference catering – courtesy of the Dorchester. We got the same pastries at afternoon break as the dowagers taking tea next door. Nice.

0 Comments Permalink
0
"I cannot imagine life without Netezza." from a tweet by "noogle" (Twitter, 11 May 2010)

 

800px-Shibuya_night.jpg

[Photo credit: 2006 photo of “the scramble” intersection in the Shibuya district of Tokyo, courtesy of "Bantosh" and Wikipedia]

 

Late in May, members of my team and I were in Tokyo's ultra-bustling Shibuya district for a few days on our "worldwide whirlwind training tour" with the global field sales teams regarding the details of the TwinFin i-Class product offering. The late-night scene, hairstyles and outfits there border on the outrageously-hip. There are high-def billboards, electronic gadgets, and of course the bright lights of retailers, bars, clubs and restaurants all through Shibuya. Advances in high technology are virtually 2nd nature to the people there. So with that as the backdrop, imagine the surprise of hearing over a beer or two (see earlier reference to bars & clubs in Shibuya) that a customer "could not live without Netezza".

 

We're proud of our highly referenceable customer base at Netezza and our "easy to do business with" relationships with our customers and partners. In my six years with the company I've met a pretty fair number of really enthusiastic customers including people who held "welcome" parties for their Netezza systems, use "Netezza" as a verb ("Did you Netezza that data?") or even an adverb ("It's Netezza easy."). But I can't recall any customer who said that they, "could not imagine living without Netezza".

 

Simple self-promotion is not the real point of this post though. What is is the thread of a dilemma that noogle presents us with in his 40+ word tweet. It's something that business managers and analysts face on a daily basis: what is more important –

  • being able look for strategic and/or tactical competitive nuggets by performing SQL OLAP analytics on their full, atomic-level dataset; or
  • looking for that guidance by using advanced analytical toolsets on subsets or aggregations of their data that are extracted from the data warehouse?

 

Here's the whole tweet by noogle in it's original form:

デー タが莫大になると分析が不可能になる。少ないデータを複雑なアルゴリズムで分析するよりも、莫大なデータを単純なアルゴリズムで分析する方が有益。統計学 とは逆。アホかという量のデータ分析の手助けするのがNetezza。もうあたしはNetezzaの無い世界では生きていけない。

 

And here's a translation of it into English [parenthetical comments and emphasis are mine]:

When data is huge, complex analytics are impossible. It’s far more beneficial analyzing massive data with simple [SQL] logic, rather than analyzing small data with complicated analysis. This is opposite of statistics [based on sampling techniques]. Analyzing data which is “crazy massive” is Netezza. I cannot imagine life without Netezza.

 

It turns out that noogle is a long-time user of advanced analytics and predictive techniques. He knows their value, but his tweet exposes of weakness of today's typical analytical environment. By not being able to perform advanced analysis inside the database, most of that work (if performed at all) is done in external servers based on data sets that are extracted (filtered, sampled and/or aggregated) from the data warehouse.

 

That adds latency to do the extraction and limits the "currency" of the data. Depending on whom you ask, it also limits the accuracy of the results. For instance, looking at aggregations or samples may give you a sense of the "big picture" but not necessarily uncover the needle in the haystack (e.g., fraud detection) or the impact of a long tail that can be exploited in a particular business.

 

So noogle's choice is to use the analytic horsepower of TwinFin over the sampling techniques. But if one is limited to the set-based logic of SQL, perhaps aided by user-defined functions, you are again limited in the predictive visibility that those tools can provide. Faced with the dilemma this customer chose being able to analyze all the data over statistically sampling and performing advanced analytics on the sample set. Having an answer to that dilemma is precisely what has driven the advent of the i-Class functionality for TwinFin.

 

We're excited about TwinFin i-Class, but I'm interested in what others may have to say about this. Does your company employ advanced analytical techniques and how have you reconciled the "sampling" versus "full data set" questions in your business? And what are the prospects and pitfalls of doing "crazy complicated analytics on crazy massive data" all in one simple, high-performance data warehouse appliance from your perspective?

0 Comments Permalink
0

I know some of you are wondering if I have "gone dark" or something. But hey, there a conference comin' up and I'm on the hot seat!

 

Much of what will be discussed, I could (and may) post here eventually, but for now it's time to burn this stuff into the presentations so all of us have a productive meeting on Day One.

 

Now some of you have also fired off to me some interesting requests for the discussion, most of which I agree with so we'll definitely make room for it. In the end, the whole of the best practice session is not really for you to hear me prattle on about derring-do - although I do have tales to tell - but more to gather the tales that you have to tell, so that all of us are better for when we return to our respective machines.

 

One of the requests is a deeper dive into the Ten Rules of Large Scale Data Processing that I added to Netezza Underground. We could probably spend all day on that stuff - because it can go really deep, like outta-hand deep on some of them. Not to worry, as I am a highly trained professional moderator and promise to keep you on track.

 

Also note that on Day Two there is a special session on TwinFin migration, with some emphasis on Netezza-To-Netezza migrations since some of us are in the middle of these kinds of conversions. I think this will be especially productive if some of the Enzees (that would be you) are present (and not shy!) so that the attendees can wrap their heads around the overall process and what to expect.So that's another invitation to not only bring your notepad, but your notebook (the one that you use for design sessions) and give us a little non-proprietary insight on how you conquered the world (or at least, your corner of it).

 

Not so strangely, the overall, high-level methodology for moving from an older Netezza to a Twinfin does not change (compared to moving from say, Oracle to a Twinfin), but the challenges can be multivariant. It is these challenges, nuances and subtlties that make life fun - so bring some of them with you! We'd like to hear the adventure.

 

And yes, the rumors are true. I have come under intense pressure to publish yet another Netezza book, and this was evident and underway as of last Fall. The real problem is in deciding what goes into the book and what should be left out - but the left-out stuff is valuable too. I think the consensus now is that the content needs to be divided into two books, and publish them both in a rapid fashion. Hey, it worked for Back to the Future, The Matrix and Lord of the Rings - but I don't have any delusions of such wild success - I just need to get them into print.

 

I am working right now on a mongraph concerning inheritable objects, namely views and such, that in some ways do, and some ways don't translate into systems of scale. After all, once the data gets really big, the physics becomes even more critical. Any inefficiency will only get worse over time. Okay, that's my hook to pique your curiosity. More later!

0 Comments Permalink
0

The big event for me last week was introductory training in TwinFin i-Class. What is does is explained in the link, what was most interesting for me was the discussion at dinner afterwards. I’m still learning the technology and the products, but the business case for what we’re doing just seemed so obvious to me right from the start. It starts with, but goes way beyond what Netezza insiders call the ‘speed and feed’ story. Speed and feed just means feed the Netezza box your data and it will deliver greater speed than your existing warehouse infrastructure. The key question then is what do you do with all that speed? What we’re seeing is a world where more and more data gets collected from more and more sources and we’re seeing our customers thinking about some very smart things to do with that data. One example is retailers tying point-of-sale data to customer loyalty cards or branded payment cards, so they can analyse individual spending patterns - scary or cool, depending on how well they turn that knowledge into really helping me with individualized advertising.

For sure, i’ve had enough of so-called eCommerce leaders spamming me with mail. Apparently there’s one global bookseller that thinks i’m interested in fantasy, gardening, learning Italian and technology. A little better analysis of the data they have about me would tell them that i only buy Italian, gardening & fantasy books in November, December or around dates that represent family birthdays. They could even work those dates out if they traced my purchases to the wishlists i pick from. Now it doesn’t matter, i can unsubscribe or i can just delete the mails unread. But in that case we’re both losers and it makes me cross to see the knowledge represented by that data go to waste. And why should it? It’s not expensive or rocket engineering to unify wishlists, purchase records, products and categories and run some simple analytics, though it might need some horsepower and a lot of historical data available.

I hear a lot of discussion about the intelligent economy, and i’m a big believer. But sometimes i think the development of the information economy it’s a bit like the development of the human being. As babies we have all the sense data of an adult but at first we just use it for basic purposes – your hand jumps back from something sharp and your brain registers distress. Later we get smarter and remember that data so we resist the urge to poke the cat in the eye. Later still we learn to coordinate so we can decide to stroke the cat instead. So we use the same data for increasingly sophisticated processes as we retain the data and analyse how we want to use it. And unlike humans, organizations can decide how fast they want to develop. How well they want to exploit those data resources.

There’s always more data you might collect and keep, but i’m thinking that right now the challenge is more about analytics. Our customers , telcos, retailers and others are pushing on hard with smarter ways to use their data. That’s why we’re focussing on moving analytical capability into the database with the i-Class, trying keep ahead of the curve as customers really start to ramp up their deep analytics. I also saw a presentation from one of our partners last week (TEOCO) who provide specific high-value applications for telcos. Exactly the same trend - doing smarter things with more data.

0 Comments Permalink
1

It’s a great honor after just two full years as a public company to be recognized in the Boston Globe Top 100 list. Making the list after a most difficult economic downturn is not only something that all of us at Netezza are proud of but it’s a testament to the tremendous support we receive from our customers, our partners, our investors, and our employees.

To be highlighted amongst the ranks of some of Massachusetts largest, most successful, and most important companies, including EMC, Genzyme and our customer TJX, is truly special. Given all the great technology companies in Massachusetts, to be chosen as one of only 19 companies categorized as “technology” by the Globe is exciting.

We are building Netezza on a strong philosophy of being a business partner that is “easy to do business with”, that simplifies the complexity of IT infrastructure, and that helps our customers gain value and insight from the data assets on which they run their businesses. Our core values permeate everything we do and our customers are continually complementing us on our approach to the business. Of course, we could not do all this without a remarkably talented group of employees and our experience has been that Massachusetts continues to produce many of the best and brightest.

Thank you, Boston Globe for this recognition. We look forward to our continued journey building a great east coast technology company, a company that Massachusetts can be proud of.

1 Comments Permalink
2

I’m drafting this as we fly ‘Compact First’ from Johannesburg to Cape Town. Actually it’s a one class plane, but if you want to get the status-conscious sales executive on a one-class plane you tell him it’s ‘Compact First’. But one class or not Kulula Air are pretty classy. Their emergency procedures leaflet in the seat pocket is entitled ‘Stuff to do in an Emergency’ and features a picture of a member of cabin crew with her hands thrown up in horror and a look of panic on her face. Inside, along with the standard sensible stuff about oxygen masks dropping and leaving your high heels behind, there are little sidebar comments like ‘we tried laughing gas instead, but they insisted on oxygen’ and ‘yes ladies I’m afraid you’re going to have to leave the Jimmy Choos behind.’ Also on Kulula Air flights the smoking lounge is on the port wing - they say if you can light it you can smoke it. It was an early start so i was slightly regretting the long discussion about South African wine the previous evening.

I have a long history of conventional RDBMS database design. You use indexes, partitions and other more esoteric features - whatever the database can do for you. And you look at table design ‘data tricks’ - denormalization, data duplication and so on. So as a Netezza neophyte I was keen to learn a bit about how the database features available differ and to learn some design tricks of the trade. I had to sign a non-disclosure before the tech account manager I was grilling would answer. Not because it’s secret, but because he doesn’t want a half-educated marketing director running round thinking he’s a ninja-level design consultant. Turns out that in Netezza it’s pretty straightforward to get going - you just select your distribution key and start loading data. I’m educated enough now to know there’s more to it than that, but not educated enough to explain it. And anyway 90% of it is not important for 90% of databases. The whole Netezza architecture of hardware, firmware & software is holistically designed so you really have to be down to the very short straws before you need to do anything significant for any particular database design.

I thought that was great and filed it away as interesting stuff I should learn more about as I go. But it was brought back into my mind when I was talking to the DW architect of a potential new customer about how we should set up our proof of concept project with them (more about Netezza Test Drive proof of concept - POC - projects here). And he was very concerned that the way his database is designed (the data tricks I mentioned above) wouldn’t add any value in a Netezza database and might even be sub-optimal (duplicated data always carries a load; you only do that if there’s a concomitant benefit). I said he was right that some of the hyper-tuning he had done to get the last ounce of performance out of his existing warehouse (before they threw in the towel and started talking to us) would no longer help. But the debate was around whether he should devote time in the proof-of-concept project to re-design. The Netezza view is don’t. Just load your data and compare the results. We don’t say this to shorten the timescale so we get to a sale quicker, though our sales folk might feel that is a good enough reason. The real reason is more scientific. In any experimental process you change one factor at a time, so when you see a difference you know what caused it. We like to just put the Netezza appliance in to replace an existing warehouse and throw the same queries at it. That way we and the customer know the performance improvement is just down to the appliance itself.

I don’t say there isn’t value in subsequently re-addressing the database design, just don’t do it as part of the POC, otherwise you’re making it impossible to calculate ROI on the appliance and you’re also delaying time-to-payback. At Netezza we’re pretty confident you will get sufficient improvement from a TwinFin that you’ll have all the time you need to address any historical design tweaks. But when I was discussing this with my friendly neighbourhood deepTechs, they said it almost never happens. Customers just never get around to removing these historical idiosyncrasies of a database design because it’s not worth it.

Maybe we should think of them as what first the students of gothic architecture and then the evolutionary biologists call spandrels. They don’t add value, but they are doing no discernable harm.

Next stop Cape Town.

2 Comments Permalink
0

Monday I flew into South Africa for the first time, just a month too   early to watch the footy. That was bad planning, but the reason I’m  here was to launch the partnership with BITanium – our new partner in the  region. The launch was this morning at a very grand hotel on the  outskirts of Johannesburg, I was a bit nervous – this was my Netezza  debut in front of customers, but i shouldn’t have bothered.  After  Calvin Low of Edcon presented there wasn’t much  more that needed saying.

 

Edcon are our first customer  in South Africa - a major retailing force in the area - who had taken  our ‘Test Drive’ challenge  and went straight ahead with their first project.  And the numbers he  shared with the audience were great. They did a really thorough job on  the Test Drive proof of concept (POC);  they set us up against the  latest version of their existing warehouse and threw a whole spectrum of  their existing queries at it.  The worst performing query ran for 48  hours on the incumbent kit, then fell over. On the Netezza box it ran to  completion in 20 minutes. Another query that had taken 5hrs+ ran in  less than 3 minute on Netezza.

It was funny watching the body  language of some of the audience who had done the ‘raise  eyebrow, lean  back in chair, fold arms, cross legs’ response when Chris O’Connell of  BITanium had put up the old ’10 to 100 times faster’ slide.  When Calvin  had finished with them they were doing the ’lean forward , elbows on  table, chin on fist, stare intently’ posture.

 

My personal triumph of the day was answering  a serious techie  question on solid state media to the satisfaction of the Netezza SE  & Tech Account Manager who were in the audience (presumably both  doing the ‘bite fingernails, clench buttocks’ response when I launched  into my answer).

 

Now it’s late after dinner at a Greek restaurant,   discussing  how conventional warehouse designs get so convoluted over time as they  are tuned and tuned to keep up with new demands and increasing data  volumes. The debate was about whether you should clean up the schema  before move it on Netezza or whether you should just lift & shift,  let the Netezza box solve your performance problem and clean up at your  leisure.  In fact we know from experience that clean up often doesn’t  happen, it becomes less of an issue when the knotty performance problems  that led to the design complexity just go away.

 

By the way it was Greek because of the vegetarian-friendly menu  for one lacto and one pescatarian (me) in the party. You get some funny  looks from the average South African confronted with a vegetarian Meze  for his starter

0 Comments Permalink
1 2 3 ... 5 Previous Next