Database Benchmarking

Benchmarking is one of the important aspect of database performance evaluation. Since years, there are quite a few number of database benchmarking tools, standards were evolved. To name some of them

•   Wisconsin
•   AS3AP
•    Debit-Credit
•   TPC
•    SAP R3 SD
•    People Soft eBill
•    Oracle AS-BM

Basically Database benchmarks can be divided into two, one is synthetic benchmark that uses artificial data and simulating some application in a particular domain. The other one is Empirical which uses real data and operations run in an actual application system.

We know that it is very difficult to control their data in scalable manner and it is also expensive to implement entire system for other databases. Hence, Synthetic benchmark tools are used mostly to measure and evaluate database performance.

Synthetic benchmarks can be further divided into three categories

The System standards benchmarks are created from academic world such as Wisconsin and AS3AP (ANSI SQL Standard Scalable and Portable).

Another is industry standard benchmarks created by organizations like TPC( Transaction Processing Performance Council ) and SPEC (Standard Performance Evaluation Corporation), these standards provides their application for benchmarking depending on the domain.

The 3^rd standards are Application standard benchmarks which are created by DBMC, ERP major vendors such as Oracle AS-BM, SAP R3 SD.

Confused! Here is pictorial view of it.

I know, now you are at the cross to decide on what is the ideal benchmarking standard for you. If you are doing it for RDBMS, you want to repeatable results for accurate comparisons then choose Synthetic type model benchmark, then further, you wanted to publish your results in the market then go for choosing latest and mostly used standard, which I prefer is TPC (if you are doing it for DBMS).

Transaction Processing Performance Council is a non-profit Organization, its main activities are to define transaction processing and database benchmarks and verifiable TPC performance data to industry.  Following picture shows you different benchmarks defined by TPC .

To read more on TPC refer to : http://www.tpc.org/default.asp