Benchmark (computing)

Source: Wikipedia, the free encyclopedia.
This article is about the use of benchmarks in computing. For other uses, see Benchmark.
A graphical demo running as a benchmark of the OGRE engine

In computing, a benchmark is the act of running a computer program, a set of programs, or other operations, in order to assess the relative performance of an object, normally by running a number of standard tests and trials against it.[1]

The term benchmark is also commonly utilized for the purposes of elaborately designed benchmarking programs themselves.

Benchmarking is usually associated with assessing performance characteristics of

database management systems
(DBMS).

Benchmarks provide a method of comparing the performance of various subsystems across different chip/system architectures. Benchmarking as a part of continuous integration is called Continuous Benchmarking.[2]

Purpose

As

Athlon XP or PowerPC processors, which did not necessarily translate to more computational power; a processor with a slower clock frequency might perform as well as or even better than a processor operating at a higher frequency. See BogoMips and the megahertz myth
.

Benchmarks are designed to mimic a particular type of workload on a component or system. Synthetic benchmarks do this by specially created programs that impose the workload on the component. Application benchmarks run real-world programs on the system. While application benchmarks usually give a much better measure of real-world performance on a given system, synthetic benchmarks are useful for testing individual components, like a

hard disk
or networking device.

Benchmarks are particularly important in

algorithms
of an application, it will contain the performance-sensitive aspects of that application. Running this much smaller snippet on a cycle-accurate simulator can give clues on how to improve performance.

Prior to 2000, computer and microprocessor architects used

SPEC
to do this, although SPEC's Unix-based benchmarks were quite lengthy and thus unwieldy to use intact.

Computer manufacturers are known to configure their systems to give unrealistically high performance on benchmark tests that are not replicated in real usage. For instance, during the 1980s some compilers could detect a specific mathematical operation used in a well-known floating-point benchmark and replace the operation with a faster mathematically equivalent operation. However, such a transformation was rarely useful outside the benchmark until the mid-1990s, when

VLIW architectures emphasized the importance of compiler technology as it related to performance. Benchmarks are now regularly used by compiler
companies to improve not only their own benchmark scores, but real application performance.

CPUs that have many execution units — such as a

VLIW CPU, or a reconfigurable computing
CPU — typically have slower clock rates than a sequential CPU with one or two execution units when built from transistors that are just as fast. Nevertheless, CPUs with many execution units often complete real-world and benchmark tasks in less time than the supposedly faster high-clock-rate CPU.

Given the large number of benchmarks available, a manufacturer can usually find at least one benchmark that shows its system will outperform another system; the other systems can be shown to excel with a different benchmark.

Manufacturers commonly report only those benchmarks (or aspects of benchmarks) that show their products in the best light. They also have been known to mis-represent the significance of benchmarks, again to show their products in the best possible light. Taken together, these practices are called bench-marketing.

Ideally benchmarks should only substitute for real applications if the application is unavailable, or too difficult or costly to port to a specific processor or computer system. If performance is critical, the only benchmark that matters is the target environment's application suite.

Functionality

Features of benchmarking software may include recording/

color-coded tiles, and pausing the process to be able to resume without having to start over. Software can have additional features specific to its purpose, for example, disk benchmarking software may be able to optionally start measuring the disk speed within a specified range of the disk rather than the full disk, measure random access reading speed and latency, have a "quick scan" feature which measures the speed through samples of specified intervals and sizes, and allow specifying a data block size, meaning the number of requested bytes per read request.[3]

Challenges

Benchmarking is not easy and often involves several iterative rounds in order to arrive at predictable, useful conclusions. Interpretation of benchmarking data is also extraordinarily difficult. Here is a partial list of common challenges:

Benchmarking principles

There are seven vital characteristics for benchmarks.[6] These key properties are:

  1. Relevance: Benchmarks should measure relatively vital features.
  2. Representativeness: Benchmark performance metrics should be broadly accepted by industry and academia.
  3. Equity: All systems should be fairly compared.
  4. Repeatability: Benchmark results can be verified.
  5. Cost-effectiveness: Benchmark tests are economical.
  6. Scalability: Benchmark tests should work across systems possessing a range of resources from low to high.
  7. Transparency: Benchmark metrics should be easy to understand.

Types of benchmark

  1. Real program
    • word processing software
    • tool software of CAD
    • user's application software (i.e.: MIS)
    • Video games
    • Chromium browser or Linux kernel
  2. Component Benchmark / Microbenchmark
  3. Kernel
    • contains key codes
    • normally abstracted from actual program
    • popular kernel: Livermore loop
    • linpack benchmark (contains basic linear algebra subroutine written in FORTRAN language)
    • results are represented in Mflop/s.
  4. Synthetic Benchmark
    • Procedure for programming synthetic benchmark:
      • take statistics of all types of operations from many application programs
      • get proportion of each operation
      • write program based on the proportion above
    • Types of Synthetic Benchmark are:
    • These were the first general purpose industry standard computer benchmarks. They do not necessarily obtain high scores on modern pipelined computers.
  5. I/O benchmarks
  6. Database benchmarks
    • measure the throughput and response times of database management systems (DBMS)
  7. Parallel benchmarks
    • used on machines with multiple cores and/or processors, or systems consisting of multiple machines

Common benchmarks

Industry standard (audited and verifiable)

Open source benchmarks

  • AIM Multiuser Benchmark – composed of a list of tests that could be mixed to create a 'load mix' that would simulate a specific computer function on any UNIX-type OS.
  • Bonnie++ – filesystem and hard drive benchmark
  • BRL-CAD – cross-platform architecture-agnostic benchmark suite based on multithreaded ray tracing performance; baselined against a VAX-11/780; and used since 1984 for evaluating relative CPU performance, compiler differences, optimization levels, coherency, architecture differences, and operating system differences.
  • Collective Knowledge – customizable, cross-platform framework to crowdsource benchmarking and optimization of user workloads (such as deep learning) across hardware provided by volunteers
  • Coremark – Embedded computing benchmark
  • DEISA Benchmark Suite
    – scientific HPC applications benchmark
  • Dhrystone – integer arithmetic performance, often reported in DMIPS (Dhrystone millions of instructions per second)
  • Command-line tool for storage benchmarking that generates a variety of requests against computer files, partitions or storage devices
  • Fhourstones – an integer benchmark
  • HINT – designed to measure overall CPU and memory performance
  • Iometer – I/O subsystem measurement and characterization tool for single and clustered systems.
  • IOzone – Filesystem benchmark
  • LINPACK benchmarks – traditionally used to measure FLOPS
  • Livermore loops
  • NAS parallel benchmarks
  • NBench – synthetic benchmark suite measuring performance of integer arithmetic, memory operations, and floating-point arithmetic
  • PAL
    – a benchmark for realtime physics engines
  • PerfKitBenchmarker – A set of benchmarks to measure and compare cloud offerings.
  • Phoronix Test Suite – open-source cross-platform benchmarking suite for Linux, OpenSolaris, FreeBSD, OSX and Windows. It includes a number of other benchmarks included on this page to simplify execution.
  • POV-Ray – 3D render
  • Tak (function) – a simple benchmark used to test recursion performance
  • TATP Benchmark – Telecommunication Application Transaction Processing Benchmark
  • TPoX
    – An XML transaction processing benchmark for XML databases
  • VUP (VAX unit of performance) – also called VAX MIPS
  • Whetstone – floating-point arithmetic performance, often reported in millions of Whetstone instructions per second (MWIPS)

Microsoft Windows benchmarks

Others

  • AnTuTu – commonly used on phones and ARM-based devices.
  • Creative Computing Benchmark – Compares the BASIC programming language on various platforms. Introduced in 1983.
  • Geekbench – A cross-platform benchmark for Windows, Linux, macOS, iOS and Android.
  • iCOMP – the Intel comparative microprocessor performance, published by Intel
  • Khornerstone
  • Performance Rating – modeling scheme used by AMD and Cyrix to reflect the relative performance usually compared to competing products.
  • SunSpider
    – a browser speed test
  • VMmark – a virtualization benchmark suite.

See also

  • Benchmarking (business perspective)
  • Figure of merit
  • Lossless compression benchmarks
  • Performance Counter Monitor
  • Test suite – a collection of test cases intended to show that a software program has some specified set of behaviors

References

  1. S2CID 1047380
    . Retrieved 2017-06-09.
  2. doi:10.1109/IC2E.2019.00039
    . Retrieved 2023-12-03.
  3. ^ Software: HDDScan, GNOME Disks
  4. ^ Krazit, Tom (2003). "NVidia's Benchmark Tactics Reassessed". IDG News. Archived from the original on 2011-06-06. Retrieved 2009-08-08.
  5. ^ Castor, Kevin (2006). "Hardware Testing and Benchmarking Methodology". Archived from the original on 2008-02-05. Retrieved 2008-02-24.
  6. doi:10.1109/CogMI48466.2019.00029
    .
  7. ^ Ehliar, Andreas; Liu, Dake. "Benchmarking network processors" (PDF). {{cite journal}}: Cite journal requires |journal= (help)
  8. ^ Transaction Processing Performance Council (February 1998). "History and Overview of the TPC". TPC. Transaction Processing Performance Council. Retrieved 2018-07-02.

Further reading

External links
Wikimedia Commons has media related to Benchmarks (computing).


Retrieved from "https://en.wikipedia.org/w/index.php?title=Benchmark_(computing)&oldid=1209074256"