1.There are three types of benchmarks: kernel, synthetic, and application.
- Kernel benchmarks extract the most significate codes as benchmarks.Those
codes usually take only 10 percent of the total codes but use 80 percent of CPU resources.
Pros: Simple
Cons: too small to fit in cache; prone to attack by compiler
- Synthetic benchmarks use instructions mix chosen by analysts.
Pros: smarter and still simple
Cons: make assumption about applicatios, prone to obsolete.
- Application benchmarks use user's own application program.
Pros: From a user's perspective, it is the best way.
Cons: Need to collect applications and make a decision.

2. SPEC
- SPEC benchmarks are Application benchmarks.
- SPEC chooses a simple measure, elapsed time, to run benchmark.
- A simple speed metric and portable codes are keys to SPEC.
- SPEC measures performance by determing the elapsed time required to run a suite of applications and then comparing the completion time with that of a reference machine.
- SPEC treats each benchmark equally. SPEC selects geometric mean,i.e, G(a1,a2)=sqrt(a1*a2).
- SPEC CPU benchmark separates the results of two fundamentally different workload(Integer and floating point)

Reference
http://www.benchmarkresources.com/handbook/contents.asp
Chapter 9: Overview of the SPEC Benchmark

3. Questions about SoapBenchmark
- which category will it belong to? (kernel, synthetic, application)
List of possible applications:
* business processing (with/without security)
* searching service,i.e.,google, map point
* scientific computation
* handy device (.NET compact framework)
- performance metric? (invocations/elapsed_time or something else)
- composite metric? (weighted or equal benchmark)
- separation of fundamentally different workload, i.e., business process, database query,scientific computation?



- posted by Pop @ 12:19 PM

 
SPEC and SOAP Benchmark
SPEC develops real application benchmarks that would exercise major system components:
technologies, architectures, implementations, memory systems, I/O subsystems, OS, clock rates, bus protocols, compilers, libraries, and application software plus scalability, graphics and networking.

SPEC chooses a simple measure, elapsed time, to run benchmark. A simple speed metric and machine-independent code are keys to providing a comprehensive and fair comparison between competing machines.

SPEC measures performance by determing the times required to run a suite of applications and then comparing the time for completion of each with the time of a reference machine.

SPEC selected geometric mean-a composite metric: each benchmark carries the same weight.

SPEC's members voted not to combine the results of two fundamentally different workloads (inter and floating-point) .

CPU benchmarks suite
SPEC CPU2000 suite includes applications from the following areas:

• AI game theory
• compilers
• interpreters
• data compression
• databases
• weather prediction
• fluid dynamics
• physics
• chemistry
• image processing

Criteria SPEC considers important for the CPU benchmark suite:

• the program can be made compute bound
• the program can be made portable across different hardware architectures and operating systems
• how close the program is to the state of the art for the given field

appPlatform

Currently under development, SPECappPlatform is designed to measure the scalability and performance of enterprise platforms (such as J2EE and .NET) running web services. The benchmark will include features that are typical in customers' enterprise applications, such as transactions, persistence services, web services, and messaging.

- posted by Pop @ 9:49 AM