1.There are three types of benchmarks: kernel, synthetic, and application.
- Kernel benchmarks extract the most significate codes as benchmarks.Those
codes usually take only 10 percent of the total codes but use 80 percent of CPU resources.
Pros: Simple
Cons: too small to fit in cache; prone to attack by compiler
- Synthetic benchmarks use instructions mix chosen by analysts.
Pros: smarter and still simple
Cons: make assumption about applicatios, prone to obsolete.
- Application benchmarks use user's own application program.
Pros: From a user's perspective, it is the best way.
Cons: Need to collect applications and make a decision.

2. SPEC
- SPEC benchmarks are Application benchmarks.
- SPEC chooses a simple measure, elapsed time, to run benchmark.
- A simple speed metric and portable codes are keys to SPEC.
- SPEC measures performance by determing the elapsed time required to run a suite of applications and then comparing the completion time with that of a reference machine.
- SPEC treats each benchmark equally. SPEC selects geometric mean,i.e, G(a1,a2)=sqrt(a1*a2).
- SPEC CPU benchmark separates the results of two fundamentally different workload(Integer and floating point)

Reference
http://www.benchmarkresources.com/handbook/contents.asp
Chapter 9: Overview of the SPEC Benchmark

3. Questions about SoapBenchmark
- which category will it belong to? (kernel, synthetic, application)
List of possible applications:
* business processing (with/without security)
* searching service,i.e.,google, map point
* scientific computation
* handy device (.NET compact framework)
- performance metric? (invocations/elapsed_time or something else)
- composite metric? (weighted or equal benchmark)
- separation of fundamentally different workload, i.e., business process, database query,scientific computation?



- posted by Pop @ 12:19 PM