Overview
On this page we summarize what and how we measure with CSiBE. Our aim is to
give sufficient information so that it is possible to reproduce the results
with the instructions given here. For additional info see
our talk at the 2004 GCC Developers' Summit.
The CSiBE benchmark currently provides three kinds of measurement data: code
size, compilation speed and execution speed. GCC source code is checked out daily from the CVS
for the supported branches, the compilers are built
for the supported targets (arm/thumb, i386, m68k, mips, ppc, sh, arm-linux and i686-linux on mainline and bfin on bfin-3.4.x branch)
and measurements are performed on the CSiBE test bed. The results are stored in a
database, which is accessible via the CSiBE website using several kinds of
queries.
Hardware and software environment
After 2007-07-01 UTC the code size, compilation time and x86_64-unknown-linux execution time measurements are performed on an IBM BladeCenter with the following main parameters:
- AMD Opteron LS20 blade
- AMD Opteron Dual Core 2.2GHz/800Mhz, 1MB L2
- 2x 512MB DDR
- IBM 36.4 GB SSF HDD
- Linux kernel version: 2.6.9-42.ELsmp, CentOS 4
Note: The compilation time data have shrinkage after 2007-07-01 UTC. The compilation time is about 35% smaller than before hardware change.
The arm-linux execution time measurements are performed on an iPaq with the following main parameters:
- Compaq iPaq 3630
- 206 MHz Intel StrongARM
- 16Mb ROM, 32Mb RAM (CompactFlash)
- Linux kernel version: 2.4.19, Familiar v0.7.2, GPE Palmtop Environment
The bfin-uclinux execution time measurements are performed on an BF537-STAMP with the following main parameters:
- 500 MHz ADSP-BF537 Rev. 0.2
- 4Mb ROM, 64Mb RAM (Flash)
- uClinux kernel version: 2006R1 RC2
Old hardware and software environment
Before 2007-07-01 UTC the code size, compilation time and i686-linux execution time measurements were performed on a PC with the following main parameters:
- Asus A7N8x Deluxe
- AMD AthlonXP 2500+ 333FSB @ 1.8GHz
- 2x 512MB DDR (200MHz)
- 2x Seagate 120GB 7200rpm HDD
- Linux kernel version: 2.4.28, Debian Linux (woody) 3.0
Test bed
The latest versions of the CSiBE test bed have the version number 2.x.x.
It consists of 18 projects and its source size is roughly 51 MB
(without the input data required for execution time measurements).
When compiled, it totals about 3-4 MB binary code. The test bed consists of
programs of various types such as media (gsm, mpeg), compiler, compressor, network, driver
programs, and parts of the Linux kernel. The source code with the scripts for the measurement are accessible
through the Download page. The test inputs required
for execution time measurement are separately available as well.
The older versions (1.x.x) contained different test programs. In some
diagrams there is a possibility to chose older measurement data
(back to May 2003) with this version.
Compilers and binaries measured for mainline
The compilers with which execution time is not measured, are configured as cross-compilers for the supported targets.
We use standalone targets for use with the newlib runtime library.
At present, binutils v2.17 and newlib v1.15 are used.
The compiler for the x86_64-unknown-linux target (used also for execution time
measurement) is configured as native for the machine that performs the measurements.
The runtime library used is glibc v2.3.4.
Before 2007-07-01 UTC we have used binutils v2.14, newlib v1.12, and glibc v2.3.2.
When we measure code size and compilation time, we do not include linking time and code size of the executable.
CVS checkout
Snapshots of GCC source code are retrieved from the mainline CVS branch
daily at 12:00:00 (UTC).
Configuration
- binutils is configured with no extra flags
- newlib is configured with the only extra flag which
enables the optimization for space:
--enable-target-optspace
- For glibc and uClibc, the stock binaries are used
- The common for configuring GCC are:
--enable-languages=c --disable-nls --disable-libgcj
--disable-multilib --disable-checking --with-gnu-as --with-gnu-ld
Additional flags for the standalone versions are:
--disable-shared --disable-threads
--with-newlib
Additional flags for the Linux versions:
--enable-shared
Compilation
A simple make is used to build binutils, newlib
and GCC.
Other GCC code base
Apart from the official GCC mainline branch, currently the following compilers are regularily measured:
- Blackfin (bfin-elf)
- CVS checkout: Snapshots of
Blackfin project source base are
retrieved from the mainline CVS branch daily at 12:00:00
- Configuration and compilation: Same as described at
Blackfin project homepage.
See the Build Script here.
External baseline
Measurement
See the specific technical pages:
|
