JANET Bandwidth Management Advisory Service
BMAS Home | About BMAS | Bandwidth Management | Technology Papers and Guidance | Product Evaluations | Calls for Partipation
Bandwidth Conservation with Apache WWW Server
Users of Apache WWW Server can conserve bandwidth over a restricted link in several ways; some of which involve restricting the way in which 'greedy' clients access your WWW server, others by applying specific Apache bandwidth conserving modules.
Greedy Clients
'Greedy' clients, are here defined as either search engines or other web robots that hit every page on your site at regular intervals. Placing a 'robots.txt' file in the root directory of your web server can prevent any client that obeys the robot exclusion file from accessing the /cgi-bin (scripts) or /webalizer (statistics) directories. A sample 'robots.txt' file might contain:
User-Agent: *
Disallow: /cgi-bin
User-Agent: *
Disallow: /webalizer
An alternative to the above configuration is:
User-Agent: *
Disallow: /
which will prevent access to all of the web directory tree.
Note that the 'robots.txt' file does not prevent access by normal browsers, just web robots.Please be aware however, that but some web robots do ignore the robots.txt file.
More details on what can be included in the 'robots.txt' file can be found at: http://www.robotstxt.org/wc/norobots.html
Apache Modules for bandwidth conservation
The Apache WWW server is modular in its construction, with some modules being specifically orientated towards Bandwidth Conservation. Depending on the Operating System or Distribution, Apache can be supplied as one executable with all modules embedded in or as an executable with separate modules that are linked in but loaded when Apache is started.
The latter method makes it a relatively simple matter to add new features to the Apache WWW server by just adding a module speciifaction to the Apache configuration file.
Some of the more popular Apache modules that fit into the 'Bandwidth Conservation' category are: mod_throttle; mod_bandwidth; mod_gzip and mod_deflate.
The mod_throttle and mod_bandwidth modules control the actual accesses by a client, whilst mod_gzip and mod_deflate are used to compress documents before delivery. The configurable directives for these modules are listed below together with an explanation of said directrives.
mod_throttle
Module Directives:
ThrottleClientIP
- - Specify global throttle pool size, policy, limit, and period for all client IP connections.
ThrottleContentType
- - Content-Type of throttle display, or use /throttle-status?content-type=text/plain
ThrottleIndicator
- - Set % threshold for green, yellow, red, and critical alerts.
ThrottleLockFile
- - The lock file used with fcntl() or flock() serialization. Must be stored on a local disk.
ThrottleMaxDelay
- - Set max. delay threshold in seconds before refusing connections.
ThrottlePolicy
- - Select policy, limit, and period.
ThrottleRefresh
- - Refresh time in seconds, or use /throttle-status?refresh=sec
ThrottleRemoteUser
- - Specify global throttle pool size, policy, limit, and period for all authenticated remote users.
ThrottleRuntimeFile
- - The runtime data file used across shutdowns and restarts.
ThrottleUser
- - Specify user to throttle according to policy, limit, and period.
Full module details can be found at: http://www.snert.com/Software/mod_throttle/
mod_bandwidth
Module Directives:
BandWidthDataDir
- - Sets the name of the root directory used by mod_bandwidth to store its internal temporary information.
BandWidthModule
- - Enable or disable totally the whole module.
BandWidthPulse
- - Change the algorithm used to calculate bandwidth adn transmit data.
BandWidth
- - Limit the bandwidth for files in a given directory tree based on the remote host name or ip address.
LargeFileLimit
- - Set a maximum rate to use for a given minimum filesize.
MaxConnection
- - Restrict the number of maximum simultaneous connections.
MinBandwidth
- - Set a minimum bandwidth to use for transferring data.
Full module details can be found at: http://www.snert.com/Software/mod_throttle/
mod_gzip
Module Directives:
mod_gzip_on
- - Yes=mod_gzip will handle requests. No=mod_gzip is disabled.
mod_gzip_add_header_count
- - Yes=Add header byte counts to Common Log Format output total(s).
mod_gzip_keep_workfiles
- - Yes=Keep any work files used. No=Automatically delete any work files used.
mod_gzip_dechunk
- - Yes=Allow removal of 'Transfer-encoding: chunked' when necessary.
mod_gzip_min_http
- - Minimum HTTP protocol value to support. 1000 = HTTP/1.0 1001 = HTTP/1.1
mod_gzip_minimum_file_size
- - Minimum size ( bytes ) of a file eligible for compression
mod_gzip_maximum_file_size
- - Maximum size ( bytes ) of a file eligible for compression
mod_gzip_maximum_inmem_size
- - Maximum size ( bytes ) to use for in-memory compression.
mod_gzip_temp_dir
- - The directory to use for work files and compression cache
mod_gzip_item_include
- - ARG1=[mime,handler,file,uri,reqheader,rspheader] ARG2=[Name of item to INCLUDE in list of things that should be compressed]
mod_gzip_item_exclude
- - ARG1=[mime,handler,file,uri,reqheader,rspheader] ARG2=[Name of item to EXCLUDE from list of things that should be compressed]
mod_gzip_add_vinfo
- - Configuration directive no longer supported.
mod_gzip_do_static_files
- - Configuration directive no longer supported.
mod_gzip_do_cgi
- - Configuration directive no longer supported.
mod_gzip_verbose_debug
- - Configuration directive no longer supported.
mod_gzip_post_on
- - Configuration directive no longer supported.
mod_gzip_command_version
- - User defined pickup string to use for mod_gzip version command.
mod_gzip_can_negotiate
- - Yes=Negotiate/send static compressed versions of files No=Do not negotiate.
Full module details can be found at: http://sourcefourge.net/projects/mod_gzip/
mod_deflate - Apache v2.xx and above
Module directives
SetOutputFilter DEFLATE
- - Enable compression for documents in a given container.
AddOutputFilterByType DEFLATE text/html
- - Compress only text/html documents in a given container.
DeflateBufferSize
- - Fragment size to be compressed at one time.
DeflateCompressionLevel
- - How much compression to apply to the output.
DeflateFilterNote
- - make a note of the compression ratio for logging.
DeflateMemLevel
- - Hoe much memory should the system use for compression.
DeflateWindowSize
- - Zlib compression window size.
Full module details can be found at: http://httpd.apache.org/docs-2.0/mod/mod_deflate.html