Mirroring NetBSD

From NetBSD Wiki

Contents 1 Mirroring NetBSD 2 Synopsis 3 Other Documentation 4 Mirroring Techniques 5 Collections 5.1 NetBSD 5.1.1 Disk Space Requirement Estimates 5.2 FreeBSD 5.2.1 Disk Space Requirement Estimates 6 Input Protocols 6.1 Example Disk Layout 6.2 Choose a Mirror Source 6.3 Mirroring Code 6.4 Crontab(5) Example 7 Output Protocols 7.1 AnonCVS SSH Access for anoncvs meta User 7.1.1 Using the AnonCVSSSH (anoncvsssh) Login Shell 7.1.2 Protecting SSH 7.1.2.1 DenySSH Setup with pf(4) 7.1.3 Example AnonCVS SSH Access 7.2 Configuring CVSUP Server 7.2.1 NetBSD CVSUP Note 7.2.2 FreeBSD CVSUP Note 7.2.3 Example CVSUP Client Configuration 7.2.4 Example cvsupd(8) rc.d/ script 7.2.5 Adding IPv6 Support to cvsupd(8) 7.3 Configuring RSYNC Server 7.3.1 Example RSYNC Client Access 7.4 Configuring an Anonymous FTP Server 7.5 Configuring an HTTP Server 7.6 Rate Limiting Link Saturation using pf(4) 7.7 Security

Mirroring NetBSD

A guide to mirroring NetBSD (and other *BSD) Sources, Binaries, Web/Documentation.

Synopsis

You may wish to host a NetBSD mirror either to support the community or for private use.

Other Documentation

The official Mirroring FAQ can be found at:

 http://www.netbsd.org/docs/mirror.html

The purpose of this document is to augment that document with system administration details.

Mirroring Techniques

The general goal of running a mirror server is to replicate file system collection hierarchies to help distribute network load, provide redundancy, etc.

To do so, this article breaks documentation into two major categories:

• Input protocols (clients on the server)
• Output protocols (system services clients consume)

Collections

Each project/distribution has a separate set of "collections", or file system hierarchies, which it wishes to distribute

NetBSD

NetBSD has the following collections:

• ftp (the /pub, /pub/NetBSD and /pub/NetBSD-archive FTP hierarches)
• daily (The /pub/NetBSD-daily FTP hierarchy subdir anchor)
• pkgsrc (the /pub/pkgsrc FTP hierarchy)
• cvs (The CVS Tree in raw form)

Disk Space Requirement Estimates

Collection	Current Size	Maximum Size
ftp	185 gig	-
daily	154 gig	250 GB
pkgsrc binaries	350 gig	400 GB
www/htdocs	180 meg	-
cvs tree	4.5 gig	-

The maximum sizes are valid February 2009.

FreeBSD

FreeBSD has the following basic collections:

• ftp (the /pub, /pub/FreeBSD and FTP hierarchies on rsync mirrors)
• ftp-archive (/pub/FreeBSD-Archive and FTP hierarchies on rsync mirrors)
• cvs (The CVS Tree in raw form)

Disk Space Requirement Estimates

Collection	Current Size	Maximum Size
ftp	700 GB (190GB w/ Ports Distfiles)	-
ftp-archive	500 GB	-
cvs tree	3.8 GB	-

Input Protocols

For mirroring NetBSD and FreeBSD servers, you can use a combination of rsync(1) and cvsup(1) to retrieve input data sets.

NOTE: Some systems, for example, FreeBSD's ports/net/anoncvs package, use cvsup(1) or csup(1) in non-checkout mode to replicate CVS repositories, as well as configuration files of themselves.

Each project has a separate architecture of tiered mirror servers. Each project documents the access and replication policies at their respective web site. In general, the diagram below sums of the general consensus amongst the various *BSD projects for server architecture, including replication protocols:

Example Disk Layout

Depending on how your logical and physical volumes are arranged, you can setup your local storage file system mountpoints in a variety of ways.

In the examples that follow, the format will be:

/export/${project}/${collection}

Where $project is = {netbsd,freebsd} Where $collection = {ftp,cvs,www}

However, symbolic links will be used to span the collections across limited disk space:

$ ls /export/*/*
lrwxr-xr-x  1 root  wheel  16 Oct  8 15:12 /export/freebsd/cvs@ -> /export1/fbsdcvs
lrwxr-xr-x  1 root  wheel  16 Oct 24 15:03 /export/freebsd/ftp@ -> /export1/fbsdftp
lrwxr-xr-x  1 root  wheel  16 Oct  8 14:34 /export/netbsd/cvs@ -> /export0/nbsdcvs
lrwxr-xr-x  1 root  wheel  16 Oct 13 18:18 /export/netbsd/ftp@ -> /export0/nbsdftp

NOTE: Symbolic links can lead to confusion in configuration of output services that utilize chroot(2), etc.

Choose a Mirror Source

• For each source, select a tier 1 or tier 2 mirror server from which to mirror. Use traceroute(1) and traceroute6(1) to find a mirror geographical or network topographically in proximity.

Mirroring Code

Example code for mirroring a collection might look like rsync-common.sh for an example shell script. Use the following example configuration file:

metrotitan% sudo more netbsd-daily.conf
BASEDIR=/export0/nbsdftp/pub/NetBSD-daily
EXCLD_FLAGS="--exclude /netbsd-3*"
RSYNC_FLAGS="-v -Wartz -4 --no-g --no-o ${EXCLD_FLAGS} "
RSYNC=/usr/pkg/bin/rsync
RSYNC_SOURCE=rsync://rsync.NetBSD.org/NetBSD-daily
LOCK_FILE=/var/run/rsync_netbsd_ftp_daily.lock
LOG_FILE=/var/log/rsync_netbsd_ftp_daily.log
STATUS_FILE=/export0/nbsdftp/.last_rsync_success_ftp_daily

Crontab(5) Example

Stagger your input cron(8) jobs:

seklecki@metrotitan:/usr/pkg/cf$ ls r*sh
-rwxr-x---  1 root  wheel  1155 Dec  9 11:20 rsync-common.sh

Cron Tabs:

seklecki@metrotitan:/usr/pkg/cf$ sudo crontab -l -u [meta-user]
[...snip...] 
0 *     *       *       *       /usr/pkg/cf/rsync-common.sh /usr/pkg/etc/netbsd-ftp.conf 
10 *    *       *       *       /usr/pkg/cf/rsync-common.sh /usr/pkg/etc/freebsd-ftp.conf 
20 *    *       *       *       /usr/pkg/cf/rsync-common.sh /usr/pkg/etc/netbsd-cvs.conf 
30 *   *       *       *       /usr/pkg/etc/cvsup/update.sh freebsd 2>&1 
40 *   *       *       *       /usr/pkg/cf/rsync-common.sh /usr/pkg/etc/netbsd-daily.conf 2>&1

Output Protocols

For each collection, you may share your local mirror via:

• AnonCVS via SSH
• AnonCVS via PServer
• cvsupd(8) server
• rsyncd(8)
• FTP Server
• HTTP Server
• NFS Exports

AnonCVS SSH Access for anoncvs meta User

• To establish a password less account, add the following line to /etc/pam.d/sshd:

% more /etc/pam.d/sshd

# $NetBSD: sshd,v 1.8 2005/09/22 01:02:12 tsarna Exp $
#
# PAM configuration for the "sshd" service
# auth
auth            sufficient      pam_guest.so    guests=anoncvs nopass
auth            required        pam_nologin.so  no_warn
auth            sufficient      pam_krb5.so     no_warn try_first_pass
auth            optional        pam_afslog.so   no_warn try_first_pass
#pam_ssh has potential security risks.  See pam_ssh(8).
#auth           sufficient      pam_ssh.so      no_warn try_first_pass
auth            required        pam_unix.so     no_warn try_first_pass

• Manually create a system POSIX meta-user account using vipw(8):

% vipw

anoncvs:*************:502:502::0:0::/:/usr/pkg/bin/anoncvssh

Using the AnonCVSSSH (anoncvsssh) Login Shell

The anoncvsssh shell can be used to secure the anoncvs user's access to permit CVS client calls to '/usr/bin/cvs server' only:

anoncvs.shar can be fetched from http://www.openbsd.org/anoncvs.shar. Run the script

Here is a small diff to anoncvsssh.c to customize the shell for your system:

$ diff -u anoncvssh.c anoncvssh.c.bas
--- anoncvssh.c 2009-02-20 21:43:09.000000000 -0500
+++ anoncvssh.c.bas     2009-02-20 21:43:05.000000000 -0500
@@ -53,14 +53,14 @@
 * home directory
 */
#ifndef LOCALROOT
-#define        LOCALROOT       "/cvs"
+#define        LOCALROOT       "/export/{netbsd,freebsd}/cvs"
#endif

/*
 * Hostname to be used when accessing the remote repository.
 */
#ifndef HOSTNAME
-#define        HOSTNAME        "anoncvs1.usa.openbsd.org"
+#define        HOSTNAME        "netbsd.mirror.pub.collaborativefusion.com"
#endif

/*
@@ -110,12 +110,12 @@
 * If you want to be able to run an alternate OpenCVS binary on your
 * anoncvs server, define OPENCVS_USER as the user who will invoke it.
 */
-#define OPENCVS_USER "opencvs"
+/* #define OPENCVS_USER "opencvs" */

int main __P((int, char *[]));

char * const env[] = {
-       "PATH="_PATH_DEFPATH,
+       "PATH=/usr/bin:/bin",
       "SHELL="_PATH_BSHELL,
       "CVSROOT="LOCALROOT,
       "HOME=/",

To compile:

$ sudo make
#   compile  seklecki/anoncvssh.o
cc -O2 -mfpmath=sse -msse2 -march=pentium4    -Werror      -c    anoncvssh.c
#      link  seklecki/anoncvssh
cc    -o anoncvssh  -Wl,-rpath-link,/lib:/usr/lib  -L/lib anoncvssh.o 

After installing manually in an sbin path of your choice, add the shell to /etc/shells

[ Notes Here TBD after INFOSEC discussion]

$ ls -al /usr/pkg/cf/anoncvssh ---s--x--x 1 root users 8729 Feb 20 13:14 /usr/pkg/cf/anoncvssh*

NOTE: Mode should be 4111 NOTE: You may wish to add:

EXCLD_FLAGS='--exclude CVSROOT/config'

To your update script and set:

 LockDir=/tmp

Protecting SSH

Options:

1. Run a separate sshd(8) on a service VIP in a chroot(2) for AnonCVS via SSH
2. Load DenySSH services

DenySSH Setup with pf(4)

You can configure the DenySSH package to read your system auth syslog(2) file and dynamically add botnets and GNU/Linux users to your pf(4) bogon list. DenySSH simply daemons a ruby(1) process which tail(1)'s & parses your /var/log/auth, then appends IPs to a pf(4) table of your choice. In this case, we call the table <wankers>

$ sudo ruby18 /usr/pkg/sbin/denyssh.rb -c /usr/pkg/etc/denyssh.conf -d start
Starting denyssh.

With the following configuration file:

$ egrep -iv '^#|^$' /usr/pkg/etc/denyssh.conf 
module DenySSHConfig # <-- do not remove this line
LOGFILE = '/var/log/security'
HOSTDATA = '/var/db/denyssh/hosts.yaml'
PFCTL = '/sbin/pfctl'
PF_TABLE = 'wankers'
DENY_THRESHOLD_VALID = 10
DENY_THRESHOLD_INVALID = 5
DENY_THRESHOLD_ROOT = 3
end # <-- do not remove this line

And the following provisions in pf.conf(5):

$ grep -i wankers /etc/pf.conf 
table <wankers> persist
block log quick from <wankers> to any

You can view the contents of the wankers table with pfctl(8):

$ sudo pfctl -t wankers -T show 
  12.152.124.2
  58.213.125.25
  58.253.67.58
  59.67.97.24
$ sudo pfctl -t wankers -T show|wc -l
    136

Additionally, one can enable chroot(2) features.

Example AnonCVS SSH Access

$ more ~/.cvsrc 
cvz -z0 -t
$ cvs -d anoncvs@netbsd.mirror.pub.collaborativefusion.com:/export/netbsd/cvs co -P -rHEAD src
 -> main loop with CVSROOT=anoncvs@netbsd.mirror.pub.collaborativefusion.com:/export/netbsd/cvs
 -> safe_location( where=(null) )
 -> Starting server: ssh -l anoncvs netbsd.mirror.pub.collaborativefusion.com cvs server 
The authenticity of host 'netbsd.mirror.pub.collaborativefusion.com (206.83.220.245)' can't be established.
DSA key fingerprint is f2:d2:3f:55:5b:ad:6e:d4:28:cc:f4:a3:0b:68:15:ad.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'netbsd.mirror.pub.collaborativefusion.com' (DSA) to the list of known hosts.
S-> my_module (src, Updating, , )
S-> safe_location( where=(null) )
S-> my_module (src, Updating, , )
S-> Create_Admin (., src, /export/netbsd/cvs/src, HEAD, , 0, 0, 1)
 -> Create_Admin (src, src, /export/netbsd/cvs/src, , , 0, 0, 1)

Configuring CVSUP Server

cvsupd(8) automatically builds prefixes of available systems using:

/usr/pkg/etc/cvsup/sup/*/{list.cvs,releases} 

This system builds a complex hierarchy based on inheritance, for example:

$ more /usr/pkg/etc/cvsup/sup/src-usrsbin/releases 
cvs     list=list.cvs prefix=prefixes/FreeBSD.cvs keywordprefix=/home/ncvs super=src-all
$ more  /usr/pkg/etc/cvsup/sup/src-usrsbin/list.cvs 
upgrade src/usr.sbin
omitany */#cvs.*
omitany */,*
omitany */.nfs*
omitany *.core
omitany */CVS

Like many UNIX system, the CVSUP System can be quite ambiguous/esoteric until you realize the simplicity of it.

NetBSD CVSUP Note

There is no central self-distribution point for NetBSD cvsupd(8) collection configuration files. However, the following files can be found in the archive: Media:NetBSD-cvsupd-lists.cvs_and_releases.tar

Also at this URL (Wiki Extension Problem): http://people.collaborativefusion.com/~seklecki/NetBSD-cvsupd-lists.cvs_and_releases.tar

$ ls /usr/pkg/etc/cvsup/sup/NetBSD-*/{list.cvs,releases} 
-rw-r--r--  1 root  cvsupin  324 Oct 23 13:47 /usr/pkg/etc/cvsup/sup/NetBSD-CVSROOT/list.cvs
-rw-r--r--  1 root  cvsupin   94 Oct 23 13:45 /usr/pkg/etc/cvsup/sup/NetBSD-CVSROOT/releases
-rw-r--r--  1 root  cvsupin  381 Oct 23 13:48 /usr/pkg/etc/cvsup/sup/NetBSD-cvs/list.cvs
-rw-r--r--  1 root  cvsupin   77 Oct 23 13:49 /usr/pkg/etc/cvsup/sup/NetBSD-cvs/releases
-rw-r--r--  1 root  cvsupin   90 Oct 24 14:49 /usr/pkg/etc/cvsup/sup/NetBSD-htdocs/list.cvs
-rw-r--r--  1 root  cvsupin   94 Oct 23 13:43 /usr/pkg/etc/cvsup/sup/NetBSD-htdocs/releases
-rw-r--r--  1 root  cvsupin   92 Oct 23 13:44 /usr/pkg/etc/cvsup/sup/NetBSD-othersrc/list.cvs
-rw-r--r--  1 root  cvsupin   94 Oct 23 13:44 /usr/pkg/etc/cvsup/sup/NetBSD-othersrc/releases
-rw-r--r--  1 root  cvsupin   90 Oct 23 13:44 /usr/pkg/etc/cvsup/sup/NetBSD-pkgsrc/list.cvs
-rw-r--r--  1 root  cvsupin   94 Oct 23 13:44 /usr/pkg/etc/cvsup/sup/NetBSD-pkgsrc/releases
-rw-r--r--  1 root  cvsupin   87 Oct 23 13:33 /usr/pkg/etc/cvsup/sup/NetBSD-src/list.cvs
-rw-r--r--  1 root  cvsupin   94 Oct 23 13:43 /usr/pkg/etc/cvsup/sup/NetBSD-src/releases
-rw-r--r--  1 root  cvsupin   88 Oct 23 13:44 /usr/pkg/etc/cvsup/sup/NetBSD-xsrc/list.cvs
-rw-r--r--  1 root  cvsupin   94 Oct 23 13:44 /usr/pkg/etc/cvsup/sup/NetBSD-xsrc/releases

FreeBSD CVSUP Note

The FreeBSD CVSUP system self-distributes distributes it ${prefix}/etc/cvsup/sup/*/* filesets, and they insist on referring /home/ncvs

However, the data is actually stored whatever data store mount point you have:

lrwxr-xr-x   1 root      wheel   16 Oct  9 13:57 ncvs@ -> /export1/fbsdcvs

Example CVSUP Client Configuration

An example supfile for fetching NetBSD srcs:

$ cat netbsd-cvsup.supfile
# Use Your Mirror Here
*default host=netbsd.mirror.pub.organization.tld
*default base=/var/db/sup/netbsd
*default prefix=/usr
*default release=cvs tag=HEAD
# To replicate the raw CVS repository in "Non CVS Checkout Mode", e.g. to simply replicate collection (for use with NetBSD-cvs collection below)
# *default release=self
*default delete use-rel-suffix
*default compress
NetBSD-src
#NetBSD-pkgsrc
#NetBSD-cvs  <- for use replicating the raw CVS repository
# All Standard FreeBSD Entries work as well:
# cvs-all
# ports-all
# src-all
# www

Example cvsupd(8) rc.d/ script

$ more /etc/rc.d/cvsupd

#!/bin/sh
#
# $NetBSD$
#
# PROVIDE: cvsupd
# REQUIRE: DAEMON
. /etc/rc.subr
name="cvsupd"
rcvar=$name
command="/usr/pkg/sbin/${name}"
#config="/usr/pkg/etc/nagios/${name}.cfg"
#pidfile="/var/run/${name}.pid"
cvsupd_user="cvsup"
command_args="-e -C 8 -l @local0 -b /usr/pkg/etc/cvsup -s sup-client-scan-deltas"
# config files are actually inin -b {dir/}/base by implication; one for each collection
load_rc_config $name
run_rc_command "$1"

Adding IPv6 Support to cvsupd(8)

IPv6 libc(3) bindings are missing in Modula-3 on *BSD, so we have to use a few cheap hacks to enable v6:

• At the network perimeter: Use faithd(8) to map a /96 of your v6 space to your v4 space at a dual-homed router near your network perimeter, then announce the AAAA of a translated IP address of your server (too much to get into here; see the NetBSD docs on keeping the faith(4))
• At the host level, use Pkgsrc's net/netcat6 to map an incoming TCP6 listener daemon into a localhost V4 TCP client

Example:

$ sudo nohup nc6 -6 -l -p 5999 --continuous --exec nc6 -4 127.0.0.1 5999
$ netstat -tan -f inet6 |grep -i 5999
 tcp6       0      0  *.5999                 *.*                    LISTEN

From a V6 client:

client$ telnet freebsd.mirror.pub.tld 5999
 Trying 2607:f000:10::4002...
 Connected to freebsd.mirror.pub.
 Escape character is '^]'.
 OK 17 0 SNAP_16_1h CVSup server ready

On the server you will see:

$ netstat -tan  |grep -i 5999|grep -i est
 tcp        0      0  127.0.0.1.5999         127.0.0.1.63610        ESTABLISHED
 tcp        0      0  127.0.0.1.63610        127.0.0.1.5999         ESTABLISHED
 tcp6       0      0  2607:f000:10::40.5999  2607:f000:10:202.40619 ESTABLISHED

Configuring RSYNC Server

Setting up rsyncd(8) collections is straightforward:

$ more rsync/rsyncd.conf 
uid = cvsup
gid = cvsup
use chroot = yes
max connections = 8
syslog facility = local1
pid file = /var/run/rsyncd.pid

[NetBSD-CVS]
        path = /export0/nbsdcvs
        comment = NetBSD SRC, XSRC, HTDOC, PKGSRC 
        read only = yes
[FreeBSD-CVS]
       path = /export1/fbsdcvs
       comment = FreeBSD SRC, WWW, Project, Distrib 
       read only = yes
[FreeBSD-FTP]
        path = /export1/fbsdftp/pub/FreeBSD
        comment = FreeBSD FTP Mirror (from FBSD US FTP #12 via rsync)
        read only = yes 
[NetBSD-FTP]
        path = /export0/nbsdftp/pub/NetBSD
        comment = NetBSD FTP Mirror (from rsync rsync.jp.NetBSD.org)
        read only = yes

Note: The net/rsync package in Pkgsrc lacks and rc.d/ script. Here is an example.

$ more /etc/rc.d/rsyncd
#!/bin/sh
#
# $NetBSD$
#
# PROVIDE: rsyncd
# REQUIRE: DAEMON

. /etc/rc.subr

name="rsyncd"
rcvar=$name
command="/usr/pkg/bin/rsync"
#config="/usr/pkg/etc/nagios/${name}.cfg"
pidfile="/var/run/${name}.pid"

command_args="--daemon"

load_rc_config $name
run_rc_command "$1"

Example RSYNC Client Access

Because rsyncd(8) doesn't support vhosting, connect to any IP:

$ rsync freebsd.mirror.pub.organization.tld::
NetBSD-CVS     	NetBSD SRC, XSRC, HTDOC, PKGSRC
FreeBSD-CVS    	FreeBSD SRC, WWW, Project, Distrib
FreeBSD-FTP    	FreeBSD FTP Mirror (from FBSD US FTP #12 via rsync)
NetBSD-FTP     	NetBSD FTP Mirror (from rsync rsync.jp.NetBSD.org)

$ rsync freebsd.mirror.pub::FreeBSD-FTP
drwxr-xr-x        1024 2009/01/19 00:32:36 .
-rw-rw-r--         263 2006/05/27 15:30:21 .message
-r--rw-r--           0 1996/11/07 14:55:33 .notar
lrwxrwxrwx          15 2007/04/19 13:28:24 CTM
lrwxrwxrwx          17 2007/04/19 13:28:24 CVSup
lrwxrwxrwx          17 2007/04/19 13:28:24 FreeBSD-current
lrwxrwxrwx          19 2007/04/19 13:28:24 FreeBSD-stable
lrwxrwxrwx          25 2007/04/19 13:28:24 ISO-IMAGES-alpha
[...]

Configuring an Anonymous FTP Server

The recommended method is to utilize an FTP daemon that supports chroot() such as ProFTPD:

$ egrep -iv '^#|^$' /usr/pkg/etc/proftpd.conf 
ServerName                      "My FOSS Mirror"
ServerType                      standalone
DefaultServer                   on
PassivePorts    65000 65500
Port                            21
UseIPv6                         on
IdentLookups off
SyslogFacility          LOCAL2
Umask                           022
MaxInstances                    30
User                            nobody
Group                           nogroup
DefaultRoot /export
AllowOverwrite          on
<VirtualHost netbsd.mirror.pub.organization.tld>
 ServerName                    "My NetBSD Mirror"
 PassivePorts  65000 65500
 <Anonymous /export0/nbsdftp>
  User                          ftp
  Group                         ftp
  # We want clients to be able to login with "anonymous" as well as "ftp"
  UserAlias                     anonymous ftp
  RequireValidShell             off
  # Limit the maximum number of anonymous logins
  MaxClients                    10
  ShowSymlinks            on
  # We want 'welcome.msg' displayed at login, and '.message' displayed in each newly chdired directory.
  DisplayLogin                  welcome.msg
  DisplayChdir                  .message
  # Limit WRITE everywhere in the anonymous chroot
  <Limit WRITE>
   DenyAll
  </Limit>
 </Anonymous>
</VirtualHost>

Note: The pkgsrc/net/proftpd package does not automatically create an FTP user, so create a restricted user:

$ finger ftp
Login: ftp                              Name: FTP User
Directory: /export                      Shell: /sbin/nologin
Never logged in.
No Mail.
No Plan.

Use vipw(8):

ftp:*************:503:100::0:0:FTP User:/export:/sbin/nologin

Configuring an HTTP Server

Create a standard VHOST and <Directory> statement for your data storage root:

$ cat /usr/pkg/etc/httpd/httpd-vhosts.conf

NameVirtualHost *:80
Include etc/httpd/httpd-autoindex.conf
<Directory /export>
       Options FollowSymLinks Indexes 
   Order allow,deny
   Allow from all
</Directory> 
<VirtualHost *:80>
   ServerAdmin bofh@collaborativefusion.com
   DocumentRoot "/export/netbsd"
   ServerName netbsd.mirror.pub.collaborativefusion.com
   ErrorLog "/var/log/httpd/mirror-netbsd-error.log"
   CustomLog "/var/log/httpd/mirror-netbsd-access.log" common
</VirtualHost>

$ grep -i http /etc/newsyslog.conf

/var/log/httpd/mirror-netbsd-error.log       640  16   *    24   Z  /var/run/httpd.pid
/var/log/httpd/mirror-netbsd-access.log      640  16   *    24   Z  /var/run/httpd.pid

Rate Limiting Link Saturation using pf(4)

Although it is not possible to rate-limit traffic for incoming traffic to a host machine, such as outbound TCP connections that pull data down from RSYNC sources, you can rate limit how much bandwidth your outgoing connections from clients consume (which are bigger bandwidth hogs the majority of the time)

With a default-block policy in your pf.conf(5), you can create explicit pass rules for your management IP and optional service VIPs. The rules related to output protocols, which are generally viewed as incoming TCP connections, can be placed in a special queue of your liking using the simple template below:

ext_if="wm0"
srvc_addr="{129.144.0.100/32, 129.144.0.101/32}"
srvc_addr_v6="{2607:6666:1000::1234/128}" 
altq on $ext_if cbq queue { default, mirror_srvc}
       queue default bandwidth 4096Kb cbq(default borrow red)
       queue mirror_srvc bandwidth 4096Kb cbq(red)
set skip on lo0
[...snip...]]
#Limit outgoing bandwidth of TCP connections accepted in to mirror serviecs
pass in quick on $ext_if inet proto tcp from any to $srvc_addr port {$p_rsync,$p_http,$p_cvsup,$p_ssh,ftp,65000:65500,$p_dns} flags S/SA keep state queue mirror_srvc
pass in quick on $ext_if inet6 proto tcp from any to $srvc_addr_v6 port {$p_rsync,$p_http,$p_cvsup,$p_ssh,ftp,65000:65500,$p_dns} flags S/SA keep state
block log inet all
block log inet6 all

Security

See Mirring NetBSD/Server Security Review Subsection