Web Proxy
| Document revision: | 1.8 (Wed Apr 28 13:56:22 GMT 2004) |
| Applies to: | V2.8 |
General Information
Summary
The MikroTik RouterOS implements the following proxy server features:
- Regular HTTP proxy
- Transparent proxy. Can be transparent and regular at the same time
- Access list by source, destination, URL and requested method
- Cache access list (specifies which objects to cache, and which not)
- Direct Access List (specifies which resources should be accessed directly, and which - through another proxy server)
- Logging facility
Specifications
Packages required: web-proxyLicense required: Level3
Submenu level: /ip web-proxy
Standards and Technologies: HTTP/1.0, HTTP/1.1, FTP
Hardware usage: uses disk space, if available (see description below)
Related Documents
Description
When setting up Web proxy, make sure it serves only your clients, and is not misused as relay. Please read the security notice in the Access List Section!
Note that it may be useful to have Web proxy running even with no cache when you want to use it as something like HTTP and FTP firewall (for example, denying access to mp3 files) or to redirect requests to external proxy transparently
Setup
Submenu level: /ip web-proxyProperty Description
src-address (IP address; default: 0.0.0.0) - the web-proxy will use this address connecting to the parent proxy or web site.rebuilding-cache - proxy is enabled and running, existing cache is being verified
running - proxy is enabled and running
stopping - proxy is shutting down (max 10s)
clearing-cache - proxy is stopped, cache files are being removed
creating-cache - proxy is stopped, cache directory structure is being created
dns-missing - proxy is enabled, but not running because of unknown DNS server (you should specify it under /ip dns)
invalid-address - proxy is enabled, but not running because of invalid address (you should change address or port)
invalid-cache-administrator - proxy is enabled, but not running because of invalid cache-administrator's e-mail address
invalid-hostname - proxy is enabled, but not running because of invalid hostname (you should set a valid hostname value)
error-logged - proxy is not running because of unknown error. This error is logged as System-Error. Please, send us this error and some description, how it happened
reserved-for-cache (integer) - maximal cache size, that is accessible to web-proxy
Notes
By default the proxy cache can use as much disk space as there is allocated for it. When the system allocates the space for the proxy cache, 1/7th of the total partition (disk) size is reserved for the system, but not less than 50MB. The rest is left for the proxy cache. The system RAM size is considered as well when allocating the cache size. The cache size is limited so, that there are at least 15MB of RAM per 1GB of cache plus 55MB of RAM is reserved for the system. max-cache-size is also taken in account, so the cache will not occupy more than it is specified in this property. The effective limit is calculated as a minimum of all three limits.
Considering the previous note, you should be aware that you will not be able to enable web proxy, if you have less than 60MB of RAM on your router
Expire time of cache entries can be different for each HTML page (specified in headers). But, if there is no such header, the entry will be considered fresh for not more than 72 hours.
The address argument has been removed so the proxy listens to all IP addresses that the router has in its IP address list now.
Example
To enable the proxy on port 8080:
[admin@MikroTik] ip web-proxy> set enabled=yes src-address=0.0.0.0:8080
[admin@MikroTik] ip web-proxy> print
enabled: no
src-address: 0.0.0.0
port: 8080
hostname: proxy
transparent-proxy: no
parent-proxy: 0.0.0.0:0
cache-administrator: webmaster
max-object-size: 4096 kB
cache-drive: system
max-cache-size: 12
status: rebuilding-cache
reserved-for-cache: 9 MB
[admin@MikroTik] ip web-proxy>
Monitoring
Command name: /ip web-proxy monitorProperty Description
status (read-only: text) - the same as for /ip web-proxy print uptime (read-only: time) - shows uptime of the proxy server clients (read-only: integer) - total number of proxy clients with different IP addresses during last uptime requests (read-only: integer) - total number of proxy requests during last uptime hits (read-only: integer) - total number of requests satisfied by proxy's cache during last uptime cache-size (read-only: integer) - current cache size in kilobytes received-from-servers (read-only: integer) - shows the amount in kilobytes the proxy received from remote servers during last uptime sent-to-clients (read-only: integer) - shows the amount in kilobytes the proxy sent to the clients to resolve their requests during last uptime hints-sent-to-clients (read-only: integer) - shows how much outgoing traffic was taken from the cache during last uptimeExample
[admin@MikroTik] > ip web-proxy monitor
status: running
uptime: 4d19h8m14s
clients: 9
requests: 10242
hits: 3839
cache-size: 328672 kB
received-from-servers: 58108 kB
sent-to-clients: 65454 kB
hits-sent-to-clients: 7552 kB
[admin@MikroTik] >
Access List
Submenu level: /ip web-proxy accessDescription
Access list is implemented in the same way as MikroTik firewall rules. Rules are processed from the top to the bottom. First matching rule specifies decision of what to do with this connection. Connections can be matched by its source address, destination address, destination port, substring of requested url or request method. If none of these parameters is specified, every connection will match this rule.
If connection is matched by a rule, action property of this rule specifies whether connection will be allowed or not. If connection does not match any rule, it will be allowed.
Property Description
src-address (IP address mask; default: 0.0.0.0/0) - source address of the IP packet dst-address (IP address mask; default: 0.0.0.0/0) - destination address of the IP packet dst-port (text; default: "") - a list of destination ports url (text) - the URL of the request (regular expression) method (any | connect | delete | get | head | options | post | put | trace; default: any) - HTTP method used in the request (see HTTP Methods section in the end of this document) action (allow | deny; default: allow) - specifies the action to perform on matched packetsNotes
There is one rule by default, that disallows connect method connections to ports other than 443 (https) and 563 (snews). connect method is a security hole that allows connections (transparent tunneling) to any computer using any protocol. It is used mostly by spammers, as they found it very convinient to use others' mail (SMTP) servers as anonymous mail relay to send spam over the Internet.
It is strongly recommended to deny all IP addresses except those behind the router as the proxy still may be used to access your internal-use-only (intranet) web servers. Also, consult examples in Firewall Manual on how to protect your router.
Details about regular expressions used in url field can be found here: http://www.cs.utah.edu/dept/old/texinfo/regex/regex_toc.html
Example
The default rule:
[admin@MikroTik] ip web-proxy access> print
Flags: X - disabled
0 ;;; allow CONNECT only to SSL ports 443 [https] and 563 [snews]
src-address=0.0.0.0/0 dst-address=0.0.0.0/0 dst-port=!443,563 url=""
method=connect action=deny
[admin@MikroTik] ip web-proxy access>
To disallow download of .MP3 files and FTP connections other than from the 10.0.0.1 server:
[admin@MikroTik] ip web-proxy access> add url="\\.mp3$" action=deny
[admin@MikroTik] ip web-proxy access> add src-address=10.0.0.1/32 action=allow
[admin@MikroTik] ip web-proxy access> add url="ftp://" action=deny
[admin@MikroTik] ip web-proxy access> print
Flags: X - disabled
0 ;;; allow CONNECT only to SSL ports 443 [https] and 563 [snews]
src-address=0.0.0.0/0 dst-address=0.0.0.0/0 dst-port=!443,563 url=""
method=connect action=deny
1 src-address=0.0.0.0/0 dst-address=0.0.0.0/0 dst-port="" url="\.mp3$"
method=any action=deny
2 src-address=10.0.0.1/32 dst-address=0.0.0.0/0 dst-port="" url=""
method=any action=allow
3 src-address=0.0.0.0/0 dst-address=0.0.0.0/0 dst-port="" url="ftp://"
method=any action=deny
[admin@MikroTik] ip web-proxy access>
Notes:
- \\ symbol sequence is used to enter \ character
- \. pattern means . only (in regular expressions single dot in pattern means any symbol)
- to show that no symbols are allowed before the given pattern, we use ^ symbol at the beginning of the pattern
- to specify that no symbols are allowed after the given pattern, we use $ symbol at the end of the pattern
Direct Access List
Submenu level: /ip web-proxy directDescription
If parent-proxy property is specified, it is possible to tell proxy server whether to try to pass the request to the parent proxy or to resolve it connecting to the requested server directly. Direct Access List is managed just like Proxy Access List described in the previous chapter except the action argument.
Property Description
src-address (IP address mask; default: 0.0.0.0/0) - source address of the IP packet dst-address (IP address mask; default: 0.0.0.0/0) - destination address of the IP packet dst-port (text; default: "") - a list of destination ports url (text) - the URL of the request (regular expression) method (any | connect | delete | get | head | options | post | put | trace; default: any) - HTTP method used in the request (see HTTP Methods section in the end of this document) action (allow | deny; default: allow) - specifies the action to perform on matched packetsdeny - resolve these requests through parent proxy if there is one. If there in no parent proxy, the action will be the same as for allow
Notes
The default action (if no rules specified or a request did not match any) is deny.
Managing the Cache
Submenu level: /ip web-proxy cacheDescription
Cache access list specifies, which requests (domains, servers, pages) have to be cached locally by web proxy, and which not.
Access list is implemented exactly the same way as web proxy access list. Default action is to cache object (if no matching rule is found).
Property Description
src-address (IP address mask; default: 0.0.0.0/0) - source address of the IP packet dst-address (IP address mask; default: 0.0.0.0/0) - destination address of the IP packet dst-port (text; default: "") - a list of destination ports url (text) - the URL of the request (regular expression) method (any | connect | delete | get | head | options | post | put | trace; default: any) - HTTP method used in the request (see HTTP Methods section in the end of this document) action (allow | deny; default: allow) - specifies the action to perform on matched packetsNotes
There is one cache access rule added by default:
[admin@MikroTik] ip web-proxy cache> print
Flags: X - disabled
0 src-address=0.0.0.0/0 dst-address=0.0.0.0/0 dst-port=""
url="cgi-bin \?" method=any action=deny
[admin@MikroTik] ip web-proxy cache>
This rule defines that all runtime generated pages (which are located within cgi-bin directories or contain ? in url) have not to be cached.
Objects, which are larger than max-object-size, are not cached.
Rebuilding the Cache
Command name: /ip web-proxy clear-cacheDescription
Web proxy will automatically detect any problems with cache and will try to solve them without loosing any cache data. But in case of a heavy damage to the file system, the web proxy can't rebuild cache data. Cache can be deleted and new cache directories created using this feature.
Example
[admin@MikroTik] ip web-proxy> set enabled=no [admin@MikroTik] ip web-proxy> clear-cache Clear all web proxy cache, yes? [y/N]: y cache will be cleared shortly [admin@MikroTik] ip web-proxy>
Transparent Mode
Description
Transparent proxying does caching of web contents "transparently" to the end-user. Id est he (or she) does not know about the proxy being enabled and therefore does not need to provide any additional setting to his (her) web client.
To enable the transparent mode, firewall rule in destination NAT has to be added, specifying which connections (to which ports) should be transparently redirected to the proxy.
Notes
Only HTTP traffic is supported in web proxy transparent mode. HTTPS and FTP are not going to work this way.
Example
For example, if we want all connections coming from ether1 interface to port 80 to be handled transparently by web proxy, and if our web proxy is listening on port 8080, then we should add the following destination NAT rule:
[admin@MikroTik] ip firewall dst-nat> add in-interface=ether1 protocol=tcp \
dst-address=!10.0.0.1/32:80 action=redirect to-dst-port=8080
[admin@MikroTik] ip firewall dst-nat> print
Flags: X - disabled, I - invalid
0 src-address=0.0.0.0/0:0-65535 in-interface=ether1
dst-address=!10.0.0.1/32:80 protocol=tcp icmp-options=any:any flow=""
src-mac-address=00:00:00:00:00:00 limit-count=0 limit-burst=0
limit-time=0s action=redirect to-dst-address=0.0.0.0 to-dst-port=8080
[admin@MikroTik] ip firewall dst-nat>
Here, the router's address and port 80 (10.0.0.1/32:80) have been excluded from redirection to preserve the Winbox functionality which uses TCP port 80 on the router. More than one redirect rule can be added to redirect more than one port.
HTTP Methods
Description
OPTIONSThis method is a request of information about the communication options available on the chain between the client and the server identified by the Request-URI. The method allows the client to determine the options and (or) the requirements associated with a resource without initiating any resource retrieval
GETThis method retrieves whatever information identified by the Request-URI. If the Request-URI refers to a data processing process than the response to the GET method should contain data produced by the process, not the source code of the process procedure(-s), unless the source is the result of the process.
The GET method can become a conditional GET if the request message includes an If-Modified-Since, If-Unmodified-Since, If-Match, If-None-Match, or If-Range header field. The conditional GET method is used to reduce the notwork traffic specifying that the transfer of the entity should occure only under circumstances described by conditional header field(-s).
The GET method can become a partial GET if the request message includes a Range header field. The partial GET method intends to reduce unnecessary network usage by requesting only parts of entities without transfering data already held by client.
The response to a GET request is cacheable if and only if it meets the requirements for HTTP caching.
HEADThis method shares all features of GET method except that the server must not return a message-body in the responce. This retrieves the metainformation of the entity implied by the request which leads to a wide usage of it for testing hypertext links for validity, accessibility, and recent modification.
The response to a HEAD request may be cacheable in the way that the information contained in the responce may be used to update priviously cached entity identified by that Request-URI.
POSTThis metod requests that the origin server accept the entity enclosed in the request as a new subordinate of the resource identified by the Request-URI.
The actual action performed by the POST method is determined by the origin server and usually is Request-URI dependent.
Responses to POST method are not cacheable, unless the response includes appropriate Cache-Control or Expires header fields.
PUTThis method requests that the enclosed entity be stored under the supplied Request-URI. If another entity exists under specified Request-URI, the enclosed entity should be considered as updated (newer) version of that residing on the origin server. If the Request-URI is not pointing to an existing resource, the origin server should create a resource with that URI.
If the request passes through a cache and the Request-URI identifies one or more currently cached entities, those entries should be treated as stale. Responses to this method are not cacheable.
TRACEThis method invokes a remote, application-layer loop-back of the request message. The final recipient of the request should reflect the message received back to the client as the entity-body of a 200 (OK) response. The final recipient is either the origin server or the first proxy or gateway to receive a Max-Forwards value of 0 in the request. A TRACE request must not include an entity.
Responses to this method MUST NOT be cached.