Taming Tomcat, Day 3: Security and Perfomance

Author and presenter: Simon Brooke.

The full text of this presentation is online at <URL: http://www.weft.co.uk/library/tomcat/>

Written March-April 2006; $Revision: 1.6 $ of $Date: 2006-04-28$

Changes to the presentation since your handouts were printed are highlighted like this.

Simon Brooke, 21 Main Street, Auchencairn, DG7 1QU, Scotland.

back Day 2: Development and Administration

Programme for Today

This morning
- Before break
  - Security (1): On the host
- After break
  - Securiity (2): Over the network
  - Security (3): hardware failure

This afternoon
- Before break
  - Performance Tuning
- After break
  - Wrap

Security on the host

File system security
Tomcat's own security
Keeping separate apps separate
Recommendations

File System Security

Tomcat exposes potential entry points to potentially hostie attackers
- So do all dynamic content systems, this isn't something that's peculiar to Tomcat
We need to limit what damage an attacker could do
We need to limit what damage a careless programmer could do
File system security is important in this

The JVM and file system security

Tomcat runs in a Java virtual machine
There should be no reason for it to run programs on the real machine
The account Tomcat runs as need not be able to access (e.g.) /usr/bin

Recommendations: users and groups

Create a special user account (e.g., tomcat) to run Tomcat
- Need not be able log in
- Need not be able to start a shell
- Must be able to read, but not write, its conf and shared directories
- Must be able to read and write its own webapps,work and temp directories
- Must be able to write to its own logs directory
- Other than that, can (and should) have very limited privilege
Create a special group of users (e.g. webapps) who are authorised to alter the content of Webapps
- Should be able to write to the webapps directory...
  - ... and subdirectories of it
- So the webapps directory should be
  - owned by user tomcat
  - belong to group webapps
  - be readable, writable and searchable by user and group
  - have setUID bit set
  - have setGID bit set
- NOTE: This is my recommendation. It is not the default!
Create a special group of users (e.g. tc-admin) who are authorised to alter Tomcat configuration
- Should be able to read and write the Tomcat conf directory

Users and groups: summary

ls -l /var/lib/tomcat4
total 16
dr-xrwx---  2 tomcat  tc-admin 4096 Apr 27 17:35 conf
drwxr-x---  2 tomcat  root     4096 Apr 27 17:35 logs 
drwxr-xr-x  4 root    root     4096 Apr 25 13:46 shared
drwxr-xr-x  2 tomcat  root     4096 Apr 27 17:35 temp
drwsrwsr-x  7 tomcat  webapps  4096 Apr 27 13:41 webapps
drwx------  3 tomcat4 root     4096 Apr 25 16:35 work

Tomcat's own security

Realms, users and roles
The Security Manager

Realms, users and roles

A 'realm' is a webapp or group of webapps within a common security cordon
To enter a realm, a user must show the right authentication tokens
- And be a member of the right role
Moving within the realm, the user should not be challenged again

We've encountered these already

We've encountered these already when we set up the manager and admin webapps
We had to add entries to the tomcat-users.xml file
The manager webapp exists in a realm which only admits people with the role manager

Type of Realm

As we've seen, realms can store authenticating information in an XML file
- known as a MemoryRealm
- Wouldn't use this in a production environment
Alternatives are
- JDBCRealm stores authenticating information in an SQL database
- DataSourceRealm is similar, but the database is connected by JNDI
- JNDIRealm stores authentication in an LDAP directory
- JAASRealm (Tomcat 5.0 and later) authenticates against a Java Authentication & Authorization Service
  - TODO: What's that?

Configuring the Realm

The realm is configured in conf/server.xml
Realm element may be used as child of
- Engine (group of virtual hosts)
  - This Realm will be shared across ALL web applications on ALL virtual hosts, UNLESS it is overridden by a Realm element nested inside a subordinate Host or Context element.
- Host (single virtual host)
  - This Realm will be shared across ALL web applications for THIS virtual host, UNLESS it is overridden by a Realm element nested inside a subordinate Context element
- Context (single webapp)
  - This Realm will be used ONLY for THIS web application.
Realm element has varying attributes depending on the realm type chosen
Realm element is empty - it has no children

Summing up on Realms

If you're running Tomcat behind Apache, remember that Apache can do the same thing
- if you are doing user authentication in Apache HTTPD
- and your users can't bypass Apache HTTPD
You probably want to do this sort of thing in as few places as possible
- You certainly want to maintain as few user authentication repositories as possible
If you have a central LDAP service, you probably want to use that

Keeping separate apps separate

Tomcat runs in a JVM
All webapps run in the same VM
- As same OS-level user
Not utterly impossible to access internal structures
For ultimate inter-application security, run separate webapps in separate Tomcat instances
- Using separate user accounts

The `SecurityManager`

Without the security manager, a component within a Webapp could damage other webapps or Tomcat itself
- e.g. if a JSP file contained <% System.exit(1) %> it would stop Tomcat immediately every time a user looked at it
- Webapps could over-write one another's configuration
The SecurityManager is there to set and enforce policy on what components can do what to what

The `SecurityManager` (ii)

The SecurityManager is not part of or unique to Tomcat
- It is the same intrinsic Java mechanism which prevents Applets having access to resources on the client a Web browser runs on
- Why Applets are much safer than DirectX components, for example
- So it is well tested
The SecurityManager grants permissions to Java classes
- Wherever they're called from
- Whoever is running them

Permission classes

Standard Java contains all the Permission classes needed to make Applet security work
- java.util.PropertyPermission - Controls read/write access to JVM properties such as java.home.
- java.lang.RuntimePermission - Controls use of some System/Runtime functions like exit() and exec(). Also control the package access/definition.
- java.io.FilePermission - Controls read/write/execute access to files and directories.
  - Note that if you grant access to files here, and deny it in the file system, you still can't access them
  - And vice versa
  - Huge scope for confusion, can makes things very difficult to debug
  - However, does allow you to set inter-application security
- java.net.SocketPermission - Controls use of network sockets.
- java.net.NetPermission - Controls use of multicast network connections.
- java.lang.reflect.ReflectPermission - Controls use of reflection to do class introspection.
- java.security.SecurityPermission - Controls access to Security methods.
- java.security.AllPermission - Allows access to all permissions, just as if you were running without a SecurityManager.

Custom Permission classes

You can write your own Permission classes
I suspect this is tricky
Tomcat comes with one custom Permission class, org.apache.naming.JndiPermission

The policy file(s)

By default there's just one, conf/catalina.policy
On Debian there's a directory /etc/tomcat[4|5]/policy.d with a number of smaller files
- These are read in alphanumeric collating sequence as Tomcat starts
- Allows small files which address particular policy areas, rather than one big one
Format is plain text, comment lines begin with double slash

Policy rule syntax

General syntax:

grant [signedBy <signer>,] [codeBase <code source>] {

  permission <class> [<name> [, <action list>]];

};

Examples:

// The permissions granted to the context root directory apply to JSP pages.
grant codeBase "file:${catalina.home}/webapps/examples/-" {
      permission java.net.SocketPermission "dbhost.mycompany.com:5432", "connect";
      permission java.net.SocketPermission "*.noaa.gov:80", "connect";
};

// These permissions apply to the privileged admin and manager web applications
grant codeBase "file:${catalina.home}/server/webapps/admin/WEB-INF/classes/-" {
      permission java.security.AllPermission;
};

Editing policy files

Plain text, so easy enough to edit
Put Java comes with a GUI policy file editor, policytool
- You'll find it in $JAVA_HOME/bin
Editing them isn't the tricky bit. Working out what permissions to set is the tricky bit.

Debugging policy problems

As you will have seen by now I find working with Tomcat's SecurityManager difficult and frustration inducing
There is an environment variable you can set which turns on extensive debugging
- CATALINA_OPTS=-Djava.security.debug=all
Unfortunately the way we normally start Tomcat it's pretty difficult to pass an environment variable
- On Debian you can set it in /etc/default/tomcat[4|5]
Very copious output appears in the catalina_[date].log
- grep for FAILED

How secure is security?

To quote Tomcat's own SecurityManager HOWTO:
WARNING - A security audit have been conducted using the Tomcat 5 codebase. Most of the critical package have been protected and a new security package protection mechanism has been implemented. Still, make sure that you are satisfied with your SecurityManager configuration before allowing untrusted users to publish web applications, JSPs, servlets, beans, or tag libraries. However, running with a SecurityManager is definitely better than running without one.

Recommendations

Tomcat's own security is extremely fine grained
- But you rarely need that granularity
- Very difficult to set good policy at that level of granularity
Conflicts between Tomcat security and file system security are extremely hard to debug
I now run with the minimum of Tomcat security, and concentrate on getting file system security right

Exercise: Set up a Realm authenticated against a database

Protect your firstjsp webapp with a JDBCRealm
Use the documentation for the version of Tomcat you are using
For bonus points, use digest passwords
Make sure you have the database driver jar file in the tomcat shared directory!

[Break]

Security on the net

Exposed ports
Secure Sockets Layer

Exposed ports

To talk HTTP to a client, you need to expose a port
Every exposed port is a potential attack point for hostiles
Don't expose more ports than you need to
(Teaching my grandmother to suck eggs)

Exposed ports: what do you have to expose?

None (recommended).
- Use Apache HTTPD as your exposed server on port 80 (443 for SSL)
  - Apache HTTPD is exceptionally well audited against vulnerabilities
- Expose one port on the Tomcat host for AJP13 (8009)
- Expose one port on the Tomcat host for shutdown (8005)
- Block both at the firewall
One.
- Expose one port on the Tomcat host for HTTP (80, or 8080, or 8180)
- Expose one port for shutdown (8005)
  - Block it at the firewall

Secure Sockets Layer

Brief introducton to Secure Sockets Layer
Using Apache HTTPD (recommended)
Using Tomcat directly

Brief Introduction to Secure Sockets Layer

Secure Sockets Layer provides encrypted communication between two agents
One or both offer the other a digitally signed certificate as proof of authenticity
Usually the server offers the certificate
- Client certificates are also useful if e.g. you have roving clients
Not just for HTTP - almost any sockets-layer protocol can be layered over SSL
- Increasingly, things like database connections are
If you are passing authentication tokens over an open network you really should use SSL

Secure Sockets Layer using Apache HTTPD

Apache HTTPD is extremely good at handling secure communications
- Handles most of the secure websites on the 'Net
Your operating system almost certainly has a package to set SSL up on Apache HTTPD
Setting it up by hand isn't hard
- Documentation here
Use AJP13 to communicate between Apache HTTPD and Tomcat
- Tomcat doesn't have to know anything about SSL
- Much easier

Secure Sockets Layer using Tomcat directly

Advise at least Tomcat 5
- can be done with older versions, but more complex
Advise JDK at least 1.4
- if you use 1.3 you must install JSSE 1.0.3
- gets messy

Secure Sockets Layer with Tomcat directly: HOWTO

Create a certificate keystore by executing the following command:
- Windows:
  - %JAVA_HOME%\bin\keytool -genkey -alias tomcat -keyalg RSA
- Unix:
  - $JAVA_HOME/bin/keytool -genkey -alias tomcat -keyalg RSA

and specify a password value of "changeit"
TODO: Check this. Sounds dodgy to me

Uncomment the SSL HTTP/1.1 Connector entry in server.xml and tweak as necessary.

This is what it should look like

<Connector port="8443" 
               maxThreads="150" minSpareThreads="25" maxSpareThreads="75"
               enableLookups="false" disableUploadTimeout="true"
               acceptCount="100" debug="0" scheme="https" secure="true"
               clientAuth="false" sslProtocol="TLS" />

Further documentation here

Exercise: Protect one of your Webapps with Secure Sockets Layer

Protect one of your webapps with Secure Sockets Layer (https)
You should have one Webapp which is protected, and one Webapp which is not protected
You may set SSL up either in Tomcat or in Apache
- If you do it in Apache you must makre sure that users can't use port 8080 or 8081 as a 'back door' to your Webapp

Lunch

Security against failures: Load balancing and failover

Clustering
Load Balancing
Failover

Clustering Tomcats

Load balancing and failover both require clustering
- Once you've got a cluster, failover comes free
To run on a cluster, your applications must all be distributable

Clustering: Quick HOWTO

All your session attributes must implement java.io.Serializable
Uncomment the Cluster element in server.xml
Uncomment the Valve(ReplicationValve) element in server.xml
If your Tomcat instances are running on the same machine, make sure the tcpListenPort attribute is unique for each instance.
- Better still, don't run them on the same machine
Make sure the web.xml for each application has the <distributable/> element
- and each application actually is distributable!

Important Gotchas

Remember that your session state is tracked by a cookie, so your URL must look the same from the out side otherwise, a new session will be created.
Clustering support currently requires the JDK version 1.4 or later.
Exactly the same URLs must work for all machines in the cluster
- and for all applications in the cluster

Session replication

To enable session replication in Tomcat, three different paths can be followed to achieve the exact same thing:
- Using session persistence, and saving the session to a shared file system (PersistenceManager)
- Using session persistence, and saving the session to a shared database (JDBCManager)
- Using in-memory-replication, using the SimpleTcpCluster that ships with Tomcat 5 (server/lib/catalina-cluster.jar)
Algorithm only efficient for quite small clusters
Detailed documentation here

Load Balancing

Share the load between many Tomcats running identical copies of identical applications
Three techniques
- With mod_jk
- With mod_proxy and mod_rewrite
- With the balancer webapp

Load balancing with `mod_jk`

class="changed"In the workers.properties file, define a load-balancing factor for each worker
class="changed"Must be greater than 0, the higher it is the more work given to this worker
worker.local.lbfactor=1

Almost certainly the simplest and best way to do it.

Load balancing with `mod_proxy` and `mod_rewrite`

mod_rewrite is one of the coolest, most powerful, and most dangerous features of Apache HTTPD
- Generalised URL rewriting engine
- Driven by pattern matching
- Despite the tons of examples and docs, mod_rewrite is voodoo. Damned cool voodoo, but still voodoo.
- The great thing about mod_rewrite is it gives you all the configurability and flexibility of Sendmail. The downside to mod_rewrite is that it gives you all the configurability and flexibility of Sendmail.
Load balancing this way is described as 'simple' and 'easy to set up'
Detailed documentation here

Load balancing with the `balancer` webapp

First appeared in Tomcat 5.0.15
The curse of open source packages...
- Sounds good from what documentation there is
- Not enough documentation to actually use it...

What the documentation doesn't say

Can't actually share identical requests between mirrored servers
Rules tried in sequential order until one matches
Rule file syntax:
- An XML file whose root element is rules with children
  - 1 or many rule with attributes
    - className (required) whose value is the fully qualified name of a Java class which implements org.apache.webapp.balancer.Rule and is available on the classpath
    - targetString (optional) whose value is a proper substring of the URL
    - paramName (optional) whose value is the name of some parameter passed in the query part of the URL
    - paramValue (optional) whose value is the value of some parameter passed in the query part of the URL
    - redirectURL (required) whose value is the URL to redirect to
  - But actual attributes read depend on the rule class used.
  - Other rule classes may expect/need different attributes

`balancer` webapp rules file: Available classes

AcceptEverythingRule
- Use last, as a default
BaseRule
- Does nothing, superclass to all the others
CharacterEncodingRule
- Redirects requests based on the character encoding they asked for
RemoteAddressRule
- Redirects requests bases on the address of the client they came from
RequestAttributeRule
- This rule accepts or rejects requests based on the presence of a attribute in the request.
RequestHeaderRule
- This rule checks for the presence of a specific request header, optionally with a specific value. The value may be null.
RequestParameterRule
- This rule accepts or rejects requests based on the presence of a parameter in the request.
SessionAttributeRule
- Documentation for this is exactly the same as for RequestAttributeRule
- This is a cut'n'paste error
- Actually accepts or rejects requests based on the presence of a attribute in the session
URLStringMatchRule
- This rule looks for a specific string in the URL for a positive match.
UserRoleRule
- This rule redirects the request based on the user's role.

`balancer` webapp rules file: what's missing?

A rule class which can redirect based on the number of requests sent to each member of the cluster in the past period
A rule class which can redirect based on the time each member of the cluster is currently taking to service requests
In other words, the actual ability to dynamically load balance
You could write your own rule class...
- But it's interesting these are not there
- Implies it's probably difficult to do right!

`balancer` webapp rules file: example

<?xml version="1.0" encoding="UTF-8"?>
<rules>
  <!-- If the URL contains News (case-sensitive), go to CNN.com -->
  <rule className="org.apache.webapp.balancer.rules.URLStringMatchRule"
    targetString="News"
    redirectUrl="http://www.cnn.com" />

  <!-- If the request contains a parameter named paramName whose value
       is paramValue, go to Yahoo.com. -->
  <rule className="org.apache.webapp.balancer.rules.RequestParameterRule"
    paramName="paramName"
    paramValue="paramValue"
    redirectUrl="http://www.yahoo.com" />

  <!-- Redirect all requests to jakarta.apache.org. -->
  <rule className="org.apache.webapp.balancer.rules.AcceptEverythingRule"
    redirectUrl="http://jakarta.apache.org" />
</rules>

[Break]

Performance Tuning

Tuning and measurement
Performance of individual components
Generic fixes

Tuning and Measurement

Without measurement you can't tune
You can't tell whether you're improving or not
Set up JMeter test plans, run them before and after each attempted improvement
Make sure your test plan exercises the component you are working on

Performance of individual components

A great deal of Tomcat's time is spent executing your components
Your Servlets, JSP pages, and code they depend on
Are some of your Servlets much faster than others? Why?

Profiling your components

It's pretty hard to profile a live Servlet
If you can build test harnesses for your components outside the Servlet context, do so.
Test, profile, improve.

Building test-harness Servlets

Don't be afraid to write special purpose Servlets just to exercise bits of code
- That do everything the 'real' Servlet does, except calling the bit of code you're investigating
- Or which call the component you're investigating multiple times when the 'real' servlet would call it just once
What happens to performance? Test again!

Generic fixes

Static content
DNS Lookups
Threads

Static content

Compared to Apache HTTPD, Tomcat is substantially slower serving static content
If performance is a problem, serve all your static stuff over to Apache HTTPD
- images
- stylesheets (both CSS and XSL)
- static html

DNS Lookups

DNS lookups are typically very slow (although they aren't processor intensive)
By default, calls to HttpRequest.getRemoteHost() cause a DNS lookup
You can turn this off by setting enableLookups="false" in the Connector elements in server.xml

Threads

Tomcat keeps a number of threads alive to respond to requests
If this number is too few, performance will suffer
If too many memory will be wasted
If too much memory being consumed causes the machine to swap, performance may suffer
Alter the minProcessor and maxProcessor values in in the Connector elements in server.xml
Test. Have you made an improvement?

Wrap

Questions and feedback

Final thought

Tomcat's core mission is serving dynamic content
- It does this extremely well
It does an awful lot of things which are not part of it's core mission
- Feature creep
- Does mean you can run Tomcat as a standalone Web server
- Other things (mainly Apache HTTPD) do the other stuff better
Keep Tomcat for what it does best.

Taming Tomcat, Day 3: Security and Perfomance

Programme for Today

Security on the host

File System Security

The JVM and file system security

Recommendations: users and groups

Users and groups: summary

Tomcat's own security

Realms, users and roles

We've encountered these already

Type of Realm

Configuring the Realm

Summing up on Realms

Keeping separate apps separate

The SecurityManager

The SecurityManager (ii)

Permission classes

Custom Permission classes

The policy file(s)

Policy rule syntax

Editing policy files

Debugging policy problems

How secure is security?

Recommendations

Exercise: Set up a Realm authenticated against a database

[Break]

Security on the net

Exposed ports

Exposed ports: what do you have to expose?

Secure Sockets Layer

Brief Introduction to Secure Sockets Layer

Secure Sockets Layer using Apache HTTPD

Secure Sockets Layer using Tomcat directly

Secure Sockets Layer with Tomcat directly: HOWTO

Exercise: Protect one of your Webapps with Secure Sockets Layer

Lunch

Security against failures: Load balancing and failover

Clustering Tomcats

Clustering: Quick HOWTO

Important Gotchas

Session replication

Load Balancing

Load balancing with mod_jk

Load balancing with mod_proxy and mod_rewrite

Load balancing with the balancer webapp

What the documentation doesn't say

balancer webapp rules file: Available classes

balancer webapp rules file: what's missing?

balancer webapp rules file: example

[Break]

Performance Tuning

Tuning and Measurement

Performance of individual components

Profiling your components

Building test-harness Servlets

Generic fixes

Static content

DNS Lookups

Threads

Wrap

Final thought

The `SecurityManager`

The `SecurityManager` (ii)

Load balancing with `mod_jk`

Load balancing with `mod_proxy` and `mod_rewrite`

Load balancing with the `balancer` webapp

`balancer` webapp rules file: Available classes

`balancer` webapp rules file: what's missing?

`balancer` webapp rules file: example