web.xml schema (simplified)

  <xsd:complexType name="web-appType">

    <xsd:choice minOccurs="0" maxOccurs="unbounded">
      <xsd:group ref="j2ee:descriptionGroup"/>
      <xsd:element name="distributable" type="j2ee:emptyType"/>
      <xsd:element name="context-param" type="j2ee:param-valueType">
        <xsd:annotation>
          <xsd:documentation> 
            The context-param element contains the declaration
            of a web application's servlet context
            initialization parameters.  
          </xsd:documentation>
        </xsd:annotation>
      </xsd:element>

      <xsd:element name="filter" type="j2ee:filterType"/>
      <xsd:element name="filter-mapping" type="j2ee:filter-mappingType"/>
      <xsd:element name="listener" type="j2ee:listenerType"/>
      <xsd:element name="servlet" type="j2ee:servletType"/>
      <xsd:element name="servlet-mapping" type="j2ee:servlet-mappingType"/>
      <xsd:element name="session-config" type="j2ee:session-configType"/>
      <xsd:element name="mime-mapping" type="j2ee:mime-mappingType"/>
      <xsd:element name="welcome-file-list" type="j2ee:welcome-file-listType"/>
      <xsd:element name="error-page" type="j2ee:error-pageType"/>
      <xsd:element name="jsp-config" type="j2ee:jsp-configType"/>
      <xsd:element name="security-constraint" type="j2ee:security-constraintType"/>
      <xsd:element name="login-config" type="j2ee:login-configType"/>
      <xsd:element name="security-role" type="j2ee:security-roleType"/>
      <xsd:group ref="j2ee:jndiEnvironmentRefsGroup"/>
      <xsd:element name="message-destination" type="j2ee:message-destinationType"/>
      <xsd:element name="locale-encoding-mapping-list" type="j2ee:locale-encoding-mapping-listType"/>
    </xsd:choice>

    <xsd:attribute name="version" type="j2ee:web-app-versionType" use="required"/>
    <xsd:attribute name="id" type="xsd:ID"/>
  </xsd:complexType>

Apache SSL + SVN notes

Recently I started to use Apache http server again. I am trying to build a SVN repository which can be accessed through HTTPS.

Environment

Ubuntu
Apache 2.2.17 source

Doc: http://httpd.apache.org/docs/2.2/

Build

  ./configure --prefix=/home/gerald/servers/httpd-2.2.17 --enable-ssl --enable-dav --enable-so
  make
  make install

add bin directory to your PATH

add man pages:

function addManPath() { 
    if (($# != 1)); then return 0; fi

    path="$1" 
    if [ "x$MANPATH" == "x" ]; then 
        export MANPATH="$(manpath):$path" 
    else 
        export MANPATH="${MANPATH}:$path" 
    fi  
}

addManPath "~/servers/httpd-2.2.17/bin/man"

start up apache server: apachectl start

benchmarking: ab -n 10000 -c 100 http://localhost:80/

Show modules:
  httpd -M  //show all loaded modules
  httpd -S      // show parsed virtual host settings
  httpd -l    //listed compiled in modules
  httpd -L   //list available configuration directives
  httpd -V  //show compile settings (not settings for compiling the whole package, the settings for compiling the server - httpd).

Configure SSL

Prepare your certificate and private key.
Uncomment line "Include conf/extra/httpd-ssl.conf" in httpd.conf.
Change file "conf/extra/httpd-ssl.conf". The most important directives are SSLCertificateFile and SSLCertificateKeyFile.
Test whether you can access your website through HTTPS.

SSL + SVN

Get modules dav_svn and authz_svn

wget http://altruistic.lbl.gov/mirrors/ubuntu/pool/universe/s/subversion/libapache2-svn_1.6.5dfsg-1ubuntu1_i386.deb

dpkg-deb -x libapache2-svn_1.6.5dfsg-1ubuntu1_i386.deb

copy two module (.so files) to apache modules directory.

Configure modules

Edit file <Apache>/conf/extra/dav_svn.load

    LoadModule dav_svn_module modules/mod_dav_svn.so
    LoadModule authz_svn_module modules/mod_authz_svn.so

Edit file <Apache>/conf/extra/httpd.conf, add following two lines

    Include conf/extra/dav_svn.load
    Include conf/extra/dav_svn.conf

Edit file <Apache>/conf/extra/dav_svn.conf

<Location /svn/> <!-- trailing / is necessary!! -->
  DAV svn
  
  SSLRequireSSL  # enforce use of HTTPS
  #SVNPath /var/lib/svn
  SVNParentPath /home/svn/projects
  SVNListParentPath on 
  
  AuthType Basic
  AuthName "Subversion Repository"
  AuthUserFile Apache_Dir/conf/dav_svn.passwd

  # To enable authorization via mod_authz_svn
  AuthzSVNAccessFile Apache_Dir/conf/dav_svn.authz

  Require valid-user
</Location>

http://stackoverflow.com/questions/488778/how-do-i-list-all-repositories-with-the-svnparentpath-directive-on-apachesvn

Create authentication and authorization files

Create password file: htpassword -cm <Apache>/conf/dav_svn.passwd gerald

Edit file <Apache>/conf/dav_svn.authz

[groups]
admin=gerald
guests=guest

[/]
@admin=rw

[repository_name:/directory]
@admin=rw

Test

Restart Apache httpd server.
Go to https://your_ip/svn/ (note: the trailing / is necessary!)

 

Permission Problem

If you see following error when you try to commit some code:

svn: Commit failed (details follow):
svn: Can't open file '/path/to/your/repo/db/txn-current-lock': Permission denied

follow these steps:

1. Execute command: ps -wwf $(pgrep httpd)
   You should say one of the processes is run as root. All other processes are run as daemon (in my case).
2. To make httpd able to access(read/write) your svn repository, you should set the file permissions of svn repository correctly.
   chown -R gerald:daemon /path/to/svn/repo
   chmod -R 770 /path/to/svn/repo

Replace token using ant in Maven 2

<plugin>
    <artifactid>maven-antrun-plugin</artifactid>
    <executions>
      <execution>
        <id>Copy and filter af file</id>
        <goals><goal>run</goal></goals>
        <phase>prepare-package</phase>
        <configuration>
          <tasks> 
            <copy file="source_file" filtering="true" failonerror="true" overwrite="true" tofile="dest_file">
              <filterset>
                <filter value="${variable_name}" token="token_to_be_replaced" />
                <filter value="value" token="token_to_be_replaced" />
               </filterset>
            </copy>
          </tasks>
        </configuration>
      </execution>
    </executions>
</plugin>

XAuth!

Meebo and some other supporters just released XAuth. The video on page http://xauth.org/info/ is really informative. XAuth provides front-end solution for registration of various web service sessions.

• If you are authenticated to a service, the service puts a registry entry into XAuth local storage.
• Other mashup apps/publisher websites can ask XAuth for a list of web services that the user has been authenticated to. Then the app can adjust UI according to the retrieved data.

In current reference implementation, it requires HTML5 features – postMessage and local storage.

Javascript code: http://github.com/xauth/xauth
Official web site: http://xauth.org/

Numbers Everyone Should Know

LADIS2009 keynote slides: http://www.cs.cornell.edu/projects/ladis2009/talks/dean-keynote-ladis2009.pdf

Following numbers are excerpted from the above slides.

L1 cache reference	0.5 ns
Branch mispredict	5 ns
L2 cache reference	7 ns
Mutex lock/unlock	25 ns
Main memory reference	100 ns
Compress 1K bytes with Zippy	3,000 ns
Send 2K bytes over 1 Gbps network	20,000 ns
Read 1 MB sequentially from memory	250,000 ns
Round trip within same datacenter	500,000 ns
Disk seek	10,000,000 ns
Read 1 MB sequentially from disk	20,000,000 ns

Clear memory cache in Linux

Read this article

http://linux-mm.org/Drop_Caches

Basically, run commands
    sync
    sudo echo 3 > /proc/sys/vm/drop_caches

If you want to control "swappiness" (to what extent apps are swapped), read following posts:
http://kerneltrap.org/node/3000
http://lwn.net/Articles/92892/

Basic command: sudo echo 0-100 > /proc/sys/vm/swappiness

Steve Huffman talks about lessons learned at Reddit

Very interesting talk. Helpful to those who want to build scalable distributed systems.

Video: http://vimeo.com/10506751
Transcript is here: http://carsonified.com/blog/dev/steve-huffman-on-lessons-learned-at-reddit/

One interesting lesson is lesson 3: open schema.
They combined relational data model and key-value store. Relational data model is powerful (in some sense) and can naturally represent real-world data models. However, as data set size grows, it seems that relational databases cannot scale up easily. As a result, many large companies develop their own storage systems - Amazon's Dynamo, Google's BigTable, Cassandra, etc. Some use key-value model which seems to be able to scale better than relational model. To change schema does NOT incur much overhead. Different rows/entities can have different number of columns/attributes.

Another post on caching in web app: http://www.mysqlperformanceblog.com/2010/05/19/beyond-great-cache-hit-ratio/

Create a file of a given size on linux

Let's say you want to create a file whose size is 1K. You can achieve that by using either of following two commands:

    dd if=/dev/zero of=file bs=1024 count=1

    dd of=file bs=1024 count=0 seek=1

In my test, the generated files are different. The second command generates a file with holes. You can try command stat file. The output in my machine is:

  File: `bigfile'
  Size: 1024            Blocks: 0          IO Block: 4096   regular file

The number of allocated blocks is 0 although size of the file is 1024.
Note: if sparse file is not supported, it may write 1024K zeros to the file.

As a user, you don't need to worry about whether the file is sparse or not. It will grow as needed.

Ubuntu package downgrade

I tried to use some Karmic sources for my Intrepid. Of course, this is bad. The reason I had to do it was the package of new version I needed only exist in Karmic repository.
After installing a package, suddenly I got the following error when I tried to use gvim
"gvim: error while loading shared library: libgailutil.so.18: cannot open shared object file: No such file or directory"

Obviously, the old libgail18 was removed which is needed by Intrepid. Because I used unmatched sources, the apt-get did not detect any problem.

I tried to install libgail18 using command
    sudo apt-get install libgail18
It did not work and the error message is

Package libgail18 is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
However the following packages replace it:
  libgtk2.0-0
E: Package libgail18 has no installation candidate

Finally, I figured out the cause is package libgtk2.0-0, libgtk2.0-0-common, etc. These packages are too new for Intrepid. They are for Karmic. So I tried to remove these packages using command
    sudo apt-get remove libgtk2.0-0 libgtk2.0-0-common etc
However, apt-get always gave error messages which told me those packages were needed by lots of other packages. It is obviously true. Also I tried command
    sudo apt-get install --reinstall libgtk2.0-0
It also did not work. The error message is the package cannot be found. Again the cause I think was the those installed packages were too new and did not match version of ubuntu.

It turned out that I need to use dpkg command
    dpkg --remove –-depends libgtk2.0-0 libgtk2.0-0-common etc

Then use following command to fix the broken dependencies:
    sudo apt-get install –f
libgtk2.0-0, libgtk2.0-0-common of correct versions are downloaded and installed.

Probably following commands are needed to clean up:
    sudo dpkg --configure –a
    dpkg-reconfigure

One big difficult I encountered was it was hard to collect detailed log about which files are created/updated/removed by which package during installation. It makes much easier to locate those packages which remove needed files.

SSH, XAuth and X11 Forward after user switch at remote site

Problem

User A connects to server S using ssh. X11 forward is enabled using –Y option. Then X11 should work smoothly.
User A connects to server S using ssh. X11 forward is enabled. Then user A switches to another User B (using command su B or ssh localhost –l B). After that, X11 forward won't work. The error should look like
"SSH gateway: X11 authentication failed. Error: Can't open display:" or
"Error: Can't open display:".
Readers may ask why user A does not directly connect to server S as user B given user A knows password of user B. The reason is that sometimes user B is a restricted user account so that he cannot log in remotely.

Solution

1. Run command
       echo ${DISPLAY}
   Sample result:
       localhost:11.0
2. Command: xauth list
   The output should be like:   

   your_host_name/unix:11  MIT-MAGIC-COOKIE-1  d1e63de6fd7bc3800d868c3b64ca4531
   your_host_name/unix:0  MIT-MAGIC-COOKIE-1  e044d47b672dcade1362cd632236f919
   your_host_name/unix:10  MIT-MAGIC-COOKIE-1  2aa3bc47d1c209fd06577f4b45f83383

   Pick the entry with the same display number as the output in step 1)
   In this example, display number is 11, so the entry we pick is

   your_host_name/unix:11  MIT-MAGIC-COOKIE-1  d1e63de6fd7bc3800d868c3b64ca4531

3. switch to another user using either of the following ways
   1) su user_name
   or su – user_name
   read "man su" for difference between these two commands.
   2) ssh localhost –l user_name
4. In step 3), if you ran command "su – user_name" or "ssh localhost –l user_name", you should run command
       export DISPLAY=localhost:11.0
   Value of DISPLAY should be the same as the output in step 1).
   
   add the entry obtained in step 2) to the .Xauthority file. You can either add it manually to the file or use tool xauth to do it. The way to use xauth to add an entry:
       xauth add :11 . d1e63de6fd7bc3800d868c3b64ca4531
   The cookie string (long string) must match the one in step 2). The display number (:11) must match the result in step 1)
5. Try command
       xclock

Or you can combine step 1), 2), 3) and 4) into one long command:

    (tmpfile=/tmp/xauth_tmp_entry; \
    xauth extract ${tmpfile} :$(echo $DISPLAY|cut -d : -f 2 ); \
    chmod a+r ${tmpfile}; \
    su user_name -c "xauth merge ${tmpfile}"; \
    rm ${tmpfile} )

Note: replace user_name with the real target user name.

Disadvantage

Each time the user reconnects the remote machine using ssh, the whole process described above must be redone :-( The reason is that sshd may choose another display number and cookie value.

How SSH X authorization works?

From ssh manual:

"ssh will also automatically set up Xauthority data on the server machine.  For this purpose, it
will generate a random authorization cookie, store it in Xauthority on the server, and verify
that any forwarded connections carry this cookie and replace it by the real cookie when the
connection is opened.  The real authentication cookie is never sent to the server machine (and
no cookies are sent in the plain)."

http://blogs.gnome.org/markmc/2005/02/25/ssh-x-forwarding-and-xauth/

Every time a user connects to a remote server using ssh, a proxy X server is created. And that X server is used by the sshd process (a new process is forked each time a new connection comes in). The process is like:

  user ---> server ---> fork a new process, 
                        create a proxy X server
                        create pseudo terminal, etc.
                                       |

                                       |

                                       V

                        a program that needs X is used

                                       |

                                       |

                                       V

  local display <--- verify <--- the X output is forwarded by sshd to client

It seems that after a user connects to a remote server using ssh, another proxy X server is not created when the user ssh to localhost or 127.0.0.1.