Donnerstag, 12. Dezember 2013

changing ulimits for jenkins (daemons started by start-stop-daemon)

Our build infrastructure consists of several build systems running jenkins as a CI platform. The hardware that hosts these servers is very powerful, that means large and fast disks and up to 48 cores and 128GB of RAM. Having such a hardware allows to us to run a higher number of parallel build tasks which is the cause why we have hit the 4k limit for open files (Ubuntu 12.04LTS with default settings). Using ulimit -a will show you all limit settings for the current user.

root@system:# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 1031036
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1031036
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

This output shows that the maximum number of open files (soft limit) is 1024, the hard limit can be show with ulimit -a -H giving the following output.

root@system:# ulimit -a -H
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 1031036
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 4096
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1031036
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

A user can increase its ulimits only until the hard limit has been reached by using the ulimit command. Ulimits can be specified system wide by using the /etc/security/limits.conf configuration file. However, these limits can only be applied to sessions that use pam daemon. We will come to this later...
In the following we will change the ulimits for a jenkins server in 2 steps:
  1. setting limits for user jenkins in /etc/security/limits.conf by adding the following two lines to the end of this file...
  2. jenkins soft nofile 64000
    jenkins hard nofile 64000
    
  3. setting limits for the jenkins process started by start-stop-daemon
  4. RUN_AS=jenkins
    COMMAND=start_jenkins.sh
    d_start() {
            ulimit -n 64000
            start-stop-daemon --start --quiet --background --chuid $RUN_AS --exec $COMMAND
    }
    
    The important part is that you have to specify the ulimits, e.g., for the number of open files before start-stop-daemon is called. The reason is that start-stop-daemon doesn't consider pam and hence will not find the limits which have been specified in /etc/security/limits.conf. Once the limits have been changed, one can check this by running a test job in jenkins that uses a shell as build step in which ulimit -a is executed...

Resources:

  • http://superuser.com/questions/454465/make-ulimits-work-with-start-stop-daemon
  • http://www.windowslinuxosx.com/q/answers-values-set-in-limits-conf-are-not-working-678783.html
  • http://posidev.com/blog/2009/06/04/set-ulimit-parameters-on-ubuntu/
  • http://serverfault.com/questions/472904/how-to-diagnose-ulimit-enforcement
  • http://www.ovirt.org/Jenkins#change_open_files
  • http://nishal-tech.blogspot.de/2013/07/how-to-set-ulimit-in-ubuntudebian-linux.html
  • http://www.altj.com/tag/ubuntu/

1 Kommentar:

  1. Danke für die Erklärung! Jetzy weiß ich wie das gemacht wird. Beiläufig gesagt, entdecke gerade Due Diligence Datenraum

    AntwortenLöschen