How to deal with Jenkins performance issues ?

by
How to deal with Jenkins performance issues ?

This article describe details about how you can make stable CI/CD infrastructure with high availability of Jenkins server and resolve stability issues for your Jenkins production application .

Problem around performance issues for Jenkins master setup :

As there are 1000+ jobs are added into production Jenkins configuration you may observe performance issues for Jenkins .

Some of issues are related to memory config and JAVA_OPTS settings . It is difficult to trace the exact problem to the production Jenkins server. Here is a step-by-step guide on how you fix performance or slowness issues for a production Jenkins server and make your deployment go smoothly without any issues.

Sometimes you may observe slowness issues , your application will stuck and you are not able to do anything . Also inside logs there are no any failure logs and only restart require to solve this problem , after restart this are working fine but there will be similar issues observe again and again . Its unexpected behaviour and difficult to debug .

In this situation use below steps to troubleshoot problem .

How to deal with Jenkins performance issues

Read Also-What are the Interview Questions for java ?

You need to check and analyse performance by implementing below config :

  • Monitor memory usage : Check and monitor RAM memory usage continuously for Jenkins master / slave node .
  • Monitor CPU performance : Check and monitor CPU usage for Jenkins slave & master node .
  • Check java memory leak : Check if there is any memory leak .
  • Install recommended jdk version : You should use correct and recommended jdk version suggested by Jenkins official documentation .
  • Add correct java opts parameters which are suggested by Jenkins official documents .
  • Add correct memory config and limits for Jenkins deployment in kubernetes .
  • Monitor with correct plugin, monitoring plugin will help you to monitor running setup with live scaled data . Install monitoring plugin and monitor Jenkins memory usage .
  • Add monitoring alerts for deadlock , threads , memory , active sessions . You can add monitoring alerts to capture threshold base details .
  • Reduce IO by twinning correct settings . Check if you have configured correct settings to reduce IO . If not correct it and monitor IO of jenkins master node closely .
  • Use recommend NFS setup . Make sure that you are using correct NFS version for your production setup . Also if the number of jobs are maximum your should use SSD storage for NFS config to reduce IO related issues.
  • Add correct cleanup config to delete old jobs from master node. For each job and build jenkins is storing data into jenkins master server , if number of jobs are large this data is getting increase rapidly . Add cleanup script to clean data continuously for old data , and keep minimum data on master server to avoid performance issues .
  • Avoid to run jobs on master , user master slave config . Ideally each jobs should run on slave . Also make sure you are executing jobs on slave to have minimum load on master node .
  • Avoid to install unwanted plugins , check before installing any plugin it should be with security compliance and not having security vulnerabilities . Install only supported plugin few plugins have memory leak issues it will create problem to your production Jenkins server . Its also difficult to debug about exact which plugin is creating memory leak issues. We have to make sure that we are installing correct plugins which are secure or you can test plugins on test server first and then only installing it in production Jenkins .
  • You can also free cache memory , where your application is running .If RAM usage is high for your application . If it’s docker image free cache on server where docker is running . There will be no any impact on application if you have free cache memory .

Recommendation about maintaining production Jenkins setup :

1.Add real time alerts for memory or CPU spike .

2.Install monitoring plugin .

3.Keep Jenkins version always updated with latest LTS version .

4.You can also create replica of your production ci/cd jenkins setup which can be use if primary setup goes down . Parallel setup with same config help more to have high available setup where builds will work all time .

Hope you like this blog….
Chat on WhatsApp
Mahesh Wabale
Latest posts by Mahesh Wabale (see all)

Leave a Comment