[JENKINS-75871] Very poor performance when source code is retained with a coverage report

[jenkins-infra-bot]

As the JaCoCo plugin I used so far is marked as deprecated I switched to the Coverage plugin. And it seems to work fine. At least for some smaller projects. But for a larger one the build time went up from 11 minutes to 4h 30m! But only if I include the source code. If I set "Source Code Retention Strategy" to "Never store source code" I am back to 11 or 12 minutes. Any thoughts on that?

---

Originally reported by htappe, imported from: Very poor performance when source code is retained with a coverage report

• status: Open
• priority: Major
• component(s): coverage-plugin
• resolution: Unresolved
• votes: 0
• watchers: 4
• imported: 20260107-220949

Raw content of original issue

As the JaCoCo plugin I used so far is marked as deprecated I switched to the Coverage plugin. And it seems to work fine. At least for some smaller projects. But for a larger one the build time went up from 11 minutes to 4h 30m! But only if I include the source code. If I set "Source Code Retention Strategy" to "Never store source code" I am back to 11 or 12 minutes. Any thoughts on that?

• environment: Jenkins 2.504.3

5 attachments

• 10000.png

  

• 2000.png

  

• coverage-log-final.txt
• coverage-log-start.txt
• jacoco-log.txt

[jenkins-infra-bot]

drulli:

• Original comment link
• Raw content of original comment:

  How many files does your project have? Can you add the logging output of the plugin? How is the agent connected to the controller? How fast is the rendering of the jacoco plugin for the same project?
  

How many files does your project have? Can you add the logging output of the plugin? How is the agent connected to the controller? How fast is the rendering of the jacoco plugin for the same project?

[jenkins-infra-bot]

htappe:

• Original comment link
• Raw content of original comment:

  Great! What a quick response! 
  
  The project roughly has 35.000 java files. I know, that's lot. Many of them are generated! 
  
  I just restarted the project with the strategy "Store source code of last build only" and uploaded the log output so far. To get the rest of the log I have to wait a couple of hours. In the meantime I will provide the other information you were asking for...
  
   
  

Great! What a quick response!

The project roughly has 35.000 java files. I know, that's lot. Many of them are generated!

I just restarted the project with the strategy "Store source code of last build only" and uploaded the log output so far. To get the rest of the log I have to wait a couple of hours. In the meantime I will provide the other information you were asking for...

 

[jenkins-infra-bot]

htappe:

• Original comment link
• Raw content of original comment:

  As for the time of the jacoco plugin: I can't tell exactly from the log. Maybe 1 or 2 minutes!? Without the plugin the build usually lasts for 11 minutes. Including the plugin it takes about 12 or 13 minutes.
  

As for the time of the jacoco plugin: I can't tell exactly from the log. Maybe 1 or 2 minutes!? Without the plugin the build usually lasts for 11 minutes. Including the plugin it takes about 12 or 13 minutes.

[jenkins-infra-bot]

htappe:

• Original comment link
• Raw content of original comment:

  As for the agent/controller connection: I am not sure I understand you correctly. The build runs directly on the machine Jenkins runs on. So no SSH or anything involved.
  

As for the agent/controller connection: I am not sure I understand you correctly. The build runs directly on the machine Jenkins runs on. So no SSH or anything involved.

[jenkins-infra-bot]

drulli:

• Original comment link
• Raw content of original comment:

  Ok, thanks. Does your log also show timestamps, to see which part actually takes so long? 
  
  When your build runs on the same machine, then no overhead is involved. So this cannot be a problem. Are the sources stored in a slow filesystem (filelinks, NFS, ?)
  

Ok, thanks. Does your log also show timestamps, to see which part actually takes so long?

When your build runs on the same machine, then no overhead is involved. So this cannot be a problem. Are the sources stored in a slow filesystem (filelinks, NFS, ?)

[jenkins-infra-bot]

htappe:

• Original comment link
• Raw content of original comment:

  Unfortunately there are only very few timestamps in the log. But almost all time is spent between the two parts of the log I uploaded. So the last message of the first part of the log is where the build does not "visually" proceed for 4.5 hours: "Making paths of 30299 source code files relative to workspace root..."
  

Unfortunately there are only very few timestamps in the log. But almost all time is spent between the two parts of the log I uploaded. So the last message of the first part of the log is where the build does not "visually" proceed for 4.5 hours: "Making paths of 30299 source code files relative to workspace root..."

[jenkins-infra-bot]

mcmics:

• Original comment link
• Raw content of original comment:

  Same happens to me. I have about 10000 files.
  Agent is connected via SSH.
  
  Ourtime for the stage  increased from ~8 Minutes to ~40 Minutes 
  
  My scenario was a little different. I update also to newer Jenkins LTS and also used JDK 21 as Runtime for master now.
  
  I can try to create  a sample project if it's help.
  

Same happens to me. I have about 10000 files.
Agent is connected via SSH.

Ourtime for the stage increased from ~8 Minutes to ~40 Minutes

My scenario was a little different. I update also to newer Jenkins LTS and also used JDK 21 as Runtime for master now.

I can try to create a sample project if it's help.

[jenkins-infra-bot]

mcmics:

• Original comment link
• Raw content of original comment:

  I have created a small sample where the files are created(currently set to 10.000). Can be changed on annotation in de.mcmics.sample.NameProvider
  Sample: https://github.com/MCMicS/GenerateJavaSources
  
  The log says that only the non-genrated sources are mapped but it still takes some time. maybe an operation after the make absolute paths.
  
  [Coverage] [-ERROR-]   ... skipped logging of 9980 additional errors ...
  [Coverage] Resolving source code files...
  [Coverage] Searching for source code files in:
  [Coverage] -> project/src/main/java
  [Coverage] -> /home/jenkins/agent/workspace/JENKINS-75871_main/src/main/java
  [Coverage] -> finished resolving of absolute paths (found: 2, not found: 10000)
  [Coverage] Making paths of 2 source code files relative to workspace root...
  
  
  it takes about 17 minutes and with disabled {{sourceCodeRetention}}it takes only 3 minutes
  also created some branches with different sizes. WIth 2000 you will see the first big difference ( 1min vs 30 sec)
  2000:   
  10000:   
  I hope it helps. I will also have a lookt into this on weekend
  

I have created a small sample where the files are created(currently set to 10.000). Can be changed on annotation in de.mcmics.sample.NameProvider
Sample: https://github.com/MCMicS/GenerateJavaSources

The log says that only the non-genrated sources are mapped but it still takes some time. maybe an operation after the make absolute paths.

[Coverage] [-ERROR-]   ... skipped logging of 9980 additional errors ...
[Coverage] Resolving source code files...
[Coverage] Searching for source code files in:
[Coverage] -> project/src/main/java
[Coverage] -> /home/jenkins/agent/workspace/JENKINS-75871_main/src/main/java
[Coverage] -> finished resolving of absolute paths (found: 2, not found: 10000)
[Coverage] Making paths of 2 source code files relative to workspace root...

it takes about 17 minutes and with disabled {{sourceCodeRetention}}it takes only 3 minutes

also created some branches with different sizes. WIth 2000 you will see the first big difference ( 1min vs 30 sec)
2000:
10000:

I hope it helps. I will also have a lookt into this on weekend

[jenkins-infra-bot]

drulli:

• Original comment link
• Raw content of original comment:

  Thanks for creating the sample project. I'm not sure when I will find time into look into this problem...
  

Thanks for creating the sample project. I'm not sure when I will find time into look into this problem...

[jenkins-infra-bot]

mcmics:

• Original comment link
• Raw content of original comment:

  the reason is here: 
  
    
  
      
  
        coverage-plugin/plugin/src/main/java/io/jenkins/plugins/coverage/metrics/source/SourceCodePainter.java
      
      
  
           Line 78
        in
        d35441e
      
    
    
  
      
  
  
          
  
            
  
            
   .map(f -> createFileModel(rootNode, f)) 
          
      
    
  
  
  
  it takes a lot time to paint the sources. I will have a further look later the week
  

the reason is here:

coverage-plugin/plugin/src/main/java/io/jenkins/plugins/coverage/metrics/source/SourceCodePainter.java

Line 78 in d35441e

.map(f -> createFileModel(rootNode, f))

it takes a lot time to paint the sources. I will have a further look later the week

[jenkins-infra-bot]

drulli:

• Original comment link
• Raw content of original comment:

  Thanks for looking into it. I never used the code for large projects yet. So I am sure there is plenty of room for improvement. 
  
  If the rendering code itself cannot be improved much a different approach would be to change the rendering to an on-demand rendering. I.e., during the build step only the sources are saved and only when a file is accessed then it is rendered on demand. 
  
  The rendering is also a part of the code that should be changed in general, see JENKINS-75796. The current implementation is taken from the former code-coverage plugin without much changes.
  

Thanks for looking into it. I never used the code for large projects yet. So I am sure there is plenty of room for improvement.

If the rendering code itself cannot be improved much a different approach would be to change the rendering to an on-demand rendering. I.e., during the build step only the sources are saved and only when a file is accessed then it is rendered on demand.

The rendering is also a part of the code that should be changed in general, see JENKINS-75796">JENKINS-75796. The current implementation is taken from the former code-coverage plugin without much changes.