-
Notifications
You must be signed in to change notification settings - Fork 324
Fix Synapse passthru spans finishing before status code is available #10478
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
ValentinZakharov
wants to merge
4
commits into
master
Choose a base branch
from
vzakharov/synapse_passthru_fix
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+44
−13
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 62 metrics, 9 unstable metrics. Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.59.0-SNAPSHOT~7bdce96bb5, baseline=1.59.0-SNAPSHOT~02cc48359b
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.056 s) : 0, 1055905
Total [baseline] (10.923 s) : 0, 10923158
Agent [candidate] (1.059 s) : 0, 1058772
Total [candidate] (10.925 s) : 0, 10924978
section appsec
Agent [baseline] (1.23 s) : 0, 1229930
Total [baseline] (10.944 s) : 0, 10944242
Agent [candidate] (1.238 s) : 0, 1237873
Total [candidate] (10.946 s) : 0, 10946295
section iast
Agent [baseline] (1.226 s) : 0, 1226222
Total [baseline] (11.196 s) : 0, 11195816
Agent [candidate] (1.227 s) : 0, 1227055
Total [candidate] (11.182 s) : 0, 11181847
section profiling
Agent [baseline] (1.188 s) : 0, 1188379
Total [baseline] (10.909 s) : 0, 10908509
Agent [candidate] (1.19 s) : 0, 1190431
Total [candidate] (10.941 s) : 0, 10940665
gantt
title petclinic - break down per module: candidate=1.59.0-SNAPSHOT~7bdce96bb5, baseline=1.59.0-SNAPSHOT~02cc48359b
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.171 ms) : 0, 1171
crashtracking [candidate] (1.17 ms) : 0, 1170
BytebuddyAgent [baseline] (623.125 ms) : 0, 623125
BytebuddyAgent [candidate] (622.164 ms) : 0, 622164
AgentMeter [baseline] (28.662 ms) : 0, 28662
AgentMeter [candidate] (28.706 ms) : 0, 28706
GlobalTracer [baseline] (256.623 ms) : 0, 256623
GlobalTracer [candidate] (256.938 ms) : 0, 256938
AppSec [baseline] (32.871 ms) : 0, 32871
AppSec [candidate] (32.775 ms) : 0, 32775
Debugger [baseline] (62.204 ms) : 0, 62204
Debugger [candidate] (60.418 ms) : 0, 60418
Remote Config [baseline] (613.589 µs) : 0, 614
Remote Config [candidate] (622.767 µs) : 0, 623
Telemetry [baseline] (9.099 ms) : 0, 9099
Telemetry [candidate] (10.697 ms) : 0, 10697
Flare Poller [baseline] (6.18 ms) : 0, 6180
Flare Poller [candidate] (9.889 ms) : 0, 9889
section appsec
crashtracking [baseline] (1.179 ms) : 0, 1179
crashtracking [candidate] (1.184 ms) : 0, 1184
BytebuddyAgent [baseline] (651.144 ms) : 0, 651144
BytebuddyAgent [candidate] (656.083 ms) : 0, 656083
AgentMeter [baseline] (11.807 ms) : 0, 11807
AgentMeter [candidate] (11.901 ms) : 0, 11901
GlobalTracer [baseline] (257.272 ms) : 0, 257272
GlobalTracer [candidate] (258.824 ms) : 0, 258824
AppSec [baseline] (166.994 ms) : 0, 166994
AppSec [candidate] (167.837 ms) : 0, 167837
Debugger [baseline] (67.492 ms) : 0, 67492
Debugger [candidate] (67.364 ms) : 0, 67364
Remote Config [baseline] (658.736 µs) : 0, 659
Remote Config [candidate] (676.808 µs) : 0, 677
Telemetry [baseline] (9.189 ms) : 0, 9189
Telemetry [candidate] (9.291 ms) : 0, 9291
Flare Poller [baseline] (3.616 ms) : 0, 3616
Flare Poller [candidate] (3.616 ms) : 0, 3616
IAST [baseline] (25.363 ms) : 0, 25363
IAST [candidate] (25.722 ms) : 0, 25722
section iast
crashtracking [baseline] (1.171 ms) : 0, 1171
crashtracking [candidate] (1.188 ms) : 0, 1188
BytebuddyAgent [baseline] (789.885 ms) : 0, 789885
BytebuddyAgent [candidate] (790.222 ms) : 0, 790222
AgentMeter [baseline] (11.187 ms) : 0, 11187
AgentMeter [candidate] (11.193 ms) : 0, 11193
GlobalTracer [baseline] (248.378 ms) : 0, 248378
GlobalTracer [candidate] (248.2 ms) : 0, 248200
AppSec [baseline] (32.12 ms) : 0, 32120
AppSec [candidate] (32.993 ms) : 0, 32993
Debugger [baseline] (68.706 ms) : 0, 68706
Debugger [candidate] (68.134 ms) : 0, 68134
Remote Config [baseline] (543.356 µs) : 0, 543
Remote Config [candidate] (552.888 µs) : 0, 553
Telemetry [baseline] (8.528 ms) : 0, 8528
Telemetry [candidate] (8.636 ms) : 0, 8636
Flare Poller [baseline] (3.535 ms) : 0, 3535
Flare Poller [candidate] (3.549 ms) : 0, 3549
IAST [baseline] (26.892 ms) : 0, 26892
IAST [candidate] (27.041 ms) : 0, 27041
section profiling
ProfilingAgent [baseline] (98.996 ms) : 0, 98996
ProfilingAgent [candidate] (99.673 ms) : 0, 99673
crashtracking [baseline] (1.214 ms) : 0, 1214
crashtracking [candidate] (1.218 ms) : 0, 1218
BytebuddyAgent [baseline] (679.932 ms) : 0, 679932
BytebuddyAgent [candidate] (680.037 ms) : 0, 680037
AgentMeter [baseline] (8.653 ms) : 0, 8653
AgentMeter [candidate] (8.642 ms) : 0, 8642
GlobalTracer [baseline] (216.613 ms) : 0, 216613
GlobalTracer [candidate] (217.393 ms) : 0, 217393
AppSec [baseline] (32.372 ms) : 0, 32372
AppSec [candidate] (32.471 ms) : 0, 32471
Debugger [baseline] (67.473 ms) : 0, 67473
Debugger [candidate] (67.821 ms) : 0, 67821
Remote Config [baseline] (591.311 µs) : 0, 591
Remote Config [candidate] (594.843 µs) : 0, 595
Telemetry [baseline] (8.674 ms) : 0, 8674
Telemetry [candidate] (8.666 ms) : 0, 8666
Flare Poller [baseline] (3.787 ms) : 0, 3787
Flare Poller [candidate] (3.754 ms) : 0, 3754
Profiling [baseline] (99.562 ms) : 0, 99562
Profiling [candidate] (100.264 ms) : 0, 100264
Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.59.0-SNAPSHOT~7bdce96bb5, baseline=1.59.0-SNAPSHOT~02cc48359b
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.056 s) : 0, 1055607
Total [baseline] (8.745 s) : 0, 8744692
Agent [candidate] (1.06 s) : 0, 1060059
Total [candidate] (8.728 s) : 0, 8727535
section iast
Agent [baseline] (1.223 s) : 0, 1222887
Total [baseline] (9.404 s) : 0, 9403867
Agent [candidate] (1.225 s) : 0, 1225097
Total [candidate] (9.399 s) : 0, 9399452
gantt
title insecure-bank - break down per module: candidate=1.59.0-SNAPSHOT~7bdce96bb5, baseline=1.59.0-SNAPSHOT~02cc48359b
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.175 ms) : 0, 1175
crashtracking [candidate] (1.185 ms) : 0, 1185
BytebuddyAgent [baseline] (622.009 ms) : 0, 622009
BytebuddyAgent [candidate] (625.27 ms) : 0, 625270
AgentMeter [baseline] (28.563 ms) : 0, 28563
AgentMeter [candidate] (28.791 ms) : 0, 28791
GlobalTracer [baseline] (256.882 ms) : 0, 256882
GlobalTracer [candidate] (258.225 ms) : 0, 258225
AppSec [baseline] (32.771 ms) : 0, 32771
AppSec [candidate] (32.779 ms) : 0, 32779
Debugger [baseline] (60.197 ms) : 0, 60197
Debugger [candidate] (60.881 ms) : 0, 60881
Remote Config [baseline] (607.69 µs) : 0, 608
Remote Config [candidate] (606.067 µs) : 0, 606
Telemetry [baseline] (9.781 ms) : 0, 9781
Telemetry [candidate] (9.112 ms) : 0, 9112
Flare Poller [baseline] (8.313 ms) : 0, 8313
Flare Poller [candidate] (7.663 ms) : 0, 7663
section iast
crashtracking [baseline] (1.179 ms) : 0, 1179
crashtracking [candidate] (1.18 ms) : 0, 1180
BytebuddyAgent [baseline] (788.77 ms) : 0, 788770
BytebuddyAgent [candidate] (790.123 ms) : 0, 790123
AgentMeter [baseline] (11.192 ms) : 0, 11192
AgentMeter [candidate] (11.153 ms) : 0, 11153
GlobalTracer [baseline] (247.78 ms) : 0, 247780
GlobalTracer [candidate] (247.989 ms) : 0, 247989
AppSec [baseline] (32.811 ms) : 0, 32811
AppSec [candidate] (33.331 ms) : 0, 33331
Debugger [baseline] (66.626 ms) : 0, 66626
Debugger [candidate] (66.698 ms) : 0, 66698
Remote Config [baseline] (541.651 µs) : 0, 542
Remote Config [candidate] (543.439 µs) : 0, 543
Telemetry [baseline] (8.451 ms) : 0, 8451
Telemetry [candidate] (8.453 ms) : 0, 8453
Flare Poller [baseline] (3.457 ms) : 0, 3457
Flare Poller [candidate] (3.473 ms) : 0, 3473
IAST [baseline] (26.881 ms) : 0, 26881
IAST [candidate] (26.798 ms) : 0, 26798
LoadParameters
See matching parameters
SummaryFound 1 performance improvements and 2 performance regressions! Performance is the same for 16 metrics, 17 unstable metrics.
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.59.0-SNAPSHOT~7bdce96bb5, baseline=1.59.0-SNAPSHOT~02cc48359b
dateFormat X
axisFormat %s
section baseline
no_agent (1.208 ms) : 1196, 1220
. : milestone, 1208,
iast (3.128 ms) : 3089, 3166
. : milestone, 3128,
iast_FULL (5.912 ms) : 5852, 5973
. : milestone, 5912,
iast_GLOBAL (3.497 ms) : 3436, 3558
. : milestone, 3497,
profiling (2.348 ms) : 2323, 2373
. : milestone, 2348,
tracing (1.796 ms) : 1781, 1812
. : milestone, 1796,
section candidate
no_agent (1.181 ms) : 1169, 1193
. : milestone, 1181,
iast (3.174 ms) : 3127, 3222
. : milestone, 3174,
iast_FULL (5.97 ms) : 5909, 6030
. : milestone, 5970,
iast_GLOBAL (3.602 ms) : 3544, 3660
. : milestone, 3602,
profiling (2.157 ms) : 2138, 2177
. : milestone, 2157,
tracing (1.885 ms) : 1869, 1902
. : milestone, 1885,
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.59.0-SNAPSHOT~7bdce96bb5, baseline=1.59.0-SNAPSHOT~02cc48359b
dateFormat X
axisFormat %s
section baseline
no_agent (17.853 ms) : 17669, 18036
. : milestone, 17853,
appsec (18.664 ms) : 18473, 18855
. : milestone, 18664,
code_origins (17.895 ms) : 17717, 18073
. : milestone, 17895,
iast (17.655 ms) : 17481, 17829
. : milestone, 17655,
profiling (18.793 ms) : 18602, 18985
. : milestone, 18793,
tracing (17.457 ms) : 17284, 17631
. : milestone, 17457,
section candidate
no_agent (18.433 ms) : 18240, 18627
. : milestone, 18433,
appsec (19.502 ms) : 19308, 19695
. : milestone, 19502,
code_origins (17.435 ms) : 17263, 17607
. : milestone, 17435,
iast (17.791 ms) : 17612, 17970
. : milestone, 17791,
profiling (18.8 ms) : 18609, 18990
. : milestone, 18800,
tracing (17.531 ms) : 17358, 17704
. : milestone, 17531,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics. Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.59.0-SNAPSHOT~7bdce96bb5, baseline=1.59.0-SNAPSHOT~02cc48359b
dateFormat X
axisFormat %s
section baseline
no_agent (1.482 ms) : 1470, 1494
. : milestone, 1482,
appsec (2.498 ms) : 2444, 2552
. : milestone, 2498,
iast (2.247 ms) : 2178, 2316
. : milestone, 2247,
iast_GLOBAL (2.286 ms) : 2217, 2355
. : milestone, 2286,
profiling (2.074 ms) : 2020, 2129
. : milestone, 2074,
tracing (2.061 ms) : 2008, 2114
. : milestone, 2061,
section candidate
no_agent (1.476 ms) : 1465, 1488
. : milestone, 1476,
appsec (3.656 ms) : 3438, 3873
. : milestone, 3656,
iast (2.257 ms) : 2188, 2326
. : milestone, 2257,
iast_GLOBAL (2.307 ms) : 2238, 2377
. : milestone, 2307,
profiling (2.094 ms) : 2039, 2149
. : milestone, 2094,
tracing (2.079 ms) : 2025, 2133
. : milestone, 2079,
Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.59.0-SNAPSHOT~7bdce96bb5, baseline=1.59.0-SNAPSHOT~02cc48359b
dateFormat X
axisFormat %s
section baseline
no_agent (14.695 s) : 14695000, 14695000
. : milestone, 14695000,
appsec (14.981 s) : 14981000, 14981000
. : milestone, 14981000,
iast (18.058 s) : 18058000, 18058000
. : milestone, 18058000,
iast_GLOBAL (17.746 s) : 17746000, 17746000
. : milestone, 17746000,
profiling (15.129 s) : 15129000, 15129000
. : milestone, 15129000,
tracing (15.086 s) : 15086000, 15086000
. : milestone, 15086000,
section candidate
no_agent (15.073 s) : 15073000, 15073000
. : milestone, 15073000,
appsec (15.105 s) : 15105000, 15105000
. : milestone, 15105000,
iast (18.015 s) : 18015000, 18015000
. : milestone, 18015000,
iast_GLOBAL (17.703 s) : 17703000, 17703000
. : milestone, 17703000,
profiling (14.995 s) : 14995000, 14995000
. : milestone, 14995000,
tracing (14.738 s) : 14738000, 14738000
. : milestone, 14738000,
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
inst: synapse
Synapse Instrumentation
tag: flaky test
Flaky tests
type: enhancement
Enhancements and improvements
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What Does This Do
Fixed flaky Synapse passthru tracing by changing when we finish the HTTP span.
responseReadyandresponseReceivedmay be called before theHttpResponse(and status code) is available.Instead of removing the stored context and finish span of the first callback, we keep the context until we see the real response status (or error), then finish the span once.
Motivation
Test
test passthru request is tracedwas sometimes failing with, because the span was finishing too early, before status code was knownAdditional Notes
With the passthru/NIO transport,
SourceHandler.responseReady()andTargetHandler.responseReceived()can be called multiple times for the same connectionThe first call can happen while the response is still null, and the actual 200 OK is set later.
Previously we removed the context and finished the span on that first call, so the later callback couldn’t tag the span
Contributor Checklist
type:and (comp:orinst:) labels in addition to any useful labelsclose,fixor any linking keywords when referencing an issue.Use
solvesinstead, and assign the PR milestone to the issueJira ticket: [PROJ-IDENT]